9:00 - 9:45 (CET)
Technology | ReDem
Quality control – you have been scammed. We need to check your data!
In this way, one or the other end-user of online survey data could perhaps be persuaded to question the quality of their data. After all, cost and speed are still at the forefront of choosing a provider. The euphoria of conducting online surveys quickly for little money has led to an enormous price competition. Hardly anyone demands transparent quality controls or doubts the reliability of the results achieved in this way. At the same time, the degree of professionalization of survey fraud has increased sharply. In the past this was primarily done in low-wage countries in a setting organized like a call center –
the so-called click farms. This amateurish click-through of surveys was easy to identify. Nowadays, phone farming is the challenge in the fight against fraud. Here, professionals operate decentral from home via a network of locally connected smartphones. Through the use of modern technologies such as AI, bots, botnets and VPNs, they automatically produce a large number of fake interviews, which are not conspicuous by their digital fingerprint or by speeding, straightlining or nonsense answers. The verification of identity by SMS or voice is also handled automatically via Internet services. This talk illustrates how AI technology can be used to protect online surveys from these new professional forms of survey fraud.
Sebastian Berger holds a PhD from the Chair of Marketing at the University of Vienna. With the Spotlight Viewer, he has developed the first online eye-tracking method that does not require cameras. Sebastian was a professor of marketing and market research at the University of Applied Sciences Krems. During this time, he has published numerous articles on advertising effectiveness research. With the Constantinus Award, Sebastian was awarded Austria’s most important prize for successful management consulting. Last year, he joined the research tech company ReDem and has been responsible for building and expanding the business ever since.
9:45 - 10:30 (CET)
Academic Research | University of Vienna
Some suggestions to safeguard the data quality of online surveys
The quality of data collected using communication heavily relies on respondents’ ability and willingness to provide the requested information when answering a questionnaire. Marketing research practice frequently finds that a substantial part of respondents has other goals than truthfully filling out a survey. In particular, when using online data administration, the phenomenon of satisficing is prevalent, i.e. rather than diligently striving for accurate responses, responses are provided quickly. We will represent several methods safeguarding against satisficing, thereby distinguishing between direct and indirect measures and measurements. Furthermore, we will consider how to safeguard against method effects, i.e. associations between variables might be biased because these variables have been measured by the same instrument, the questionnaire.
Udo Wagner is a full professor at the School of International Management, Modul University of Vienna, and Professor Emeritus at the University of Vienna, Austria. His research focuses on market research, consumer behavior, marketing models, retail marketing and pricing. His articles have appeared in academic journals such as the Journal of Marketing Research, Marketing Science, International Journal of Research in Marketing, Journal of Business Research, and Psychology & Marketing.
10:30 -11:15 (CET)
Global Data Quality Initiative – 1/2 | GDQ
Online Data Quality – Joining up the dots
The online data collection and research community has been hit hard over the last couple of years with an influx of fraudulent participants. The industry under the Global Data Quality initiative with leadership from the likes of MRS, Insights Association, ESOMAR have come together to address ongoing and emerging risks to data quality in the market and social research, consumer insights and analytics industry with the goal of increasing information and building trust.
MRS is overseeing work streams focussed on fraud detection. One key area is to understand some aspects of “poor data” and MRS have been conducting an in-depth audit of a badly hit survey. The results so far have indicated the need for additional views on the data and more transparency on the data flowing into the survey.
The presentation will cover
- Results from the audit – including what additional data points highlighted the issues
- Sample ecosystem
- Information that can help improve the quality of data
- Potential impact on the research process
Chris Stevens brings 30+ years’ experience with expertise in panels, operational process, research methods and what impacts on quality. In prior roles such as Head of Quality at Kantar Profiles, Research Director at TNS, Chris focussed on identifying, analysing and implementing process around quality. He continues to support industry quality initiatives such as the MRS Mobile Optimization project, ESOMAR’s recommendation for standardisation of demographics and is a contributor to the MRS program of activities that is part of the Global Data Quality (GDQ) initiative.
11:15 - 12:00 (CET)
Global Data Quality Initiative - 2/2 | Market Research Society
Meeting the Data Quality Challenge
Since the launch of the Global Data Quality initiative in April 2023, MRS has been leading a programme of activities addressing the impact of fraud and technology on data quality and integrity.
Debrah Harding will present a review of the findings from the project, the outputs that have been developed so far, including new MRS guidance on practical approaches to addressing data quality, and the plans for the year ahead.
Debrah Harding is Managing Director of the Market Research Society (MRS), Chair of the Global Research Business Network (GRBN) and Vice-President of the European Research Federation, EFAMRO. Debrah leads the MRS’ contribution to the Global Data Quality (GDQ) initiative and is an expert on research standards, legislation and ethics.
13:00 - 13:45 (CET)
Institutional research | GfK / NIQ
Always one step ahead of fraudsters - how to automatically combat bad quality in online surveys
A major threat for online surveys are click-farms that systematically attack surveys on a large scale. With millions of interviews that are conducted by GfK, it is necessary to have scalable solutions that identify click-farms in a standardized and automated way.
Besides standard checks at the survey entry like captchas, de-duplication or geo-fencing we introduced automated quality field reports that focus on additional technical flags as well as open-ends, outlier and grid analyses that reliable identify click-farm patterns. With these additional reports we are able to provide high-level and standardized data quality across our products and services to ensure clients’ trust in our data.
Oliver Hülser joined GfK’s Marketing and Data Science department more than 20 years ago and has been leading the global Data Quality Science team since 2018. The goal of this team is to ensure high and standardized quality across all GfK/NIQ products and services in the ad-hoc and panel world in order to ensure trust in our data. As early as 2002, he started developing the first data quality tool.
13:45 - 14:30 (CET)
Online Access Panels | horizoom
Enhancing Data Quality: AI in Online Access Panels
Generative AI models have quickly found their way into the market research landscape along the entire value chain. For online access panels, AI applications have an impact on data collection, synthetic data and data cleansing. Supported by empirical research results, we show how we assess quality on a panelist level by developing a unique panelist quality score in conjunction with ReDem in order to continuously and systematically improve data quality in online-surveys.
As Chief Innovation Officer at horizoom, Malte Friedrich-Freksa drives the development of data-driven market research solutions. Previously, he worked for eight years in management positions at GapFish GmbH and before that as a market researcher for various agencies and institutes in the field of digital research. He has been a speaker for the BVM since 2012 and gives presentations at national and international conferences.
14:30 -15:15 (CET)
User Perspective | Procter & Gamble
Survey Fraud: The Implications of Cheaters and Repeaters in Online Research
Online surveying has become the universal way to conduct quantitative market research. Not only does the internet offer researchers a cost-efficient way to collect data and insights from a large and diverse audience. Researchers now have access to DIY surveying platforms and suppliers are guaranteeing data delivery in less than 24 hours! What business isn’t looking for ways to generate faster insights these days? As suppliers moved to online surveying, they instituted changes in sample management. Many stopped validating sample, reduced incentive payments, began using routers, and dropped past participation guidelines. The panelist survey experience declined, and fraud started to surface. Researchers rely on genuine respondents who are interested in offering thoughtful answers to our survey questions. Unfortunately, fraud coming from an increasing number of non-validated, real time sourcing methods is having a negative impact on our insights and business decisions. Join us to learn about the issues impacting the industry’s ability to ensure valid and reliable research findings and the steps researchers can take to ensure data quality.
Tia is an experienced product researcher with over 25 years of experience in consumer centric research and product development at the Procter & Gamble Company. She has designed, developed, and piloted many emerging consumer research methodologies to fuel product innovation for the company’s Home Care and Beauty organizations. She is currently working in Surface Care on the Mr. Clean and Swiffer brands. She has a passion for data quality, and as a member of CASE 4 Quality, she worked on a groundbreaking industry study related to measuring and combatting fraud in market research. In her spare time, she is a ballroom dance instructor and serves as a trustee at the Miami Valley Veterans Museum.
15:15 - 16:15 (CET)
Panel Discussion
Online surveys: Is there life in the old dog yet?
The euphoria of conducting online surveys quickly, easily and inexpensively is fading against the backdrop of a declining willingness to participate, bored participants and the increase in the extent and degree of professionalization of survey fraud. The reliability of the results obtained is increasingly being questioned. Are online surveys here to stay? Or will they be replaced in the future by synthetic data or other means? What are the challenges for online surveys and what measures need to be taken to overcome them?
The speakers of the Quality Day discuss this with each other from their respective perspectives and openly share their experiences and assessments with the audience. Additionally, Florian Kögl from ReDem is joining the panel discussion.
Participants:
