"When Fraud Stings" – Cases of Survey Fraud in Market Research and What We Can Learn from Them
The rise in fraudulent responses has sparked a heated debate within the industry. While analyses show that on average, between 30% and 40% of market research data is compromised (Kantar, 2022), there is disagreement about the true extent of the impact on business outcomes. On one hand, reports suggest that survey fraud undermines the foundation of data-driven business decision-making, costing billions of dollars annually. For example, Procter & Gamble reported at the ReDem Quality Day (June 2024) about a failed product launch that was based on compromised data, leading to losses in the millions (watch the video recordings here). On the other hand, some argue that professional market research firms are successfully mitigating this issue, keeping the damage contained.
What is clear, however, is that while many market researchers continue to rely on traditional and increasingly ineffective methods like speeding checks, straightlining, and geolocation to combat fraud in their data, the field of survey fraud is evolving rapidly. Researchers warn that these methods alone are no longer sufficient to combat the increasingly sophisticated fraud strategies (Goodrich et al., 2023; Pinzón et al., 2023; Storozuk et al., 2020), leading to a gradual erosion of buyers' trust in the reliability of market research data.
This article delves into the various facets of online survey fraud, examining its impact on market research and showcasing real-world examples.
Three real-life cases show how survey fraud challenges market research
Case 1: Of Bees and Bytes: Decoding the Results of the US Beekeepers Survey
Researchers from the University of California, along with colleagues from three other universities, conducted a survey targeting the U.S. beekeeping industry. This survey, published in an article in the Applied Economic Perspectives and Policy journal in 2023, aimed to evaluate commercial beekeepers' preferences for different almond pollination contract attributes. They offered a $20 Amazon gift card as an incentive to attract the target population of large commercial beekeeping operations managing more than 300 colonies. The estimated size of the target population was ~1,200 beekeepers. However, after promoting the survey on several channels, it attracted over 2,500 responses—more than double the estimated target population.
Upon closer examination of the data, only around 4% of the responses were classified as legitimate by the researchers.In other words, 96% of these responses were fraudulent, primarily driven by bots and other malicious actors trying to exploit the incentive. As a result, the initial survey's findings were rendered worthless, providing no meaningful insights into beekeepers' contract preferences.
Case 2: Did 4% of Americans Really Drink Bleach? Unpacking the Data Behind Sensational Headlines
In the summer of 2020, shocking headlines claimed that 4% of Americans had ingested bleach to prevent COVID-19—a figure that would represent a staggering 12 million people. This alarming statistic, based on a Centers for Disease Control and Prevention (CDC) survey, was quoted by over 150 news outlets worldwide. However, a deeper investigation tells a different story.
A 2021 Harvard Business Review article takes a closer look at this alarming statistic and reveals how misleading data can easily make headlines. The problem? The original survey had significant flaws. Researchers who replicated the study, accounting for stricter data quality issues to eliminate inaccurate data, got very different results. They found that almost 80% of respondents who claimed to have ingested bleach failed basic attention checks or gave implausible answers to simple reality check questions. After filtering out these questionable responses, the number of confirmed bleach drinkers dropped dramatically—from 4% to 0%.
“To be clear, the takeaway here isn’t that all survey data is garbage. But especially when that data is used to support claims with serious societal repercussions, it’s essential to validate results with quality control interventions”
— Harvard Business Review (2021)
For more information, you can read the Harvard Business Review’s coverage of this story.
Case 3: The Virginia Farm Survey Fraud
To assess the economic impact of COVID-19 on Virginia farms and agribusinesses, researchers from leading U.S. universities launched a survey, offering a $10 Amazon gift card lottery as an incentive. Initially, the survey seemed successful, garnering 444 responses. However, a closer examination revealed that 72% of the responses were fraudulent. Faced with this high level of fraud, the researchers decided to cancel plans for another round of the survey, highlighting the significant challenges of preserving data integrity in incentivized online surveys.
For more information, you can read the Goodrich et al. (2023) im article in the Applied Economic Perspectives and Policy. lesen.
Conclusio
In the end, it's not the data itself that drives success —it's the quality of the data. The real power of data lies in its accuracy, relevance, and reliability. As we move forward, we must be proactive in tackling the challenges of survey fraud, ensuring that it does not erode the trust in market research. According to the 2024 GRIT Insights Practice Report, 80% of market research buyers prioritize data quality as a top factor when selecting partners, underscoring the critical importance of reliable survey data in today’s business environment.
With the right tools and a proactive mindset, we can ensure that our insights are not only insightful but also trustworthy. Survey research continues to thrive, and now is the perfect time to reinforce its integrity. As JFK wisely put it, "The best time to repair the roof is when the sun is shining."
