Challenge
Ensuring the quality of collected panel data has always been a top priority at (r)evolution. Before implementing ReDem, the company employed methods like oversampling, trap questions, and filtering out speeders and flatliners to combat “lazy respondents” and systematic fraud. However, these approaches were time-consuming and inconsistent, as they varied from project to project. A major challenge was assessing open-ended responses, a particularly labor-intensive process, with fraudulent answers becoming harder to spot. Over the past one to two years, (r)evolution witnessed a significant rise in fraud cases within their online samples, particularly in heavily researched markets like the U.S., where large incentives are offered. Traditional methods for data validation and cleaning began to hit their limits, as detecting fraud became increasingly difficult, and oversampling alone was no longer enough to guarantee data quality.
Solution
To tackle these challenges, (r)evolution began using ReDem for data cleaning and quality assurance — a process now applied to the majority of the company’s studies. Through a live integration with keyingress, the survey software used by (r)evolution, each interview is evaluated in real time against predefined quality criteria. Low-quality data is flagged as a "Quality Fail" and reported back to the panel, ensuring it does not affect quotas. A key aspect of this process is the automated review of open-ended responses, used across all studies. Additional checks include content analysis and duplicate detection within and across interviews, along with monitoring response times and patterns in grid questions.
Impact
(r)evolution has seen a marked improvement in data quality, particularly evident in the review of open-ended responses.
A striking example of this is an analysis on product usage intent, which revealed how significantly fraud can distort study results. The comparison between the raw data and the cleaned data showed that the seemingly high usage intent in the raw data was primarily the result of fraudulent responses—cases that would have been nearly impossible to detect through manual review. Without a technology-based data quality review, these skewed data could have led to misguided decisions, such as launching products that ultimately would have failed due to inaccurate insights on customer demand.
In addition to improving data quality, (r)evolution has also achieved efficiency gains through automated data checking, cleaning, and quota management. The time required for project management, quality assurance, and data analysis has been significantly reduced.
"We are now reaching a level of data quality that would be impossible to maintain manually, given the growing sophistication and automation of fraud." - Kim Svete (r)evolution