Rebuilding Trust: 4 Industry Leaders Tackle Data Quality Crisis

Survey Fraud & Data Quality: Defense or Offense?

Question: Are stronger data quality tools the solution — or is better respondent recruitment the real answer to the $10M survey fraud scandal that has shaken the industry?

This defense-versus-offense dichotomy set the stage for a thought-provoking exchange. Steve Male opened the discussion by emphasizing the need to play both offense and defense when combating fraud. "It's great that we're actually having these conversations now, we're being more transparent than ever," he noted. While acknowledging the industry's progress in developing tools to validate and cleanse survey data, he highlighted a critical gap: "One of the areas that we haven't really talked a lot about is how we do a better job of recruiting and validating respondents at the source."

While Tia Maurer agreed, she took the discussion a step further by stating bluntly, "The ecosystem is broken." What she meant by this is that traditional verification methods have deteriorated, from physical mail verification to simple email confirmations and this has allowed fraudsters to easily create multiple personas. Her stark reality check resonated when she asked the audience if they knew anyone who sits around taking surveys all day, with no hands raised in response. But we know this from multiple sources within the industry, that professional survey takers are doing just that. What was a line of defense needs to be reconsidered and that requires some offensive plays.

It was agreed that multiple tools are needed to examine different aspects of data quality, from duplications to bot detections and everything in between. Chuck Miller emphasized that demanding transparency isn't enough. He added that demanding transparency from your providers about what they are doing to combat fraud is not enough. Instead, he insisted that we all have to “do our own homework," and collectively raise the bar on the standards of both offensive and defensive approaches.

Random Digital Recruitment: Opportunity or Risk?

Question: Some are considering forms of random digital recruitment (river sampling) as an option. What are the benefits and drawbacks compared to traditional panel assets?

As the industry wrestles with panel quality concerns, many researchers have turned their attention to random digital recruitment—commonly known as river sampling—as a potential solution. This approach, which sources respondents from across the internet rather than from established panels, initially promised fresher perspectives from people less conditioned by the research process. The question posed to our panelists: Could this method offer benefits that outweigh its potential drawbacks when compared to traditional panel assets?

Drawing on his extensive experience with river sampling during his time at AOL, Chuck offered a balanced perspective. "When we started sourcing from all over the Internet, we drew a correlation that it's just like our RDD (Random Digit Dialing)," he explained. "You're going out, you're touching people, you're bringing them in." This approach can yield respondents who aren't as invested in the research process, potentially providing fresher, less rehearsed opinions.

However, Chuck cautioned about practical challenges that come with river sampling. "The economics can be challenging," he noted, and researchers need to be mindful of how they implement quality checks. "You need to look out for things in terms of your red herrings and your trap questions," he warned, explaining that inexperienced respondents may be confused by validation questions that seem out of context, leading to higher drop-off rates.

Tia more directly challenged the notion that river sampling inherently produces better quality with compelling data from her own research. She described a simple experiment comparing river samples to validated panels: "I sent one question out to a river sample source and to verified panelists." The question asked respondents which fast food chains they had heard of, including major brands and a fictional one. The results were striking—while the validated panel showed 95-97% awareness of major chains like McDonald's and Burger King, the river sample reported only 45% awareness. Even more telling, 33% of river sample respondents claimed they had heard of none of the brands, compared to less than 1% in the validated panel. The fictional brand was recognized by 34% of river respondents versus just 11% in the validated panel.