Before AI-generated insights reach a client, check seven things: every quote exists verbatim in the source, every segment and market is covered, counter-evidence was searched for, theme sizes are counted rather than guessed, claims match the sample behind them, outliers are labelled as outliers, and a human can retrace any finding to its evidence.
AI has made it possible to turn 50 interview transcripts into a polished summary in minutes. That speed is real and useful. The risk is that the summary reads equally well whether it is right or wrong. Some in the industry have started calling this "ultra-processed insight": smooth, plausible, easy to consume, and stripped of the evidence that made it trustworthy. A client who catches one invented quote will question every finding you have ever sent them, which is why agencies and in-house insight teams need a defence.
The fix is a quality gate between the AI's output and the client, the same way a good agency always had a senior researcher review a junior's debrief.
Why do AI-generated insights need a separate quality check?
Traditional qualitative analysis has built-in friction. A researcher who codes 30 transcripts by hand has read every one, and their findings are limited by what they saw. AI analysis removes the friction, and with it the guarantee that someone has looked.
General-purpose AI tools fail in predictable ways on qualitative data. They paraphrase quotes and present them as verbatim. They over-weight the first and last documents in a long context. They state a theme confidently because it is a common theme in the category, whether or not your respondents said it. We describe the mechanics in hallucinations, limited context and black boxes.
Purpose-built tools reduce these risks but do not remove the need to check. The researcher still decides what the findings mean, and the client still needs to trust them. The seven checks below take about an hour on a typical study and catch most of what goes wrong.
What are the 7 checks?
1. Does every quote exist verbatim in the source?
Pick every quote that appears in the report and find it in the transcript. Not a similar sentence: the exact words. If your tool cannot show you where a quote came from, treat every quote as unverified.
This is the single most important check, because a fabricated quote is the error clients remember. Skimle verifies each quote against the source document with ordinary string matching before it is stored, and every insight links to its passage, so the check takes seconds per quote, as the screenshot below shows.
2. Is every segment and market covered?
List the segments, markets or respondent groups in the sample. For each key finding, check that it draws on more than one of them, or that the report says clearly that it applies to one group only. AI summaries tend to echo the largest or most articulate group. A finding that comes entirely from the UK interviews should not be presented as a European finding.
3. Did anyone look for counter-evidence?
For each headline finding, ask: who said the opposite? Most datasets contain respondents who disagree with the majority view. If the report never mentions them, either they do not exist (worth saying) or nobody looked. Asking an AI assistant "which respondents contradict this finding?" is a quick way to do this, provided the answers come with quotes you can check.
4. Are theme sizes counted, not guessed?
"Many respondents" and "a common theme" are claims about frequency. Check them against counts. If a theme appears in 4 of 40 interviews, "many" is wrong. Good tools count the respondents behind each category; in a spreadsheet you can count by hand. Our guide to quantifying qualitative data covers when counts help and when they mislead.
5. Do the claims match the sample that made them?
A study of 20 existing customers in two countries can say a lot about those customers. It cannot say what "the market" thinks, and it cannot say what non-customers think. Read each finding and ask whether the sample supports its scope. AI-written summaries generalise readily because generalised sentences are what they have been trained on.
6. Are outliers labelled as outliers?
A vivid quote from one respondent can carry a whole slide. That is fine if the slide says it is one voice. It is a problem if one dramatic interview becomes "customers feel betrayed". Mark each finding as widespread, segment-specific or individual.
7. Can a human retrace any finding to its evidence?
Choose three findings at random and ask a colleague who did not do the analysis to trace each one back to the transcripts. If they can do it in a few minutes, the analysis is auditable. If they cannot, the client cannot either, and neither can you in six months' time when someone asks how you reached that conclusion. This is the idea we call two-way transparency: from finding to evidence, and from any document to the findings it shaped.
Running these checks on every project is easier when the analysis tool keeps the evidence attached. See how that works for market research and customer insights teams.
Who should run the quality check?
Ideally someone other than the person who produced the analysis. In an agency that is usually the project director; in an in-house team, a peer researcher. The reviewer does not need to re-read every transcript. They need access to the evidence behind each claim and an hour of focused time.
Two practical rules make this work:
- Build the check into the timeline. If the debrief is on Thursday, the quality review is on Wednesday morning, not Thursday at 8am.
- Review the evidence, not the prose. A well-written slide is persuasive by design. Open the quotes and the counts first, then read the narrative.
How do you use AI to help with the quality check itself?
AI is good at the search parts of quality control: finding contradicting quotes, listing which segments contributed to a theme, spotting duplicate phrasing. It is weak at judging whether a finding is meaningful. Use it for the first and keep humans on the second.
In Skimle the agentic chat reads the same coded analysis the researcher works on, so you can ask "which respondents disagree with the onboarding finding?" or "is the pricing theme driven by one market?" and get answers that cite real quotes, as in the screenshot below. Every cited quote opens in its original context.
Quality check summary table
Copy this into your project template:
| Check | Question | Fail signal |
|---|---|---|
| Verbatim quotes | Can I find every quote word for word? | Paraphrased or missing quotes |
| Coverage | Which segments feed each finding? | One group presented as everyone |
| Counter-evidence | Who said the opposite? | No dissent mentioned anywhere |
| Counts | How many respondents sit behind "many"? | Frequency words without numbers |
| Scope | Does the sample support this claim? | "The market" from 20 customers |
| Outliers | Is this one voice or many? | A single quote carrying a slide |
| Traceability | Can a colleague retrace it? | "The AI found it" as the only source |
Frequently asked questions
Do I need to check AI insights if I use a purpose-built qualitative tool?
Yes, though it takes less time. Purpose-built tools can guarantee that quotes are verbatim and that every document was read. They cannot guarantee that the researcher's interpretation is right or that a claim fits the sample. Those checks stay with people.
How long should an AI insight quality check take?
For a typical study of 20 to 60 interviews, about an hour if the tool links each finding to its evidence, and most of a day if the reviewer has to search transcripts by hand.
What is the most common error in AI-generated research reports?
Unverified quotes, followed closely by claims that are broader than the sample. Both are easy to catch with checks 1 and 5 above.
Can clients audit AI-assisted analysis themselves?
They can if you share the coded analysis with them. Many agencies now give clients view access to the project, so stakeholders can open any finding and read the quotes behind it. In Skimle this is done through project sharing.
Want AI analysis that passes the check by design? Try Skimle for free and see every insight linked to the verbatim quote it came from.
Related reading:
- Responsible AI in qualitative market research
- AI qualitative data analysis checklist
- The commoditisation trap for market research agencies
- A citation is not an analysis
About the author
Olli Salo is a former Partner at McKinsey & Company where he spent 18 years helping clients understand the markets and themselves, develop winning strategies and improve their operating models. He has done over 1000 client interviews and published over 10 articles on McKinsey.com and beyond. LinkedIn profile
