To analyse concept test interviews, code every reaction against a fixed set of evaluation criteria (clarity, relevance, believability, difference, price fit, purchase intent), keep each concept and segment separate, count how many respondents sit behind each reaction, read the reasons behind the strongest and weakest scores, and end with a recommendation per concept: go, fix or stop.
Concept tests are among the most common studies in commercial research, and among the most commonly over-summarised. A team shows three new product ideas to 30 or 100 people, collects pages of reactions, and the debrief ends up as "Concept B was the favourite". That answer is rarely wrong, but it throws away the part that matters most: why the others failed and what would fix them.
What is qualitative concept testing?
Concept testing shows a target audience an early version of an idea (a product, a pack, a service, a campaign line) and collects their reactions before money is spent on development or launch. It is a staple for brand teams and product managers alike. Quantitative concept tests ask everyone the same rating questions and compare scores. Qualitative concept tests ask open questions, follow up on reactions and capture the reasons.
Most teams need both: survey scores show which concept wins, and interviews explain why and what to change. With AI-moderated interviews, the line between the two has blurred: you can now run 100 qualitative concept interviews in a few days, and count reactions across them while keeping the depth. Our guide to sizing AI-moderated interview studies covers how many you need.
How should you design a concept test so it can be analysed?
Analysis is easier when the interview guide is built for it. Four design choices matter.
- Fix the evaluation criteria in advance. Decide which dimensions you will judge each concept on, and ask about each one for every concept. A common set: first impression, clarity, relevance to me, believability, difference from what I use now, price fit and likelihood to buy.
- Rotate the order. The first concept shown is judged differently from the third. Rotate the order across respondents and record which order each person saw.
- Separate unprompted from prompted. Ask "what is your first reaction?" before any specific question. Unprompted reactions are the purest signal of what the concept communicates.
- Capture metadata. Record segment, current brand, usage level and concept order for every respondent. These become the columns you compare later.
For general guidance on writing the questions themselves, see how to write the perfect interview guide.
How do you analyse concept test interviews in 5 steps?
Step 1: code each concept separately against the criteria
Treat each concept as its own analysis. For every respondent, code what they said about Concept A against each criterion, then do the same for B and C. A useful structure is a top-level category per criterion, with positive and negative reactions as subcategories:
- Clarity: understood immediately / confused by X
- Believability: credible / sceptical because Y
- Difference: new to me / "just like Brand Z"
Use deductive coding for the criteria (they are fixed) and inductive coding inside each one (the reasons are not). We explain the combination in inductive, deductive and abductive coding.
Step 2: count respondents, not mentions
For each concept and criterion, count how many respondents reacted positively and negatively. Count people, not mentions: one enthusiastic respondent who praises the packaging five times is still one respondent. The result is a scorecard like the one on the cover of this post: concepts across the top, criteria down the side, and the share of respondents behind each reaction.
The chart below shows the kind of frequency view Skimle produces for each category, which you can filter by concept or segment.
Step 3: compare segments
A concept that scores averagely overall may be loved by one segment and rejected by another. That is often the most valuable finding in the whole study. Compare reactions by the metadata you captured: target segment versus others, current users versus non-users, and concept order (to check for order effects).
In Skimle, metadata analysis ranks every possible split by the size of its effect and writes what separates the groups, as in the screenshot below, so you do not have to test every combination by hand.
Step 4: read the reasons behind the extremes
Use the counts to decide where to look, then read the quotes. For each concept, read every quote behind its strongest positive and strongest negative reaction. Look for:
- Fixable objections: misunderstandings caused by wording, a confusing visual, a missing proof point.
- Fundamental objections: "I don't have this problem", "I would never pay for this".
- Unexpected appeal: reasons for liking the concept that the team did not intend, which may point to a better positioning.
Step 5: write a recommendation per concept
End with a clear verdict for every concept, not just the winner:
| Verdict | When to use it | What to include |
|---|---|---|
| Go | Strong on relevance and difference, objections minor | The two or three fixes to make before launch |
| Fix and retest | Appeal is there, but a fixable objection blocks it | The specific objection, with quotes, and the suggested change |
| Stop | Fundamental objections in the target segment | The core reason, so the team does not revive it next year |
If your team runs concept and product tests for clients or brands, see how this fits the market research and customer insights workflow.
What are the most common mistakes in concept test analysis?
- Averaging across segments. "60% liked it" hides a concept that 90% of the target audience loved and 30% of everyone else disliked.
- Ignoring order effects. If Concept C was always shown last and always scored lowest, you may be measuring fatigue.
- Mistaking politeness for appeal. Respondents are kind to new ideas. "It's interesting" is not purchase intent. Weight specific, unprompted enthusiasm more heavily than general approval.
- Reporting only the winner. The reasons a concept failed are often more useful to the product team than the reasons another succeeded.
- Losing the language. The words respondents use to describe a concept they like are often better copy than the concept's own headline. Keep a list of them.
How does AI change concept test analysis?
The structure above has not changed in decades. What AI changes is the scale at which you can apply it and how quickly.
Coding 100 interviews against seven criteria for three concepts means placing roughly 2,000 reactions. By hand that takes days, and it is where shortcuts creep in. A systematic AI analysis codes every reaction, keeps the verbatim quote behind each one and counts respondents per category, leaving the researcher to review the structure and make the judgement calls. Skimle supports this with predefined categories for the criteria, inductive themes inside them, and every insight traceable to its quote.
The same applies to collecting the data. AI-moderated interviews can show each respondent the concepts in rotated order and ask the same follow-ups every time, which makes the analysis cleaner. See our comparison of AI and human interviewing for when a human moderator is still the better choice.
Frequently asked questions
How many respondents do you need for a qualitative concept test?
For a classic qualitative concept test, 20 to 30 respondents per target segment. If you want to count reactions and compare segments reliably, 30 to 50 per segment, which AI-moderated interviews make affordable.
Should each respondent see all concepts?
Usually yes, in rotated order (a monadic-sequential design). It allows direct comparison and is efficient. If concepts are long or very similar, show each respondent only one or two and increase the sample.
What is the difference between a concept test and a usability test?
A concept test asks whether an idea is appealing, relevant and believable before it is built. A usability test asks whether people can use something that already exists. Concept tests come earlier and focus on meaning; usability tests focus on behaviour.
How do I present concept test results to stakeholders?
Lead with the recommendation for each concept, show the scorecard, then give two or three quotes per key finding. Our guide to presenting qualitative findings to executives covers the structure, and Skimle can export the analysis to PowerPoint with quotes included.
Running a concept test soon? Try Skimle for free and code every reaction across concepts and segments, with each one linked to the respondent's own words.
Related reading:
- How to analyse customer interviews for market research
- How to write a qualitative topline report in 24 hours
- Qualitative research for CPG brands
About the author
Olli Salo is a former Partner at McKinsey & Company where he spent 18 years helping clients understand the markets and themselves, develop winning strategies and improve their operating models. He has done over 1000 client interviews and published over 10 articles on McKinsey.com and beyond. LinkedIn profile
