9 design criteria for AI-assisted qualitative analysis tools

In late 2025, 419 experienced qualitative researchers from 32 countries put their names to an article in Qualitative Inquiry rejecting generative AI for reflexive qualitative research outright. Among the signatories were Virginia Braun and Victoria Clarke, whose work most of the field's thematic analysis rests on.

That is not a fringe position, and it should not be dismissed as one. It is also not the whole story: in a survey of 150 research professionals published a few months later, 80% reported using AI regularly in their research work. The field is simultaneously adopting these tools at speed and formally rejecting them.

Both things can be true, because "AI in qualitative research" is not one thing. Pasting transcripts into a chatbot and asking for themes is a different practice from running a structured coding workflow whose every output links back to a verified quote. The debate stays stuck because it argues about a category ("AI") rather than about what a tool actually does and how a researcher can use it.

So this post attempts something more useful: taking the critiques seriously enough to convert them into design criteria. Nine of them, drawn from the published objections, from professional guidance, and from our own working paper on the subject. Then, since we build one of these tools and you should discount accordingly, an account of how Skimle addresses each and where it does not.

What are the critics of AI in qualitative analysis objecting to?

The objections fall into three families, and they are not equally answerable by better engineering.

Capability objections. Generative models do not understand what they read. Jowsey and colleagues put it directly: GenAI is "simulated intelligence only, based on statistical predictive algorithms without any understanding of the world". Duc Cuong Nguyen and Catherine Welch's analysis in Organizational Research Methods is the most thorough version of this argument, identifying five epistemic risks: a category error that mistakes next-word prediction for analysis, unstable outputs from a probabilistic architecture, anthropomorphic fallacies that cast the chatbot as a collaborator, causal misattribution that blames the user's prompting for the technology's limits, and an "oracle effect" whereby users grant outputs an authority they have not earned.

Practice objections. Even where output quality is adequate, the tool changes the researcher. The concern is cognitive offloading: the gradual surrender of interpretive work to a system that is happy to do it. Jowsey and colleagues note the trap precisely, that "our desire for GenAI to be reliable reduces our capacity to critically appraise GenAI outputs." Coding is not merely a means to a coded dataset. Reading transcripts closely, arguing with yourself about a label, and revising a scheme are how a researcher comes to know their material. A tool that removes that work removes the knowing.

Ethical objections. These divide again. Some are addressable by design: participant privacy, consent, re-identification, disclosure. Asher Beckwitt, a medical and cultural anthropologist with 25 years in qualitative research who now advises teams on responsible AI practice, makes the practical version of this case, pointing out that removing names is often not enough, since a participant may remain identifiable through position, location, diagnosis, relationships, or a combination of demographic details. Others are not addressable by any tool design at all: the environmental cost of data centres, and the conditions of the data workers who moderate training content.

There is also a strong professional-practice strand outside academia. The ESOMAR-affiliated Research World published five rules for AI in market research in August 2025 covering duty of care, data minimisation, transparency, bias, and keeping humans in the loop. In health research, a team led by Leonard Fehring has begun developing COREQ+LLM, an extension to the standard qualitative reporting checklist, on the reasoning that COREQ's single item about software is inadequate for documenting how a language model was used.

Read together, these sources converge on a fairly consistent set of demands. Here they are as criteria.

The 9 design criteria for AI-assisted qualitative analysis tools

#	Criterion	What it demands of a tool	Skimle's approach
1	Non-fabrication	No quote, citation or claim that is not present in the source	Every extracted quote is verified verbatim against the source document by non-AI (mechanistic) software
2	Complete, inspectable coverage	Analyse all the data, and show what was left out	Determinate and iterative workflows over the full corpus passage by passage rather than opaque retrieval in one go; uncoded passages remain visible in the document view
3	Predictability	The same actions produce the same shape of result, every time	A fixed five-step workflow (focus, extract, categorise, assign, summarise); all analysis run through a human expert designed pipeline, not an improvised one
4	Two-way transparency	Conclusion to source, and source to conclusion	Every insight, category and summary links to its passages; every document shows which passages were used
5	Substantive control	Low-friction, high-useability editing of everything the AI produced	Rename, merge, split, move, reassign and restructure, by click or chat, with undo
6	Method-agnosticism	The tool must not impose a methodology	Inductive, theory-driven and abductive modes; category restructuring to an alternative theoretical frame; MCP access for external agents
7	Adversarial rigour	Actively seek counter-evidence and negative cases	Agentic analysis runs a dedicated counter-evidence search per theme and revises claims to be conditional
8	Privacy and governance	Known data location, no training on your data, real anonymisation	EU processing and storage, GDPR compliance, no customer data used for model training, built-in pseudonymisation options
9	Auditability	A record of what was done, in a form you can report	Analysis log memo recording every step in agentic analysis model; full export including REFI-QDA for all analysed text

Criteria 1, 2, 8 and 9 are hygiene. A tool that fails any of them should not be used for research at all, and most general-purpose chatbots fail at least two. Criteria 3 to 7 are where tools genuinely differ, and where the design argument lives.

1. Non-fabrication

Hallucination is the objection everyone knows, and the one that is closest to solved in purpose-built tools. Confabulation is not an occasional glitch but a property of how these models work, so the answer cannot be a better prompt. It has to be verification outside the model.

Skimle's extraction pipeline checks every quote the model produces against the source text and discards anything that does not appear verbatim. That eliminates invented quotations, which is the failure mode that would destroy a study's credibility fastest. What it does not eliminate is a badly chosen code name or an unwise category boundary, and it is worth being precise about that limit rather than implying that verification makes the output correct. It makes the evidence real. Judging whether the evidence supports the label is still yours. We go through the mechanics in hallucinations, context and the black box.

2. Complete, inspectable coverage

A subtler failure than fabrication is silent omission. Many consumer tools retrieve a sample of your documents through an embedding search and analyse that sample, without telling you which documents never made it in. The output looks like an analysis of your corpus and is an analysis of an unknown subset of it.

Skimle processes the full corpus through a determinate workflow rather than an opaque retrieval step, which is the argument we made at length in why RAG does not work for qualitative analysis. Just as important is the second half of the criterion: showing what was not used. The document view highlights coded passages, which means the uncoded text is visible too. Being able to scan a transcript and notice that a whole section went uncoded is how you catch a framework that is missing a category, and it is a capability that a summary-only tool cannot offer at all.

3. Predictability

A tool becomes an instrument when its behaviour is stable enough that you can develop skill with it. A hammer is useful because it responds consistently. A tool that decomposes your task differently each time cannot be learned, only supervised. When trying to analyse qualitative data with vanilla chatbots or tools like Claude Code, they will improvise ways to do it on the go (e.g., count words) that change every time and generally attempt to minimise token costs rather than find the right answer.

Skimle follows a generic workflow modelled on ordinary qualitative practice: set the focus, extract and code passages, develop categories, assign material to them, summarise. The same documents and the same instructions produce substantively the same analysis, and users learn how their prompting shapes the outcome. This has a real cost, which is that it constrains the system: a fully agentic tool that plans its own approach each time can do things a fixed pipeline cannot. Our agentic analysis is the attempt to have both, running a declared pipeline of named steps rather than an improvised one, so that even the most autonomous mode has a shape you can anticipate. Whether that is enough remains an open question, and we would rather say so than pretend the tension is resolved.

4. Two-way transparency

Transparency at the level of model internals is unattainable, and demanding it is a distraction (just like human judgement and thinking is largely a black box). What is attainable is transparency of behaviour: showing how inputs relate to outputs, which is exactly the standard we hold human researchers to. Nobody can see how a co-author's reasoning works either; we ask them for a methods section and an evidence table.

That transparency runs in two directions, and most tools offer at most one. Conclusion to source is the familiar one: from any claim, get to the quotations behind it, and read each in the context of the full document. Source to conclusion is the neglected one: for any document, see what was drawn from it and what was passed over. The second direction is what lets you judge whether an analysis is balanced or systematically blind. Skimle implements both, and the argument for why the second matters is in two-way transparency.

For agentic workflows the requirement extends further, into what our working paper calls end-to-end transparency: being able to travel the path from initial analytical choices to final conclusions, seeing what alternatives the system considered. Skimle's agentic analyses write a log memo recording every step, including the setup conversation, but a genuinely navigable map of an agentic run is still ahead of us rather than behind.

5. Substantive control

Control is the criterion that most directly counteracts cognitive offloading, and it is mostly a question of friction. If renaming a category takes one click, researchers rename categories. If it takes a new run of the full analysis taking 5 minutes and costing thousands of tokens, they accept whatever the model produced. Analysis drifts towards whatever is easy.

Skimle is built so that revising the AI's output is routine rather than exceptional: codes can be edited, renamed, merged and moved, passages reassigned, and the category structure reorganised wholesale. Nearly everything can be done by clicking or by chat, and nearly everything can be undone. The framing we want is that an AI-produced analysis is a draft you are expected to reshape. Interpretive authority stays with the researcher not because we say so in a marketing sentence but because changing the analysis is faster than arguing with it. And the AI assisted features support also manual editing, for example you can remove a category and ask Skimle to automatically reassign the insights to second-best fitting categories.

6. Method-agnosticism

There is a reasonable assumption that because these tools do symbolic processing, each must implement one qualitative method. We think a good tool should not. The methodological range in qualitative research is wide, from Glaserian open coding that approaches data with as few preconceptions as possible, through Charmaz's view of coding as an interpretive act shaped by the researcher's perspective, to reflexive thematic analysis where the researcher's theoretical position is the engine of the work.

A tool bound to one of these forecloses the others, and it does so quietly. If inductive coding is effortless and theory-driven coding is cumbersome, users drift towards inductive work regardless of what their study needs. Skimle's predefined categories mode covers both ends: start from suggested categories for the data-driven end, or write theory-informed category descriptions for the deductive end. It adds category restructuring so a coded corpus can be reorganised under a different conceptual frame entirely. The grounded theory tradition and a Gioia-style structure can both be run on the same material.

The MCP interface goes further, letting external agents read and write project data, so the tool does not become the ceiling on what you can do.

7. Adversarial rigour

This criterion answers the objection about dominant patterns. Language models are drawn to the frequent and the consensual, and the concern that they will quiet marginal voices is not speculative. Valentin Hofmann and colleagues showed in Nature that models hold raciolinguistic stereotypes about speakers of African American English more negative than any human stereotype ever experimentally recorded, and that alignment training on overt bias widened rather than closed the gap with the covert kind.

A tool that only aggregates will therefore systematically flatten your data towards its majority. The design response is to make contradiction a step in the process rather than an act of researcher virtue. Skimle's agentic analysis runs a dedicated counter-evidence search against every analytical theme, then revises the theme prose to make claims conditional where the counter-evidence warrants. Themes with strong counter-evidence are flagged.

Metadata slicing exists for the same reason: a finding that holds in aggregate and fails in a subgroup is the finding you most need to see. We wrote about the underlying commitment in rigour means being able to return a negative result, and about the mechanisms in bias in AI-assisted qualitative analysis. In Skimle, when you analyse data by segment (for example "men vs. women"), we minimise bias by never labelling the groups with the names when using LLMs to tease out the differences. The model is asked to compare "group A" versus "group B", so that patterns it spots would stem from the data not its prejudices or common stereotypes.

8. Privacy and governance

Qualitative data is usually the most sensitive material a researcher holds. Before anything is uploaded, the answerable questions are concrete: where is it stored, is it retained, is it used to train models, who can access it, and is the intended use consistent with the consent participants gave and the approval an ethics board granted.

Skimle processes and stores data in the EU, is GDPR-compliant, and does not use customer data to train models. Data processing agreements are available, and single-tenant deployments exist for teams whose security requirements demand them. On the re-identification point Beckwitt raises, we would agree that stripping names is insufficient, which is why anonymisation is a built-in step that pseudonymises consistently across a corpus rather than a find-and-replace per document. Skimle also spots and flags combinations of descriptors that might identify a unique entity: for example "a great qualitative analysis tool developed in Finland" doesn't say Skimle, but its evident what it refers to...

What no vendor can do for you is decide whether your consent language and ethics approval actually cover processing by a third-party service. That determination is yours, and it should be made before the upload rather than after.

9. Auditability

The reporting-standards work now under way, COREQ+LLM among it, points at the same gap: a study that used AI needs to be able to say what the AI did, how outputs were checked, and who took responsibility. Beckwitt's argument that organisations need qualitative-specific protocols rather than a general instruction to "use AI responsibly" is the practical version of the same point. A protocol needs artefacts to point at.

Skimle produces those artefacts as a by-product rather than as extra work. Agentic analyses write an analysis log memo narrating every step the system took, including the setup conversation that shaped the framework, so a methods section can be written from the record rather than from memory. Everything exports, including to REFI-QDA so coded material can move into NVivo or ATLAS.ti for a colleague to check independently. Our 20-question checklist is a reasonable pre-publication review to run against whatever record you have.

What these criteria cannot fix

A piece like this could easily end by implying that nine well-implemented criteria dissolve the objections. They do not, and the gaps deserve naming.

The meaning-making objection stands where it stands. If your position is that reflexive thematic analysis is constitutively a human interpretive practice, then no amount of verification, transparency or control makes a machine a legitimate participant in it, including at the initial coding stage. Jowsey and colleagues are explicit that they consider GenAI inappropriate in all phases of reflexive analysis. Design criteria are answers to questions about tool quality. That is a question about what the method is, and it is not ours to settle by shipping features. Researchers working in the Big Q tradition who accept that argument should not use our software for that work, and we would rather say so plainly than sell around it.

What we do believe is that different types of tools serve human meaning-making in very different ways. A tool like Skimle with full transparency, user control and rigorous methods can give a better starting point for human analysis and reflection than a black box AI tool spitting out a list of themes and quotes. Allowing the researcher to explore alternative ways to code the corpus without spending a week recoding each interview can make it easier to be curious about what the data is telling them.

The environmental and labour objections are real and unaddressed here. Data centre water and energy use, and the conditions of the workers who moderate training content, are costs of the technology that a well-designed interface does not offset. They belong in the decision, and a tool vendor listing nine criteria that happen to exclude the two it cannot influence should be read with that in mind.

In our opinion there is a wide spectrum of LLM usage globally, and assisting a serious qualitative researcher in discovering new insights would likely be among the most beneficial use cases for the technology.

Cognitive offloading is mitigated, not eliminated. Every criterion above is aimed at keeping the researcher engaged, and none of them forces engagement. A determined user can accept every AI-produced category without reading a transcript, and the output will look identical to careful work on the surface. The literature on this is not comforting: heavy reliance on AI tools is associated with reduced critical engagement and weaker independent judgement, mediated precisely by how much offloading the tool invites. We can lower the friction of intervening. We cannot make anyone intervene. On which, see is AI destroying the ability to think.

We hope cognitive offloading will ultimately be weeded out by raising standards for quality. A great researcher equipped with great tools should be able to, and expected to, dig deeper and more rigorously to their data and before. They should be able to collect the sample they want, transcript it with high quality, anonymise it well and explore different analytical lenses to produce results worth celebrating. The bar is raising, and that is a good thing.

Questions to ask as you are evaluating your tools

The criteria are more useful as a purchasing instrument than as an argument, so here is the version to take into a vendor conversation. None of these questions is answerable with a demo video.

Show me a quote in a finding, and show me where it appears in the source document.
Show me a document, and tell me for each passage where it was used or was it used at all.
Run the same analysis twice. How different are the results?
Remove a category and reassign 20 insights from it to a best fit. How long did that take?
Can I impose my own theoretical framework, or only accept the one you generate?
What did the system find that contradicts its own conclusions?
Where is my data stored, is it used for training, and can I have that in a DPA?
Can I export the full coded project to another tool, for example via REFI-QDA format?

A tool that answers all eight is not thereby appropriate for every study. It is merely a tool whose behaviour you can describe to a reviewer, which is the minimum for research use and rarer than it should be. Our own comparison of qualitative analysis tools covers how the main options fare on the practical dimensions.

If you work in academic research, how Skimle fits an academic workflow covers the ethics and reporting side in more detail. For commercial insights work, responsible AI in qualitative market research sets out the equivalent rules for teams without an ethics board to answer to.

Frequently asked questions

Do these criteria make AI-assisted analysis acceptable for reflexive thematic analysis?

Not by themselves, and possibly not at all. The rejection published in Qualitative Inquiry is a methodological argument about what reflexive analysis is, not a complaint about tool quality, and it applies to all phases including initial coding. Researchers who hold that position should not use AI coding for that work. Our criteria are aimed at the large territory of qualitative analysis that does not make manual reflexivity constitutive: content analysis, framework analysis, applied policy and market research, large-corpus work, and much of grounded theory practice.

Is a general-purpose chatbot ever adequate for qualitative analysis?

It fails the hygiene criteria, which is the more useful way to put it than arguing about capability. A chatbot fabricates quotes without verification, samples your corpus without telling you which parts, produces different results on identical inputs, keeps no audit trail, and in most consumer configurations offers no usable answer about data retention. Those are disqualifying for research regardless of how good the prose looks. For exploratory reading of a handful of documents you own outright, it is fine. Read more in our article on how ChatGPT actually works.

How do I document AI use in a methods section?

State the tool and version, what it did and what you did, how outputs were verified, and who is accountable for the interpretation. A concrete example: coding was AI-assisted against a researcher-defined framework, all coded passages were verified verbatim against source documents, the framework and category boundaries were reviewed and revised by the named researcher, and every reported theme is traceable to source. The COREQ+LLM work will formalise this; until it lands, that shape covers most reviewer concerns.

You build Skimle. Why should I trust this list?

You should not, on our say-so. The criteria are drawn from published critics who have no interest in our commercial success, and the fair test is to apply them to us adversarially. Ask us the eight questions above and check the answers in a trial account rather than a demo. The parts of the critique we cannot answer are stated in this post, which is the only real evidence a vendor can offer that the list was not reverse-engineered from a feature set.

Want to test these criteria on your own data? Try Skimle for free and run the eight questions above against a real project. Every finding traces back to a verified quote, every document shows what was and was not used, and everything the AI produced can be changed.

Related reading: See designing AI that augments qualitative researchers for the underlying argument, two-way transparency for the interface case, and the AI qualitative data analysis checklist for a pre-publication review.

About the authors

Henri Schildt is a Professor of Strategy at Aalto University School of Business and co-founder of Skimle. He has published over a dozen peer-reviewed articles using qualitative methods, including work in Academy of Management Journal, Organisation Science, and Strategic Management Journal. His research focuses on organisational strategy, innovation, and qualitative methodology. Google Scholar profile

Olli Salo is a former Partner at McKinsey & Company where he spent 18 years helping clients understand the markets and themselves, develop winning strategies and improve their operating models. He has done over 1000 client interviews and published over 10 articles on McKinsey.com and beyond. LinkedIn profile

Disclosure: the authors have a financial interest in Skimle. The criteria above are drawn from independent published critiques; the assessments of Skimle against them are our own.