A public or regulatory consultation can generate hundreds — sometimes thousands — of written responses in a matter of weeks. The challenge for policy teams is not collecting the responses. It is analysing them fairly, systematically, and in a way that can withstand scrutiny if the policy decision is ever challenged.
Standard approaches break down at scale. Manually reading 400 responses takes several analysts weeks and produces inconsistent summaries. Automated sentiment analysis loses the nuance of detailed stakeholder arguments. Excel-based coding becomes unmanageable above 50 responses and is difficult to audit. And pure AI summarisation — feeding everything into a large language model — produces a fluent summary of the whole corpus but no traceability from conclusion to specific respondent.
Traceability is not a technical nicety. Regulatory decisions are subject to judicial review. Policy ministers are challenged in parliamentary committees. Responses are covered by freedom of information requests. The analysis must be defensible, not just useful.
This guide covers the framework for analysing consultation responses at scale: coding structure, respondent metadata, handling volume, documentation requirements, and where AI tools can appropriately accelerate the work without undermining its defensibility.
What types of consultation data are you dealing with?
Consultation exercises vary significantly in scope and structure, and the appropriate analysis approach differs accordingly.
| Consultation type | Typical scale | Typical format | Primary challenge |
|---|---|---|---|
| Open public consultation | 100-5,000+ responses | Online form, email, letter | Volume; variable quality; ensuring minority voices are counted |
| Targeted stakeholder consultation | 20-100 organisations | Written submission (often long, structured) | Depth; identifying substantive vs procedural concerns |
| Parliamentary/committee evidence | 30-150 submissions | Structured written evidence | Citation accuracy; connecting positions to evidence quality |
| Regulatory impact assessment | 50-300 responses | Online form or structured template | Consistency of coding; comparability across categories |
| Citizen panel or deliberative forum | 20-80 participants | Transcript, notes, survey | Combination of types; tracking how views evolved |
Each type requires a similar core methodology but different handling at the edges. A targeted consultation with 40 detailed organisational submissions requires careful reading and synthesis of complex arguments. An open consultation with 2,000 responses requires a sampling and prioritisation strategy alongside systematic coding.
How to build a coding framework for consultation responses
Before you begin coding, you need a framework that captures what you need to know from each response. The standard dimensions:
Position: Does the respondent support, oppose, or have a conditional/mixed view on the proposal? For multi-part consultations, position may differ by element. Do not force ambiguous responses into binary categories — code "mixed" or "conditional support" as a distinct position.
Issues raised: What substantive concerns, arguments, or evidence does the respondent present? These should be coded thematically, not just listed. A good coding scheme for issues groups related arguments: "implementation concerns" might include resourcing, timeline, and technical capability; "legal or procedural concerns" might include competence, proportionality, and consultation adequacy.
Evidence cited: Does the respondent cite external evidence? Is the evidence primary research, secondary analysis, anecdote, or assertion? This dimension is important when evaluating the weight to give different responses.
Respondent type: Who is this? Individual citizen, trade association, NGO, local authority, academic institution, regulated entity, professional body? This metadata is essential for reporting. Policy teams need to be able to say "of 120 industry respondents, 85% raised implementation timeline concerns, compared with 23% of civil society organisations." Without respondent metadata, you cannot make that claim.
New issues: Does the response raise something not anticipated in the consultation document? These are often the most analytically valuable inputs — they surface blind spots in the original policy thinking.
How to handle volume: sampling versus full analysis
The traditional approach to consultation volume was to sample. For consultations with more than 200-300 responses, comprehensive individual coding of every response was rarely feasible within typical policy timelines. The standard advice was:
- Code all organisational submissions in full; sample individual responses systematically using random stratified sampling by response length or stated position
- Prioritise substantively novel responses, and flag late submissions that introduce new arguments regardless of sequence
- Be transparent about sampling: "We coded all 124 organisational submissions in full and conducted systematic random sampling of the 723 individual responses"
This approach is principled and still defensible when sampling is genuinely unavoidable. But it carries a real cost: sampled responses can miss rare but important voices, particularly from individuals in affected communities who submit fewer, shorter responses than organised stakeholders. A response that falls outside the sample may contain the most significant challenge to the policy proposal — and the analysis will never surface it.
The better approach, now practically achievable, is full analysis of every response. AI-assisted tools like Skimle can process hundreds of submissions in hours rather than analyst-weeks, coding each document against the same framework consistently without fatigue or drift. The predefined categories analysis mode lets teams define their coding framework in advance and apply it across the full corpus, producing frequency counts, cross-tabulations by respondent type, and traceable source links for every finding.
Full analysis changes the defensibility position entirely. Rather than "we sampled 723 responses using stratified random sampling," the report can say "we coded all 847 responses against a pre-registered framework." That is a materially stronger claim in a judicial review, a parliamentary committee hearing, or a freedom of information request. It is also more likely to reflect the genuine distribution of views, including the minority positions that policy teams are obligated to acknowledge.
For teams that must still sample due to resource constraints, the traditional principles remain sound. But the threshold for when sampling is genuinely necessary has shifted considerably.
Documentation and audit trail requirements
Consultation analyses are subject to multiple accountability mechanisms: judicial review, parliamentary scrutiny, FOI requests, and public reporting requirements. The analysis needs to maintain a clear, accessible link from each policy conclusion back to the specific responses that support it.
Minimum documentation standards:
- Each coded response must be traceable to its source document
- The coding framework must be documented and applied consistently
- Any changes to the coding framework mid-analysis must be recorded with rationale
- The overall summary must distinguish between what was said by all respondents and what was said predominantly by particular respondent types
- Minority views must be explicitly noted, even if they do not drive the policy recommendation
This is where AI tools that summarise but do not provide traceability create a problem. A tool that produces "most respondents supported the proposal with concerns about implementation" is not analysis — it is a summary that cannot be checked. The analysis must support a claim like "47 of 124 organisational respondents explicitly raised implementation timeline concerns, including [specific organisations]."
How Skimle supports consultation analysis
Skimle was used to analyse submissions to the EU Digital Omnibus consultation — a case study covered in Skimle in action: EU Digital Omnibus consultation. The consultation workflow uses several features that are particularly relevant to policy teams:
Document loading at scale: Import PDFs, Word documents, and text files representing all consultation submissions in one project. Each document retains its source identity for attribution.
Metadata by respondent type: Each document can be tagged with respondent type (trade association, NGO, individual, academic), sector, country, and other relevant variables. The metadata analysis feature then lets you compare theme patterns across respondent types — essential for defensible reporting.
Predefined categories: The predefined categories analysis mode allows you to define your coding framework in advance and apply it consistently across the corpus — producing a deductive analysis that matches the consultation's stated themes.
Traceable insights: Every coded theme is linked back to the specific document excerpts that support it. This provides the audit trail that policy analysis requires.
Export to office: The export to Word and Excel produces a structured summary that can feed directly into the policy report, with quotes and source attribution intact.
For consultation teams handling sensitive personal data or politically sensitive material, the anonymisation feature can pseudonymise identifying information across a corpus before analysis. See how to anonymise qualitative research data for IRB and compliance for the relevant methodology.
Frequently asked questions
Can AI analysis be used in legally significant consultation processes?
Yes, but with care. AI tools that provide traceable, auditable analysis — where every conclusion can be linked back to specific source documents — are appropriate. AI tools that produce summarised outputs without source traceability create a documentation problem. The test is whether your analysis could be submitted as evidence in a legal challenge. If you cannot show which responses support which conclusions, the analysis is vulnerable.
How do you report on consultation responses when the results are mixed or contradictory?
Report the full picture, including disagreements. A consultation report that presents a consensus where significant dissent exists will be challenged. The appropriate framing: "The majority of respondents supported the proposal overall, but there was strong disagreement on [specific aspect], particularly among [respondent type]." Where evidence is in genuine conflict, acknowledge the conflict and explain how the policy decision weighs it.
How should anonymous responses be treated?
Anonymously submitted responses are valid inputs to a public consultation and should be included in the analysis. However, you cannot make claims about the respondent's sector or affiliation if it was not provided. Code what you can (position, issues raised) and note the anonymity. For your totals, separate out anonymous responses so your reporting is transparent: "Of 847 responses, 93 were anonymous; these have been included in the overall analysis but cannot be attributed by respondent type."
What is the difference between a consultation analysis and a summary report?
A summary report tells stakeholders what the consultation found. A consultation analysis is the structured work that supports that summary — the coding, the categorisation, the frequency counts, and the traceable links from findings to source documents. The analysis should be retained as a working document even if only the summary report is published. It is the analysis, not the report, that would need to be produced in a FOI request or legal challenge.
How long does consultation analysis typically take?
For a targeted consultation with 50-100 organisational submissions, thorough analysis takes 2-4 analyst weeks manually. With AI-assisted tools, this can be compressed to 3-7 days while maintaining traceability. For large-scale public consultations (500+ responses), manual analysis is rarely feasible within policy timelines — structured AI-assisted analysis with sampling is the practical approach.
Analysing a consultation and need traceable, auditable findings? Try Skimle for free or book a demo to discuss how the platform handles consultation analysis at your scale.
Related reading:
- Skimle in action: EU Digital Omnibus consultation
- How to anonymise qualitative research data for compliance
- How to analyse open-text responses at scale
About the authors
Henri Schildt is a Professor of Strategy at Aalto University School of Business and co-founder of Skimle. He has published over a dozen peer-reviewed articles using qualitative methods, including work in Academy of Management Journal, Organisation Science, and Strategic Management Journal. His research focuses on organisational strategy, innovation, and qualitative methodology. Google Scholar profile
Olli Salo is a former Partner at McKinsey & Company where he spent 18 years helping clients understand the markets and themselves, develop winning strategies and improve their operating models. He has done over 1000 client interviews and published over 10 articles on McKinsey.com and beyond. LinkedIn profile