Patient experience and member experience surveys generate thousands of free-text comments alongside their quantitative scores. To analyse them at scale: de-identify the comments before analysis given the sensitivity of health information, run systematic thematic analysis rather than manual reading, and map every theme back to a specific care touchpoint (admissions, a named unit, discharge) so clinical and operations leaders can act on it. Tools like Skimle do this with AI-assisted theme discovery and a dedicated anonymisation workspace, while keeping every insight traceable to its source quote.
A health system runs its quarterly HCAHPS survey. The results land in a dashboard: an overall rating of 3.7 stars, a slight dip in the "communication with nurses" composite, a note that discharge information scores are below the state median. A committee discusses the numbers. Someone proposes a nurse communication training refresh. The meeting ends.
Sitting next to those scores, often in the same export, are hundreds of comments patients typed into the "Is there anything else you would like to tell us about your stay?" box. "The night shift nurse on 4 West was the only person who explained what was happening." "I waited three hours after my discharge order before anyone brought my paperwork." "Nobody asked about my pain until I asked them to." This is where patients tell you, specifically, what happened to them, on which unit, at which point in their stay. Most of it goes unread, because reading it properly, consistently, and safely is harder than it looks.
Why patient and member free text gets ignored
The problem is not that healthcare organisations lack patient voice data. CMS reports that the current HCAHPS correlations table draws on 2.2 million completed surveys from patients discharged between July 2023 and June 2024, and Press Ganey notes that over 7,000 patients respond to HCAHPS survey questions every day. Health plans run a parallel programme, the CAHPS Health Plan Survey, which measures member experience across getting needed care, getting care quickly, doctor communication, and health plan customer service.
Both survey families typically include open-ended questions, and patients use them. A 2025 topic-modelling study in JMIR Medical Informatics found that 43.4% of adult patients and 46.9% of pediatric caregivers who completed a patient experience survey included a free-text response. That is a study, not an anecdote, and the proportion is too large to treat as a footnote to the quantitative score.
So why does this data sit unanalysed? Three reasons come up again and again in conversations with quality and patient experience teams:
The volume outpaces manual reading. A health system running quarterly HCAHPS surveys across a dozen facilities can accumulate thousands of comments per cycle. Reading all of them, consistently, and writing up themes by hand is not a task one analyst can do properly within the reporting window.
The data is sensitive in a way generic customer feedback is not. A comment about a missed medication dose, a specific diagnosis, or a named clinician on a named unit carries protected health information. Before anyone outside a small compliance-cleared group can see it, work the same way NPS verbatim comments or product reviews can be worked, it typically needs review and de-identification.
Findings rarely map to an action owner. "Patients are unhappy with communication" is a theme nobody can act on directly. "Patients on the cardiac step-down unit describe nurses not explaining medication changes during shift handoff" is a finding the unit's nurse manager can take to a huddle. Getting from the first sentence to the second requires the comment to carry its touchpoint with it through the whole analysis, not just the topic.
What is the difference between patient experience and member experience research?
Patient experience research (HCAHPS, the NHS Friends and Family Test, CAHPS Clinician & Group surveys, and equivalents) asks people who received care to describe specific touchpoints: admission, the inpatient stay, communication with doctors and nurses, discharge planning, follow-up. The HCAHPS Survey is a 32-item, CMS-administered instrument that has been used since 2008 to produce standardised, publicly reported comparisons across hospitals, covering measures such as communication with nurses, communication with doctors, discharge information, and cleanliness of the hospital environment.
Member experience research asks people enrolled in a health plan to describe their experience of the plan itself, not a single care episode: ease of finding a doctor, claims processing, customer service responsiveness, prior authorisation friction. The CAHPS Health Plan Survey is the standard instrument here, producing composite measures across getting needed care, getting care quickly, doctor communication, and customer service.
In the UK, the closest equivalents are the NHS Friends and Family Test, which asks patients about their overall experience of a specific service, and the national patient surveys run by the Care Quality Commission across inpatient, maternity, and community settings. The methodology differs in detail from CAHPS, but the structural problem is identical: a quantitative score plus an open-text box, and an organisation that needs to act on both.
The practical distinction that matters for analysis: patient experience comments are anchored to a specific encounter and unit, so the analysis needs to preserve that touchpoint detail. Member experience comments are anchored to interactions with the plan over time, so the analysis needs to preserve the channel and process step (the call centre, the claims portal, the prior authorisation request).
How established players approach this problem
It is worth being clear-eyed about who already operates in this space, because the gap Skimle fills is narrower than "nobody analyses patient narrative comments."
Press Ganey is the longest-established patient experience measurement vendor in US healthcare, administering CAHPS surveys and providing national benchmarking across thousands of hospitals. It has also moved into narrative analysis: its GenAI-powered Narrative HX tool is built specifically to summarise and surface themes from the free-text portion of patient satisfaction surveys.
NRC Health runs its Human Understanding programme, combining CAHPS administration with AI-powered analytics, EHR-integrated patient feedback (MyView), and reputation management across patient reviews. It has operated in healthcare consumer experience since 1981.
Qualtrics offers a broader experience management (XM) platform with a healthcare vertical, unifying patient, member, and caregiver survey data with operational data from existing systems, and using AI to route and act on feedback signals in near real time.
| Vendor | Core strength | Where it sits in the workflow |
|---|---|---|
| Press Ganey | CAHPS/HCAHPS survey administration, national benchmarking, GenAI narrative summaries | Survey design through CMS submission and improvement programmes |
| NRC Health | Patient and consumer experience measurement, EHR-integrated feedback capture | Real-time feedback collection plus longitudinal relationship data |
| Qualtrics | General-purpose XM platform with a healthcare vertical | Multi-channel signal capture (surveys, calls, digital, reviews) across an enterprise |
| Skimle | AI-assisted qualitative analysis with source-level traceability and a dedicated anonymisation workspace | Deep analysis of free-text comments, transcripts, and AI interview responses once collected, with care-touchpoint and metadata mapping |
The distinction worth understanding: Press Ganey, NRC Health, and Qualtrics are platforms for collecting and administering patient and member experience surveys at the scale CAHPS programmes require, and several now offer narrative summarisation as one feature within that platform. Skimle is not a survey administration tool. It is built for the deeper qualitative analysis layer: when you have the comments (from any of these platforms, from an internal survey tool, or from interview transcripts) and need a rigorous, source-traceable thematic analysis that maps to specific units and processes rather than a high-level summary paragraph. Organisations already running CAHPS through one of the platforms above can still bring the resulting free-text export into a tool built specifically for systematic qualitative analysis at scale.
How does AI-assisted analysis handle patient narrative comments differently from manual review?
Manual review of patient comments runs into the same problems as manual NPS verbatim review: the analyst notices vivid, memorable comments more than statistically common ones, the process does not scale past a few hundred comments without weeks of effort, and ad hoc tagging categories ("communication issue", "nursing issue") are too broad to support a specific action.
Systematic AI-assisted thematic analysis works differently. Rather than starting from a list of categories you expect to find, the analysis reads the full set of comments and builds a theme structure from what is actually there. For a quarter's worth of HCAHPS comments, this typically surfaces the themes you expected (communication with nurses, discharge information, wait times) and some you did not: a specific pattern of confusion about a new electronic check-in process, a cluster of comments about call-light response time concentrated on one unit during overnight shifts, or recurring praise for a particular team that the quantitative scores never isolate.
This is the same structural problem Skimle already solves for NPS verbatim comments and broader open-text customer feedback at scale. The healthcare layer adds two requirements on top: the comments usually need de-identification before anyone outside a small group can see them, and the findings need to carry enough structural detail (unit, shift, service line, discharge stage) to be actionable by a specific operations or clinical lead, not just a quality committee.
Skimle's automatic thematic analysis builds that theme structure directly from the comment text, and every theme links back to the specific quotes that generated it. When a chief nursing officer asks "where does this finding come from?", the answer is the actual patient comment, not an AI-generated paraphrase nobody can verify. That two-way traceability from theme to source quote is what makes a finding usable in a clinical governance meeting, where unverifiable claims do not survive scrutiny.
Why anonymisation needs to happen before analysis, not after
Patient and member comments routinely contain protected health information even when nobody intended to include it. A patient describing their stay will often name a clinician, mention a diagnosis or medication, describe a specific room or unit, or reference dates that, combined with admission records, make the comment identifiable to anyone with access to both data sources.
This is a different sensitivity profile from customer feedback or employee survey data. A comment like "the night nurse on 4 West, Sarah, was wonderful but the resident who came in at 2am didn't introduce himself before changing my IV" names a real member of staff, a real unit, and describes a specific clinical interaction. Before this comment goes into a shared analysis workspace that a quality team, an external consultant, or a vendor's AI tool can access, it needs review.
Skimle Anonymise is built for exactly this kind of de-identification work. It scans documents for identifiers across six categories (names, titles and roles, locations, organisations, dates, and other contextual identifiers), flags them for review, and lets you choose how each category is transformed: pseudonymised, generalised, redacted, or kept. For patient comments, this typically means pseudonymising clinician and patient names, generalising specific room or unit references where the unit itself is not the analytical point, and reviewing dates that could combine with other data to narrow identification to one admission.
The process matters because it is auditable. Skimle Anonymise produces a PDF audit report documenting every identifier detected and every transformation applied, which is the kind of record a privacy or compliance officer can point to when asked how a dataset was de-identified, rather than relying on an analyst's verbal assurance that "we took the names out." Read the full mechanics in our piece introducing Skimle Anonymise, and the step-by-step process in how to anonymise interview transcripts.
How do you map free-text findings to specific care touchpoints?
The difference between an interesting theme and an actionable one is almost always the metadata attached to it. A finding that "12% of comments mention discharge confusion" is a starting point. A finding that "discharge confusion appears in 34% of comments from the orthopaedic unit, concentrated among patients discharged on weekends" is a brief someone can act on by Monday.
This works the same way it does for NPS verbatim analysis cut by score segment, but the relevant metadata dimensions are healthcare-specific:
- Care setting or unit (emergency department, a named inpatient unit, ambulatory clinic, telehealth) so a theme can be routed to the right operational owner
- Stage of the care journey (admission, during stay, discharge, post-discharge follow-up) so findings map to a specific process rather than "the hospital experience" generally
- Shift or time of day, which surfaces patterns that aggregate daily reporting hides, such as overnight staffing concerns
- Service line (medical, surgical, maternity, the three HCAHPS service lines) for organisations that need to compare experience across very different care types
- Score band (top-box, mid-range, bottom-box on the relevant HCAHPS composite, or promoter/detractor for member NPS-style measures), to see which themes are actually associated with dissatisfaction versus which appear regardless of score
Skimle's metadata analysis lets you attach these fields to every comment at import and then cut any theme by any combination of them. The comment about the night nurse on 4 West is not just an instance of a "nursing communication" theme. It is an instance of that theme, on that unit, on the overnight shift, from a bottom-box respondent, which is a different priority and a different owner than the same theme appearing evenly across all units and shifts.
A practical workflow for analysing patient or member free text
Export with every field intact. Pull the free-text comment column alongside response ID, the relevant composite or overall score, unit or service line, discharge or contact date, and any other structural fields your survey vendor captures. Do not strip these before import. They become the metadata that makes the analysis actionable later, exactly as described in end-to-end import and export workflows.
De-identify before the data leaves a controlled environment. Run the export through Skimle Anonymise before it reaches a wider analysis team, an external consultant, or any tool that was not part of the original data collection chain. Choose the protection level appropriate to who will see the output: a closed quality team reviewing internally needs less aggressive transformation than a dataset that will be shared with an external improvement consultancy.
Let the AI build the initial theme structure. Import the de-identified comments into Skimle and let automatic thematic analysis read the full dataset and propose a theme structure from what is actually there, rather than starting from a category list someone wrote two years ago. Expect this to surface both the themes you expected (communication, wait times, cleanliness) and a few you did not.
Review and refine with clinical and operational knowledge. Spend time checking whether the AI's groupings make distinctions that matter to your organisation. "Slow response to call light" and "nobody answered the call light at all" might need to be split if one is a staffing capacity issue and the other is closer to a safety event. Skimle lets you merge, split, and rename themes, and read the underlying comments behind any category to verify the grouping holds up.
Cut by touchpoint metadata. Break the confirmed theme structure down by unit, shift, service line, and score band. This is the step that turns "patients mention communication issues" into a brief a specific unit can act on.
Write the recommendation with the evidence attached. For each significant theme, state what it is, how prevalent it is, which unit or stage it concentrates in, three to five representative (de-identified) quotes, and the action you are recommending. A quality committee or board safety committee will act on "discharge confusion concentrates on the orthopaedic unit on weekends, when senior staffing is reduced, recommendation: weekend discharge planning checklist" far more readily than on a percentage in a slide.
Frequently asked questions
Can AI tools analyse patient experience comments without exposing protected health information?
The comments need to be de-identified before they reach any tool or team outside the original data collection chain, regardless of which AI tool performs the thematic analysis. A dedicated anonymisation step, such as the one Skimle Anonymise provides, detects names, roles, locations, organisations, and dates across a batch of comments and lets a reviewer choose how each is handled before the data is used for broader analysis. Running unredacted comments directly through a general-purpose AI chat tool is a data handling risk, not a shortcut.
How is analysing HCAHPS free-text comments different from analysing customer NPS comments?
The underlying thematic analysis method is the same: read the comments, build a theme structure from what is there, code the full dataset, and segment by metadata to find where themes concentrate. The differences are the sensitivity of the data (protected health information requires de-identification before broader analysis) and the metadata that matters (care unit, shift, and stage of the care journey rather than product line or customer segment). Our guide on analysing NPS verbatim comments covers the shared methodology in detail.
What is the difference between HCAHPS and the CAHPS Health Plan Survey?
HCAHPS measures patient experience of a specific hospital stay: communication with doctors and nurses, discharge information, responsiveness of staff, and the cleanliness of the environment. The CAHPS Health Plan Survey measures member experience with a health insurance plan over time: getting needed care, getting care quickly, doctor communication, and health plan customer service. Both produce a quantitative score and typically include open-ended questions, and the analysis approach for the free text is structurally similar even though the touchpoints being measured differ.
How many patient comments do you need before thematic analysis produces reliable findings?
There is no fixed minimum, but below roughly 100 to 150 comments it becomes difficult to distinguish a real pattern from one or two vivid but unrepresentative comments. A 2025 study in JMIR Medical Informatics worked with tens of thousands of free-text responses collected over several years, which is well beyond what most facility-level quarterly reports will have, but the same topic modelling and coding logic scales down to smaller, more typical reporting volumes without losing validity, provided the theme structure is reviewed rather than accepted automatically.
Does Skimle replace our CAHPS survey vendor?
No. Skimle is not a survey design, administration, or CMS submission platform, and it does not replace vendors like Press Ganey or NRC Health for that part of the programme. It sits downstream: once you have the free-text comments, whichever platform collected them, Skimle provides AI-assisted thematic analysis with care-touchpoint metadata and a dedicated anonymisation step. Many organisations use a CAHPS-certified vendor for survey administration and a separate tool for the deeper qualitative analysis of the narrative comments.
Ready to turn patient or member free-text feedback into findings your operations and clinical teams can act on? Try Skimle for free and bring your next survey export into a workflow built for sensitive data, source-level traceability, and care-touchpoint metadata.
Want to go deeper on the adjacent methodology? Read our guides on analysing NPS verbatim comments, building a voice of customer programme, and how to anonymise interview transcripts. If your team works in consumer or market research more broadly, see how Skimle fits customer and market research workflows.
About the authors
Henri Schildt is a Professor of Strategy at Aalto University School of Business and co-founder of Skimle. He has published over a dozen peer-reviewed articles using qualitative methods, including work in Academy of Management Journal, Organisation Science, and Strategic Management Journal. His research focuses on organisational strategy, innovation, and qualitative methodology. Google Scholar profile
Olli Salo is a former Partner at McKinsey & Company where he spent 18 years helping clients understand the markets and themselves, develop winning strategies and improve their operating models. He has done over 1000 client interviews and published over 10 articles on McKinsey.com and beyond. LinkedIn profile
Sources
- HCAHPS Fact Sheet (CAHPS Hospital Survey), CMS, January 2025
- HCAHPS Summary Analyses - hcahpsonline.org
- HCAHPS 101: What HCAHPS Surveys mean for hospitals - Press Ganey
- CAHPS Health Plan Survey - Agency for Healthcare Research and Quality
- Friends and Family Test - NHS England
- Identifying Patient-Reported Care Experiences in Free-Text Survey Comments: Topic Modeling Study - JMIR Medical Informatics, 2025