How does ChatGPT actually work? LLMs, reasoning and AI agents explained

How ChatGPT works under the hood: next-token prediction, reasoning models, tools and AI agents, and what happens when you ask it to analyse your data.

Cover Image for How does ChatGPT actually work? LLMs, reasoning and AI agents explained
Share this article:

ChatGPT works by predicting text one token (roughly a word fragment) at a time, using patterns learned from training on huge amounts of text. Modern versions add three layers on top: hidden reasoning before answering, tools such as web search and a Python sandbox, and agent loops that plan, act and check results. None of them reads your data systematically unless the workflow forces it to.

You have probably used ChatGPT, Claude, Gemini or Copilot hundreds of times. You type a question, press enter, and an answer appears that often sounds remarkably intelligent. Since we first wrote this article, the tools have changed a lot: they now "think" before answering, browse the web, run code and complete multi-step tasks as agents. What happens when you hit enter has become more interesting, and more important to understand.

This guide explains how large language models (LLMs) actually work, what reasoning models and AI agents add, and what happens when you ask one to analyse your interview transcripts, survey responses or spreadsheets. We have discussed parts of this on Skimlecast Episode 6 and Episode 8, and written about the challenges of using ChatGPT for analysing qualitative data. Here we go deeper, and finish with practical rules for using these tools well. If you are new to AI tools and want to see how they apply to your own work, see how Skimle fits curious professionals.

How does ChatGPT work in two minutes?

The two-minute video below walks through the whole picture: tokens, training, context windows, reasoning, agents and what happens when you ask ChatGPT to analyse your data.

A modern AI assistant is best understood as five layers stacked on top of each other. Most confusion comes from mixing them up.

LayerWhat it doesWhat it is not
The language modelPredicts the next token, over and over, based on everything in the conversationA database, a search engine or a mind
ReasoningGenerates hidden working-out tokens before the visible answerA separate logic engine; it is still token prediction
ToolsLets the model call a search engine, a Python sandbox, a file reader or an external appSomething the model does by itself; the product runs the tool and feeds back the result
The agent loopRepeats plan, act, observe until a goal is metGuaranteed to be thorough; it decides when it has done enough
Product featuresMemory, projects, uploaded files, connectorsLearning; these are notes and documents fed back into the prompt

The rest of this article takes each layer in turn and then looks at what they mean for analysing data.

What happens when you press enter?

At its core, a large language model has one job: predict the next token. A token is a word or part of a word; "analysing" might be split into "analys" and "ing". Here is the full sequence when you send a message:

  1. Everything in the conversation (the system instructions, your earlier messages, any pasted or retrieved documents and your new question) is turned into one long sequence of tokens.
  2. The model calculates a probability for every possible next token and picks one, with a little randomness.
  3. That token is appended to the sequence.
  4. The model reads the whole, slightly longer sequence again and predicts the next token.
  5. This repeats until the model produces a token that means "stop".

The model has no other channel. Every answer, every "thinking" trace, every tool call and every summary of your data is produced through this loop, one token at a time.

Diagram of how ChatGPT works: the conversation becomes tokens, the model predicts one token, appends it and repeats

Attention decides what the model looks at

The breakthrough behind modern LLMs is the attention mechanism, introduced with the transformer architecture in 2017. For every token it generates, the model weighs how relevant each earlier token is. If you ask about "pricing complaints from retail customers", the tokens about pricing and retail get more weight than the rest of the conversation.

Attention is what makes LLMs seem to remember earlier parts of a chat. They do not remember anything: the entire conversation is fed back in every time, and attention picks out what matters. That also explains why long conversations get slower and less reliable. More tokens compete for attention on every step.

The context window is the model's whole world

The context window is the maximum number of tokens the model can read at once. Current models accept anywhere from about 128,000 tokens to over a million, which sounds like plenty: 128,000 tokens is roughly 100,000 words, or about 15 hour-long interview transcripts.

A large window does not mean everything in it is used equally well. The well-known "Lost in the Middle" study found that models use information at the beginning and end of a long input far better than information in the middle, with multi-document question answering accuracy dropping by more than 30% when the relevant document moved from the start to the middle (Liu et al., 2024). Chroma's 2025 "Context Rot" study tested 18 frontier models and found that every one of them became less reliable as input length grew, even on simple retrieval tasks (Chroma Research, 2025).

How is a model like ChatGPT trained?

Training happens in stages, and it all happens before you ever talk to the model.

  1. Pre-training. The model reads a huge collection of text (books, websites, code, articles) and learns to predict the next token. After seeing "The capital of Finland is" followed by "Helsinki" thousands of times, it learns that pattern, and billions of far subtler ones about how arguments, explanations and analyses are structured. This is why "fancy autocomplete" undersells it: predicting text well at this scale produces surprisingly capable behaviour.
  2. Supervised fine-tuning. The model is trained on curated examples of good assistant behaviour, so it answers questions rather than continuing your text in random directions.
  3. Reinforcement learning. Humans, and increasingly automated graders, score the model's answers, and the model is adjusted so high-scoring answers become more likely. Reinforcement learning from human feedback (RLHF) is why assistants are helpful and polite, and also why they can be sycophantic. Reasoning models are additionally trained with rewards for getting checkable answers right, such as maths problems and code that passes tests.

Then the model is frozen. The ChatGPT you talk to today is a fixed snapshot. It does not learn from your conversation, however many corrections you give it.

What changed with reasoning models?

When this article first appeared, we wrote that a message like "Thinking..." was just more predicted text. For today's reasoning models that is no longer the whole story. Since OpenAI's o1 in 2024, and now in most leading models from OpenAI, Anthropic and Google, the model generates a stream of reasoning tokens before its visible answer: it breaks the problem down, tries approaches and checks its own work (OpenAI, 2024). On maths, logic and coding problems this makes a large difference.

Three things are worth knowing about it:

  • Reasoning is still next-token prediction. The working-out is produced by the same loop described above. It is trained to be useful, but it is text, not a separate logic engine.
  • The visible summary may not be the real reasoning. Anthropic found that reasoning models often fail to mention information that actually shaped their answer: in their tests, Claude 3.7 Sonnet mentioned a hint it had been given only 25% of the time on average (Anthropic, 2025). Treat the "thought process" as a helpful explanation, not an audit trail.
  • More thinking is not more reading. A model that reasons for two minutes about 40 interview transcripts is still reasoning over whatever the transcripts look like in its context, with the attention limits described above. Reasoning improves logic. It does not make the model check every document.

What is an AI agent, and how is it different from a chatbot?

A chatbot answers. An agent pursues a goal. Anthropic describes agents as systems where LLMs "dynamically direct their own processes and tool usage", as opposed to fixed workflows where code decides the steps (Anthropic, 2024).

The mechanics are simple once you know about tokens. The model is told which tools exist (search the web, run Python, read a file, query a database, send an email). When it wants to use one, it outputs a specially formatted tool call instead of an answer. The surrounding software runs the tool and pastes the result back into the conversation. The model reads the result and decides what to do next. That cycle is the agent loop:

  1. Plan: decide the next step towards the goal.
  2. Act: call a tool.
  3. Observe: read what the tool returned.
  4. Repeat until the model judges the goal is met, then answer.

Diagram comparing a chatbot, which answers in one pass, with an AI agent, which loops through plan, act and observe using tools

This is what powers ChatGPT's agent mode and deep research, Claude Code and Codex for software, and the agentic features appearing in most business tools. The Model Context Protocol (MCP), an open standard introduced by Anthropic in 2024 and now supported across the major assistants, lets any app offer its tools to any agent in the same way (modelcontextprotocol.io). That is how an assistant can query your CRM, your files or your qualitative analysis in Skimle.

Agents are a real step forward because tools replace guessing with doing. A model that calls a calculator gets the right number; a model that searches gets real pages. The weak point is the stopping decision: the model, not you, decides when it has searched enough, read enough or checked enough. A deep research agent that reads 30 sources has not read the other 3,000, and it will still fill gaps with plausible text from training.

How does ChatGPT analyse your data?

When you ask ChatGPT to "analyse" something, one of four quite different mechanisms is doing the work, depending on what you gave it. Knowing which one is running tells you how far to trust the result.

What you give itWhat actually happensGood forWatch out for
Text pasted into the chatThe whole text goes into the context window and the model reads it in one passShort documents, a handful of interviewsLost-in-the-middle effects and context rot as length grows
Uploaded documents (PDF, Word)Long files are split into chunks and searched; only the passages that match your question are placed in contextFinding a specific fact in a long documentThe model only sees the retrieved chunks, not the whole file, so "what are all the themes?" is answered from a sample
Spreadsheets and CSV filesThe model writes Python code and runs it in a sandbox, then reads the output (OpenAI)Counting, statistics and charts on numeric columnsOpen-text columns are usually summarised from samples or keyword counts, not read row by row
Agent mode or deep researchThe agent loops through searches, page reads and code runsDesk research and multi-step tasksThe agent decides when to stop; coverage is whatever it chose to look at

The spreadsheet row is the most reassuring one. When ChatGPT computes an average or a cross-tab with Python, the arithmetic is real, and you can ask to see the code. The uploaded-document row is the most misunderstood. Many people believe ChatGPT has "read all 50 interviews" when it has retrieved a dozen passages that looked relevant to the wording of their question.

Seven misconceptions about ChatGPT, updated for 2026

1. ChatGPT stores my data in a smart database

Language models are not databases. The model itself stores nothing from your conversation. Products now add project files and memory, but these work by retrieving text and pasting it back into the prompt. Nothing is being organised into a structured representation of your data that the model can query reliably.

2. The "thinking" panel shows how it reached the answer

Reasoning tokens are real, but the displayed reasoning is a summary that can leave out what actually drove the answer, as the Anthropic study above showed. It is a useful window, not proof.

3. It learns from my corrections

Chatting is not training. We covered this at length in Skimlecast Episode 4. If you tell ChatGPT "that theme should be split in two", the model has learned nothing. Memory features may save a note about your preference, which is fed back into future prompts, but the model will make the same kind of mistake in the next analysis unless the note happens to cover it.

4. It understands meaning the way people do

LLMs are extraordinarily capable pattern learners, and whether they "understand" is still debated. In practice they often catch sarcasm, hedging and implied meaning, and sometimes miss them badly. When you analyse interview transcripts yourself, you notice when an offhand remark reveals something important. A model may notice too, but you cannot tell from the fluent output whether it did.

5. Newer models are always better

Not for every task. A study of 4,900 AI-generated summaries of scientific research found that leading models overgeneralised findings in 26-73% of cases depending on the model, were nearly five times more likely than human-written summaries to make broad generalisations, and that some newer models performed worse than older ones (Peters and Chin-Yee, 2025). Asking for accuracy did not fix it.

6. A bigger context window means it reads everything

As the Lost in the Middle and Context Rot studies show, a model can accept a million tokens and still pay uneven attention to them. Fitting your data into the window is necessary for a complete analysis, but it is not sufficient.

7. An agent that searched the web has checked everything

Deep research and agent modes visit the sources they choose, usually the top results, within a time and step budget. Their reports are much better than answers from memory, but they are not systematic reviews, and the citations still need checking.

Why do LLMs struggle with rigorous analysis?

Understanding the mechanics explains the failure modes researchers keep running into when they use ChatGPT or Copilot for qualitative analysis.

  • Every answer is generated from scratch. There is no persistent coding frame. Ask "what were the main themes again?" and you get a fresh prediction, which may differ from the first one.
  • Plausible beats accurate. The model is trained to produce text that looks like a good answer. Summaries drift towards confident, general claims that go beyond the data, which is exactly the overgeneralisation the Peters and Chin-Yee study measured.
  • Sycophancy. Models lean towards agreeing with you. If you discussed a theory earlier in the chat, the model will often bridge from it to your data, even when the data does not support the link. This makes confirmation bias, already the main threat to rigour in thematic analysis, worse.
  • Coverage is invisible. A summary of 100 survey responses looks the same whether the model attended to all of them or to 30. Nothing in the output tells you which.
  • No audit trail. Without quotes linked to their sources, you cannot show a reviewer, a client or yourself where a finding came from.

How should you use ChatGPT and AI agents for analysis?

None of this means you should avoid general-purpose AI. It means you should use it in ways that work with the mechanics rather than against them. Eight rules we use ourselves:

  1. Keep inputs small and complete. Work through documents a few at a time rather than uploading 50 and asking for "the themes". A small input the model can read in full beats a large one it samples.
  2. Ask for verbatim quotes with their source, and check some. A finding without a quote is an opinion. Spot-check quotes against the original, because models sometimes paraphrase while presenting text as a quotation.
  3. Ask for disconfirming evidence. Follow every summary with "which responses contradict this?" to counter sycophancy and overgeneralisation.
  4. Use code for counting. For anything numeric, ask the assistant to compute it with Python and show the code. Let the model interpret, not tally.
  5. Fix the coding frame in writing. Paste the same category definitions into every request, so you are not relying on the model to remember them.
  6. Run it twice. If two runs of the same request produce different themes, the result is not yet stable enough to report.
  7. Start fresh conversations. A long chat full of earlier ideas contaminates later answers. New task, new chat.
  8. Use a purpose-built harness for anything you will publish or act on. For a quick look at three interviews, a chat is fine. For a study you need to defend, use a tool designed for systematic coverage and traceability. Our ChatGPT prompts for qualitative analysis show how far careful prompting can take you.

Why does the harness matter more than the model?

The same model can be a mediocre chatbot or an excellent specialist tool, depending on the software around it. That surrounding software is often called the harness: the tools, memory, workflow and checks that decide what the model sees and what happens to its output.

Claude Code is a good example. The model inside it is the same family you can chat with, but the harness gives it tools to read and edit files, run tests and check its own work, so it can change a real codebase rather than describe changes. In software, the harness is what turned language models from autocomplete into colleagues.

How Skimle builds a harness for qualitative analysis

We built Skimle because we tried every chat-style tool on interview transcripts for research and public consultation statements, and found that none had the harness rigorous analysis needs. Skimle uses LLMs for what they do well, small and well-defined reading tasks, and handles everything else in a structured system:

  • Systematic processing that reads every document one by one, so coverage does not depend on retrieval or attention.
  • Structured storage of categories, insights and quote-to-category links outside the model, so the coding frame persists and does not drift between questions.
  • Two-way transparency, so you can trace every insight to its source quotes and see every quote behind a category.
  • Human control to merge, split, rename and review categories, so analytical decisions stay with the researcher.
  • Agents on top of structure. Agentic analysis and the in-app agentic chat work over the coded data rather than raw text, and through MCP your own AI assistant can query the same structured analysis.

Skimle structured qualitative analysis showing categories built document by document, with each insight linked to its source quotes

The screenshot above shows the difference in practice: themes held as a structure, with counts and quotes behind each one, rather than a paragraph generated from whatever was in context. You can read more about why structure beats dumping documents into embeddings, how Skimle handles hallucinations and context limits and why two-way transparency matters.

Frequently asked questions

How does ChatGPT generate its answers?

ChatGPT generates answers by predicting one token at a time. It reads the whole conversation, predicts the most suitable next token, appends it and repeats until the answer is complete. Reasoning models first generate hidden working-out tokens, and agentic modes can pause to call tools such as web search or Python before continuing.

Does ChatGPT read all of my uploaded documents?

Not necessarily. Short texts fit in the context window and are read in one pass. Long uploaded documents are usually split into chunks, and only the chunks that match your question are passed to the model. Spreadsheets are analysed by running Python code, which handles numbers well but usually samples open-text columns.

What is the difference between ChatGPT and an AI agent?

A chatbot answers in one pass from what is in the conversation. An AI agent works towards a goal in a loop: it plans a step, calls a tool such as search or code execution, reads the result and decides what to do next. ChatGPT itself now includes agent features, such as agent mode and deep research.

Does ChatGPT learn from my conversations?

No. The model is frozen after training and does not change when you correct it. Memory features save short notes that are added to future prompts, and your conversations may be used to train future models depending on your settings, but the model you are talking to does not learn during the chat.

Can ChatGPT analyse qualitative data reliably?

For a few short documents, with careful prompting and checked quotes, it can help. For larger datasets it cannot guarantee that every document was read, that the coding frame stays consistent or that every finding traces to a source. Purpose-built tools such as Skimle add that structure around the language model.

The bottom line

ChatGPT and its rivals are remarkable, and they keep improving: reasoning makes them better at logic, tools make them better at facts, and agent loops let them complete real tasks. Underneath, they still predict text one token at a time over whatever is in their context, and they still decide for themselves when they have done enough.

For drafting, brainstorming and explaining, a chat window is fine. For analysing interviews, open-ended survey responses or expert feedback, ask four questions of any AI tool: did it read everything, can I trace each finding to its source, does it give the same answer twice, and who decides the categories? Our guide to AI for data analysis with text and unstructured data and our comparison of qualitative analysis tools show what good answers look like. These questions separate AI-assisted analysis from AI slop, and help keep your own thinking in the loop.


Ready to analyse your qualitative data with a harness built for serious work? Try Skimle for free and see every insight linked back to the quotes it came from.

About the authors

Henri Schildt is a Professor of Strategy at Aalto University School of Business and co-founder of Skimle. He has published over a dozen peer-reviewed articles using qualitative methods, including work in Academy of Management Journal, Organisation Science, and Strategic Management Journal. His research focuses on organisational strategy, innovation, and qualitative methodology. Google Scholar profile

Olli Salo is a former Partner at McKinsey & Company where he spent 18 years helping clients understand the markets and themselves, develop winning strategies and improve their operating models. He has done over 1000 client interviews and published over 10 articles on McKinsey.com and beyond. LinkedIn profile

Sources and further reading

Note: Skimle articles may contain text, videos and images created, edited or translated using AI tools.

Dig deeper to your data with Skimle

Skimle collects, analyses and categorises interviews, survey responses, reports and other qualitative data automatically. Our modern qualitative analysis software combines a rigorous and transparent workflow with the speed of AI.

Upload text or audio, remove sensitive data with Skimle Anonymise, automatically create categories and sub-categories, explore the data across documents and export the data to seamlessly fit your workflow. Built by professionals for professionals, with full privacy and GDPR compliance.

Free trial · No credit card required · Full plans from €20/month