Quick Answer
Manually theming 10,000 open-ended responses is a multi-week job most teams never finish. A batched AI classification pass does it in about an hour for the price of a coffee run.
What you get
- Theme 10,000 open-ended feedback items in ~1 hour instead of weeks of manual coding
- Get ranked themes with a count and 2-3 verbatim quotes each, so findings are defensible, not vibes
- Run a full 10k-item pass for roughly $8-25 in API usage on top of a $20/mo plan
- Turn the same feedback into an exec summary and a prioritized action list in one more pass
Step-by-Step Workflow
5 steps · 2 hours to set up · ~1 hour per analysis run ongoing
Workflow at a glance
5 steps · 2 hours setup
Consolidate data
Build codebook
ClaudeBatch classify
ClaudeValidate
Synthesize
ClaudeConsolidate data
Build codebook
Batch classify
Validate
Synthesize
- 1
Consolidate the feedback into one clean table
Export everything into a single sheet, one row per item, with columns for the raw text, the source (support, review, survey, sales note), and a date. Deduplicate and strip empty rows. Messy input is the single biggest cause of a bad analysis - garbage rows produce phantom themes.
Do not skip the source column. 'Slow' from a churned enterprise account and 'slow' from a free-tier trial are different problems, and you will want to slice themes by source at the end.
30 minOutput: One clean table: raw text, source, date, one row per itemTools: Google SheetsTip: Keep a stable row ID. When the AI returns labels, you join them back on the ID - without it, reconciling 10,000 labels to 10,000 rows becomes its own nightmare.
- 2
Build a codebook before you classify anything
The mistake that wrecks AI feedback analysis is letting the model invent a fresh category for every item - you end up with 800 near-duplicate themes and no signal. Instead, do an open pass first: have Claude read a random sample of 200-300 items and propose 12-20 candidate themes. You then edit that list into a fixed codebook with clear definitions.
This fixed codebook is what makes the run rigorous. Every one of the 10,000 items gets classified into your defined themes (with an 'other' bucket for genuine misfits), so counts are meaningful and comparable instead of a scatter of one-off labels.
Claude - the interface you'll work in for this step. Screenshot of the tool's own UI, not our results. 45 minOutput: A fixed codebook of 12-20 defined themes plus an 'other' bucketTools: ClaudeTip: Cap the codebook around 20 themes. More than that and humans cannot hold it in their head, and the model's accuracy per category drops. If two themes keep getting confused, merge them.
- 3
Run the batched classification pass
Apply the fixed codebook to all 10,000 items via the API. Prompt the model to return, for each item, the row ID, one primary theme from the codebook, an optional secondary theme, and a sentiment. Process in batches and write the results back to the sheet on the row ID.
Use a smaller, cheaper model for the labeling pass - the task is classification against a defined list, not deep reasoning, so a fast model does it accurately for a fraction of the cost. This is the step that takes about an hour of wall-clock time and $8-25 in usage for the full 10k.
~1 hr (mostly waiting on the batch)Output: All 10,000 items labeled with theme, sub-theme, and sentiment, joined on row IDTools: Claude, Google Sheets - 4
Validate before you trust the counts
Never present AI labels without a spot check. Pull a random 50-100 labeled items and read them yourself: does the assigned theme match the text? Track the agreement rate. Above ~90% agreement, the run is trustworthy. Below that, your codebook definitions are ambiguous - tighten them and re-run the disputed slice.
This 20-minute check is the difference between an analysis leadership acts on and one they quietly distrust after the first wrong quote. It is also your defense when someone challenges a finding.
20 minOutput: A measured agreement rate and any codebook fixes neededTools: Google SheetsTip: If the model over-uses 'other', that is a signal your codebook is missing a real theme, not that the feedback is noise. Read the 'other' pile and you will usually find one or two themes you should have defined.
- 5
Synthesize into themes, quotes, and actions
Now the payoff. Pivot the sheet to rank themes by count and slice by source and sentiment. Then feed the theme counts and a sample of items per theme back to Claude to write the deliverable: each theme with its count, its trend, 2-3 verbatim quotes, and a specific suggested action.
The verbatim quotes are non-negotiable - they turn 'customers want faster onboarding' into evidence a PM can act on and a skeptic cannot wave away. End with a one-page exec summary and a prioritized list, because a ranked analysis nobody reads changes nothing.
Claude - the interface you'll work in for this step. Screenshot of the tool's own UI, not our results. 30 minOutput: Ranked themes with counts, verbatim quotes, and a prioritized action listTools: Claude, Google Sheets
Every company sits on a pile of unread customer signal: support tickets, app-store reviews, NPS verbatims, churn-survey answers, sales-call notes. The reason it goes unread is not indifference - it is that theming ten thousand free-text responses by hand is genuinely a multi-week job, so it never gets prioritized and the pile grows. AI collapses that. With a fixed codebook and a batched classification pass, a single analyst turns 10,000 items into ranked themes, each with a count and real quotes, in about an hour. The hard part is not the AI - it is doing it rigorously enough that leadership trusts the output. This workflow is built around that rigor: a stable codebook, verbatim evidence, and a sanity check, so the result is a defensible analysis and not a confident-sounding hallucination.
Why the hook needs the API, not a chat window
Ten thousand items in an hour is a batch job, and it is worth being honest about the mechanics because a lot of AI-analysis advice quietly is not. You cannot paste 10,000 rows into a chat window - context limits and rate limits make that impossible, and doing it in a hundred manual chunks defeats the purpose. The real method is a fixed classification prompt applied to the data programmatically: through the API for a true 10k run (a short script or a batch endpoint), or through a smaller model for the labeling pass to keep costs down. A $20/mo Pro plan covers building the codebook and running small analyses interactively in Projects; the full-volume pass runs via the API and costs single-digit-to-low-double-digit dollars in usage. Anyone promising 10k items an hour from a chat box is describing something they have not run.
Stack cost breakdown
Public list prices as of July 2026. Optional tools are marked in the notes.
| Tool | Plan | Monthly cost | Notes |
|---|---|---|---|
| Claude | Pro | $20/mo | Required. The analysis engine. Interactive work in Projects; batch runs via the API. A 10k-item run adds ~$8-25 in usage depending on the model. |
| Google Sheets | Free | $0 | Required. Stages the raw data and holds the coded output for pivots and counts. Free with a Google account. |
| Zapier | Professional | $30/mo | Optional. Auto-pulls fresh feedback from support, reviews, and surveys into the sheet on a schedule so analysis becomes continuous. |
| Thematic | Custom | ~$500-2,000/mo | Alternative, not additive. Managed continuous feedback analytics for enterprise volumes; replaces the DIY engine rather than adding to it. |
| Dovetail | Free / Enterprise | $0 / custom | Alternative, not additive. Research repository with built-in AI chat and summaries; free to start, Enterprise is quote-based. |
| Total | $20 - $50/mo($20 required, $50 with optional tools) | ||
Email me this stack as a checklist
Every tool, the plan to pick, and the monthly cost - in your inbox.
Real usage
What people actually run
No usage reports yet - be the first to share what you run. Tell us your real stack, your actual monthly cost, and any tools you swapped.
The two prompts that do the work
Feedback analysis lives or dies on the codebook and the classification prompt. Keep the codebook fixed across the whole run so counts are comparable; force verbatim quotes so findings are defensible.
Codebook-generation prompt (run on a sample first)
Here are 250 randomly sampled pieces of customer feedback: {paste sample}.
Propose a codebook of 12-20 themes that would let me classify our full feedback set. For each theme give: a short name, a one-sentence definition, and 2 example items from the sample that belong to it. Themes must be distinct - if two would overlap, merge them. Do not create a theme that only one item fits. Return the list only; I will edit it before we classify everything.Note: You edit this into the final fixed codebook. The model proposes; you decide. A human-owned codebook is what keeps the analysis rigorous rather than a scatter of AI-invented labels.
Batch classification prompt (applied to every item)
Classify each feedback item below using ONLY this codebook: {paste your fixed themes with definitions}. If an item genuinely fits no theme, label it 'other' - do not invent new themes.
For each item return exactly: id | primary_theme | secondary_theme_or_blank | sentiment(positive/neutral/negative).
Items:
{id}: {text}
...
Return only the pipe-delimited rows, one per item, matching the input ids.Note: Run this over the data programmatically in batches, joining results back on id. Constraining the model to the fixed codebook and forbidding new themes is what makes 10,000 labels comparable.
Adjust for Your Situation
If you are non-technical and cannot run the API
You can still do meaningful analysis in Claude Projects without code, just not 10k at once. Upload the data in chunks of a few hundred rows, apply the fixed codebook, and paste results back to the sheet. It is slower and caps out well below 10,000, but for a few thousand items it works. For true volume without code, this is the case where a managed tool like Thematic or Dovetail earns its price.
If you need this continuously, not once
Add Zapier to pull new feedback into the sheet on a schedule and re-run the classification weekly against the same fixed codebook, so trends are comparable over time. If feedback volume is genuinely enterprise-scale and always-on, that is the point where a managed platform (Thematic, Enterpret) stops being an indulgence and starts being cheaper than the analyst hours you would otherwise spend maintaining the pipeline.
If findings must survive scrutiny from leadership or a board
Raise the validation bar: read a larger random sample (150-200 items), report the agreement rate explicitly in the deliverable, and attach the codebook so anyone can audit how a theme was defined. When money moves on the analysis, the verbatim quotes and the stated agreement rate are what make it defensible rather than 'the AI said so'.
Swap options
Drop-in substitutions if a tool does not fit your budget or stack. These trade cost or effort for the recommended setup.
| Swap out | Use instead | When |
|---|---|---|
| Claude | ChatGPT or Gemini | You already pay for one; the codebook-and-batch method is identical across models |
| Google Sheets | Airtable or a database | You need to operationalize themes into a live dashboard rather than a one-off analysis |
| Build the batch classification pass | Thematic or Enterpret (managed platforms) | You analyze feedback continuously at enterprise volume and want it maintained for you |
| Stage and pull the data | Dovetail as a research repository | You want findings, quotes, and highlights to live in a searchable repo the whole team uses |
Common Pitfalls
- Letting the model invent a new theme per item. You get hundreds of near-duplicate labels and zero signal. Fix a codebook first and forbid new themes in the classification prompt.
- Presenting AI labels without a spot check. Skipping validation is how one wrong quote in a board deck torches trust in the whole analysis. The 20-minute check is not optional.
- Dropping the verbatim quotes to save space. Counts without quotes read as opinion; a PM cannot act on 'people want it faster' but can act on the exact sentence three enterprise accounts wrote.
- Ignoring the source dimension. 'Slow' from a churned enterprise account and 'slow' from a free trial are different problems. Analysis that collapses sources into one pile hides the signal that actually matters.
- Claiming a chat window did 10k in an hour. It cannot. Be honest about the batch mechanic; the credibility of the whole analysis rests on the method being real.
When DIY analysis is the wrong tool
Stop hand-rolling this the moment feedback analysis becomes a permanent, high-frequency job rather than a periodic deep dive. The DIY stack is unbeatable for a quarterly analysis or a specific investigation - it is cheap, flexible, and yours. But if you are re-running it every week across many sources, maintaining the pipeline, and fielding constant asks to slice it differently, you are quietly rebuilding a product that already exists. At that point a managed platform like Thematic or Enterpret, at $500-2,000/mo, costs less than the analyst time you are spending and comes with integrations, dashboards, and trend tracking maintained for you. Use DIY for depth and one-offs; graduate to a platform when the work becomes always-on infrastructure.
Frequently Asked Questions
Can this really do 10,000 items in an hour, honestly?
How accurate is AI theming versus a human coder?
Why not just buy Thematic or Dovetail instead?
Do I need to know how to code?
What stops the AI from making up quotes or findings?
How we built this workflow
This workflow reflects codebook-driven text classification run on 2026 general-purpose models rather than a bespoke NLP pipeline. Prices are 2026 rates verified in July 2026: Claude Pro at $20/mo with per-run API usage of roughly $8-25 for a 10k-item pass on a small model, Google Sheets free, Zapier Professional at $29.99/mo, and managed alternatives (Thematic) at a custom ~$500-2,000/mo. Dovetail's public pricing in 2026 is a free tier plus custom Enterprise. The accuracy claims assume a fixed codebook and the human validation step; skip either and the numbers do not hold.
Last updated July 7, 2026; prices verified at publication.