AI Keyword Research That Beats the Old Playbook (2026)

What you get

• Build a 30-50 keyword cluster mapped to search intent in about 90 minutes
• Filter out the head terms AI Overviews already answer, before wasting a brief on them
• Run keyword research for $69/mo instead of the $140+/mo an all-in-one suite costs
• Find long-tail, low-difficulty queries with buying or tutorial intent that still convert clicks

Step-by-Step Workflow

6 steps · 2 hours to set up · ~90 min per topic cluster ongoing

Workflow at a glance

6 steps · 2 hours setup

120 min

AI seed topics

Claude

220 min

Ahrefs data

Ahrefs

320 min

Intent clustering

Claude

420 min

SERP recon

Perplexity

515 min

Filter dead terms

Claude

Perplexity

620 min

Map to pages

Claude

1

AI seed topics

20 minClaude

2

Ahrefs data

20 minAhrefs

3

Intent clustering

20 minClaude

4

SERP recon

20 minPerplexity

5

Filter dead terms

15 minClaude · Perplexity

6

Map to pages

20 minClaude

1. 1

   Generate seed topics with AI, not a keyword tool

   Start from the customer, not the database. Ask Claude: 'I sell [product] to [audience]. List 15 problems they search for, phrased the way they would actually type them, grouped by buying stage (unaware, problem-aware, solution-aware, ready-to-buy).' This produces intent-rich seeds a volume-sorted export never surfaces, because the best long-tail queries have too little volume to rank in a standard tool's suggestions.

   

   20 minOutput: 15-20 seed topics grouped by buying stageTools: Claude

   Tip: The ready-to-buy and solution-aware seeds are worth 10x the unaware ones. Those are where AI Overviews are weakest and clicks still convert.

2. 2

   Pull volume and difficulty for the seeds

   Drop your seeds into Ahrefs Keywords Explorer and export the matching-terms and related-terms with volume and keyword difficulty (KD). Starter tier limits rows, so work one seed cluster at a time rather than dumping everything at once. Filter to KD under 30 and volume over 200. You are gathering raw data here, not making decisions yet.

   

   20 minOutput: A raw keyword list with volume and KD per seedTools: Ahrefs

   Tip: Do not sort by volume and stop there - that is the old playbook. Volume is one input; intent and AI Overview presence matter more for actual clicks.

3. 3

   Cluster by intent with Claude

   Paste the raw keyword list into Claude: 'Cluster these keywords by search intent, not by keyword similarity. For each cluster give it a label, the primary intent (informational, commercial, transactional), the single best target keyword, and the page type that should target it (guide, comparison, tool, product).' This replaces the clustering feature you used to pay a suite for, and it groups by what the searcher wants rather than by matching words.

   

   20 minOutput: 30-50 keywords grouped into intent clusters with a target page type eachTools: Claude

   Tip: For thousands of keywords, hand this step to Keyword Insights ($58/mo). For a few hundred, Claude does it well for free.

4. 4

   Recon the SERP and AI Overview for each target

   For each cluster's target keyword, run it through Perplexity to see what the live SERP and AI Overview currently say. Prompt: 'For the query [keyword], summarize what the top results and any AI Overview cover, and what specific angle or data they all miss. Cite the pages.' This tells you whether the query is already fully answered (skip it) or has a gap you can win (target it).

   

   20 minOutput: A go/no-go and content angle for each target keywordTools: Perplexity
5. 5

   Kill the terms AI Overviews already own

   Cut any target where the AI Overview fully and specifically answers the query - those send almost no clicks now regardless of your rank. Keep tutorial ('how to'), comparison ('X vs Y'), decision ('best X for Y'), and anything needing current data, tested specifics, or a real opinion. This filter is the entire difference between the new playbook and the old one.

   

   15 minOutput: A pruned keyword map of only click-worthy targetsTools: Claude, Perplexity

   Tip: A query where the AI Overview is present but shallow is the sweet spot: it proves demand, and a page with specifics can get cited inside the Overview itself.

6. 6

   Assign each cluster a page and priority

   Turn the surviving clusters into a build queue. For each: the target keyword, the page type, the angle from SERP recon, and a priority score (KD, intent value, and gap size). Prioritize solution-aware and ready-to-buy clusters with low KD. This is your content roadmap, derived from intent and opportunity rather than a volume sort.

   20 minOutput: A prioritized content roadmap mapped to keyword clustersTools: Claude

Why volume-first keyword research stopped working

Two things broke the old model. First, AI Overviews now fully answer broad and definitional queries, so the head terms that topped volume-sorted lists send collapsing click-through. Second, LLMs made clustering and intent classification - the parts that used to require a $140/mo tool's built-in features - something you can do better with a $20/mo model. The expensive suite's remaining edge is the raw index, and you can rent a thin slice of that for $29/mo on Ahrefs Starter.