---
title: "Did Your Page Change Earn an AI Citation? Run a Control-Group Test"
description: "A six-week protocol for testing whether a page change earned citations in AI Overviews, ChatGPT, Perplexity, or Gemini—or whether the results were noise."
url: "https://groas.com/post/did-that-page-change-get-you-cited-or-di"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/dddf07f5-f281-4bdd-b584-4b7df48c1cf3.png"
published: "2026-10-16T05:48:02.139Z"
modified: "2026-10-16T05:48:02.213Z"
---

October 16, 2026 · 11 min read

# Did Your Page Change Earn an AI Citation? Run a Control-Group Test

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![Tiny harbour model: a crew cheers in a freshly painted red boat, but the untouched boat beside it rose on the same tide, so the paint earned no credit.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/dddf07f5-f281-4bdd-b584-4b7df48c1cf3.png)

In this article

1. [Write the question before you touch the page](#write-the-question-before-you-touch-the-page)
2. [Build a prompt set and two comparable page groups](#build-a-prompt-set-and-two-comparable-page-groups)
3. [Collect a two-week baseline, not a screenshot](#collect-a-two-week-baseline-not-a-screenshot)
4. [Make one change, then leave both groups alone](#make-one-change-then-leave-both-groups-alone)
5. [Measure the lift after subtracting the control](#measure-the-lift-after-subtracting-the-control)
6. [Read the result without moving the goalposts](#read-the-result-without-moving-the-goalposts)
7. [Skip this protocol when you cannot protect the comparison](#skip-this-protocol-when-you-cannot-protect-the-comparison)
8. [Decide what each outcome changes](#decide-what-each-outcome-changes)

You rewrite a pricing page on Tuesday. By Friday, someone drops a ChatGPT screenshot in Slack: your page is cited. The rewrite gets credit before anyone asks whether the same answer would have appeared without it.

I have made that mistake with ad copy. For AI citations, I use the same correction I learned there: **a control group, repeated samples, and a threshold set before I look at the result**. One screenshot is one draw from a slot machine. Ask the same buyer question twice and the sources can change. In [repeat runs of identical prompts](https://dev.to/robin_geo/one-ai-citation-check-is-a-sample-how-much-answers-change-between-identical-runs-and-how-to-log-e1i), overlap between two runs was 0.664 for OpenAI, 0.515 for Claude, and 0.319 for Gemini. On 10 buyer price questions asked twice, Google AI Overviews re-cited only 32 of 51 pages; ChatGPT kept just 5 of 17.

The question for this test is not _Can I get a screenshot?_ It is _Did my change make a citation more likely than it would have been anyway?_ Here is how to find out.

## Write the question before you touch the page

Put this in your test sheet: _If I change page A, will engines cite page A more often for prompt P than if I had left it alone?_ The last clause is the test. I have seen ad-copy variants both improve when demand rose that week. Without an unchanged group, the calendar gets mistaken for creative genius.

There is a useful precedent in [Ahrefs’ schema test](https://www.seroundtable.com/study-schema-citations-study-41311.html): 1,885 pages added JSON-LD while 4,000 matched pages did not. AI Overviews citations fell 4.6% for the treated group, while movement in AI Mode and ChatGPT looked like noise. The control pages made that result readable. A before-and-after screenshot would not have.

The problem is especially sharp with AI answers. [AI Overviews expand a visible query into multiple sub-queries](https://askairank.com/blog/how-google-ai-overviews-pick-sources), pulling candidates for questions such as what a product is, what it costs, and who it suits. The mix can shift between runs. Location, personalization, and engine updates can shift it again. If treated and untouched pages both rise, do not hand your rewrite the trophy. **The untouched pages tell you what happened without your change.**

Practical rule: write down what would count as evidence against your idea, not just evidence for it.

## Build a prompt set and two comparable page groups

Start with **20 to 30 buyer-language prompts**, not the phrases from your marketing deck. Pull wording from sales calls, support tickets, and questions prospects ask when price, fit, comparison, or risk matters. Practitioners who audit visibility recommend [25 to 40 real buyer prompts across problem, comparative, and brand categories](https://wellows.com/blog/audit-brand-visibility-on-llms/). Keep the wording identical throughout the test.

Give every prompt a row with its ID, exact text, target page, and engines. If you cannot say which page on your site ought to answer a prompt, cut the prompt. A broad question might produce an interesting answer, but it will not give you a clean verdict on a particular page change.

Now make two matched piles of pages:

- **Treatment:** pages that receive the change.
- **Control:** pages that remain untouched.

Pair pages by topic and current visibility. That is the same principle used in [SEO split tests](https://shopify.com/blog/seo-ab-testing): compare pages with similar intent and authority rather than asking a pricing page to stand in for an unrelated blog post. If you have 24 mapped service and pricing pages, put 12 in each pile, paired by the kind of question they answer. Do not put every page you already expect to win in treatment.

Choose the engines now: AI Overviews, ChatGPT, Perplexity, and Gemini if those are the surfaces you intend to judge. Record the split before collecting post-change results. Future you will be remarkably good at explaining why one awkward page never really belonged in the test. Do not give future you the option.

Practical rule: **lock the prompts, engines, page pairs, and treatment assignment before baseline sampling begins.**

## Collect a two-week baseline, not a screenshot

Run each prompt **three times per engine per session, twice a week for two weeks**. That is 12 runs per prompt per engine before the edit. Use the same prompt wording and engine each time, starting a fresh session and using a private window where you can. Log every run, including the dull ones where nothing cites anything.

The [identical-prompt tests](https://dev.to/robin_geo/one-ai-citation-check-is-a-sample-how-much-answers-change-between-identical-runs-and-how-to-log-e1i) are the reason for that repetition. Their 1,200-question API test reported Jaccard overlap of 0.664 for OpenAI and 0.319 for Gemini across two runs. Their practical advice was to run each prompt two or three times per engine per session and report rates rather than screenshots. You are trying to measure how often something happens, not catch it happening once.

Use **one row per run**. Record:

- Date, engine, prompt ID, and run number.
- Whether the answer included any source citations.
- Whether your brand appeared in the answer text.
- Whether your target URL appeared in the cited sources.
- Which URL was cited, and whether any figures in the answer were correct.

Keep _mention_ and _citation_ separate. A mention names your brand in the answer. A citation points to your URL in the sources. You can have one without the other, and calling both a win makes the sheet useless.

Decide how you will handle answers with no citations at all, then keep that rule fixed. I log them separately as **uncited answers**, rather than treating them as evidence that another page beat mine. For a citation rate, count target-URL citations against answers that supplied citations, and report the uncited count alongside it. That way, a shift toward fewer sourced answers cannot hide inside an apparently improving rate. In your baseline notes, write down both counts, not just a percentage that loses its denominator when copied into a slide.

A page cited in 3 of 12 eligible runs has a 25% baseline citation rate. It does not have a permanent seat in the answer. Practical rule: if you cannot state the baseline rate and what went into its denominator, you are not ready to edit.

## Make one change, then leave both groups alone

Pick **one intervention** and apply it only to treatment pages. My default is a standalone answer block near the top of each page: 40 to 60 words that answer its mapped prompt directly, including a current number, date, or condition where the answer calls for one. A passage that makes sense on its own has a better shot when an engine lifts it into an answer than a paragraph that depends on three earlier paragraphs for context.

Do not pair that block with a new title, schema push, internal-link campaign, and page redesign. If citations rise, you will not know which change mattered. The ordinary [split-testing rule applies](https://shopify.com/blog/seo-ab-testing): change one variable and decide what counts as success before measuring it.

Record the date and hour you publish the treatment. Freeze the control pages for the four-week post-change period. No copy tweaks, new internal links, or quiet title tests. Leave the treatment pages alone too, apart from the planned edit. The point is to compare a changed group with a group that could have changed but did not.

I used to tell clients two weeks was enough to notice a page change in Google. That is not how I would call an AI-citation test. Citation pools can update on a different timetable from the index; a day-10 reading can look different at day 28. Keep sampling while the engines have time to recrawl, rather than calling an early fluctuation a result.

Practical rule: **if someone edits a control page mid-test, log the breach and do not pretend the original comparison survived.**

## Measure the lift after subtracting the control

For four weeks after publication, sample on the same schedule: identical prompts, engines, sessions, and three-run routine. Calculate a citation rate for each group using the denominator rule you set during baseline. Keep the uncited-answer count visible beside it.

The primary measure is **the treatment lift minus the control lift**:

1. Subtract the treatment group’s baseline citation rate from its post-change rate.
2. Do the same for the control group.
3. Subtract the control lift from the treatment lift.

Suppose treated pages move from 25% to 45%, a 20-point gain, while controls move from 25% to 30%, a 5-point gain. The net lift is 15 points. That second subtraction is the reason you ran the test. It helps separate your edit from a broader shift that affected both groups. It will not make noisy answers deterministic; it gives you a comparison worth reading.

Then inspect your own server or edge logs for a mechanism check. Filter for [GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, and PerplexityBot](https://siteseoaiaudit.com/ai-crawlers-server-logs/). Check the exact treated URLs, status codes, and published IP ranges, because a user-agent can be faked. You want verified 200 responses rather than 403s, 429s, or 5xx errors. Analytics alone will not show you this traffic.

Log two secondary signals: verified answer-bot fetches of treated pages and time to first citation for each prompt. Fetches rising in week two before citations rise in week three would be consistent with the mechanism you expected. **A fetch does not prove that the engine used your passage; an absent fetch in your logs does not, by itself, prove where a citation came from.** Logs help you catch access problems and check whether the story behind a claimed win holds together. They are not a second citation counter.

Practical rule: report the net lift first, then use the logs to challenge your explanation of it.

## Read the result without moving the goalposts

Set the threshold before opening the post-change sheet. Mine is **a net 10-point citation-rate lift for treated pages over controls, on at least two engines, sustained for two weeks**. I also want verified 200 fetches on the treated URLs before I attribute that result to the intended path. Anything less, I call inconclusive and test again.

Then read the pattern:

- **Treated up, control flat:** This is the result you were looking for. Check that the cited URL is the page you changed, that the lift clears your pre-set threshold, and that the logs do not show an access problem. Then consider extending the change.
- **Both up:** The broader citation environment may have moved. The control group has stopped you from taking credit for a rising tide. Do not roll out the edit on this evidence alone.
- **Nothing moved:** Check access first, then the number of eligible answers, then the passage. No verified 200s from the relevant bots points toward a plumbing problem. A five-point wobble across a handful of runs is not a reason to rewrite the site.

A mixed result deserves an equally narrow claim. If Perplexity moves and AI Overviews does not, keep the Perplexity result; do not sell the edit as a universal fix. If nothing moves despite verified fetches, the passage may have been available without being selected. A tighter answer block is a reasonable next intervention, but that is a new test, not a retroactive explanation that turns this one into a win.

Practical rule: **a result belongs to the prompts, pages, engines, and period you actually tested.** Do not let a good screenshot speak for the rest.

## Skip this protocol when you cannot protect the comparison

There are three cases where I would not start this test yet:

- **Too few comparable pages.** With fewer than 10 pages that map cleanly to buyer prompts, matched pairs become hard to build and one odd result can swing the rate.
- **Unstable questions.** News-driven or local-pack answers can change because of a competitor launch, a review wave, or a local update. Your page edit can disappear under that movement.
- **Controls nobody can leave alone.** If the team cannot freeze control pages for four weeks, the comparison breaks as soon as an unlogged title tweak ships.

That last one sounds administrative until a weekly ship cycle meets a test sheet. Say you spend $20k a month and have a ten-person marketing team. Someone will have a perfectly sensible reason to edit a control page. Give one owner the authority to say no or to log the change and stop the test. I let [groas](https://groas.com/for-businesses) handle the ongoing search work for clients; the experimental split still needs a person accountable for keeping it intact.

Practical rule: if you cannot protect the controls, fix that before you collect a baseline. More sampling will not repair a broken comparison.

## Decide what each outcome changes

Before baseline week one, write your decision rule in the sheet: _I will call this a win only if treated pages beat controls by a net 10 points of citation rate, on at least two engines, sustained for two weeks, with verified 200 fetches in the logs._ Keep the uncited-answer counts beside those rates so you can see what you actually measured.

A win means you can roll the answer blocks to the control pages and choose the next change. A null result with fetches means you can try a tighter passage next cycle. If both groups rise, leave the edit uncredited and wait for a cleaner read. This is the control-group habit I kept from ad copy, applied to answers that rewrite themselves daily.

Dice land. Baselines shift. **Do not spend budget scaling a screenshot until you know which one you saw.**

![Small tabletop lab with treated and untouched control pages in separate groups, with dice beside them](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/e799f20e-7e29-4419-9f2c-04745fddb6a5.png)

![Pen-and-ink cartoon of a marketer studying one framed citation screenshot while repeated test logs pile up behind](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/6f7894d7-e0dd-4ae1-8601-f549218e940b.png)

## Frequently Asked Questions

### Why isn't a single ChatGPT screenshot enough proof that my page change worked?

A single screenshot is one draw from a slot machine. Asking the same buyer question twice can return different sources, with overlap between two identical runs measured at 0.664 for OpenAI, 0.515 for Claude, and 0.319 for Gemini. A control-group test with repeated samples is needed instead.

### How do I test whether a page edit caused an increase in AI citations?

Write the question as: if I change page A, will engines cite page A more often for prompt P than if I had left it alone? Run a treatment group and an unchanged control group, then compare citation rates between them rather than judging a before-and-after screenshot.

### What result should I treat as a win in a citation test?

Set the threshold before looking at results. A win is a net 10-point citation-rate lift for treated pages over controls, on at least two engines, sustained for two weeks, with verified 200 fetches on the treated URLs. Anything less should be called inconclusive and tested again.

### What does it mean if both my treated and control pages get more AI citations?

It suggests the broader citation environment moved, and the control group has stopped you from taking credit for a rising tide. Do not roll out the edit on this evidence alone; wait for a cleaner read.

### If Perplexity started citing my page but AI Overviews did not, can I claim the edit worked everywhere?

No. A result belongs to the prompts, pages, engines, and period you actually tested. Keep the Perplexity result and do not sell the edit as a universal fix for all engines.

### When should I not run a control-group citation test?

Skip it when you have fewer than 10 pages that map cleanly to buyer prompts, when the questions are unstable such as news-driven or local-pack answers, or when the team cannot freeze control pages for four weeks. If you cannot protect the controls, fix that before collecting a baseline.

### How do I stop teammates from editing my control pages during the test?

Give one owner the authority to say no to edits or to log a change and stop the test. In an active ship cycle, someone will always have a sensible reason to tweak a control page, and an unlogged title tweak breaks the comparison.

## Pay For Results, Not For Hours

Businesses buy the outcome, agencies resell it, and groas answers for it either way.

[See If You Qualify](https://groas.typeform.com/to/xC1bQNUT)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/did-that-page-change-get-you-cited-or-di#webpage","url":"https://groas.com/post/did-that-page-change-get-you-cited-or-di","name":"Did Your Page Change Earn an AI Citation? Run a Control-Group Test","description":"A six-week protocol for testing whether a page change earned citations in AI Overviews, ChatGPT, Perplexity, or Gemini—or whether the results were noise.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/dddf07f5-f281-4bdd-b584-4b7df48c1cf3.png"},"breadcrumb":{"@id":"https://groas.com/post/did-that-page-change-get-you-cited-or-di#breadcrumb"},"dateModified":"2026-10-16T05:48:02.213Z","mainEntity":{"@id":"https://groas.com/post/did-that-page-change-get-you-cited-or-di#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/did-that-page-change-get-you-cited-or-di#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"Did Your Page Change Earn an AI Citation? Run a Control-Group Test","item":"https://groas.com/post/did-that-page-change-get-you-cited-or-di"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/did-that-page-change-get-you-cited-or-di#article","url":"https://groas.com/post/did-that-page-change-get-you-cited-or-di","mainEntityOfPage":{"@id":"https://groas.com/post/did-that-page-change-get-you-cited-or-di#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"Did Your Page Change Earn an AI Citation? Run a Control-Group Test","description":"A six-week protocol for testing whether a page change earned citations in AI Overviews, ChatGPT, Perplexity, or Gemini—or whether the results were noise.","datePublished":"2026-10-16T05:48:02.139Z","dateModified":"2026-10-16T05:48:02.213Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/dddf07f5-f281-4bdd-b584-4b7df48c1cf3.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"page, change, pages, citation, test, page change, answer, run","wordCount":2221,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/did-that-page-change-get-you-cited-or-di#faq","mainEntity":[{"@type":"Question","name":"Why isn't a single ChatGPT screenshot enough proof that my page change worked?","acceptedAnswer":{"@type":"Answer","text":"A single screenshot is one draw from a slot machine. Asking the same buyer question twice can return different sources, with overlap between two identical runs measured at 0.664 for OpenAI, 0.515 for Claude, and 0.319 for Gemini. A control-group test with repeated samples is needed instead."}},{"@type":"Question","name":"How do I test whether a page edit caused an increase in AI citations?","acceptedAnswer":{"@type":"Answer","text":"Write the question as: if I change page A, will engines cite page A more often for prompt P than if I had left it alone? Run a treatment group and an unchanged control group, then compare citation rates between them rather than judging a before-and-after screenshot."}},{"@type":"Question","name":"What result should I treat as a win in a citation test?","acceptedAnswer":{"@type":"Answer","text":"Set the threshold before looking at results. A win is a net 10-point citation-rate lift for treated pages over controls, on at least two engines, sustained for two weeks, with verified 200 fetches on the treated URLs. Anything less should be called inconclusive and tested again."}},{"@type":"Question","name":"What does it mean if both my treated and control pages get more AI citations?","acceptedAnswer":{"@type":"Answer","text":"It suggests the broader citation environment moved, and the control group has stopped you from taking credit for a rising tide. Do not roll out the edit on this evidence alone; wait for a cleaner read."}},{"@type":"Question","name":"If Perplexity started citing my page but AI Overviews did not, can I claim the edit worked everywhere?","acceptedAnswer":{"@type":"Answer","text":"No. A result belongs to the prompts, pages, engines, and period you actually tested. Keep the Perplexity result and do not sell the edit as a universal fix for all engines."}},{"@type":"Question","name":"When should I not run a control-group citation test?","acceptedAnswer":{"@type":"Answer","text":"Skip it when you have fewer than 10 pages that map cleanly to buyer prompts, when the questions are unstable such as news-driven or local-pack answers, or when the team cannot freeze control pages for four weeks. If you cannot protect the controls, fix that before collecting a baseline."}},{"@type":"Question","name":"How do I stop teammates from editing my control pages during the test?","acceptedAnswer":{"@type":"Answer","text":"Give one owner the authority to say no to edits or to log a change and stop the test. In an active ship cycle, someone will always have a sensible reason to tweak a control page, and an unlogged title tweak breaks the comparison."}}]}]}
```