# If AI citations move 40%–60% in a month, how many runs should an experiment need?

Canonical: https://mudpie.ai/blog/citation-volatility-experiment-sample-design/
Breadcrumb: [Home](https://mudpie.ai/) / [Field notes](https://mudpie.ai/blog/) / [If AI citations move 40%–60% in a month, how many runs should an experiment need?](https://mudpie.ai/blog/citation-volatility-experiment-sample-design/)
Author: Ali Abouelatta (https://mudpie.ai/authors/ali-abouelatta/)
Published: 2026-09-19
Updated: 2026-09-19
Research type: research-methods-article
Method: Use observed volatility figures only as motivation; label the two-proportion arithmetic and intra-prompt correlation example synthetic; propose clustered, engine-specific, holdout-based measurement rather than a universal run-count rule.

AI-search experiments can manufacture a win by accident.

[Profound](https://www.tryprofound.com/blog/ai-search-volatility) compared June 11–13 with July 11–13, 2025, reporting domain-level citation drift of 59.3% for Google AI Overviews, 54.1% for ChatGPT, 53.4% for Copilot, and 40.5% for Perplexity. Its sample contained roughly 80,000 prompts per platform. [AirOps](https://www.airops.com/report/how-citations-mentions-impact-visibility-in-ai-search) reported 30% back-to-back brand persistence across 800 queries and more than 45,000 citations. A [2026 preprint](https://arxiv.org/abs/2603.08924), sampling three platforms and three consumer-product topics, found many apparent differences fell inside the measurement noise floor. These are different measures and samples, not interchangeable failure rates.

Those numbers do not tell us how many runs every team needs. They tell us why one pre/post screenshot is weak evidence.

## What a clean experiment has to separate

There are at least four moving parts:

1. **The prompt set.** Did the question change, or did the page change?
2. **The engine.** ChatGPT, Perplexity, Google AI Overviews, and Copilot do not share source pools.
3. **The response draw.** Repeating the same prompt can produce different citations.
4. **The web and model.** A model update, index refresh, or source-page change can move the result during the test.

[Product Hunt’s published case study](https://www.producthunt.com/p/producthunt/case-study-how-product-hunt-can-improve-ai-visibility-in-2026) illustrates the problem. It reported a roughly 0.4% baseline citation rate and a 10× increase on ChatGPT after community-derived FAQs. The authors also noted that the timing aligned with a model update, leaving causality unclear. That is a useful experiment lead, not a controlled estimate of FAQ impact.

## A synthetic arithmetic example

The following numbers are **synthetic, not observed**. They show the arithmetic a team should do before calling a small lift real.

Assume:

- Baseline citation rate: 20%.
- Minimum effect worth acting on: 24%.
- Two-sided alpha: 0.05.
- Power: 80%.

Define one observation as a binary citation outcome from an independent response draw. Let p̄ be the average of p0 and p1. For two independent proportions, the rough sample-size calculation is:

```text
n per arm ≈
[(1.96 × √(2p̄(1−p̄))) + (0.84 × √(p0(1−p0) + p1(1−p1)))]²
÷ (p1 − p0)²
```

With `p0 = 0.20` and `p1 = 0.24`, this produces roughly **1,680 independent observations per arm**.

Now imagine 100 prompts run 10 times each. The raw count is 1,000 observations per arm. They are not necessarily 1,000 independent observations. If repeated answers to the same prompt have an illustrative intra-prompt correlation of 0.30, the design effect is:

```text
1 + (10 − 1) × 0.30 = 3.7
```

The effective sample is roughly `1,000 ÷ 3.7 = 270` observations per arm. Under that synthetic assumption, about **622 prompts per arm at 10 runs each** would approach 1,680 effective observations. Other shared conditions can add dependence. A real repeated pre/post design may need a paired or clustered treatment-by-time model; this independent-arm approximation is not a prescribed experimental design.

The arithmetic is not a claim about the true correlation of any engine. It is a reminder to estimate the design effect instead of treating repeated answers as free independent data.

## A workable design

| Component | Minimum design | Why |
|---|---|---|
| Prompt set | Freeze a versioned set. Assign comparable pages or content groups to treatment and holdout where feasible, and map prompts to those groups. | Prevents prompt drift and avoids pretending every prompt is an independent treatment unit. |
| Engines | Analyze each engine and search mode separately. | Profound’s 11% ChatGPT/Perplexity citation overlap shows why blended scores hide behavior. |
| Runs | Use a fixed schedule and a duration chosen from pilot variance and the effect worth detecting; log model, locale, date, and failures. | A calendar rule alone cannot establish adequate power. |
| Treatment | Change one page element or one content group at a time. | Keeps the intervention interpretable. |
| Primary outcome | Citation rate or mention rate, chosen before the test. | Keeps the test tied to one answer-level unit. |
| Secondary outcomes | Citation position, cited passage, referral sessions, branded search, and conversions. | Shows whether visibility travels downstream. |
| Holdout | Preserve comparable pages or prompts with no change. | Provides a counterfactual for model-wide movement. |
| Analysis | Cluster by prompt and engine; report intervals and raw counts. | Stops repeated answers from inflating confidence. |

## What to publish

A credible experiment report needs the prompt list or a reproducible description, the run dates, the exact treatment, the response-level definitions, the failure rate, and the holdout result. It should show the result by engine rather than only as a blended percentage.

If the result moves only on ChatGPT after a model update, say that. If citation rate rises while referral traffic stays flat, say that. If the treatment and holdout move together, the experiment detected platform movement, not a page effect.

Publish a 10× result from one post-change run as a reason to re-run the experiment, not as the result.


## Author disclosure

I cofound Lazyweb and publish Mudpie. This is an owner-written publication, not an independent testing organization. Research notes distinguish observations, sourced reporting and editorial judgment.