How often is a cited quote actually on the page?
We take 633 quote attributions our product produced in real use, fetch every source page again, and check each quote character by character. This page is the method, the result, and the part where the answer was partly our own fault.
Sample frozen 20 August 2026 · raw per-triple judgements (JSON)
Why this exists. An outside critic pointed out that "every quoted sentence is machine-checked" was a claim, not a measured property, and specified a test we could not grade ourselves: freeze the sample first, count failure classes separately, let an independent judge fetch the pages, publish the raw decisions. That is what this is. Reading our own code to build it, we also found the claim was stronger than the code, corrected the same day.
The sample
633 quote attributions from real production use, 29 July to 19 August 2026, across 310 different websites. Not selected by outcome, every attribution the system emitted in that window, with only exact duplicates (same URL and same quote) removed.
The fingerprint of the frozen sample was committed to the repository before the judge ran: sha256 f3c6f112ee8a9ef98252a5fc2832e160c732b93b18f7436f43a16a9548632e34. Touching the sample afterwards changes it.
No personal data. Only the triple (quote, source URL, timestamp) was exported, no licence key, no installation, and deliberately not the user's claim. The quote itself is text from a public page.
What the judge does
It sees only the triple. It fetches the URL itself, extracts the text, and compares character by character after Unicode and whitespace normalisation. Failure classes are counted separately and never added into one rate, that is the whole point, and our own first attempt got it wrong (see below).
Result
| Outcome | n | share | meaning |
|---|---|---|---|
| Verbatim on the page | 463 | 73.1% | found exactly as cited |
| Page not readable to us | 120 | 19.0% | neither confirmed nor refuted |
| Mostly not findable | 30 | 4.7% | under half the quote matched |
| On the page, our reader splits it | 17 | 2.7% | a link or footnote breaks the sentence |
| Not findable at all | 3 | 0.5% | strongest indicator of a wrong attribution |
Of the 513 pages we could actually read, 33 did not carry the quote: 6.43%, upper 95% bound 8.56%.
Why 19% of pages could not be read
| Reason | n |
|---|---|
| HTTP 403, automated readers blocked | 50 |
| PDF, we do not read PDFs | 21 |
| HTTP 429, rate limited | 12 |
| Timeout | 11 |
| HTTP 401 / 406 / 404 / 402 / 410 | 16 |
| Network error, page too short, paywall pattern | 10 |
This is why a live fetch is not a hard requirement in the product. Making it one would discard roughly one in five perfectly good citations. It is reported as a third state instead: search index only, with the reason.
The uncomfortable part: a quarter of the failures are ours
Reading the 33 failures individually rather than as a number: 8 of them contain markdown artefacts, headings starting with #, line breaks inside the "sentence", prefixes like Repository:. Those quotes were never page prose. Our search provider returns a cleaned, markdown-ish rendering of a page, not its literal text, and the model quoted from that rendering.
Excluding them leaves 25 real divergences out of 513, 4.87%. We are reporting the higher number as the headline anyway, because the artefact is still our problem: we chose the provider and we passed its output on.
The three that were not findable at all
Two of the three are exactly that artefact, a GitHub page quoted as Repository: … and a USA Today page quoted as # Fact check: …. Neither string exists on the page in that form. The third is a German newspaper article whose page returned only 5,263 characters to us, consistent with a truncated or gated delivery.
Not one of the 633 was an invented quote attributed to a page that never contained anything like it. That is the failure mode the check exists to prevent, and in this sample it did not occur.
Does the web move under us? Measured: no.
Attributions were between 1 and 22 days old when re-checked. If pages were quietly changing, older ones should fail more. They do not: July 90.5%, early August 91.0%, mid August 87.4% found on the page (of readable pages). Whatever produces the failures, it is not the passage of time.
Adversarial cases: 15 of 15
Separately from the production sample, fifteen constructed traps, each testing one way a naive checker breaks. All passed.
| Trap | Correct behaviour |
|---|---|
| Quote spanning a paragraph boundary | found |
| Sentence with a link inserted mid-way | found |
| Footnote marker breaking the sentence | not found, our reader's limit, reported as such |
Text that exists only inside a <script> | not counted as page text |
| Arabic, mixed script direction | found |
| Curly quotes, non-breaking spaces, dashes | found |
| Quote longer than 400 characters | found, nothing silently truncated |
| Soft-404, PDF, image URL, malformed URL | not verifiable, never "confirmed", never "refuted" |
| Quote under 8 words | refused, a short string matches by accident |
| Invented sentence on a readable page | correctly not found |
The distinction those traps enforce: a paywall must produce neither "verified" nor "not on the page". A tool that treats a paywall as proof lies upward; one that treats it as refutation lies downward and throws away correct sources.
Our first attempt at this number was wrong, and it is instructive
An earlier run on 40 triples reported "18.2% of reachable pages did not carry the quote". That number mixed three unrelated things: a bot wall returning 226 characters that our threshold accepted as a real page, a sentence split by an inserted link, and one genuine absence. It would have made our own tool look worse than it is, by measuring our fetcher and calling it our accuracy.
That is why the table above separates every class. A single blended "error rate" is almost always hiding this.
What this does not measure
- Temporal drift proper. We check against the page today, not against the page as it was at citation time. We had no archived HTML from then, this study is where archiving starts. The age analysis above suggests drift is not the driver, but it does not rule it out for individual cases.
- Wrong document. If a quote lives on a syndicated copy rather than the cited URL, it counts here as a failure. We did not go looking for the twin.
- Whether the source is right. A quote can be verbatim on the page and the page can still be wrong. This measures citation integrity, nothing more.
- The fact-check side under load. The sample is what production emitted; it is not a stress test.
What changed because of it
- The product now fetches the source page and shows one of three states per citation: verified on the page, search index only (with the reason), or, where the page is readable and the quote is absent, the finding is discarded and never shown.
- The wording on the site was corrected from "every source is opened and read" to what the code actually does.
- The bot-wall threshold was raised from 200 to 900 characters, after a 226-character shell was accepted as a page.
Reproduce it
The raw data holds all 633 judgements: source host, outcome, match fraction, page length, failure reason, and the age of the attribution. Quotes and full URLs stay in the repository rather than on this page, they are other people's content, and the host plus outcome is what makes the numbers checkable.