How Accurate Is This AI Detector?

How we know the human samples are human

Every sample was published between 2010 and 2019, i.e. before any general-purpose LLM was publicly available. Human authorship here is a date fact, not a stylistic judgement. Each sample records how the date was verified, preferring identifiers that cannot be back-dated: DOIs (checked against Crossref), PMC ids and Europe PMC firstPublicationDate, Federal Register document numbers, MediaWiki revision ids, git tags, and Stack Exchange server-assigned creation timestamps.

On the non-native English samples

READ THIS BEFORE PUBLISHING ANY ESL NUMBER. The five "esl-nonnative" samples are NOT learner-corpus essays. No learner corpus with a license compatible with commercial use could be sourced: PELIC is CC BY-NC-SA (NC excludes us), the Cambridge Learner Corpus FCE set, ICLE, ICNALE, NUCLE and the BEA-2019 W&I+LOCNESS data are all behind restrictive or registration-gated licenses, and the ETS TOEFL11 corpus is a paid LDC product. WHO and FAO publications were also rejected as CC BY-NC-SA. Rather than substitute something unlicensed, these five are CC BY / CC BY-SA scholarly prose written by authors whose institutional affiliations are all in non-anglophone countries (Indonesia, Turkey, Iran, China), selected because visible L2-transfer features survived peer review uncorrected. That is evidence of non-native authorship, not proof of any individual author's first language, and the register is academic rather than the undergraduate essay that produces our worst false positives. Treat any FPR computed on this slice as a floor, not as the ESL-essay false-positive rate.

Which models the AI samples came from

Read this before quoting any FNR computed from this file. WHICH MODELS ARE ACTUALLY IN HERE (24 samples): - claude-opus-5 (Anthropic), generated 2026-08-26 for this benchmark: 18 samples. - OpenAI gpt-4o-mini, as the engine of the Codaone production humanizer, rewriting claude-opus-5 text: 3 samples. Mixed lineage — Claude wrote it, GPT rewrote it. - OpenAI ChatGPT, December 2022 web release (gpt-3.5 era), via the HC3 dataset: 3 samples. WHICH ARE NOT: Google Gemini. Meta Llama. Mistral. DeepSeek. Qwen. xAI Grok. Cohere. Every current-generation OpenAI model (GPT-4o, GPT-5 family) except gpt-4o-mini in its humanizer role. Every current-generation Claude except opus-5. Nothing from a consumer "undetectable AI" service (StealthGPT, Undetectable.ai, Phrasly) — those are the strongest evasion tools in the wild and we have zero samples of their output. THEREFORE: an FNR computed over this corpus is an FNR *for Claude Opus 5 output, for our own humanizer's output, and for three-year-old ChatGPT output*. It is not an estimate of how often we miss AI text in general, and it must not be published as one. If the number is quoted on /ai-detector/accuracy, this sentence has to travel with it. A SECOND BIAS, LESS OBVIOUS: 18 of the samples were written by the same model that assembled this corpus, while it knew it was building a detector benchmark. That is a demand-characteristic risk in both directions — the model may have written more stereotypically "AI" prose for the default-register samples, and may have tried harder than a naive user would on the evasion samples. The prompts are recorded verbatim so a third party can rerun them on any model and check. Doing that on a non-Claude model is the cheapest available fix for both biases. THIRD, THE REGISTER MIX IS NOT A USAGE DISTRIBUTION: {"default-assistant":7,"instructed-casual":5,"instructed-varied-rhythm":5,"humanizer-output":3,"domain-specific":4}. It is deliberately weighted toward evasion (13 of 24 samples are instructed-casual, instructed-varied-rhythm, or humanizer output) because §1.4 of AI-DETECTOR-COMPETITIVE-2026-08-25 measured that all of our false negatives come from disguised AI. A corpus weighted this way will report a WORSE FNR than a corpus of default-register text would. That is intentional and honest; it is not comparable to a competitor's headline accuracy number, which is measured on whatever mix flatters them. FOURTH, ON THE HUMANIZER SAMPLES: the free anonymous quota is 3/day/IP and all three were used on 2026-08-26, so there are exactly three and no reruns. They are the highest-value rows in the file — a false negative on any of them means our own paid product defeats our own free product, which is the central question the accuracy page exists to answer honestly.

On how few voices some registers have

Registers are unevenly hard to source under an open license. Government and technical-documentation samples come from a small number of institutional voices (three Federal Register rules, two Kubernetes docs pages, three English Wikipedia revisions), so within-register variance understates the real world.

The score floors make it worse

Five rules can raise a score above what the weighted blend produced, to catch documents the blend misses. On this corpus they cost 0.036 AUC — they misfire more often than they fire.

The benchmark does not exercise the engine

The phrase table is 70 patterns across 22 named categories. 17 of those categories never fire on any of the 48 samples. Each of them does fire on text matching its own pattern, so this is a gap in the corpus rather than a dead rule — but it means the error rates above were measured without two thirds of what the detector looks for ever appearing.

Where the ranking inverts

Mean score by register. The registers most likely to be accused sit near the top, and AI given a one-line instruction to vary its rhythm sits at the bottom.

Why it fails: Statistical detection reads simpler vocabulary and regular sentence rhythm as machine-like. This is the best-documented bias in the entire detector category, and it lands on the people with the most at stake. In the run above, the 5 samples written by non-native English speakers averaged 23.8, the highest of them reached 42, and none was flagged.

What we do: Nothing in the score is register-aware. We cannot tell that English is your second language, so we do not claim to adjust for it — the number you get is the same number any other writer would get for the same prose. One rule does read register, and it lowers our stated confidence rather than your score: prose with contractions and lowercase sentence starts caps confidence at low below 40, because our own feature screen showed we cannot separate casual-prompted AI from casual humans. What the code does do: scores from 40 to 59 return "Mixed — some AI-sounding passages" at low confidence, scores under 20 return "No obvious AI tells" at low confidence, and any input under 100 words drops one confidence notch. The per-sentence view names the exact phrases that moved the score. If you write in English as a second language and got flagged, that context outweighs our number.

Why it fails: Connectors like "Furthermore" and "In conclusion" are also among the most reliable LLM tells, so a regulation, a contract, or a press release can out-score actual AI text on the pattern model. In the run above the 4 government-bureaucratic samples averaged 17.8 and the 3 business-marketing samples 14.3, none of them flagged — but the 4 academic abstracts averaged 37.3 and produced the single false positive in the whole run, at 50.

What we do: Nothing caps this band, and we should not have implied otherwise. Four or more stacked formal connectors in one document trip a floor that sets the score to at least 78 whatever the other signals say: a 203-word human regulatory passage returns 78, "Reads like AI in places", medium confidence. What we can offer is visibility — the deep report now prints the weighted blend of the four models and then names the rule that overrode it ("5 stacked formal connectors raised the score to 78%"), so you can see that the number came from counting connectors rather than from four signals agreeing.

Why it fails: Good documentation is deliberately uniform: same sentence shape, same structure, restrained vocabulary. A variance-based model reads that discipline as generation. In the run above the 4 technical-documentation samples averaged 21.8 and the highest reached 42.

What we do: Uniform rhythm has a floor of its own, and it is not capped either: at four or more sentences, a sentence length that varies by under 15% of the average forces the score to at least 50, and under 25% to at least 42. That is why disciplined documentation drifts upward even with no AI phrasing anywhere in it. The deep report names that rule when it fires, and the per-sentence breakdown will show you an empty flag column — which is the tell that no actual phrasing was found.

Why it fails: We used to over-weight punctuation habits as an AI tell: an em-dash-heavy human essay scored 80 on an earlier version of the scoring. Plenty of humans write like that; so do LLMs.

What we do: We downgraded punctuation from primary evidence to corroboration: dashes now need an independent signal before they can raise a verdict. The em-dash-heavy human sample kept for this scores 12 today. It is not in the table below — it is one of 11 regression cases, each encoding a specific defect this detector was caught making, which run on every scoring change but are excluded from the rates above because they were written to trap a bug rather than sampled from anywhere. They are in the raw data file under "regressionResults".

Why it fails: Prompt an LLM to write casually — or run its output through a paraphraser — and the formal tells our pattern model keys on are stripped away. 13 of the 24 AI samples above are evasion attempts of that kind; every one of them passed as human, and the highest any of them scored was 16. This is the main way to beat us, and every other statistical detector.

What we do: Mostly, we cannot fix this within our approach, and we say so. Treat a low score on text you already suspect as weak evidence of anything. Uneven per-sentence scores on a polished piece are sometimes the residue of partial rewriting — a lead worth following, not a verdict.

Why it fails: Statistics computed over a handful of sentences are close to noise. Sentence-length variance, vocabulary richness, and starter diversity all need material to measure.

What we do: Under 30 words we refuse to give a verdict at all. Just above that line, read the score as a hint, not a finding.

Why it fails: Every signal here is English-shaped: words split on spaces, sentences end on a full stop, and the phrase list is English. In a script that does not space its words, a whole paragraph tokenises to roughly one "word", so vocabulary richness pins at 1 and length variance at 0 — and the arithmetic still produces a confident-looking number. It was not merely weak, it was inverted: AI-written Chinese scored 9 ("No obvious AI tells") while human-written Chinese scored 55.

What we do: Chinese, Japanese, Korean, Thai, Lao, Khmer and Burmese input is now refused before scoring — the tool returns an error explaining why, not a number. Other space-separated scripts (Arabic, Hebrew, Cyrillic, Greek, Devanagari) still score, because their structural signals survive; those results carry a note saying the phrase-level checks are English-only and the score is weaker evidence than usual.