An editor receives a 1,400-word submission with three em dashes, a paragraph that begins “In today’s rapidly evolving landscape”, and a bulleted list of exactly five items. Every instinct says machine. The writer says no. Both of them are arguing about the wrong evidence, and the argument will be settled badly unless the editor changes what counts as proof.
The style tells are real but weak, because style is the cheapest thing to change. Anyone can delete an em dash in four seconds, and anyone who has used a chatbot twice has already learned to. A test that fails after one editing pass is not a test; it is a mood.
This is a working protocol for the editor who has to decide, and it is built on what detection research actually measured rather than on what style guides repeat.
The em-dash test fails for a mechanical reason
Punctuation is a surface feature. It is generated after the model has chosen words, and it is the first thing a human rewrites. Any signal that lives entirely in punctuation and vocabulary has a half-life measured in minutes.
The deeper problem is selection. Editors who believe in the em-dash test notice em dashes. They do not notice that the same submission contains a named source on line 12 and a fabricated one on line 40, because they were not looking there. The test does not just fail to detect; it consumes the attention that would have detected something else.
What survives editing is not style. It is structure that the writer would have to rebuild, and facts the writer cannot invent. Those are the two categories worth checking.
What a detector is actually computing
The best-known zero-shot method, DetectGPT, does not look at prose style at all. It measures a property of the model’s probability function: text sampled from a language model tends to sit in negative-curvature regions of that model’s log-probability function, so a passage can be scored by perturbing it and comparing likelihoods. The authors report raising detection of machine-written fake news from 0.81 AUROC for the strongest zero-shot baseline to 0.95 for their method. FACT (arXiv:2301.11305, ICML 2023)
Two consequences follow, and both matter to an editor.
First, the method needs log-probabilities from the model of interest. Only the provider hosting that model can compute them honestly. A third-party detector scoring text from a model it does not host is approximating, and how well it approximates is not published. INFERENCE
Second, even the good number is a discrimination score on a benchmark, not a verdict about the paragraph in front of you. A detector that separates two populations at 0.95 AUROC still has a false-positive rate at whatever threshold you choose, and the person who pays for that false positive is the writer.
The paraphrase result that ends the text-only argument
In 2023 a team built DIPPER, an 11B-parameter paraphraser designed to rewrite paragraphs while preserving meaning and controlling lexical diversity. Used against text from three large models, it evaded several detectors, including watermarking, GPTZero, DetectGPT and OpenAI’s classifier. The headline number: at a constant 1% false-positive rate, paraphrase dropped DetectGPT’s detection accuracy from 70.3% to 4.6%. FACT (arXiv:2303.13408, NeurIPS 2023)
Read that as an editor, not a researcher. The same paper that reports the failure also reports a defence: retrieval, where the provider keeps a database of generations and checks whether a candidate passage matches something the API previously produced. That caught 80% to 97% of paraphrased generations across settings while flagging 1% of human-written text. FACT
The defence works only for the provider, because it requires the generation database. No editor has one. So the state of the art in text detection is either (a) a statistical test that a paraphrase can drive to near-chance, or (b) a provider-side lookup that the public cannot run.
The bias is not a footnote, it is the main risk
The same literature documents the failure mode that matters most in a newsroom or a university: detectors systematically misclassify non-native English writing as machine-generated, while native writing is identified accurately. The authors also found that simple prompting strategies both mitigate the bias and bypass the detectors. FACT (arXiv:2304.02819)
That produces an asymmetric error. A confident editor who relies on a detector will accuse the careful non-native writer more often than the native writer, and the accusation will feel objective because a number produced it. The harm does not fall evenly, and it does not show up in the detector’s accuracy score.
If a detection tool’s output is going to influence a person’s grade, job, or byline, the burden of proof is on the tool, and no tool in this category has published a per-demographic error rate that would justify the decision alone. INFERENCE
Where to look instead: four signals with different failure modes
The signals below are ordered by how expensive they are for a writer to fake. Expensive is the point. A signal that costs nothing to produce decides nothing.
| Signal | What it measures | How it fails |
|---|---|---|
| Citation behaviour | Whether claims point to checkable sources | A careful human can also cite badly; a machine can be prompted to cite |
| Claim-to-source ratio | How many assertions carry evidence | Some legitimate genres are assertion-heavy by design |
| Process artefacts | Whether a draft has a history | Ghostwritten human work also lacks one |
| Specificity under questioning | Whether the writer owns the details | Anxiety and language barriers read as evasion |
Citation behaviour
Open ten factual claims at random and check whether each one points somewhere a reader could verify. Machine drafts tend to produce citations that are plausible in shape and wrong in content: real-sounding journal names, correct-looking URLs that 404, statistics with no origin. The tell is not the presence of citations but their behaviour under a click.
This is the strongest single signal available to an editor, because it is cheap to run and it produces evidence that does not depend on a detector’s opinion. A draft with eleven citations, of which four resolve and seven do not, has a factual problem regardless of who or what wrote it.
Claim-to-source ratio
Count assertions that would change a reader’s decision and divide by the number that carry a source. A ratio of 30 to 2 is a red flag in an analysis piece and unremarkable in a personal essay, so the benchmark has to be genre-specific. Set it per section type rather than per publication.
The failure mode is that this measures rigour, not authorship. A human writer with a deadline can produce the same ratio, and the correct response to a low ratio is editing, not accusation.
Process artefacts
Ask for the document’s history: the version trail, the research notes, the interview recording, the earlier draft with the wrong first paragraph. Machine-first drafts still have these if the human did any work afterwards; what they lack is the intermediate thinking that never becomes prose.
A writer who cannot produce any intermediate artefact for a 2,000-word analytical piece is telling you something, but it may only be that they work in one pass. Treat this as corroboration, never as proof.
Specificity under questioning
Ask three questions that only the author could answer: why that source and not the obvious one, what the counter-argument was, what got cut. The author who wrote the piece answers in specifics and volunteers corrections. The author who assembled it answers in generalities and agrees with everything you suggest.
This is an interview, not a test. It produces a conversation you can quote and a judgement you can defend, and it works on the same principle as a reference check: the person who did the work has texture.
What to do with the answer
Three decisions are available, and most organisations only use the worst one.
If the evidence is strong and the piece is otherwise sound, edit it. The question “was a model involved” is not the same as “is this accurate”. A sourced, checkable piece is publishable even if a model helped draft it, provided your disclosure policy says so.
If the evidence is strong and the claims do not check out, reject on accuracy. You never need to prove authorship; you need to prove the piece is wrong. This is the cheapest defensible route and the one that keeps the argument about the work.
If the evidence is weak, say nothing about authorship and ask for the sources. Accusations made on style alone are unwinnable in public and unfair in private, and the research above explains why: the signal is weak, the paraphrase attack is trivial, and the error is not distributed evenly across writers.
The policy that saves you the argument
Write the rule down before the dispute, not during it. A workable one has four lines:
- Disclosure is required for generated text that survives into the published draft, not for research, outlining or ideation.
- Every factual claim must resolve to a source the editor can open.
- The author is responsible for accuracy whether or not a model was involved.
- No authorship decision is made on a detector score alone.
Those four lines settle almost every case without a detection argument, because they move the question from “how was this made” to “can this be checked”. The first question is unanswerable with current tools. The second is answerable in ten minutes with a browser.
Why this is the honest position
It would be satisfying to end with a reliable test. The published evidence does not support one. What it supports is narrower and more useful: style signals are surface features that survive minutes, statistical detectors degrade to near-chance under paraphrase, provider-side retrieval works but is not available to you, and the available detectors carry a documented bias against exactly the writers most likely to be accused.
So the durable signals are the expensive ones: citations that resolve, claims that carry evidence, artefacts that show thinking, and an author who can talk about their own work. None of them proves a machine was involved. All of them tell you whether the piece is worth publishing, which was the decision you actually had to make.
If you want the writer’s side of this — what a machine draft looks like before an editor touches it — the companion piece on why AI content reads as fake is the place to start, and the full tool coverage is where the generator comparisons live. For the mechanics of why a model produces the same paragraph twice, the prompt-to-answer walkthrough explains the sampling step.
Sources
All URLs fetched 26 October 2026.
- https://arxiv.org/abs/2301.11305 — DetectGPT, probability curvature, 0.81 → 0.95 AUROC on GPT-NeoX fake news, ICML 2023, requires log-probabilities from the model of interest.
- https://arxiv.org/abs/2303.13408 — DIPPER paraphrase attack, DetectGPT accuracy 70.3% → 4.6% at 1% FPR, evasion of watermarking/GPTZero/OpenAI classifier, retrieval defence at 80–97% detection with 1% human false positives, NeurIPS 2023.
- https://arxiv.org/abs/2304.02819 — detectors systematically misclassify non-native English writing as AI-generated; simple prompting both mitigates the bias and bypasses the detectors.
Independent technology writer focused on artificial intelligence, emerging technologies, and digital innovation. Covers AI applications in sports, productivity, and online business.
