Can you really tell whether a text was written by an artificial intelligence?

Your daughter comes home and says her teacher has accused her of getting ChatGPT to write her essay. She swears she did not. The teacher is holding a number: a 92% probability that the text was generated.

Who is right? And more to the point, what does that 92% actually mean?

This is one of the few questions about technology where getting the answer right changes someone’s life. Students have been expelled, disciplinary proceedings opened, school records ruined. So it is worth understanding properly, and not by hearsay.

How a program can tell

There are three families of method, and mixing them up is the first source of confusion.

The first looks at predictability. A language model picks, word after word, from among the likelier continuations: the result is text that is smoother and more regular than what a person writes, since a person has swerves, second thoughts, sentences that come out crooked. The earliest detectors measured that.

The trouble is that predictability depends on what you are writing about. A group of researchers from four universities showed this with an example that has become famous: ask a model to write about an astrophysicist capybara and the output is so unusual that it scores as barely predictable, and the detector files it as human. That is why modern methods compare two models against each other instead of looking at just one.

The second family is made of trained classifiers: a neural network fed millions of human and generated texts until it learned to tell them apart. This is where today’s commercial products sit, and they are clearly better than those of three years ago.

The third is the watermark, and I am saving it for the end because it is the only one that works in a different way from the other two.

Why a number is not proof

Two arguments, and they do not carry the same weight.

The first is practical. When you accuse someone of old-fashioned plagiarism, there is a piece of evidence you can look at: the source they copied from. With generated text that evidence does not exist. There is a score, and a score cannot be inspected.

The second is theoretical, and to my mind it is the more important of the two. A group of researchers from Maryland and Harvard proved a mathematical result: the best quality any detector can reach, now or in the future, depends on how closely the model’s writing resembles human writing. The better models get, the more detection becomes impossible. Not hard: impossible, in the technical sense of the word.

The same researchers showed that having generated text rewritten by another program wrecks almost every detector. In the case of watermarks, recognition collapsed from 99.3% to 9.7%, and in some runs as far down as 4%.

There is also a precedent worth a line. OpenAI released a detector of its own in January 2023 and withdrew it six months later for a “low rate of accuracy”. Their own published figures: it caught 26% of generated texts and wrongly flagged 9% of human ones.

What was measured, and what was claimed

Here is the heart of it, and there is one distinction to hold steady: who measured what.

The newer detectors claim impressive numbers. The most cited one right now claims roughly one false positive in every twenty-four thousand texts, and that figure comes from the company that sells it, measured on an archive of its own that is not public. It does not mean the figure is false. It means nobody can verify it.

Two serious independent evaluations do exist, and it is only fair to say they come out in that product’s favour.

The first is a working paper by two researchers at the University of Chicago, that is, a study circulated before formal publication, carried out on almost two thousand English texts across six different genres: the product in question gets very little wrong in either direction, while other detectors flag as generated up to 78% of human texts. Be careful, though: a working paper has not been through peer review, and I have seen that study presented as if it had.

The second is a study published in June 2026 in the International Journal for Educational Integrity, this one peer reviewed, carried out by three researchers at the Free University of Brussels on 160 academic essays in English. The result here points two ways. On one side it confirms: zero false positives for three detectors out of four, and the authors write that false positives are rare, an improvement on earlier studies. On the other side there is a limit the authors state themselves and almost nobody reports: there were forty human texts. Zero errors on forty texts does not demonstrate one error in twenty-four thousand. It only shows the real rate is plausibly under 7%.

To validate a number like one in twenty-four thousand you need tens of thousands of verified human texts. For every language. For every genre of writing.

The part that gets left out: Italian

For readers in Italy, and for any language that is not English, this is the part that matters. The general lesson travels even if the specific findings do not: a figure measured in one language says nothing about another.

So, here is the question an Italian reader should be asking first, and which I have not seen put anywhere.

Those numbers were measured on texts in English. All four of the serious independent studies I could find, from 2023 to today, are entirely in English. One of them includes seven languages other than English, but only machine-translated, and Italian is not among the seven.

The multilingual research benchmarks, the ones that compare detectors across many languages, do not include Italian: not the most cited one from 2023, with eleven languages, and not the 2025 one, with twenty-two. There are Catalan, Portuguese, Romanian, Spanish. Italian, no.

As for the products, Turnitin’s official documentation lists the languages its detection model supports: English, Spanish and Japanese. Italian is not there. GPTZero’s support page lists around twenty languages and Italian does not appear. For the most accurate product I could not find any public list of supported languages.

One evaluation on Italian does exist, though, and it is an interesting one. Three researchers at the CNR in Pisa ran it inside EVALITA 2026, the evaluation campaign for the Italian language: over forty-six thousand news articles, half human and half generated, fed to systems from six research groups. The best reaches 94.6% accuracy with under 0.5% false positives, which sounds good.

But there is the result that really counts. When the Italian text had been generated by a model trained specifically on Italian, rather than by a model built for English, recognition drops to around 50%. That is, a coin toss.

To sum up in a form you could repeat out loud: as of 5 October 2026 no independent evaluation of commercial detectors on Italian texts appears to have been published, the two products most used in schools do not list Italian among their supported languages, and the one Italian academic evaluation says that Italian generated by Italian models currently slips through half the time. Anyone running a detector over an essay written in Italian is using a tool calibrated on another language.

Who pays for the mistakes

In 2023 five Stanford researchers ran a simple test. They took 91 essays written by non-native speakers, from TOEFL exams, and 88 essays by American eighth-graders, and put them through seven detectors.

On the American children’s essays the detectors worked: almost nothing flagged. On the non-native speakers’ texts the average false positive rate was 61%. Eighteen essays out of ninety-one were flagged by all seven detectors at once, and eighty-nine out of ninety-one by at least one.

Then they ran the check that clinches the argument. They asked ChatGPT to rewrite those essays “with more native-sounding vocabulary”, and the error rate collapsed from 61% to 11%. They did the reverse, simplifying the vocabulary of the American essays, and the rate climbed from 5% to 57%.

Those detectors were not measuring who had done the writing. They were measuring how rich the vocabulary was.

Two things I owe that study. It is published in a peer reviewed journal, but in the section given over to opinion, not as a full research article. And the authors themselves warn that the detectors tested were the 2023 generation, built on older models, and that they did not know whether the problem would persist in later products.

A partial answer to that question: the Belgian study of 2026 no longer finds that disaster, but it does not measure the difference between native and non-native writers, so it is not a refutation. And another peer reviewed study, from 2023, measured something close: human texts written in another language and machine-translated into English were wrongly flagged twice as often as texts written in English directly. Machine translation leaves traces that look like a machine’s, because a machine did go through them.

On top of that there are two effects that pile up quietly.

The first is scale. Turnitin states under 1% error on a whole document, but 4% on a single sentence. A detector that highlights sentences multiplies the chances of getting one wrong: across twenty sentences, at that rate, almost every piece of work will contain a sentence highlighted in error.

The second is repetition. With a 1% error per text and forty submissions over the course of a school career, the probability that a student gets flagged wrongly at least once is about a third. Across a thousand students, that means a few hundred people who sooner or later will hear that sentence said to them.

It is the sum Vanderbilt University did too when, in August 2023, it turned off Turnitin’s detector: with a stated 1% error and the 75,000 pieces of work it received in a year, that came to roughly 750 students wrongly accused. Among their reasons they also put the tool’s opacity and the problem with non-native speakers.

What the people selling them say

This is worth reading, because it is more cautious than what many schools say.

Turnitin’s technical documentation writes that false positives are a possibility, that scores between 1% and 19% are not even shown, precisely in order to reduce the risk of wrong accusations, and one sentence that ought to be pinned up in every staff room: the AI writing score should not be used as the sole basis for action against a student. The same company stated, in a post of its own in 2023, that it is not Turnitin that establishes whether misconduct occurred, and that the teacher’s professional judgement is required.

So the people selling the tool say it is not proof. Then it reaches the classroom and becomes proof.

In Italy there is a gap here that is itself a piece of information. The Ministry’s 2025 guidelines on artificial intelligence in schools talk about human oversight, transparency and bias, but say not a word about detectors, their reliability or their worth as evidence. The policy of a major university like Bologna governs what a student may do and asks them to declare their use of AI, but makes no mention of detectors. I could not find a single document from an Italian university, or from the Ministry, that takes a position.

If it happens to you

1. Ask what the accusation rests on, and in particular whether it rests on a score alone. If the answer is yes, what you are facing is an indication, not proof, and the manual written by the people selling the tool says so.
2. Pull up the document's history. Google Docs keeps every revision, Word with OneDrive keeps versions. A text that grew in pieces, with sentences deleted and rewritten, looks very different from a block pasted in all at once. It is not conclusive, because you can paste in several goes, but it is the strongest indication you have.
3. Gather everything around it: notes, outlines, intermediate drafts, the sources you consulted, the emails where you asked for clarification.
4. Offer a direct check. Writing something similar in person, or talking the content through out loud, clears the situation up in ten minutes.
5. If you are foreign, or writing in a language that is not your own, say so, and bring the figures above. It is documented that the burden of these errors falls exactly there.

And if you are on the other side, the one who has to decide: check whether the tool has been validated on the language the work you are assessing is written in, talk to the person as a first step and not a last one, and keep the burden of proof on the side doing the accusing. The conclusion of the 2026 Belgian study, which does give detectors credit, is precisely that: they should not be used as sole proof in decisions that carry weight.

The watermark, and why it does not solve it

That leaves the third family, the only one that is technically solid.

Google DeepMind published in Nature, in October 2024, how SynthID-Text works: during generation, the choice between near-equivalent words is steered by a secret key, leaving a statistical signature in the text that anyone holding the key can recognise. It was tried on around twenty million Gemini responses without users noticing any difference in quality.

It works, but the limits are written down by the authors themselves. Paraphrase erases it. Short texts carry little signal. It does not work when the answer is near-obligatory, as with a definition or a translation. It covers only the texts of whoever applies it, so it would really work only if everybody applied it. And it does not apply to the open models people install on their own computer, because anyone can rebuild them without the watermark.

The practical consequence, worth remembering: the absence of a watermark proves nothing.

On the regulatory side, the European AI Act has been in force on this point since 2 August 2026. Article 50 requires providers of these systems to mark generated content in a machine-readable format, and those publishing generated text on matters of public interest to tell the public. Two caveats, though. The duty to declare falls away where there has been human review and someone takes editorial responsibility, which in journalistic practice is close to the norm. And none of it touches a student’s essay: a school gains no extra tool from this law.

To understand what sits underneath

A lot of this gets clearer once you know how a language model works, that is, why it writes more predictably than a person does. I wrote about it in What a language model is, and why it sometimes makes things up, and the part about how predictable words are is exactly what the first family of detectors stands on.