Protocol

Run your own detector test

Twenty texts, one afternoon, two numbers that tell you whether to trust a detector on your writing.

01 · Setup

The 20 texts

SET H10Your human textsSame genre and length as what you will check. Half written before November 2022.
SET A5AI, uneditedSame topics as your human texts, straight from a model.
SET E3AI, editedThree of set A, edited by you for ten minutes each.
SET P2AI, paraphrasedTwo of set A, run once through a paraphrasing tool.

02 · Steps

The protocol

  1. Collect 10 human texts

    Writing like the writing you will check: same genre, 300 to 800 words. Use work written before November 2022 where you can, so no AI was involved.

    Set H · 10 texts · label: human

  2. Make 10 AI texts on the same topics

    Five straight from a model, three you edit for ten minutes, two run through a paraphraser. Same topics stop the test measuring subject instead of authorship.

    Set A 5 · Set E 3 · Set P 2 · label: AI

  3. Fix your rule before you look

    Decide what counts as 'called AI': the tool's own AI label, or a score above a number you write down now.

    rule: called_ai = tool label == 'AI' (or score >= your threshold)

  4. Shuffle, then run each text once

    Paste each text in a fresh session in shuffled order. Record the score and label exactly as shown, plus the passages it flags.

    One run per text. No re-runs to 'see if it changes'.

  5. Count the two error rates

    False-positive rate is the share of your human texts it called AI. Detection rate is the share of AI texts it called AI.

    fp_rate = human_called_ai / 10 · detection = ai_called_ai / 10

  6. Compare on false positives first

    A tool that accuses your human writers is the costlier mistake. Between tools with the same false-positive rate, take the higher detection rate.

    Re-run the same 20 texts whenever the vendor ships a new model.

03 · Record

Your results sheet

Paste into any spreadsheet. One row per text, one sheet per detector.

id,set,truth,words,tool,score,label,called_ai,flagged_passages
h01,H,human,612,,,,,
h02,H,human,455,,,,,
a01,A,ai,580,,,,,
e01,E,ai,601,,,,,
p01,P,ai,540,,,,,

04 · Count

Work out the two rates

–False-positive rate: human writing it would have accused
–Detection rate: AI writing it caught

Detectors change their models

Get an email when the ranking changes