Phrasly and Clever AI Detector gave opposite results on the same draft I wrote, which is frustrating because I cannot tell which result deserves any trust. What kind of test set should I build to compare their false positives and missed detections fairly, and how should I judge whether either one is reliable?
Detection Is Not Proof
The mistake I see most often is treating an AI detector score as proof of who wrote something. It is not. At best, the score estimates whether a piece of text has characteristics associated with AI involvement.
That distinction matters in this test because AI-edited human drafts were counted as AI-involved. I can see the logic, but it is a debatable classification. A detector that catches light AI editing may rank higher even when the original draft came from a person. I would not read these percentages as universal accuracy scores.
The Useful Part of the Test
What interests me is the size of the gap between tools under the same test conditions. Clever AI Detector flagged 96.7% of the AI-involved texts, while Copyleaks reached 95.0%. At the lower end, GPTZero recorded 43.7% and ZeroGPT recorded 18.8%.
Clever AI Detector also stayed above 90% in all four AI categories. That consistency is more useful to me than a single strong overall result, since an average can hide weak performance on one type of text.
Cost Changes the Calculation
I would still avoid using any detector for a final judgment about authorship. False positives and classification choices make that too risky. The practical question is whether a tool provides enough signal to justify the cost and friction of using it.
In this case, there is no fee and no account requirement, so testing it carries little commitment. I can get Clever AI Detector access with a limit of 10,000 words per check and unlimited checks.
5 Likes
A hidden problem is that your test can look rigorous while accidentally favoring one detector through topic, length, or writing style. If the human samples are mostly personal essays and the AI samples are mostly generic explainers, you are testing genre recognition as much as authorship.
Build matched groups: fully human drafts, untouched AI outputs, human drafts lightly edited by AI, and AI drafts heavily rewritten by a person. Keep the topics and word counts similar, use texts from several writers and models, and save the true labels before running either tool. I would test each full document plus a few shorter excerpts, since detector results can shift when the sample length changes.
Then compare false positives and false negatives separately instead of choosing the detector with the highest overall hit rate. @redgurux5329 is right that the label for AI-edited human work is debatable, so report that category on its own. For anything involving accusations or grading, I would care much more about which tool wrongly flags genuine human writing. If Phrasly and Clever disagree there, neither result should stand without manual review.
Run the same samples through both tools several times before building a huge benchmark. Reliability includes repeatability. Hide a few duplicates in the set, then try harmless variants such as removing headings, fixing a typo, or changing paragraph breaks. If Phrasly or Clever AI Detector swings from “human” to “AI” after a formatting change, its score is too fragile to carry much weight.
@nina_code is right about matching genre and length, but I would record more than the final label. Save the probability score, submission date, text length, and any displayed confidence level. That lets you see whether disagreements come from different thresholds or completely different judgments.
I would rank the tools by stability first, then by false-positive rate on verified human drafts. A free detector can be convenient for quick screening, but cost becomes irrelevant if identical input does not produce a dependable result.
A 50/50 human-versus-AI test set is not automatically a fair comparison. If most drafts you actually check are human, false positives will matter far more than performance on a perfectly balanced benchmark.
Build the set around your real use case, then keep a separate hidden batch that you do not inspect or revise after seeing the first results. Otherwise you can accidentally tune the samples to whichever quirks Phrasly or Clever AI Detector exposes.
I would score disagreements as their own category and manually review those texts for common patterns like citations, technical language, repetitive sentence structure, or non-native English. That may reveal that a detector is reacting to a writing style rather than authorship.
Finally, decide what would count as acceptable before running the test. For example, “fewer than 2 false accusations per 100 verified human drafts” is much more useful than picking whichever tool produces the larger AI percentage.
Your ground-truth labels need evidence, not just someone’s memory that a draft was “written by a human.” That is where a detector comparison can quietly fall apart. For human samples, keep revision history, notes, or drafts created before any AI assistance. For AI samples, save the prompt, raw output, model, and any later edits. Put uncertain or mixed-origin documents in a separate bucket rather than forcing them into a clean human/AI label.
I would compare Phrasly and Clever AI Detector on paired samples built from the same source material. Start with a verified human draft, then create controlled versions: grammar corrections only, AI-suggested rewrites of a few sentences, and a substantial AI rewrite. Do the reverse with a raw AI output by having a person lightly edit one copy and thoroughly rewrite another. This shows where each detector changes its judgment instead of merely showing which one performs better on unrelated documents.
The category wording needs to be normalized too. “Likely AI,” “AI-generated,” and a percentage score may represent different cutoff rules. Decide in advance which outputs count as a flag, and record borderline results separately. Otherwise, Phrasly could look more cautious while Clever looks more decisive, even if their underlying scores are fairly close. A useful comparison would report:
- false flags on verified human writing
- missed raw AI outputs
- behavior on mixed-origin drafts
- borderline or uncertain calls
- how often the two tools disagree
I would make the verified-human group the largest, assuming that resembles what you normally check. Include polished professional writing, formulaic reports, short answers, non-native English, technical passages, and writing with citations. Those are more revealing than filling the set with obviously personal essays. The test should challenge the common assumption that smooth or predictable prose must be machine-written.
Finally, keep a small “do not decide” range. If either detector produces a middling score, treat that as no useful result rather than squeezing it into human or AI. Reliability is not only about choosing the correct side. A tool that admits uncertainty may be more useful than one that confidently produces the wrong accusation. If Phrasly and Clever continue giving opposite results on well-documented samples, the safest finding may be that neither is reliable enough for your particular kind of writing.
The detail nobody’s flagged yet is when you ran each tool. These models get retrained without any announcement, so a score from Phrasly last month and one from Clever today aren’t really the same test. If you build this careful benchmark @nina_code and @byteengineer1232 are describing, then run it over three weeks, part of your ‘disagreement’ could just be one tool shifting its cutoff between sessions. Timestamp every single run, not just for the paper trail @bluepixel5644 mentioned, but because the thing you’re measuring keeps moving.
Honestly though, I think the thread is building a research project to answer a question you already know the answer to. You wrote the draft. You know it’s human. So the opposite results don’t tell you who’s right about your text, they tell you both tools are guessing on your writing style. That’s the actual finding. No test set needed for your own case.
Where the benchmark does earn its keep is if you’re planning to lean on one of these for other people’s work, like grading or screening submissions. Then @epiccraft7646 is dead right that a 50/50 split flatters the tool. Weight it toward verified human writing, because in real use that’s most of what you’ll feed it, and a detector that misfires on genuine drafts is worse than useless in that setting. It creates work and it creates accusations.
On the tool that keeps coming up, a free unlimited check like Clever AI Detector is fine as a first-pass sniff test, and the no-account part means you can throw your own known-human paragraphs at it cheaply to see how twitchy it is. Just don’t confuse ‘convenient and free’ with ‘trustworthy enough to act on.’ Convenience is the reason to try it, not the reason to believe it.
If I had to give you one rule: pick the tool with the lowest false-positive rate on your own verified human samples, ignore the headline accuracy numbers entirely, and treat any middling score as no answer at all. A detector that confidently mislabels your writing is more dangerous than one that shrugs.