No published number answers the question “how accurate is AI contract review” for your contracts, because accuracy is not one measurement — it is a precision score and a recall score on a specific task, against a specific answer key, on a specific contract type. A vendor quoting one figure has collapsed all four of those choices into a number and kept the choices private. The useful question is never “what is your accuracy?” It is “on which task, scored how, against whose answer key, on which contracts?” This page gives you the five questions that recover those, with thresholds for what a good answer looks like.
The three substitutions hiding inside one number
Task substitution. The strongest independent evidence available is the Vals Legal AI Report (February 2025), the first benchmark to score legal AI tools against a lawyer control group on Am Law 100-derived work. On document extraction, the tools beat the humans: Harvey Assistant 75.1, CoCounsel 73.2, against a lawyer baseline of 71.1. On redlining — the task that is actually contract review — the ranking inverts: lawyer baseline 79.7, Harvey Assistant 65.0, Vincent AI 53.6. Same vendors, same study, opposite conclusion. A vendor is free to quote the extraction number and call it contract review accuracy.
Metric substitution. Many vendors benchmark on CUAD, the Atticus Project’s 510-contract, 41-clause-category dataset with 13,000+ expert annotations. Ivo publishes 97% on CUAD. CUAD’s own designers scored models by area under the precision-recall curve, because clause extraction is a needle-in-a-haystack problem where class imbalance makes plain accuracy meaningless — a model that flags nothing scores well on a metric that counts correct silences. Research models on CUAD’s intended metric have sat in the 40s. A high-90s “accuracy” figure on the same dataset is measuring something else, and the vendor rarely says what.
Grader substitution. Vals graded with an LLM-as-judge against reference answers decomposed into individual checks. Harvey’s LAB-AA benchmark, open-sourced May 2026 and reimplemented by Artificial Analysis across 120 private tasks in 24 practice areas, reports two numbers for every model: criterion pass rate (share of individual rubric criteria met) and all-pass rate (share of tasks where every criterion passes, no partial credit). Top criterion pass rates sit around 93-95%. The all-pass rate for the same systems is far lower, because professional deliverables fail on one bad clause. Which number a vendor quotes is a marketing decision.
The five questions
Ask these in the order below. Each has a threshold for a passing answer.
| # | Question | Passing answer | Failing answer |
|---|---|---|---|
| 1 | Which task was scored? | Named and narrow: “flagging deviations from a 24-position NDA playbook" | "Contract review” |
| 2 | Precision or recall, and what’s the other one? | Both quoted, or F-measure with the components available | A single “accuracy” figure |
| 3 | Who wrote the answer key, and can you see it? | Named annotators, disclosed disagreement rate, sample viewable | ”Expert-validated” |
| 4 | Which contracts, and were they unseen? | Contract type stated, held out from training | Dataset unnamed |
| 5 | Who ran it? | Independent third party, or vendor-run with published method | Vendor-run, method private |
Recall is the one that matters for review. Precision is the share of flags that were real; recall is the share of real issues that got flagged. A missed indemnity cap is a liability; a false flag costs 30 seconds of a lawyer’s attention. Demand recall, then check precision is high enough that reviewers don’t start ignoring the flags — under about 70% precision, teams stop reading them, and the recall number stops mattering.
Playbook fidelity is the metric nobody publishes. Every benchmark above scores against a general notion of legal correctness. Your team’s question is narrower: when your playbook says liability caps under 12 months of fees are unacceptable, does the tool flag a 9-month cap every time? That is playbook fidelity — agreement with your positions, not with a reasonable lawyer’s. No public benchmark measures it, and no vendor number predicts it. It has to be measured on your paper.
The canonical number, and why to stop citing it
The stat behind most AI contract review marketing is the 2018 LawGeex study: 20 US-trained lawyers against the vendor’s engine on 5 previously unseen NDAs containing 153 paragraphs, scored by F-measure with academic oversight from Stanford’s Roland Vogl and USC’s Gillian Hadfield. The AI scored 94% to the lawyers’ 85%, in 26 seconds against a human average of 92 minutes.
It is a real study, and it is eight years old, run by the vendor, on one contract type, against a fixed answer key of pre-defined risks. It also describes a product that no longer exists — LawGeex was dismantled in 2023. Treat “94% accurate” in a 2026 deck as a citation to a dead company’s marketing, and ask question 1.
Worked example: running your own 50-contract test
Budget one week of a contract manager’s time.
- Pull 50 executed contracts of one type — the type you actually process most, not a mix. Fewer than 30 and your confidence interval swallows the result.
- Have two reviewers independently mark issues against your playbook. Where they disagree, record it. Inter-annotator disagreement of 15-25% is normal; above 30% your playbook is the problem, not the AI, and no tool will score well against an answer key that two of your own people can’t reproduce.
- Run all 50 through the tool with your playbook loaded, not the vendor’s demo config.
- Count four buckets per contract: true positives, false positives, false negatives, and — the one teams forget — issues both reviewers missed that the tool caught.
- Compute recall and precision. Recall = TP / (TP + FN). Precision = TP / (TP + FP).
Calibrated expectations for a first-party test on your own paper: recall of 85-95% on high-frequency playbook positions (liability, indemnity, term, payment), 60-75% on positions appearing in under a fifth of your contracts, and precision of 70-85%. A tool clearing those on your contracts is worth deploying with human review. A tool scoring 97% in a vendor demo and 62% recall here is telling you the demo set was not your paper.
Pitfalls and guards
- Benchmarks migrate across tasks. A legal research accuracy figure gets quoted as contract review accuracy. Guard: the Stanford RegLab study of legal research tools found Lexis+ AI accurate on 65% of 202 queries and Westlaw AI-Assisted Research on 42%, with hallucination rates of 17% and 33% — those are research numbers, and they do not transfer to clause extraction in either direction. Make the vendor name the task before you write the number down.
- Retrieval grounding gets sold as a hallucination fix. Guard: the same study found tools built on retrieval-augmented generation still fabricating, which is the finding that contradicted vendor claims at the time. Ask for the fabrication rate on cited clause text specifically, and see grounding vs hallucination.
- The demo set is the training set. Guard: run the pilot on contracts the vendor has never seen, chosen by you after the demo.
- Accuracy is measured, adoption is not. A tool at 90% recall that reviewers stop opening delivers zero. Guard: track flags-reviewed alongside flags-raised for the first 60 days.
- One number covers all clause types. Aggregate recall hides that the tool finds payment terms and misses assignment-on-change-of-control. Guard: report recall per playbook position, not per contract.
When this framework breaks down
It assumes you have a written playbook and enough volume of one contract type to test against. Under roughly 200 contracts a year, or with no codified positions, the 50-contract test costs more than the tool saves — start with the NDA playbook and a documented contract review SOP, then benchmark. It also assumes review is the job. For extraction from an executed portfolio, the Vals result points the other way: that is the task where these tools already beat the lawyer baseline, and the bar for adopting them is lower.