AI スクリーナーのスコア分散を測定し、根拠のあるカットオフを設定する

---
name: screening-score-variance-audit
description: Measure how much your AI candidate screener's score moves when the same application is scored repeatedly, decompose the movement per rubric criterion, and convert it into a defensible cutoff plus a manual-review band. Reports the single-draw flip rate at your cutoff — the probability an identical application clears the bar on one run and fails on the next — not the standard error of an average nobody uses in production. Produces a measurement report, never a validity or fairness conclusion.
---

# Screening score variance audit

## When to invoke

Use this skill when an AI screener produces a numeric score, a rank, or a pass/fail on candidate applications, and that output gates a stage — auto-advance, auto-reject, or a recruiter working a sorted list top-down. It answers one question: how much of the score is the candidate, and how much is the draw.

The trigger is usually one of four. A screener is about to go live and someone asked what the cutoff should be. A recruiter noticed the same candidate scoring differently across two reqs. Counsel asked whether the score is reproducible. Or a vendor quoted a reliability figure computed on their data, and you need one computed on yours.

Inputs are a re-runnable screener and a candidate sample. The skill drives replications, then reads the resulting score matrix.

Do NOT invoke this skill for:

- **The bias audit.** NYC Local Law 144 requires an independent auditor and measures selection rates across sex, race, and ethnicity categories. That is disparate impact. This is consistency. A screener can be perfectly consistent and still produce a selection-rate disparity, and it can be wildly inconsistent with no measurable disparity at all. The two audits share no statistics. Running this one does not discharge that duty.
- **Proving the screener works.** Reliability is not validity. A criterion that returns the same number on every run and the same number for every candidate is perfectly stable and measures nothing — see the `experience` case in step 4. This skill can tell you a score is noise. It cannot tell you a stable score is signal.
- **Deterministic screeners.** Keyword matching, boolean filters, and rules engines return the same output on the same input by construction. Run one 3-run sanity check; if all three agree exactly, stop.
- **Screeners you cannot re-run without writing to the candidate record.** See the third watch-out. Get a sandbox first.
- **A stack already under a charge or demand letter.** At that point the score history is discovery material and counsel drives. Generating a parallel internal variance study of the same decisions creates a document you did not need.

## Inputs

- Required: `screener` — a callable that takes one application and returns the score payload, with the rubric criteria broken out and not only the total. An API endpoint, a CLI, or a sandbox job. If the screener returns only a total, say so at invocation; the skill drops step 4 and marks the report accordingly.
- Required: `harness_freeze` — the pinned configuration the replications run against, filled from `references/1-harness-freeze-sheet.md`. Model ID, prompt version, rubric version, parsing code, and the sampling settings the endpoint actually accepts.
- Required: `sample` — the candidate applications and their first-pass scores, framed per `references/2-sample-frame.md`.
- Required: `cutoff` — the score that gates the stage today, and what it gates. Give the real operating number, not the configuration default. If a recruiter works the top 40 of a ranked list of 400, the effective cutoff is the 40th score, not the threshold field.
- Optional: `n_runs` — replications per candidate. Defaults to 15. Raise to 30 when the number goes to counsel or a regulator; see step 3.
- Optional: `criteria_weights` — the rubric's weighting, if the total is not a plain sum.

## Reference files

- `references/1-harness-freeze-sheet.md` — what to pin before the first replication, and the cache-defeat checklist that stops the run from measuring zero variance for the wrong reason.
- `references/2-sample-frame.md` — the stratified sample template: band candidates, high anchors, low anchors, and why the anchors are not optional.
- `references/3-variance-report-template.md` — the output scaffold, with the per-criterion table, the flip-rate table, and the two statements the report is not allowed to make.

## Method

Six steps. The order matters: the harness freezes before anything runs, because a variance number computed across two prompt versions measures your deploy history rather than your screener.

### 1. Freeze the harness, and record what the endpoint will not let you fix

Fill `references/1-harness-freeze-sheet.md`. Pin the model ID exactly — a floating alias that resolves to a new snapshot mid-run splits the sample into two populations without announcing it.

Then record the sampling regime honestly, because there are three and they license different claims:

- **Temperature is set and settable.** Record the value. A low temperature narrows the output distribution; it does not close it.
- **Temperature is not exposed** by the vendor's screening product. Record it as unknown and treat the endpoint as the black box it is.
- **Temperature cannot be set at all.** On current frontier Claude models — Claude Opus 5, Opus 4.8, Opus 4.7, Sonnet 5, and Fable 5 — `temperature`, `top_p`, and `top_k` were removed from the Messages API, and a request carrying them returns a 400. If your screener runs on one of those, "we pinned temperature to 0" is not a statement anyone can make about it, and a vendor who claims it is describing a different model than the one they are running.

One more line goes on the sheet, because it is the fact that makes this audit necessary rather than paranoid: **temperature 0 is not determinism.** Thinking Machines Lab sampled 1,000 completions from Qwen3-235B-A22B-Instruct-2507 at temperature 0 with greedy decoding and got 80 distinct completions, first diverging at token 103. The cause is not floating-point luck — inference kernels are not batch-invariant, so the reduction tree a request runs through depends on how many other requests shared its batch. Their batch-invariant kernels do produce 1,000 identical completions, at roughly 1.6x the wall clock after optimization (26s to 42s on Qwen3-8B for 1,000 sequences). No hosted screening product ships them. Your screener's score depends, in small part, on how busy the endpoint was.

### 2. Frame a stratified sample, not a convenience sample

Pull K candidates per `references/2-sample-frame.md`. Default K is 40, split three ways:

- **24 in the band** — candidates whose first-pass score sits within roughly 1.5 points of the cutoff for every 1 point of expected score spread. These are the only candidates whose outcome can flip, so they carry the decision-relevant signal.
- **8 high anchors** and **8 low anchors** — candidates well clear of the cutoff in each direction.

The anchors are the part teams skip, and skipping them breaks the arithmetic rather than merely weakening it. Reliability is between-candidate variance divided by total variance. Sample only from a narrow band around the cutoff and you have truncated the between-candidate term by construction, so the ratio collapses toward zero and the screener looks worse than it is. The skill refuses to emit a reliability ratio when the sample has fewer than 6 anchors on either side; it emits the flip rates alone and says why.

Use real applications from a closed req. Synthetic resumes score differently, and the difference is not a constant you can subtract out.

### 3. Run N replications per candidate, and pick N for the interval you need

Default N is 15. That is not a round number chosen for comfort — it is where the estimate of the within-candidate spread becomes worth quoting. For a normal sample, the 95% confidence interval on a standard deviation estimated from 15 runs spans about 0.73 to 1.58 times the estimate. Wide, and honest about it. At 30 runs it tightens to about 0.80 to 1.34.

So: 15 when the output drives an internal cutoff decision, 30 when someone outside the team will quote the number back at you. Going past 30 buys little — the interval narrows with the square root of N, and the fourth digit of a noise estimate is not the constraint on the decision.

Run the replications independently. Do not batch a candidate's 15 runs into one prompt and ask for 15 scores; the model conditions on its own earlier answers and the spread you measure will be far narrower than production, where every application arrives cold.

### 4. Decompose per criterion before touching the total

The total's spread is an aggregate, and aggregates hide the two distinct pathologies that need different fixes. For each criterion, compute the within-candidate standard deviation (averaged across candidates) and the between-candidate standard deviation, then classify:

- **Noisy** — high within-candidate spread. The criterion is unstable and drags the total around. Its rubric language is the thing to fix.
- **Dead** — near-zero between-candidate spread. The criterion returns the same value for everyone. It is perfectly stable, contributes nothing to ranking, and inflates the apparent reliability of the total by padding the denominator with a constant.
- **Working** — low within, meaningful between.

The public HackerRank hiring-agent teardown shows both pathologies in one rubric. Running an unchanged resume PDF through it 100 times on `gemma3:4b` at temperature 0.1 returned totals from 66 to 99 out of 100. Underneath that: `technical skills` returned 8/10 on 98 of the 100 runs — working. `experience` returned 25/25 on every single run, which reads as rock-solid and is in fact dead, because it awards full marks position-agnostically and therefore never separates two candidates. And `projects`, the criterion with the most detailed rubric and the worked examples, was the noisiest of the set. More rubric text did not buy more stability.

That last observation is the one to carry into remediation: a criterion is unstable because it asks for a judgment the model resolves differently on different draws, and adding paragraphs of guidance to an open judgment widens the search rather than narrowing it. Split it into checkable sub-questions or drop it.

### 5. Compute the flip rate on single draws — never on the mean

This is the step that makes the audit useful, and the step every spreadsheet version gets wrong.

The audit has N scores per candidate. The temptation is to report the standard error of that mean, which shrinks by the square root of N and yields a reassuring figure. It is the wrong statistic, because production does not average 15 runs. Production takes exactly one draw and decides. The decision-relevant spread is the single-draw spread, undivided.

For each candidate, compute the empirical flip rate: the share of the N runs landing on the minority side of the cutoff. Report the maximum across the sample, the count of candidates with any flip at all, and the score interval those candidates occupy. That interval is the honest width of the machine's indecision.

The public example again, because it shows how completely cutoff placement drives the answer: with the score distribution ranging 66 to 99 and an 85-point cutoff, the same resume was rejected on about 65% of runs and accepted on the rest. The same teardown's second configuration — `Gemini-3.1-flash-lite`, 50 runs, scores clustered 45 to 65 — put the failure rate at 28% against a 60-point cutoff. Same class of instability, wildly different decision consequences, because the cutoff sat in a different part of the density. This is why a vendor-supplied reliability coefficient cannot answer your question: the number that matters is a joint property of their model and your cutoff.

### 6. Emit a cutoff and a manual-review band

The output is two numbers and a routing rule.

Set the manual-review band to the cutoff plus and minus twice the pooled within-candidate standard deviation. Everything inside the band routes to a human. Everything outside keeps its automated disposition.

The 2-sigma choice is the standard normal-approximation argument: for a candidate whose true score sits exactly at a band edge, a single draw lands on the wrong side of the cutoff about 2.3% of the time. But screener scores are usually not normal — they clump hard on rubric integers, as the 8/10-on-98-runs case shows — so the skill computes the empirical rate at the band edge as well and widens the band to whichever rule is more conservative. The report states which rule bound it.

Then state the throughput cost in applications, not in percentages, because that is the number the recruiting manager will push back on: with 400 applications per req and a band that catches 12% of them, someone reviews 48 applications by hand per req. If nobody will, the band is theatre, and the honest move is to widen the automated reject threshold instead, so fewer decisions ride on a coin the machine flipped.

## Output format

```
SCREENING SCORE VARIANCE AUDIT — req ENG-2291 "Senior Backend Engineer"
Harness: acme-screener v4.2 / claude-haiku-4-5 / rubric r7 / temperature 0.1
Sample: 40 candidates (24 band, 8 high anchor, 8 low anchor) x 15 runs = 600 evaluations
Cutoff audited: 78 (effective; config field reads 70, recruiter works top 40 of ~410)

PER-CRITERION
  criterion            within-SD   between-SD   verdict
  technical_skills          0.4          2.1    working
  experience                0.0          0.1    DEAD — full marks for 39/40 candidates
  projects                  3.8          2.4    NOISY — within exceeds between
  communication             1.1          1.7    working
  TOTAL                     4.6          5.2

FLIP RATE AT CUTOFF 78 (single draw, not mean)
  candidates with any flip across 15 runs: 11 of 40
  worst candidate: c-0318, 7/15 runs pass  (flip rate 47%)
  flip interval: scores 71 - 86

RECOMMENDED
  cutoff:              78  (unchanged — the cutoff is not the problem)
  manual-review band:  69 - 87  (empirical rule; 2-SD rule gave 69-87, same width)
  throughput cost:     ~14% of applications -> ~57 manual reviews per req at n=410

REMEDIATE FIRST
  1. projects — within-SD 3.8 is 83% of the total's spread. Split into checkable
     sub-questions or drop from the weighted total.
  2. experience — contributes 0 ranking signal. Removing it changes no ordering
     and stops it inflating the reliability of the total.

NOT ESTABLISHED BY THIS REPORT
  - whether the screener measures job performance (validity — not tested)
  - whether selection rates differ across protected categories (LL 144 bias
    audit — different statistic, independent auditor required)
```

## Watch-outs

- **Reporting the standard error of the mean.** It shrinks with N, it looks great, and it describes a procedure your production screener does not run. Guard: the report template has no field for it, and step 5 computes flip rates from raw single draws only.
- **Caching returning zero variance.** If the screener sits behind a result cache keyed on candidate ID, the replications return stored answers and the audit concludes the screener is perfectly stable. This failure is silent, and it is the most likely way this audit produces a confidently wrong answer. Guard: `references/1-harness-freeze-sheet.md` requires distinct request IDs across runs and a recorded count of distinct raw payloads before any statistic is computed. If all N payloads are byte-identical, the run is a cache test, not a variance test. Caching the *prompt prefix* is fine and cuts cost sharply — cached input reads bill at roughly a tenth of the input rate — because prefix reuse does not remove sampling variance. It is result-level caching that destroys the measurement.
- **Re-running live candidates through the production screener.** Fifteen replications of a real applicant can write 15 score events onto that person's record, fire 15 webhook deliveries into the ATS, and in the worst case send automated rejection mail 15 times. Guard: run against a sandbox req or a staging instance, and confirm before the first replication that the screener call is side-effect free. If it is not, this audit is blocked until it is — that is the correct outcome, not a reason to proceed carefully.
- **Auditing one req and generalizing.** Spread is a property of the rubric plus the model plus the applicant pool, and a req with a tight pool produces different between-candidate variance than an open one. Guard: the report names the req in its header, and the template's scope line states that the numbers bind to that req only.
- **Treating a stable score as a good score.** The `experience` case is the whole warning: 25/25 on every run for nearly every candidate. Guard: the per-criterion table always shows between-SD next to within-SD, and the skill labels a low-between criterion `DEAD` rather than leaving a zero to be read as excellence.
- **Letting the band become permanent.** A manual-review band is a mitigation for a screener you have not fixed. Guard: the report's remediation list is ordered by each criterion's share of total spread, so the next fix is always named, and the header carries the harness version so the audit's expiry is visible the moment the prompt changes.

# 1 — Harness freeze sheet

Fill this before the first replication runs. Everything below is a fact about the configuration the audit's numbers will describe. If any of it changes mid-run, the run is void — restart rather than patch, because a spread computed across two configurations is a measure of your deploy history.

```
frozen_at:      2026-__-__T__:__      # timestamp of the freeze, not of the report
frozen_by:
req_id:                               # the audit binds to one req; see watch-out 4
audit_owner:
```

## A. Screener identity

| Field | Value | Notes |
|---|---|---|
| Product / service name | | vendor product, or internal service name |
| Version or build | | the exact deployed build, not "latest" |
| Prompt version | | commit SHA or vendor prompt ID |
| Rubric version | | the criteria list the scores decompose into |
| Response parser version | | a parser change moves scores without the model moving |
| Scoring scale | | e.g. 0-100 total, 4 criteria |
| Criteria weights | | leave blank if the total is a plain sum |

## B. Model and sampling regime

Pin the model ID **exactly**. A floating alias (`-latest`, an unversioned product name) that resolves to a new snapshot part-way through 600 calls silently splits your sample into two populations.

| Field | Value |
|---|---|
| Model ID (exact string) | |
| Model ID is pinned, not a floating alias | ☐ yes ☐ no |
| Hosting | ☐ vendor SaaS ☐ our API key ☐ self-hosted |

Then tick exactly one sampling regime. They license different claims, and writing down which one applies is the point of this section:

- ☐ **Settable and set.** `temperature = ______`, `top_p = ______`. A low temperature narrows the output distribution. It does not close it.
- ☐ **Not exposed.** The vendor's screening product gives no sampling controls. Record as unknown and treat the endpoint as a black box. This is the most common case for bought screeners, and it is not a blocker — the audit measures the endpoint you actually run.
- ☐ **Cannot be set.** The model rejects sampling parameters outright. On current frontier Claude models — Claude Opus 5, Opus 4.8, Opus 4.7, Sonnet 5, Fable 5 — `temperature`, `top_p`, and `top_k` were removed from the Messages API and a request carrying them returns a 400. If a vendor tells you they pinned temperature to 0 on one of these, ask which model they are actually running.

**Determinism note, to be copied into the report verbatim:** temperature 0 does not mean reproducible. Thinking Machines Lab sampled 1,000 completions from Qwen3-235B-A22B-Instruct-2507 at temperature 0 with greedy decoding and got 80 distinct completions, diverging first at token 103, because inference kernels are not batch-invariant — the reduction tree depends on how many other requests shared the batch. Batch-invariant kernels fix it at roughly 1.6x wall clock, and no hosted screening product ships them.

## C. Side-effect clearance — blocking

Every box must be ticked before replication 1. A "no" anywhere blocks the audit; it does not license care.

- ☐ The screener call writes **no** score event to the candidate record
- ☐ The screener call fires **no** webhook into the ATS
- ☐ The screener call sends **no** candidate-facing mail
- ☐ The screener call increments **no** metered billing counter, or the cost is accepted and budgeted below
- ☐ The run targets a sandbox req / staging instance, **or** the four boxes above are all confirmed on production

Environment used: ______________________

## D. Cache-defeat checklist — blocking

A result cache keyed on candidate ID returns the stored answer to every replication, and the audit reports perfect stability. This is the single most likely way this audit produces a confidently wrong answer, and it produces no error while doing it.

- ☐ Result-level caching is disabled for the audit path, **or** each replication carries a cache-busting field the screener ignores for scoring
- ☐ Each replication carries a distinct request ID, and the IDs are logged
- ☐ After the run: count of **distinct raw response payloads** = ______ out of ______ total calls

If distinct payloads = 1, stop. You measured a cache. Fix the path and re-run.

Prompt-*prefix* caching is fine and is worth leaving on — cached input reads bill at roughly a tenth of the input rate, and prefix reuse does not remove sampling variance. Only result-level caching destroys the measurement.

## E. Run budget

Fill before starting so nobody discovers the bill afterwards. Default plan is 40 candidates x 15 runs = 600 evaluations.

| Field | Value |
|---|---|
| Candidates (K) | 40 |
| Replications (N) | 15 |
| Total evaluations | 600 |
| Cost per evaluation | $ |
| **Total estimated cost** | $ |
| Concurrency | |
| Estimated wall clock | |

Two cost regimes, and they differ by orders of magnitude:

- **Own API key.** A screening prompt of roughly 6,000 input and 800 output tokens costs about $0.01 per evaluation on Claude Haiku 4.5 ($1.00 / $5.00 per million input / output tokens) — about **$6** for the full 600-call run. On Claude Opus 5 ($5.00 / $25.00 per million) the same prompt runs about $0.05 per evaluation, or about **$30**. Either is a rounding error against one bad hire.
- **Per-screen vendor SKU.** If the screener bills per assessment at, say, $2, then 600 evaluations is **$1,200**. That is the reason K is 40 and not 400. Negotiate audit-mode calls, or drop N to 15 and K to 24 band candidates plus 6 anchors each side (216 calls) and accept the wider interval.

## F. Sign-off

```
harness frozen and side-effect cleared:  ______________  date: __________
cache-defeat verified post-run:          ______________  date: __________
```

# 2 — Sample frame

The sample decides what the audit can conclude. A convenience sample — the last 40 applications, or the 40 the recruiter remembers — produces numbers that describe that accident rather than the screener.

Pull from a **closed req** with real applications. Synthetic resumes score differently from real ones, and the difference is not a constant you can subtract back out.

## Step 1 — Establish the effective cutoff

Write down what actually gates the stage, which is rarely the number in the configuration screen.

```
config threshold field:        ______
applications per req:          ______
how many a human actually reads: ______
effective cutoff:              ______
what it gates:  ☐ auto-reject  ☐ auto-advance  ☐ recruiter works a sorted list top-down
```

If a recruiter works the top 40 of a ranked list of 410, the effective cutoff is the 40th score. Rank-ordering is a filter whatever the settings page calls it, and auditing the unused threshold field measures nothing.

## Step 2 — Estimate the band half-width

You need a rough score spread before you can pick band candidates, and you do not have one yet. Bootstrap it: pick any 3 applications near the cutoff and run each 5 times (15 calls, a few cents). Take the largest of the three per-candidate standard deviations, call it `s0`.

```
pilot candidate 1 scores: ___ ___ ___ ___ ___    SD: ____
pilot candidate 2 scores: ___ ___ ___ ___ ___    SD: ____
pilot candidate 3 scores: ___ ___ ___ ___ ___    SD: ____

s0 (largest)     = ______
band half-width  = 1.5 x s0 = ______
```

If all 15 pilot scores are identical, do not celebrate. Go back to `references/1-harness-freeze-sheet.md` §D and confirm you are not reading a cache. Only after distinct raw payloads are confirmed does an identical set mean the screener is stable.

## Step 3 — Fill the three strata

Default K is 40. Every row needs a first-pass score before it can be assigned to a stratum.

### Band — 24 candidates

First-pass score inside `effective cutoff ± band half-width`. These are the only candidates whose outcome can flip, so they carry the decision-relevant signal.

| # | candidate ref | first-pass score | distance from cutoff |
|---|---|---|---|
| 1 | | | |
| 2 | | | |
| … | | | |
| 24 | | | |

### High anchors — 8 candidates

First-pass score at least `3 x s0` **above** the cutoff.

| # | candidate ref | first-pass score |
|---|---|---|
| 1 | | |
| … | | |
| 8 | | |

### Low anchors — 8 candidates

First-pass score at least `3 x s0` **below** the cutoff.

| # | candidate ref | first-pass score |
|---|---|---|
| 1 | | |
| … | | |
| 8 | | |

## Why the anchors are not optional

Reliability is between-candidate variance divided by total variance. Sample only from a narrow band around the cutoff and the between-candidate term is truncated by construction — every candidate has nearly the same score, because that is how you selected them. The ratio then collapses toward zero and the screener reads as far worse than it is.

The skill refuses to emit a reliability ratio with fewer than 6 anchors on either side. It emits the flip rates alone and prints the reason, because flip rates survive a truncated sample and the ratio does not.

The anchors carry a second job: an anchor that flips is a much louder finding than a band candidate that flips. A candidate 3 standard deviations clear of the cutoff should never land on the wrong side. If one does, the spread is not normal-shaped and step 6's 2-sigma band rule will under-protect — the empirical rule takes over.

## Step 4 — Record the scope line

This sentence goes verbatim into the report header. It is the guard against the audit being quoted about a req it never touched.

```
These numbers describe req ________ , screened by ________ version ________
against rubric ________ , between ________ and ________ .
They do not transfer to another req, another rubric version, or another model snapshot.
```

## Sample-size variants

| Situation | K | N | Calls | Why |
|---|---|---|---|---|
| Default internal decision | 40 | 15 | 600 | SD interval ~0.73-1.58x |
| Number goes to counsel or a regulator | 40 | 30 | 1,200 | SD interval ~0.80-1.34x |
| Per-screen vendor SKU, tight budget | 36 | 6 | 216 | 24 band + 6 anchors each side; wider interval, stated in the report |
| Screener returns total only, no criteria | 40 | 15 | 600 | Step 4 of the method is skipped; report says so |

# 3 — Variance report template

The scaffold the skill fills. Two sections at the end are load-bearing: the remediation list, which names the next fix, and the not-established list, which stops the report being read as a clean bill of health.

Replace every `<>` placeholder. A placeholder left in the delivered report means that number was never computed, and a reader has no way to tell the difference unless it is still visibly a blank.

---

## Header

```
SCREENING SCORE VARIANCE AUDIT
req:            <req id> "<req title>"
screener:       <product> <version>
model:          <exact model id>       sampling: <set N / not exposed / cannot be set>
rubric:         <version>              parser: <version>
run window:     <start> to <end>
sample:         <K> candidates (<band> band, <hi> high anchor, <lo> low anchor) x <N> runs = <calls> evaluations
distinct raw payloads: <d> of <calls>          <- if d = 1, the run is void; see freeze sheet §D
cutoff audited: <effective cutoff>  (<how it was derived>)
```

Scope line, verbatim from `2-sample-frame.md` step 4:

```
These numbers describe req <> , screened by <> version <> against rubric <> ,
between <> and <> . They do not transfer to another req, another rubric
version, or another model snapshot.
```

## Per-criterion decomposition

`within-SD` is the average across candidates of each candidate's standard deviation over its N runs. `between-SD` is the standard deviation across candidates of their per-candidate means.

```
criterion              within-SD   between-SD   share of total spread   verdict
<name>                    <>           <>              <>%             working
<name>                    <>           <>              <>%             DEAD
<name>                    <>           <>              <>%             NOISY
TOTAL                     <>           <>             100%
```

Verdict rules, applied mechanically:

- `NOISY` — within-SD is at least as large as between-SD. The criterion moves more between runs of one candidate than it does between candidates.
- `DEAD` — between-SD is under 5% of the total's between-SD. The criterion returns near-identical values for everyone. It adds no ranking signal and pads the reliability denominator with a constant.
- `working` — neither.

A `DEAD` verdict on a zero-variance criterion is the finding most likely to be argued with, because a column of identical numbers reads as precision. Note in-line how many of the K candidates received the identical value; "full marks for 39 of 40" ends the argument faster than the standard deviation does.

## Flip rate at the cutoff

Computed on **single draws**. There is deliberately no field here for the standard error of the mean: production takes one draw and decides, so an average of N runs describes a procedure nobody runs.

```
candidates with at least one flip across N runs:  <> of <K>
worst candidate:  <ref>, <k>/<N> runs pass       (flip rate <>%)
flip interval:    scores <lo> - <hi>
anchors that flipped: <> of <hi+lo>              <- any non-zero value is a loud finding
```

Per-candidate detail, band candidates only:

```
candidate    mean    min   max   runs passing   flip rate
<ref>         <>     <>    <>       <>/<N>         <>%
```

## Recommendation

```
cutoff:               <>    (<changed / unchanged, and why>)
manual-review band:   <lo> - <hi>
band rule applied:    <2-SD rule / empirical rule>   <- name the more conservative one
throughput cost:      <>% of applications -> ~<n> manual reviews per req at <applications per req>
```

The throughput number goes in applications, not percentages. "12% of applications" gets nodded at; "48 resumes a week, by hand, per req" gets a decision. If nobody will staff it, say so in the report and recommend widening the automated reject threshold instead — fewer decisions then ride on a coin flip, which is the actual goal. A band nobody works is worse than no band, because it launders the same automated decision through a queue that reads as human review.

## Remediate first

Ordered by each criterion's share of total spread, so the largest reducible source of noise is always the top line.

```
1. <criterion> — within-SD <> is <>% of the total's spread.
   Fix: <split into checkable sub-questions / drop from the weighted total / re-word>
2. <criterion> — contributes <>% of ranking signal.
   Fix: <>
```

For a `NOISY` criterion, resist adding rubric text. The public HackerRank teardown found its noisiest criterion was `projects` — the one with the most detailed rubric and worked examples. More guidance on an open judgment widens the model's search rather than narrowing it. Split the judgment into questions with checkable answers, or drop it.

## Not established by this report

Copy this section unchanged. It is the guard against the report being screenshotted into a deck as a clearance.

```
- Whether the screener predicts job performance. That is validity. This audit
  measured consistency only. A perfectly consistent screener can be consistently
  measuring the wrong thing.
- Whether selection rates differ across sex, race, or ethnicity categories.
  That is a bias audit, it is a different statistic, and where NYC Local Law 144
  applies it requires an independent auditor. Nothing here discharges it.
- Whether the screener behaves the same on any other req, rubric version, or
  model snapshot. See the scope line.
```

## Expiry

```
this audit expires when any of these change:
  model id  <>            prompt version  <>
  rubric    <>            parser version  <>
re-run trigger recorded with: <owner / ticket / change-management hook>
```

An audit with no expiry hook becomes a permanent artifact describing a configuration that shipped over months ago. Wire the re-run to whatever gates prompt changes; if nothing gates them, that is the first finding.