A Claude Skill that turns a finished AI SDR pilot into a keep, kill, or extend memo, scored against criteria the team wrote down before the first send. It computes four axes — fully-loaded cost per qualified meeting, meetings per positive reply, human rework hours, and sending-reputation delta — and refuses to return a verdict when the criteria were amended mid-pilot or the sample is too small to carry one. The bundle at apps/web/public/artifacts/ai-sdr-pilot-scorecard-skill/ ships SKILL.md plus four reference files: the pre-registration template the team fills in before the pilot starts, the five-line cost model, the metric definitions, and two literal sample memos.
The problem it solves is not measurement. It is that almost nobody writes the kill line down first. A pilot ends, the vendor’s dashboard shows meetings booked, somebody divides the license fee by that number, and the result clears a threshold nobody registered in advance. Three months later the contract auto-renews because no one produced a document that said what failure would have looked like.
When to use
Run it 14 days after the last pilot send, with the full export. The 14-day tail is deliberate: meetings booked in the final week have not held yet, and complaint-driven filtering degrades on a lag, so a pilot scored on its end date is scored before its own consequences arrive.
Two other modes. A mid-pilot checkpoint with checkpoint: true computes the axes and returns no_verdict — that run exists to catch a reputation breach at week three, not to decide anything. And a renewal review on an already-running deployment treats the trailing 90 days as the window; it is the one mode where a pre-registration written after the fact is acceptable, and the memo carries retrospective: true on its face forever after.
The skill fits 11x, Artisan, aisdr, and homegrown agents equally, because it scores your systems rather than the vendor’s. Everything it reads comes from your CRM, your finance export, your timesheets, and Google Postmaster Tools.
When NOT to use
No pre-registration exists. The skill returns no_preregistration and stops. Writing the criteria now and backdating them produces a worse artifact than no memo — it launders a post-hoc judgment into something that looks like a measurement. The renewal-review mode above is the only exception, and it labels itself.
Below the registered minimum sample. Returns insufficient_sample with observed and required counts. With n qualified meetings, adding or removing one moves cost per meeting by roughly 1/n — 12.5% at eight meetings, 5% at twenty. A six-week pilot that produced nine held meetings cannot distinguish a 4% reply-to-meeting rate from a 7% one, and a verdict computed on it is noise with a decimal point.
Choosing between two vendors. This scores one deployment against a threshold, not two against each other. A head-to-head needs both running on comparable segments with a shared exclusion list, which is a different design.
A human SDR team’s quarterly review. The cost model assumes a per-lead or per-contact meter and a rework line that does not exist in the same shape for headcount.
Setup
Two to four hours, most of it deciding thresholds rather than wiring anything.
Fill in the pre-registration before the first send. Copy references/1-preregistration-template.md, replace every placeholder, commit it. Set min_held_meetings so that one meeting moving in or out changes cost per meeting by less than the margin between your threshold and your expected value — if you expect to land near 700 against a ceiling of 750, that margin is about 7% and twenty meetings is the floor.
Set max_cost_per_qualified_meeting against your current motion, not the vendor’s list price. Published prices only give you the subscription line. AiSDR publishes 250 per month for 200 AI-researched contacts, 900 for 800, and 2,500 for 2,500, with the two larger tiers on quarterly commitment. 11x publishes 3,750 per month billed annually for Growth — 45,000 for the year — covering 2,000 new prospects per month and up to five end users, and states it charges per lead rather than per send. Artisan publishes no price and scopes its tiers at roughly 2,500 and roughly 6,000 leads contacted per month.
Set max_spam_rate to 0.0030 and leave it there. That is Google’s published requirement for senders of more than 5,000 messages per day to Gmail accounts: keep the spam rate reported in Postmaster Tools under 0.30%. Google separately recommends staying under 0.10% so an ordinary complaint spike does not reach the hard line. On a Gmail-heavy list set the working threshold at 0.0010 and treat 0.0030 as the stop.
Set the exclusion rules now.exclude_open_opp_at_booking and exclude_prior_human_touch_within_days: 90 are the two that decide what counts as the agent’s meeting. Loosen them here, in writing, before you know which way it cuts.
Instrument rework hours from day one. A weekly self-reported number from the two people doing the work is imprecise and honest. A number reconstructed at the end by the person advocating for the tool is precise and useless.
Install the skill. Drop SKILL.md and references/ into .claude/skills/ai-sdr-pilot-scorecard/, or upload it as a Skill in claude.ai. Assemble the input payload from your CRM export, finance, timesheets, and Postmaster.
Why the four axes are built this way
Cost per qualified meeting is emitted twice. Once on all five cost lines — subscription, data and enrichment, sending infrastructure, amortized implementation, and rework hours priced at a loaded rate — and once on subscription alone. Both go on the memo. The subscription-only figure is the number in the vendor’s business case; it is not wrong, it is answering a different question, and printing them side by side makes the basis visible rather than arguable. In the worked example in references/2-cost-model.md, a 42-day pilot with 14 qualified meetings comes to 851 per meeting loaded and 370 on subscription alone, and 6,035 of the 6,735 that separates them is a single line: 71 hours of human rework. That distinction changes the next step. “Too expensive” is a negotiation; “six hours of human work per meeting produced” is a product problem a discount does not fix.
Reply-to-meeting counts positive replies only. A negative reply, an out-of-office, and an unsubscribe request are not conversion opportunities, and pooling them into one denominator is how a reply-rate chart stays flat while the list burns. Where the vendor’s own classifier labeled the replies, references/3-metric-definitions.md applies a 0.80 discount and marks the figure unaudited — a convention that file sets so runs stay comparable, not a measured industry number. Supply a 50-reply human-labeled stratified sample and the skill computes the real precision instead.
Rework is scored on trend as well as level. Falling week over week means the team is learning the tool; flat or rising means the tool is not converging. Same level, opposite verdicts.
Reputation is a hard stop, evaluated independently of the economics. If the tail window crosses the absolute line, the verdict is kill no matter what cost per meeting says. Sender reputation is shared across everything the company sends from those domains — invoices, renewals, support replies — so a pilot cannot spend it and book the saving as a result.
Failure modes and guards
The kill line moves mid-pilot. This is the most common way a failing pilot survives, and it is rarely dishonest — it is a threshold quietly softened in a doc by someone who has already started to believe. Guard: step 1 hashes the pre-registration and refuses to score against an amended one, returning a field-level diff instead of a verdict. Amending is allowed; re-register with a new hash and the memo carries retrospective: true permanently.
Attribution inflation. Vendors count meetings on accounts already in cycle or recently human-touched. Guard: excluded meetings leave the numerator by rule rather than by negotiated discount, and the excluded count and reason distribution are reported as a finding. A vendor whose attributed meetings are 32% already-open accounts is telling you something about its sourcing that no discount addresses.
The subscription-only denominator.Guard: the run is rejected with incomplete_cost_model if any of the five lines is null. A line you believe is zero must be entered as 0, which is a claim rather than a gap.
Damage landing after the scoring date.Guard: the 14-day tail window is mandatory and the reputation axis can kill a pilot that cleared every cost threshold.
keep read as “scale”. The skill scores one configuration at one volume, and volume is the input most likely to break the reputation axis. Guard: the memo’s final section names the scored volume and makes re-scoring a condition of any increase.
Against the alternatives
The vendor’s QBR deck reports the numerator. It has no access to your rework hours, no incentive to apply your exclusion rules, and it labels its own positive replies. Read it, then run this.
A RevOps analyst’s spreadsheet gets the arithmetic right and still loses to criteria drift, because the sheet is built after the results are visible and the thresholds get chosen alongside them. The hash is the only part of this workflow that a spreadsheet structurally cannot replicate.
Letting the contract auto-renew is the actual default, and it is the one this exists to displace. On an 11x Growth commitment that default costs 45,000 for a year nobody decided to buy.
---
name: ai-sdr-pilot-scorecard
description: Scores a finished or in-flight AI SDR / GTM-agent pilot against criteria registered before the pilot started, and emits a keep / kill / extend memo. Computes fully-loaded cost per qualified meeting, reply-to-meeting conversion split by reply polarity, human rework hours, and sending-reputation delta. Refuses to score against amended criteria or below the registered minimum sample. Use at the end of a time-boxed pilot, before the contract auto-renews.
---
# AI SDR pilot scorecard
## When to invoke
Invoke when a time-boxed AI SDR or GTM-agent pilot reaches its end date, or at a mid-pilot checkpoint the pre-registration named. Production patterns:
- **End-of-pilot decision.** Run once, 14 days after the last send, with the full export. The 14-day tail exists because meetings booked in the final week have not yet held, and reputation damage lands after the sending stops.
- **Mid-pilot checkpoint.** Run at the halfway mark against the same registered criteria, with `checkpoint: true`. The skill returns axis-level readings and an explicit `no_verdict` — a mid-pilot run is for catching a reputation breach early, not for deciding.
- **Renewal review on a running deployment.** Treat the trailing 90 days as the pilot window. This is the only mode where a pre-registration written after the fact is acceptable, and the memo says so on its face.
Do NOT invoke this skill for:
- **A pilot with no pre-registration.** The skill returns `no_preregistration` and stops. Scoring criteria chosen after seeing the results is the failure this skill exists to prevent — writing them now and backdating them produces a worse artifact than no memo at all. The exception is the renewal-review mode above, which stamps the memo accordingly.
- **A pilot below the registered minimum sample.** Returns `insufficient_sample` with the observed and required counts. A six-week pilot that produced 9 held meetings cannot distinguish a 4% reply-to-meeting rate from a 7% one, and a verdict computed on it is noise wearing a decimal point.
- **Choosing between two vendors.** This skill scores one deployment against a threshold, not two against each other. A head-to-head needs both running on comparable segments with a shared exclusion list, which is a different design.
- **A fully human SDR team's quarterly review.** The cost model assumes a per-lead or per-contact meter and a human-rework line that does not exist in the same shape for a headcount-based team.
## Inputs
Required:
- `preregistration` — object. The parsed contents of the pre-registration file written before the pilot started. Required shape: `{ registered_at, window_start, window_end, min_held_meetings, thresholds: { max_cost_per_qualified_meeting, min_reply_to_meeting_rate, max_rework_hours_per_meeting, max_spam_rate }, exclusion_rules, qualification_gate }`.
- `preregistration_sha256` — string. SHA-256 of the pre-registration file as committed. The skill recomputes it and refuses to score on a mismatch.
- `meetings` — array. One entry per meeting the vendor or your CRM attributes to the pilot: `{ account_id, booked_at, held, held_at, stage_at_plus_30d, prior_human_touch_at, open_opp_at_booking, source_claimed_by }`.
- `costs` — object. All five cost lines, in the currency of the contract: `{ subscription, data_and_enrichment, sending_infrastructure, implementation_amortized, human_rework_hours }`. A `null` in any line is a hard rejection, not a zero.
- `loaded_hourly_rate` — number. Fully-loaded cost of an hour of the human doing the rework. Used to price `human_rework_hours`.
- `replies` — array. `{ account_id, replied_at, polarity, labeled_by }` where `polarity` is `positive` / `negative` / `out_of_office` / `unsubscribe_request` and `labeled_by` is `vendor_classifier` or `human`.
- `reputation` — object. `{ baseline_window, pilot_window, tail_window }`, each `{ spam_rate, domain, daily_volume_peak }` read from Google Postmaster Tools or the equivalent for your recipient mix.
Optional:
- `checkpoint` — boolean. Default `false`. When `true`, the skill computes the axes and returns `no_verdict`.
- `reply_label_audit` — array. A stratified human-labeled sample of replies, used to compute the vendor classifier's precision on `positive`. Without it the skill discounts vendor-labeled positives by the documented default and says so in the memo.
- `counterfactual` — object. `{ meetings_held, window }` from a matched human-SDR segment or the same segment's prior period. Turns the memo's incremental-lift section from a caveat into a number.
## Reference files
Load these from `references/` before first run. All four are stable within a deployment.
- `references/1-preregistration-template.md` — the file the team fills in **before** the pilot starts. Contains the four thresholds with the reasoning for each default, the exclusion-rule scaffolding, and the qualification-gate definition.
- `references/2-cost-model.md` — the five cost lines, what belongs in each, and the amortization rule for implementation fees. Contains a worked example against a published vendor price.
- `references/3-metric-definitions.md` — exact definitions and the arithmetic for each axis, including the reply-polarity split and the reputation window construction.
- `references/4-sample-memo.md` — a literal `kill` memo and a literal `keep` memo, plus the structured-field contract for parsers.
## Method
Run these steps in order. Steps 1 and 2 gate everything after them.
### 1. Pre-registration integrity check
Recompute SHA-256 over the `preregistration` object as serialized in the template's canonical form and compare to `preregistration_sha256`. On mismatch, return `result: criteria_amended` with a diff of the changed threshold fields — do not score. On a missing pre-registration, return `result: no_preregistration`.
The hash is the whole point of the step. Criteria drift is not usually dishonest; it is a threshold quietly softened in a doc three weeks into a pilot that is not going well, by someone who has already started to believe. A hash makes the softening a visible event rather than an invisible one.
Also verify `registered_at` precedes `window_start`. A pre-registration written after the first send is not a pre-registration; flag it as `retrospective: true` and carry that flag onto the memo.
### 2. Sample-adequacy gate
Count meetings that pass both the exclusion rules and the qualification gate. If the count is below `min_held_meetings`, return `result: insufficient_sample` with observed and required counts, and a note on how many more weeks at the observed rate would reach it. Do not return a verdict.
This gate is upstream of the economics for a reason. Cost per qualified meeting on a denominator of 7 moves by more than 14% every time one meeting is added or removed, which is wider than the gap most kill thresholds are set at.
### 3. Attribution filtering
Apply the exclusion rules from the pre-registration to every entry in `meetings`. The defaults in the template exclude a meeting when any of these holds:
- `open_opp_at_booking` is true — the account already had an open opportunity.
- `prior_human_touch_at` falls inside the 90 days before `booked_at`.
- The account appears on a named-account list the pre-registration excluded.
Excluded meetings are removed from the numerator entirely, not discounted by a factor. A discount is a judgment call that gets negotiated; a rule is a rule. Report the excluded count and the exclusion reason distribution — the ratio of excluded to included is itself a finding, and a vendor whose attributed meetings are 40% already-open accounts is telling you something about its sourcing.
### 4. Qualification gate
Apply `qualification_gate` from the pre-registration to the surviving meetings. The template default requires both `held` is true and `stage_at_plus_30d` has advanced past the initial stage. Booked-but-no-show meetings count as zero and stay out of the numerator; they are still counted separately and reported, because a high no-show rate on a healthy booking rate is a targeting problem the cost figure would otherwise hide.
### 5. Cost assembly
Sum the five lines from `costs`, pricing `human_rework_hours` at `loaded_hourly_rate`. Reject the run with `result: incomplete_cost_model` if any line is `null`.
Divide by the qualified-meeting count from step 4 to get fully-loaded cost per qualified meeting. Report the same figure computed on subscription alone, labeled as the vendor-facing number, so the memo shows both and the gap between them is legible. That gap is the argument.
### 6. Reply-to-meeting conversion, split by polarity
Compute replies per contacted account, then meetings per reply. Split the reply count by `polarity`. A `negative` reply and an `unsubscribe_request` are not conversion opportunities and must not sit in the same denominator as a `positive` one — collapsing them is how a reply-rate chart stays flat while the underlying list burns.
Where `labeled_by` is `vendor_classifier` and `reply_label_audit` is supplied, compute the classifier's precision on `positive` and multiply the vendor-labeled positive count by it. Where the audit is absent, apply the documented discount from `references/3-metric-definitions.md` and mark the figure `unaudited` in the memo.
### 7. Reputation delta
Compare `spam_rate` across the three windows. Two independent readings:
- **Trend.** Pilot and tail against baseline, on the same domains.
- **Absolute line.** Whether any window crossed the bulk-sender complaint threshold your recipient mix is judged against. For Gmail recipients at more than 5,000 messages per day, Google's published requirement is to keep the reported spam rate under 0.30%, with under 0.10% as the recommended operating point.
The tail window is where the damage usually shows. A pilot that ends on day 42 and is scored on day 42 is scored before its own consequences arrive.
**This axis is a hard stop, evaluated independently of the economics.** If the tail window crossed the absolute line, the verdict is `kill` regardless of cost per meeting, and the memo says the pilot was cheap because it was spending an asset that is not on the pilot's ledger — the deliverability of every mailbox in the company.
### 8. Verdict assembly
Compare each axis against its registered threshold and return one verdict:
- `keep` — every axis inside threshold.
- `extend` — at most one axis outside threshold, the reputation axis inside its hard stop, and the observed trend on the failing axis improving across the pilot window. The memo names the single axis and the specific reading that would make it a `keep`.
- `kill` — the reputation hard stop was crossed, or two or more axes are outside threshold, or one axis is outside threshold with a flat or worsening trend.
- `no_verdict` — `checkpoint: true`.
The `extend` verdict is deliberately narrow. A verdict of `extend` available on any failing configuration is not a verdict, it is a way of never deciding, and it is the outcome pilots default to when nobody wrote the thresholds down.
## Output format
Literal JSON the skill emits for a `kill` verdict:
```json
{
"verdict": "kill",
"result": "ok",
"retrospective": false,
"window": { "start": "2026-05-04", "end": "2026-06-15", "tail_end": "2026-06-29" },
"sample": {
"meetings_attributed": 34,
"excluded": 11,
"exclusion_reasons": { "open_opp_at_booking": 7, "prior_human_touch_90d": 4 },
"held": 19,
"qualified": 12,
"min_required": 12
},
"axes": [
{
"axis": "cost_per_qualified_meeting",
"observed": 1042.00,
"threshold": 750.00,
"status": "outside",
"trend": "flat",
"subscription_only_figure": 433.33,
"note": "Vendor-facing figure omits 71 rework hours and the enrichment line."
},
{
"axis": "reply_to_meeting_rate",
"observed": 0.061,
"threshold": 0.050,
"status": "inside",
"trend": "improving",
"positive_replies_audited": false,
"note": "Vendor-classifier positives discounted by the default factor; no reply_label_audit supplied."
},
{
"axis": "rework_hours_per_qualified_meeting",
"observed": 5.9,
"threshold": 3.0,
"status": "outside",
"trend": "worsening"
},
{
"axis": "reputation_delta",
"observed_tail_spam_rate": 0.0034,
"absolute_line": 0.0030,
"baseline_spam_rate": 0.0008,
"status": "hard_stop_crossed",
"note": "Tail window on sending domain crossed the bulk-sender complaint line 11 days after the last pilot send."
}
],
"counterfactual": null,
"memo_markdown": "...",
"run_metadata": {
"preregistration_sha256": "…",
"model": "claude-opus-5",
"definitions_version": "1.0.0"
}
}
```
A `keep` verdict has every axis at `status: "inside"` and a populated `memo_markdown`. An `extend` verdict has exactly one axis `outside`, `reputation_delta` inside its line, and a `path_to_keep` object naming the required reading. A `no_verdict` response carries populated `axes` and a `null` verdict.
The `memo_markdown` field is the deliverable a human reads. Its structure is fixed in `references/4-sample-memo.md` — decision first, the axis that drove it second, the arithmetic third, and what would change the answer last.
## Watch-outs
- **Criteria drift mid-pilot.** The most common way a pilot survives is that its kill line moves. **Guard:** step 1 hashes the pre-registration and refuses to score against an amended one, returning a field-level diff instead of a verdict. Amending is allowed — the team re-registers with a new hash, and the memo carries `retrospective: true` forever after.
- **Attribution inflation.** Vendors count meetings on accounts that were already in cycle or already human-touched. **Guard:** step 3 removes them from the numerator by rule rather than discounting them by negotiation, and reports the excluded count and reason distribution as a finding in its own right.
- **The subscription-only denominator.** Cost per meeting computed on the license price alone reliably understates the real figure, because the rework hours and the enrichment meter are the lines nobody put on the pilot budget. **Guard:** step 5 rejects the run when any of the five cost lines is `null`, and the memo prints the vendor-facing figure next to the loaded one so the gap is visible rather than arguable.
- **Reputation damage arriving after the scoring date.** Complaint-driven filtering degrades on a lag, so a pilot scored on its end date is scored before its cost lands. **Guard:** the tail window is mandatory, and the reputation axis is a hard stop evaluated independently of the economics — it can kill a pilot that cleared every cost threshold.
- **Vendor-classified positive replies.** "Positive reply" labeled by the system being evaluated is a self-graded exam. **Guard:** step 6 computes classifier precision against `reply_label_audit` when supplied and applies the documented discount when it is not, marking the figure `unaudited` on the memo rather than silently trusting it.
- **A verdict on a sample too small to carry one.** **Guard:** step 2 gates on the registered minimum and returns `insufficient_sample` — with the weeks-to-adequacy estimate — rather than a confident number computed on nine meetings.
- **`keep` read as "scale".** The skill scores the pilot's configuration at the pilot's volume. Send volume is the input most likely to break the reputation axis, and a `keep` at 2,000 contacts per month is not a `keep` at 10,000. **Guard:** the memo's final section states the scored volume explicitly and names re-scoring at the new volume as a condition of any increase.
# 1 — Pre-registration template
Fill this in **before the first send**. Commit it. The skill hashes it and refuses to score against a version that changed afterwards.
Replace every `<…>` placeholder. Do not delete fields you think do not apply — set them explicitly, because a missing threshold and a threshold of "we did not care" are different claims and the memo distinguishes them.
```yaml
registered_at: <YYYY-MM-DD> # must precede window_start
registered_by: <name, role>
vendor: <11x | artisan | aisdr | unify | homegrown | other>
plan_and_meter: <e.g. "Growth, 2,000 new prospects/mo, 5 end users">
window_start: <YYYY-MM-DD> # first send
window_end: <YYYY-MM-DD> # last send
tail_days: 14 # do not lower; see reputation note below
min_held_meetings: <integer> # see sizing note below
thresholds:
max_cost_per_qualified_meeting: <number> # your currency
min_reply_to_meeting_rate: <decimal> # meetings per positive reply
max_rework_hours_per_meeting: <number>
max_spam_rate: 0.0030 # absolute line; see note
qualification_gate:
require_held: true
require_stage_advance_by_days: 30
exclusion_rules:
exclude_open_opp_at_booking: true
exclude_prior_human_touch_within_days: 90
excluded_named_account_lists:
- <list name or CRM view id>
loaded_hourly_rate: <number> # fully loaded, not salary/2080
```
## Sizing `min_held_meetings`
Set it so that one meeting moving in or out changes cost per qualified meeting by less than the margin between your threshold and your expected value. The arithmetic: with `n` qualified meetings, adding or removing one moves the per-meeting cost by roughly `1/n`. At `n = 8` that is 12.5%; at `n = 20` it is 5%; at `n = 40` it is 2.5%.
If your threshold is 750 and you expect to land near 700, your margin is about 7% and `n = 20` is the floor. If you expect to land near 400 against a threshold of 750, the margin is wide and `n = 12` is defensible.
If the pilot cannot plausibly produce your `n` in the window, the honest move is to lengthen the window before starting, not to score a short one. Write the required `n` down anyway — an `insufficient_sample` return at the end is a real finding, and it is the finding that the pilot was not designed to answer the question.
## The four thresholds
**`max_cost_per_qualified_meeting`.** Anchor it to what the same meeting costs you today through your existing motion, not to the vendor's list price. Published entry prices give you the floor of the subscription line only: AiSDR publishes 250 per month for 200 AI-researched contacts, 900 for 800, and 2,500 for 2,500, with the middle and top tiers on a quarterly commitment. 11x publishes 3,750 per month billed annually for its Growth plan, covering 2,000 new prospects per month and up to five end users, and states it charges per lead rather than per send. Artisan publishes no price and scopes its tiers at roughly 2,500 and roughly 6,000 leads contacted per month. Those are subscription lines, not costs per meeting — the cost model in `2-cost-model.md` adds the four other lines that turn one into the other.
**`min_reply_to_meeting_rate`.** Meetings per *positive* reply, not per reply. Set it from your own historical rate on the same segment if you have one. If you do not, set it and note that you do not — an unanchored threshold you registered is still better than a rate you interpret after the fact.
**`max_rework_hours_per_meeting`.** The hours a human spends editing drafts, correcting bad targeting, cleaning the CRM after the agent, and handling replies the agent mishandled. This is the line that is never in the business case. Instrument it from day one, even crudely: a weekly self-reported number from the two people doing it beats a precise number you reconstruct at the end.
**`max_spam_rate`.** The default of 0.0030 is Google's published bulk-sender requirement for senders of more than 5,000 messages per day to Gmail accounts, which is to keep the spam rate reported in Postmaster Tools under 0.30%. Google separately recommends staying under 0.10% so that an ordinary complaint spike does not push you over. If your recipient mix is Gmail-heavy, set the threshold at 0.0010 and treat 0.0030 as the hard stop. Do not raise this field. It is not a preference; it is the line above which a mailbox provider starts making decisions about your domain.
## The exclusion rules
These decide what counts as the agent's meeting. Set them now, because after the pilot every excluded meeting is an argument.
`exclude_open_opp_at_booking` and `exclude_prior_human_touch_within_days: 90` are the two that matter most. An agent that books a meeting on an account your AE has been working for a month did not source that meeting; it scheduled it. Both defaults are deliberately strict — if you want them looser, loosen them here, in writing, before you know which way it cuts.
`excluded_named_account_lists` exists for the top-of-house accounts you would never let an agent source into. If that list is empty, say so explicitly by leaving the key with an empty list rather than deleting it.
## Amending
You may amend. Re-register with a new `registered_at`, commit, and re-hash. The skill will score against the new file and stamp `retrospective: true` on the memo permanently. That stamp is not a punishment — it is the correct label for a number produced against criteria chosen with partial knowledge of the outcome, and a reader of the memo six months from now needs it.
# 2 — Cost model
Five lines. All five are required; the skill rejects the run with `incomplete_cost_model` if any is `null`. A line you believe is zero must be entered as `0`, which is a claim you are making, not a gap you are leaving.
## Line 1 — `subscription`
The contracted license cost for the pilot window, prorated if the contract is annual.
Vendor meters differ in kind, not just in price, and the difference changes what "more volume" costs you:
- **Per lead or per prospect.** 11x publishes a Growth plan at 3,750 per month billed annually — 45,000 for the year — covering 2,000 new prospects per month and up to five end users, and states plainly that it charges per lead, not per send. Under a per-lead meter, sending more touches to the same 2,000 prospects is free and widening the list is not.
- **Per AI-researched contact.** AiSDR publishes 250 per month for 200 contacts, 900 per month for 800, and 2,500 per month for 2,500, with the two larger tiers on a quarterly commitment and a 20% discount for annual billing. The unit price falls from 1.25 per contact at the entry tier to 1.00 at the top published one.
- **Quote-only, scoped by leads contacted.** Artisan publishes no price and describes its tiers at roughly 2,500 and roughly 6,000 leads contacted per month.
Prorating an annual commitment across a six-week pilot: `subscription = annual_total × (pilot_days / 365)`. For 11x Growth at 45,000 per year and a 42-day window, that is 45,000 × 42/365, or about 5,178. Use the prorated figure, and note in the memo that the real commitment is annual — a pilot that cost 5,178 to run may have cost 45,000 to enter, and if the contract is already signed, the kill decision is about the next term, not this one.
## Line 2 — `data_and_enrichment`
Credits consumed for enrichment, verification, and intent data during the window. Include spend on tools the agent called even when they are billed elsewhere — a Clay or ZoomInfo meter that spiked during the pilot is a pilot cost regardless of which cost centre it landed in.
Bundled-plan trap: several vendors bundle contact data into the subscription, which does not make it free, it makes it invisible. When it is bundled, enter `0` and note the bundling in the memo, because the moment you scale past the bundled allowance it becomes a variable line and your cost per meeting moves.
## Line 3 — `sending_infrastructure`
Domains, mailboxes, warmup services, and any deliverability tooling bought or expanded for the pilot. Also invisible when bundled — 11x, for instance, states that every plan bundles deliverability, mailbox warmup, and inbox rotation at no additional cost.
The reason this line exists even when it is small: it is the line that grows fastest under volume increases, and a `keep` verdict that leads to tripled volume needs the reader to know which lines are fixed and which are not.
## Line 4 — `implementation_amortized`
One-time onboarding, implementation, and integration fees, amortized across the term you would actually commit to if you keep. Not across the pilot window.
A 10,000 implementation fee charged against a six-week pilot makes the pilot look catastrophic; charged across a twelve-month term at 833 per month, it makes the decision you are actually making. If you would only ever commit to twelve months, amortize across twelve. If the fee is refundable on non-renewal, enter `0` and note it.
## Line 5 — `human_rework_hours`
Hours, not currency. The skill prices them at `loaded_hourly_rate`.
What counts:
- Editing or rewriting agent drafts before send.
- Correcting targeting — removing accounts the agent should not have touched, fixing segment definitions after the fact.
- CRM cleanup attributable to the agent: duplicate contacts, wrong-account activity, stage changes that had to be reversed.
- Handling replies the agent mishandled, including apology emails and internal escalations.
- Meetings that were booked and then rescheduled or unwound by a human.
What does not count: initial setup and configuration, which belongs in line 4; and ordinary AE preparation for a meeting that would have happened anyway.
Instrument this from day one. A weekly number self-reported by the two or three people doing the work is imprecise and honest. A number reconstructed at the end of the pilot by the person advocating for the tool is precise and useless.
## Worked example
A 42-day pilot on a published entry plan, 14 qualified meetings after exclusions and the qualification gate:
| Line | Value |
|---|---|
| `subscription` (annual 45,000, prorated 42/365) | 5,178 |
| `data_and_enrichment` (bundled) | 0 |
| `sending_infrastructure` (bundled) | 0 |
| `implementation_amortized` (6,000 over 12 months, 1.4 months elapsed) | 700 |
| `human_rework_hours` (71 hours at 85) | 6,035 |
| **Total** | **11,913** |
Cost per qualified meeting: 11,913 / 14 = **851**.
Subscription-only figure: 5,178 / 14 = **370**.
Both go on the memo. The 370 is the number in the vendor's business case and it is not wrong, it is answering a different question. The gap between the two is 481 per meeting, and 6,035 of the 6,735 that separates them is one line: the hours a human spent making the output usable. If that line is the reason the pilot fails its threshold, the memo should say so plainly, because "the tool is too expensive" and "the tool needs six hours of human work per meeting it produces" lead to different next steps — the first is a negotiation, the second is a product problem that a discount does not fix.
# 3 — Metric definitions
Version `1.0.0`. The skill stamps this version into `run_metadata.definitions_version`. Change the version when you change a definition, so two memos with different arithmetic are never silently compared.
## Axis 1 — Cost per qualified meeting
```
cost_per_qualified_meeting = total_cost / qualified_meetings
```
`total_cost` is the sum of the five lines in `2-cost-model.md`, with `human_rework_hours × loaded_hourly_rate` substituted for line 5.
`qualified_meetings` is the count surviving both the exclusion rules and the qualification gate. Booked-but-not-held meetings are excluded from the denominator and reported separately as `held_rate`.
Also emit `subscription_only_figure = subscription / qualified_meetings`. This is the vendor-facing number. It is not a competing estimate — it is the same arithmetic on a narrower cost basis, and printing both is how the memo makes the basis visible instead of arguable.
## Axis 2 — Reply-to-meeting rate
```
reply_to_meeting_rate = qualified_meetings / positive_replies
```
Not meetings per reply. `positive_replies` counts only `polarity: positive`. A `negative`, `out_of_office`, or `unsubscribe_request` reply is not a conversion opportunity, and pooling them into one denominator is how a reply-rate line stays flat while the list is being burned. Report all four polarity counts on the memo, and report `unsubscribe_request` prominently — a rising unsubscribe count against a flat positive count is the leading indicator for axis 4.
### Classifier discount
When `labeled_by: vendor_classifier` and `reply_label_audit` is supplied:
```
adjusted_positive_replies = vendor_positive_count × classifier_precision
```
where `classifier_precision` is the fraction of a human-labeled stratified sample that the classifier called `positive` and a human agreed was positive. Sample at least 50 replies, stratified across the window so a mid-pilot prompt change does not sit entirely inside or outside the sample.
When no audit is supplied, apply `default_classifier_discount = 0.80` and mark the axis `unaudited` on the memo. This default is a convention this file sets so that runs are comparable to each other, not a measured industry figure — it is deliberately conservative because the classifier is a component of the system under evaluation and the direction of its error is not neutral. Any deployment that runs this skill more than once should replace it with its own audited precision and note the change here.
## Axis 3 — Rework hours per qualified meeting
```
rework_hours_per_qualified_meeting = human_rework_hours / qualified_meetings
```
Trend matters as much as level. Compute it per week across the window and report the direction. Rework hours falling week over week means the team is learning the tool, and a level above threshold with a falling trend is the canonical `extend` case. Rework hours flat or rising means the tool is not converging, and the same level with a rising trend is a `kill`.
## Axis 4 — Reputation delta
Three windows, same domains throughout:
- `baseline_window` — the 30 days ending the day before `window_start`.
- `pilot_window` — `window_start` to `window_end`.
- `tail_window` — `window_end + 1` to `window_end + tail_days`, default 14 days.
Two independent readings:
**Trend.** `pilot_spam_rate - baseline_spam_rate` and `tail_spam_rate - baseline_spam_rate`. A baseline near zero and a tail at 0.0018 has not crossed any line and has still moved by more than a factor of two, which is a finding.
**Absolute line.** Whether any window crossed `thresholds.max_spam_rate`. For a Gmail-heavy recipient mix at more than 5,000 messages per day, Google's published requirement is that senders keep the spam rate reported in Postmaster Tools under 0.30%, with under 0.10% recommended so that an ordinary spike does not reach 0.30%. The template therefore sets 0.0030 as the hard stop and suggests 0.0010 as the working threshold for Gmail-heavy lists.
The tail window is mandatory because complaint-driven filtering degrades on a lag. A pilot scored on its last send date is scored before its own consequence arrives, which is why the default end-of-pilot run is 14 days after the last send rather than on the end date.
**This axis is a hard stop.** Crossing the absolute line in any window produces `kill` regardless of every other axis. The reason is that the cost is not on the pilot's ledger: sender reputation is shared across everything the company sends from those domains, including invoices, renewals, and support replies, and a pilot cannot spend it and call the saving a result.
## Denominators, stated once
- Contacted accounts: distinct accounts that received at least one agent-sent message inside the window.
- Replies: distinct accounts that replied at least once. A second reply from the same account does not add to the count.
- Meetings: distinct scheduled events. A reschedule is the same meeting; a second meeting on the same account inside 30 days is the same meeting.
- Qualified meetings: meetings surviving exclusion rules and the qualification gate.
Every rate on the memo names its denominator. A rate without a stated denominator is the format in which most pilot results are reported and the reason most of them cannot be compared to anything.
## Counterfactual
Optional and worth the effort. Cost per qualified meeting answers "what did this cost", not "what did this add". With a `counterfactual` object from a matched human-SDR segment or the same segment's prior period:
```
incremental_meetings = qualified_meetings - counterfactual_meetings_held_normalized
```
normalized to the same account count and window length. Where it is absent, the memo states explicitly that the figure is total, not incremental, and that a `keep` verdict therefore rests on the assumption that the meetings would not have happened otherwise. Say it plainly rather than letting the reader assume incrementality — that assumption is the one most often wrong and least often written down.
# 4 — Sample memo and field contract
The `memo_markdown` field is the deliverable a human reads. Structure is fixed: decision first, the axis that drove it second, the arithmetic third, what would change the answer last. Do not lead with methodology — a reader who trusts the method will not read it, and a reader who does not will read it before the number either way.
Both memos below are literal output. Copy the structure, not the values.
---
## Sample A — `kill`
```markdown
# AI SDR pilot: kill
**Decision: do not renew.** Scored 2026-06-29 against criteria registered 2026-04-28, before the first send. Criteria hash verified; no amendments.
## What drove it
The sending domain's spam complaint rate reached 0.34% in the 14 days after the last pilot send, crossing the 0.30% bulk-sender line. Baseline before the pilot was 0.08%. This is a hard stop independent of the economics: the cost of crossing it is borne by every mailbox the company sends from, including billing and support, and it is not on this pilot's ledger.
Two economic axes were also outside threshold. Fully-loaded cost per qualified meeting came in at 851 against a registered ceiling of 750, and rework ran 5.9 hours per qualified meeting against a ceiling of 3.0, with a worsening trend across the six weeks.
## The arithmetic
42-day window, 14-day tail. 34 meetings attributed by the vendor; 11 excluded (7 on accounts with an open opportunity at booking, 4 with human contact inside the prior 90 days); 19 held; 12 qualified after the 30-day stage-advance gate.
| Line | Value |
|---|---|
| Subscription (annual, prorated) | 5,178 |
| Data and enrichment (bundled) | 0 |
| Sending infrastructure (bundled) | 0 |
| Implementation (amortized over 12 months) | 700 |
| Rework (71 hours at 85) | 6,035 |
| **Total** | **11,913** |
Cost per qualified meeting: **851**. The subscription-only figure is **370** — that is the number in the vendor's business case, and the 481 gap is almost entirely the rework line.
Reply-to-meeting was inside threshold at 6.1% against a floor of 5.0%, and improving. Positive replies were labeled by the vendor's own classifier and discounted by the default factor, unaudited.
## What would change the answer
Nothing available inside this contract term. The reputation reading is a hard stop and would need a full baseline recovery plus a re-run at lower volume on isolated domains before the economics are worth re-testing. If that re-test is worth running, it needs its own pre-registration, separate sending domains, and a volume cap set below the level at which the complaint rate moved.
Note also that 32% of vendor-attributed meetings were on accounts already in cycle or recently human-touched. That ratio is a sourcing finding on its own and would not improve with a discount.
```
---
## Sample B — `extend`
```markdown
# AI SDR pilot: extend, 4 weeks
**Decision: extend for four weeks at current volume, then re-score.** Scored 2026-06-29 against criteria registered 2026-04-28. Criteria hash verified.
## What drove it
One axis outside threshold: rework at 3.8 hours per qualified meeting against a ceiling of 3.0. It is falling — 5.9 in weeks one and two, 3.4 in weeks five and six — which is the pattern of a team learning the tool rather than a tool that does not converge. Reputation is well inside its line, with the tail window at 0.09% against a 0.08% baseline.
## The arithmetic
Cost per qualified meeting: **604** against a ceiling of 750, on 18 qualified meetings from 41 attributed and 9 excluded. Reply-to-meeting 7.2% against a 5.0% floor, on an audited classifier precision of 0.86 from a 60-reply stratified sample.
## What would change the answer
Rework at or below 3.0 hours per qualified meeting across the four extension weeks, with cost per qualified meeting holding under 750 and the tail spam rate staying under 0.10%. If rework flattens above 3.0 rather than continuing to fall, this becomes a kill — the extension is testing the trend, not waiting for a better mood.
Volume is held flat deliberately. This memo scores the configuration at 2,000 contacts per month. Any increase invalidates the reputation reading and requires re-scoring at the new volume.
```
---
## Field contract for parsers
| Field | Type | Notes |
|---|---|---|
| `verdict` | `keep` / `extend` / `kill` / `null` | `null` only when `checkpoint: true` |
| `result` | string | `ok`, `no_preregistration`, `criteria_amended`, `insufficient_sample`, `incomplete_cost_model` |
| `retrospective` | boolean | `true` when `registered_at` does not precede `window_start` |
| `axes[].status` | `inside` / `outside` / `hard_stop_crossed` | `hard_stop_crossed` only on `reputation_delta` |
| `axes[].trend` | `improving` / `flat` / `worsening` | Absent on `reputation_delta`, which reports windows instead |
| `path_to_keep` | object | Present only on `extend` |
| `memo_markdown` | string | Always present when `result` is `ok` |
A non-`ok` `result` carries a populated diagnostic field and a `null` verdict. Never render a memo for a non-`ok` result — surface the diagnostic and the specific field that caused it, because "the tool would not score this" is a legible outcome and a fabricated verdict is not.
