Ein Claude Skill, der einen abgeschlossenen AI SDR Pilot in ein Keep-, Kill- oder Extend-Memo verwandelt, bewertet gegen Kriterien, die das Team vor dem ersten Versand aufgeschrieben hat. Er berechnet vier Achsen — vollständig belastete Kosten pro qualifiziertem Meeting, Meetings pro positiver Antwort, menschliche Nacharbeitsstunden und Delta der Versandreputation — und verweigert ein Urteil, wenn die Kriterien mitten im Pilot geändert wurden oder die Stichprobe zu klein ist, um eines zu tragen. Das Bundle unter apps/web/public/artifacts/ai-sdr-pilot-scorecard-skill/ liefert SKILL.md sowie vier Referenzdateien: das Vorregistrierungs-Template, das das Team vor Pilotstart ausfüllt, das Kostenmodell mit fünf Zeilen, die Metrikdefinitionen und zwei wörtliche Beispiel-Memos.
Das Problem, das er löst, ist nicht die Messung. Es ist, dass fast niemand die Abbruchlinie zuerst aufschreibt. Ein Pilot endet, das Dashboard des Anbieters zeigt gebuchte Meetings, jemand teilt die Lizenzgebühr durch diese Zahl, und das Ergebnis übertrifft eine Schwelle, die niemand vorab registriert hat. Drei Monate später verlängert sich der Vertrag automatisch, weil niemand ein Dokument erstellt hat, das beschreibt, wie Scheitern ausgesehen hätte.
Wann Sie ihn einsetzen
Führen Sie ihn 14 Tage nach dem letzten Pilotversand mit dem vollständigen Export aus. Der 14-Tage-Nachlauf ist Absicht: In der letzten Woche gebuchte Meetings haben noch nicht stattgefunden, und beschwerdegetriebene Filterung verschlechtert sich zeitversetzt. Ein Pilot, der an seinem Enddatum bewertet wird, wird also bewertet, bevor seine eigenen Folgen eintreffen.
Zwei weitere Modi. Ein Checkpoint zur Halbzeit mit checkpoint: true berechnet die Achsen und liefert no_verdict — dieser Lauf existiert, um in Woche drei einen Reputationsbruch zu erkennen, nicht um etwas zu entscheiden. Und ein Renewal-Review auf einem bereits laufenden Deployment behandelt die letzten 90 Tage als Fenster. Das ist der einzige Modus, in dem eine nachträglich geschriebene Vorregistrierung akzeptabel ist, und das Memo trägt retrospective: true dauerhaft.
Der Skill passt gleichermaßen zu 11x, Artisan, aisdr und selbst gebauten Agenten, weil er Ihre Systeme bewertet und nicht die des Anbieters. Alles, was er liest, kommt aus Ihrem CRM, Ihrem Finanzexport, Ihren Zeiterfassungen und Google Postmaster Tools.
Wann Sie ihn NICHT einsetzen
Es gibt keine Vorregistrierung. Der Skill gibt no_preregistration zurück und stoppt. Die Kriterien jetzt zu schreiben und rückzudatieren erzeugt ein schlechteres Artefakt als gar kein Memo — es tarnt ein nachträgliches Urteil als Messung. Der Renewal-Review-Modus oben ist die einzige Ausnahme, und er kennzeichnet sich selbst.
Unterhalb der registrierten Mindeststichprobe. Liefert insufficient_sample mit beobachteter und geforderter Anzahl. Bei n qualifizierten Meetings verschiebt das Hinzufügen oder Entfernen eines einzelnen die Kosten pro Meeting um etwa 1/n — 12,5% bei acht Meetings, 5% bei zwanzig. Ein sechswöchiger Pilot mit neun stattgefundenen Meetings kann eine Antwort-zu-Meeting-Rate von 4% nicht von einer von 7% unterscheiden, und ein darauf berechnetes Urteil ist Rauschen mit Nachkommastelle.
Auswahl zwischen zwei Anbietern. Dies bewertet ein Deployment gegen eine Schwelle, nicht zwei gegeneinander. Ein direkter Vergleich braucht beide auf vergleichbaren Segmenten mit gemeinsamer Ausschlussliste, was ein anderes Design ist.
Das Quartalsreview eines menschlichen SDR-Teams. Das Kostenmodell setzt einen Zähler pro Lead oder pro Kontakt voraus sowie eine Nacharbeitszeile, die es für Headcount so nicht gibt.
Setup
Zwei bis vier Stunden, überwiegend Schwellenentscheidungen statt Verkabelung.
Füllen Sie die Vorregistrierung vor dem ersten Versand aus. Kopieren Sie references/1-preregistration-template.md, ersetzen Sie jeden Platzhalter, committen Sie. Setzen Sie min_held_meetings so, dass ein Meeting mehr oder weniger die Kosten pro Meeting um weniger verändert als die Marge zwischen Ihrer Schwelle und Ihrem Erwartungswert — wenn Sie nahe 700 gegen eine Obergrenze von 750 erwarten, beträgt diese Marge rund 7% und zwanzig Meetings sind der Boden.
Setzen Sie max_cost_per_qualified_meeting gegen Ihre heutige Motion, nicht gegen den Listenpreis des Anbieters. Veröffentlichte Preise geben nur die Abo-Zeile. AiSDR veröffentlicht 250 pro Monat für 200 AI-recherchierte Kontakte, 900 für 800 und 2.500 für 2.500, wobei die beiden größeren Tiers eine Quartalsbindung haben. 11x veröffentlicht 3.750 pro Monat bei jährlicher Abrechnung für Growth — 45.000 im Jahr — für 2.000 neue Prospects pro Monat und bis zu fünf Endnutzer, und erklärt, pro Lead statt pro Versand abzurechnen. Artisan veröffentlicht keinen Preis und beschreibt seine Tiers mit rund 2.500 und rund 6.000 kontaktierten Leads pro Monat.
Setzen Sie max_spam_rate auf 0,0030 und lassen Sie es dort. Das ist die veröffentlichte Anforderung von Google für Absender von mehr als 5.000 Nachrichten pro Tag an Gmail-Konten: die in Postmaster Tools gemeldete Spam-Rate unter 0,30% halten. Google empfiehlt zusätzlich, unter 0,10% zu bleiben, damit ein gewöhnlicher Beschwerdeausschlag die harte Linie nicht erreicht. Bei einer Gmail-lastigen Liste setzen Sie die Arbeitsschwelle auf 0,0010 und behandeln 0,0030 als Stopp.
Setzen Sie die Ausschlussregeln jetzt.exclude_open_opp_at_booking und exclude_prior_human_touch_within_days: 90 entscheiden, was als Meeting des Agenten zählt. Lockern Sie sie hier, schriftlich, bevor Sie wissen, wie es ausgeht.
Erfassen Sie Nacharbeitsstunden ab Tag eins. Eine wöchentliche Selbstmeldung der zwei Personen, die die Arbeit machen, ist ungenau und ehrlich. Eine am Ende rekonstruierte Zahl von der Person, die das Tool befürwortet, ist genau und nutzlos.
Installieren Sie den Skill. Legen Sie SKILL.md und references/ in .claude/skills/ai-sdr-pilot-scorecard/ ab oder laden Sie ihn als Skill in claude.ai hoch. Bauen Sie das Eingabe-Payload aus CRM-Export, Finanzen, Zeiterfassung und Postmaster.
Warum die vier Achsen so gebaut sind
Kosten pro qualifiziertem Meeting werden zweimal ausgegeben. Einmal über alle fünf Kostenzeilen — Abo, Daten und Anreicherung, Versandinfrastruktur, amortisierte Implementierung und Nacharbeitsstunden zum belasteten Stundensatz — und einmal nur über das Abo. Beide stehen im Memo. Die Nur-Abo-Zahl ist die Zahl aus dem Business Case des Anbieters. Sie ist nicht falsch, sie beantwortet eine andere Frage, und beide nebeneinander zu drucken macht die Basis sichtbar statt strittig. Im ausgearbeiteten Beispiel in references/2-cost-model.md ergibt ein 42-Tage-Pilot mit 14 qualifizierten Meetings 851 pro Meeting belastet und 370 nur auf Abo-Basis, und 6.035 der 6.735 Differenz sind eine einzige Zeile: 71 Stunden menschliche Nacharbeit. Dieser Unterschied ändert den nächsten Schritt. “Zu teuer” ist eine Verhandlung; “sechs Stunden menschliche Arbeit pro erzeugtem Meeting” ist ein Produktproblem, das ein Rabatt nicht löst.
Antwort-zu-Meeting zählt nur positive Antworten. Eine negative Antwort, eine Abwesenheitsnotiz und eine Abmeldeanfrage sind keine Konversionschancen, und sie in einen Nenner zu werfen ist der Grund, warum eine Antwortratenkurve flach bleibt, während die Liste verbrennt. Wo der Klassifikator des Anbieters selbst die Antworten gelabelt hat, wendet references/3-metric-definitions.md einen Abschlag von 0,80 an und markiert die Zahl als unaudited — eine Konvention, die diese Datei setzt, damit Läufe vergleichbar bleiben, keine gemessene Branchenzahl. Liefern Sie eine geschichtete Stichprobe von 50 menschlich gelabelten Antworten, und der Skill berechnet die tatsächliche Precision.
Nacharbeit wird nach Trend und nicht nur nach Niveau bewertet. Wöchentlich fallend heißt, das Team lernt das Tool; flach oder steigend heißt, das Tool konvergiert nicht. Gleiches Niveau, entgegengesetzte Urteile.
Reputation ist ein harter Stopp, unabhängig von der Ökonomie bewertet. Überschreitet das Nachlauffenster die absolute Linie, lautet das Urteil kill, egal was die Kosten pro Meeting sagen. Absenderreputation wird von allem geteilt, was das Unternehmen über diese Domains versendet — Rechnungen, Verlängerungen, Support-Antworten — ein Pilot kann sie also nicht ausgeben und die Ersparnis als Ergebnis verbuchen.
Fehlermodi und Guards
Die Abbruchlinie verschiebt sich mitten im Pilot. Das ist die häufigste Art, wie ein scheiternder Pilot überlebt, und sie ist selten unehrlich — es ist eine Schwelle, die jemand in einem Doc leise aufweicht, der bereits angefangen hat zu glauben. Guard: Schritt 1 hasht die Vorregistrierung und verweigert die Bewertung gegen eine geänderte Fassung, mit einem Diff auf Feldebene statt eines Urteils. Ändern ist erlaubt; registrieren Sie mit neuem Hash neu, und das Memo trägt dauerhaft retrospective: true.
Attributionsinflation. Anbieter zählen Meetings auf Accounts, die bereits im Zyklus waren oder kürzlich menschlich berührt wurden. Guard: Ausgeschlossene Meetings verlassen den Zähler per Regel statt per verhandeltem Abschlag, und die Ausschlussanzahl samt Gründeverteilung wird als eigener Befund berichtet. Ein Anbieter, dessen zugeschriebene Meetings zu 32% bereits offene Accounts sind, sagt Ihnen etwas über sein Sourcing, das kein Rabatt behebt.
Der Nur-Abo-Nenner.Guard: Der Lauf wird mit incomplete_cost_model abgelehnt, wenn eine der fünf Zeilen null ist. Eine Zeile, die Sie für null halten, muss als 0 eingetragen werden — das ist eine Aussage, keine Lücke.
Schaden, der nach dem Bewertungsstichtag eintrifft.Guard: Das 14-Tage-Nachlauffenster ist Pflicht, und die Reputationsachse kann einen Piloten killen, der jede Kostenschwelle bestanden hat.
keep gelesen als “skalieren”. Der Skill bewertet eine Konfiguration bei einem Volumen. Volumen ist der Input, der die Reputationsachse am ehesten bricht. Guard: Der Schlussabschnitt des Memos nennt das bewertete Volumen und macht eine Neubewertung zur Bedingung jeder Erhöhung.
Gegen die Alternativen
Das QBR-Deck des Anbieters berichtet den Zähler. Es hat keinen Zugriff auf Ihre Nacharbeitsstunden, keinen Anreiz, Ihre Ausschlussregeln anzuwenden, und labelt seine eigenen positiven Antworten. Lesen Sie es, und führen Sie dann dies aus.
Die Tabelle eines RevOps-Analysten rechnet richtig und verliert trotzdem gegen Kriteriendrift, weil die Tabelle gebaut wird, nachdem die Ergebnisse sichtbar sind, und die Schwellen daneben gewählt werden. Der Hash ist der einzige Teil dieses Workflows, den eine Tabelle strukturell nicht nachbilden kann.
Den Vertrag automatisch verlängern lassen ist der tatsächliche Default, und genau den soll dies verdrängen. Bei einer 11x-Growth-Bindung kostet dieser Default 45.000 für ein Jahr, das niemand zu kaufen entschieden hat.
---
name: ai-sdr-pilot-scorecard
description: Scores a finished or in-flight AI SDR / GTM-agent pilot against criteria registered before the pilot started, and emits a keep / kill / extend memo. Computes fully-loaded cost per qualified meeting, reply-to-meeting conversion split by reply polarity, human rework hours, and sending-reputation delta. Refuses to score against amended criteria or below the registered minimum sample. Use at the end of a time-boxed pilot, before the contract auto-renews.
---
# AI SDR pilot scorecard
## When to invoke
Invoke when a time-boxed AI SDR or GTM-agent pilot reaches its end date, or at a mid-pilot checkpoint the pre-registration named. Production patterns:
- **End-of-pilot decision.** Run once, 14 days after the last send, with the full export. The 14-day tail exists because meetings booked in the final week have not yet held, and reputation damage lands after the sending stops.
- **Mid-pilot checkpoint.** Run at the halfway mark against the same registered criteria, with `checkpoint: true`. The skill returns axis-level readings and an explicit `no_verdict` — a mid-pilot run is for catching a reputation breach early, not for deciding.
- **Renewal review on a running deployment.** Treat the trailing 90 days as the pilot window. This is the only mode where a pre-registration written after the fact is acceptable, and the memo says so on its face.
Do NOT invoke this skill for:
- **A pilot with no pre-registration.** The skill returns `no_preregistration` and stops. Scoring criteria chosen after seeing the results is the failure this skill exists to prevent — writing them now and backdating them produces a worse artifact than no memo at all. The exception is the renewal-review mode above, which stamps the memo accordingly.
- **A pilot below the registered minimum sample.** Returns `insufficient_sample` with the observed and required counts. A six-week pilot that produced 9 held meetings cannot distinguish a 4% reply-to-meeting rate from a 7% one, and a verdict computed on it is noise wearing a decimal point.
- **Choosing between two vendors.** This skill scores one deployment against a threshold, not two against each other. A head-to-head needs both running on comparable segments with a shared exclusion list, which is a different design.
- **A fully human SDR team's quarterly review.** The cost model assumes a per-lead or per-contact meter and a human-rework line that does not exist in the same shape for a headcount-based team.
## Inputs
Required:
- `preregistration` — object. The parsed contents of the pre-registration file written before the pilot started. Required shape: `{ registered_at, window_start, window_end, min_held_meetings, thresholds: { max_cost_per_qualified_meeting, min_reply_to_meeting_rate, max_rework_hours_per_meeting, max_spam_rate }, exclusion_rules, qualification_gate }`.
- `preregistration_sha256` — string. SHA-256 of the pre-registration file as committed. The skill recomputes it and refuses to score on a mismatch.
- `meetings` — array. One entry per meeting the vendor or your CRM attributes to the pilot: `{ account_id, booked_at, held, held_at, stage_at_plus_30d, prior_human_touch_at, open_opp_at_booking, source_claimed_by }`.
- `costs` — object. All five cost lines, in the currency of the contract: `{ subscription, data_and_enrichment, sending_infrastructure, implementation_amortized, human_rework_hours }`. A `null` in any line is a hard rejection, not a zero.
- `loaded_hourly_rate` — number. Fully-loaded cost of an hour of the human doing the rework. Used to price `human_rework_hours`.
- `replies` — array. `{ account_id, replied_at, polarity, labeled_by }` where `polarity` is `positive` / `negative` / `out_of_office` / `unsubscribe_request` and `labeled_by` is `vendor_classifier` or `human`.
- `reputation` — object. `{ baseline_window, pilot_window, tail_window }`, each `{ spam_rate, domain, daily_volume_peak }` read from Google Postmaster Tools or the equivalent for your recipient mix.
Optional:
- `checkpoint` — boolean. Default `false`. When `true`, the skill computes the axes and returns `no_verdict`.
- `reply_label_audit` — array. A stratified human-labeled sample of replies, used to compute the vendor classifier's precision on `positive`. Without it the skill discounts vendor-labeled positives by the documented default and says so in the memo.
- `counterfactual` — object. `{ meetings_held, window }` from a matched human-SDR segment or the same segment's prior period. Turns the memo's incremental-lift section from a caveat into a number.
## Reference files
Load these from `references/` before first run. All four are stable within a deployment.
- `references/1-preregistration-template.md` — the file the team fills in **before** the pilot starts. Contains the four thresholds with the reasoning for each default, the exclusion-rule scaffolding, and the qualification-gate definition.
- `references/2-cost-model.md` — the five cost lines, what belongs in each, and the amortization rule for implementation fees. Contains a worked example against a published vendor price.
- `references/3-metric-definitions.md` — exact definitions and the arithmetic for each axis, including the reply-polarity split and the reputation window construction.
- `references/4-sample-memo.md` — a literal `kill` memo and a literal `keep` memo, plus the structured-field contract for parsers.
## Method
Run these steps in order. Steps 1 and 2 gate everything after them.
### 1. Pre-registration integrity check
Recompute SHA-256 over the `preregistration` object as serialized in the template's canonical form and compare to `preregistration_sha256`. On mismatch, return `result: criteria_amended` with a diff of the changed threshold fields — do not score. On a missing pre-registration, return `result: no_preregistration`.
The hash is the whole point of the step. Criteria drift is not usually dishonest; it is a threshold quietly softened in a doc three weeks into a pilot that is not going well, by someone who has already started to believe. A hash makes the softening a visible event rather than an invisible one.
Also verify `registered_at` precedes `window_start`. A pre-registration written after the first send is not a pre-registration; flag it as `retrospective: true` and carry that flag onto the memo.
### 2. Sample-adequacy gate
Count meetings that pass both the exclusion rules and the qualification gate. If the count is below `min_held_meetings`, return `result: insufficient_sample` with observed and required counts, and a note on how many more weeks at the observed rate would reach it. Do not return a verdict.
This gate is upstream of the economics for a reason. Cost per qualified meeting on a denominator of 7 moves by more than 14% every time one meeting is added or removed, which is wider than the gap most kill thresholds are set at.
### 3. Attribution filtering
Apply the exclusion rules from the pre-registration to every entry in `meetings`. The defaults in the template exclude a meeting when any of these holds:
- `open_opp_at_booking` is true — the account already had an open opportunity.
- `prior_human_touch_at` falls inside the 90 days before `booked_at`.
- The account appears on a named-account list the pre-registration excluded.
Excluded meetings are removed from the numerator entirely, not discounted by a factor. A discount is a judgment call that gets negotiated; a rule is a rule. Report the excluded count and the exclusion reason distribution — the ratio of excluded to included is itself a finding, and a vendor whose attributed meetings are 40% already-open accounts is telling you something about its sourcing.
### 4. Qualification gate
Apply `qualification_gate` from the pre-registration to the surviving meetings. The template default requires both `held` is true and `stage_at_plus_30d` has advanced past the initial stage. Booked-but-no-show meetings count as zero and stay out of the numerator; they are still counted separately and reported, because a high no-show rate on a healthy booking rate is a targeting problem the cost figure would otherwise hide.
### 5. Cost assembly
Sum the five lines from `costs`, pricing `human_rework_hours` at `loaded_hourly_rate`. Reject the run with `result: incomplete_cost_model` if any line is `null`.
Divide by the qualified-meeting count from step 4 to get fully-loaded cost per qualified meeting. Report the same figure computed on subscription alone, labeled as the vendor-facing number, so the memo shows both and the gap between them is legible. That gap is the argument.
### 6. Reply-to-meeting conversion, split by polarity
Compute replies per contacted account, then meetings per reply. Split the reply count by `polarity`. A `negative` reply and an `unsubscribe_request` are not conversion opportunities and must not sit in the same denominator as a `positive` one — collapsing them is how a reply-rate chart stays flat while the underlying list burns.
Where `labeled_by` is `vendor_classifier` and `reply_label_audit` is supplied, compute the classifier's precision on `positive` and multiply the vendor-labeled positive count by it. Where the audit is absent, apply the documented discount from `references/3-metric-definitions.md` and mark the figure `unaudited` in the memo.
### 7. Reputation delta
Compare `spam_rate` across the three windows. Two independent readings:
- **Trend.** Pilot and tail against baseline, on the same domains.
- **Absolute line.** Whether any window crossed the bulk-sender complaint threshold your recipient mix is judged against. For Gmail recipients at more than 5,000 messages per day, Google's published requirement is to keep the reported spam rate under 0.30%, with under 0.10% as the recommended operating point.
The tail window is where the damage usually shows. A pilot that ends on day 42 and is scored on day 42 is scored before its own consequences arrive.
**This axis is a hard stop, evaluated independently of the economics.** If the tail window crossed the absolute line, the verdict is `kill` regardless of cost per meeting, and the memo says the pilot was cheap because it was spending an asset that is not on the pilot's ledger — the deliverability of every mailbox in the company.
### 8. Verdict assembly
Compare each axis against its registered threshold and return one verdict:
- `keep` — every axis inside threshold.
- `extend` — at most one axis outside threshold, the reputation axis inside its hard stop, and the observed trend on the failing axis improving across the pilot window. The memo names the single axis and the specific reading that would make it a `keep`.
- `kill` — the reputation hard stop was crossed, or two or more axes are outside threshold, or one axis is outside threshold with a flat or worsening trend.
- `no_verdict` — `checkpoint: true`.
The `extend` verdict is deliberately narrow. A verdict of `extend` available on any failing configuration is not a verdict, it is a way of never deciding, and it is the outcome pilots default to when nobody wrote the thresholds down.
## Output format
Literal JSON the skill emits for a `kill` verdict:
```json
{
"verdict": "kill",
"result": "ok",
"retrospective": false,
"window": { "start": "2026-05-04", "end": "2026-06-15", "tail_end": "2026-06-29" },
"sample": {
"meetings_attributed": 34,
"excluded": 11,
"exclusion_reasons": { "open_opp_at_booking": 7, "prior_human_touch_90d": 4 },
"held": 19,
"qualified": 12,
"min_required": 12
},
"axes": [
{
"axis": "cost_per_qualified_meeting",
"observed": 1042.00,
"threshold": 750.00,
"status": "outside",
"trend": "flat",
"subscription_only_figure": 433.33,
"note": "Vendor-facing figure omits 71 rework hours and the enrichment line."
},
{
"axis": "reply_to_meeting_rate",
"observed": 0.061,
"threshold": 0.050,
"status": "inside",
"trend": "improving",
"positive_replies_audited": false,
"note": "Vendor-classifier positives discounted by the default factor; no reply_label_audit supplied."
},
{
"axis": "rework_hours_per_qualified_meeting",
"observed": 5.9,
"threshold": 3.0,
"status": "outside",
"trend": "worsening"
},
{
"axis": "reputation_delta",
"observed_tail_spam_rate": 0.0034,
"absolute_line": 0.0030,
"baseline_spam_rate": 0.0008,
"status": "hard_stop_crossed",
"note": "Tail window on sending domain crossed the bulk-sender complaint line 11 days after the last pilot send."
}
],
"counterfactual": null,
"memo_markdown": "...",
"run_metadata": {
"preregistration_sha256": "…",
"model": "claude-opus-5",
"definitions_version": "1.0.0"
}
}
```
A `keep` verdict has every axis at `status: "inside"` and a populated `memo_markdown`. An `extend` verdict has exactly one axis `outside`, `reputation_delta` inside its line, and a `path_to_keep` object naming the required reading. A `no_verdict` response carries populated `axes` and a `null` verdict.
The `memo_markdown` field is the deliverable a human reads. Its structure is fixed in `references/4-sample-memo.md` — decision first, the axis that drove it second, the arithmetic third, and what would change the answer last.
## Watch-outs
- **Criteria drift mid-pilot.** The most common way a pilot survives is that its kill line moves. **Guard:** step 1 hashes the pre-registration and refuses to score against an amended one, returning a field-level diff instead of a verdict. Amending is allowed — the team re-registers with a new hash, and the memo carries `retrospective: true` forever after.
- **Attribution inflation.** Vendors count meetings on accounts that were already in cycle or already human-touched. **Guard:** step 3 removes them from the numerator by rule rather than discounting them by negotiation, and reports the excluded count and reason distribution as a finding in its own right.
- **The subscription-only denominator.** Cost per meeting computed on the license price alone reliably understates the real figure, because the rework hours and the enrichment meter are the lines nobody put on the pilot budget. **Guard:** step 5 rejects the run when any of the five cost lines is `null`, and the memo prints the vendor-facing figure next to the loaded one so the gap is visible rather than arguable.
- **Reputation damage arriving after the scoring date.** Complaint-driven filtering degrades on a lag, so a pilot scored on its end date is scored before its cost lands. **Guard:** the tail window is mandatory, and the reputation axis is a hard stop evaluated independently of the economics — it can kill a pilot that cleared every cost threshold.
- **Vendor-classified positive replies.** "Positive reply" labeled by the system being evaluated is a self-graded exam. **Guard:** step 6 computes classifier precision against `reply_label_audit` when supplied and applies the documented discount when it is not, marking the figure `unaudited` on the memo rather than silently trusting it.
- **A verdict on a sample too small to carry one.** **Guard:** step 2 gates on the registered minimum and returns `insufficient_sample` — with the weeks-to-adequacy estimate — rather than a confident number computed on nine meetings.
- **`keep` read as "scale".** The skill scores the pilot's configuration at the pilot's volume. Send volume is the input most likely to break the reputation axis, and a `keep` at 2,000 contacts per month is not a `keep` at 10,000. **Guard:** the memo's final section states the scored volume explicitly and names re-scoring at the new volume as a condition of any increase.
# 1 — Pre-registration template
Fill this in **before the first send**. Commit it. The skill hashes it and refuses to score against a version that changed afterwards.
Replace every `<…>` placeholder. Do not delete fields you think do not apply — set them explicitly, because a missing threshold and a threshold of "we did not care" are different claims and the memo distinguishes them.
```yaml
registered_at: <YYYY-MM-DD> # must precede window_start
registered_by: <name, role>
vendor: <11x | artisan | aisdr | unify | homegrown | other>
plan_and_meter: <e.g. "Growth, 2,000 new prospects/mo, 5 end users">
window_start: <YYYY-MM-DD> # first send
window_end: <YYYY-MM-DD> # last send
tail_days: 14 # do not lower; see reputation note below
min_held_meetings: <integer> # see sizing note below
thresholds:
max_cost_per_qualified_meeting: <number> # your currency
min_reply_to_meeting_rate: <decimal> # meetings per positive reply
max_rework_hours_per_meeting: <number>
max_spam_rate: 0.0030 # absolute line; see note
qualification_gate:
require_held: true
require_stage_advance_by_days: 30
exclusion_rules:
exclude_open_opp_at_booking: true
exclude_prior_human_touch_within_days: 90
excluded_named_account_lists:
- <list name or CRM view id>
loaded_hourly_rate: <number> # fully loaded, not salary/2080
```
## Sizing `min_held_meetings`
Set it so that one meeting moving in or out changes cost per qualified meeting by less than the margin between your threshold and your expected value. The arithmetic: with `n` qualified meetings, adding or removing one moves the per-meeting cost by roughly `1/n`. At `n = 8` that is 12.5%; at `n = 20` it is 5%; at `n = 40` it is 2.5%.
If your threshold is 750 and you expect to land near 700, your margin is about 7% and `n = 20` is the floor. If you expect to land near 400 against a threshold of 750, the margin is wide and `n = 12` is defensible.
If the pilot cannot plausibly produce your `n` in the window, the honest move is to lengthen the window before starting, not to score a short one. Write the required `n` down anyway — an `insufficient_sample` return at the end is a real finding, and it is the finding that the pilot was not designed to answer the question.
## The four thresholds
**`max_cost_per_qualified_meeting`.** Anchor it to what the same meeting costs you today through your existing motion, not to the vendor's list price. Published entry prices give you the floor of the subscription line only: AiSDR publishes 250 per month for 200 AI-researched contacts, 900 for 800, and 2,500 for 2,500, with the middle and top tiers on a quarterly commitment. 11x publishes 3,750 per month billed annually for its Growth plan, covering 2,000 new prospects per month and up to five end users, and states it charges per lead rather than per send. Artisan publishes no price and scopes its tiers at roughly 2,500 and roughly 6,000 leads contacted per month. Those are subscription lines, not costs per meeting — the cost model in `2-cost-model.md` adds the four other lines that turn one into the other.
**`min_reply_to_meeting_rate`.** Meetings per *positive* reply, not per reply. Set it from your own historical rate on the same segment if you have one. If you do not, set it and note that you do not — an unanchored threshold you registered is still better than a rate you interpret after the fact.
**`max_rework_hours_per_meeting`.** The hours a human spends editing drafts, correcting bad targeting, cleaning the CRM after the agent, and handling replies the agent mishandled. This is the line that is never in the business case. Instrument it from day one, even crudely: a weekly self-reported number from the two people doing it beats a precise number you reconstruct at the end.
**`max_spam_rate`.** The default of 0.0030 is Google's published bulk-sender requirement for senders of more than 5,000 messages per day to Gmail accounts, which is to keep the spam rate reported in Postmaster Tools under 0.30%. Google separately recommends staying under 0.10% so that an ordinary complaint spike does not push you over. If your recipient mix is Gmail-heavy, set the threshold at 0.0010 and treat 0.0030 as the hard stop. Do not raise this field. It is not a preference; it is the line above which a mailbox provider starts making decisions about your domain.
## The exclusion rules
These decide what counts as the agent's meeting. Set them now, because after the pilot every excluded meeting is an argument.
`exclude_open_opp_at_booking` and `exclude_prior_human_touch_within_days: 90` are the two that matter most. An agent that books a meeting on an account your AE has been working for a month did not source that meeting; it scheduled it. Both defaults are deliberately strict — if you want them looser, loosen them here, in writing, before you know which way it cuts.
`excluded_named_account_lists` exists for the top-of-house accounts you would never let an agent source into. If that list is empty, say so explicitly by leaving the key with an empty list rather than deleting it.
## Amending
You may amend. Re-register with a new `registered_at`, commit, and re-hash. The skill will score against the new file and stamp `retrospective: true` on the memo permanently. That stamp is not a punishment — it is the correct label for a number produced against criteria chosen with partial knowledge of the outcome, and a reader of the memo six months from now needs it.
# 2 — Cost model
Five lines. All five are required; the skill rejects the run with `incomplete_cost_model` if any is `null`. A line you believe is zero must be entered as `0`, which is a claim you are making, not a gap you are leaving.
## Line 1 — `subscription`
The contracted license cost for the pilot window, prorated if the contract is annual.
Vendor meters differ in kind, not just in price, and the difference changes what "more volume" costs you:
- **Per lead or per prospect.** 11x publishes a Growth plan at 3,750 per month billed annually — 45,000 for the year — covering 2,000 new prospects per month and up to five end users, and states plainly that it charges per lead, not per send. Under a per-lead meter, sending more touches to the same 2,000 prospects is free and widening the list is not.
- **Per AI-researched contact.** AiSDR publishes 250 per month for 200 contacts, 900 per month for 800, and 2,500 per month for 2,500, with the two larger tiers on a quarterly commitment and a 20% discount for annual billing. The unit price falls from 1.25 per contact at the entry tier to 1.00 at the top published one.
- **Quote-only, scoped by leads contacted.** Artisan publishes no price and describes its tiers at roughly 2,500 and roughly 6,000 leads contacted per month.
Prorating an annual commitment across a six-week pilot: `subscription = annual_total × (pilot_days / 365)`. For 11x Growth at 45,000 per year and a 42-day window, that is 45,000 × 42/365, or about 5,178. Use the prorated figure, and note in the memo that the real commitment is annual — a pilot that cost 5,178 to run may have cost 45,000 to enter, and if the contract is already signed, the kill decision is about the next term, not this one.
## Line 2 — `data_and_enrichment`
Credits consumed for enrichment, verification, and intent data during the window. Include spend on tools the agent called even when they are billed elsewhere — a Clay or ZoomInfo meter that spiked during the pilot is a pilot cost regardless of which cost centre it landed in.
Bundled-plan trap: several vendors bundle contact data into the subscription, which does not make it free, it makes it invisible. When it is bundled, enter `0` and note the bundling in the memo, because the moment you scale past the bundled allowance it becomes a variable line and your cost per meeting moves.
## Line 3 — `sending_infrastructure`
Domains, mailboxes, warmup services, and any deliverability tooling bought or expanded for the pilot. Also invisible when bundled — 11x, for instance, states that every plan bundles deliverability, mailbox warmup, and inbox rotation at no additional cost.
The reason this line exists even when it is small: it is the line that grows fastest under volume increases, and a `keep` verdict that leads to tripled volume needs the reader to know which lines are fixed and which are not.
## Line 4 — `implementation_amortized`
One-time onboarding, implementation, and integration fees, amortized across the term you would actually commit to if you keep. Not across the pilot window.
A 10,000 implementation fee charged against a six-week pilot makes the pilot look catastrophic; charged across a twelve-month term at 833 per month, it makes the decision you are actually making. If you would only ever commit to twelve months, amortize across twelve. If the fee is refundable on non-renewal, enter `0` and note it.
## Line 5 — `human_rework_hours`
Hours, not currency. The skill prices them at `loaded_hourly_rate`.
What counts:
- Editing or rewriting agent drafts before send.
- Correcting targeting — removing accounts the agent should not have touched, fixing segment definitions after the fact.
- CRM cleanup attributable to the agent: duplicate contacts, wrong-account activity, stage changes that had to be reversed.
- Handling replies the agent mishandled, including apology emails and internal escalations.
- Meetings that were booked and then rescheduled or unwound by a human.
What does not count: initial setup and configuration, which belongs in line 4; and ordinary AE preparation for a meeting that would have happened anyway.
Instrument this from day one. A weekly number self-reported by the two or three people doing the work is imprecise and honest. A number reconstructed at the end of the pilot by the person advocating for the tool is precise and useless.
## Worked example
A 42-day pilot on a published entry plan, 14 qualified meetings after exclusions and the qualification gate:
| Line | Value |
|---|---|
| `subscription` (annual 45,000, prorated 42/365) | 5,178 |
| `data_and_enrichment` (bundled) | 0 |
| `sending_infrastructure` (bundled) | 0 |
| `implementation_amortized` (6,000 over 12 months, 1.4 months elapsed) | 700 |
| `human_rework_hours` (71 hours at 85) | 6,035 |
| **Total** | **11,913** |
Cost per qualified meeting: 11,913 / 14 = **851**.
Subscription-only figure: 5,178 / 14 = **370**.
Both go on the memo. The 370 is the number in the vendor's business case and it is not wrong, it is answering a different question. The gap between the two is 481 per meeting, and 6,035 of the 6,735 that separates them is one line: the hours a human spent making the output usable. If that line is the reason the pilot fails its threshold, the memo should say so plainly, because "the tool is too expensive" and "the tool needs six hours of human work per meeting it produces" lead to different next steps — the first is a negotiation, the second is a product problem that a discount does not fix.
# 3 — Metric definitions
Version `1.0.0`. The skill stamps this version into `run_metadata.definitions_version`. Change the version when you change a definition, so two memos with different arithmetic are never silently compared.
## Axis 1 — Cost per qualified meeting
```
cost_per_qualified_meeting = total_cost / qualified_meetings
```
`total_cost` is the sum of the five lines in `2-cost-model.md`, with `human_rework_hours × loaded_hourly_rate` substituted for line 5.
`qualified_meetings` is the count surviving both the exclusion rules and the qualification gate. Booked-but-not-held meetings are excluded from the denominator and reported separately as `held_rate`.
Also emit `subscription_only_figure = subscription / qualified_meetings`. This is the vendor-facing number. It is not a competing estimate — it is the same arithmetic on a narrower cost basis, and printing both is how the memo makes the basis visible instead of arguable.
## Axis 2 — Reply-to-meeting rate
```
reply_to_meeting_rate = qualified_meetings / positive_replies
```
Not meetings per reply. `positive_replies` counts only `polarity: positive`. A `negative`, `out_of_office`, or `unsubscribe_request` reply is not a conversion opportunity, and pooling them into one denominator is how a reply-rate line stays flat while the list is being burned. Report all four polarity counts on the memo, and report `unsubscribe_request` prominently — a rising unsubscribe count against a flat positive count is the leading indicator for axis 4.
### Classifier discount
When `labeled_by: vendor_classifier` and `reply_label_audit` is supplied:
```
adjusted_positive_replies = vendor_positive_count × classifier_precision
```
where `classifier_precision` is the fraction of a human-labeled stratified sample that the classifier called `positive` and a human agreed was positive. Sample at least 50 replies, stratified across the window so a mid-pilot prompt change does not sit entirely inside or outside the sample.
When no audit is supplied, apply `default_classifier_discount = 0.80` and mark the axis `unaudited` on the memo. This default is a convention this file sets so that runs are comparable to each other, not a measured industry figure — it is deliberately conservative because the classifier is a component of the system under evaluation and the direction of its error is not neutral. Any deployment that runs this skill more than once should replace it with its own audited precision and note the change here.
## Axis 3 — Rework hours per qualified meeting
```
rework_hours_per_qualified_meeting = human_rework_hours / qualified_meetings
```
Trend matters as much as level. Compute it per week across the window and report the direction. Rework hours falling week over week means the team is learning the tool, and a level above threshold with a falling trend is the canonical `extend` case. Rework hours flat or rising means the tool is not converging, and the same level with a rising trend is a `kill`.
## Axis 4 — Reputation delta
Three windows, same domains throughout:
- `baseline_window` — the 30 days ending the day before `window_start`.
- `pilot_window` — `window_start` to `window_end`.
- `tail_window` — `window_end + 1` to `window_end + tail_days`, default 14 days.
Two independent readings:
**Trend.** `pilot_spam_rate - baseline_spam_rate` and `tail_spam_rate - baseline_spam_rate`. A baseline near zero and a tail at 0.0018 has not crossed any line and has still moved by more than a factor of two, which is a finding.
**Absolute line.** Whether any window crossed `thresholds.max_spam_rate`. For a Gmail-heavy recipient mix at more than 5,000 messages per day, Google's published requirement is that senders keep the spam rate reported in Postmaster Tools under 0.30%, with under 0.10% recommended so that an ordinary spike does not reach 0.30%. The template therefore sets 0.0030 as the hard stop and suggests 0.0010 as the working threshold for Gmail-heavy lists.
The tail window is mandatory because complaint-driven filtering degrades on a lag. A pilot scored on its last send date is scored before its own consequence arrives, which is why the default end-of-pilot run is 14 days after the last send rather than on the end date.
**This axis is a hard stop.** Crossing the absolute line in any window produces `kill` regardless of every other axis. The reason is that the cost is not on the pilot's ledger: sender reputation is shared across everything the company sends from those domains, including invoices, renewals, and support replies, and a pilot cannot spend it and call the saving a result.
## Denominators, stated once
- Contacted accounts: distinct accounts that received at least one agent-sent message inside the window.
- Replies: distinct accounts that replied at least once. A second reply from the same account does not add to the count.
- Meetings: distinct scheduled events. A reschedule is the same meeting; a second meeting on the same account inside 30 days is the same meeting.
- Qualified meetings: meetings surviving exclusion rules and the qualification gate.
Every rate on the memo names its denominator. A rate without a stated denominator is the format in which most pilot results are reported and the reason most of them cannot be compared to anything.
## Counterfactual
Optional and worth the effort. Cost per qualified meeting answers "what did this cost", not "what did this add". With a `counterfactual` object from a matched human-SDR segment or the same segment's prior period:
```
incremental_meetings = qualified_meetings - counterfactual_meetings_held_normalized
```
normalized to the same account count and window length. Where it is absent, the memo states explicitly that the figure is total, not incremental, and that a `keep` verdict therefore rests on the assumption that the meetings would not have happened otherwise. Say it plainly rather than letting the reader assume incrementality — that assumption is the one most often wrong and least often written down.
# 4 — Sample memo and field contract
The `memo_markdown` field is the deliverable a human reads. Structure is fixed: decision first, the axis that drove it second, the arithmetic third, what would change the answer last. Do not lead with methodology — a reader who trusts the method will not read it, and a reader who does not will read it before the number either way.
Both memos below are literal output. Copy the structure, not the values.
---
## Sample A — `kill`
```markdown
# AI SDR pilot: kill
**Decision: do not renew.** Scored 2026-06-29 against criteria registered 2026-04-28, before the first send. Criteria hash verified; no amendments.
## What drove it
The sending domain's spam complaint rate reached 0.34% in the 14 days after the last pilot send, crossing the 0.30% bulk-sender line. Baseline before the pilot was 0.08%. This is a hard stop independent of the economics: the cost of crossing it is borne by every mailbox the company sends from, including billing and support, and it is not on this pilot's ledger.
Two economic axes were also outside threshold. Fully-loaded cost per qualified meeting came in at 851 against a registered ceiling of 750, and rework ran 5.9 hours per qualified meeting against a ceiling of 3.0, with a worsening trend across the six weeks.
## The arithmetic
42-day window, 14-day tail. 34 meetings attributed by the vendor; 11 excluded (7 on accounts with an open opportunity at booking, 4 with human contact inside the prior 90 days); 19 held; 12 qualified after the 30-day stage-advance gate.
| Line | Value |
|---|---|
| Subscription (annual, prorated) | 5,178 |
| Data and enrichment (bundled) | 0 |
| Sending infrastructure (bundled) | 0 |
| Implementation (amortized over 12 months) | 700 |
| Rework (71 hours at 85) | 6,035 |
| **Total** | **11,913** |
Cost per qualified meeting: **851**. The subscription-only figure is **370** — that is the number in the vendor's business case, and the 481 gap is almost entirely the rework line.
Reply-to-meeting was inside threshold at 6.1% against a floor of 5.0%, and improving. Positive replies were labeled by the vendor's own classifier and discounted by the default factor, unaudited.
## What would change the answer
Nothing available inside this contract term. The reputation reading is a hard stop and would need a full baseline recovery plus a re-run at lower volume on isolated domains before the economics are worth re-testing. If that re-test is worth running, it needs its own pre-registration, separate sending domains, and a volume cap set below the level at which the complaint rate moved.
Note also that 32% of vendor-attributed meetings were on accounts already in cycle or recently human-touched. That ratio is a sourcing finding on its own and would not improve with a discount.
```
---
## Sample B — `extend`
```markdown
# AI SDR pilot: extend, 4 weeks
**Decision: extend for four weeks at current volume, then re-score.** Scored 2026-06-29 against criteria registered 2026-04-28. Criteria hash verified.
## What drove it
One axis outside threshold: rework at 3.8 hours per qualified meeting against a ceiling of 3.0. It is falling — 5.9 in weeks one and two, 3.4 in weeks five and six — which is the pattern of a team learning the tool rather than a tool that does not converge. Reputation is well inside its line, with the tail window at 0.09% against a 0.08% baseline.
## The arithmetic
Cost per qualified meeting: **604** against a ceiling of 750, on 18 qualified meetings from 41 attributed and 9 excluded. Reply-to-meeting 7.2% against a 5.0% floor, on an audited classifier precision of 0.86 from a 60-reply stratified sample.
## What would change the answer
Rework at or below 3.0 hours per qualified meeting across the four extension weeks, with cost per qualified meeting holding under 750 and the tail spam rate staying under 0.10%. If rework flattens above 3.0 rather than continuing to fall, this becomes a kill — the extension is testing the trend, not waiting for a better mood.
Volume is held flat deliberately. This memo scores the configuration at 2,000 contacts per month. Any increase invalidates the reputation reading and requires re-scoring at the new volume.
```
---
## Field contract for parsers
| Field | Type | Notes |
|---|---|---|
| `verdict` | `keep` / `extend` / `kill` / `null` | `null` only when `checkpoint: true` |
| `result` | string | `ok`, `no_preregistration`, `criteria_amended`, `insufficient_sample`, `incomplete_cost_model` |
| `retrospective` | boolean | `true` when `registered_at` does not precede `window_start` |
| `axes[].status` | `inside` / `outside` / `hard_stop_crossed` | `hard_stop_crossed` only on `reputation_delta` |
| `axes[].trend` | `improving` / `flat` / `worsening` | Absent on `reputation_delta`, which reports windows instead |
| `path_to_keep` | object | Present only on `extend` |
| `memo_markdown` | string | Always present when `result` is `ok` |
A non-`ok` `result` carries a populated diagnostic field and a `null` verdict. Never render a memo for a non-`ok` result — surface the diagnostic and the specific field that caused it, because "the tool would not score this" is a legible outcome and a fabricated verdict is not.
