Um Claude Skill que transforma um piloto de AI SDR encerrado em um memo de manter, matar ou estender, avaliado contra critérios que o time escreveu antes do primeiro envio. Ele calcula quatro eixos — custo totalmente carregado por reunião qualificada, reuniões por resposta positiva, horas humanas de retrabalho e delta de reputação de envio — e se recusa a devolver um veredito quando os critérios foram alterados no meio do piloto ou quando a amostra é pequena demais para sustentá-lo. O bundle em apps/web/public/artifacts/ai-sdr-pilot-scorecard-skill/ traz SKILL.md mais quatro arquivos de referência: o template de pré-registro que o time preenche antes de o piloto começar, o modelo de custos de cinco linhas, as definições de métricas e dois memos de exemplo literais.
O problema que ele resolve não é medição. É que quase ninguém escreve a linha de corte primeiro. Um piloto termina, o dashboard do vendor mostra reuniões agendadas, alguém divide a licença por esse número, e o resultado passa por um limiar que ninguém registrou antes. Três meses depois o contrato renova automaticamente porque ninguém produziu um documento dizendo como o fracasso teria sido.
Quando usar
Rode 14 dias após o último envio do piloto, com o export completo. A cauda de 14 dias é deliberada: reuniões agendadas na semana final ainda não aconteceram, e a filtragem por reclamações se degrada com atraso, então um piloto avaliado na data de encerramento é avaliado antes de suas próprias consequências chegarem.
Outros dois modos. Um checkpoint no meio do piloto com checkpoint: true calcula os eixos e devolve no_verdict — essa rodada existe para pegar uma quebra de reputação na semana três, não para decidir nada. E uma revisão de renovação sobre um deploy já em produção trata os últimos 90 dias como a janela; é o único modo em que um pré-registro escrito depois dos fatos é aceitável, e o memo carrega retrospective: true permanentemente.
O skill serve igualmente a 11x, Artisan, aisdr e agentes próprios, porque ele avalia seus sistemas e não os do vendor. Tudo o que ele lê vem do seu CRM, do seu export de finanças, dos seus apontamentos de horas e do Google Postmaster Tools.
Quando NÃO usar
Não existe pré-registro. O skill devolve no_preregistration e para. Escrever os critérios agora e retroagir a data produz um artefato pior do que nenhum memo — transforma um julgamento posterior em algo que parece medição. O modo de revisão de renovação é a única exceção, e ele se rotula sozinho.
Abaixo da amostra mínima registrada. Devolve insufficient_sample com as contagens observada e exigida. Com n reuniões qualificadas, adicionar ou remover uma move o custo por reunião em cerca de 1/n — 12,5% com oito reuniões, 5% com vinte. Um piloto de seis semanas que produziu nove reuniões realizadas não consegue distinguir uma taxa resposta-para-reunião de 4% de uma de 7%, e um veredito calculado sobre isso é ruído com uma casa decimal.
Escolher entre dois vendors. Isto avalia um deploy contra um limiar, não dois entre si. Um confronto direto exige os dois rodando em segmentos comparáveis com uma lista de exclusão compartilhada, o que é outro desenho.
A revisão trimestral de um time humano de SDR. O modelo de custos assume um medidor por lead ou por contato e uma linha de retrabalho que não existe do mesmo jeito para headcount.
Setup
De duas a quatro horas, na maior parte decidindo limiares em vez de conectar coisas.
Preencha o pré-registro antes do primeiro envio. Copie references/1-preregistration-template.md, substitua cada placeholder, faça commit. Defina min_held_meetings de modo que mover uma reunião para dentro ou para fora mude o custo por reunião menos do que a margem entre seu limiar e seu valor esperado — se você espera cair perto de 700 contra um teto de 750, essa margem é de cerca de 7% e vinte reuniões é o piso.
Defina max_cost_per_qualified_meeting contra o seu motion atual, não contra o preço de tabela do vendor. Preços publicados só dão a linha de assinatura. A AiSDR publica 250 por mês para 200 contatos pesquisados por AI, 900 para 800 e 2.500 para 2.500, com os dois tiers maiores em compromisso trimestral. A 11x publica 3.750 por mês faturado anualmente no Growth — 45.000 no ano — cobrindo 2.000 prospects novos por mês e até cinco usuários finais, e declara que cobra por lead e não por envio. A Artisan não publica preço e descreve seus tiers em aproximadamente 2.500 e aproximadamente 6.000 leads contatados por mês.
Defina max_spam_rate em 0,0030 e deixe assim. Esse é o requisito publicado do Google para remetentes de mais de 5.000 mensagens por dia para contas Gmail: manter a taxa de spam reportada no Postmaster Tools abaixo de 0,30%. O Google recomenda separadamente ficar abaixo de 0,10% para que um pico normal de reclamações não alcance a linha dura. Numa lista com muito Gmail, defina o limiar de trabalho em 0,0010 e trate 0,0030 como o corte.
Defina as regras de exclusão agora.exclude_open_opp_at_booking e exclude_prior_human_touch_within_days: 90 são as duas que decidem o que conta como reunião do agente. Afrouxe aqui, por escrito, antes de saber para que lado isso pesa.
Instrumente as horas de retrabalho desde o primeiro dia. Um número semanal auto-reportado pelas duas pessoas que fazem o trabalho é impreciso e honesto. Um número reconstruído no fim por quem defende a ferramenta é preciso e inútil.
Instale o skill. Coloque SKILL.md e references/ em .claude/skills/ai-sdr-pilot-scorecard/, ou faça upload como Skill no claude.ai. Monte o payload de entrada a partir do export do CRM, de finanças, dos apontamentos de horas e do Postmaster.
Por que os quatro eixos são construídos assim
O custo por reunião qualificada é emitido duas vezes. Uma sobre as cinco linhas de custo — assinatura, dados e enriquecimento, infraestrutura de envio, implementação amortizada e horas de retrabalho precificadas a uma taxa carregada — e outra só sobre assinatura. As duas vão para o memo. A cifra de só-assinatura é o número do business case do vendor; ela não está errada, responde outra pergunta, e imprimir as duas torna a base visível em vez de discutível. No exemplo trabalhado em references/2-cost-model.md, um piloto de 42 dias com 14 reuniões qualificadas dá 851 por reunião carregado e 370 só sobre assinatura, e 6.035 dos 6.735 que separam os dois são uma única linha: 71 horas de retrabalho humano. Essa distinção muda o próximo passo. “Caro demais” é uma negociação; “seis horas de trabalho humano por reunião produzida” é um problema de produto que desconto nenhum resolve.
Resposta-para-reunião conta apenas respostas positivas. Uma resposta negativa, um fora-do-escritório e um pedido de descadastro não são oportunidades de conversão, e juntá-los no mesmo denominador é como um gráfico de taxa de resposta fica plano enquanto a lista queima. Quando o classificador do próprio vendor rotulou as respostas, references/3-metric-definitions.md aplica um desconto de 0,80 e marca a cifra como unaudited — uma convenção que aquele arquivo estabelece para que as rodadas continuem comparáveis, não um número medido do mercado. Forneça uma amostra estratificada de 50 respostas rotuladas por humanos e o skill calcula a precisão real.
O retrabalho é avaliado por tendência além do nível. Cair semana a semana significa que o time está aprendendo a ferramenta; plano ou subindo significa que a ferramenta não converge. Mesmo nível, veredictos opostos.
Reputação é um corte duro, avaliado independentemente da economia. Se a janela de cauda cruzar a linha absoluta, o veredito é kill independentemente do que o custo por reunião disser. A reputação de remetente é compartilhada com tudo o que a empresa envia daqueles domínios — faturas, renovações, respostas de suporte — então um piloto não pode gastá-la e lançar a economia como resultado.
Modos de falha e guardas
A linha de corte se move no meio do piloto. É a forma mais comum de um piloto que fracassa sobreviver, e raramente é desonesta — é um limiar suavizado em silêncio num doc por alguém que já começou a acreditar. Guarda: o passo 1 faz hash do pré-registro e se recusa a avaliar contra um alterado, devolvendo um diff em nível de campo em vez de um veredito. Alterar é permitido; re-registre com um hash novo e o memo carrega retrospective: true permanentemente.
Inflação de atribuição. Vendors contam reuniões em contas que já estavam em ciclo ou com contato humano recente. Guarda: reuniões excluídas saem do numerador por regra e não por desconto negociado, e a contagem de exclusões e a distribuição de motivos são reportadas como achado. Um vendor cujas reuniões atribuídas são 32% contas já abertas está te dizendo algo sobre o sourcing dele que desconto nenhum resolve.
O denominador de só-assinatura.Guarda: a rodada é rejeitada com incomplete_cost_model se qualquer uma das cinco linhas for null. Uma linha que você acredita ser zero precisa ser informada como 0, o que é uma afirmação e não uma lacuna.
O dano que chega depois da data de avaliação.Guarda: a janela de cauda de 14 dias é obrigatória e o eixo de reputação pode matar um piloto que passou por todos os limiares de custo.
keep lido como “escalar”. O skill avalia uma configuração em um volume. Volume é o input com maior chance de quebrar o eixo de reputação. Guarda: a seção final do memo nomeia o volume avaliado e torna a reavaliação uma condição de qualquer aumento.
Contra as alternativas
O deck de QBR do vendor reporta o numerador. Ele não tem acesso às suas horas de retrabalho, nenhum incentivo para aplicar suas regras de exclusão, e rotula as próprias respostas positivas. Leia, e depois rode isto.
A planilha de um analista de RevOps acerta a aritmética e ainda assim perde para o desvio de critérios, porque a planilha é montada depois que os resultados são visíveis e os limiares são escolhidos junto com eles. O hash é a única parte deste workflow que uma planilha estruturalmente não consegue replicar.
Deixar o contrato renovar sozinho é o default de verdade, e é o que isto existe para deslocar. Sobre um compromisso 11x Growth, esse default custa 45.000 por um ano que ninguém decidiu comprar.
---
name: ai-sdr-pilot-scorecard
description: Scores a finished or in-flight AI SDR / GTM-agent pilot against criteria registered before the pilot started, and emits a keep / kill / extend memo. Computes fully-loaded cost per qualified meeting, reply-to-meeting conversion split by reply polarity, human rework hours, and sending-reputation delta. Refuses to score against amended criteria or below the registered minimum sample. Use at the end of a time-boxed pilot, before the contract auto-renews.
---
# AI SDR pilot scorecard
## When to invoke
Invoke when a time-boxed AI SDR or GTM-agent pilot reaches its end date, or at a mid-pilot checkpoint the pre-registration named. Production patterns:
- **End-of-pilot decision.** Run once, 14 days after the last send, with the full export. The 14-day tail exists because meetings booked in the final week have not yet held, and reputation damage lands after the sending stops.
- **Mid-pilot checkpoint.** Run at the halfway mark against the same registered criteria, with `checkpoint: true`. The skill returns axis-level readings and an explicit `no_verdict` — a mid-pilot run is for catching a reputation breach early, not for deciding.
- **Renewal review on a running deployment.** Treat the trailing 90 days as the pilot window. This is the only mode where a pre-registration written after the fact is acceptable, and the memo says so on its face.
Do NOT invoke this skill for:
- **A pilot with no pre-registration.** The skill returns `no_preregistration` and stops. Scoring criteria chosen after seeing the results is the failure this skill exists to prevent — writing them now and backdating them produces a worse artifact than no memo at all. The exception is the renewal-review mode above, which stamps the memo accordingly.
- **A pilot below the registered minimum sample.** Returns `insufficient_sample` with the observed and required counts. A six-week pilot that produced 9 held meetings cannot distinguish a 4% reply-to-meeting rate from a 7% one, and a verdict computed on it is noise wearing a decimal point.
- **Choosing between two vendors.** This skill scores one deployment against a threshold, not two against each other. A head-to-head needs both running on comparable segments with a shared exclusion list, which is a different design.
- **A fully human SDR team's quarterly review.** The cost model assumes a per-lead or per-contact meter and a human-rework line that does not exist in the same shape for a headcount-based team.
## Inputs
Required:
- `preregistration` — object. The parsed contents of the pre-registration file written before the pilot started. Required shape: `{ registered_at, window_start, window_end, min_held_meetings, thresholds: { max_cost_per_qualified_meeting, min_reply_to_meeting_rate, max_rework_hours_per_meeting, max_spam_rate }, exclusion_rules, qualification_gate }`.
- `preregistration_sha256` — string. SHA-256 of the pre-registration file as committed. The skill recomputes it and refuses to score on a mismatch.
- `meetings` — array. One entry per meeting the vendor or your CRM attributes to the pilot: `{ account_id, booked_at, held, held_at, stage_at_plus_30d, prior_human_touch_at, open_opp_at_booking, source_claimed_by }`.
- `costs` — object. All five cost lines, in the currency of the contract: `{ subscription, data_and_enrichment, sending_infrastructure, implementation_amortized, human_rework_hours }`. A `null` in any line is a hard rejection, not a zero.
- `loaded_hourly_rate` — number. Fully-loaded cost of an hour of the human doing the rework. Used to price `human_rework_hours`.
- `replies` — array. `{ account_id, replied_at, polarity, labeled_by }` where `polarity` is `positive` / `negative` / `out_of_office` / `unsubscribe_request` and `labeled_by` is `vendor_classifier` or `human`.
- `reputation` — object. `{ baseline_window, pilot_window, tail_window }`, each `{ spam_rate, domain, daily_volume_peak }` read from Google Postmaster Tools or the equivalent for your recipient mix.
Optional:
- `checkpoint` — boolean. Default `false`. When `true`, the skill computes the axes and returns `no_verdict`.
- `reply_label_audit` — array. A stratified human-labeled sample of replies, used to compute the vendor classifier's precision on `positive`. Without it the skill discounts vendor-labeled positives by the documented default and says so in the memo.
- `counterfactual` — object. `{ meetings_held, window }` from a matched human-SDR segment or the same segment's prior period. Turns the memo's incremental-lift section from a caveat into a number.
## Reference files
Load these from `references/` before first run. All four are stable within a deployment.
- `references/1-preregistration-template.md` — the file the team fills in **before** the pilot starts. Contains the four thresholds with the reasoning for each default, the exclusion-rule scaffolding, and the qualification-gate definition.
- `references/2-cost-model.md` — the five cost lines, what belongs in each, and the amortization rule for implementation fees. Contains a worked example against a published vendor price.
- `references/3-metric-definitions.md` — exact definitions and the arithmetic for each axis, including the reply-polarity split and the reputation window construction.
- `references/4-sample-memo.md` — a literal `kill` memo and a literal `keep` memo, plus the structured-field contract for parsers.
## Method
Run these steps in order. Steps 1 and 2 gate everything after them.
### 1. Pre-registration integrity check
Recompute SHA-256 over the `preregistration` object as serialized in the template's canonical form and compare to `preregistration_sha256`. On mismatch, return `result: criteria_amended` with a diff of the changed threshold fields — do not score. On a missing pre-registration, return `result: no_preregistration`.
The hash is the whole point of the step. Criteria drift is not usually dishonest; it is a threshold quietly softened in a doc three weeks into a pilot that is not going well, by someone who has already started to believe. A hash makes the softening a visible event rather than an invisible one.
Also verify `registered_at` precedes `window_start`. A pre-registration written after the first send is not a pre-registration; flag it as `retrospective: true` and carry that flag onto the memo.
### 2. Sample-adequacy gate
Count meetings that pass both the exclusion rules and the qualification gate. If the count is below `min_held_meetings`, return `result: insufficient_sample` with observed and required counts, and a note on how many more weeks at the observed rate would reach it. Do not return a verdict.
This gate is upstream of the economics for a reason. Cost per qualified meeting on a denominator of 7 moves by more than 14% every time one meeting is added or removed, which is wider than the gap most kill thresholds are set at.
### 3. Attribution filtering
Apply the exclusion rules from the pre-registration to every entry in `meetings`. The defaults in the template exclude a meeting when any of these holds:
- `open_opp_at_booking` is true — the account already had an open opportunity.
- `prior_human_touch_at` falls inside the 90 days before `booked_at`.
- The account appears on a named-account list the pre-registration excluded.
Excluded meetings are removed from the numerator entirely, not discounted by a factor. A discount is a judgment call that gets negotiated; a rule is a rule. Report the excluded count and the exclusion reason distribution — the ratio of excluded to included is itself a finding, and a vendor whose attributed meetings are 40% already-open accounts is telling you something about its sourcing.
### 4. Qualification gate
Apply `qualification_gate` from the pre-registration to the surviving meetings. The template default requires both `held` is true and `stage_at_plus_30d` has advanced past the initial stage. Booked-but-no-show meetings count as zero and stay out of the numerator; they are still counted separately and reported, because a high no-show rate on a healthy booking rate is a targeting problem the cost figure would otherwise hide.
### 5. Cost assembly
Sum the five lines from `costs`, pricing `human_rework_hours` at `loaded_hourly_rate`. Reject the run with `result: incomplete_cost_model` if any line is `null`.
Divide by the qualified-meeting count from step 4 to get fully-loaded cost per qualified meeting. Report the same figure computed on subscription alone, labeled as the vendor-facing number, so the memo shows both and the gap between them is legible. That gap is the argument.
### 6. Reply-to-meeting conversion, split by polarity
Compute replies per contacted account, then meetings per reply. Split the reply count by `polarity`. A `negative` reply and an `unsubscribe_request` are not conversion opportunities and must not sit in the same denominator as a `positive` one — collapsing them is how a reply-rate chart stays flat while the underlying list burns.
Where `labeled_by` is `vendor_classifier` and `reply_label_audit` is supplied, compute the classifier's precision on `positive` and multiply the vendor-labeled positive count by it. Where the audit is absent, apply the documented discount from `references/3-metric-definitions.md` and mark the figure `unaudited` in the memo.
### 7. Reputation delta
Compare `spam_rate` across the three windows. Two independent readings:
- **Trend.** Pilot and tail against baseline, on the same domains.
- **Absolute line.** Whether any window crossed the bulk-sender complaint threshold your recipient mix is judged against. For Gmail recipients at more than 5,000 messages per day, Google's published requirement is to keep the reported spam rate under 0.30%, with under 0.10% as the recommended operating point.
The tail window is where the damage usually shows. A pilot that ends on day 42 and is scored on day 42 is scored before its own consequences arrive.
**This axis is a hard stop, evaluated independently of the economics.** If the tail window crossed the absolute line, the verdict is `kill` regardless of cost per meeting, and the memo says the pilot was cheap because it was spending an asset that is not on the pilot's ledger — the deliverability of every mailbox in the company.
### 8. Verdict assembly
Compare each axis against its registered threshold and return one verdict:
- `keep` — every axis inside threshold.
- `extend` — at most one axis outside threshold, the reputation axis inside its hard stop, and the observed trend on the failing axis improving across the pilot window. The memo names the single axis and the specific reading that would make it a `keep`.
- `kill` — the reputation hard stop was crossed, or two or more axes are outside threshold, or one axis is outside threshold with a flat or worsening trend.
- `no_verdict` — `checkpoint: true`.
The `extend` verdict is deliberately narrow. A verdict of `extend` available on any failing configuration is not a verdict, it is a way of never deciding, and it is the outcome pilots default to when nobody wrote the thresholds down.
## Output format
Literal JSON the skill emits for a `kill` verdict:
```json
{
"verdict": "kill",
"result": "ok",
"retrospective": false,
"window": { "start": "2026-05-04", "end": "2026-06-15", "tail_end": "2026-06-29" },
"sample": {
"meetings_attributed": 34,
"excluded": 11,
"exclusion_reasons": { "open_opp_at_booking": 7, "prior_human_touch_90d": 4 },
"held": 19,
"qualified": 12,
"min_required": 12
},
"axes": [
{
"axis": "cost_per_qualified_meeting",
"observed": 1042.00,
"threshold": 750.00,
"status": "outside",
"trend": "flat",
"subscription_only_figure": 433.33,
"note": "Vendor-facing figure omits 71 rework hours and the enrichment line."
},
{
"axis": "reply_to_meeting_rate",
"observed": 0.061,
"threshold": 0.050,
"status": "inside",
"trend": "improving",
"positive_replies_audited": false,
"note": "Vendor-classifier positives discounted by the default factor; no reply_label_audit supplied."
},
{
"axis": "rework_hours_per_qualified_meeting",
"observed": 5.9,
"threshold": 3.0,
"status": "outside",
"trend": "worsening"
},
{
"axis": "reputation_delta",
"observed_tail_spam_rate": 0.0034,
"absolute_line": 0.0030,
"baseline_spam_rate": 0.0008,
"status": "hard_stop_crossed",
"note": "Tail window on sending domain crossed the bulk-sender complaint line 11 days after the last pilot send."
}
],
"counterfactual": null,
"memo_markdown": "...",
"run_metadata": {
"preregistration_sha256": "…",
"model": "claude-opus-5",
"definitions_version": "1.0.0"
}
}
```
A `keep` verdict has every axis at `status: "inside"` and a populated `memo_markdown`. An `extend` verdict has exactly one axis `outside`, `reputation_delta` inside its line, and a `path_to_keep` object naming the required reading. A `no_verdict` response carries populated `axes` and a `null` verdict.
The `memo_markdown` field is the deliverable a human reads. Its structure is fixed in `references/4-sample-memo.md` — decision first, the axis that drove it second, the arithmetic third, and what would change the answer last.
## Watch-outs
- **Criteria drift mid-pilot.** The most common way a pilot survives is that its kill line moves. **Guard:** step 1 hashes the pre-registration and refuses to score against an amended one, returning a field-level diff instead of a verdict. Amending is allowed — the team re-registers with a new hash, and the memo carries `retrospective: true` forever after.
- **Attribution inflation.** Vendors count meetings on accounts that were already in cycle or already human-touched. **Guard:** step 3 removes them from the numerator by rule rather than discounting them by negotiation, and reports the excluded count and reason distribution as a finding in its own right.
- **The subscription-only denominator.** Cost per meeting computed on the license price alone reliably understates the real figure, because the rework hours and the enrichment meter are the lines nobody put on the pilot budget. **Guard:** step 5 rejects the run when any of the five cost lines is `null`, and the memo prints the vendor-facing figure next to the loaded one so the gap is visible rather than arguable.
- **Reputation damage arriving after the scoring date.** Complaint-driven filtering degrades on a lag, so a pilot scored on its end date is scored before its cost lands. **Guard:** the tail window is mandatory, and the reputation axis is a hard stop evaluated independently of the economics — it can kill a pilot that cleared every cost threshold.
- **Vendor-classified positive replies.** "Positive reply" labeled by the system being evaluated is a self-graded exam. **Guard:** step 6 computes classifier precision against `reply_label_audit` when supplied and applies the documented discount when it is not, marking the figure `unaudited` on the memo rather than silently trusting it.
- **A verdict on a sample too small to carry one.** **Guard:** step 2 gates on the registered minimum and returns `insufficient_sample` — with the weeks-to-adequacy estimate — rather than a confident number computed on nine meetings.
- **`keep` read as "scale".** The skill scores the pilot's configuration at the pilot's volume. Send volume is the input most likely to break the reputation axis, and a `keep` at 2,000 contacts per month is not a `keep` at 10,000. **Guard:** the memo's final section states the scored volume explicitly and names re-scoring at the new volume as a condition of any increase.
# 1 — Pre-registration template
Fill this in **before the first send**. Commit it. The skill hashes it and refuses to score against a version that changed afterwards.
Replace every `<…>` placeholder. Do not delete fields you think do not apply — set them explicitly, because a missing threshold and a threshold of "we did not care" are different claims and the memo distinguishes them.
```yaml
registered_at: <YYYY-MM-DD> # must precede window_start
registered_by: <name, role>
vendor: <11x | artisan | aisdr | unify | homegrown | other>
plan_and_meter: <e.g. "Growth, 2,000 new prospects/mo, 5 end users">
window_start: <YYYY-MM-DD> # first send
window_end: <YYYY-MM-DD> # last send
tail_days: 14 # do not lower; see reputation note below
min_held_meetings: <integer> # see sizing note below
thresholds:
max_cost_per_qualified_meeting: <number> # your currency
min_reply_to_meeting_rate: <decimal> # meetings per positive reply
max_rework_hours_per_meeting: <number>
max_spam_rate: 0.0030 # absolute line; see note
qualification_gate:
require_held: true
require_stage_advance_by_days: 30
exclusion_rules:
exclude_open_opp_at_booking: true
exclude_prior_human_touch_within_days: 90
excluded_named_account_lists:
- <list name or CRM view id>
loaded_hourly_rate: <number> # fully loaded, not salary/2080
```
## Sizing `min_held_meetings`
Set it so that one meeting moving in or out changes cost per qualified meeting by less than the margin between your threshold and your expected value. The arithmetic: with `n` qualified meetings, adding or removing one moves the per-meeting cost by roughly `1/n`. At `n = 8` that is 12.5%; at `n = 20` it is 5%; at `n = 40` it is 2.5%.
If your threshold is 750 and you expect to land near 700, your margin is about 7% and `n = 20` is the floor. If you expect to land near 400 against a threshold of 750, the margin is wide and `n = 12` is defensible.
If the pilot cannot plausibly produce your `n` in the window, the honest move is to lengthen the window before starting, not to score a short one. Write the required `n` down anyway — an `insufficient_sample` return at the end is a real finding, and it is the finding that the pilot was not designed to answer the question.
## The four thresholds
**`max_cost_per_qualified_meeting`.** Anchor it to what the same meeting costs you today through your existing motion, not to the vendor's list price. Published entry prices give you the floor of the subscription line only: AiSDR publishes 250 per month for 200 AI-researched contacts, 900 for 800, and 2,500 for 2,500, with the middle and top tiers on a quarterly commitment. 11x publishes 3,750 per month billed annually for its Growth plan, covering 2,000 new prospects per month and up to five end users, and states it charges per lead rather than per send. Artisan publishes no price and scopes its tiers at roughly 2,500 and roughly 6,000 leads contacted per month. Those are subscription lines, not costs per meeting — the cost model in `2-cost-model.md` adds the four other lines that turn one into the other.
**`min_reply_to_meeting_rate`.** Meetings per *positive* reply, not per reply. Set it from your own historical rate on the same segment if you have one. If you do not, set it and note that you do not — an unanchored threshold you registered is still better than a rate you interpret after the fact.
**`max_rework_hours_per_meeting`.** The hours a human spends editing drafts, correcting bad targeting, cleaning the CRM after the agent, and handling replies the agent mishandled. This is the line that is never in the business case. Instrument it from day one, even crudely: a weekly self-reported number from the two people doing it beats a precise number you reconstruct at the end.
**`max_spam_rate`.** The default of 0.0030 is Google's published bulk-sender requirement for senders of more than 5,000 messages per day to Gmail accounts, which is to keep the spam rate reported in Postmaster Tools under 0.30%. Google separately recommends staying under 0.10% so that an ordinary complaint spike does not push you over. If your recipient mix is Gmail-heavy, set the threshold at 0.0010 and treat 0.0030 as the hard stop. Do not raise this field. It is not a preference; it is the line above which a mailbox provider starts making decisions about your domain.
## The exclusion rules
These decide what counts as the agent's meeting. Set them now, because after the pilot every excluded meeting is an argument.
`exclude_open_opp_at_booking` and `exclude_prior_human_touch_within_days: 90` are the two that matter most. An agent that books a meeting on an account your AE has been working for a month did not source that meeting; it scheduled it. Both defaults are deliberately strict — if you want them looser, loosen them here, in writing, before you know which way it cuts.
`excluded_named_account_lists` exists for the top-of-house accounts you would never let an agent source into. If that list is empty, say so explicitly by leaving the key with an empty list rather than deleting it.
## Amending
You may amend. Re-register with a new `registered_at`, commit, and re-hash. The skill will score against the new file and stamp `retrospective: true` on the memo permanently. That stamp is not a punishment — it is the correct label for a number produced against criteria chosen with partial knowledge of the outcome, and a reader of the memo six months from now needs it.
# 2 — Cost model
Five lines. All five are required; the skill rejects the run with `incomplete_cost_model` if any is `null`. A line you believe is zero must be entered as `0`, which is a claim you are making, not a gap you are leaving.
## Line 1 — `subscription`
The contracted license cost for the pilot window, prorated if the contract is annual.
Vendor meters differ in kind, not just in price, and the difference changes what "more volume" costs you:
- **Per lead or per prospect.** 11x publishes a Growth plan at 3,750 per month billed annually — 45,000 for the year — covering 2,000 new prospects per month and up to five end users, and states plainly that it charges per lead, not per send. Under a per-lead meter, sending more touches to the same 2,000 prospects is free and widening the list is not.
- **Per AI-researched contact.** AiSDR publishes 250 per month for 200 contacts, 900 per month for 800, and 2,500 per month for 2,500, with the two larger tiers on a quarterly commitment and a 20% discount for annual billing. The unit price falls from 1.25 per contact at the entry tier to 1.00 at the top published one.
- **Quote-only, scoped by leads contacted.** Artisan publishes no price and describes its tiers at roughly 2,500 and roughly 6,000 leads contacted per month.
Prorating an annual commitment across a six-week pilot: `subscription = annual_total × (pilot_days / 365)`. For 11x Growth at 45,000 per year and a 42-day window, that is 45,000 × 42/365, or about 5,178. Use the prorated figure, and note in the memo that the real commitment is annual — a pilot that cost 5,178 to run may have cost 45,000 to enter, and if the contract is already signed, the kill decision is about the next term, not this one.
## Line 2 — `data_and_enrichment`
Credits consumed for enrichment, verification, and intent data during the window. Include spend on tools the agent called even when they are billed elsewhere — a Clay or ZoomInfo meter that spiked during the pilot is a pilot cost regardless of which cost centre it landed in.
Bundled-plan trap: several vendors bundle contact data into the subscription, which does not make it free, it makes it invisible. When it is bundled, enter `0` and note the bundling in the memo, because the moment you scale past the bundled allowance it becomes a variable line and your cost per meeting moves.
## Line 3 — `sending_infrastructure`
Domains, mailboxes, warmup services, and any deliverability tooling bought or expanded for the pilot. Also invisible when bundled — 11x, for instance, states that every plan bundles deliverability, mailbox warmup, and inbox rotation at no additional cost.
The reason this line exists even when it is small: it is the line that grows fastest under volume increases, and a `keep` verdict that leads to tripled volume needs the reader to know which lines are fixed and which are not.
## Line 4 — `implementation_amortized`
One-time onboarding, implementation, and integration fees, amortized across the term you would actually commit to if you keep. Not across the pilot window.
A 10,000 implementation fee charged against a six-week pilot makes the pilot look catastrophic; charged across a twelve-month term at 833 per month, it makes the decision you are actually making. If you would only ever commit to twelve months, amortize across twelve. If the fee is refundable on non-renewal, enter `0` and note it.
## Line 5 — `human_rework_hours`
Hours, not currency. The skill prices them at `loaded_hourly_rate`.
What counts:
- Editing or rewriting agent drafts before send.
- Correcting targeting — removing accounts the agent should not have touched, fixing segment definitions after the fact.
- CRM cleanup attributable to the agent: duplicate contacts, wrong-account activity, stage changes that had to be reversed.
- Handling replies the agent mishandled, including apology emails and internal escalations.
- Meetings that were booked and then rescheduled or unwound by a human.
What does not count: initial setup and configuration, which belongs in line 4; and ordinary AE preparation for a meeting that would have happened anyway.
Instrument this from day one. A weekly number self-reported by the two or three people doing the work is imprecise and honest. A number reconstructed at the end of the pilot by the person advocating for the tool is precise and useless.
## Worked example
A 42-day pilot on a published entry plan, 14 qualified meetings after exclusions and the qualification gate:
| Line | Value |
|---|---|
| `subscription` (annual 45,000, prorated 42/365) | 5,178 |
| `data_and_enrichment` (bundled) | 0 |
| `sending_infrastructure` (bundled) | 0 |
| `implementation_amortized` (6,000 over 12 months, 1.4 months elapsed) | 700 |
| `human_rework_hours` (71 hours at 85) | 6,035 |
| **Total** | **11,913** |
Cost per qualified meeting: 11,913 / 14 = **851**.
Subscription-only figure: 5,178 / 14 = **370**.
Both go on the memo. The 370 is the number in the vendor's business case and it is not wrong, it is answering a different question. The gap between the two is 481 per meeting, and 6,035 of the 6,735 that separates them is one line: the hours a human spent making the output usable. If that line is the reason the pilot fails its threshold, the memo should say so plainly, because "the tool is too expensive" and "the tool needs six hours of human work per meeting it produces" lead to different next steps — the first is a negotiation, the second is a product problem that a discount does not fix.
# 3 — Metric definitions
Version `1.0.0`. The skill stamps this version into `run_metadata.definitions_version`. Change the version when you change a definition, so two memos with different arithmetic are never silently compared.
## Axis 1 — Cost per qualified meeting
```
cost_per_qualified_meeting = total_cost / qualified_meetings
```
`total_cost` is the sum of the five lines in `2-cost-model.md`, with `human_rework_hours × loaded_hourly_rate` substituted for line 5.
`qualified_meetings` is the count surviving both the exclusion rules and the qualification gate. Booked-but-not-held meetings are excluded from the denominator and reported separately as `held_rate`.
Also emit `subscription_only_figure = subscription / qualified_meetings`. This is the vendor-facing number. It is not a competing estimate — it is the same arithmetic on a narrower cost basis, and printing both is how the memo makes the basis visible instead of arguable.
## Axis 2 — Reply-to-meeting rate
```
reply_to_meeting_rate = qualified_meetings / positive_replies
```
Not meetings per reply. `positive_replies` counts only `polarity: positive`. A `negative`, `out_of_office`, or `unsubscribe_request` reply is not a conversion opportunity, and pooling them into one denominator is how a reply-rate line stays flat while the list is being burned. Report all four polarity counts on the memo, and report `unsubscribe_request` prominently — a rising unsubscribe count against a flat positive count is the leading indicator for axis 4.
### Classifier discount
When `labeled_by: vendor_classifier` and `reply_label_audit` is supplied:
```
adjusted_positive_replies = vendor_positive_count × classifier_precision
```
where `classifier_precision` is the fraction of a human-labeled stratified sample that the classifier called `positive` and a human agreed was positive. Sample at least 50 replies, stratified across the window so a mid-pilot prompt change does not sit entirely inside or outside the sample.
When no audit is supplied, apply `default_classifier_discount = 0.80` and mark the axis `unaudited` on the memo. This default is a convention this file sets so that runs are comparable to each other, not a measured industry figure — it is deliberately conservative because the classifier is a component of the system under evaluation and the direction of its error is not neutral. Any deployment that runs this skill more than once should replace it with its own audited precision and note the change here.
## Axis 3 — Rework hours per qualified meeting
```
rework_hours_per_qualified_meeting = human_rework_hours / qualified_meetings
```
Trend matters as much as level. Compute it per week across the window and report the direction. Rework hours falling week over week means the team is learning the tool, and a level above threshold with a falling trend is the canonical `extend` case. Rework hours flat or rising means the tool is not converging, and the same level with a rising trend is a `kill`.
## Axis 4 — Reputation delta
Three windows, same domains throughout:
- `baseline_window` — the 30 days ending the day before `window_start`.
- `pilot_window` — `window_start` to `window_end`.
- `tail_window` — `window_end + 1` to `window_end + tail_days`, default 14 days.
Two independent readings:
**Trend.** `pilot_spam_rate - baseline_spam_rate` and `tail_spam_rate - baseline_spam_rate`. A baseline near zero and a tail at 0.0018 has not crossed any line and has still moved by more than a factor of two, which is a finding.
**Absolute line.** Whether any window crossed `thresholds.max_spam_rate`. For a Gmail-heavy recipient mix at more than 5,000 messages per day, Google's published requirement is that senders keep the spam rate reported in Postmaster Tools under 0.30%, with under 0.10% recommended so that an ordinary spike does not reach 0.30%. The template therefore sets 0.0030 as the hard stop and suggests 0.0010 as the working threshold for Gmail-heavy lists.
The tail window is mandatory because complaint-driven filtering degrades on a lag. A pilot scored on its last send date is scored before its own consequence arrives, which is why the default end-of-pilot run is 14 days after the last send rather than on the end date.
**This axis is a hard stop.** Crossing the absolute line in any window produces `kill` regardless of every other axis. The reason is that the cost is not on the pilot's ledger: sender reputation is shared across everything the company sends from those domains, including invoices, renewals, and support replies, and a pilot cannot spend it and call the saving a result.
## Denominators, stated once
- Contacted accounts: distinct accounts that received at least one agent-sent message inside the window.
- Replies: distinct accounts that replied at least once. A second reply from the same account does not add to the count.
- Meetings: distinct scheduled events. A reschedule is the same meeting; a second meeting on the same account inside 30 days is the same meeting.
- Qualified meetings: meetings surviving exclusion rules and the qualification gate.
Every rate on the memo names its denominator. A rate without a stated denominator is the format in which most pilot results are reported and the reason most of them cannot be compared to anything.
## Counterfactual
Optional and worth the effort. Cost per qualified meeting answers "what did this cost", not "what did this add". With a `counterfactual` object from a matched human-SDR segment or the same segment's prior period:
```
incremental_meetings = qualified_meetings - counterfactual_meetings_held_normalized
```
normalized to the same account count and window length. Where it is absent, the memo states explicitly that the figure is total, not incremental, and that a `keep` verdict therefore rests on the assumption that the meetings would not have happened otherwise. Say it plainly rather than letting the reader assume incrementality — that assumption is the one most often wrong and least often written down.
# 4 — Sample memo and field contract
The `memo_markdown` field is the deliverable a human reads. Structure is fixed: decision first, the axis that drove it second, the arithmetic third, what would change the answer last. Do not lead with methodology — a reader who trusts the method will not read it, and a reader who does not will read it before the number either way.
Both memos below are literal output. Copy the structure, not the values.
---
## Sample A — `kill`
```markdown
# AI SDR pilot: kill
**Decision: do not renew.** Scored 2026-06-29 against criteria registered 2026-04-28, before the first send. Criteria hash verified; no amendments.
## What drove it
The sending domain's spam complaint rate reached 0.34% in the 14 days after the last pilot send, crossing the 0.30% bulk-sender line. Baseline before the pilot was 0.08%. This is a hard stop independent of the economics: the cost of crossing it is borne by every mailbox the company sends from, including billing and support, and it is not on this pilot's ledger.
Two economic axes were also outside threshold. Fully-loaded cost per qualified meeting came in at 851 against a registered ceiling of 750, and rework ran 5.9 hours per qualified meeting against a ceiling of 3.0, with a worsening trend across the six weeks.
## The arithmetic
42-day window, 14-day tail. 34 meetings attributed by the vendor; 11 excluded (7 on accounts with an open opportunity at booking, 4 with human contact inside the prior 90 days); 19 held; 12 qualified after the 30-day stage-advance gate.
| Line | Value |
|---|---|
| Subscription (annual, prorated) | 5,178 |
| Data and enrichment (bundled) | 0 |
| Sending infrastructure (bundled) | 0 |
| Implementation (amortized over 12 months) | 700 |
| Rework (71 hours at 85) | 6,035 |
| **Total** | **11,913** |
Cost per qualified meeting: **851**. The subscription-only figure is **370** — that is the number in the vendor's business case, and the 481 gap is almost entirely the rework line.
Reply-to-meeting was inside threshold at 6.1% against a floor of 5.0%, and improving. Positive replies were labeled by the vendor's own classifier and discounted by the default factor, unaudited.
## What would change the answer
Nothing available inside this contract term. The reputation reading is a hard stop and would need a full baseline recovery plus a re-run at lower volume on isolated domains before the economics are worth re-testing. If that re-test is worth running, it needs its own pre-registration, separate sending domains, and a volume cap set below the level at which the complaint rate moved.
Note also that 32% of vendor-attributed meetings were on accounts already in cycle or recently human-touched. That ratio is a sourcing finding on its own and would not improve with a discount.
```
---
## Sample B — `extend`
```markdown
# AI SDR pilot: extend, 4 weeks
**Decision: extend for four weeks at current volume, then re-score.** Scored 2026-06-29 against criteria registered 2026-04-28. Criteria hash verified.
## What drove it
One axis outside threshold: rework at 3.8 hours per qualified meeting against a ceiling of 3.0. It is falling — 5.9 in weeks one and two, 3.4 in weeks five and six — which is the pattern of a team learning the tool rather than a tool that does not converge. Reputation is well inside its line, with the tail window at 0.09% against a 0.08% baseline.
## The arithmetic
Cost per qualified meeting: **604** against a ceiling of 750, on 18 qualified meetings from 41 attributed and 9 excluded. Reply-to-meeting 7.2% against a 5.0% floor, on an audited classifier precision of 0.86 from a 60-reply stratified sample.
## What would change the answer
Rework at or below 3.0 hours per qualified meeting across the four extension weeks, with cost per qualified meeting holding under 750 and the tail spam rate staying under 0.10%. If rework flattens above 3.0 rather than continuing to fall, this becomes a kill — the extension is testing the trend, not waiting for a better mood.
Volume is held flat deliberately. This memo scores the configuration at 2,000 contacts per month. Any increase invalidates the reputation reading and requires re-scoring at the new volume.
```
---
## Field contract for parsers
| Field | Type | Notes |
|---|---|---|
| `verdict` | `keep` / `extend` / `kill` / `null` | `null` only when `checkpoint: true` |
| `result` | string | `ok`, `no_preregistration`, `criteria_amended`, `insufficient_sample`, `incomplete_cost_model` |
| `retrospective` | boolean | `true` when `registered_at` does not precede `window_start` |
| `axes[].status` | `inside` / `outside` / `hard_stop_crossed` | `hard_stop_crossed` only on `reputation_delta` |
| `axes[].trend` | `improving` / `flat` / `worsening` | Absent on `reputation_delta`, which reports windows instead |
| `path_to_keep` | object | Present only on `extend` |
| `memo_markdown` | string | Always present when `result` is `ok` |
A non-`ok` `result` carries a populated diagnostic field and a `null` verdict. Never render a memo for a non-`ok` result — surface the diagnostic and the specific field that caused it, because "the tool would not score this" is a legible outcome and a fabricated verdict is not.
