Un Claude Skill que convierte un piloto de AI SDR terminado en un memo de continuar, matar o extender, calificado contra criterios que el equipo escribió antes del primer envío. Calcula cuatro ejes — costo totalmente cargado por reunión calificada, reuniones por respuesta positiva, horas humanas de retrabajo y delta de reputación de envío — y se niega a devolver un veredicto cuando los criterios fueron modificados a mitad del piloto o cuando la muestra es demasiado pequeña para sostenerlo. El bundle en apps/web/public/artifacts/ai-sdr-pilot-scorecard-skill/ incluye SKILL.md más cuatro archivos de referencia: la plantilla de pre-registro que el equipo completa antes de empezar el piloto, el modelo de costos de cinco líneas, las definiciones de métricas y dos memos de ejemplo literales.
El problema que resuelve no es la medición. Es que casi nadie escribe primero la línea de corte. Un piloto termina, el dashboard del vendor muestra reuniones agendadas, alguien divide la licencia entre ese número, y el resultado supera un umbral que nadie registró de antemano. Tres meses después el contrato se renueva automáticamente porque nadie produjo un documento que dijera cómo se vería el fracaso.
Cuándo usarlo
Ejecútalo 14 días después del último envío del piloto, con el export completo. La cola de 14 días es deliberada: las reuniones agendadas en la última semana todavía no se han celebrado, y el filtrado por quejas se degrada con retraso, así que un piloto calificado en su fecha de cierre se califica antes de que lleguen sus propias consecuencias.
Hay otros dos modos. Un checkpoint a mitad del piloto con checkpoint: true calcula los ejes y devuelve no_verdict — esa corrida existe para detectar una ruptura de reputación en la semana tres, no para decidir nada. Y una revisión de renovación sobre un despliegue ya en marcha trata los últimos 90 días como la ventana; es el único modo donde un pre-registro escrito después de los hechos es aceptable, y el memo lleva retrospective: true de forma permanente.
El skill encaja igual con 11x, Artisan, aisdr y agentes propios, porque califica tus sistemas y no los del vendor. Todo lo que lee viene de tu CRM, tu export de finanzas, tus registros de horas y Google Postmaster Tools.
Cuándo NO usarlo
No existe pre-registro. El skill devuelve no_preregistration y se detiene. Escribir los criterios ahora y antedatarlos produce un artefacto peor que ningún memo — convierte un juicio posterior en algo que parece una medición. El modo de revisión de renovación es la única excepción, y se etiqueta a sí mismo.
Por debajo de la muestra mínima registrada. Devuelve insufficient_sample con los conteos observado y requerido. Con n reuniones calificadas, agregar o quitar una mueve el costo por reunión en aproximadamente 1/n — 12,5% con ocho reuniones, 5% con veinte. Un piloto de seis semanas que produjo nueve reuniones celebradas no puede distinguir una tasa de respuesta-a-reunión de 4% de una de 7%, y un veredicto calculado sobre eso es ruido con un decimal.
Elegir entre dos vendors. Esto califica un despliegue contra un umbral, no dos entre sí. Un cara a cara necesita ambos corriendo sobre segmentos comparables con una lista de exclusión compartida, que es otro diseño.
La revisión trimestral de un equipo humano de SDR. El modelo de costos asume un medidor por lead o por contacto y una línea de retrabajo que no existe de la misma forma para headcount.
Setup
De dos a cuatro horas, en su mayoría decidiendo umbrales más que conectando nada.
Completa el pre-registro antes del primer envío. Copia references/1-preregistration-template.md, reemplaza cada placeholder, haz commit. Define min_held_meetings de modo que mover una reunión hacia dentro o hacia fuera cambie el costo por reunión menos que el margen entre tu umbral y tu valor esperado — si esperas aterrizar cerca de 700 contra un techo de 750, ese margen es de alrededor del 7% y veinte reuniones es el piso.
Define max_cost_per_qualified_meeting contra tu motion actual, no contra el precio de lista del vendor. Los precios publicados solo te dan la línea de suscripción. AiSDR publica 250 al mes por 200 contactos investigados por AI, 900 por 800 y 2.500 por 2.500, con los dos tiers mayores en compromiso trimestral. 11x publica 3.750 al mes facturado anualmente para Growth — 45.000 al año — cubriendo 2.000 prospects nuevos al mes y hasta cinco usuarios finales, y declara que cobra por lead y no por envío. Artisan no publica precio y describe sus tiers en aproximadamente 2.500 y aproximadamente 6.000 leads contactados al mes.
Define max_spam_rate en 0,0030 y déjalo ahí. Ese es el requisito publicado por Google para remitentes de más de 5.000 mensajes diarios a cuentas de Gmail: mantener la tasa de spam reportada en Postmaster Tools por debajo de 0,30%. Google recomienda además quedarse por debajo de 0,10% para que un pico ordinario de quejas no alcance la línea dura. En una lista con mucho Gmail, fija el umbral de trabajo en 0,0010 y trata 0,0030 como el corte.
Define las reglas de exclusión ahora.exclude_open_opp_at_booking y exclude_prior_human_touch_within_days: 90 son las dos que deciden qué cuenta como reunión del agente. Aflójalas aquí, por escrito, antes de saber hacia qué lado te conviene.
Instrumenta las horas de retrabajo desde el día uno. Un número semanal auto-reportado por las dos personas que hacen el trabajo es impreciso y honesto. Un número reconstruido al final por quien defiende la herramienta es preciso e inútil.
Instala el skill. Coloca SKILL.md y references/ en .claude/skills/ai-sdr-pilot-scorecard/, o súbelo como Skill en claude.ai. Arma el payload de entrada desde tu export de CRM, finanzas, registros de horas y Postmaster.
Por qué los cuatro ejes están construidos así
El costo por reunión calificada se emite dos veces. Una sobre las cinco líneas de costo — suscripción, datos y enriquecimiento, infraestructura de envío, implementación amortizada, y horas de retrabajo valuadas a tarifa cargada — y otra solo sobre suscripción. Ambas van al memo. La cifra de solo-suscripción es el número del business case del vendor; no está mal, responde otra pregunta, e imprimir ambas hace visible la base en lugar de discutible. En el ejemplo trabajado de references/2-cost-model.md, un piloto de 42 días con 14 reuniones calificadas da 851 por reunión cargado y 370 solo sobre suscripción, y 6.035 de los 6.735 que las separan son una sola línea: 71 horas de retrabajo humano. Esa distinción cambia el siguiente paso. “Demasiado caro” es una negociación; “seis horas de trabajo humano por reunión producida” es un problema de producto que un descuento no resuelve.
Respuesta-a-reunión cuenta solo respuestas positivas. Una respuesta negativa, un fuera-de-oficina y una solicitud de baja no son oportunidades de conversión, y agruparlas en un mismo denominador es cómo un gráfico de tasa de respuesta se mantiene plano mientras la lista se quema. Cuando el clasificador del propio vendor etiquetó las respuestas, references/3-metric-definitions.md aplica un descuento de 0,80 y marca la cifra como unaudited — una convención que ese archivo establece para que las corridas sigan siendo comparables, no un número medido de la industria. Aporta una muestra estratificada de 50 respuestas etiquetadas por humanos y el skill calcula la precisión real.
El retrabajo se califica por tendencia además de por nivel. Que caiga semana a semana significa que el equipo está aprendiendo la herramienta; plano o en alza significa que la herramienta no converge. Mismo nivel, veredictos opuestos.
La reputación es un corte duro, evaluado independientemente de la economía. Si la ventana de cola cruza la línea absoluta, el veredicto es kill sin importar lo que diga el costo por reunión. La reputación de remitente se comparte con todo lo que la empresa envía desde esos dominios — facturas, renovaciones, respuestas de soporte — así que un piloto no puede gastarla y anotar el ahorro como resultado.
Modos de fallo y guardas
La línea de corte se mueve a mitad del piloto. Es la forma más común en que sobrevive un piloto que fracasa, y rara vez es deshonesta — es un umbral suavizado en silencio en un doc por alguien que ya empezó a creer. Guarda: el paso 1 hashea el pre-registro y se niega a calificar contra uno modificado, devolviendo un diff a nivel de campo en lugar de un veredicto. Modificar está permitido; re-registra con un hash nuevo y el memo lleva retrospective: true de forma permanente.
Inflación de atribución. Los vendors cuentan reuniones sobre cuentas que ya estaban en ciclo o con contacto humano reciente. Guarda: las reuniones excluidas salen del numerador por regla y no por descuento negociado, y el conteo de exclusiones y la distribución de razones se reportan como hallazgo. Un vendor cuyas reuniones atribuidas son 32% cuentas ya abiertas te está diciendo algo sobre su sourcing que ningún descuento arregla.
El denominador de solo-suscripción.Guarda: la corrida se rechaza con incomplete_cost_model si alguna de las cinco líneas es null. Una línea que crees que es cero debe ingresarse como 0, lo cual es una afirmación y no un vacío.
El daño que aterriza después de la fecha de calificación.Guarda: la ventana de cola de 14 días es obligatoria y el eje de reputación puede matar un piloto que superó todos los umbrales de costo.
keep leído como “escalar”. El skill califica una configuración a un volumen. El volumen es el input con más probabilidad de romper el eje de reputación. Guarda: la sección final del memo nombra el volumen calificado y hace del recálculo una condición de cualquier aumento.
Contra las alternativas
El deck de QBR del vendor reporta el numerador. No tiene acceso a tus horas de retrabajo, ningún incentivo para aplicar tus reglas de exclusión, y etiqueta sus propias respuestas positivas. Léelo, y después ejecuta esto.
La hoja de cálculo de un analista de RevOps hace bien la aritmética y aun así pierde ante la deriva de criterios, porque la hoja se arma después de que los resultados son visibles y los umbrales se eligen junto a ellos. El hash es la única parte de este workflow que una hoja de cálculo estructuralmente no puede replicar.
Dejar que el contrato se renueve solo es el default real, y es lo que esto existe para desplazar. Sobre un compromiso 11x Growth, ese default cuesta 45.000 por un año que nadie decidió comprar.
---
name: ai-sdr-pilot-scorecard
description: Scores a finished or in-flight AI SDR / GTM-agent pilot against criteria registered before the pilot started, and emits a keep / kill / extend memo. Computes fully-loaded cost per qualified meeting, reply-to-meeting conversion split by reply polarity, human rework hours, and sending-reputation delta. Refuses to score against amended criteria or below the registered minimum sample. Use at the end of a time-boxed pilot, before the contract auto-renews.
---
# AI SDR pilot scorecard
## When to invoke
Invoke when a time-boxed AI SDR or GTM-agent pilot reaches its end date, or at a mid-pilot checkpoint the pre-registration named. Production patterns:
- **End-of-pilot decision.** Run once, 14 days after the last send, with the full export. The 14-day tail exists because meetings booked in the final week have not yet held, and reputation damage lands after the sending stops.
- **Mid-pilot checkpoint.** Run at the halfway mark against the same registered criteria, with `checkpoint: true`. The skill returns axis-level readings and an explicit `no_verdict` — a mid-pilot run is for catching a reputation breach early, not for deciding.
- **Renewal review on a running deployment.** Treat the trailing 90 days as the pilot window. This is the only mode where a pre-registration written after the fact is acceptable, and the memo says so on its face.
Do NOT invoke this skill for:
- **A pilot with no pre-registration.** The skill returns `no_preregistration` and stops. Scoring criteria chosen after seeing the results is the failure this skill exists to prevent — writing them now and backdating them produces a worse artifact than no memo at all. The exception is the renewal-review mode above, which stamps the memo accordingly.
- **A pilot below the registered minimum sample.** Returns `insufficient_sample` with the observed and required counts. A six-week pilot that produced 9 held meetings cannot distinguish a 4% reply-to-meeting rate from a 7% one, and a verdict computed on it is noise wearing a decimal point.
- **Choosing between two vendors.** This skill scores one deployment against a threshold, not two against each other. A head-to-head needs both running on comparable segments with a shared exclusion list, which is a different design.
- **A fully human SDR team's quarterly review.** The cost model assumes a per-lead or per-contact meter and a human-rework line that does not exist in the same shape for a headcount-based team.
## Inputs
Required:
- `preregistration` — object. The parsed contents of the pre-registration file written before the pilot started. Required shape: `{ registered_at, window_start, window_end, min_held_meetings, thresholds: { max_cost_per_qualified_meeting, min_reply_to_meeting_rate, max_rework_hours_per_meeting, max_spam_rate }, exclusion_rules, qualification_gate }`.
- `preregistration_sha256` — string. SHA-256 of the pre-registration file as committed. The skill recomputes it and refuses to score on a mismatch.
- `meetings` — array. One entry per meeting the vendor or your CRM attributes to the pilot: `{ account_id, booked_at, held, held_at, stage_at_plus_30d, prior_human_touch_at, open_opp_at_booking, source_claimed_by }`.
- `costs` — object. All five cost lines, in the currency of the contract: `{ subscription, data_and_enrichment, sending_infrastructure, implementation_amortized, human_rework_hours }`. A `null` in any line is a hard rejection, not a zero.
- `loaded_hourly_rate` — number. Fully-loaded cost of an hour of the human doing the rework. Used to price `human_rework_hours`.
- `replies` — array. `{ account_id, replied_at, polarity, labeled_by }` where `polarity` is `positive` / `negative` / `out_of_office` / `unsubscribe_request` and `labeled_by` is `vendor_classifier` or `human`.
- `reputation` — object. `{ baseline_window, pilot_window, tail_window }`, each `{ spam_rate, domain, daily_volume_peak }` read from Google Postmaster Tools or the equivalent for your recipient mix.
Optional:
- `checkpoint` — boolean. Default `false`. When `true`, the skill computes the axes and returns `no_verdict`.
- `reply_label_audit` — array. A stratified human-labeled sample of replies, used to compute the vendor classifier's precision on `positive`. Without it the skill discounts vendor-labeled positives by the documented default and says so in the memo.
- `counterfactual` — object. `{ meetings_held, window }` from a matched human-SDR segment or the same segment's prior period. Turns the memo's incremental-lift section from a caveat into a number.
## Reference files
Load these from `references/` before first run. All four are stable within a deployment.
- `references/1-preregistration-template.md` — the file the team fills in **before** the pilot starts. Contains the four thresholds with the reasoning for each default, the exclusion-rule scaffolding, and the qualification-gate definition.
- `references/2-cost-model.md` — the five cost lines, what belongs in each, and the amortization rule for implementation fees. Contains a worked example against a published vendor price.
- `references/3-metric-definitions.md` — exact definitions and the arithmetic for each axis, including the reply-polarity split and the reputation window construction.
- `references/4-sample-memo.md` — a literal `kill` memo and a literal `keep` memo, plus the structured-field contract for parsers.
## Method
Run these steps in order. Steps 1 and 2 gate everything after them.
### 1. Pre-registration integrity check
Recompute SHA-256 over the `preregistration` object as serialized in the template's canonical form and compare to `preregistration_sha256`. On mismatch, return `result: criteria_amended` with a diff of the changed threshold fields — do not score. On a missing pre-registration, return `result: no_preregistration`.
The hash is the whole point of the step. Criteria drift is not usually dishonest; it is a threshold quietly softened in a doc three weeks into a pilot that is not going well, by someone who has already started to believe. A hash makes the softening a visible event rather than an invisible one.
Also verify `registered_at` precedes `window_start`. A pre-registration written after the first send is not a pre-registration; flag it as `retrospective: true` and carry that flag onto the memo.
### 2. Sample-adequacy gate
Count meetings that pass both the exclusion rules and the qualification gate. If the count is below `min_held_meetings`, return `result: insufficient_sample` with observed and required counts, and a note on how many more weeks at the observed rate would reach it. Do not return a verdict.
This gate is upstream of the economics for a reason. Cost per qualified meeting on a denominator of 7 moves by more than 14% every time one meeting is added or removed, which is wider than the gap most kill thresholds are set at.
### 3. Attribution filtering
Apply the exclusion rules from the pre-registration to every entry in `meetings`. The defaults in the template exclude a meeting when any of these holds:
- `open_opp_at_booking` is true — the account already had an open opportunity.
- `prior_human_touch_at` falls inside the 90 days before `booked_at`.
- The account appears on a named-account list the pre-registration excluded.
Excluded meetings are removed from the numerator entirely, not discounted by a factor. A discount is a judgment call that gets negotiated; a rule is a rule. Report the excluded count and the exclusion reason distribution — the ratio of excluded to included is itself a finding, and a vendor whose attributed meetings are 40% already-open accounts is telling you something about its sourcing.
### 4. Qualification gate
Apply `qualification_gate` from the pre-registration to the surviving meetings. The template default requires both `held` is true and `stage_at_plus_30d` has advanced past the initial stage. Booked-but-no-show meetings count as zero and stay out of the numerator; they are still counted separately and reported, because a high no-show rate on a healthy booking rate is a targeting problem the cost figure would otherwise hide.
### 5. Cost assembly
Sum the five lines from `costs`, pricing `human_rework_hours` at `loaded_hourly_rate`. Reject the run with `result: incomplete_cost_model` if any line is `null`.
Divide by the qualified-meeting count from step 4 to get fully-loaded cost per qualified meeting. Report the same figure computed on subscription alone, labeled as the vendor-facing number, so the memo shows both and the gap between them is legible. That gap is the argument.
### 6. Reply-to-meeting conversion, split by polarity
Compute replies per contacted account, then meetings per reply. Split the reply count by `polarity`. A `negative` reply and an `unsubscribe_request` are not conversion opportunities and must not sit in the same denominator as a `positive` one — collapsing them is how a reply-rate chart stays flat while the underlying list burns.
Where `labeled_by` is `vendor_classifier` and `reply_label_audit` is supplied, compute the classifier's precision on `positive` and multiply the vendor-labeled positive count by it. Where the audit is absent, apply the documented discount from `references/3-metric-definitions.md` and mark the figure `unaudited` in the memo.
### 7. Reputation delta
Compare `spam_rate` across the three windows. Two independent readings:
- **Trend.** Pilot and tail against baseline, on the same domains.
- **Absolute line.** Whether any window crossed the bulk-sender complaint threshold your recipient mix is judged against. For Gmail recipients at more than 5,000 messages per day, Google's published requirement is to keep the reported spam rate under 0.30%, with under 0.10% as the recommended operating point.
The tail window is where the damage usually shows. A pilot that ends on day 42 and is scored on day 42 is scored before its own consequences arrive.
**This axis is a hard stop, evaluated independently of the economics.** If the tail window crossed the absolute line, the verdict is `kill` regardless of cost per meeting, and the memo says the pilot was cheap because it was spending an asset that is not on the pilot's ledger — the deliverability of every mailbox in the company.
### 8. Verdict assembly
Compare each axis against its registered threshold and return one verdict:
- `keep` — every axis inside threshold.
- `extend` — at most one axis outside threshold, the reputation axis inside its hard stop, and the observed trend on the failing axis improving across the pilot window. The memo names the single axis and the specific reading that would make it a `keep`.
- `kill` — the reputation hard stop was crossed, or two or more axes are outside threshold, or one axis is outside threshold with a flat or worsening trend.
- `no_verdict` — `checkpoint: true`.
The `extend` verdict is deliberately narrow. A verdict of `extend` available on any failing configuration is not a verdict, it is a way of never deciding, and it is the outcome pilots default to when nobody wrote the thresholds down.
## Output format
Literal JSON the skill emits for a `kill` verdict:
```json
{
"verdict": "kill",
"result": "ok",
"retrospective": false,
"window": { "start": "2026-05-04", "end": "2026-06-15", "tail_end": "2026-06-29" },
"sample": {
"meetings_attributed": 34,
"excluded": 11,
"exclusion_reasons": { "open_opp_at_booking": 7, "prior_human_touch_90d": 4 },
"held": 19,
"qualified": 12,
"min_required": 12
},
"axes": [
{
"axis": "cost_per_qualified_meeting",
"observed": 1042.00,
"threshold": 750.00,
"status": "outside",
"trend": "flat",
"subscription_only_figure": 433.33,
"note": "Vendor-facing figure omits 71 rework hours and the enrichment line."
},
{
"axis": "reply_to_meeting_rate",
"observed": 0.061,
"threshold": 0.050,
"status": "inside",
"trend": "improving",
"positive_replies_audited": false,
"note": "Vendor-classifier positives discounted by the default factor; no reply_label_audit supplied."
},
{
"axis": "rework_hours_per_qualified_meeting",
"observed": 5.9,
"threshold": 3.0,
"status": "outside",
"trend": "worsening"
},
{
"axis": "reputation_delta",
"observed_tail_spam_rate": 0.0034,
"absolute_line": 0.0030,
"baseline_spam_rate": 0.0008,
"status": "hard_stop_crossed",
"note": "Tail window on sending domain crossed the bulk-sender complaint line 11 days after the last pilot send."
}
],
"counterfactual": null,
"memo_markdown": "...",
"run_metadata": {
"preregistration_sha256": "…",
"model": "claude-opus-5",
"definitions_version": "1.0.0"
}
}
```
A `keep` verdict has every axis at `status: "inside"` and a populated `memo_markdown`. An `extend` verdict has exactly one axis `outside`, `reputation_delta` inside its line, and a `path_to_keep` object naming the required reading. A `no_verdict` response carries populated `axes` and a `null` verdict.
The `memo_markdown` field is the deliverable a human reads. Its structure is fixed in `references/4-sample-memo.md` — decision first, the axis that drove it second, the arithmetic third, and what would change the answer last.
## Watch-outs
- **Criteria drift mid-pilot.** The most common way a pilot survives is that its kill line moves. **Guard:** step 1 hashes the pre-registration and refuses to score against an amended one, returning a field-level diff instead of a verdict. Amending is allowed — the team re-registers with a new hash, and the memo carries `retrospective: true` forever after.
- **Attribution inflation.** Vendors count meetings on accounts that were already in cycle or already human-touched. **Guard:** step 3 removes them from the numerator by rule rather than discounting them by negotiation, and reports the excluded count and reason distribution as a finding in its own right.
- **The subscription-only denominator.** Cost per meeting computed on the license price alone reliably understates the real figure, because the rework hours and the enrichment meter are the lines nobody put on the pilot budget. **Guard:** step 5 rejects the run when any of the five cost lines is `null`, and the memo prints the vendor-facing figure next to the loaded one so the gap is visible rather than arguable.
- **Reputation damage arriving after the scoring date.** Complaint-driven filtering degrades on a lag, so a pilot scored on its end date is scored before its cost lands. **Guard:** the tail window is mandatory, and the reputation axis is a hard stop evaluated independently of the economics — it can kill a pilot that cleared every cost threshold.
- **Vendor-classified positive replies.** "Positive reply" labeled by the system being evaluated is a self-graded exam. **Guard:** step 6 computes classifier precision against `reply_label_audit` when supplied and applies the documented discount when it is not, marking the figure `unaudited` on the memo rather than silently trusting it.
- **A verdict on a sample too small to carry one.** **Guard:** step 2 gates on the registered minimum and returns `insufficient_sample` — with the weeks-to-adequacy estimate — rather than a confident number computed on nine meetings.
- **`keep` read as "scale".** The skill scores the pilot's configuration at the pilot's volume. Send volume is the input most likely to break the reputation axis, and a `keep` at 2,000 contacts per month is not a `keep` at 10,000. **Guard:** the memo's final section states the scored volume explicitly and names re-scoring at the new volume as a condition of any increase.
# 1 — Pre-registration template
Fill this in **before the first send**. Commit it. The skill hashes it and refuses to score against a version that changed afterwards.
Replace every `<…>` placeholder. Do not delete fields you think do not apply — set them explicitly, because a missing threshold and a threshold of "we did not care" are different claims and the memo distinguishes them.
```yaml
registered_at: <YYYY-MM-DD> # must precede window_start
registered_by: <name, role>
vendor: <11x | artisan | aisdr | unify | homegrown | other>
plan_and_meter: <e.g. "Growth, 2,000 new prospects/mo, 5 end users">
window_start: <YYYY-MM-DD> # first send
window_end: <YYYY-MM-DD> # last send
tail_days: 14 # do not lower; see reputation note below
min_held_meetings: <integer> # see sizing note below
thresholds:
max_cost_per_qualified_meeting: <number> # your currency
min_reply_to_meeting_rate: <decimal> # meetings per positive reply
max_rework_hours_per_meeting: <number>
max_spam_rate: 0.0030 # absolute line; see note
qualification_gate:
require_held: true
require_stage_advance_by_days: 30
exclusion_rules:
exclude_open_opp_at_booking: true
exclude_prior_human_touch_within_days: 90
excluded_named_account_lists:
- <list name or CRM view id>
loaded_hourly_rate: <number> # fully loaded, not salary/2080
```
## Sizing `min_held_meetings`
Set it so that one meeting moving in or out changes cost per qualified meeting by less than the margin between your threshold and your expected value. The arithmetic: with `n` qualified meetings, adding or removing one moves the per-meeting cost by roughly `1/n`. At `n = 8` that is 12.5%; at `n = 20` it is 5%; at `n = 40` it is 2.5%.
If your threshold is 750 and you expect to land near 700, your margin is about 7% and `n = 20` is the floor. If you expect to land near 400 against a threshold of 750, the margin is wide and `n = 12` is defensible.
If the pilot cannot plausibly produce your `n` in the window, the honest move is to lengthen the window before starting, not to score a short one. Write the required `n` down anyway — an `insufficient_sample` return at the end is a real finding, and it is the finding that the pilot was not designed to answer the question.
## The four thresholds
**`max_cost_per_qualified_meeting`.** Anchor it to what the same meeting costs you today through your existing motion, not to the vendor's list price. Published entry prices give you the floor of the subscription line only: AiSDR publishes 250 per month for 200 AI-researched contacts, 900 for 800, and 2,500 for 2,500, with the middle and top tiers on a quarterly commitment. 11x publishes 3,750 per month billed annually for its Growth plan, covering 2,000 new prospects per month and up to five end users, and states it charges per lead rather than per send. Artisan publishes no price and scopes its tiers at roughly 2,500 and roughly 6,000 leads contacted per month. Those are subscription lines, not costs per meeting — the cost model in `2-cost-model.md` adds the four other lines that turn one into the other.
**`min_reply_to_meeting_rate`.** Meetings per *positive* reply, not per reply. Set it from your own historical rate on the same segment if you have one. If you do not, set it and note that you do not — an unanchored threshold you registered is still better than a rate you interpret after the fact.
**`max_rework_hours_per_meeting`.** The hours a human spends editing drafts, correcting bad targeting, cleaning the CRM after the agent, and handling replies the agent mishandled. This is the line that is never in the business case. Instrument it from day one, even crudely: a weekly self-reported number from the two people doing it beats a precise number you reconstruct at the end.
**`max_spam_rate`.** The default of 0.0030 is Google's published bulk-sender requirement for senders of more than 5,000 messages per day to Gmail accounts, which is to keep the spam rate reported in Postmaster Tools under 0.30%. Google separately recommends staying under 0.10% so that an ordinary complaint spike does not push you over. If your recipient mix is Gmail-heavy, set the threshold at 0.0010 and treat 0.0030 as the hard stop. Do not raise this field. It is not a preference; it is the line above which a mailbox provider starts making decisions about your domain.
## The exclusion rules
These decide what counts as the agent's meeting. Set them now, because after the pilot every excluded meeting is an argument.
`exclude_open_opp_at_booking` and `exclude_prior_human_touch_within_days: 90` are the two that matter most. An agent that books a meeting on an account your AE has been working for a month did not source that meeting; it scheduled it. Both defaults are deliberately strict — if you want them looser, loosen them here, in writing, before you know which way it cuts.
`excluded_named_account_lists` exists for the top-of-house accounts you would never let an agent source into. If that list is empty, say so explicitly by leaving the key with an empty list rather than deleting it.
## Amending
You may amend. Re-register with a new `registered_at`, commit, and re-hash. The skill will score against the new file and stamp `retrospective: true` on the memo permanently. That stamp is not a punishment — it is the correct label for a number produced against criteria chosen with partial knowledge of the outcome, and a reader of the memo six months from now needs it.
# 2 — Cost model
Five lines. All five are required; the skill rejects the run with `incomplete_cost_model` if any is `null`. A line you believe is zero must be entered as `0`, which is a claim you are making, not a gap you are leaving.
## Line 1 — `subscription`
The contracted license cost for the pilot window, prorated if the contract is annual.
Vendor meters differ in kind, not just in price, and the difference changes what "more volume" costs you:
- **Per lead or per prospect.** 11x publishes a Growth plan at 3,750 per month billed annually — 45,000 for the year — covering 2,000 new prospects per month and up to five end users, and states plainly that it charges per lead, not per send. Under a per-lead meter, sending more touches to the same 2,000 prospects is free and widening the list is not.
- **Per AI-researched contact.** AiSDR publishes 250 per month for 200 contacts, 900 per month for 800, and 2,500 per month for 2,500, with the two larger tiers on a quarterly commitment and a 20% discount for annual billing. The unit price falls from 1.25 per contact at the entry tier to 1.00 at the top published one.
- **Quote-only, scoped by leads contacted.** Artisan publishes no price and describes its tiers at roughly 2,500 and roughly 6,000 leads contacted per month.
Prorating an annual commitment across a six-week pilot: `subscription = annual_total × (pilot_days / 365)`. For 11x Growth at 45,000 per year and a 42-day window, that is 45,000 × 42/365, or about 5,178. Use the prorated figure, and note in the memo that the real commitment is annual — a pilot that cost 5,178 to run may have cost 45,000 to enter, and if the contract is already signed, the kill decision is about the next term, not this one.
## Line 2 — `data_and_enrichment`
Credits consumed for enrichment, verification, and intent data during the window. Include spend on tools the agent called even when they are billed elsewhere — a Clay or ZoomInfo meter that spiked during the pilot is a pilot cost regardless of which cost centre it landed in.
Bundled-plan trap: several vendors bundle contact data into the subscription, which does not make it free, it makes it invisible. When it is bundled, enter `0` and note the bundling in the memo, because the moment you scale past the bundled allowance it becomes a variable line and your cost per meeting moves.
## Line 3 — `sending_infrastructure`
Domains, mailboxes, warmup services, and any deliverability tooling bought or expanded for the pilot. Also invisible when bundled — 11x, for instance, states that every plan bundles deliverability, mailbox warmup, and inbox rotation at no additional cost.
The reason this line exists even when it is small: it is the line that grows fastest under volume increases, and a `keep` verdict that leads to tripled volume needs the reader to know which lines are fixed and which are not.
## Line 4 — `implementation_amortized`
One-time onboarding, implementation, and integration fees, amortized across the term you would actually commit to if you keep. Not across the pilot window.
A 10,000 implementation fee charged against a six-week pilot makes the pilot look catastrophic; charged across a twelve-month term at 833 per month, it makes the decision you are actually making. If you would only ever commit to twelve months, amortize across twelve. If the fee is refundable on non-renewal, enter `0` and note it.
## Line 5 — `human_rework_hours`
Hours, not currency. The skill prices them at `loaded_hourly_rate`.
What counts:
- Editing or rewriting agent drafts before send.
- Correcting targeting — removing accounts the agent should not have touched, fixing segment definitions after the fact.
- CRM cleanup attributable to the agent: duplicate contacts, wrong-account activity, stage changes that had to be reversed.
- Handling replies the agent mishandled, including apology emails and internal escalations.
- Meetings that were booked and then rescheduled or unwound by a human.
What does not count: initial setup and configuration, which belongs in line 4; and ordinary AE preparation for a meeting that would have happened anyway.
Instrument this from day one. A weekly number self-reported by the two or three people doing the work is imprecise and honest. A number reconstructed at the end of the pilot by the person advocating for the tool is precise and useless.
## Worked example
A 42-day pilot on a published entry plan, 14 qualified meetings after exclusions and the qualification gate:
| Line | Value |
|---|---|
| `subscription` (annual 45,000, prorated 42/365) | 5,178 |
| `data_and_enrichment` (bundled) | 0 |
| `sending_infrastructure` (bundled) | 0 |
| `implementation_amortized` (6,000 over 12 months, 1.4 months elapsed) | 700 |
| `human_rework_hours` (71 hours at 85) | 6,035 |
| **Total** | **11,913** |
Cost per qualified meeting: 11,913 / 14 = **851**.
Subscription-only figure: 5,178 / 14 = **370**.
Both go on the memo. The 370 is the number in the vendor's business case and it is not wrong, it is answering a different question. The gap between the two is 481 per meeting, and 6,035 of the 6,735 that separates them is one line: the hours a human spent making the output usable. If that line is the reason the pilot fails its threshold, the memo should say so plainly, because "the tool is too expensive" and "the tool needs six hours of human work per meeting it produces" lead to different next steps — the first is a negotiation, the second is a product problem that a discount does not fix.
# 3 — Metric definitions
Version `1.0.0`. The skill stamps this version into `run_metadata.definitions_version`. Change the version when you change a definition, so two memos with different arithmetic are never silently compared.
## Axis 1 — Cost per qualified meeting
```
cost_per_qualified_meeting = total_cost / qualified_meetings
```
`total_cost` is the sum of the five lines in `2-cost-model.md`, with `human_rework_hours × loaded_hourly_rate` substituted for line 5.
`qualified_meetings` is the count surviving both the exclusion rules and the qualification gate. Booked-but-not-held meetings are excluded from the denominator and reported separately as `held_rate`.
Also emit `subscription_only_figure = subscription / qualified_meetings`. This is the vendor-facing number. It is not a competing estimate — it is the same arithmetic on a narrower cost basis, and printing both is how the memo makes the basis visible instead of arguable.
## Axis 2 — Reply-to-meeting rate
```
reply_to_meeting_rate = qualified_meetings / positive_replies
```
Not meetings per reply. `positive_replies` counts only `polarity: positive`. A `negative`, `out_of_office`, or `unsubscribe_request` reply is not a conversion opportunity, and pooling them into one denominator is how a reply-rate line stays flat while the list is being burned. Report all four polarity counts on the memo, and report `unsubscribe_request` prominently — a rising unsubscribe count against a flat positive count is the leading indicator for axis 4.
### Classifier discount
When `labeled_by: vendor_classifier` and `reply_label_audit` is supplied:
```
adjusted_positive_replies = vendor_positive_count × classifier_precision
```
where `classifier_precision` is the fraction of a human-labeled stratified sample that the classifier called `positive` and a human agreed was positive. Sample at least 50 replies, stratified across the window so a mid-pilot prompt change does not sit entirely inside or outside the sample.
When no audit is supplied, apply `default_classifier_discount = 0.80` and mark the axis `unaudited` on the memo. This default is a convention this file sets so that runs are comparable to each other, not a measured industry figure — it is deliberately conservative because the classifier is a component of the system under evaluation and the direction of its error is not neutral. Any deployment that runs this skill more than once should replace it with its own audited precision and note the change here.
## Axis 3 — Rework hours per qualified meeting
```
rework_hours_per_qualified_meeting = human_rework_hours / qualified_meetings
```
Trend matters as much as level. Compute it per week across the window and report the direction. Rework hours falling week over week means the team is learning the tool, and a level above threshold with a falling trend is the canonical `extend` case. Rework hours flat or rising means the tool is not converging, and the same level with a rising trend is a `kill`.
## Axis 4 — Reputation delta
Three windows, same domains throughout:
- `baseline_window` — the 30 days ending the day before `window_start`.
- `pilot_window` — `window_start` to `window_end`.
- `tail_window` — `window_end + 1` to `window_end + tail_days`, default 14 days.
Two independent readings:
**Trend.** `pilot_spam_rate - baseline_spam_rate` and `tail_spam_rate - baseline_spam_rate`. A baseline near zero and a tail at 0.0018 has not crossed any line and has still moved by more than a factor of two, which is a finding.
**Absolute line.** Whether any window crossed `thresholds.max_spam_rate`. For a Gmail-heavy recipient mix at more than 5,000 messages per day, Google's published requirement is that senders keep the spam rate reported in Postmaster Tools under 0.30%, with under 0.10% recommended so that an ordinary spike does not reach 0.30%. The template therefore sets 0.0030 as the hard stop and suggests 0.0010 as the working threshold for Gmail-heavy lists.
The tail window is mandatory because complaint-driven filtering degrades on a lag. A pilot scored on its last send date is scored before its own consequence arrives, which is why the default end-of-pilot run is 14 days after the last send rather than on the end date.
**This axis is a hard stop.** Crossing the absolute line in any window produces `kill` regardless of every other axis. The reason is that the cost is not on the pilot's ledger: sender reputation is shared across everything the company sends from those domains, including invoices, renewals, and support replies, and a pilot cannot spend it and call the saving a result.
## Denominators, stated once
- Contacted accounts: distinct accounts that received at least one agent-sent message inside the window.
- Replies: distinct accounts that replied at least once. A second reply from the same account does not add to the count.
- Meetings: distinct scheduled events. A reschedule is the same meeting; a second meeting on the same account inside 30 days is the same meeting.
- Qualified meetings: meetings surviving exclusion rules and the qualification gate.
Every rate on the memo names its denominator. A rate without a stated denominator is the format in which most pilot results are reported and the reason most of them cannot be compared to anything.
## Counterfactual
Optional and worth the effort. Cost per qualified meeting answers "what did this cost", not "what did this add". With a `counterfactual` object from a matched human-SDR segment or the same segment's prior period:
```
incremental_meetings = qualified_meetings - counterfactual_meetings_held_normalized
```
normalized to the same account count and window length. Where it is absent, the memo states explicitly that the figure is total, not incremental, and that a `keep` verdict therefore rests on the assumption that the meetings would not have happened otherwise. Say it plainly rather than letting the reader assume incrementality — that assumption is the one most often wrong and least often written down.
# 4 — Sample memo and field contract
The `memo_markdown` field is the deliverable a human reads. Structure is fixed: decision first, the axis that drove it second, the arithmetic third, what would change the answer last. Do not lead with methodology — a reader who trusts the method will not read it, and a reader who does not will read it before the number either way.
Both memos below are literal output. Copy the structure, not the values.
---
## Sample A — `kill`
```markdown
# AI SDR pilot: kill
**Decision: do not renew.** Scored 2026-06-29 against criteria registered 2026-04-28, before the first send. Criteria hash verified; no amendments.
## What drove it
The sending domain's spam complaint rate reached 0.34% in the 14 days after the last pilot send, crossing the 0.30% bulk-sender line. Baseline before the pilot was 0.08%. This is a hard stop independent of the economics: the cost of crossing it is borne by every mailbox the company sends from, including billing and support, and it is not on this pilot's ledger.
Two economic axes were also outside threshold. Fully-loaded cost per qualified meeting came in at 851 against a registered ceiling of 750, and rework ran 5.9 hours per qualified meeting against a ceiling of 3.0, with a worsening trend across the six weeks.
## The arithmetic
42-day window, 14-day tail. 34 meetings attributed by the vendor; 11 excluded (7 on accounts with an open opportunity at booking, 4 with human contact inside the prior 90 days); 19 held; 12 qualified after the 30-day stage-advance gate.
| Line | Value |
|---|---|
| Subscription (annual, prorated) | 5,178 |
| Data and enrichment (bundled) | 0 |
| Sending infrastructure (bundled) | 0 |
| Implementation (amortized over 12 months) | 700 |
| Rework (71 hours at 85) | 6,035 |
| **Total** | **11,913** |
Cost per qualified meeting: **851**. The subscription-only figure is **370** — that is the number in the vendor's business case, and the 481 gap is almost entirely the rework line.
Reply-to-meeting was inside threshold at 6.1% against a floor of 5.0%, and improving. Positive replies were labeled by the vendor's own classifier and discounted by the default factor, unaudited.
## What would change the answer
Nothing available inside this contract term. The reputation reading is a hard stop and would need a full baseline recovery plus a re-run at lower volume on isolated domains before the economics are worth re-testing. If that re-test is worth running, it needs its own pre-registration, separate sending domains, and a volume cap set below the level at which the complaint rate moved.
Note also that 32% of vendor-attributed meetings were on accounts already in cycle or recently human-touched. That ratio is a sourcing finding on its own and would not improve with a discount.
```
---
## Sample B — `extend`
```markdown
# AI SDR pilot: extend, 4 weeks
**Decision: extend for four weeks at current volume, then re-score.** Scored 2026-06-29 against criteria registered 2026-04-28. Criteria hash verified.
## What drove it
One axis outside threshold: rework at 3.8 hours per qualified meeting against a ceiling of 3.0. It is falling — 5.9 in weeks one and two, 3.4 in weeks five and six — which is the pattern of a team learning the tool rather than a tool that does not converge. Reputation is well inside its line, with the tail window at 0.09% against a 0.08% baseline.
## The arithmetic
Cost per qualified meeting: **604** against a ceiling of 750, on 18 qualified meetings from 41 attributed and 9 excluded. Reply-to-meeting 7.2% against a 5.0% floor, on an audited classifier precision of 0.86 from a 60-reply stratified sample.
## What would change the answer
Rework at or below 3.0 hours per qualified meeting across the four extension weeks, with cost per qualified meeting holding under 750 and the tail spam rate staying under 0.10%. If rework flattens above 3.0 rather than continuing to fall, this becomes a kill — the extension is testing the trend, not waiting for a better mood.
Volume is held flat deliberately. This memo scores the configuration at 2,000 contacts per month. Any increase invalidates the reputation reading and requires re-scoring at the new volume.
```
---
## Field contract for parsers
| Field | Type | Notes |
|---|---|---|
| `verdict` | `keep` / `extend` / `kill` / `null` | `null` only when `checkpoint: true` |
| `result` | string | `ok`, `no_preregistration`, `criteria_amended`, `insufficient_sample`, `incomplete_cost_model` |
| `retrospective` | boolean | `true` when `registered_at` does not precede `window_start` |
| `axes[].status` | `inside` / `outside` / `hard_stop_crossed` | `hard_stop_crossed` only on `reputation_delta` |
| `axes[].trend` | `improving` / `flat` / `worsening` | Absent on `reputation_delta`, which reports windows instead |
| `path_to_keep` | object | Present only on `extend` |
| `memo_markdown` | string | Always present when `result` is `ok` |
A non-`ok` `result` carries a populated diagnostic field and a `null` verdict. Never render a memo for a non-`ok` result — surface the diagnostic and the specific field that caused it, because "the tool would not score this" is a legible outcome and a fabricated verdict is not.
