Braintrust

What it is

Braintrust is where you find out whether the change you just made to an AI feature made it better or worse. You instrument the application, traces land in Braintrust, and the failures in those traces become datasets, scorers and experiments that re-run on every subsequent change. Ankur Goyal founded it in summer 2023 after Figma acquired his previous company, Impira. It raised a $5.1M seed led by Greylock, then an $80M Series B led by ICONIQ on 2026-02-17 — Andreessen Horowitz, Greylock and Elad Gil participating — at an $800M valuation and roughly $121M raised. Notion, Replit, Cloudflare, Ramp and Dropbox are named customers.

The scoped problem, in the words an ops leader actually uses: your support agent or contract-review assistant was fine last month, someone edited a prompt or swapped a model, and nobody can say what regressed or by how much. Braintrust’s answer is a scored regression suite for non-deterministic systems — the discipline of a unit-test suite, except the assertions are graders rather than equality checks, and the grading is done by an LLM judge, a code scorer, or a human reviewer.

What you are actually buying

Three pieces do the work, and most teams need only the first two on day one.

Tracing into Brainstore. Braintrust wrote its own store for trace data rather than putting spans in a general-purpose observability backend, and the reason shows up when you filter a few million traces on a text field. It ingests OpenTelemetry — the BraintrustSpanProcessor drops into an existing tracer provider with a filter so only AI spans are forwarded, and it implements the OTel GenAI semantic conventions. SDKs cover Python, TypeScript, Go, Ruby, C# and Java. The AI proxy puts OpenAI, Anthropic, Google and the open-weight models behind one endpoint.

Loop. The part that changed most during 2026, and the part to evaluate hardest. Loop is an agent pointed at your own trace data: describe a failure in prose, and it reads the traces, proposes a dataset of the cases that exhibit it, and drafts a scorer. In September 2026 it moved into a managed runtime with persistent threads and project-wide scope, and gained scheduled automations that run on their own instructions and write permissions. Patterns and Debugger shipped alongside it — the first hunts for recurring failures no scorer measures yet, the second reports likely failure modes for a single trace with its evidence.

Human review. Annotation queues with assignment and Slack notification, plus blind reviews that hide peer scores until a reviewer finishes their own — which matters more than it sounds, because agreement statistics anchored on a colleague’s visible answer are not agreement statistics.

Pricing

Published at braintrust.dev/pricing, checked 2026-09-20. The structural fact that decides most evaluations: every plan has unlimited users. You are metered on data processed and scores run, never on seats.

• Starter — $0/month. $10 of model credits, 1 GB of processed data then $4/GB, 10k scores then $2.50 per 1k, 14-day retention. Unlimited users, projects, datasets, playgrounds and experiments. No card required.
• Pro — $249/month. $100 of credits, 5 GB then $3/GB, 50k scores then $1.50 per 1k, 30-day retention with longer retention at $0.50/GB/month — extended to 180 days in July 2026. Adds custom charts, environments, RBAC and priority support.
• Enterprise — custom. Custom retention and export, premium support, and self-hosted or bring-your-own-cloud deployment.

Compare the shape against LangSmith, which publishes $39 per seat per month on Plus with 10k base traces included and pay-as-you-go above. On a 12-person team that is $468/month in seats before a single trace is stored; the same 12 people cost nothing on Braintrust and the bill is entirely volume. It inverts at low headcount and high traffic — two people pushing heavy production volume will pay Braintrust more than two LangSmith seats. Model the ratio of people to traces before assuming seatless is cheaper.

Best for

An engineering or ops team that has already shipped an AI feature to production, has a quality complaint it cannot quantify, and has more reviewers than engineers. The reviewer count is the signal: subject-matter experts — the CSM who knows what a good support answer looks like, the paralegal who knows a bad clause summary — need accounts to label, and on a per-seat platform every one of them is a line item. Braintrust prices that team correctly, and nothing in it assumes a framework, so a stack that is not LangChain costs no extra integration work.

Not for

Teams that have not shipped yet. Before production traffic there are no traces, and an eval platform without traces is a spreadsheet with a worse editor — write 20 test cases by hand and run them in a script until real usage exists. Not for teams whose actual need is classical model monitoring on tabular ML: drift, feature skew and PSI are Arize’s and W&B’s territory. And not for an organization whose security review will not accept a third party holding model provider credentials — read the first watch-out before starting that procurement.

Versus the alternatives

• LangSmith — the default if you already build on LangChain or LangGraph, because the tracing is free of integration work there and the deployment products sit adjacent. Pick it when the framework is LangChain and the team is small. Pick Braintrust when the team is framework-agnostic, or when the reviewer headcount makes per-seat billing the dominant cost.
• Langfuse — the open-source option and the fastest-growing entrant in the segment. The core repository is MIT-licensed and self-hosting is free, which is the answer when data residency is non-negotiable or when volume is high enough that any metered vendor loses on arithmetic. Pick Langfuse when you have a platform team willing to operate it. Pick Braintrust when you do not, and specifically when Loop’s investigation loop is the feature you want — that is the clearest capability gap between them.
• Arize Phoenix — open source and OpenTelemetry-native, and the pick for teams already running Arize on traditional ML who want one vendor across both. Weaker fit when LLM-as-judge development is the daily job.
• Weights & Biases Weave — pick it when the same team fine-tunes models and already lives in W&B for experiment tracking.

If none fit, the honest fallback is a checked-in directory of test cases and a CI job scoring them with a function you wrote. That is roughly what Braintrust automates, it costs nothing, and for one feature with 50 test cases it competes.

Watch-outs

• An attacker reached a Braintrust AWS account in May 2026 and org-level AI provider keys were likely exposed. Braintrust detected the activity on 2026-05-04, notified customers on 05-05, disclosed publicly on 05-06, and told everyone to rotate org-level provider keys. One customer was confirmed affected; three more reported unexplained spikes in provider usage. The disclosure was fast and direct, which is the right behaviour — the exposure class is the point. Guard: the self-hosting docs state that project-level provider secrets stay in your data plane while org-level secrets stay in Braintrust’s control plane, so configure keys per project rather than per org, scope them to a spend-capped provider account, and set an expiry — key expiration and a control stopping members from minting new keys both shipped in July 2026. Do this on day one, not after the second incident.
• Two independent meters and a retention clock make the bill hard to forecast. Data processed and scores run bill separately, and every online scorer you enable multiplies the second against production traffic: four LLM judges at full sampling quadruples score volume without one line of application code changing. Guard: start online scoring on a sampled percentage, put the per-project usage breakdown that shipped in September 2026 on a recurring review, and export anything you need past 30 days before retention takes it.
• Loop’s automations hold write permissions. Scheduled runs that create and update objects on their own instructions are convenient, and are also an agent editing your scoring infrastructure unattended. If a scorer changes quietly, every experiment after it is measured on a different ruler. Guard: keep annotated version history on scorers and prompts so each save carries a description, review automation-authored changes at code-review cadence, and pause automations during an incident — pause and resume shipped in August 2026 so configuration survives that.
• Two credible open-source rivals accept the same wire format you are sending. Langfuse and Phoenix are permissively licensed and both ingest OpenTelemetry, so the switching cost is whatever Braintrust-specific SDK calls you wrote. Guard: instrument with vendor-neutral OTel spans wherever the SDK gives you the choice, and test a full trace export during the trial. Portability is real only if you built for it.