Build or adopt for AI systems is the decision between adopting an existing platform, gateway or model service and building your own. It comes up for customer-facing agents, internal assistants, retrieval systems and the infrastructure in front of model traffic. Most teams settle it with a feature comparison, a demo or an engineering preference. The essays here treat it as a measurement question.
The core method has six parts:
- Build the gold set first. Collect blind-labelled items from your own work, split into the slices where a mistake does damage, and split by time into dev and test sets.
- Measure the simplest trusted option first. Start with what your organisation already runs, with identity, data handling and ownership already settled. Score it per slice with ranges and pinned versions.
- Compare candidates with a paired test. Run every candidate on the same items, count the disagreements and read every case a challenger broke.
- Build only on a named trigger. Three triggers count: a slice that keeps failing after every supported configuration change, a policy or residency rule the option cannot enforce, or a capability you need to own long term.
- Keep any build thin. Own identity, policy enforcement, logging and evals, and reuse the commodity parts.
- Plan for switching providers from day one. Use one internal schema, thin adapters for each provider and contract tests for each adapter. Re-run the decision when vendors or models change.
The numbers that make this work are small and checkable. To show a slice is at least 95% correct with no failures observed, you need 73 clean items. You need 189 for 98% and 381 for 99%. Report ranges instead of single points, and read a paired comparison case by case. Weight overall scores by traffic and harm, never by a plain average.
Start here
- Your answer key is the product: why the gold set is the asset that survives any vendor decision.
- Counting honestly: ranges, zero failures, paired changes: the arithmetic behind per-slice bars and paired comparisons.
- A lucky week is not a better model: how chance produces false winners in a comparison.
- Make model failover preserve the agent contract: why a switching plan needs contracts, not just a second provider.
Questions this topic answers
- Should we adopt an agent platform or build our own?
- What evidence actually justifies a build?
- How do we compare two candidates fairly on our own data?
- How small can a build be and still do its job?
- How do we avoid lock-in and keep the decision current when models change?
If you are working through a build or adopt decision for AI systems, connect with me on LinkedIn to compare notes and discuss.