Marain's pitch is that you should run AI public where it makes sense and private where it matters. On its own that is an assertion, and any consultancy could make it. So we built a bake-off harness that runs the same business tasks through three model configurations and scores what happens.
What it runs
The harness is a single n8n workflow (bakeoff-harness, internal, with no public webhook). It loads a fixed 10-case evaluation set and runs every case through three legs:
- Public — every case goes to the hosted Anthropic API (
claude-sonnet-5). This is the baseline for always using the frontier model. - Private — every case goes to a 7B model (
qwen2.5:7b-instruct) running on our own Mac Studio, reached over Tailscale. Marginal cost is zero because the hardware is already paid for. - Hybrid — the same deterministic, local classifier our
model-routerworkflow runs in production decides per case. It matches emails, phone numbers, IBANs, card and account numbers, SSNs, and a capitalized name next to a date, and routes those cases private; everything else goes public. The routing decision costs no model call of its own, and each case reuses whichever of the two calls above already ran.
The eval set is 10 short tasks: classification, extraction, one-line summarization, and question-answering over a supplied paragraph. Four of them plant fake PII (a name with email and SSN, a name with phone number, a bank account number, a name with date of birth) to exercise the router and the leakage reporting. Scoring is deterministic — exact or fuzzy string match against a gold answer, or the fraction of required keywords present. No model judges its own output.
What the numbers said
We ran it against the live instance on 2026-08-10. Execution 468, run from the CLI, status success, 40.5 seconds of wall clock for 20 real HTTP calls: 10 to api.anthropic.com and 10 to the Mac Studio's Ollama.
| Leg | Accuracy | Avg latency | Total cost | Cases leaked | |---|---|---|---|---| | Public | 100% (10/10) | 2730ms | $0.0057 | 5/10 | | Private | 100% (10/10) | 1137ms | $0.0000 | 0/10 | | Hybrid | 100% (10/10) | 1764ms | $0.0037 | 0/10 |
All three legs scored 10 out of 10. The 7B model matched Sonnet 5 on every task in this set, which is a statement about these particular tasks — short, well-scoped extraction and classification — and not a general claim about model capability. The classifier flagged five of the ten cases as sensitive and routed them private in the hybrid leg: the four planted PII cases, plus one false positive described below.
Private was faster on average than public, 1137ms against 2730ms. The likely explanation is that a Tailscale round trip to our own hardware beats a round trip to Anthropic's API plus its inference time, for prompts this short. That is one measured run on one task mix, not evidence that self-hosted inference is generally faster; the answer depends on model size, prompt length, and network path. What it does show is that private deployment is not automatically the slow option, which is the assumption we most often have to argue clients out of.
Cost behaves the same way. Private inference on owned hardware costs nothing per call, and the public leg cost about half a cent for all ten cases. Both figures are small in isolation. What matters is that the tradeoff between what leaves the building and what it costs can be measured per workload, rather than settled once as a company-wide policy.
The false positive
The classifier's name_plus_dob heuristic looks for a capitalized "First Last" pattern next to a date-shaped token. It fired on a plain invoice ("Bill to: Acme Corp... Due date: 2026-09-01") because "Acme Corp" matches the name pattern. That case routed to private in the hybrid leg despite carrying no real PII.
We are reporting it rather than quietly editing the eval set around it, because it is the same heuristic our production router runs, with the same known failure mode. The direction of the error is what matters. An over-eager classifier sends an ordinary invoice to the private model when it did not need to; it never sends something sensitive to the public one. The failure worth engineering against is the false negative that leaks a customer's SSN, and a harness that hides its own false positives is not one to trust on false negatives.
What this is for
This harness is what an audit now produces alongside its recommendations. Instead of "route your HR extraction workflow privately" as a recommendation, a client gets a scored table of what that choice costs in accuracy, latency, and dollars on representative tasks, before anyone commits engineering time. Ten cases is a small set, and deliberately so for a demonstration; a real engagement runs against a client's own document types and volumes, because placement is task-dependent rather than a policy set once for the whole business.
The instrument matters more than any single scoreboard it produces. Run it against a different task mix and the numbers will move: the private model may struggle with something more open-ended, or the latency gap may narrow or reverse. Either way the placement decision comes with measurements attached. If you want to see how your own workloads score, that is where the audit starts.