Why is running premium models in tests a cost defect?

Because test volume is decoupled from product value: CI runs the suite on every commit, every branch, every retry, so a frontier-model call in a test is billed hundreds of times a week to verify plumbing — that a response parses, that a field exists — which a far cheaper model or a recorded response would verify identically.

How common are premium models in agent test suites?

In Capsera's July 2026 scan of 133 public Python agent repositories, 5 of the 51 making direct LLM calls — about one in ten — had opus-, gpt-4o- or o1-class model literals inside test files, 28 instances in total. One of them was an LLM observability company, which is the strongest evidence that the pattern is hard to see rather than careless.

Should tests call LLM APIs at all?

Most shouldn't. Structural tests — parsing, retries, error handling — should run against recorded or mocked responses that cost nothing and never flake. Reserve live calls for a small, deliberately scheduled evaluation suite, run it on the cheapest model that exercises the behaviour, and keep frontier models for evaluations whose subject is the frontier model.

Cost defect

Premium models in your test suite

Tests run on every commit, every branch, and every flaky retry — and a test asserting that a function returns a string does not need a frontier model to do it. An opus- or gpt-4o-class literal inside a test file turns your CI cadence into a billing cadence: the spend scales with how often engineers push, which has nothing to do with what the test verifies.

Why it happens

Almost never by decision. A test gets written by copying the nearest working call site — which uses the production model — and the literal comes along. It passes, it’s green, nobody looks again. Unlike production model choice, which gets debated, test model choice is invisible: no dashboard breaks spend down by “came from CI”, so the cost lands in the same bucket as everything else on the key.

Broken — frontier model verifying plumbing

def test_summarise_returns_string():
    # A frontier model, billed on every commit, every branch, every
    # flaky retry — to check that the function returns a string.
    result = summarise(SAMPLE_DOC, model="claude-opus-4-6")
    assert isinstance(result, str)

How common it is

In our July 2026 scan of 133 public agent repositories, 5 of the 51 LLM-active repos — one in ten — had premium model literals in test files, 28 instances in total. The most instructive case was an LLM observability company with eleven of them in its own suite. We read that as evidence about visibility, not diligence: if the companies building cost tooling carry this pattern, the problem is that nothing surfaces it, and that is worth fixing with tooling rather than blame.

How to tell if it's happening to you

Statically: grep your test directories for model literals — grep -rn "opus\|gpt-4o\|o1-" tests/ finds most of it in seconds, and it is exactly the predicate the benchmark scanner runs. At runtime: attribute spend by environment, so CI traffic carries its own identity instead of blending into the production key’s total — with per-agent attribution, a tests environment tag makes the number visible on a dashboard instead of in an archaeology session.

How to fix it

Route model choice through one switch, default it cheap, and take the network out of structural tests entirely:

Fixed — one switch, cheap default, recordings for structure

# conftest.py — one switch for the whole suite
TEST_MODEL = os.environ.get("TEST_MODEL", "claude-haiku-4-5")

def test_summarise_returns_string():
    result = summarise(SAMPLE_DOC, model=TEST_MODEL)
    assert isinstance(result, str)

# Better still for structural tests: no network at all.
def test_summarise_parses_response(recorded_response):
    result = parse_summary(recorded_response)
    assert result.title

The division of labour that works: structural tests (parsing, retries, error paths) run against recorded responses — free, fast, and deterministic. Behavioural checks run live on the cheapest model that exercises the code path. And genuine quality evaluation — the only place a frontier model belongs — runs as a scheduled evaluation suite with its own budget, not as a side effect of every push. This is how Capsera’s own SDK evaluation pipeline works: the provider is mocked, so the suite spends nothing and never flakes on a provider outage.

Questions this page answers

Why is running premium models in tests a cost defect?: Because test volume is decoupled from product value: CI runs the suite on every commit, every branch, every retry, so a frontier-model call in a test is billed hundreds of times a week to verify plumbing — that a response parses, that a field exists — which a far cheaper model or a recorded response would verify identically.
How common are premium models in agent test suites?: In Capsera's July 2026 scan of 133 public Python agent repositories, 5 of the 51 making direct LLM calls — about one in ten — had opus-, gpt-4o- or o1-class model literals inside test files, 28 instances in total. One of them was an LLM observability company, which is the strongest evidence that the pattern is hard to see rather than careless.
Should tests call LLM APIs at all?: Most shouldn't. Structural tests — parsing, retries, error handling — should run against recorded or mocked responses that cost nothing and never flake. Reserve live calls for a small, deliberately scheduled evaluation suite, run it on the cheapest model that exercises the behaviour, and keep frontier models for evaluations whose subject is the frontier model.