The model-routing rubric

I run a fleet of AI agents every day. The most expensive mistake I made early on had nothing to do with prompts or context windows. It was treating model choice as a technical preference when it’s actually a purchasing decision.

The frame that fixed it is simple. Every task that flows through my stack gets scored on three axes - cost, intelligence, and taste - and routed to the cheapest model that clears the bar. That’s the whole system. The discipline is in actually doing it.

The three axes

Cost means whatever is scarce for you - a subscription quota, an API budget, the hours in your own day. The sticker price per token barely matters. My frontier model is the most rationed thing I own, so a task it does that a cheaper model could have done isn’t a convenience. It’s a loss.

Intelligence is the axis everyone already optimizes, so I’ll only say this: most work doesn’t need the smartest model. A clear spec plus a mid-tier model beats a vague spec plus a frontier model, and it isn’t close.

Taste took me longest to price. Some work is mechanical - rename these files, write this migration, sweep these logs. Some work is judgment - what should this look like, what’s worth building at all, what would embarrass me if it shipped. Models differ more on taste than on intelligence, and taste is what you’re actually buying when you pay frontier prices. Once I scored it as its own axis, the routing table wrote itself.

The actual scorecard

So this isn’t theory, here’s the table I route with today. Cost is scored against my real quotas, not list price.

They get re-scored every time a new model ships.

Routing in practice

A receipt from this week. A bulk model triaged 135 files from my knowledge base. A frontier model read only the 60 it flagged as unsure. 39 reached my desk. Every layer read less and cost more.

Intentions drift, so I run tripwires instead. If I catch a model grinding through more than five similar files, or past 150 lines of work someone already specced, that batch gets stopped and delegated down, then the diff gets reviewed. And I track one number in the other direction: what fraction of delegated work survives review. Below 80%, the routing is miscalibrated - the spec was too thin or the tier was too low - and the rubric gets tightened, not the delegation abandoned.

Where the scores come from

The seed of this system was a question I asked myself while building eval infrastructure: if I’m already running the same tasks across models to catch regressions, why am I still routing by folklore? Evals aren’t just a quality gate. Run them across models and you accumulate a scoreboard of who wins which class of task. That scoreboard is the routing table. The question stops being which model do I like and becomes which model earns this task.

Why I call it a P&L decision

Every company already knows this logic under a different name: make or buy. You don’t put your most expensive hire on data entry. They’d do it fine - their hours are just the constraint the whole business is planned around.

Model routing is the same decision at a smaller grain, made hundreds of times a day. Judgment is the scarce input. Spend it where it compounds, buy everything else at the market rate, and audit the results like you’d audit any vendor.

The saved money is real, but it was never the point. The expensive model finally has room to do the only work that was ever worth paying it for.

⁂