A habit I’ve been building as more of our pipeline started running LLM agents, and I’m curious how people here handle it.
Build time, bundle size, test coverage — we gate all of those in CI and fail the PR when they regress. But the token/compute cost of the agent steps themselves? For a long time we only saw that on the monthly bill, well after the fact. A refactor that quietly ~5x’d the tokens a review-agent burned per PR was completely invisible until someone squinted at the invoice.
A few tool-agnostic habits that have helped:
-
Treat token cost as a CI metric, not an ops afterthought. The same way you’d fail a build that balloons bundle size, you can flag a PR whose agent step jumps in tokens/cost versus the base branch. Making it a check turns a surprise bill into a surprise review comment — much cheaper to act on.
-
Attribute cost to a diff, not a month. “This PR adds 40k tokens/run to the review agent because it dumped a 900-line fixture into context” is actionable; “we spent $X in July” is not.
-
Budget before merge, not after deploy. A cheap pre-run estimate over just the changed files catches the 10x cases before they land.
The CI-gate part is the one I still wire up by hand every time. For folks running agents in CI or pipelines: do you gate the token/dollar cost of an agent step the way you’d gate build time or bundle size — and if so, how do you wire it up? Or is it still a monthly-invoice surprise for everyone else too?
If you’re trying to shortcut the dev process by running an llm agent for reviewing PRs, you deserve every cost overrun you get. Letting an llm review anything written by an llm is one of the dumbest things I can imagine. If one of my devs suggested that to me, they’d have their access removed. I have zero sympathy for cost overrun, the litany of production incidents, or the amount of unmaintainable code that will inevitability come from such a decision.