An LLM Feature Is a Margin Decision Before It Is a Model Decision
Forty support conversations an hour, each opening with a 90,000-token context, is 3.6 million input tokens before the first clock hour closes, and the counter runs whether or not one customer has read a useful answer. Change the arithmetic and the conclusion stands: input context is paid per request, repeated content is most of it, and the feature that was free on a laptop starts running on a meter the day customers arrive. Which model answered is downstream of the decisions that meter imposes.
So the planning question is not model choice first. It is unit cost first, and unit cost is a product decision, made before launch and written into the price. We build LLM-backed features for data products, and the features that survived their first quarter in production were the ones whose token bill was modeled and assigned an owner before the feature had users. The ones that surprised somebody are the ones you read about the following quarter, in a retitling like "experimental", which is the polite word for unpriced.
Count the tokens, not the dollars
Prices move monthly, by model, and any dollar figure in this post would be stale by the time you read it, so the durable exercise counts tokens and keeps the shape. Every request spends its budget in four places: the shared instructions copied into every call, the pages of retrieved context, the conversation history so far, and the output the model writes back. The proportions differ, but the anatomy does not, and one line of it explains most of the pain: the shared instructions cost the same on the thousandth request as on the first, and they are paid every single time.
A conversation-shaped product then multiplies the whole anatomy by turns, because each turn re-sends the history of every earlier turn. The feature that felt like one question is, on the bill, a question re-asked twenty times with an ever-growing suitcase of context behind it:
def input_tokens(request: dict, history: list[dict]) -> int:
"""Input-token bill for one request, in the three places it is spent."""
return (
len(SHARED_INSTRUCTIONS) # same every request
+ sum(p.tokens for p in retrieved(request)) # fresh per request
+ sum(m.tokens for m in history) # grows with every turn
)
def daily_bill(traffic, model_rate, cached_rate=None):
tokens = sum(input_tokens(r, h) for r, h in traffic)
if cached_rate: # if the prefix repeats
repeated = tokens_repeated(traffic)
tokens -= repeated * (1 - cached_rate / model_rate)
return tokens * model_rate
Put a plausible shape on the opening arithmetic and the model shows where the money is. In an ordinary support conversation, the shared instructions tower over the parts that actually change, which is precisely the part a cache knows how to shorten.
The cache eats the repeated half
Prompt caching is published, documented behavior, not a trick: the provider stores a prefix of your context on its side and prices a read of it far below a fresh read. Two facts you can check on any of these vendors' current price sheets: Anthropic prices a prompt-cache read at a fraction of the base input rate, a tenth by default on most models, and batch processing halves input prices across the major API houses. On a bill where the shared instructions are most of the input, a tenth of the rate on most of the input is not an optimization. It is the difference between a feature and a resignation letter.
The honest caveats, all three in the docs and all three load-bearing. The prefix must be byte-identical to the stored one, so a timestamp in line one of the system prompt quietly ruins the whole discount; a dynamic greeting costs more than it sounds. The stored prefix expires, on the order of minutes to an hour depending on the vendor and the option you pay for, so traffic arriving in steady streams is a better cache customer than a trickle of one-off requests. And a cache does nothing for the retrieved pages and the growing history, which means the discount floor is set by how much of the bill is repetition:
A cache pays the bill for what you repeat. Retrieval and history are still fresh every time, and they are where the remaining optimization lives.
Route by difficulty, not by habit
The second decision is per-call: which tier of model answers this request. The constraint that decides it is that most of the turns are mechanical. Classifying an incoming question, extracting a field, deciding whether the hard model is needed at all: those read-and-sort calls are the majority of traffic, they are cheap by design, and sending them to whatever frontier model the team is proud of is a margin leak with no visible benefit. In the ordinary pattern we land on, a small model handles the mechanical majority and hands off the hard remainder, and the hard remainder is where the product's reputation is actually made.
The dead end on this road is truncation. Our first instinct, and one we have watched teams repeat, was to shrink the retrieved pages and cap the history hard, which cut the bill and, within days, started producing answers the queue could no longer trust: summaries citing pages the model never saw. The context was not padding. It was the evidence. That experiment was reverted, and the saving came instead from the tier routing above, which cut the mechanical majority off the expensive path without touching the evidence the hard answers rest on.
Cut tokens only where an eval lets you
Real token cutting exists, and it is worth doing in its place, which is in history and in retrieval, after eval, never in the shared instructions' clarity for the model's benefit. Compacting conversation history by summarizing turns the model no longer needs verbatim is a real saving, and so is retrieving fewer, better-chosen pages. Both change what the model sees, and what the model sees is what answers, so each cut runs against a failing-case review before it merges, in the way we describe in a prompt change is a code review. A token saved by making the feature wrong is not a token saved. It is a token the customer pays for twice, once in money and once in trust.
Price it in, or the quarter will
None of the above is finished until the product says who pays. Three shapes exist, and choosing is not an accounting exercise. Per-use metering passes cost through and puts a ceiling on your worst customer, the one who treats the feature as an oracle; a monthly cap or credit gives predictable spending and quietly subsidizes heavy use; a fixed tier prices the average and needs the headroom modeled rather than hoped. Which shape fits depends on what the feature is for, and the decision belongs in the launch plan, not in the quarter when finance notices. Each shape is defensible on its own. What fails is launching without choosing one. The owner matters as much as the shape: a token bill without a name attached grows until somebody with a spreadsheet finds it.
The takeaway, compressed to a sentence you can put in a plan: model the token bill before the feature has users, cache the prefix before anything else, route the mechanical majority to the cheap tier, and give the bill an owner and a price on day one. An LLM feature that does that is a product. One that does not is a demo with an invoice.
AngoraTek prices LLM features before they ship. If yours is about to meet its first production quarter, walk the bill with us and we will model it together.Thanks for reading this far. If this sounds like the problem you're wrestling with, tell us about it. The team reads every message.
Start a conversation