AI cost control begins with the product experience

AI cost is often treated as an infrastructure problem: negotiate provider rates, tune serving capacity, then watch a spend dashboard. That starts too late. By the time a request reaches a model, the product has already determined how often it will run, how much context it will carry, how long its answer may be, and whether one task becomes five. Those choices define unit economics before infrastructure tuning can help.

Start with the unit of value

“Cost per request” is rarely the right product metric. A request can be cheap yet useless, while a more expensive call may complete a valuable task. Define the unit users actually care about: a support case resolved, a document reviewed, a report accepted, or a form completed.

Then model the full cost of that unit:

cost per completed task = attempts × model calls per attempt × average call cost

This exposes design problems hidden by token averages. An ambiguous interface may cause users to retry. A conversational flow may ask the model to rediscover facts already captured in structured fields. An agent may perform several tool calls for a task that could have been a single deterministic operation.

Instrument attempts, retries, input and output tokens, cache reads, model selected, tool calls, latency, abandonment, and successful completion. Segment them by task rather than reporting one blended AI total. The goal is not the lowest token count; it is the lowest cost for an acceptable completed outcome.

Design interactions that prevent unnecessary inference

Every open text box invites uncertainty. If the task has known parameters, collect them with controls before calling a model. A date picker, category selector, document scope, or desired output format can replace a clarification turn and improve the first response.

The interface should also distinguish operations that need generation from those that do not. Search, filtering, arithmetic, permissions checks, and record updates usually belong in conventional code. Use a model where interpretation, synthesis, or language variability creates value.

Set useful output boundaries in the interaction itself. “Summarize this contract” leaves length and audience undefined. Asking for a five-point risk summary for a procurement reviewer gives the system a stopping condition. The user receives a more predictable artifact, and the model produces fewer speculative tokens.

Streaming belongs to this design layer too. It does not inherently reduce token charges, but it can improve perceived responsiveness and show progress before a full answer is ready. That may remove pressure to route every request to the fastest premium model. Stream when partial text is useful; for short classifications or structured outputs, a loading state may be clearer.

Treat context as a product budget

Long context windows make it possible to send an entire conversation, document collection, or customer history. They do not make that choice economical or accurate.

Give each task a context policy. Separate permanent instructions, relevant source material, recent interaction state, and the current request. Apply limits to each layer. Retrieve only material that can affect the answer, summarize older conversation turns when appropriate, and discard interface events that have no semantic value.

Context selection should be visible in the experience. Let users choose the files, accounts, date range, or conversation branch in scope. This both reduces tokens and makes the system’s basis easier to understand. When scope is unclear, ask once rather than silently attaching everything available.

The same discipline applies to output. The official OpenAI guidance on latency optimization identifies fewer input tokens, fewer output tokens, fewer requests, and avoiding an LLM when it is unnecessary as distinct performance controls. Those are product decisions as much as engineering techniques.

Build caching around stable product states

Caching works best when the product creates repetition deliberately. Stable instructions, tool definitions, schemas, and shared reference material should precede variable user content. OpenAI’s prompt-caching documentation explains that reuse depends on matching prompt prefixes, so a timestamp or user-specific field inserted near the beginning can defeat an otherwise reusable structure.

There are several cache layers to consider:

• Cache deterministic application data before it enters a prompt.
• Cache retrieval results when the query, permissions, and source version match.
• Reuse stable prompt prefixes through provider-supported prompt caching.
• Cache complete responses only when freshness, personalization, and safety rules permit it.

A cache key must represent the answer’s real dependencies: tenant or user scope, permissions, locale, model and prompt versions, source versions, and any freshness boundary. A cheap incorrect hit is not a saving. Track cache eligibility and hit rate by task, then compare the cost of writes and misses with the savings from reuse.

Route by task risk rather than user tier

Using one flagship model everywhere is easy to implement and hard to sustain. Model selection should follow the difficulty and consequence of the task.

Start with explicit routes. Extraction into a fixed schema, intent classification, and short rewriting can be evaluated on smaller models. Complex synthesis across conflicting evidence may justify a stronger model. High-impact actions need stricter validation and may require human confirmation regardless of model capability.

Escalation should be measurable. Route to a stronger model when required fields are missing, confidence checks fail, sources conflict, or a validator rejects the output. Do not let the model repeatedly retry without a ceiling. Set maximum calls, tool steps, context, output, and elapsed time for every task boundary.

This is the approach reflected in Alfcode’s AI product work: model routing and budgets are paired with caching, streaming, latency control, evaluation, and load testing. The important product lesson is the combination. Routing without evaluations can cut quality; caching without stable prompt structure produces few hits; streaming without concise outputs can merely display an expensive answer sooner. Alfcode also connects this engineering practice to product design, where the flow determines what action the system is being asked to support.

Set an economic acceptance test before launch

Create a small evaluation set for each task using representative inputs and expected outcomes. Run every candidate route against it, recording completion quality, retries, input and output tokens, cache behavior, latency, and total model calls. Test common cases separately from difficult and high-risk cases; averages can conceal an expensive tail.

Before shipping an AI interaction, apply one test: can you state the completed user outcome, maximum calls, context limit, output limit, cache boundary, default model, and escalation rule on a single page? If any field is missing, the product has not yet defined its unit economics.