deskthebuildAI COST
← All notes · 5 min read

Our OpenAI bill is $50,000/month. Where would you start?

Treat the $50,000 monthly bill as a hypothetical system-design constraint, not a benchmark. The amount matters less than the question behind it: can the team connect model spend to useful product outcomes and reduce waste without quietly reducing answer quality?

Why the common answer is incomplete

Most candidates start with: “Use a cheaper model.”

That can be useful, but it treats the invoice as if every token produces equal value. A cheaper model may reduce the unit price while increasing retries, support escalations, or task failures. It may also leave the largest cost drivers untouched: oversized context, duplicated requests, uncontrolled agent loops, and features that call a model even when deterministic code would work.

The first job is not model selection. It is building a cost model that explains which product behavior creates the spend and what the business receives in return.

A production approach

1. Measure cost per successful task

Instrument every model call with the product feature, customer tier, model, prompt version, input tokens, output tokens, latency, cache status, retry count, and final task outcome. Aggregate cost per successful task rather than cost per request. A request that costs two cents and fails twice may be more expensive than a ten-cent request that completes reliably on the first attempt.

2. Find the dominant cost path

Break monthly spend down by feature and request stage. One workflow often dominates because it has a large context window, high traffic, or an agent that makes several dependent calls. Use a Pareto view to identify the first controllable source. Optimizing a small autocomplete feature is irrelevant if document analysis or background summarization creates most of the bill.

3. Remove tokens that do not change the decision

Inspect real prompts and traces. Repeated policy text, full conversation histories, duplicate retrieved passages, verbose tool output, and unbounded documents all consume input tokens. Replace raw history with a bounded summary, deduplicate retrieval results, select only fields required for the current step, and enforce explicit context budgets. Re-evaluate quality after every reduction because some “extra” context may carry essential constraints.

4. Route work by difficulty

Use the least expensive path that meets the task’s quality requirement. Deterministic validation, formatting, and database lookups should not require an LLM. Simple classification may use a smaller model, while ambiguous planning or high-risk responses may justify a stronger one. A routing policy needs evaluation data and fallback rules; otherwise it becomes an undocumented collection of guesses.

5. Cache stable work at the correct layer

Cache results when equivalent inputs should produce reusable outputs. Exact prompt caching can help repeated system prefixes. Application caching can reuse document summaries, embeddings, or normalized tool results. The cache key must include every input that changes correctness, including prompt and model versions. Set retention and invalidation from the underlying data’s volatility and privacy requirements, not from an arbitrary time-to-live.

6. Control retries and agent loops

Retries are necessary for transient failures, but they can multiply spend during an outage. Retry only errors likely to recover, apply exponential backoff with jitter, and enforce a per-task attempt budget. Agent workflows need maximum steps, token ceilings, tool-call allowlists, and a terminal fallback. Store idempotency keys for side-effecting tools so a repeated model call cannot repeat a payment or external mutation.

7. Move suitable work off the interactive path

Batch background summarization, evaluation, and enrichment when users do not need the answer immediately. Batching may improve provider utilization or pricing, but it adds queue delay and operational complexity. Use it for workloads with clear deadlines and durable job state. Do not move a user-facing dependency to a queue merely to make the synchronous trace look cheaper.

8. Create a quality-and-cost release gate

For every optimization, compare task success, groundedness, latency, user escalation rate, and cost on the same representative dataset. Roll changes out gradually and segment results by workflow. A global average can hide a severe regression for long documents, non-English inputs, or a customer tier with different context. Optimization is complete only when the product outcome remains within its agreed quality boundary.

Engineering trade-offs

Aggressive context reduction lowers cost and often latency, but it can remove information needed for a correct answer. Smaller models improve unit economics but may require more routing logic and fallbacks. Caching saves repeated work but introduces staleness, invalidation, and privacy concerns. Batching reduces interactive pressure but increases time-to-result. Model consolidation simplifies operations, while model routing can reduce spend at the cost of more evaluations and deployment paths.

The right architecture depends on traffic shape, failure cost, data sensitivity, and the quality threshold of each feature. A low-risk drafting assistant can accept different trade-offs from a system that generates financial or medical decisions.

Common mistakes

  • Optimizing token price before measuring cost by feature and outcome.
  • Cutting the context window globally instead of examining which tokens are useful.
  • Caching responses without prompt-version, authorization, or tenant boundaries in the key.
  • Retrying validation errors and policy rejections that will not recover.
  • Allowing agents to continue until the model decides it is finished.
  • Declaring savings from an offline benchmark while production escalation rates increase.
  • Ignoring background jobs because they are not visible in request latency dashboards.

Failure handling and observability

Set budgets at request, workflow, tenant, and daily-system levels. Emit a structured event for each model call and link it to a trace covering retrieval, tools, retries, and outcome evaluation. Alert on cost per successful task, unexpected model mix, token growth, cache-hit collapse, retry amplification, and agent-step distribution. When a budget is exhausted, degrade intentionally: use a smaller model for low-risk work, postpone background enrichment, or return a transparent partial result. Do not fail silently or let an outage create unlimited retries.

Keep billing reconciliation separate from request telemetry. Provider invoices are the source of truth for spend, while traces explain behavior. Regularly compare them so missing instrumentation does not create imaginary savings.

Interview insight

The interviewer is not testing whether you know a cheaper model name. They are testing whether you can connect system behavior, product quality, and operational cost. A staff-level answer starts with measurement, finds the dominant path, applies the smallest safe optimization, and proves the outcome with production evaluations.