The easiest AI budget is also the one most likely to be wrong: requests times input tokens times the model price. That misses the real system. A product call may include a long system prompt, retrieved documents, tool schemas, output tokens, retries, safety checks, logs, storage, and sometimes a human who reviews the answer before it reaches a customer.
For a small team, the right artifact is not a spreadsheet full of guesses. It is a cost ledger with named lines, measured from real traces, updated whenever a model, price page, retention term, or workflow changes.
The ledger
| Line item | How to estimate it | What teams often miss |
|---|---|---|
| Online API calls | requests x (input tokens x input rate + output tokens x output rate + reasoning/cache/tool/search/retry costs) | Output tokens and reasoning or thinking tokens can dominate chatty workflows. |
| Context | static prompt + conversation + tools + retrieved documents | Large context is billed even when only a small part is useful. |
| Caching | cache-write tokens, cache-read tokens, hit rate, TTL, and prompt-prefix stability | A small change near the top of the prompt can destroy the hit rate. |
| Batch jobs | offline requests that can wait up to the provider's batch window | Batch is unsuitable when a user is waiting, but strong for evals, enrichment, classification, and embeddings. |
| RAG | embedding/indexing + vector search + retrieved-context tokens + rerank if used | The model still pays for the text you put back into context. |
| Vector database | stored vectors, dimensions, query volume, metadata, backups, region, and minimum plan | Some providers price by dimensions, some by plan minimums, some by hosted file-search calls. |
| Evaluation | test set size x model candidates x judge model x human labels | Evals are recurring product cost, not a one-time launch chore. |
| Human review | review minutes x loaded hourly cost x escalation rate | The expensive part may be handling uncertain, high-risk, or customer-visible outputs. |
| Logs and observability | token logs, traces, error events, retention period, query volume | Verbose traces help debugging but can create storage, privacy, and redaction costs. |
| Infrastructure | gateway, queue, workers, database, object storage, secrets, monitoring, and failover | The model bill may be smaller than the product plumbing around it. |
A simple formula
Start with this monthly estimate: online inference + offline batch + embeddings + retrieval/storage + evals + human review + logs/infra + 20% variance buffer. The buffer is not pessimism. It covers retries, prompt changes, traffic spikes, provider routing changes, and newly discovered edge cases.
What changed by July 2026
- OpenAI, Anthropic, and Gemini all expose batch processing as a formal path for non-urgent work. The discount, expiry window, and eligible endpoints must be checked against the current provider docs before launch.
- Prompt or context caching is now a budget lever, not an implementation detail. The provider-specific rules matter because cache writes, reads, TTL, and prompt-prefix stability shape the bill.
- Hosted retrieval tools are lowering setup effort but changing the bill shape. Storage, tool calls, indexing embeddings, and retrieved document tokens may be charged on different lines.
- Vector storage is no longer one pricing model. Cloudflare Vectorize, Pinecone, and database-backed pgvector designs move cost between provider usage, plan minimums, and operations work.
- Evaluation tooling moves over time, so the durable asset is the dataset, rubric, and traces, not a specific dashboard.
Route by model capability
| Workload | Model lane | Cost rule |
|---|---|---|
| Routing, tagging, extraction, short classification | Low-cost fast model | Keep prompts short, require structured output, batch when possible. |
| Customer-facing RAG answer | Standard model with citations and retrieval tests | Spend on source quality and evals before upgrading the model. |
| Code review, complex policy analysis, high-value synthesis | Frontier reasoning model | Use only when the task value justifies latency, output cost, and review time. |
| Tool-using agent | Model plus product controls | Budget for retries, permissions, audit logs, and human approval on irreversible actions. |
RAG costs are not just the vector database
A RAG system starts spending before the first user asks a question: cleaning documents, chunking, embedding, indexing, metadata design, and deletion workflows. At query time, it spends again on embedding or search, vector reads, possible reranking, and the retrieved text sent back to the model. If the answer must cite sources, you also need evals for recall, precision, citation accuracy, and conflict handling.
The cheapest RAG design is often the one that retrieves less but retrieves better. A smaller top-k, cleaner chunking, metadata filters, and a refusal path for missing evidence can reduce both hallucination risk and context cost.
Budget confidence score
| Score | Meaning |
|---|---|
| 0 | No token logs, no estimate, no owner. |
| 1 | A spreadsheet based on public prices, but no real traces. |
| 2 | Real token usage is logged for a small pilot; output tokens and retries are visible. |
| 3 | Caching, batch, RAG, vector storage, and human review are separate ledger lines. |
| 4 | The team has task-specific evals, failure categories, and per-workflow unit economics. |
| 5 | Costs are monitored in production with alerts, margins, provider fallback, and a monthly recheck of dynamic prices and terms. |
What to do before launch
- Capture real traces for at least one representative week or a deliberately varied pilot set.
- Separate synchronous user-facing calls from batchable offline work.
- Measure cache hit rate instead of assuming a vendor's best-case saving applies.
- Put RAG retrieval quality in the same review as model quality.
- Write a human-review policy for low-confidence, regulated, or irreversible outputs.
- Set spend limits, anomaly alerts, log retention, and deletion workflows before inviting real users.
- Keep eval data, prompts, and provider adapters portable so the team can rerun the comparison when prices or terms change.