AI bills grow quietly
Most AI features launch on the largest model available with generous prompts, because that’s the fastest way to get something working. Then usage grows, and so does the invoice — often faster than revenue. Optimising later is entirely possible, but only if you can see where the money goes and can prove that cheaper doesn’t mean worse.
Measure first
We start by instrumenting every request: which feature, which model, how many input and output tokens, how long it took and whether the outcome was successful. The breakdown usually surprises people — a background job nobody watches, a retry loop, a feature that stuffs whole documents into every prompt. Alongside cost, we build a quality baseline, so every optimisation can be evaluated rather than hoped about.
The levers
- Routing. Send simple requests to small, cheap models and reserve frontier models for the hard ones, escalating automatically when the small model is unsure.
- Caching. Use provider prompt caching for repeated system prompts and context, and semantic caching for frequently repeated questions.
- Context discipline. Retrieve fewer, better passages; trim boilerplate; ask for structured, concise outputs.
- Batching. Move non-urgent work to batch APIs, which providers typically price lower.
- Fine-tuned small models. For a high-volume narrow task, a fine-tuned open model can replace an expensive general one.
- Right-sized infrastructure. For self-hosted models: the right GPU, quantisation, continuous batching and autoscaling to avoid paying for idle hardware.
Savings that stick
We make changes one at a time, each with a measured effect on cost and quality, and finish with budgets, per-feature alerts and a simple monthly review so spend stays under control as usage grows. We do this for our own products too: OptyCV runs its LLM pipeline on low-cost and free-tier models with fallbacks, which forced us to get good at it.