AI systems 0/10

Lesson 5 of 10

Cost and latency for AI features

Price a task by input, cache-write, cache-read and output tokens, then cut cost and latency via prompt caching, streaming, smaller models and routing.

1,945 words · 9 min read Ultra Premium Useful background: How an LLM call works, LLM serving: batching and the KV cache

In this lesson

  1. Budget the request
  2. Count the full task
  3. How prompt caching works
  4. Reasoning and batch pricing
  5. A latency equation
  6. Adjust the budget
  7. A cost ledger
  8. What stays true
  9. Costs and trade-offs
  10. Common mistakes
  11. Related topics and practice

Ultra Premium

Unlock this lesson

Ultra Premium opens the 10 premium lessons in AI systems, along with the rest of WhiteBox's premium lessons and problems.

Not ready yet? Read the free lesson: Backpressure