Optimizing LLM Token Costs in Production: A Practical Engineering Playbook [Part 3]

Author(s): Garvit Agarwal Originally published on Towards AI. Optimizing LLM Token Costs in Production: A Practical Engineering Playbook [Part 3] In Part 2, we focused on optimizing how requests are constructed before they reach the language model. We explored how techniques like Model Routing, Prompt Caching, and Conversation Summarization reduce unnecessary token usage without affecting the user experience.Those optimizations alone can significantly reduce production costs. But here’s something that surprised me when I started studying production AI systems. Many applications continue to spend thousands — or even millions — of unnecessary tokens after the request has already been optimized. How? Because they still: Retrieve far more context than the model actually needs. Process requests one at a time instead of efficiently batching them. Generate responses that are much longer than users require. None of these problems originate from the language model itself. They’re engineering decisions. Three optimization techniques working together to build an efficient AI pipeline. And just like the techniques we discussed in Part 2, they can often be improved without changing models or sacrificing response quality.Let’s look at three more production optimization techniques that help AI systems become faster, cheaper, and more scalable. Lever 4- Adaptive Retrieval Retrieval-Augmented Generation (RAG) has become one of the most common architectures for production AI applications. Instead of relying solely on the model’s training data, a RAG pipeline retrieves relevant information from an external knowledge base before generating a response. The idea is simple: Give the model the right context so it can produce a more accurate answer. The challenge is deciding how much context to retrieve. The Hidden Cost of Fixed Retrieval Imagine you’re building an internal company chatbot.A user asks: “What are your office hours?”Your vector database retrieves 10 documents because the retrieval pipeline is configured with: documents = vectorstore.similarity_search(query, k=10) Those documents might include: Employee handbook, HR policy, Security guidelines, Travel policy and so on. The answer only needs one sentence. Yet thousands of tokens are sent to the language model. Now imagine this happens for 50,000 requests every day.Most of those retrieved tokens contribute nothing to the final answer — but you’re still paying for them. One Size Doesn’t Fit Every Query Not every question deserves the same amount of context.Compare these two requests. Query 1: “What are your office hours?”A couple of relevant documents are enough. Now consider, Query 2: “Compare our healthcare reimbursement policy with last year’s finance guidelines.”This question requires multiple documents from different sources. Both queries are important. But they shouldn’t retrieve the same amount of information. Making Retrieval AdaptiveInstead of treating every query equally, production systems first estimate its complexity. Simple factual questions retrieve a small amount of context. Broader analytical questions retrieve more. A simplified implementation looks like this: if query_type == "simple": top_k = 2elif query_type == "medium": top_k = 5else: top_k = 10documents = vectorstore.similarity_search(query, k=top_k)p The logic isn’t complicated. But over millions of requests, this small engineering decision can eliminate a huge amount of unnecessary token processing. Reranking: Quality Matters More Than Quantity Retrieving more documents doesn’t necessarily improve answer quality. Production systems often perform a second filtering step called reranking. Instead of passing every retrieved document to the model:Retriever — > Top 10 Documents — > Reranker — > Top 3 Relevant Documents — >LLM The reranker scores each document according to its relevance and forwards only the best matches. This has two advantages:First, the model processes fewer tokens.Second, it receives higher-quality context. In many cases, fewer documents actually produce better responses because the model isn’t distracted by irrelevant information. Adaptive retrieval sends only the most relevant context to the LLM. Production Insight Many teams spend weeks experimenting with better embedding models.Sometimes the biggest improvement comes from something much simpler:Stop sending documents the model doesn’t need.A smaller, cleaner context often improves both accuracy and cost efficiency. Lever 5- Batch Inference So far, we’ve optimized what reaches the language model. But optimization isn’t just about reducing tokens. It’s also about how efficiently requests are processed.This becomes particularly important when your AI application isn’t serving a single user — it might be processing thousands of documents, emails, product descriptions, or customer reviews every hour. At this scale, sending one request at a time can become surprisingly expensive. The Hidden Cost of Sequential Processing Imagine you’re building a document search system. Before users can search your documents, each one needs to be converted into an embedding and stored in a vector database.Suppose you have 1,000 PDF documents waiting to be indexed. A straightforward implementation might look like this: for document in documents: embedding = embedding_model.embed(document) vector_db.insert(embedding) It works. But behind the scenes, you’re making 1,000 separate API calls. Each request carries its own: Network latency Authentication overhead Request initialization Response processing The model spends almost as much time handling requests as it does generating embeddings. A Better Approach Instead of sending documents individually, production systems process them in batches. batch_size = 100for i in range(0, len(documents), batch_size): batch = documents[i:i + batch_size] embeddings = embedding_model.embed(batch) vector_db.insert(embeddings) Now, instead of making 1,000 API calls, you’re making only 10. The number of tokens remains almost the same, but the infrastructure becomes far more efficient. Why Batching Improves Performance Think of ordering coffee for your team.Would you rather: Walk to the café twenty times and order one coffee each trip? or Collect everyone’s order and make a single visit? Both approaches produce the same result. One simply wastes much less time. Batch inference works in exactly the same way. Instead of repeatedly setting up new requests, the system processes multiple inputs together, reducing overhead and improving throughput. Where Batch Inference Works Best Batching is most effective for workloads that don’t require an immediate response. Some common examples include: Generating embeddings for large document collections Indexing knowledge bases Classifying customer feedback Offline summarization Processing support tickets Content moderation These are background jobs where processing speed matters more than instant user interaction. For real-time chatbots, however, batching is often less suitable because users expect responses […]