LLM Inference Optimization: 10 Techniques That Cut Costs

Introduction

Large Language Models (LLMs) have transformed how businesses build AI-powered applications—but serving them at scale is brutally expensive. OpenAI reportedly spends millions of dollars per day on compute costs, and organizations running private deployments often face GPU bills that threaten to dwarf their entire engineering budget.

The good news? LLM inference optimization has matured rapidly. Teams are now achieving 2x to 10x throughput improvements while slashing latency by 50–80%, all without sacrificing meaningful output quality. Whether you're running a startup chatbot or a Fortune 500 internal knowledge base, these techniques can dramatically change the economics of your AI stack.

In this post, we'll break down the most impactful inference optimization strategies available today—explaining the "what," "why," and "how" for each, with real benchmarks and production examples.

What Is LLM Inference and Why Does Optimization Matter?

Inference refers to the process of running a trained model to generate outputs (text, embeddings, etc.) in response to user inputs. Unlike training—which happens once—inference happens millions of times per day in production systems.

Inference costs are driven by:

Latency: Time to first token (TTFT) and total response time
Throughput: Number of requests served per second
Memory: How much GPU VRAM the model consumes
Hardware utilization: How efficiently GPUs are actually being used

Optimizing inference means squeezing better performance out of the same hardware—or achieving the same performance on cheaper hardware. Given that a single H100 GPU costs roughly $30,000–$40,000 and cloud GPU instances run $3–$8/hour, the financial stakes are enormous.

Technique 1: Quantization — Shrink Without Breaking

Quantization reduces the numerical precision of model weights, typically from 32-bit floats (FP32) down to 16-bit (FP16/BF16), 8-bit (INT8), or even 4-bit (INT4) representations.

How It Works

Think of it like compressing a high-resolution image to JPEG. You lose some pixel-level detail, but the overall picture is still recognizable and useful. Similarly, quantized weights take less memory and enable faster arithmetic operations.

Real-World Impact

INT8 quantization on Llama 2 70B reduces model size from ~140GB to ~70GB, fitting on 2× A100 80GB GPUs instead of 4
GPTQ 4-bit quantization achieves ~3.5x memory reduction with less than 1–2% perplexity degradation on standard benchmarks
Hugging Face's bitsandbytes library enables plug-and-play 4-bit and 8-bit loading with a single parameter change

Tools to know: GPTQ, AWQ (Activation-aware Weight Quantization), bitsandbytes, llama.cpp

Technique 2: KV Cache Optimization — Stop Repeating Yourself

The Key-Value (KV) Cache is one of the most important memory structures in transformer inference. During autoregressive generation, each new token attends to all previous tokens—and storing those attention keys and values in a cache prevents recomputation.

The Problem

KV cache memory scales linearly with sequence length and batch size. For a 70B model with a 4096-token context, the KV cache can consume tens of gigabytes per request.

Solutions

PagedAttention (developed by the vLLM team) revolutionized this by treating KV cache like virtual memory in an operating system—allocating it in non-contiguous "pages." The result? 2–4x higher throughput on the same hardware compared to naive implementations.

vLLM, which implements PagedAttention, has become a production staple. Companies like Anyscale and Databricks have integrated it into their serving infrastructure, reporting up to 24x higher throughput compared to HuggingFace's naive text generation.

Technique 3: Continuous Batching — Keep the GPU Busy

Traditional static batching waits for a fixed batch of requests before processing them. This leads to GPU idleness whenever requests complete at different times.

Continuous batching (also called iteration-level scheduling) processes requests dynamically—inserting new requests into the batch as soon as a slot frees up. This keeps GPU utilization close to 100%.

Why It Matters

Throughput improvements of 5–10x over static batching in high-traffic scenarios
Latency for individual requests drops significantly because they don't wait in queue
NVIDIA's TensorRT-LLM and vLLM both implement continuous batching as a first-class feature

Technique 4: Speculative Decoding — Guess and Verify

Speculative decoding is one of the most elegant tricks in the inference optimization playbook. Here's the core idea:

A small, fast "draft" model generates several candidate tokens
The large "target" model verifies all candidates in a single parallel forward pass
Accepted tokens are kept; the first rejected token triggers a correction

The Numbers

Google Research demonstrated that speculative decoding with a small draft model achieves 2–3x speedup on latency-sensitive tasks with zero quality degradation (because rejected tokens are always replaced correctly by the target model).

DeepMind applied this technique internally, and Google's Gemini API uses variants of speculative decoding to power its fastest response tiers. The technique shines most on tasks where output is more predictable—code completion, summarization, and structured data generation.

Technique 5: Model Pruning and Distillation

Pruning

Pruning removes weights or entire attention heads that contribute minimally to model outputs. Structured pruning (removing whole neurons or layers) yields the best hardware speedups.

SparseGPT can prune GPT-class models to 50% sparsity with less than 1% accuracy loss using a one-shot method requiring no retraining
NVIDIA's Ampere architecture supports 2:4 structured sparsity natively, delivering 2x compute speedup with no software changes

Knowledge Distillation

Distillation trains a smaller "student" model to mimic a larger "teacher" model's behavior.

DistilBERT retains 97% of BERT's performance at 60% of the size and 40% faster
Microsoft's Phi series (Phi-2, Phi-3) demonstrates that carefully curated training data enables tiny models (2.7B parameters) to match models 10x their size on reasoning benchmarks

If you want to go deeper on the theory behind neural network compression, Efficient Deep Learning: A Comprehensive Guide for Practitioners is an excellent starting point that covers pruning, distillation, and quantization in a unified framework.

Technique 6: Flash Attention — Rewriting the Core Primitive

FlashAttention (developed by Tri Dao and colleagues at Stanford) rewrites the attention mechanism to be IO-aware—minimizing slow reads and writes to GPU High Bandwidth Memory (HBM).

Standard attention requires materializing the full N×N attention matrix in HBM. FlashAttention computes attention in tiles that fit in the much faster SRAM, then writes results back.

Results

2–4x faster attention computation
5–20x memory reduction for long sequences
Enables longer context windows on the same GPU
Adopted by virtually every major LLM serving framework: vLLM, TensorRT-LLM, HuggingFace Transformers

FlashAttention-2 and FlashAttention-3 (targeting H100 Tensor Cores specifically) push these gains even further, with FlashAttention-3 achieving up to 740 TFLOPs/s on H100—close to theoretical peak.

Technique 7: Tensor Parallelism and Pipeline Parallelism

For very large models that don't fit on a single GPU, distributed inference is essential.

Tensor Parallelism

Splits individual weight matrices across multiple GPUs. Each GPU computes a portion of each matrix multiplication, then results are combined via an AllReduce operation.

Megatron-LM pioneered this approach at NVIDIA
A 70B model can be split across 4×A100 80GB GPUs with near-linear throughput scaling

Pipeline Parallelism

Assigns different layers of the model to different GPUs. Requests flow through the pipeline stage by stage.

Best for batch inference workloads (lower latency sensitivity)
DeepSpeed provides robust pipeline parallelism with ZeRO memory optimization

Understanding the trade-offs between these strategies is crucial for production ML engineers—topics covered thoroughly in books like Designing Machine Learning Systems: An Iterative Process, which provides practical frameworks for deploying models at scale.

Technique 8: Prompt Caching and Prefix Sharing

Many LLM applications share long system prompts across thousands of requests. Without optimization, the same prompt tokens are processed from scratch every single time.

Prompt caching stores the KV cache of repeated prefixes:

Anthropic's Claude API offers prompt caching, reducing costs by up to 90% and latency by up to 85% for repeated prefix patterns
OpenAI introduced automatic prompt caching for prompts >1,024 tokens, with cache hits billed at 50% of the normal token price
vLLM's RadixAttention