Inference Engineering in a nutshell

With the global Nvidia GPU shortage causing lead time up to 12-18 months, the AI race is shifting from who has the most GPUs to who can extract the most intelligence from every GPU.
AI infrastructure generally has four layers: energy (electricity and cooling), hardware (GPUs and memory), systems software (CUDA, PyTorch, kernels), and inference serving (vLLM, SGLang).
Today, I will focus on the inference serving optimization layer, which is all about running AI models faster, cheaper, and more efficiently to serve users at scale.
TL;DR: Inference serving optimization is about using GPU compute and memory more efficiently, minimizing wasted computation and waiting time, and maximizing tokens per second, per dollar, and per watt.
I'll dive deeper into the GPU supply chain and data center bottlenecks in my next article.
Four Ways Inference Engines Make GPUs More Efficient
Before diving into the optimizations, it's important to understand that LLM inference has two stages, each with different hardware bottlenecks.
Prefill (Compute-Bound): The GPU processes the entire input prompt in parallel, performing large matrix calculations and building the Key-Value (KV) Cache. This stage is typically limited by GPU computing power.
Decode (Memory-Bandwidth-Bound): The model generates tokens one at a time, repeatedly accessing model weights and the growing KV Cache from GPU memory (HBM).
This creates an interesting challenge: the same GPU needs to handle two very different workloads efficiently. That's where inference serving optimizations come in.
1. Stop Recalculating: KV Cache, vLLM, and SGLang

When an LLM generates text, it stores previous attention calculations in something called KV Cache, so it doesn't need to recompute them for every new token. The problem is that these caches consume substantial GPU memory.
vLLM introduced PagedAttention, dividing KV Cache into smaller memory blocks allocated on demand. Imagine a hotel assigning rooms only when guests need them, instead of reserving ten rooms for everyone. Its original research demonstrated 2–4x higher throughput than the systems it compared against.
SGLang introduced RadixAttention, organizing reusable KV Cache in a tree structure. Imagine 1,000 coding agents sharing identical system prompts and tool descriptions. Instead of recalculating everything, they can reuse the same cached computations.
Both engines now support overlapping techniques, but efficient memory management and computation reuse were their defining early breakthroughs.
2. Stop Waiting: Continuous Batching and Prefill/Decode Separation
Traditional batching groups requests together, but shorter requests may finish while longer ones are still running. Continuous batching allows completed requests to leave and new ones to join dynamically, keeping GPUs productive.
Since Prefill and Decode have different computing requirements, Prefill/Decode separation runs them on different GPU pools to reduce interference and improve efficiency.
3. Compute More Efficiently: Quantization and Speculative Decoding
During decoding, GPUs can spend more time moving data than performing calculations, leaving powerful computing units underutilized.
Quantization reduces numerical precision, such as from 16-bit to 8-bit, requiring less memory and data movement. Speculative decoding lets a smaller model draft several tokens before a larger model verifies them together, potentially speeding up generation by 2–3x.
Both help generate more tokens using the same hardware.
4. Scale Across GPUs: Cache-Aware Routing and Distributed Scheduling
Optimizing one GPU is only half the challenge. Production inference often involves hundreds or thousands of GPUs working together.
Imagine GPU 1 already has a system prompt cached, but the next request gets sent to GPU 3. GPU 3 might need to recalculate everything.
Cache-aware routing sends requests to GPUs that already hold useful cached computations, while distributed scheduling balances workloads across machines to avoid overloaded and underutilized GPUs.
Ultimately, all four strategies work toward the same goal: more tokens per second, per dollar, and per watt. The fastest inference system isn't necessarily the cheapest or most energy-efficient, which is why optimizing the entire system matters more than any single technique.
My Take: From Inference to Specialized Intelligence

What's interesting is that more tokens are increasingly being consumed by AI agents rather than read by humans. 10x faster inference doesn't just mean getting the same answer 10x faster. It could allow an AI agent to reason 10x more within the same time, potentially delivering much better results.
Instinct's founder shared that its compute demand was effectively doubling every week, making inference capacity one of its biggest bottlenecks. Fireworks and Base10, the inference optimization companies are also the hottest companies in the bay area that have shown hyper growth rate.
Meanwhile, enterprises are exploring using proprietary data to fine-tune or train specialized models through reinforcement learning. RadixArk is already expanding beyond SGLang into RL infrastructure through Miles, connecting inference and training more efficiently.
We've spent years making AI models smarter. The next big opportunity might be making intelligence cheap enough to deploy everywhere, and specialized enough to solve real business problems.
Get new posts by email
An email whenever I publish. Unsubscribe anytime.
No comments yet