Insights
Ten hours inside NVIDIA's inference stack
Notes from NVIDIA’s Inference Co-Design Day in Berlin, read against the traffic we serve: long agent sessions, most of them already in cache.

Notes from NVIDIA’s Inference Co-Design Day in Berlin, read against the traffic we serve: long agent sessions, most of them already in cache.
A coding agent sends its whole session back to the model on every turn. On our Kimi K3 deployment that prefix grows to between 100,000 and 400,000 tokens, and nearly all of it is text the model read on the previous turn. Two thirds of what we serve is long-context, and our cache hit rate runs around 85%. That shape decides most of how we serve: where each request goes, how far ahead a draft model can guess, whether splitting prefill from decode pays, and where the cache lives.
NVIDIA’s Inference Co-Design Day in Berlin on 19th October, the day before GTC Berlin, spent much of its agenda on the same questions. The morning covered the engines (vLLM, SGLang and TensorRT-LLM) and a panel on speculative decoding. After lunch came nearly three hours of NVIDIA Dynamo, with a panel on request routing in the middle, then sessions on NVIDIA’s new LPX and Vera racks, resiliency and autoscaling, and the networking and Kubernetes underneath.
We went as an NVIDIA inference partner and enjoyed every second. PrivateMind runs open-weight models on our own Blackwell GPUs for financial firms that won’t send their data to a public API, with trillions of tokens a month. KV-aware routing, tiered KV cache and scheduling already run in production here, so most of what follows compares NVIDIA’s choices with ours.
Drafts past 128K
Speculative decoding lets a small draft model guess the next few tokens, and the big model checks the guesses in one pass. The catch is load. Each guess is one more token the GPU has to verify, and vLLM’s documentation warns that past a critical batch size speculation “negatively impacts the decode speed.” In the EAGLE-3 paper’s tests on vLLM, Llama 3.1 8B ran 1.75 times faster with two requests in flight and 1% faster with 56. NVIDIA’s own DeepSeek-V4 Pro settings on GB300 drop from three draft tokens at most latency points to one at the highest concurrency.
Long context changes the arithmetic. With long sequences, each step spends most of its time reading the KV cache, and the MagicDec paper found speculation still paying up to 2.51 times at batch sizes from 32 to 256. Our traffic sits in that regime. Most drafters don’t; they’re trained on text under 4K tokens. Our Kimi K3 drafter averaged about 2.3 accepted tokens per step up to 128K and 1.3 past 256K. We let it read only the last 32K tokens, which brought it back above 2.2 and raised output speed 23 to 31% on those prompts, and sent the fix to vLLM. Last week a second PR proposed the same setting for EAGLE-3 drafters.
Routing on the cache
The KV cache is the model’s working memory for a conversation. A round-robin load balancer throws it away every time a follow-up lands on a worker that doesn’t hold its prefix, and with prompts this long, a miss means reading a few hundred thousand tokens again before the first word comes back.
The routing panel was billed as gateway decisions and KV-cache-aware routing. Our gateway hashes each prompt, remembers which engine got each prefix, and sends follow-ups back there unless load makes another engine cheaper. In production tests with long agent sessions at moderate load, 91% of follow-up turns hit a warm cache. The gateway also decides who goes first, sharing capacity between customers and passing each request’s place to vLLM as a priority.
Dynamo, the open-source layer NVIDIA puts above the engines to decide where each request runs, has a router that works from better information. Workers publish an event every time they store or drop a cache block, so the router knows what each one actually holds. For each request it counts the prompt blocks each worker would still have to compute, adds the decode work already running there, and picks the cheapest. If you come from trading, it’s a smart order router for tokens: send the order where the liquidity sits, but price in the queue first.
Under heavy load, engines evict blocks faster than a gateway’s bookkeeping can follow, and worker events fix the picture. They don’t fix the evictions. In our tests, two Kimi K3 engines kept 80% of follow-up turns on a warm cache at 32 long sessions and 39% at 48. Agents take the worst of it. NVIDIA’s agent deck puts it as “an agent is a program, not a stream of chats”: while an agent waits on a tool, its cache sits idle and gets evicted to make room, and when it comes back it reads its whole history again. ThunderAgent, a research scheduler now in Dynamo as an experimental plugin, pauses whole agent programs once cache use reaches 95% and resumes them on the same worker when there’s room. Its paper reports 1.7 to 2.1 times the throughput at 48 to 72 concurrent programs.
Hybrid models are still an open problem. Qwen3.8 and Kimi K3 mix linear-attention layers, which carry one running state per sequence, with ordinary attention layers that cache per token, and Dynamo’s router only sees the ordinary ones. When we tested it on Qwen3.8-27B, the router expected 34% to 70% of each follow-up prompt to come from cache. The engine reused 12% to 18%, with or without Dynamo in front.
Splitting prefill from decode
Take pasting a 50-page contract into a chatbot and ask for a summary. The model first reads all 50 pages in one parallel pass. That’s prefill, which is bound by compute. Then it writes the summary a token at a time, looking back over everything it stored on every step. That’s decode, and it’s bound by memory bandwidth. On a shared GPU one phase keeps getting in the other’s way.
Disaggregated serving gives each phase its own workers and copies the cache from one to the other, about 320 KB per token for a 70B-class model in FP16, or 2.7 GB for an 8K prompt. NVIDIA’s early numbers were 30% more throughput per GPU on one node and over twice as much on two, measured with roughly 3,000 tokens in and 150 out. NVIDIA also ships a tool, AIConfigurator, that predicts whether splitting wins for your model, hardware and latency targets. You don’t build that for a sure thing.
Our traffic pulls both ways. Long prefills are the case for splitting: while an engine works through a 300K-token prompt, every other conversation on it slows down. But most of our prompts are already cached, and NVIDIA’s own recipe for Qwen3.5-122B shows what that does. It’s tuned for agents: 64K tokens in, 400 out, 90% of the prompt already cached. On B200, unsplit serving won, 1,173 output tokens per second per GPU against 917, and on H200 it won by almost 3x. The recipe gives the reason in one line: “At 90% KV-cache hit the dedicated prefill worker has little to do yet emits no output tokens.” Dynamo now has an experimental mode, off by default, that skips the prefill pool when the decode worker already holds most of the prompt, and that may be the version that suits us.
Splitting also costs something underneath. It needs RDMA, even between two pods on one server, because Kubernetes isolation blocks NVLink between pods; over TCP, NVIDIA’s guide puts the first token at roughly 98 seconds, against 200 to 500 milliseconds with RDMA. And a split model only serves with both halves running, so a Kubernetes scheduler that places each role on its own can deadlock. NVIDIA’s Grove deck walks through two services on a tight cluster that each place their prefill pods first, leaving both decodes waiting forever: 32 GPUs held, nothing served.
Where the cache lives
Sessions this long fill GPU memory quickly, so the cache gets pushed down a ladder: host RAM, local SSD, shared storage, and now a rung of BlueField-4 flash NVIDIA calls CMX. Dynamo had its own manager for that ladder until version 1.5 deprecated it in favor of “the engine’s native KV offloading for host and disk tiering.” That’s what we already run: vLLM’s own offloading into a 480 GiB RAM tier and local NVMe on each Kimi host. On a normal day 94 to 99% of our cached tokens still come straight from GPU memory, and at peak the lower tiers serve about half. We’ve also tested shared storage for this with Everpure FlashBlade (white paper).
Long context on new silicon
The session on inference beyond GPUs covered NVIDIA’s two new racks. LPX is built on Groq’s LPUs, each with 500 MB of on-chip SRAM and no HBM. A rack of 256 holds 128 GB in total, less memory than a single B200. In NVIDIA’s design the GPUs keep the KV cache and run attention, and the LPUs run the feed-forward and expert layers, trading activations once per attention layer for every token. On an LPX system alone, Artificial Analysis measured 3,431 output tokens per second for Gemma 4 31B with 100K tokens of context. NVIDIA’s numbers for GPUs and LPUs working together are still projections. Vera CPU racks sit alongside to run agent sandboxes, over 22,500 per rack.
Dynamo is open source, and NVIDIA’s own slides list AMD GPUs and Intel XPUs among the hardware it runs on, plus Google TPUs and Amazon Trainium through vLLM. The parts tied to NVIDIA’s racks are the newest ones, like splitting a token between GPUs and LPUs, and that’s where I’d watch NVIDIA’s moat.
Starting , failing, hanging
Big models start slowly. One replica of DeepSeek-V4-Pro takes 14 minutes 43 seconds to start in NVIDIA’s measurement on eight B200s. Loading the 806 GiB of weights is two and a half minutes of that. The other twelve go on compiling kernels, autotuning and capturing CUDA graphs, work that comes out identical on every replica, and an autoscaler waiting fifteen minutes for a worker is always reacting to traffic that already happened. Dynamo’s ModelExpress copies weights from a GPU that already holds them, and its Snapshot feature restores a warmed-up worker, GPU state included, on another node. For Qwen3-Next-80B on one B200, the pair took a new replica from 180 seconds to 14. Snapshot is still experimental.
For crashes, NVIDIA’s fault-tolerance deck has a shadow engine, still in preview. A small sidecar holds the weights in GPU memory so they survive the engine process, and a second, fully warmed engine waits on a plain file lock. When the active process dies, the kernel releases the lock and the shadow takes over. On GLM-5.2 across eight B200s, a worker was serving again in 7.3 seconds, against 283 for a cold restart.
Hangs are harder, and the same deck calls canaries “the only detector for a hung engine with a live process.” Our gateway follows the same logic. A hung engine looks idle, and a router that balances on load would send it more work, so it takes an engine out of rotation when its metrics go stale or when it has requests running and no tokens coming out, and a canary sends a real completion to every model every five minutes.
What we're taking home
We plan to adopt Dynamo, with our gateway staying in front for keys, budgets, fairness between customers and metering. When we first installed it into our vLLM image, its Python dependencies clashed with the image’s own. The sidecar, experimental since 1.5, avoids that by leaving the engine on its stock image, with Dynamo in a small CPU-only process beside it. Before we move a model onto it, we’ll measure the router on our hybrid models and the split on our own traffic.
Thanks to the NVIDIA team for a seriously technical day. If you’re running inference at scale, especially anywhere latency costs real money, we'd love to compare notes.
Want to see how this would work with your firm's data?


