NVIDIA Launches Dynamo with KV-Aware Routing for Agentic Coding Inference

Details

• NVIDIA AI announced Dynamo, a new inference stack optimized for agentic coding tools that make hundreds of API calls per session with recomputed context, addressing cost-per-token bottlenecks.
• Key feature is KV-aware routing, which directs requests to GPU workers with the highest KV cache overlap to reuse precomputed context and skip redundant prefill computation.
• Supports backends like SGLang, TensorRT-LLM, and vLLM across multinode, disaggregated GPU fleets; uses hashing and Radix Trees for scalable cache tracking.
• Routing modes include KV (evaluates cache hit and load), round-robin, random, least-loaded, and device-aware weighted; configurable via CLI like --router-mode kv.
• Multimodal KV routing extends to images by computing hashes for cache overlap; Baseten reported 2x faster time-to-first-token (TTFT) and 1.6x throughput on Qwen3 Coder 480B.
• Open-source on GitHub; workers auto-report KV events for real-time global view of cache usage and load.

Impact

NVIDIA Dynamo's KV-aware routing boosts inference efficiency for agentic AI workloads, delivering up to 2x faster TTFT and higher throughput as shown by Baseten on large models. This pressures rivals like vLLM and TensorRT-LLM by integrating smarter cache reuse in distributed setups, potentially lowering costs for high-context coding agents. As agentic tools proliferate, Dynamo narrows performance gaps in multi-GPU environments, enabling broader adoption without custom optimizations. It aligns with trends toward disaggregated serving, positioning NVIDIA ahead in optimizing frontier-model inference for real-world API-heavy use cases.