NVIDIA has announced that they have broken yet another new record on Meta’s Llama 4 Maverick 4B model through the power of Blackwell servers.
Achieved via a single DGX B200 node equipped with eight NVIDIA Blackwell GPUs, marking the first time any platform has broken the 1,000 TPS/user threshold for such a large-scale model, according to independent measurements by Artificial Analysis.
With a total throughput of up to 72,000 TPS per server, NVIDIA positions Blackwell as the optimal platform for deploying Llama 4 for both maximizing overall throughput and minimizing latency for critical, time-sensitive tasks. This record-setting performance was made possible by extensive software optimization through TensorRT-LLM, as well as the use of speculative decoding techniques based on the EAGLE-3 architecture. Together, these improvements delivered a 4x speed-up over the best prior Blackwell baseline.
NVIDIA ensured that these optimizations didn’t come at the cost of accuracy. By adopting FP8 precision for key operations like GEMMs, Attention, and Mixture of Experts (MoE), the team reduced memory load while maintaining performance parity with BF16-based reference models. Benchmark comparisons across multiple datasets, including LiveCodeBench and MATH-500, showed comparable or even improved scores in most cases.
Minimizing latency is critical for applications like real-time assistants and AI agents, where every millisecond counts. NVIDIA focused heavily on reducing per-user latency through CUDA kernel optimizations, including low-latency GEMM kernels and operation fusions like AllReduce with RMSNorm. The company also used warp specialization and memory shuffling strategies to better utilize the DGX system’s massive 64TB/s HBM3e bandwidth.
A key contributor to the record performance was Programmatic Dependent Launch (PDL), a CUDA feature that allows overlapping kernel execution to avoid GPU idle time. Instead of waiting for a kernel to finish before starting the next, PDL enables secondary kernels to launch during the tail end of the first kernel’s execution, boosting GPU utilization and throughput.
The research team also doubled down on speculative decoding, which uses a smaller draft model to predict sequences of tokens that are later verified by the main model. By tuning the draft length and optimizing inference via CUDA Graphs and the TensorRT-LLM overlap scheduler, NVIDIA managed to reduce latency and boost speed without introducing significant overhead. Draft length 3 was found to offer the best balance, delivering a peak speed-up of 2.06x.
To further streamline speculative decoding, NVIDIA avoided breaking the CUDA Graph with host-device sync points by keeping the verification logic entirely on the device side. Meanwhile, they used torch.compile() with OpenAI’s Triton backend to auto-fuse many small Torch native kernels in the draft model, cutting the overhead from 25% down to 18%.
In summary, NVIDIA’s achievement of real-time, low-latency inference at over 1,000 TPS/user on Llama 4 Maverick underscores how hardware-software co-design continues to push the limits of generative AI performance. The Blackwell architecture that combines with cutting-edge optimizations in CUDA, TensorRT-LLM, and speculative decoding sets a new bar for deploying ultra-large models in production environments, enabling more responsive and scalable AI systems for enterprise and research use alike.
