NVIDIA Blackwell has broken some new records in the latest MLPerf Inference V5.0 benchmarks.
Tested with the flagship rack scale AI reasoning solution GB200 NLV72 system, it has achieved 30x higher throughput on the Llama 3.1 405B benchmark compared to the NVIDIA H200 NVL8 results, thanks to the interconnected 72 units of NVIDIA Blackwell GPUs to function as a single, high-performance GPU on top of the 9x larger NVIDIA NVLink interconnect domain.
In terms of the test, the latest MLPerf Inference version has included new challenges, including the aforementioned Llama 3.1 405B mode,l which is the one of the largest and open-weight models to date, in addition to the revised Llama 2 70B Interactive benchmark with tighter latency constraints that reflects on actual real-world deployment environment.
The responsiveness benchmark also highlighted the importance of 2 key metrics: Time to First Token (TTFT), which indicates how quickly users start seeing responses, and Time Per Output Token (TPOT), which measures token delivery speed. The newly introduced Llama 2 70B Interactive benchmark enforces a 5x shorter TPOT and 4.4x lower TTFT, ensuring a more responsive AI experience. NVIDIA’s submission, featuring an NVIDIA DGX B200 system with eight Blackwell GPUs, achieved triple the performance compared to a system with eight NVIDIA H200 GPUs, setting a new standard for this demanding benchmark.
By integrating Blackwell architecture with an optimized software stack, NVIDIA continues to push inference performance boundaries, enabling AI factories to achieve greater intelligence, higher throughput, and faster token generation.
On the other hand, the Hopper architecture might not be the “latest and greatest” as of now, but it is still the one behind the beginning of the AI era. Through continuous software enhancements, NVIDIA has increased the throughput of Hopper-based AI factories, further amplifying their value with examples of Llama 2 70B benchmark getting 1.5x improvements on H100 GPU and 1.6x for H200 GPU respectively, on top of full software compatibility by supporting the latest Llama 3.1 405B, Llama 2 70B Interactive, and graph neural network tests.
MLCommons’ continuous refinement of the MLPerf Inference benchmark suite ensures that AI infrastructure keeps pace with rapid advancements. By providing rigorous, peer-reviewed performance data, these benchmarks serve as a valuable resource for IT decision-makers selecting optimal AI solutions.
