Llama 2 70B · Offline · v4.1 → v5.0 → v5.1 → v6.0 → v6.1 · same GPU, tracked across every MLPerf round
The H100 in a datacenter today is the same H100 that was there a year ago — no refresh, no new silicon.
What's counterintuitive is that its usefulness hasn't degraded the way aging hardware usually does; the
tokens/sec it delivers keeps climbing instead, purely from better kernels, quantization, and
inference-serving software. That's throughput procurement already paid for and hasn't claimed. This page
tracks it directly: the same three flagship GPUs, the same workload, every MLPerf round since we started
tracking. The first chart is raw speed. The second holds today's price constant and asks what a token
actually costs — which is where the story gets less predictable.
+14.0%
B200-SXM tok/s·GPU, v4.1→v6.1 — same silicon, pure software/firmware gain
4.2×
more tok/s per GPU — earliest H100 result vs latest B200
9–22%
range of decline in $/M tokens across flagships since v4.1, at today's floor price
3
MLPerf rounds since H100-SXM last got a llama2-70b submission (v5.0)
Throughput — tokens/sec per GPU, by MLPerf round
H100 SXMH200 SXMB200 SXM
Cost per million tokens — at today’s floor price
H100 SXMH200 SXMB200 SXM
B200 SXM costs $0.145 per million tokens at today's floor
pricing — more than H200 SXM at $0.144 — despite B200 SXM being
the newer, faster chip (12,838 vs 4,415 tok/s·GPU). B200 SXM's price premium
over H200 SXM outpaces the throughput it delivers.
GB200 NVL72's per-GPU throughput (12,417 tok/s, v6.1) is
in line with, not ahead of standalone B200-SXM (12,838 tok/s, v6.1) —
the rack's real advantage is 72 GPUs sharing one NVLink domain, not faster silicon per chip.
No pricing data yet for GB200, so it's excluded from the charts above.
GB300 NVL72's per-GPU throughput (16,532 tok/s, v6.1) is
ahead of standalone B300-SXM (14,441 tok/s, v6.1) —
the rack's real advantage is 8 GPUs sharing one NVLink domain, not faster silicon per chip.
No pricing data yet for GB300, so it's excluded from the charts above.