LLM Inference

Generative text inference — tokens produced per second per GPU. · llama2-70b · Offline + Server scenarios · v4.1 / v5.0 / v5.1 / v6.0 / v6.1 · on-demand pricing

Throughput — per GPU

GPUScenariotok/s per GPU
NVIDIA GB300Offline16,532 tok/s
NVIDIA GB300Server15,998 tok/s
NVIDIA B300-SXM-270GBOffline14,441 tok/s
NVIDIA B300-SXM-270GBServer14,322 tok/s
AMD Instinct MI355X 288GB HBM3eOffline13,315 tok/s
AMD Instinct MI355X 288GB HBM3eServer12,992 tok/s
NVIDIA B200-SXM-180GBOffline12,838 tok/s
NVIDIA B200-SXM-180GBServer12,800 tok/s
NVIDIA H200-SXM-141GBOffline4,415 tok/s
AMD Instinct MI325X 256GB HBM3EOffline4,319 tok/s
NVIDIA H200-SXM-141GBServer4,254 tok/s
AMD Instinct MI325X 256GB HBM3EServer4,003 tok/s
NVIDIA H200-NVL-141GBOffline4,000 tok/s
NVIDIA H100-SXM-80GBOffline3,913 tok/s
NVIDIA H100-SXM-80GBServer3,888 tok/s
NVIDIA H200-NVL-141GBServer3,636 tok/s
NVIDIA L40SOffline435 tok/s
NVIDIA L40SServer400 tok/s

Efficiency — tok/$ · ranked

ProviderGPUScenarioPricetok/$
Lambda LabsNVIDIA H200-SXM-141GBOffline$2.29/GPU·hr
48m ago		6,939,963
Lambda LabsNVIDIA B200-SXM-180GBOffline$6.69/GPU·hr
48m ago		6,908,274
Lambda LabsNVIDIA B200-SXM-180GBServer$6.69/GPU·hr
48m ago		6,887,733
Lambda LabsNVIDIA H200-SXM-141GBServer$2.29/GPU·hr
48m ago		6,686,996
RunPodNVIDIA B200-SXM-180GBOffline$7.99/GPU·hr
48m ago		5,784,274
RunPodNVIDIA B200-SXM-180GBServer$7.99/GPU·hr
48m ago		5,767,076
GCP (us-central1)NVIDIA B200-SXM-180GBOffline$8.05/GPU·hr
48m ago		5,737,598
GCP (us-central1)NVIDIA B200-SXM-180GBServer$8.05/GPU·hr
48m ago		5,720,538
HyperstackNVIDIA H200-SXM-141GBOffline$2.79/GPU·hr
48m ago		5,696,242
OCIAMD Instinct MI355X 288GB HBM3eOffline$8.60/GPU·hr
48m ago		5,573,564
HyperstackNVIDIA H200-SXM-141GBServer$2.79/GPU·hr
48m ago		5,488,610
NebiusNVIDIA B300-SXM-270GBOffline$9.50/GPU·hr
48m ago		5,472,474
OCIAMD Instinct MI355X 288GB HBM3eServer$8.60/GPU·hr
48m ago		5,438,327
NebiusNVIDIA B200-SXM-180GBOffline$8.50/GPU·hr
48m ago		5,437,218
NebiusNVIDIA B300-SXM-270GBServer$9.50/GPU·hr
48m ago		5,427,267
NebiusNVIDIA B200-SXM-180GBServer$8.50/GPU·hr
48m ago		5,421,051
CoreWeaveNVIDIA B200-SXM-180GBOffline$8.60/GPU·hr
48m ago		5,373,994
CoreWeaveNVIDIA B200-SXM-180GBServer$8.60/GPU·hr
48m ago		5,358,016
HyperstackNVIDIA H100-SXM-80GBOffline$2.72/GPU·hr
48m ago		5,179,434
HyperstackNVIDIA H100-SXM-80GBServer$2.72/GPU·hr
48m ago		5,146,260
DigitalOceanAMD Instinct MI325X 256GB HBM3EOffline$3.80/GPU·hr
48m ago		4,092,039
DigitalOceanAMD Instinct MI325X 256GB HBM3EServer$3.80/GPU·hr
48m ago		3,792,739
DigitalOceanNVIDIA H200-SXM-141GBOffline$4.47/GPU·hr
48m ago		3,555,372
RunPodNVIDIA H100-SXM-80GBOffline$3.99/GPU·hr
48m ago		3,530,842
Lambda LabsNVIDIA H100-SXM-80GBOffline$3.99/GPU·hr
48m ago		3,530,842
RunPodNVIDIA H100-SXM-80GBServer$3.99/GPU·hr
48m ago		3,508,227
Lambda LabsNVIDIA H100-SXM-80GBServer$3.99/GPU·hr
48m ago		3,508,227
OCINVIDIA B300-SXM-270GBOffline$15.00/GPU·hr
48m ago		3,465,900
OCINVIDIA B300-SXM-270GBServer$15.00/GPU·hr
48m ago		3,437,269
DigitalOceanNVIDIA H200-SXM-141GBServer$4.47/GPU·hr
48m ago		3,425,777
OCINVIDIA GB300Offline$18.00/GPU·hr
48m ago		3,306,325
OCINVIDIA B200-SXM-180GBOffline$14.00/GPU·hr
48m ago		3,301,168
OCINVIDIA B200-SXM-180GBServer$14.00/GPU·hr
48m ago		3,291,352
OCINVIDIA GB300Server$18.00/GPU·hr
48m ago		3,199,597
DigitalOceanNVIDIA H100-SXM-80GBOffline$4.41/GPU·hr
48m ago		3,194,571
DigitalOceanNVIDIA H100-SXM-80GBServer$4.41/GPU·hr
48m ago		3,174,110
CrusoeNVIDIA H100-SXM-80GBOffline$4.50/GPU·hr
48m ago		3,130,680
NebiusNVIDIA H100-SXM-80GBOffline$4.50/GPU·hr
48m ago		3,130,680
CrusoeNVIDIA H100-SXM-80GBServer$4.50/GPU·hr
48m ago		3,110,628
NebiusNVIDIA H100-SXM-80GBServer$4.50/GPU·hr
48m ago		3,110,628
RunPodNVIDIA H200-SXM-141GBOffline$5.29/GPU·hr
48m ago		3,004,256
CrusoeNVIDIA H200-SXM-141GBOffline$5.40/GPU·hr
48m ago		2,943,058
NebiusNVIDIA H200-SXM-141GBOffline$5.40/GPU·hr
48m ago		2,943,058
RunPodNVIDIA H200-SXM-141GBServer$5.29/GPU·hr
48m ago		2,894,749
CrusoeNVIDIA H200-SXM-141GBServer$5.40/GPU·hr
48m ago		2,835,782
NebiusNVIDIA H200-SXM-141GBServer$5.40/GPU·hr
48m ago		2,835,782
CoreWeaveNVIDIA H200-SXM-141GBOffline$6.31/GPU·hr
48m ago		2,518,624
CoreWeaveNVIDIA H200-SXM-141GBServer$6.31/GPU·hr
48m ago		2,426,818
CoreWeaveNVIDIA H100-SXM-80GBOffline$6.16/GPU·hr
48m ago		2,287,023
CoreWeaveNVIDIA H100-SXM-80GBServer$6.16/GPU·hr
48m ago		2,272,374
CrusoeNVIDIA H200-NVL-141GBOffline$8.50/GPU·hr
48m ago		1,694,329
OCINVIDIA H200-SXM-141GBOffline$10.00/GPU·hr
48m ago		1,589,251
CrusoeNVIDIA H200-NVL-141GBServer$8.50/GPU·hr
48m ago		1,539,787
OCINVIDIA H200-SXM-141GBServer$10.00/GPU·hr
48m ago		1,531,322
Azure (eastus2)NVIDIA H200-SXM-141GBOffline$10.60/GPU·hr
48m ago		1,499,294
GCP (us-central1)NVIDIA H200-SXM-141GBOffline$10.84/GPU·hr
48m ago		1,465,425
Azure (eastus2)NVIDIA H200-SXM-141GBServer$10.60/GPU·hr
48m ago		1,444,643
GCP (us-central1)NVIDIA H200-SXM-141GBServer$10.84/GPU·hr
48m ago		1,412,009
AWS (us-east-1)NVIDIA H200-NVL-141GBOffline$10.60/GPU·hr
48m ago		1,358,660
OCINVIDIA H100-SXM-80GBOffline$10.75/GPU·hr
48m ago		1,310,517
OCINVIDIA H100-SXM-80GBServer$10.75/GPU·hr
48m ago		1,302,123
AWS (us-east-1)NVIDIA H200-NVL-141GBServer$10.60/GPU·hr
48m ago		1,234,735
GCP (us-central1)NVIDIA H100-SXM-80GBOffline$11.53/GPU·hr
48m ago		1,222,202
GCP (us-central1)NVIDIA H100-SXM-80GBServer$11.53/GPU·hr
48m ago		1,214,374
AWS (us-east-1)NVIDIA H100-SXM-80GBOffline$12.29/GPU·hr
48m ago		1,146,303
Azure (eastus)NVIDIA H100-SXM-80GBOffline$12.29/GPU·hr
48m ago		1,146,303
Azure (eastus2)NVIDIA H100-SXM-80GBOffline$12.29/GPU·hr
48m ago		1,146,303
AWS (us-east-1)NVIDIA H100-SXM-80GBServer$12.29/GPU·hr
48m ago		1,138,961
Azure (eastus)NVIDIA H100-SXM-80GBServer$12.29/GPU·hr
48m ago		1,138,961
Azure (eastus2)NVIDIA H100-SXM-80GBServer$12.29/GPU·hr
48m ago		1,138,961
VultrNVIDIA L40SOffline$1.54/GPU·hr
48m ago		1,018,521
DigitalOceanNVIDIA L40SOffline$1.57/GPU·hr
48m ago		997,891
VultrNVIDIA L40SServer$1.54/GPU·hr
48m ago		936,490
DigitalOceanNVIDIA L40SServer$1.57/GPU·hr
48m ago		917,521

Generation Efficiency Ratios

AWS (us-east-1) · Offline · H200 vs H1001.19×more tok/$
AWS (us-east-1) · Server · H200 vs H1001.08×more tok/$
Azure (eastus2) · Offline · H200 vs H1001.31×more tok/$
Azure (eastus2) · Server · H200 vs H1001.27×more tok/$
CoreWeave · Offline · B200 vs H1002.35×more tok/$
CoreWeave · Offline · H200 vs H1001.10×more tok/$
CoreWeave · Server · B200 vs H1002.36×more tok/$
CoreWeave · Server · H200 vs H1001.07×more tok/$
Crusoe · Offline · H200 vs H1000.94×more tok/$
Crusoe · Server · H200 vs H1000.91×more tok/$
DigitalOcean · Offline · H200 vs H1001.11×more tok/$
DigitalOcean · Server · H200 vs H1001.08×more tok/$
GCP (us-central1) · Offline · B200 vs H1004.69×more tok/$
GCP (us-central1) · Offline · H200 vs H1001.20×more tok/$
GCP (us-central1) · Server · B200 vs H1004.71×more tok/$
GCP (us-central1) · Server · H200 vs H1001.16×more tok/$
Hyperstack · Offline · H200 vs H1001.10×more tok/$
Hyperstack · Server · H200 vs H1001.07×more tok/$
Lambda Labs · Offline · B200 vs H1001.96×more tok/$
Lambda Labs · Offline · H200 vs H1001.97×more tok/$
Lambda Labs · Server · B200 vs H1001.96×more tok/$
Lambda Labs · Server · H200 vs H1001.91×more tok/$
Nebius · Offline · B200 vs H1001.74×more tok/$
Nebius · Offline · H200 vs H1000.94×more tok/$
Nebius · Server · B200 vs H1001.74×more tok/$
Nebius · Server · H200 vs H1000.91×more tok/$
OCI · Offline · B200 vs H1002.52×more tok/$
OCI · Offline · H200 vs H1001.21×more tok/$
OCI · Server · B200 vs H1002.53×more tok/$
OCI · Server · H200 vs H1001.18×more tok/$
RunPod · Offline · B200 vs H1001.64×more tok/$
RunPod · Offline · H200 vs H1000.85×more tok/$
RunPod · Server · B200 vs H1001.64×more tok/$
RunPod · Server · H200 vs H1000.83×more tok/$