Study 01 · Benchmark research · October 8, 2026
Capacity has
conditions.
How workload and response targets change GPU economics.
A measured comparison of H100, H200 and L40S serving Llama 3.1 8B—and a transparent look at which findings can inform a purchase.
The business question
What does meeting your customer experience actually cost?
Peak throughput is not the same as capacity that meets your response requirements. We filtered published configurations by first-token delay and token spacing, compared the remaining capacity, and tested how demand and whole-GPU counts affect a hypothetical fleet bill.
Evidence boundary. Performance figures are published NVIDIA benchmarks. Prices, capacity allowances and replica scaling are assumptions. No customer deployment, realized savings, production p95/p99 guarantee or reliable universal predictor is claimed.
Avoid chasing unusable throughput
Higher load can buy more waiting.
Near the H100 curve’s plateau, raising simultaneous requests from 200 to 250 adds no measured throughput advantage, while first-token delay rises from 1.26 to 1.89 seconds.
Figure 01 · Published measurements
More requests. Almost no extra output.
Engineering decision
Shortlist concurrency settings that meet the response requirement. Investigate the plateau with repeated load tests before increasing the request limit.
Financial decision
Ask whether the additional load delivers usable output. Screening out settings that miss requirements can focus proof-of-concept effort; testing time saved has not been measured.
Compare the workload you actually serve
The upgrade advantage is not a constant.
Under illustrative limits of 500 ms time to first token and 10 ms inter-token latency, the H200’s qualifying throughput advantage over H100 ranges from 2.4% to 193.4% across the five workloads.
Figure 02 · Published measurements
The workload changes the premium.
Qualifying aggregate output tokens per second
H200 delivers 61.7% more qualifying throughput in this comparison.
Engineering decision
Specify input and output lengths, model quality, software and response targets in the vendor workload brief. A benchmark from another workload may not transfer.
Financial decision
At equal operating time and full qualifying utilization, the capacity ratio gives a break-even hourly cost ratio. For 1,000 / 1,000, H200 can cost 61.7% more and have equal compute cost per equivalent answer. Actual demand and operating costs can change that result.
Translate capacity into a bill
You pay for whole GPUs.
In our scenario, 8 equivalent answers per second needs two H100 replicas at $4/hour or one H200 at $3/hour. At 10 answers per second, the comparison flips: $4/hour for H100 versus $6/hour for H200.
Figure 03 · Hypothetical cost scenario
A faster GPU does not always make a cheaper fleet.
| Demand answers/sec | H100 fleet | H200 fleet | Lower cost |
|---|---|---|---|
| 1 | 1 GPU$2/hour | 1 GPU$3/hour | H100 |
| 5 | 1 GPU$2/hour | 1 GPU$3/hour | H100 |
| 8 | 2 GPUs$4/hour | 1 GPU$3/hour | H200 |
| 10 | 2 GPUs$4/hour | 2 GPUs$6/hour | H100 |
| 12 | 3 GPUs$6/hour | 2 GPUs$6/hour | Tie |
At 8 answers/sec in this scenario
25%lower modeled fleet cost$1/hour less than an all-H100 baseline. At 720 continuously billed hours, that is $720 per 30-day month before other costs. These are hypothetical savings, not an observed customer outcome.
One assumption can reverse the choice.
H200: $6/h
H100 is cheaper.
H200: $3/h
H200 is cheaper.
Same demand of 8 answers/sec and same hypothetical prices. At 70%, one H200 falls short, triggering a second replica. The percentage is a planning allowance, not measured hardware utilization or a validated safe limit.
Engineering decision
Measure usable capacity under representative arrival rates, bursts and latency percentiles. Do not assume a fixed allowance guarantees a response target.
Financial decision
Compare quotes using demand, capacity headroom and whole-unit counts. Request a range of outcomes rather than accepting one savings estimate. Add failover, billing constraints and migration costs before choosing.
Test predictions before relying on them
A useful calculator does not require a trusted ML predictor.
We compared benchmark lookup, log-linear Ridge, Extra Trees and gradient boosting. Boosting achieved 7.14% mean absolute percentage error for the held-out 1,000 / 1,000 workload—but 122.23% for 20,000 / 2,000. All predictions in that long-input fold overestimated throughput.
Figure 04 · Development validation
One good prediction result is not enough.
| Input / output | Lookup | Ridge | Extra Trees | Boosting |
|---|---|---|---|---|
| 200 / 200 | 15.94% | 100.35% | 25.28% | 21.03% |
| 500 / 2,000 | 5.09% | 47.19% | 20.10% | 32.44% |
| 1,000 / 1,000 | 17.33% | 38.29% | 22.82% | 7.14% |
| 5,000 / 500 | 145.86% | 33.02% | 14.74% | 41.01% |
| 20,000 / 2,000 | 106.69% | 121.54% | 154.04% | 122.23% |
Engineering decision
Keep the predictor experimental. Collect independently varied token lengths and request-level logs; reserve fresh runs for final testing. Refuse unsupported predictions.
Financial decision
Use measured configurations to narrow the quote comparison now. Do not base a purchase on the best ML score from one workload. Overestimated throughput can lead to insufficient capacity; underestimated throughput can inflate the fleet.
Method and reproducibility
Measured inputs. Explicit assumptions.
Performance selection
c* = arg max T(c)
subject to TTFT(c) ≤ target
and ITL(c) ≤ target
Choose the highest-throughput qualifying tested setting. This finite-set optimum says nothing about untested settings. Missing required measurements remain unverified.
Fleet scenario
Replicas = ceil[demand / (capacity × allowance)]
Fleet cost = replicas × hourly price
Equivalent answers/sec = aggregate output tokens/sec ÷ fixed output tokens per answer. This is a conversion, not a separately measured completion rate. Separate replicas are assumed to scale ideally.
The source workbook contains 136 NIM 1.8 configurations for Llama 3.1 8B, TP1 on a single GPU: 113 FP8 and 23 BF16. We analyzed FP8 separately. The workbook has no repeated configuration keys or missing required fields. Coverage is uneven; absent settings do not establish failure.
Five token-length pairs were held out one at a time for throughput prediction. GPU labels, token lengths and concurrency were inputs; measured latency was not used as a predictor. Preprocessing was fitted within each training fold. These related configuration summaries are not independent production trials, and the explored folds do not form an untouched final test.
What this report does not establish
Request-level tail latency, measured end-to-end completion distributions, mixed-workload capacity, quality differences between precision profiles, fleet scaling efficiency and real customer savings. Aggregate response seconds saved do not automatically become paid labor hours saved.
Retrieved October 8, 2026. Published measurements, not Rack City Capital hardware tests.
Bring your actual inputs
Turn a vendor quote into
a defined comparison.
Start with your model, token lengths, peak demand, response targets and dated quote. A scoped review can identify comparable options, assumptions and the measurements still needed.
Request a workload review