Skip to content
All case studies

Study 01 · Benchmark research · October 8, 2026

Capacity has
conditions.

How workload and response targets change GPU economics.

A measured comparison of H100, H200 and L40S serving Llama 3.1 8B—and a transparent look at which findings can inform a purchase.

136Published configurations
5FP8 workload pairs
3GPU types

The business question

What does meeting your customer experience actually cost?

Peak throughput is not the same as capacity that meets your response requirements. We filtered published configurations by first-token delay and token spacing, compared the remaining capacity, and tested how demand and whole-GPU counts affect a hypothetical fleet bill.

Evidence boundary. Performance figures are published NVIDIA benchmarks. Prices, capacity allowances and replica scaling are assumptions. No customer deployment, realized savings, production p95/p99 guarantee or reliable universal predictor is claimed.

01

Avoid chasing unusable throughput

Higher load can buy more waiting.

Near the H100 curve’s plateau, raising simultaneous requests from 200 to 250 adds no measured throughput advantage, while first-token delay rises from 1.26 to 1.89 seconds.

Figure 01 · Published measurements

More requests. Almost no extra output.

Output tokens/secFirst-token delay06,50013,0000 ms1000 ms2000 ms150100150200250Simultaneous requests
Throughput · left axis First-token delay · right axis
H100, Llama 3.1 8B FP8, 1,000 input / 1,000 output tokens. Concurrency 200 → 250: throughput −0.22%; TTFT +49.8%. Different vertical scales; lines connect tested points, not predictions. Repeated-run uncertainty is unavailable.

Engineering decision

Shortlist concurrency settings that meet the response requirement. Investigate the plateau with repeated load tests before increasing the request limit.

Financial decision

Ask whether the additional load delivers usable output. Screening out settings that miss requirements can focus proof-of-concept effort; testing time saved has not been measured.

02

Compare the workload you actually serve

The upgrade advantage is not a constant.

Under illustrative limits of 500 ms time to first token and 10 ms inter-token latency, the H200’s qualifying throughput advantage over H100 ranges from 2.4% to 193.4% across the five workloads.

Figure 02 · Published measurements

The workload changes the premium.

H100
6,515
H200
10,534

Qualifying aggregate output tokens per second

H200 delivers 61.7% more qualifying throughput in this comparison.

Highest-throughput published setting per GPU meeting TTFT ≤ 500 ms and ITL ≤ 10 ms. One GPU, FP8, NIM 1.8. No tested L40S setting qualifies for these five workloads. Compare GPUs within a workload; output lengths differ across workloads.

Engineering decision

Specify input and output lengths, model quality, software and response targets in the vendor workload brief. A benchmark from another workload may not transfer.

Financial decision

At equal operating time and full qualifying utilization, the capacity ratio gives a break-even hourly cost ratio. For 1,000 / 1,000, H200 can cost 61.7% more and have equal compute cost per equivalent answer. Actual demand and operating costs can change that result.

03

Translate capacity into a bill

You pay for whole GPUs.

In our scenario, 8 equivalent answers per second needs two H100 replicas at $4/hour or one H200 at $3/hour. At 10 answers per second, the comparison flips: $4/hour for H100 versus $6/hour for H200.

Figure 03 · Hypothetical cost scenario

A faster GPU does not always make a cheaper fleet.

Demand
answers/sec
H100 fleetH200 fleetLower cost
11 GPU$2/hour1 GPU$3/hourH100
51 GPU$2/hour1 GPU$3/hourH100
82 GPUs$4/hour1 GPU$3/hourH200
102 GPUs$4/hour2 GPUs$6/hourH100
123 GPUs$6/hour2 GPUs$6/hourTie
1,000 / 1,000 workload. Hypothetical $2/hour H100 and $3/hour H200; 80% of qualifying benchmark capacity; steady demand; independent replicas with ideal scaling. Compares homogeneous fleets only, excluding migration, other costs and failover.

At 8 answers/sec in this scenario

25%lower modeled fleet cost

$1/hour less than an all-H100 baseline. At 720 continuously billed hours, that is $720 per 30-day month before other costs. These are hypothetical savings, not an observed customer outcome.

One assumption can reverse the choice.

70% usable capacityH100: $4/h
H200: $6/h

H100 is cheaper.

80% usable capacityH100: $4/h
H200: $3/h

H200 is cheaper.

Same demand of 8 answers/sec and same hypothetical prices. At 70%, one H200 falls short, triggering a second replica. The percentage is a planning allowance, not measured hardware utilization or a validated safe limit.

Engineering decision

Measure usable capacity under representative arrival rates, bursts and latency percentiles. Do not assume a fixed allowance guarantees a response target.

Financial decision

Compare quotes using demand, capacity headroom and whole-unit counts. Request a range of outcomes rather than accepting one savings estimate. Add failover, billing constraints and migration costs before choosing.

04

Test predictions before relying on them

A useful calculator does not require a trusted ML predictor.

We compared benchmark lookup, log-linear Ridge, Extra Trees and gradient boosting. Boosting achieved 7.14% mean absolute percentage error for the held-out 1,000 / 1,000 workload—but 122.23% for 20,000 / 2,000. All predictions in that long-input fold overestimated throughput.

Figure 04 · Development validation

One good prediction result is not enough.

Input / outputLookupRidgeExtra TreesBoosting
200 / 20015.94%100.35%25.28%21.03%
500 / 2,0005.09%47.19%20.10%32.44%
1,000 / 1,00017.33%38.29%22.82%7.14%
5,000 / 500145.86%33.02%14.74%41.01%
20,000 / 2,000106.69%121.54%154.04%122.23%
Mean absolute percentage error in output throughput; lower is better. 113 FP8 configurations, leave-one-workload-out across five groups. These explored folds are development validation, not an untouched final test. Input and output lengths were not independently varied.

Engineering decision

Keep the predictor experimental. Collect independently varied token lengths and request-level logs; reserve fresh runs for final testing. Refuse unsupported predictions.

Financial decision

Use measured configurations to narrow the quote comparison now. Do not base a purchase on the best ML score from one workload. Overestimated throughput can lead to insufficient capacity; underestimated throughput can inflate the fleet.

Method and reproducibility

Measured inputs. Explicit assumptions.

Performance selection

c* = arg max T(c)
subject to TTFT(c) ≤ target
and ITL(c) ≤ target

Choose the highest-throughput qualifying tested setting. This finite-set optimum says nothing about untested settings. Missing required measurements remain unverified.

Fleet scenario

Replicas = ceil[demand / (capacity × allowance)]
Fleet cost = replicas × hourly price

Equivalent answers/sec = aggregate output tokens/sec ÷ fixed output tokens per answer. This is a conversion, not a separately measured completion rate. Separate replicas are assumed to scale ideally.

The source workbook contains 136 NIM 1.8 configurations for Llama 3.1 8B, TP1 on a single GPU: 113 FP8 and 23 BF16. We analyzed FP8 separately. The workbook has no repeated configuration keys or missing required fields. Coverage is uneven; absent settings do not establish failure.

Five token-length pairs were held out one at a time for throughput prediction. GPU labels, token lengths and concurrency were inputs; measured latency was not used as a predictor. Preprocessing was fitted within each training fold. These related configuration summaries are not independent production trials, and the explored folds do not form an untouched final test.

What this report does not establish

Request-level tail latency, measured end-to-end completion distributions, mixed-workload capacity, quality differences between precision profiles, fleet scaling efficiency and real customer savings. Aggregate response seconds saved do not automatically become paid labor hours saved.

Primary sourceNVIDIA NIM 1.8 Llama 3.1 8B performance tables

Retrieved October 8, 2026. Published measurements, not Rack City Capital hardware tests.

Bring your actual inputs

Turn a vendor quote into
a defined comparison.

Start with your model, token lengths, peak demand, response targets and dated quote. A scoped review can identify comparable options, assumptions and the measurements still needed.

Request a workload review