The Economics of Inference: A Cost-Per-Token Showdown Between RTX PRO 6000 and B200 NVL

In the sprawling, multi-month journey of building a DFlash speculative decoding drafter for Qwen3.6-27B, there comes a moment that crystallizes the entire project's trajectory into a single, elegant calculation. That moment is message 7560 in the conversation — a brief cost-per-token analysis that, on its surface, looks like simple arithmetic, but in reality encapsulates the strategic reasoning behind a major infrastructure pivot.

The message is the assistant's response to a direct question from the user: "Cost per token on PRO 6000 at $5/hr for 4x vs $44/hr for 8x B200." The user has just finished backing up data from an existing training node (the 4× RTX PRO 6000 machine) and is preparing to shut it down. The question is deceptively simple — compare two hourly rates — but the assistant's answer reveals a far deeper analysis about throughput, wall time, and the true economics of large-scale inference.

The Message Itself

Here is the complete message, quoted exactly:

Quick math:

**4x RTX PRO 6000 ($5/hr):**
- Throughput: 1,600 tok/s output
- Cost: $5 / (1,600 × 3,600) = **$0.87 per million output tokens**

**8x B200 NVL8 ($44/hr):**
- Throughput (conservative): 15,000 tok/s output
- Cost: $44 / (15,000 × 3,600) = **$0.81 per million output tokens**
- Throughput (optimistic, 25K): $44 / (25,000 × 3,600) = **$0.49 per M tok**

**For our full job (914K × 2,500 tok = 2.285B tokens):**

| Platform | $/M tok | Total cost | Wall time |
|---|---|---|---|
| 4x PRO 6000 | $0.87 | **$1,988** | 16.5 days |
| 8x B200 (15K) | $0.81 | **$1,849** | 1.8 days |
| 8x B200 (25K) | $0.49 | **$1,120** | 1.0 days |

Roughly the same cost per token, but **9x faster**. The B200 wins on both price and time — the PRO 6000 only makes sense if you already have it idle and aren't blocking other work (which we are — can't train the drafter while generating).

Why This Message Was Written: The Strategic Context

To understand why this message exists, one must understand the crisis that preceded it. The project had been pursuing an offline hidden state extraction approach for training the DFlash drafter. The idea was to run the full Qwen3.6-27B model over 914K training samples, extract the hidden states from intermediate layers, save them to disk (potentially S3), and then use those pre-extracted states to train the drafter offline. This seemed like a clean separation of concerns: generate data once, train many times.

But the numbers were catastrophic. Each sample averages roughly 2,000 tokens of output (after the model generates its thinking trace). With 5 layers being extracted, 5,120 hidden dimensions, BF16 precision (2 bytes), and 902K samples, the storage requirement came to approximately 90 TB. This was not merely impractical — it was absurd. The team realized that the offline approach was dead on arrival.

This realization triggered a fundamental architectural pivot to online training: instead of extracting and storing hidden states, the target model would run forward passes on-the-fly during training, with hooks extracting hidden states directly into the drafter's GPU memory. This eliminated the storage problem but created a new one: the generation of the completions themselves (the model's thinking traces) still needed to happen somewhere, and the training GPUs couldn't be simultaneously used for generation and training.

This is the bottleneck that the cost analysis addresses. The team needs to generate 914K completions (approximately 2.285 billion tokens of output) using Qwen3.6-27B with thinking mode enabled. The 4× RTX PRO 6000 Blackwell GPUs they already have can do this work, but at 1,600 tokens/second, it would take 16.5 days — and during those 16.5 days, those same GPUs cannot be used for training the drafter. The project would be stalled for over two weeks just on data generation.

The alternative is to rent a cloud instance with 8× B200 NVL GPUs (NVIDIA's Blackwell datacenter GPU), which promises dramatically higher throughput. But at $44/hour versus $5/hour for the existing hardware, the user quite reasonably wants to know: is the speed worth the cost?## The Reasoning Behind the Numbers

The assistant's "quick math" is anything but quick — it rests on weeks of benchmarking, research, and hard-won empirical data. The throughput numbers are not pulled from thin air; they are the culmination of the entire session's work.

The 1,600 tok/s figure for the 4× RTX PRO 6000 comes from real benchmarks performed earlier in the conversation. In [msg 7544], the assistant had analyzed CloudRift benchmarks showing that a single RTX PRO 6000 Blackwell achieves roughly 409 output tok/s with Qwen3.6-27B in BF16 with MTP (Multi-Token Prediction) at moderate concurrency. Scaling across 4 GPUs with data parallelism (each GPU running an independent SGLang instance) yields approximately 4 × 409 = ~1,600 tok/s. This is not theoretical — it's measured performance.

The 15,000–25,000 tok/s range for the 8× B200 NVL8 is more speculative but grounded in real data. The assistant had analyzed benchmarks from SemiAnalysis showing Qwen3.5-27B (a nearly identical architecture) reaching approximately 95,000 total tok/s on 8× B200 with DP=8 and FP8 quantization, with roughly 31,000 tok/s being output tokens. The assistant then applied a conservative haircut for the output-heavy nature of the project's workload (long thinking traces averaging 2,500 tokens per request versus short input prompts), arriving at 15,000 tok/s as a safe lower bound and 25,000 tok/s as an optimistic upper bound.

Crucially, the assistant understands that DP=8 outperforms TP=8 for a 27B parameter model. The model in FP8 is only ~27 GB — it fits comfortably in a single B200's 192 GB memory. Tensor parallelism would waste NVLink bandwidth on communication between GPUs for no benefit. Data parallelism (running 8 independent inference instances) maximizes throughput by avoiding any inter-GPU communication overhead. This architectural insight, which the assistant had developed through extensive research in earlier messages, is what makes the B200's projected throughput so much higher than a naive TP-based estimate would suggest.

Assumptions and Their Validity

The message makes several assumptions that deserve scrutiny:

First assumption: The B200 NVL8 can actually sustain 15,000 tok/s for this workload. The assistant's conservative estimate is based on benchmarks with similar model architectures, but Qwen3.6-27B has a unique hybrid architecture combining Mamba state-space model layers with traditional attention layers. This hybrid design may not achieve the same throughput as a pure transformer of equivalent size, particularly because Mamba's recurrent computation doesn't batch as efficiently as attention. The assistant implicitly assumes the hybrid architecture doesn't materially impact throughput, which is a reasonable but unverified assumption.

Second assumption: The hourly cost of $44/hr for 8× B200 NVL8 is accurate. This is a market-dependent assumption. B200 instances are scarce and pricing fluctuates wildly. The $44/hr figure likely comes from a specific provider's spot pricing or a negotiated rate. At retail rates, B200 NVL8 instances can cost $80–$160/hr, which would dramatically change the cost-per-token calculation. The assistant's analysis is sensitive to this input — at $80/hr, the B200 would cost $1.48/M tok at 15K throughput, making the PRO 6000 cheaper per token.

Third assumption: The throughput scales linearly with GPU count. The assistant assumes that 8 independent DP instances each achieve the same per-GPU throughput as a single instance. This is generally true for DP inference (no communication between instances), but it assumes perfect load balancing across GPUs and no resource contention (CPU, disk I/O, network for S3 uploads). In practice, there may be some degradation at the system level.

Fourth assumption: The generation job is purely output-bound. The 914K samples have an average input length of ~250 tokens and output length of ~2,500 tokens. The throughput numbers are quoted in "output tok/s," which implicitly assumes input processing is negligible relative to output generation. For autoregressive decoding, this is generally true — input processing is a single forward pass, while output generation requires one forward pass per token. However, the ratio matters: at 250 input tokens versus 2,500 output tokens, input processing accounts for roughly 9% of total computation, which is small but not zero.

What the Message Doesn't Say: Hidden Costs and Trade-offs

The assistant's analysis focuses on the direct compute cost, but the decision involves several hidden factors that the message implicitly acknowledges but doesn't quantify:

Opportunity cost of the PRO 6000s. The assistant notes that "can't train the drafter while generating" — this is the critical insight. The 4× PRO 6000 GPUs are not just a generation platform; they are also the intended training platform. Using them for 16.5 days of generation means 16.5 days of zero training progress. If the drafter training itself takes, say, 5 days, the total project timeline with the PRO 6000 path is 16.5 + 5 = 21.5 days (sequential). With the B200 path, generation takes 1–2 days on rented hardware while the PRO 6000s sit idle or do other work, then training takes 5 days on the PRO 6000s — total timeline: ~7 days. The B200 path saves 14.5 days of calendar time.

Data transfer and setup costs. Switching to a B200 NVL8 instance requires provisioning a new machine, installing software (SGLang, dependencies, model weights), downloading the 914K prompts, and configuring the generation pipeline. The assistant had already prepared for this by writing generate_completions.py and start_sglang_mtp_hicache.sh scripts that work on any hardware — but there's still a non-trivial setup overhead of several hours. The message doesn't account for this engineering time.

The FP8 quantization assumption. The B200 throughput estimate assumes FP8 model weights, which halves memory and doubles throughput versus BF16. But FP8 quantization can introduce accuracy degradation, particularly for the thinking traces that the project needs. If FP8 produces lower-quality completions (shorter or less coherent thinking), the training data quality suffers. The assistant implicitly assumes FP8 is lossless for this purpose, which is a reasonable assumption for generation (as opposed to training) but not proven.

The risk of the B200 being unavailable or unreliable. Cloud GPU instances, especially from smaller providers, can be preempted, suffer hardware failures, or have network issues. A 1.8-day job on rented hardware carries a non-trivial risk of interruption. The PRO 6000s are owned hardware — they run until the job is done, no questions asked. The message doesn't quantify this reliability premium.## The Thinking Process: From Raw Data to Decision

The assistant's reasoning, visible in the preceding messages, reveals a multi-step analytical process that culminates in this cost analysis.

Step 1: Establish real-world baselines. In [msg 7544], the assistant systematically gathered benchmarks from multiple sources: CloudRift's GLM-4.6-FP8 comparison between B200 and RTX PRO 6000 (showing a 4.87× per-GPU advantage), SemiAnalysis's Qwen3.5-27B DP=8 results (95K total tok/s on 8× B200), and the team's own benchmarks (409 tok/s per PRO 6000 GPU). These baselines ground the analysis in empirical data rather than theoretical FLOPs.

Step 2: Understand the architectural advantage. The assistant recognized that DP=8 dramatically outperforms TP=8 for a 27B model because the model fits in a single GPU. This is a non-obvious insight — many practitioners default to tensor parallelism for multi-GPU inference. The assistant's research into the SemiAnalysis results showed that DP was 4× faster than TP for this model size, which fundamentally changes the throughput calculation.

Step 3: Apply workload-specific adjustments. The assistant didn't just take the raw benchmarks — it adjusted for the project's specific workload characteristics. The SemiAnalysis benchmark used short sequences (ISL=1024, OSL=512), while this project has long outputs (average 2,500 tokens). The assistant estimated that only about 33% of the total throughput in the SemiAnalysis benchmark was output tokens, and then further adjusted downward for the output-heavy nature of thinking-mode generation.

Step 4: Calculate cost-per-token. This is the final step, and it's the one the user explicitly asked about. But the assistant doesn't just report the numbers — it contextualizes them. The key finding is that the cost-per-token is roughly the same ($0.87 vs $0.81 per million tokens), but the B200 is 9× faster. This is a remarkable result: the newer hardware doesn't just save time; it saves time at the same cost per unit of work.

Step 5: Frame the decision. The assistant concludes that "the B200 wins on both price and time." This is a stronger claim than "the B200 is faster" — it's saying the B200 is Pareto-optimal: better on both dimensions. The only caveat is the opportunity cost of the PRO 6000s being tied up, which the assistant explicitly calls out.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. The project's data generation requirements. The 914K samples, each requiring ~2,500 output tokens of thinking trace, totaling 2.285 billion tokens. Without this context, the cost-per-token numbers are abstract.
  2. The throughput numbers' provenance. The 1,600 tok/s for PRO 6000 and 15,000–25,000 tok/s for B200 come from earlier analysis in [msg 7544], which itself draws on CloudRift benchmarks, SemiAnalysis reports, and the team's own empirical testing. A reader who hasn't seen that analysis might think these numbers are arbitrary.
  3. The architectural difference between DP and TP. The B200's advantage comes from running 8 independent DP instances, not from raw GPU speed alone. The 4.87× per-GPU advantage from CloudRift's TP benchmark is amplified to a ~9–15× node-level advantage because DP=8 avoids inter-GPU communication overhead.
  4. The FP8 quantization context. The B200 throughput assumes FP8 model weights, which halve memory bandwidth requirements and double throughput. The PRO 6000 benchmarks are in BF16. This is an apples-to-oranges comparison that the assistant implicitly makes by using different precision for each platform.
  5. The hourly pricing assumptions. The $5/hr for 4× PRO 6000 and $44/hr for 8× B200 NVL8 are specific to the user's situation. The PRO 6000 pricing likely reflects the amortized cost of owned hardware (electricity, cooling, depreciation), while the B200 pricing is a cloud rental rate. Different assumptions would yield different conclusions.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

  1. A decision framework for the infrastructure pivot. The message provides a clear, quantitative basis for choosing the B200 NVL8 over the existing PRO 6000s. The user can now make an informed decision rather than a gut feeling.
  2. A cost-per-token benchmark for both platforms. These numbers ($0.87/M tok for PRO 6000, $0.49–0.81/M tok for B200) serve as reference points for future cost modeling. The assistant has effectively established the unit economics of inference on these two platforms.
  3. A total job cost estimate. The projected $1,120–$1,988 total cost for generating 2.285 billion tokens gives the user a concrete budget figure. This is essential for procurement approval and resource planning.
  4. A wall-time comparison. The 1.0–1.8 day estimate for B200 versus 16.5 days for PRO 6000 quantifies the time savings. This is arguably more important than the cost comparison — the project's timeline depends on this data being generated before training can begin.
  5. An implicit recommendation. The assistant's framing — "the B200 wins on both price and time" — is a clear recommendation to proceed with the cloud rental, despite the higher hourly rate.

Mistakes and Potential Pitfalls

While the analysis is sound, several potential issues deserve mention:

The throughput numbers may not be simultaneously achievable. The B200's 15,000 tok/s assumes sustained throughput over 1.8 days of continuous generation. In practice, throughput can degrade over time due to memory fragmentation, KV cache management overhead, or S3 upload bottlenecks. The assistant's scripts include progress tracking and resume support, which mitigates this risk, but the throughput estimate doesn't account for degradation.

The cost comparison ignores data transfer costs. Uploading 2.285 billion tokens of generated data to S3 (as the project does) incurs egress and storage costs. From a cloud provider, egress is often free, but from a rented bare-metal machine, it may not be. The $44/hr likely doesn't include bandwidth costs.

The $5/hr for PRO 6000 may be an underestimate. If this is truly the amortized cost (electricity + cooling + hardware depreciation / uptime), it's reasonable. But if the user is paying for a dedicated server at $5/hr for 4 GPUs, that's remarkably cheap — RTX PRO 6000 Blackwell cards retail for ~$10,000 each, so $5/hr for 4 cards implies a 10-month payback period at 24/7 utilization, which is aggressive.

The analysis doesn't consider the value of engineering time. Setting up the B200 instance, debugging any issues, and monitoring the 1.8-day run requires engineering attention. If the engineer's time is valued at $100/hr, the setup and monitoring overhead could add $200–$500 to the B200 path that isn't captured in the cost-per-token calculation.

Conclusion

Message 7560 is a masterclass in applied cost analysis for ML infrastructure. It takes a simple question — "compare these two hourly rates" — and produces a nuanced, multi-dimensional answer that accounts for throughput, wall time, total cost, and opportunity cost. The assistant's reasoning is grounded in real benchmarks, informed by architectural insights (DP vs TP), and contextualized for the specific workload.

The message's power lies not in its arithmetic (which is trivial) but in the weeks of work that produced the throughput numbers feeding into it. Every number in the table represents dozens of benchmarks, hours of research, and hard-won empirical knowledge. The cost-per-token calculation is the tip of an iceberg — visible, simple, and elegant, but supported by a massive body of work beneath the surface.

For the project, this message marks the decisive moment. The old path (16.5 days on owned hardware, blocking training) is abandoned. The new path (1–2 days on rented B200s, freeing the PRO 6000s for training) is chosen. The economics justify the pivot, and the project can move forward with confidence. The DFlash drafter will be trained, the speculative decoding will accelerate inference, and it all traces back to this moment — a few lines of "quick math" that changed the trajectory of the entire effort.