Pushing Against the Ceiling: When Scaling an Inference Service Reveals Hidden Limits

In the high-stakes world of large language model inference deployment, every tok/s matters. The assistant had just achieved a remarkable milestone: deploying Kimi K2.6 with DFlash speculative decoding on 8× NVIDIA B300 SXM6 GPUs connected via NVLink, reaching 303 tok/s at single-request concurrency — a 2.15× speedup over the autoregressive baseline. With NVLS (NVIDIA Scalable Hierarchical Aggregation and Broadcast) enabled, throughput at C=128 hit 4723 tok/s. The system was humming, but the power draw told a tantalizing story: only 450W out of a 1100W ceiling. If there was headroom, why not push further?

This message captures the moment when the assistant decided to find the ceiling — and the unexpected failure that followed. It is a study in engineering ambition, the assumptions we make when success breeds confidence, and the hard lessons that come when a system refuses to cooperate.

The Context: A Winning Configuration

The path to this moment had been long and winding. Earlier in the session, the assistant had systematically benchmarked four parallelism strategies — TP8, PP8, EP8, and EP4 — on the PCIe-connected RTX PRO 6000 Blackwell GPUs, confirming the user's intuition that expert parallelism avoids PCIe AllReduce bottlenecks. But on the B300 SXM6 machine with NVLink, the calculus changed. NVLink's high-bandwidth GPU-to-GPU communication made tensor parallelism (TP8) competitive again, and when combined with NVLS — a hardware-accelerated multicast for AllReduce — TP8 became the clear winner.

The benchmark results that immediately preceded this message were impressive. At concurrency 1, TP8+NVLS DDTree achieved 303.2 tok/s, compared to 132.7 tok/s for the autoregressive baseline. At concurrency 64, it reached 4006.7 tok/s. And at concurrency 128, throughput scaled to 4723.2 tok/s — a 24% improvement over the C=64 result, confirming that the system had been under-loaded at lower concurrency levels. The assistant had also confirmed that the GPUs were running at 100% utilization but drawing only ~360–450W out of a possible 1100W, a classic signature of a memory-bandwidth-bound workload.

The assistant's reasoning in this message reflects a careful analysis of these results. It notes the ~5-6% gain from NVLS, the power climb to ~450W, and the fundamental insight that "the INT4 MoE decode is fundamentally memory-bandwidth constrained rather than compute-bound—the tensor cores aren't fully utilized because the arithmetic intensity is low." This is a crucial observation: the tensor cores are idle because the arithmetic intensity of INT4 weight-only decoding is low — most of the time is spent waiting for HBM to deliver expert weights.

The Reasoning: Why Push Harder?

The assistant's reasoning section reveals a logical chain that any performance engineer would recognize:

  1. Observation: Power consumption is only 450W out of 1100W available.
  2. Inference: There is headroom to push further.
  3. Evidence: Throughput scaled from 3795 tok/s at C=64 to 4723 tok/s at C=128.
  4. Hypothesis: Higher concurrency will continue to improve throughput by amortizing HBM weight reads across more tokens.
  5. Decision: Increase max-running-requests from 128 to 256 and benchmark at C=128 and C=256. This is textbook performance tuning: find the bottleneck, push against it, see where it breaks. The assistant had already confirmed the bottleneck was memory bandwidth, not compute. Higher concurrency means more tokens sharing the same weight-loading cost, which should improve throughput — up to the point where either the GPU's memory bandwidth is fully saturated or some other resource (like CPU-side request handling) becomes the bottleneck. The assistant also makes an important architectural decision here: "I'm confident TP8 + NVLS is the winning configuration on B300, and the EP + DDTree + cudagraph approach isn't worth pursuing further given the NVLink advantages." This closes the loop on the earlier EP8 experiments, which had failed with CUBLAS errors and NoneType bugs. On NVLink, TP8's AllReduce is cheap enough that the complexity of expert parallelism isn't justified. This is a sound engineering judgment, grounded in empirical evidence from the benchmark matrix. The reasoning also contains a subtle but important shift in framing. Earlier in the session, the assistant had been debugging why EP8+DDTree+cudagraph crashed. Now, having found a working configuration, the focus shifts from "why does this fail?" to "how far can we push this?" This is the natural rhythm of engineering work: diagnose, fix, optimize. The assistant had successfully diagnosed and fixed the TP8+NVLS path, and was now in optimization mode.

The Execution: Bumping max-running-requests

The tool call that follows the reasoning is a bash command with several stages:

First, it modifies the systemd service file:

sed -i 's|--max-running-requests 128|--max-running-requests 256|' /etc/systemd/system/sglang-k26-ddtree-nvls.service
systemctl daemon-reload && systemctl restart sglang-k26-ddtree-nvls.service

Then it enters a polling loop, checking every 15 seconds whether the service is ready by making a lightweight curl request to the /v1/chat/completions endpoint. The loop runs up to 50 iterations — a total of 12.5 minutes of waiting.

Once the service is (presumed) ready, the assistant launches a benchmark harness:

timeout 1200 ssh ... /root/venv/bin/python bench_ddtree_matrix.py --label 'B300 TP8+NVLS maxreq256' --contexts 60 --concurrency 128 256 --max-tokens 512 --out /root/bench_results_opt/B300_nvls_c256.json

Finally, it attempts to sample power under load by firing 200 concurrent requests and reading GPU power draw via nvidia-smi.

The structure is clean and methodical: change configuration, wait for readiness, benchmark, measure power. It's the same pattern the assistant had used successfully throughout the session.## The Failure: Connection Refused and Power Collapse

The results are stark and immediate. The polling loop never reports readiness — instead, after an unspecified number of iterations, it outputs FAILED. The benchmark then runs but both concurrency levels return ERROR: URLError: <urlopen error [Errno 111] Connection refused>. The service is down. The power sample shows a paltry ~186W — barely above idle — confirming that the GPUs are doing essentially nothing.

What went wrong? The assistant's reasoning had made several assumptions that proved incorrect:

Assumption 1: The service would restart successfully. The assistant had successfully restarted the service many times before — when deploying DDTree, when adding NVLS, when switching between TP8 and EP8 configurations. Each time, the service came back within 2-3 minutes. This time, it didn't. The max-running-requests parameter controls how many concurrent requests the SGLang server will accept before queuing. Bumping it from 128 to 256 changes memory allocation patterns — the server pre-allocates KV cache slots and other buffers proportional to this value. On an 8-GPU system with 275 GB per GPU, doubling the request limit might push memory allocation into a regime that triggers an OOM or other initialization failure.

Assumption 2: The polling loop would catch the failure. The loop checks systemctl is-active and the curl endpoint every 15 seconds. But if the service crashes during initialization — say, during model weight loading or CUDA graph capture — the systemd state might briefly show "activating" before flipping to "failed." The loop's logic checks for failed explicitly, but if the transition is fast enough, it might miss it. More importantly, the loop doesn't print the service status on every iteration — it only prints on every 6th iteration (90 seconds) and when the status is failed. The user sees FAILED but not the journalctl output that would explain why.

Assumption 3: The benchmark harness would gracefully handle an unavailable service. The benchmark script bench_ddtree_matrix.py catches the URLError and records it as an error entry in the output JSON. This is good engineering — the harness doesn't crash, it records the failure. But the power sample that follows is meaningless because there are no requests being processed — the 200 concurrent curl commands are all failing with connection refused, so the GPUs are idle.

Assumption 4: The power draw would be informative. The assistant fires 200 concurrent requests in the background, waits 12 seconds, then reads nvidia-smi. But if the service is down, those requests all fail immediately, and the GPUs never see any work. The power reading of ~186W is the idle power of 8× B300 GPUs, confirming the service is dead.

The Deeper Mistake: Not Checking the Journal

The most significant mistake is visible in what the assistant didn't do. In previous failure modes — the EP8+NVLS CUBLAS crash, the EP8 NoneType error — the assistant immediately fetched journalctl output to diagnose the root cause. Here, the assistant does not. The bash command includes the polling loop and the benchmark, but there is no journalctl call to capture the service failure reason.

Why? The reasoning section provides a clue: the assistant was "confident" in the TP8+NVLS configuration. It had worked before. The only change was max-running-requests from 128 to 256. This seemed like a safe, incremental change — just increasing a buffer size. The assistant likely expected the service to come up without issue, and the failure was unexpected enough that the diagnostic step was omitted.

This is a classic cognitive bias in engineering: past success with a configuration leads to overconfidence in its stability. The assistant had successfully restarted the DDTree service multiple times — when adding NVLS, when switching from EP8 back to TP8, when testing different parallelism strategies. Each time, the service came up. The assumption that "it will work this time too" was implicit and unexamined.

There's also a structural issue with the polling loop. The loop checks systemctl is-active and the curl endpoint, but it doesn't capture the service's stdout/stderr. If the service crashes during initialization, the systemd journal contains the error message, but the loop doesn't retrieve it. The FAILED message is printed, but without context. In previous iterations, the assistant had separate bash commands to fetch journalctl output — here, that step was merged into the polling loop but only triggers on failed. However, the journalctl command in the loop only runs if the status is failed, and it only shows the last 20 lines filtered for error keywords. If the error is something subtle — like a memory allocation failure that doesn't produce a Python traceback — it might be missed.

The Input Knowledge Required

To fully understand this message, the reader needs knowledge across several domains:

System administration: Understanding systemd service management (systemctl daemon-reload, restart, is-active), the service file format, and the polling loop pattern for service readiness.

GPU inference architecture: Understanding tensor parallelism (TP8), NVLS (NVIDIA's hardware-accelerated AllReduce), speculative decoding with DDTree, and the concept of max-running-requests as a concurrency limiter that affects KV cache pre-allocation.

Performance analysis: Understanding the relationship between GPU utilization, power draw, and memory-bandwidth-bound workloads. The insight that 100% utilization with only 450W out of 1100W indicates a memory-bound (not compute-bound) workload is non-trivial.

Benchmark methodology: Understanding the benchmark harness's output format (context length, concurrency, aggregate tok/s, per-request tok/s, wall time, prompt tokens) and what each metric reveals about system behavior.

The conversation history: The reader needs to know that TP8+NVLS had been working successfully, that EP8 had failed with CUBLAS errors, and that the assistant had already established the memory-bandwidth-bound nature of the workload. Without this context, the decision to push concurrency higher seems arbitrary.

The Output Knowledge Created

This message creates several important pieces of knowledge, even in failure:

  1. A confirmed upper bound for max-running-requests: The failure at 256 establishes that the working configuration (128) is near or at the limit for this model and hardware combination. Future experiments should not exceed this value without understanding why 256 fails.
  2. A negative result that validates the previous configuration: The failure reinforces that the TP8+NVLS config with maxreq=128 was already at a good operating point. The 4723 tok/s at C=128 is a validated peak, not an intermediate point on a curve that continues upward.
  3. A diagnostic gap: The lack of journalctl output means the root cause of the failure is unknown. It could be OOM during KV cache allocation, a CUDA graph capture failure at larger batch sizes, or a SGLang internal limit. This gap is itself knowledge — it tells the next iteration to instrument the service startup more carefully.
  4. A pattern for future experiments: The polling loop and benchmark harness are reusable. The failure mode (service doesn't come up, benchmark records URLError) is documented and can be recognized in future runs.

The Thinking Process: Visible Reasoning

The assistant's reasoning section is particularly revealing because it shows the thought process in real-time. The assistant:

  1. Reviews the benchmark results and identifies the key metrics (303 tok/s at C=1, 4723 at C=128, 450W power).
  2. Draws the architectural conclusion that the workload is memory-bandwidth-bound.
  3. Makes the strategic decision to abandon EP8+DDTree on NVLink.
  4. Formulates the hypothesis that higher concurrency will improve throughput.
  5. Designs the experiment: increase maxreq to 256, test C=128 and C=256. The reasoning is linear and confident. There's no hedging, no "this might fail," no contingency plan for what to do if the service doesn't come up. The assistant is in optimization mode, not debugging mode, and the reasoning reflects that mindset. This is both a strength and a weakness. The strength is that it enables fast iteration — the assistant doesn't second-guess itself. The weakness is that it can lead to blind spots, as the failure demonstrates. The assistant's confidence in the TP8+NVLS configuration — earned through multiple successful restarts — led to the assumption that any change would be safe.

The Broader Lesson: Success Breeds Assumptions

This message is a microcosm of a universal engineering experience: the moment when a winning streak meets an unexpected wall. The assistant had been on a roll — fixing CUDA toolkit issues, benchmarking parallelism strategies, deploying DDTree, achieving 2.15× speedup, scaling to 4723 tok/s. Each experiment built on the last, each success validated the approach. The natural next step was to push harder.

But the system had limits that weren't visible from the outside. The max-running-requests parameter is not just a concurrency limiter — it's a memory allocation directive. Doubling it from 128 to 256 doesn't just allow more requests; it pre-allocates more KV cache slots, more CUDA graph buffers, more internal state. On an 8-GPU system running a 590 GB model with speculative decoding, those allocations can push past hardware limits in ways that aren't obvious from the parameter name.

The failure is not a mistake in the traditional sense — it's a boundary discovery. The assistant learned that the system's ceiling is somewhere between 128 and 256 max-running-requests. That's valuable information. The only real mistake was not capturing the diagnostic output that would explain why 256 fails. With that information, the assistant could distinguish between "OOM during KV cache allocation" (which might be fixable by reducing mem-fraction-static) and "CUDA graph capture failure at larger batch sizes" (which would require code changes) and "SGLang internal limit" (which would be a hard cap).

The message ends with the system in a broken state — the service is down, the benchmark failed, the power reading is meaningless. But this is not the end of the story. The next message in the conversation will likely diagnose the failure and either fix it or adjust the approach. That's the nature of systems engineering: you push, you break, you learn, you fix. This message captures the "push" and the "break" — the learning and fixing come next.