The Capacity Calculation: A Pivotal Moment in Model Deployment

In the sprawling arc of an opencode coding session that spans infrastructure setup, driver installation, model downloads, and speculative decoding research, some messages serve as quiet hinges — moments where the trajectory of the entire session pivots. Message [msg 6810] is precisely such a hinge. It is the moment when the assistant, having successfully migrated the Qwen3.6-27B model to a new host (kpro5) and downloaded its 52GB of weights, pauses to calculate whether the hardware can actually run what has been assembled. This message is a study in capacity planning, architectural reasoning, and the assumptions that precede failure.

The Context of the Moment

To understand why this message matters, one must appreciate what preceded it. The assistant had just completed a complex migration: moving the Qwen3.6-27B deployment from the decommissioned kpro6 host to kpro5, installing NVIDIA driver 580.126.09 on the host, unbinding two RTX A6000 GPUs from vfio-pci, updating the CT129 LXC container configuration, and installing matching userspace libraries inside the container. The model itself — a 27.78-billion-parameter BF16 model weighing approximately 55GB — had been downloaded after a stalled snapshot_download process was killed and resumed via huggingface-cli. All 15 safetensor shards were confirmed present at 52GB on disk.

With the infrastructure in place and the model on disk, the assistant now faced the critical question: Can this model actually run on two RTX A6000 GPUs with 48GB each? The answer was not obvious. The model uses Gated DeltaNet, a hybrid architecture combining linear attention layers with full attention layers (every fourth layer is full attention). It supports Multi-Token Prediction (MTP) with draft heads. The assistant needed to determine the right parameters for the SGLang server launch — context length, memory fraction, tensor parallelism, and speculative decoding configuration.

The Reasoning Process Visible in the Message

The message opens with a chain of explicit reasoning. The assistant writes: "Now let me figure out the right parameters. 2× A6000 = 98GB VRAM. Model is ~55GB BF16 with TP=2. With MTP draft heads, we need some extra memory. Let me calculate context length."

This is a textbook capacity planning exercise. The assistant is performing a mental budget:

Total VRAM: 2 × 48GB = 96GB (the assistant says 98GB, likely accounting for slight differences in reported vs. actual memory)
Model weights: ~55GB in BF16 with tensor parallelism across 2 GPUs, meaning each GPU holds roughly half the weights (~27.5GB)
MTP draft heads: An unknown additional memory cost for the speculative decoding heads
KV cache: The remaining memory after weights and overhead must accommodate the key-value cache for the chosen context length
Torch compile overhead: Additional memory used by JIT-compiled CUDA graphs The assistant then makes an architectural assumption: "Qwen3.6-27B uses Gated DeltaNet (hybrid linear+full attention) — same as Qwen3.5, so SGLang should handle it with triton attention backend." This assumption is critical. The assistant is asserting that because the model's model_type is qwen3_5 (as confirmed in the config.json output), SGLang's existing support for Qwen3.5 models will transparently extend to Qwen3.6. The triton attention backend is expected to handle the hybrid linear+full attention pattern correctly. The bash command that follows — reading the model's config.json — is the assistant's attempt to ground-truth its assumptions. It wants to verify that MTP heads are indeed present in the model configuration before committing to a launch command that enables speculative decoding.

Assumptions and Their Consequences

This message is built on several assumptions, some of which prove incorrect in the subsequent messages:

Memory budget assumption: The assistant assumes that 0.85 memory fraction (the --mem-fraction-static parameter) will leave enough room for model weights, MTP heads, KV cache, and torch compile overhead. In the very next message ([msg 6813]), this assumption fails catastrophically with a RuntimeError: Not enough memory error, forcing the assistant to reduce context length from 65536 to 32768 and eventually to debug further.
SGLang version compatibility: The assistant assumes that SGLang 0.5.9 (the version installed) will handle Qwen3.6 correctly because it handles Qwen3.5. The model card recommends SGLang >=0.5.10, but the assistant could not upgrade due to a flash-attn-4 dependency conflict. This assumption proves partially correct — the model loads — but the assistant will later discover that SGLang 0.5.9 produces degenerate output with this model, requiring an upgrade to 0.5.11.
MTP head presence: The assistant assumes MTP heads exist in the model. The config.json output (partially visible in the message) confirms mtp_num_hidden_layers: 1 in the subsequent message ([msg 6811]), validating this assumption.
Triton attention backend suitability: The assistant assumes the triton backend will handle Gated DeltaNet correctly. This is a reasonable assumption given the model's qwen3_5 architecture tag, but it glosses over the complexity of hybrid attention kernels.

Input Knowledge Required

To fully understand this message, a reader needs:

Knowledge of GPU VRAM budgeting: Understanding that a 55GB BF16 model with TP=2 distributes ~27.5GB per GPU, leaving ~20.5GB per GPU for KV cache and overhead at 0.85 memory fraction
Familiarity with MTP (Multi-Token Prediction): Knowing that MTP adds draft heads that consume additional memory proportional to the number of draft tokens and layers
Understanding of Gated DeltaNet: Recognizing that this hybrid architecture (linear attention + full attention at intervals) has different memory characteristics than pure transformer architectures — linear attention requires O(1) KV cache per layer rather than O(sequence_length)
SGLang server parameters: Knowing what --mem-fraction-static, --tp-size, --context-length, and --speculative-algo NEXTN control
The hardware topology: Two RTX A6000 GPUs with NVLink (48GB each, Ampere architecture SM86)

Output Knowledge Created

This message produces several forms of knowledge:

The config.json inspection: The bash command reads the model's configuration, revealing the qwen3_5 model type, the hybrid attention pattern (full_attention_interval: 4), and the presence of MTP configuration. This output becomes the basis for the launch command in the next message.
A capacity planning template: The assistant's reasoning — "98GB VRAM, 55GB model, TP=2, MTP overhead" — establishes a reusable mental model for deploying large language models on multi-GPU setups.
An implicit test hypothesis: The assistant is implicitly testing whether SGLang 0.5.9 can serve Qwen3.6-27B. The launch command that follows in [msg 6811] is the experiment; the OOM error in [msg 6813] is the result.

The Thinking Process and Its Significance

What makes this message remarkable is what it reveals about the assistant's cognitive process. The assistant is not blindly launching a server with default parameters. It is actively reasoning about resource constraints, making architectural inferences from model type compatibility, and verifying those inferences against the model configuration. The phrase "Let me figure out the right parameters" signals a deliberate planning phase.

The assistant also demonstrates awareness of the model's architectural novelty. Gated DeltaNet is a relatively new architecture (introduced in the Qwen3.5 family), and the assistant explicitly notes that it uses "hybrid linear+full attention." This shows that the assistant understands the implications of this architecture for memory usage — linear attention layers require far less KV cache than full attention layers, which is why the assistant is optimistic about supporting a reasonable context length despite the tight memory budget.

The message also reveals a subtle tension between confidence and verification. The assistant states "SGLang should handle it with triton attention backend" — a confident assertion — but immediately follows with "Let me check if MTP heads are present in the model config" — a verification step. This pattern of assert-then-verify is characteristic of the assistant's operating style throughout the session.

The Broader Narrative Arc

Message [msg 6810] sits at the boundary between preparation and execution. Everything before it was about setting up the environment — installing drivers, configuring the container, downloading the model. Everything after it is about the actual deployment — launching the server, hitting OOM errors, tuning parameters, and eventually discovering that SGLang 0.5.9 produces degenerate output for this model. The message is the moment where theory meets practice, where the assistant's mental model of the system collides with the messy reality of GPU memory management and software compatibility.

In this sense, the message is emblematic of the entire opencode session's theme: the gap between what should work in principle and what actually works in practice. The assistant's careful capacity calculation was correct in its arithmetic but wrong in its assumptions about memory overhead. The model should have fit in 96GB with 0.85 memory fraction, but it didn't — and that discrepancy drove the next several rounds of debugging.

This message, brief as it is, captures the essence of deploying large language models on constrained hardware: the constant negotiation between ambition (long context, speculative decoding, torch compile) and reality (finite VRAM, software bugs, architectural quirks). It is a moment of quiet competence before the storm of errors that follows.