Pivoting to Qwen3.5-122B: A Strategic Model Swap Under Infrastructure Constraints

At message index 6100 in this opencode session, the user issued a concise but consequential instruction:

Setup https://huggingface.co/Qwen/Qwen3.5-122B-A10B fp16 tp4, tool calling, thinking, MTP like previous. Deploy (don't use /data), run benchmarks

This single message represents a major strategic pivot point in the conversation. To understand its full significance, we must examine the context that led to it, the assumptions embedded within it, and the cascade of decisions it triggered.

The Context: Why This Message Was Written

The immediate preceding messages reveal a critical infrastructure and quality assessment. The user had just instructed the assistant to inspect what was stored on /data — a large ZFS volume mounted on the LXC container — because it was "going to be retired soon to cold backup, unmounted." The assistant's audit ([msg 6090]) revealed that the primary occupant was the Qwen3.5-397B-A17B-NVFP4 model (223 GB), along with 4.8 TB of EAGLE-3 training data from earlier Kimi-K2.5 work.

The user's response was blunt and decisive ([msg 6091]): after confirming the filesystem layout with df -h, they said "rm qwen, in use it's actually very low quality, we'll be deploying a similarly sized model." This is the direct antecedent to message 6100. The 397B parameter model — despite its enormous size and the weeks of effort spent deploying it with FP4 quantization, SM120 patches, and custom SGLang builds — was deemed unsatisfactory in output quality. The user wanted to replace it with something better.

The "similarly sized model" turned out to be the Qwen3.5-122B-A10B — a 122 billion parameter Mixture-of-Experts model with only 10 billion active parameters per token. Despite having roughly one-third the total parameters of the 397B, this model uses native BF16 precision (no quantization) and represents a fundamentally different architectural approach: hybrid GDN (Gated Dense Network) with 48 layers, 256 experts, 8 routed plus 1 shared expert per token.

The Reasoning and Motivation

The user's motivation appears to be a quality-versus-size tradeoff. The 397B NVFP4 model, while enormous, was quantized to FP4 — a 4-bit floating point format that aggressively compresses weights. Quantization at this extreme level inevitably introduces accuracy degradation, which the user experienced as "very low quality." The 122B model, by contrast, runs in native BF16 (16-bit) precision. Despite having fewer total parameters, its weights are preserved with much higher fidelity. For many tasks — especially reasoning, tool calling, and structured output — a smaller model at full precision can outperform a larger quantized model.

The user also explicitly wanted to preserve the advanced inference features from the previous deployment: tool calling (the model can invoke external functions), thinking/reasoning (the model produces chain-of-thought before answering), and MTP (Multi-Token Prediction, also known as speculative decoding, where a smaller draft model predicts multiple future tokens in parallel to accelerate inference). The phrase "like previous" is a shorthand that assumes the assistant understands the exact configuration flags used for the Kimi-K2.5 EAGLE-3 setup — a reasonable assumption given the extensive history.

Assumptions Embedded in the Message

This message makes several implicit assumptions:

  1. VRAM sufficiency: The user assumes 4× 96 GB GPUs (384 GB total) can accommodate a 122B BF16 model (~250 GB) plus KV cache. This leaves ~134 GB for cache — a reasonable margin.
  2. Storage availability: The user assumes /shared (895 GB free) can hold the model, since /data is forbidden. The model is ~234 GB on disk, so this fits comfortably.
  3. Infrastructure readiness: The GPU split (4 GPUs for LXC, 4 for VM) was already configured in the preceding messages (<msg id=6077-6084>), so TP=4 is the natural choice.
  4. MTP compatibility: The user assumes the 122B model supports the same MTP/speculative decoding setup as the Kimi-K2.5. This turns out to be correct — the model's config.json includes mtp_num_hidden_layers: 1 — but the implementation details differ enough to cause a crash on first attempt.
  5. The assistant's knowledge: The user assumes the assistant can infer the complete SGLang command-line flags from the phrase "MTP like previous." The assistant does so correctly, adding --speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4, but misses the critical SGLANG_ENABLE_SPEC_V2=1 environment variable and --mamba-scheduler-strategy extra_buffer flag needed for hybrid GDN models with MTP.

Input Knowledge Required

To fully understand this message, a reader needs:

Output Knowledge Created

This message triggers a multi-hour deployment workflow that produces:

The Thinking Process Visible in the Assistant's Response

The assistant's response to this message (visible in the subsequent tool calls) reveals a structured, methodical approach. First, it researches the model by fetching the Hugging Face page and config.json (<msg id=6102, 6109>), extracting key parameters: 48 layers, 256 experts, BF16 dtype, MTP support with 1 hidden layer. It estimates VRAM: "125B params × 2 bytes = ~250 GB. With 4× 96 GB = 384 GB total, that leaves ~134 GB for KV cache. Should work."

The assistant then checks available storage on /shared (895 GB) and initiates a parallel download while preparing the service file. It carefully adapts the previous 397B NVFP4 service: removing FP4-specific flags (--quantization modelopt_fp4, --moe-runner-backend flashinfer_cutlass, --fp4-gemm-backend flashinfer_cudnn), updating the model path, and adding MTP flags.

However, the assistant makes an incorrect assumption about MTP compatibility. When the first server attempt crashes with an error about MTP requiring --mamba-scheduler-strategy extra_buffer and SGLANG_ENABLE_SPEC_V2=1, the assistant quickly diagnoses the issue from the journal logs and fixes it. This reveals a gap in the assistant's knowledge: while it knew the MTP flags from the previous setup, it didn't know that hybrid GDN models (which combine Mamba-like selective state updates with attention) require the speculative decoding v2 path.

Mistakes and Incorrect Assumptions

The most notable mistake is the MTP configuration gap. The assistant assumed that the same --speculative-algo NEXTN flags that worked for the Kimi-K2.5 EAGLE-3 setup would work unchanged for the Qwen3.5-122B. But the 122B uses a hybrid GDN architecture (alternating linear and full attention layers with Mamba-like selective state updates), which requires the speculative decoding v2 infrastructure. The error message from SGLang explicitly states this requirement, and the assistant corrects it in the next round.

A subtler issue is the NCCL hang that occurs after the MTP fix. The server gets stuck at "Init torch distributed begin" for over 10 minutes with zero GPU memory usage. The assistant initially suspects the MTP configuration but later, prompted by the user's question about driver updates ([msg 6138]), discovers a driver version mismatch: the Proxmox host runs NVIDIA driver 590.48.01, but the container still has userspace libraries from version 565. This mismatch can cause NCCL initialization failures, especially under the SEV-SNP IOMMU configuration that was previously shown to break P2P DMA.

Conclusion

Message 6100 is a pivotal moment that encapsulates the iterative nature of real-world ML infrastructure work. It represents a strategic decision — abandon a massive quantized model for a smaller native-precision one — driven by empirical quality assessment. It tests assumptions about model compatibility, storage constraints, and infrastructure readiness. And it triggers a deployment workflow that, while encountering expected bumps (MTP configuration, NCCL hangs, driver mismatches), ultimately succeeds through systematic debugging. The message is a masterclass in concise technical specification: in 18 words, it communicates a model choice, precision requirement, parallelism strategy, feature set, infrastructure constraint, and validation criteria — all of which the assistant must unpack and execute.