The SCP Pivot: Overcoming SSH Heredoc Quoting to Deploy MTP Speculative Decoding

Introduction

In the midst of a high-stakes machine learning deployment session, message [msg 7499] represents a quiet but critical inflection point: the moment when a persistent technical obstacle—shell quoting mangled across an SSH connection—was finally sidestepped through a simple but effective workaround. The message itself is deceptively brief: a single bash command that uses scp to upload a launch script to a remote server, followed by an SSH command to verify its contents. But the story behind this message reveals a cascade of failed attempts, a deepening understanding of the system's constraints, and a series of architectural decisions about how to configure Multi-Token Prediction (MTP) speculative decoding for the Qwen3.6-27B model on a memory-constrained GPU.

The Context: An OOM Nightmare

To understand why this message matters, we must trace the thread of failures that preceded it. The session had been working toward deploying Qwen3.6-27B with MTP speculative decoding using SGLang on a 4× RTX PRO 6000 Blackwell node. The goal was to achieve high-throughput generation for a large-scale dataset creation pipeline—902,087 completions needed to be generated with full thinking traces.

The core problem was memory. Each RTX PRO 6000 GPU has 96 GB of VRAM, and the Qwen3.6-27B model alone consumes approximately 51 GB in BF16. This leaves roughly 43 GB for KV caches, Mamba state caches, and speculative decoding buffers. MTP speculative decoding, however, requires the extra_buffer Mamba scheduler strategy, which doubles the Mamba state cache allocation. Combined with speculative draft tokens and the need to support hundreds of concurrent requests, the memory budget was being stretched to its breaking point.

The assistant had tried multiple configurations: reducing mem-fraction-static from 0.95 to 0.90, enabling hierarchical cache with 200 GB of CPU memory overflow, disabling the extra_buffer strategy (which caused a ValueError because speculative decoding is incompatible with the no_buffer strategy when radix cache is enabled), and capping max-running-requests at various levels. Each attempt either failed silently or produced an OOM error. The server logs consistently showed that the Mamba cache allocation was exceeding available GPU memory.

The Heredoc Quoting Problem

The immediate technical obstacle that message [msg 7499] addresses is not about GPU memory at all—it is about shell quoting. In the preceding messages ([msg 7489] through [msg 7498]), the assistant had been attempting to write a launch script to the remote server using SSH heredocs. The approach was straightforward: use ssh ... "cat > script.sh << 'EOF' ... EOF" to create a file with the desired content. However, the script contained shell variables ($1, $2) and the heredoc approach was failing to preserve them correctly.

In [msg 7497], the assistant verified the script content and discovered the problem:

#!/bin/bash
export LD_LIBRARY_PATH=/usr/local/cuda/lib64
export SGLANG_ENABLE_SPEC_V2=1
export CUDA_VISIBLE_DEVICES=$1

exec /workspace/dflash/venv/bin/python3 -m sglang.launch_server \
  --model-path /workspace/dflash/models/Qwen3.6-27B \
  --reasoning-parser qwen3 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --mem-fraction-static 0.90 \
  --host 0.0.0.0 \
  --port $2 \
  --context-length 8192 \
  --trust...

The script was missing the critical --mamba-scheduler-strategy extra_buffer, --max-mamba-cache-size 24, and --mamba-full-memory-ratio 0.4 flags that the assistant had carefully added in [msg 7496]. The heredoc had silently dropped or mangled these additions. The assistant's reasoning in [msg 7498] correctly diagnosed the issue: "Heredoc quoting is getting mangled over SSH."

The Message Itself: A Pragmatic Workaround

Message [msg 7499] implements a clean solution: instead of fighting with SSH quoting, write the script to a local file and use scp to transfer it. The command is:

scp -P 19248 /data/dflash/scripts/launch_mtp.sh root@154.59.156.20:/workspace/dflash/scripts/ && ssh -p 19248 root@154.59.156.20 "chmod +x /workspace/dflash/scripts/launch_mtp.sh && cat /workspace/dflash/scripts/launch_mtp.sh"

The output confirms success—the script now contains the correct configuration:

#!/bin/bash
export LD_LIBRARY_PATH=/usr/local/cuda/lib64
export SGLANG_ENABLE_SPEC_V2=1
export CUDA_VISIBLE_DEVICES=$1

exec /workspace/dflash/venv/bin/python3 -m sglang.launch_server \
  --model-path /workspace/dflash/models/Qwen3.6-27B \
  --reasoning-parser qwen3 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --mamba-scheduler-strategy extra_buffer \
  --max-mamba-cache-size 24 \
  --mamba-full-memory-rat...

The --mamba-scheduler-strategy extra_buffer flag is now present, along with --max-mamba-cache-size 24 (capping the Mamba cache at 24 GB) and --mamba-full-memory-ratio (which controls how much of the available memory is pre-allocated for the Mamba state). These three parameters together represent the assistant's strategy for fitting MTP speculative decoding into the available GPU memory: keep the required extra_buffer strategy, but cap its memory consumption explicitly rather than letting it use the default allocation.

Technical Decisions Embedded in the Script

The script encodes several important architectural decisions:

  1. Mamba Scheduler Strategy: extra_buffer is required because SGLang's speculative decoding implementation for Qwen3.5-based models (which Qwen3.6 inherits from) is incompatible with the no_buffer strategy when radix cache is enabled. The extra_buffer strategy pre-allocates a larger Mamba state buffer to handle speculative decoding's need for multiple candidate states simultaneously.
  2. Mamba Cache Cap: --max-mamba-cache-size 24 limits the Mamba state cache to 24 GB. This is a compromise: it provides enough space for the doubled buffer requirement of extra_buffer while leaving approximately 19 GB for the KV cache and other overhead (model: 51 GB + Mamba: 24 GB = 75 GB out of 96 GB, leaving ~21 GB for KV cache and speculative buffers).
  3. Memory Fraction: --mamba-full-memory-ratio 0.4 (visible in the truncated output as --mamba-full-memory-rat...) controls what fraction of the memory pool is allocated to the Mamba full state versus the radix cache. A lower ratio means less memory is reserved for Mamba states, which constrains the number of concurrent speculative decoding slots but prevents OOM.
  4. Speculative Decoding Parameters: --speculative-algorithm EAGLE, --speculative-num-steps 3, --speculative-eagle-topk 1, and --speculative-num-draft-tokens 4 configure the MTP engine to generate 4 draft tokens per step using the EAGLE (Efficient AGgregation of Language Embeddings) architecture with top-1 sampling and 3 lookahead steps.

The Thinking Process

The assistant's reasoning in the preceding messages reveals a methodical debugging process. When the initial MTP launch failed with an OOM error, the assistant systematically explored the configuration space:

First, it identified that the extra_buffer strategy was the root cause of excessive memory consumption. Then it discovered that removing extra_buffer caused a hard error because speculative decoding requires it. This created a dilemma: the required feature caused OOM, but disabling it was not allowed.

The assistant then considered several alternatives: using tensor parallelism across 2 GPUs (which would halve the per-GPU model footprint but reduce the number of available instances), disabling radix cache (which would break speculative decoding), or reducing the Mamba cache allocation. It chose the last option—capping the Mamba cache size and reducing the memory ratio—as the least disruptive change.

The shift to scp was itself a product of frustration with SSH heredocs. After multiple failed attempts (messages [msg 7489] through [msg 7498]), the assistant recognized that the quoting issue was a separate problem from the memory configuration and needed a different solution. The decision to write locally and transfer with scp eliminated the quoting problem entirely.

Assumptions and Potential Mistakes

The assistant made several assumptions in this message:

  1. The local file exists: The command assumes /data/dflash/scripts/launch_mtp.sh exists on the local machine. This file was created in [msg 7498] with a [write] tool call, so this assumption is valid.
  2. SCP will preserve the file correctly: Unlike heredocs, scp transfers binary-identical copies, so there is no risk of quoting or encoding issues. This is a safe assumption.
  3. The memory configuration will work: The assistant assumes that capping the Mamba cache at 24 GB with a 0.4 memory ratio will be sufficient to prevent OOM while still allowing MTP to function. This is an educated guess—the exact memory requirements depend on the number of concurrent requests, the context length distribution, and the speculative decoding acceptance rate, all of which are workload-dependent.
  4. The script's exec usage is correct: The script uses exec to replace the shell process with the Python process, which ensures proper signal handling and process management. This is a best practice for wrapper scripts. A potential mistake is that the --mamba-full-memory-ratio value might be too conservative. If the ratio is too low, the Mamba cache might not have enough pre-allocated slots to handle the speculative decoding workload, causing requests to be queued or rejected. Conversely, if it's too high, the OOM error would reappear. The 0.4 value represents a starting point for iterative tuning.

Input and Output Knowledge

Input knowledge required to understand this message includes:

Conclusion

Message [msg 7499] is a study in pragmatic engineering. It does not solve the fundamental memory constraint problem—that remains to be tested in subsequent messages. But it solves the immediate blocking issue: getting the correct configuration onto the remote server. The shift from SSH heredocs to scp is a small tactical adjustment, but it reflects a larger pattern in complex system deployment: when a tool repeatedly fails, change the tool rather than continuing to fight it.

The message also captures a moment of synthesis. The assistant had spent several messages exploring the configuration space, understanding the constraints, and formulating a strategy. The script that gets transferred is the embodiment of that strategy: keep extra_buffer (required by spec decode), cap its memory consumption (to prevent OOM), and tune the memory ratio (to find the right balance). Whether this configuration actually works will be determined by the next launch attempt—but at least the script is now correctly in place.