The Art of the First Step: Decommissioning Before Deployment in Multi-Node AI Inference

Introduction

In the world of large-scale AI inference deployment, the most critical decisions are often not about which model to run or which framework to choose—they are about what to stop before you can start. Message 6562 in this opencode session captures a pivotal moment in a complex multi-node deployment: the transition from an existing GLM-4.7-Flash inference service to a new Qwen3.5-122B-A10B-FP8 deployment across two NVIDIA DGX Spark systems. This message, seemingly a simple reconnaissance step, reveals the deep interplay between infrastructure awareness, resource management, and the careful choreography required when working with cutting-edge hardware and models.

The Message in Full

The assistant begins with a structured reasoning block, then issues a bash command to survey the running services on the head Spark node:

## Agent Reasoning The user wants me to: 1. Stop the GLM model on the head spark (but keep other services that don't conflict) 2. Keep the embeddings vLLM running on the secondary spark 3. Continue with the Qwen3.5 deployment

>

Let me first stop the GLM vLLM service on the head, and also note that the current vLLM in Docker (0.14) does NOT have Qwen3.5 support - it only has Qwen3 models, not Qwen3_5MoeForConditionalGeneration. So I'll need a newer vLLM or SGLang.

>

I'll stop the GLM service now and check what's currently using GPU memory on the head node.

>

[bash] ssh aurora@10.1.230.180 'echo "=== Services ===" && systemctl --user list-units --type=service --state=running 2>/dev/null | grep -iE "vllm|sglang|llm" ; sudo systemctl list-units --type=service --state=running 2>/dev/null | grep -iE "vllm|sglang|llm"' 2>&1 === Services === vllm-cluster.service loaded active running vLLM Cluster (GLM-4.7-Flash on GPU) vllm-embeddings.service loaded active running vLLM Embedding Server (gte-Qwen2 on GPU) vllm-proxy.service loaded active running Unified LLM Proxy (port 8000) with Load Balancing vllm-reranker.service loaded active running ZeRank-2 Reranker (standalone container)

This is the complete content of the message. On its surface, it is a straightforward acknowledgment of the user's instruction followed by a service inventory command. But beneath this simplicity lies a wealth of strategic thinking and technical depth.

WHY This Message Was Written: The Reasoning and Motivation

The immediate trigger for this message is the user's instruction in message 6561: "note head is running some glm model now, stop that; on secondary keep embeddings vllm running." This is a classic resource management directive. The DGX Spark systems have 120GB of unified memory each—memory that is shared between the CPU and GPU rather than being dedicated VRAM. The Qwen3.5-122B-A10B-FP8 model, with its 125 billion total parameters in FP8 quantization, requires approximately 125GB of memory to load. Even with tensor parallelism across two nodes (roughly 62.5GB per node), the head Spark's available memory must be freed from any competing workloads.

However, the assistant's motivation goes deeper than simply following an instruction. The reasoning block reveals a sophisticated understanding of the deployment landscape. The assistant recognizes that stopping the GLM service is only the first of several prerequisites. It notes that the current vLLM version (0.14.0rc2) lacks support for Qwen3_5MoeForConditionalGeneration—the model architecture class needed for Qwen3.5. This observation, carried forward from earlier exploration in messages 6559 and 6560, means that even after freeing GPU memory, the assistant will need to either upgrade vLLM or pivot to SGLang. The message thus serves as a bridge between the user's immediate request and the assistant's broader deployment strategy.

The motivation is also defensive: before stopping any services, the assistant first inventories what is running. This is a classic systems administration best practice—never kill a process without first understanding what it is and what depends on it. The assistant explicitly notes that other services (embeddings, proxy, reranker) should be preserved, showing an awareness that the Spark node runs a multi-service inference stack, not just a single model.

HOW Decisions Were Made

The decision-making in this message operates on several levels. The most visible decision is the choice of command: rather than immediately running systemctl stop vllm-cluster.service, the assistant first runs a discovery command. This reflects a deliberate "measure before act" philosophy. The command itself is carefully constructed: it checks both systemctl --user (user-level systemd services, common in containerized deployments) and sudo systemctl (system-level services), ensuring no running service is missed. The grep pattern -iE "vllm|sglang|llm" is broad enough to catch any inference-related service while excluding unrelated system services.

A subtler decision is the assistant's choice to surface the vLLM compatibility issue in this message, even though the user's instruction was only about stopping the GLM model. The reasoning block mentions that "the current vLLM in Docker (0.14) does NOT have Qwen3.5 support." This is a forward-looking decision: by raising this issue now, the assistant prepares the user for the next challenge. It also implicitly sets expectations—stopping the GLM service is necessary but not sufficient; a framework upgrade will also be required.

The decision to keep the embeddings vLLM running on the secondary Spark is also noteworthy. The user explicitly instructed this, but the assistant's reasoning shows it has internalized this constraint. The embeddings service (gte-Qwen2) serves a different purpose (text embedding generation) and does not conflict with the generative inference workload. By preserving it, the assistant maintains the secondary node's utility for other tasks while dedicating its GPU memory to the Qwen3.5 deployment.

Assumptions Made by the User and Agent

Several assumptions underpin this message, some explicit and some implicit.

The user assumes that stopping the GLM model on the head Spark is a straightforward operation—a simple systemctl stop command. This assumption is largely correct, but it glosses over the complexity of what "stopping" means in a multi-node, multi-service environment. The GLM service (vllm-cluster.service) is described as "vLLM Cluster (GLM-4.7-Flash on GPU)," suggesting it may be part of a Ray cluster spanning both nodes. Stopping it might require careful cluster teardown to avoid orphaned processes on the secondary node.

The user assumes that the embeddings service on the secondary Spark can coexist with the Qwen3.5 deployment. This is a reasonable assumption given the memory budget: the embeddings model (gte-Qwen2) is relatively small, and with 120GB of unified memory, there should be room for both—provided the Qwen3.5 model's memory footprint is accurately estimated.

The assistant assumes that the current vLLM version definitively lacks Qwen3.5 support. This assumption is based on a direct import test in message 6560: from vllm.model_executor.models.qwen3_5_moe import * raised ModuleNotFoundError. However, this is a narrow test—it only checks for a specific module name. It's possible that Qwen3.5 support is integrated into an existing model class (e.g., Qwen3MoeForCausalLM might handle the new architecture with updated configuration). The assistant does not test whether the existing vLLM can load the Qwen3.5 model by attempting an actual model load, which would be the definitive test. This is a minor but notable gap in the assumption.

The assistant assumes that the service names discovered by the grep command accurately reflect the running services. The systemctl list-units output shows human-readable descriptions (e.g., "vLLM Cluster (GLM-4.7-Flash on GPU)"), but these descriptions are set by the service file author and could be inaccurate or outdated. The assistant does not verify, for example, that the GLM container is actually using GPU memory by running nvidia-smi or docker stats.

Input Knowledge Required

To fully understand this message, the reader needs a substantial body of context from the preceding conversation:

  1. Hardware topology: Two NVIDIA DGX Spark systems (GB10), each with 120GB unified memory, ARM Cortex-X925 CPU, SM121 Blackwell GPU, connected via InfiniBand RoCE at 192.168.200.x addresses. The head Spark (spark-1a32) is at 10.1.230.180, the secondary (spark-5be3) at 192.168.200.13, reachable via SSH jump.
  2. Existing software stack: The head Spark runs Docker with a vllm-node image containing vLLM 0.14.0rc2, PyTorch 2.9.1+cu130, CUDA 13.0, and transformers 5.0.0rc3. The secondary Spark has the same Docker image but only runs an embeddings service.
  3. Model characteristics: Qwen3.5-122B-A10B is a Mixture-of-Experts architecture with 125B total parameters but only 10B active per token. The FP8 quantized version is approximately 125GB on disk. It requires a recent inference framework (vLLM main branch or SGLang nightly) that supports the Qwen3_5MoeForConditionalGeneration architecture.
  4. Previous exploration: Messages 6554-6560 established that the existing vLLM Docker image supports Qwen3 models but not Qwen3.5 specifically. The assistant had already searched for deployment guides, checked the HuggingFace model page, and verified InfiniBand connectivity between the two Sparks.
  5. User's deployment goal: Deploy the Qwen3.5-122B-A10B-FP8 model across both DGX Sparks with tensor parallelism (TP=2) over InfiniBand, achieving high-throughput inference for this cutting-edge reasoning model.

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. Service inventory: The head Spark runs four inference-related services: vllm-cluster.service (GLM-4.7-Flash), vllm-embeddings.service (gte-Qwen2), vllm-proxy.service (unified proxy on port 8000), and vllm-reranker.service (ZeRank-2). This inventory is critical for understanding which services to stop, which to preserve, and how they interrelate.
  2. Service naming convention: The services follow a consistent naming pattern (vllm-{role}.service) and include human-readable descriptions. This suggests a well-organized deployment with systemd service files, likely managed by the spark-vllm-docker scripts discovered earlier.
  3. Proxy architecture: The presence of vllm-proxy.service with "Load Balancing" indicates that the Spark deployment uses a proxy to route requests between the GLM model, embeddings, and reranker. Stopping the GLM service may require updating or disabling the proxy to avoid routing errors.
  4. GPU memory pressure: The vllm-cluster.service description says "GLM-4.7-Flash on GPU," confirming that GPU memory is actively consumed. The earlier nvidia-smi output in message 6555 showed 104GB used out of 119GB, with the GLM model likely being the primary consumer. Freeing this memory is essential for loading the 125GB Qwen3.5 model.
  5. Compatibility gap confirmed: The reasoning block reiterates that vLLM 0.14 lacks Qwen3.5 support, setting up the next phase of the deployment where the assistant will need to find or build a compatible framework.

The Thinking Process Visible in Reasoning

The reasoning block in this message is a window into the assistant's cognitive process. It reveals a structured, methodical approach to problem-solving:

Step 1: Parse the user's instruction. The assistant breaks the user's command into three actionable items: stop GLM on head, keep embeddings on secondary, continue deployment. This decomposition turns a vague directive into a concrete checklist.

Step 2: Identify prerequisites. The assistant recognizes that stopping the GLM service is not sufficient—the vLLM framework itself is incompatible with the target model. This shows systems-level thinking: the assistant is not just executing commands but evaluating the entire deployment pipeline.

Step 3: Plan the immediate action. The assistant decides to "first stop the GLM vLLM service on the head" but precedes this with a discovery command. This is the hallmark of careful operations: verify before modify.

Step 4: Execute with appropriate tooling. The bash command uses systemctl --user and sudo systemctl to cover both user-level and system-level services. The grep pattern is intentionally broad to catch any inference-related service. The command is designed to be informative without being destructive.

The reasoning also reveals what the assistant is not doing. It does not immediately stop the GLM service in this message—it only checks what's running. This restraint is deliberate: by first confirming the service names, the assistant can construct a precise stop command in the next message, targeting only the GLM service while leaving embeddings, proxy, and reranker untouched.

Broader Significance

Message 6562, while brief, exemplifies the kind of careful orchestration required when deploying large language models on specialized hardware. The DGX Spark, with its unified memory architecture and ARM CPU, is not a standard server—it demands a deployment approach that accounts for memory sharing between CPU and GPU, ARM-specific Docker images, and InfiniBand-based multi-node communication. The assistant's methodical approach—inventory before action, forward planning of compatibility issues, and preservation of non-conflicting services—is a template for how to manage complex AI infrastructure.

The message also highlights a recurring theme in this session: the tension between the cutting edge and the stable. The Qwen3.5 model is bleeding-edge, requiring framework support that may not exist in released versions. The DGX Spark hardware is also cutting-edge (SM121 Blackwell, ARM architecture). When the frontier of model development meets the frontier of hardware, the deployment engineer must be prepared to build, patch, and adapt—and to start by clearing the ground of what came before.