The Unprobeable Container: When a Docker Image's Entrypoint Blocks Inspection

Introduction

In the course of deploying a large language model across two NVIDIA DGX Spark systems, an AI assistant encountered a peculiar obstacle: a Docker image so tightly coupled to GPU hardware that even listing its files became impossible. Message [msg 6653] captures a moment of diagnostic frustration — the assistant's third attempt to inspect a purpose-built vLLM image, each time thwarted by the same Python import error. This seemingly minor failure reveals deep truths about container design, hardware dependency chains, and the assumptions we make when probing unfamiliar infrastructure.

The Context: A Multi-Node Deployment Under Siege

The broader session involved deploying Qwen3.5-122B-A10B-FP8, a 119-billion-parameter reasoning model, across two DGX Spark nodes connected via InfiniBand RoCE. Each Spark is an ARM64 system with a single Blackwell GB10 GPU and 120GB of unified memory — powerful but architecturally unusual. The assistant had already attempted SGLang multi-node serving, only to watch NCCL initialization hang indefinitely after channel setup ([msg 6636] through [msg 6647]). The logs showed CustomAllreduce is disabled because this process group spans across nodes followed by dead silence — no weight loading, no progress, just two processes staring at each other across the InfiniBand link.

After killing the stalled SGLang containers, the assistant pivoted. The existing vllm-node Docker image on the Spark (vLLM 0.14.0rc2) was checked and found lacking — it had transformers 5.0.0rc3 but no Qwen3.5 model support in its registry ([msg 6648]). A web search then revealed hellohal2064/vllm-qwen3.5-gb10, a Docker image explicitly built for running Qwen3.5 on DGX Spark GB10 systems ([msg 6649]). The image was pulled successfully ([msg 6650]), and the assistant now needed to verify it actually contained the Qwen3.5 model files before deploying it.

The Probing Attempts: A Cascade of Failures

The first probe ([msg 6651]) ran python3 -c "import vllm; ..." inside the container to check the version and model registry. It failed with a Python traceback — vLLM's engine configuration tried to initialize during import and crashed because no GPU was available in the container (the --rm flag was used without --gpus all).

The assistant correctly diagnosed this: "The import fails without GPU" ([msg 6652]). The logical next step was to sidestep Python entirely and use a pure shell command. Instead of importing vLLM, the assistant would run bash -c "ls /opt/vllm/vllm/model_executor/models/ | grep qwen3" — a filesystem probe that should work without any GPU interaction.

Message [msg 6653]: The Third Strike

Here is the message in full:

[assistant] [bash] ssh aurora@10.1.230.180 'docker run --rm hellohal2064/vllm-qwen3.5-gb10 bash -c "ls /opt/vllm/vllm/model_executor/models/ | grep qwen3"' 2>&1 | tail -10
    return copy.deepcopy(_compute_kwargs(cls))
                         ^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm/vllm/engine/arg_utils.py", line 270, in _compute_kwargs
    default = default.default_factory()  # type: ignore[call-arg]
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm-env/lib/python3.12/site-packages/pydantic/_internal/_dataclasses.py", line 121, in __init__
    s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
  File "/opt/vllm/vllm/config...

The command is straightforward: SSH into the head Spark node, run a Docker container from the hellohal2064/vllm-qwen3.5-gb10 image with --rm (auto-cleanup), execute bash -c "ls ... | grep qwen3" inside it, pipe stderr to stdout, and show only the last 10 lines. The expected output would be something like qwen3_5.py or qwen3_5_moe.py — a simple directory listing confirming the model files exist.

Instead, the output is the same Python traceback from vllm/engine/arg_utils.py. The ls command never ran. The grep never ran. Even bash itself may never have started — at least, not as the primary process.

The Revelation: A Custom ENTRYPOINT

The identical error across three different commands (python3 -c, bash -c "pip show", and bash -c "ls") reveals something fundamental about the Docker image's design. The image's ENTRYPOINT is not a shell — it is a Python script that imports vLLM before doing anything else. When you run docker run hellohal2064/vllm-qwen3.5-gb10 bash -c "ls ...", the bash -c "ls ..." string is passed as arguments to the entrypoint script, not executed directly. The entrypoint script likely does something like:

# Entrypoint script
import vllm  # fails without GPU
# ... then maybe exec the passed command

This is a common pattern in ML container images. The entrypoint sets up the environment, validates hardware, initializes the runtime, and then executes the user's command. It provides a safety net — ensuring the container is never used in a broken state. But it also means the image is unprobeable without GPU access. You cannot inspect its contents, check its version strings, or verify its model support without passing --gpus all to Docker.

Assumptions and Their Consequences

The assistant made several assumptions in this message, each reasonable but ultimately incorrect:

1. That bash -c bypasses Python initialization. This assumes the Docker image uses a standard shell as its ENTRYPOINT, or at least that passing a shell command overrides whatever the image normally does. In reality, the ENTRYPOINT intercepts all commands.
2. That ls is a GPU-independent operation. File listing should not require GPU access. But because the entrypoint runs before any command, even ls triggers the GPU-dependent initialization chain.
3. That the error from [msg 6651] was caused by Python import specifically. The assistant correctly identified that python3 -c "import vllm" failed without GPU, but assumed a shell command would avoid that path. The error source was not the Python command itself but the container's entrypoint wrapping.
4. That tail -10 would capture the relevant output. The traceback is truncated, showing only the tail end of a deep call stack. The actual root cause — the entrypoint script — is hidden above the visible lines.

Input Knowledge Required

To understand this message, the reader needs:

• Docker fundamentals: How ENTRYPOINT vs CMD work, and that docker run image command passes command as arguments to the entrypoint.
• The deployment context: That SGLang multi-node failed with NCCL deadlock, forcing a pivot to vLLM.
• The hardware constraint: DGX Spark has a single GPU per node, and the assistant is running these probes from the Proxmox host (not the Spark itself), so GPU access requires --gpus all which wasn't used.
• The model architecture: Qwen3.5-122B-A10B-FP8 is a Mixture-of-Experts model with FP8 quantization, requiring specific vLLM support.

Output Knowledge Created

Despite the failure, this message produces valuable knowledge:

1. The image's ENTRYPOINT requires GPU. This is an important constraint — the image cannot be used for offline inspection or validation. Any interaction must happen with GPU passthrough.
2. The error signature is consistent. The same traceback across different commands confirms the entrypoint hypothesis, ruling out intermittent failures or command-specific issues.
3. A new approach is needed. The assistant must either run the container with --gpus all (which would give it real GPU access and allow inspection) or simply trust the image's description and deploy it directly. The chunk summary confirms the assistant chose the latter path — deploying the image directly and building a Ray-based multi-node setup.

The Thinking Process

The assistant's reasoning is visible in the progression from [msg 6651] to [msg 6653]. After the first Python import failure, the assistant says "The import fails without GPU. Let me just check the model files" — a clear diagnosis followed by a workaround attempt. The workaround (using bash -c "ls") is logical: if Python import is the problem, avoid Python. But the workaround fails because the entrypoint is the problem, not the command.

The assistant does not explicitly acknowledge the entrypoint hypothesis in the visible reasoning. The message is purely the command and its output — the assistant's next action (not shown in the provided context) would reveal whether the entrypoint issue was recognized. Based on the chunk summary, the assistant moved on to deploying the image directly, which suggests either recognition of the entrypoint constraint or a pragmatic decision to stop probing and start testing.

Broader Implications

This message illustrates a tension in container design for ML workloads. On one hand, a custom entrypoint that validates hardware and initializes the runtime provides safety and consistency — it prevents users from accidentally running in a broken state. On the other hand, it makes the image opaque and hard to debug. A user who wants to verify the image contents, check version compatibility, or inspect model files before deployment is blocked.

The design choice reflects a tradeoff: production robustness vs. explorability. For a well-known deployment target (DGX Spark with known GPU configuration), the entrypoint makes sense. But for troubleshooting and integration work, it creates friction. The assistant's repeated attempts to probe the image — each failing identically — demonstrate this friction in practice.

Conclusion

Message [msg 6653] is a small moment in a complex deployment, but it captures a universal experience in infrastructure engineering: the assumption that a simpler command will bypass a complex failure, only to discover the failure is deeper than expected. The assistant's three failed probes of the hellohal2064/vllm-qwen3.5-gb10 image reveal a container designed for production use, not exploration — an ENTRYPOINT that guards the image contents behind GPU access. The lesson is both specific (this image needs --gpus all for any interaction) and general (container entrypoints can transform even trivial commands into complex initialization sequences). The assistant would go on to deploy the image successfully, but not before learning that some containers refuse to be inspected — they demand to be trusted.