Crossing the Bridge: Deploying a Custom INT4 Inference Engine to Production Blackwell Hardware

Introduction

In the lifecycle of any serious machine learning infrastructure project, there is a moment when carefully crafted code must leave the safety of the development environment and face the unforgiving reality of production hardware. Message [msg 12080] captures precisely such a moment. After building a complete native C/C++/CUDA speculative decoding engine from scratch—including custom CUDA kernels for tree construction, MLA-absorb attention, and greedy acceptance, plus a full DeepSeekV3/Kimi-style transformer with INT4 W4A16 group-quantized MoE support—the assistant takes the decisive step of deploying this work to the CT200 box, an 8× RTX PRO 6000 Blackwell GPU machine that represents the actual target hardware for the inference stack.

This message is deceptively brief. On its surface, it contains only a reasoning paragraph and two shell commands followed by their output. But within that brevity lies a dense web of engineering decisions, implicit assumptions, and a carefully calibrated deployment strategy that reveals much about how the assistant approaches the transition from development to validation on production-like infrastructure.

The Context: What Was Just Completed

To understand why this message matters, one must appreciate what preceded it. The assistant had just committed Phase 3 of the kdtree-engine project—a milestone captured in commit 6a046bb with the subject line "Phase 3: native INT4 W4A16 group-quant MoE path in the engine (validated)" ([msg 12079]). This commit represented the culmination of an intense development effort spanning multiple chunks within segment 65.

The Phase 3 work included: a custom w4a16_gemm CUDA kernel that performs matrix multiplication between FP32 activations and INT4 packed weights with per-group float32 scales (8 nibbles per int32 along the input dimension, symmetric with value = nibble - 8); Python reference implementations for quantization (quantize_w4a16 and quantize_weight_dict in model_ref.py); a model bundle generator (gen_model_int4.py) that produces KDTR-format bundles with packed .qw (int32) and .sc (float32) weight files alongside INT4-dequantized golden tokens for validation; and modifications to model.cu/model.h to load these quantized weights, dispatch through a linear() function that selects between the W4A16 kernel and FP32 cuBLAS depending on whether quantized weights exist for a given layer.

Crucially, the MoE experts, shared expert, and dense MLP layers all run through the INT4 path, while the MLA attention and lm_head remain in FP32. The engine transparently handles both formats, with the same code path serving FP32 and INT4 model bundles. All four model bundles (two FP32, two INT4, across two model configurations) passed validation locally, with the critical invariant confirmed: the DDTree greedy output matches the autoregressive greedy output token-for-token, and the INT4 golden output differs from the FP32 golden output (confirming real quantization is applied, not a no-op).

But this validation happened on the assistant's local development machine—a 5070 Ti GPU (sm_89 architecture). The target hardware is the CT200 box with 8× RTX PRO 6000 Blackwell GPUs (sm_120 architecture). The INT4 kernel and the engine's CUDA code were compiled for sm_120, but they had never actually run on that architecture. This is the gap that message [msg 12080] aims to close.

The Deployment Strategy: Controlled and Conservative

The assistant's reasoning reveals a clear two-phase plan: "validate the INT4 path on actual hardware, then benchmark INT4 against FP32 MoE performance." The deployment itself is executed through two carefully constructed shell commands, each reflecting deliberate engineering choices.

The first command is an rsync invocation:

rsync -az --delete --exclude build/ --exclude .git/ --exclude 'tests/refs/model_*.kdtr' -e "ssh -o StrictHostKeyChecking=no" kdtree-engine/ root@10.1.230.171:/root/kdtree-engine/

The flags tell a story. --delete ensures that any stale artifacts on the target are removed, preventing confusion from old builds. The exclusions are equally deliberate: build/ is excluded because the build must happen on the target hardware with its specific CUDA installation; .git/ is excluded to avoid transferring repository metadata that isn't needed for execution; and tests/refs/model_*.kdtr is excluded because these large model bundles (the INT4 bundles are ~5 MB each, the FP32 bundles ~10-15 MB) are regenerable on the target via the Python generation scripts. This last exclusion is particularly telling—it reflects the assistant's earlier decision (in [msg 12079]) to gitignore the model bundles and treat them as build artifacts rather than tracked assets.

The second command chains a build and test execution under a 180-second timeout:

timeout 180 ssh -o StrictHostKeyChecking=no root@10.1.230.171 'cd /root/kdtree-engine && CUDA_HOME=/usr/local/cuda-13.0 bash scripts/build_nvcc.sh >/dev/null 2>&1 && PYTHON=/root/venv_sglang211/bin/python bash scripts/run_tests_nvcc.sh 2>&1 | grep -E "model_int4|ALL TESTS|FAIL"'

The timeout 180 wrapper is a defensive measure—if the build hangs or takes abnormally long, the command will be killed rather than leaving the assistant waiting indefinitely. The >/dev/null 2>&1 on the build command suppresses all build output, since the assistant only cares about the test results. The grep filter on the test output is designed to show only the INT4 model generation lines and the final pass/fail status, keeping the output concise.

The Results: Validation Across the Architecture Gap

The output confirms success on both fronts. The INT4 model bundles were regenerated on the target machine:

wrote /root/kdtree-engine/python/../tests/refs/model_int4_tiny.kdtr  quantized 84 weights (G=32)  golden[:8]=[499, 128, 58, 76, 462, 243, 474, 353]
wrote /root/kdtree-engine/python/../tests/refs/model_int4_tiny2.kdtr  quantized 90 weights (G=32)  golden[:8]=[187, 328, 252, 328, 121, 158, 129, 137]

The golden token prefixes match exactly what was produced on the local development machine (visible in [msg 12077]), confirming deterministic regeneration. And the final line—"ALL TESTS PASSED"—confirms that the INT4 path, the FP32 path, and all kernel tests work correctly on the Blackwell PRO 6000 hardware.

This is a non-trivial result. CUDA kernels compiled for sm_120 may behave differently on actual hardware than on an emulator or a different architecture. Memory alignment requirements, warp scheduling behavior, and numerical precision can all vary. The fact that all tests pass on the target hardware validates not just the INT4 quantization path but the entire engine's correctness on the production architecture.

Assumptions and Their Implications

The message rests on several implicit assumptions that are worth examining. First, the assistant assumes that the CT200 box is accessible and in a consistent state—that the SSH connection will succeed, that CUDA 13.0 is installed at /usr/local/cuda-13.0, that the Python venv at /root/venv_sglang211/bin/python exists with the correct dependencies. These are reasonable assumptions given that the assistant has been working with this machine throughout the session, but they are assumptions nonetheless.

Second, the assistant assumes that the build scripts (scripts/build_nvcc.sh) will work on the target without modification. This script was designed to work with nvcc directly (as opposed to CMake, which is not available on CT200), and it has been tested on the local machine, but the target has a different CUDA installation path and potentially different environment variables.

Third, the assistant assumes that the INT4 golden tokens generated on the target will match those generated locally. The output confirms this, but the assumption was made before the result was known—it reflects confidence in the deterministic nature of the quantization and generation scripts.

Fourth, and perhaps most significantly, the assistant assumes that passing the model tests (which use tiny synthetic model configurations with 6 layers, 192 hidden dimensions, 400 vocabulary tokens) is sufficient validation before proceeding to benchmark the INT4 path at realistic K2.6 scale. This is a standard engineering practice—validate correctness on small, fast-to-run tests before scaling up—but it's worth noting that the tiny models may not exercise all the edge cases that a full-scale model would encounter.

The Thinking Process: What the Reasoning Reveals

The assistant's reasoning paragraph is notably candid about uncertainty: "though I'll need to check what metrics it actually outputs since it may not directly show inference timing." This admission is revealing. The assistant is planning to benchmark INT4 vs FP32 MoE performance using the in-engine demo, but it isn't certain that the demo tool provides the timing information needed for a meaningful comparison. This uncertainty drives the decision to first validate correctness (via the test suite) and then figure out the benchmarking approach.

The reasoning also reveals the assistant's mental model of the deployment pipeline as a linear sequence: rsync → build → test → benchmark. Each step depends on the previous one succeeding. The timeout 180 wrapper and the && chaining enforce this dependency—if the build fails, the tests don't run; if the tests fail, the assistant will see the failure in the grep output and can decide how to proceed.

The Broader Significance

Message [msg 12080] sits at a critical juncture in the kdtree-engine project. The engine has been built from scratch—custom CUDA kernels, a full transformer implementation, INT4 quantization, and the DDTree speculative decoding loop all wired together. But until this message, it had only been proven on a development GPU. The deployment to CT200 is the first step toward the engine's ultimate purpose: running on the 8× PRO 6000 Blackwell machine to accelerate Kimi K2.6 inference.

The "ALL TESTS PASSED" result is therefore more than just a green checkmark. It is evidence that the custom CUDA kernels, written for sm_120 and compiled with CUDA 13.0, actually work correctly on Blackwell hardware. It is evidence that the INT4 quantization path, with its custom packing format and dequantization logic, produces bit-exact results on the target architecture. It is evidence that the deployment pipeline—rsync with careful exclusions, remote build, remote test—is reliable and reproducible.

In the broader narrative of the session, this message represents the transition from development to validation. The subsequent chunks in segment 65 will deal with diagnosing throughput regressions, extending context length, and ultimately benchmarking the engine against production requirements. But none of that work would be possible without the foundational validation that message [msg 12080] accomplishes: proving that the custom engine runs correctly on the hardware it was built for.