Two Bottlenecks, One Pipeline: Diagnosing and Addressing the DFlash Training Slowdown

Introduction

In the high-stakes world of multi-GPU training for speculative decoding, performance degradation is rarely caused by a single factor. More often, it is a cascade of independent failures—each one compounding the others—that transforms a smoothly running pipeline into a grinding, underutilized mess. This is the story of one such cascade, captured across a concentrated debugging session within a sprawling opencode conversation about training a DFlash block-diffusion speculative decoding drafter.

The training pipeline had achieved a peak throughput of 21.5K tokens per second, with flat GPU memory utilization and stable operation. Then, in a series of well-intentioned but ultimately destructive actions—installing SGLang, reverting torch versions, deleting compile caches—the assistant broke everything. Throughput collapsed to 4.3K tok/s, GPU memory became volatile, and a multi-threaded torch.compile race condition, previously invisible because the compile cache was warm, blocked every training attempt.

What followed was a multi-front debugging campaign that would uncover not one but two independent root causes for the slowdown, each requiring a fundamentally different remediation strategy. The first was a missing-dependency problem so mundane that it had been overlooked for days: 48 out of 64 layers in the target model were running a slow PyTorch fallback because two CUDA extension packages were not installed. The second was a deep, architectural thread-safety issue in PyTorch's torch.compile infrastructure that would resist multiple attempted fixes and ultimately force a complete redesign of the training pipeline's input shape strategy.

This article traces the arc of that debugging effort, examining the reasoning, assumptions, decisions, and knowledge created across dozens of messages. It is a case study in systematic diagnosis under pressure, and a cautionary tale about the fragility of modern ML training infrastructure.

The First Bottleneck: Missing CUDA Extensions in the Target Model

The Discovery

The first breakthrough came not from a complex profiling tool, but from a simple model inspection script. In [msg 9996], the assistant ran a diagnostic probe that loaded the target model's configuration and checked its attention implementation. The output contained a single warning line that would reshape the entire investigation:

[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation.

The target model—Qwen3.6-27B, a 27-billion-parameter hybrid architecture—uses a mix of standard attention layers and GatedDeltaNet linear attention layers. Of its 64 layers, 48 are linear_attention layers that require two external packages for their fast CUDA kernel paths: flash-linear-attention (fla) and causal-conv1d. Without these packages, every one of the 48 layers falls back to a pure-PyTorch implementation that is dramatically slower.

This discovery was the smoking gun for the target model's poor performance. The assistant had been wrestling with complex multi-threaded torch.compile race conditions, CUDA graph capture failures, and intricate attention mask implementations. Yet the biggest performance win came from a missing pip install—a dependency so basic that it had been overlooked in the rush to build and deploy the training pipeline.

The Installation Saga

Installing these packages proved far from trivial. The training environment ran inside a Proxmox container that lacked the CUDA compiler (nvcc), which is required to compile causal-conv1d from source. The assistant had to:

Install the NVIDIA CUDA keyring package to access the apt repository
Install cuda-nvcc-12-8 and cuda-cudart-dev-12-8 (matching the container's CUDA 12.8 runtime)
Install flash-linear-attention via uv pip install flash-linear-attention
Compile causal-conv1d from source—a process that took over four minutes The compilation step was particularly fraught. The system uses NVIDIA RTX PRO 6000 Blackwell GPUs with SM 12.0 compute capability, which means prebuilt wheels for causal-conv1d were unlikely to exist. The assistant had to ensure that the CUDA toolkit version matched PyTorch's cu128 suffix, that the CUDA_HOME environment variable was set correctly, and that the build process could find the necessary headers and libraries. After the installation, the assistant verified that all four fast-path functions (causal_conv1d_fn, causal_conv1d_update, chunk_gated_delta_rule, fused_recurrent_gated_delta_rule) were available and non-None. A subsequent benchmark confirmed the fix: the target model's forward pass achieved approximately 6,000 tokens per second per GPU, yielding a theoretical throughput of ~30K tok/s across five GPUs ([msg 10036]). The target model bottleneck was resolved.

What This Reveals About ML Infrastructure

The GatedDeltaNet slow-path episode is a textbook example of a silent performance killer. The model loaded, ran, and produced correct outputs—just 10x slower. No error was raised, no crash occurred. The warning from HuggingFace Transformers was easy to miss, especially when it appeared alongside dozens of other log lines during model loading.

This pattern is distressingly common in modern ML engineering. Model architectures that mix standard and custom layer types create hidden dependency chains. The model appears to work, but silently degrades to a fraction of its potential performance when optional CUDA extensions are absent. The lesson is clear: when diagnosing performance issues in complex pipelines, always verify that the expected CUDA kernels are actually being used, not just that the model loads without errors.

The Second Bottleneck: The FX Tracing Race Condition

The Race Condition Explained

While the target model bottleneck was being resolved, a second, more stubborn problem plagued the drafter model. The DFlash drafter uses a custom attention pattern implemented via PyTorch's flex_attention function, wrapped with torch.compile(mode="reduce-overhead") for performance. In the multi-threaded training pipeline—where three drafter threads each call the compiled function simultaneously on their first forward pass—the threads collided on a module-level global flag (_is_fx_tracing_flag in torch.fx._symbolic_trace).

This flag is not thread-local. When one thread begins FX tracing to compile the function, it sets the flag to True. Another thread's compile_wrapper check sees the flag and raises:

RuntimeError: Detected that you are using FX to symbolically trace a dynamo-optimized function

Two of the three drafter threads crashed on this error, leaving only one drafter consuming hidden states from the shared queue. The pipeline entered a death spiral of queue backpressure: with only one drafter consuming, the target threads blocked on queue.put() when the queue filled up, causing GPU utilization to plummet across all eight GPUs.

The assistant's root cause analysis, documented in [msg 9962], traced the entire chain of causality:

The old working run (step 690, 21.5K tok/s) had a warm compile cache at /tmp/torchinductor_root/
During first compilation of torch.compile(flex_attention), FX tracing sets the module-level global flag
This flag is not thread-local—when multiple drafter threads trigger first compilation simultaneously, the race condition is exposed
The old run never hit this because the compile cache was warm—torch.compile loaded the cached graph and never entered the FX tracing path
The assistant deleted the cache while troubleshooting, and the race condition was exposed

Attempt 1: Replacing flex_attention with Per-Block Batched SDPA

The assistant's first major attempt to fix the race condition was to remove torch.compile and flex_attention entirely, replacing them with a hand-rolled per-block batched SDPA (Scaled Dot-Product Attention) implementation. The plan, articulated in [msg 9978], was elegant:

For the four sliding-window attention (SWA) layers, each anchor block of 32 query tokens would attend to its prefix (up to 2048 tokens) plus the block itself, totaling 2080 KV tokens. All 1024 blocks could be batched together into a single SDPA call using flash attention. For the final full-attention layer, the same approach would be used but chunked by sorted anchor position to keep memory bounded.

The assistant implemented this across four sequential edits ([msg 9979] through [msg 9982]), rewriting the attention mask infrastructure, the compiled function wrapper, the DFlashAttention class, and the DFlashDecoderLayer wiring. A fifth edit ([msg 9983]) updated DFlashDrafter.forward to use the new index builder instead of BlockMask.

However, this approach was ultimately reverted. The per-block batched SDPA introduced variable memory allocation and GQA (Grouped Query Attention) expansion overhead that proved problematic. The assistant returned to the flex_attention approach.

Attempt 2: Per-Thread Execution Lock

The assistant's second attempt was to add a per-thread execution lock (_exec_lock) to serialize the first call to torch.compile(flex_attention) across drafter threads, and to switch gradient checkpointing from use_reentrant=True to use_reentrant=False to avoid secondary FX tracing conflicts.

While this allowed one drafter thread to compile and run successfully, the other threads still hit the race condition. The lock was insufficient because the FX tracing state is global and not fully isolated by Python-level locks. The race condition is inherent to per-device compilation in a multi-threaded environment, and a simple mutex cannot prevent the internal dynamo state from being corrupted.

The Deeper Architectural Problem

The FX tracing race condition exposed a fundamental tension in the training pipeline's architecture. The single-process, multi-threaded design—where multiple drafter threads share a Python process and call torch.compile-decorated functions—is inherently incompatible with PyTorch's current compilation infrastructure. The _is_fx_tracing_flag global is a design flaw in PyTorch's dynamo system, but it is not the only issue. Even if the flag were made thread-local, other aspects of the compilation state (the trace cache, the guards system, the kernel cache) are not designed for concurrent access.

The assistant's eventual pivot—to a fixed-shape pipeline with padded batches, persistent GPU buffers, and CUDA graph capture—represented an acknowledgment that the multi-threaded torch.compile approach was fundamentally broken for this use case. By making all tensor shapes static, the pipeline could use torch.compile(mode="reduce-overhead") without triggering retracing, and could capture CUDA graphs for maximum performance. However, this approach introduced its own thread-safety issues: CUDA graphs captured in the main thread cannot be safely replayed in drafter worker threads, triggering a CUDAGraph Trees thread-local assertion.

The Interplay Between the Two Bottlenecks

One of the most challenging aspects of this debugging session was the interaction between the two bottlenecks. The target model's slow forward pass (due to missing CUDA extensions) and the drafter's thread crashes (due to the FX tracing race) created a cascading failure pattern that was difficult to diagnose:

The target model was slow, but not catastrophically so—it could still produce hidden states
The drafter threads crashed, reducing consumption to 1/3 of capacity
The queue filled up, causing target threads to block on put()
Blocked target threads showed low GPU utilization, making it look like the target model was the bottleneck
The one surviving drafter showed moderate utilization, struggling to keep up This cascade made the symptoms appear far from their causes. The uneven GPU utilization (8-86% across GPUs) looked like a load balancing problem, but the root cause was a thread crash in a different part of the pipeline. The assistant's ability to trace through this causal chain—from dead drafters to queue backpressure to target starvation—was essential to correct diagnosis.

The Knowledge Created

This debugging session produced several forms of knowledge that persist beyond the immediate conversation:

A causal model of the FX tracing race condition: The assistant constructed a complete causal chain from "deleted compile cache" to "training crashes with FX tracing error." This model includes the role of the module-level global flag, the per-process nature of dynamo wrapping, and the insufficiency of warmup scripts that run in separate processes.

A taxonomy of attention implementation strategies: The assistant evaluated dense SDPA with materialized masks, per-block batched SDPA, chunked full attention, and flex_attention with torch.compile, calculating FLOP budgets, memory requirements, and dispatch paths for each. This taxonomy is a reusable resource for any engineer designing sparse attention mechanisms.

A documented infrastructure dependency chain: The installation of flash-linear-attention and causal-conv1d revealed the full dependency chain for GatedDeltaNet layers, including the CUDA toolkit version requirements, the compilation process for Blackwell GPUs, and the verification steps needed to confirm the fast path is active.

A refined understanding of thread safety in PyTorch compilation: The session demonstrated that torch.compile is not thread-safe for first-time compilation, that per-thread locks are insufficient to isolate FX tracing state, and that CUDA graphs captured in one thread cannot be safely replayed in another. These are hard-won insights that will inform future architectural decisions.

The Broader Lessons

Lesson 1: The Compile Cache Is Fragile

The entire chain of failures in this session traces back to deleting the compile cache (rm -rf /tmp/torchinductor_root/). The cache is not just a performance optimization—it is a critical piece of infrastructure for multi-threaded torch.compile workflows. Without it, every process must recompile from scratch, and the race condition is exposed.

Best practice: Never delete the compile cache unless you are certain you can regenerate it safely. If you must clear it, do so in a single-threaded warmup run before launching the multi-threaded training pipeline.

Lesson 2: Profile Before Optimizing

The assistant spent significant effort optimizing the drafter's attention mechanism, but the user's complaint was about hidden state extraction being 10x slow. The attention optimization might have improved drafter throughput by 17%, but the target model bottleneck was a 10x issue.

Best practice: Always profile to identify the actual bottleneck before optimizing. A quick inspection of the target model's attention implementation would have revealed the missing CUDA extensions much earlier.

Lesson 3: Thread Safety of torch.compile

torch.compile is not thread-safe for first-time compilation. The FX tracing flag is a global variable, not thread-local. This is a known limitation that the PyTorch team is working on, but as of PyTorch 2.11, it is still present.

Best practice: If you are using torch.compile in a multi-threaded pipeline, pre-compile all functions in the main thread before spawning worker threads. Use a per-device lock to serialize the first compilation call if pre-compilation is not possible. Even then, be aware that dynamic shapes can trigger retracing during training, potentially re-exposing the race condition.

Lesson 4: Silent Performance Killers Are the Most Dangerous

The missing flash-linear-attention and causal-conv1d packages caused a 10x slowdown without raising any errors. The model loaded, ran, and produced correct outputs—just at a fraction of its potential speed. This silent degradation is far more dangerous than a crash because it can persist indefinitely, masquerading as a fundamental algorithmic limitation rather than a simple installation oversight.

Best practice: When deploying a model with custom layer types (GatedDeltaNet, Mamba, etc.), verify that the expected CUDA kernels are actually being used. Check for warning messages from the transformers library, inspect the attention implementation flag, and benchmark the forward pass to confirm expected throughput.

Lesson 5: Queue Dynamics Can Mask Root Causes

The uneven GPU utilization (8-86%) looked like a load balancing problem, but the root cause was a thread crash in a different part of the pipeline. Queue-based backpressure creates cascading effects that can make symptoms appear far from their causes.

Best practice: When diagnosing uneven GPU utilization in a pipeline, check whether all stages are running at full capacity. A bottleneck in one stage can cause underutilization in upstream stages due to backpressure. Do not assume the problem is in the underutilized stage.

Conclusion

The DFlash training debugging session is a microcosm of modern ML engineering. It demonstrates that performance debugging requires understanding the full stack—from Python-level threading to GPU kernel execution, from package dependencies to compiler internals. The two bottlenecks uncovered in this session—missing CUDA extensions and a multi-threaded compilation race condition—are archetypal problems that will recur across many training pipelines.

The assistant's methodical approach—diagnose, plan, implement, verify, pivot when necessary—reflects a disciplined engineering process. The willingness to abandon promising approaches (the SDPA replacement) when they hit hard constraints, and to trace causal chains across multiple system layers, are the hallmarks of effective debugging.

In the end, the target model bottleneck was resolved through straightforward package installation. The drafter bottleneck proved more stubborn, requiring a fundamental redesign of the pipeline's input shape strategy. But the knowledge created in the process—about the fragility of torch.compile in multi-threaded environments, about the cascading effects of thread failures in queue-based pipelines, about the necessity of matching attention implementations to hardware capabilities—will outlast any single fix. That is the true value of deep diagnostic work.## References

[1] "The Moment of Reckoning: A Status Message That Saved a Machine Learning Training Pipeline" — Analysis of message 9962, the comprehensive status document that organized the debugging effort.

[2] "The Breaking Point: A User's Directive to Abandon FX Tracing Workarounds and Confront the Real Bottleneck" — Message 9963, the user's directive to remove FX tracing.

[3] "The Investigation That Almost Wasn't: Diagnosing a 10× Slowdown in Multi-GPU DFlash Training" — Message 9964, the opening diagnostic move.

[4] "The Todo List That Saved a Training Run: Strategic Debugging Under Pressure" — Message 9965, structured todo management.

[5] "Reading the Source: A Diagnostic Pivot in Multi-GPU DFlash Training" — Message 9966, code reading for diagnostic purposes.

[6] "Reading the Code: A Diagnostic Deep-Dive into DFlash Training Bottlenecks" — Message 9967, deeper code analysis.

[7] "The Anatomy of a Diagnostic Read: Tracing a 10x Slowdown in Multi-GPU DFlash Training" — Message 9968, diagnostic read of the pipeline.

[8] "The Quiet Read: How a Single File Inspection Revealed the Shape of a Debugging Crisis" — Message 9969, file inspection.

[9] "Reading the Bottleneck: How One read Command Pivoted the Diagnosis of a Multi-GPU Training Slowdown" — Message 9970, diagnostic pivot via reading.

[10] "Reading the Code: A Diagnostic Pivot in DFlash Training Debugging" — Message 9971, code reading for bottleneck analysis.

[11] "Reading the Loss Function: A Diagnostic Deep-Dive into DFlash Training Performance" — Message 9972, loss function inspection.

[12] "The Diagnostic Snapshot: Reading GPU Telemetry in the Heat of a Training Debug" — Message 9973, GPU telemetry analysis.

[13] "The FX Tracing Trap: Diagnosing Multi-Threaded torch.compile Failures in Custom Training Pipelines" — Message 9974, the deep diagnostic reasoning about the FX tracing race condition.

[14] "The Power of 'Continue': A Single Word That Unlocks Hours of Reasoning" — Message 9975, continuation of reasoning.

[15] "The Moment of Diagnosis: When a Training Pipeline's Bottleneck is Revealed by Dead Drafters" — Message 9976, diagnosis of dead drafter threads.

[16] "The Moment of Diagnosis: A Todo List That Tells a Story" — Message 9977, todo list as diagnostic tool.

[17] "Diagnosing the Dead Drafters: A Turning Point in Multi-GPU Training Debugging" — Message 9978, the diagnosis that led to the SDPA pivot.

[18] "The Moment of Action: Replacing flex_attention with Per-Block SDPA in a Multi-GPU Training Pipeline" — Message 9979, the first edit in the SDPA replacement.

[19] "The SDPA Pivot: Replacing torch.compile(flex_attention) to Escape the FX Tracing Race Condition" — Message 9980, the second edit in the SDPA replacement.

[20] "The Pivot: Replacing torch.compile(flex_attention) with Per-Block Batched SDPA" — Message 9981, the third edit replacing the DFlashAttention class.

[21] "The Final Piece: Updating DFlashDecoderLayer to Complete the SDPA Transition" — Message 9982, the fourth edit wiring the decoder layer.

[22] "The Final Integration: Replacing BlockMask with an Index Builder in DFlashDrafter.forward" — Message 9983, the fifth edit integrating the index builder.

[23] "The Diagnostic Read: A Pivot Point in the DFlash Training Debugging Saga" — Message 9984, diagnostic read.

[24] "The Verification Read: A Quiet Pivot in Multi-Threaded DFlash Training" — Message 9985, verification read.

[25] "The Quiet Edit That Killed a Race Condition: Replacing flex_attention with Per-Block SDPA in DFlash Training" — Message 9986, edit to replace flex_attention.

[26] "The Layer Loop Edit: A Pivotal Moment in the DFlash Drafter Refactoring" — Message 9987, layer loop edit.

[27] "The Last Fossil: Cleaning Up After a Multi-Threaded torch.compile War" — Message 9988, cleanup edit.

[28] "The Final Cut: Removing flex_attention from the DFlash Drafter" — Message 9989, final removal of flex_attention.

[29] "The Final Touch: Cleaning Up Unused Imports After a Major Refactoring" — Message 9990, import cleanup.

[30] "The Syntax Check That Almost Goes Unnoticed: Engineering Discipline in AI-Assisted Development" — Message 9991, syntax verification.

[31] "The Verification That Confirmed a Pivot: Grepping Away flex_attention in DFlash Training" — Message 9992, grep verification.

[32] "The Verification That Confirms a Surgical Refactor: Grep as Quality Gate" — Message 9993, grep as quality gate.

[33] "The Pivot: A Transitional Message That Marks a Shift in Debugging Strategy" — Message 9994, transitional message.

[34] "The Shell That Swallowed the Diagnostic: A Case Study in Nested Escaping and Multi-Threaded Training Debugging" — Message 9995, shell diagnostic.

[35] "The Missing Library That Was Costing 10x Performance: A Diagnostic Deep Dive" — Message 9996, the model inspection script that revealed the missing fast-path warning.

[36] "The Missing Kernel: How a Single Warning Message Revealed the True Bottleneck in DFlash Training" — Message 9997, reasoning about the warning and follow-up investigation.

[37] "The Hidden Bottleneck: How Missing CUDA Extensions Sabotaged a Multi-GPU Training Pipeline" — Message 9998, further investigation.

[38] "The 10x Bottleneck: How a Missing CUDA Extension Was Silently Crippling 75% of a Model's Layers" — Message 9999, the discovery that 48/64 layers were affected.

[39] "The Missing Kernel: How a Single Bash Command Exposed the Root Cause of a 10x Training Slowdown" — Message 10000, bash command exposing the bottleneck.

[40] "The Silence That Speaks Volumes: A Simple Grep That Exposed a 10x Training Bottleneck" — Message 10001, grep verification.

[41] "The Triton Version Check: A Diagnostic Pivot in Multi-GPU Training Debugging" — Message 10002, Triton version check.

[42] "The Moment of Discovery: Identifying the GatedDeltaNet Slow Path Bottleneck" — Message 10003, GatedDeltaNet discovery.

[43] "The Moment the Bottleneck Was Found: Installing causal-conv1d in a DFlash Training Environment" — Message 10004, installation attempt.

[44] "The Missing Package Manager: How Installing uv Unblocked a 10x Performance Bottleneck" — Message 10005, uv installation.

[45] "The Missing Kernel: When a Single Build Error Exposes the Fragility of ML Infrastructure" — Message 10006, build error.

[46] "The Blackwell Build Barrier: Diagnosing CUDA Extension Failures in a Multi-GPU Training Pipeline" — Message 10007, Blackwell build issues.

[47] "The Quiet Diagnostic: How a Single ls Command Exposed the Limits of Environment Reconnaissance" — Message 10008, ls diagnostic.

[48] "A Single Package Installation That Unblocked 75% of a Model's Performance" — Message 10009, flash-linear-attention installation.

[49] "The Missing Wheel: A Microcosm of ML Infrastructure Debugging" — Message 10010, missing wheel.

[50] "The Missing Compiler: When Installing a Python Package Reveals a Deeper Infrastructure Gap" — Message 10011, missing nvcc discovery.

[51] "The Missing Compiler: How a Single which nvcc Command Exposed the CUDA Dependency Gap in Multi-GPU Training" — Message 10012, nvcc check.

[52] "The Missing CUDA Compiler: A Diagnostic Pivot in the DFlash Training Pipeline" — Message 10013, diagnostic pivot for CUDA compiler.

[53] "Probing the Depths: How a Single Diagnostic Command Uncovered the Kernel Architecture of a 27B-Parameter Model" — Message 10014, kernel architecture probe.

[54] "The Moment of Verification: Diagnosing a Silent Performance Killer in Multi-GPU Training" — Message 10015, verification of fast path.

[55] "Tracing the Fast Path: A Diagnostic Deep Dive into Missing CUDA Extensions" — Message 10016, tracing the fast path.

[56] "The Silent Diagnostic: When a Verification Command Returns Nothing" — Message 10017, silent diagnostic.

[57] "The Silence That Changed Direction: An Empty Message as a Pivot Point in AI-Assisted Debugging" — Message 10018, empty message pivot.

[58] '"Just Stop the Current Bad Run": A Turning Point of Frustration and Decisive Action' — Message 10019, user directive to stop the run.

[59] "The Kill Command: A Single Message That Reveals the Fragility of Multi-GPU Training" — Message 10020, kill command.

[60] "The Reset: When a Training Run Must Die" — Message 10021, reset.

[61] "The Confirmation of Silence: A Single Bash Command That Marks a Pivot Point" — Message 10022, confirmation.

[62] "The Checkpoint After the Kill: Diagnosing GatedDeltaNet's Slow Fallback" — Message 10023, checkpoint after kill.

[63] "The Missing CUDA Extension: Diagnosing a 10x Training Slowdown at the Boundary of Hardware and Software" — Message 10024, missing CUDA extension.

[64] "The Diagnostic Pivot: How a Single Bash Command Uncovered the CUDA Compilation Barrier in DFlash Training" — Message 10025, diagnostic pivot.

[65] "The Silence of nvcc: A Diagnostic Dead End in Multi-GPU Training" — Message 10026, nvcc silence.

[66] "The Missing CUDA Toolkit: A Pivotal Discovery in Containerized ML Engineering" — Message 10027, missing CUDA toolkit.

[67] "The Diagnostic Pivot: When a Single Bash Command Reveals the Depth of an Infrastructure Problem" — Message 10028, diagnostic pivot.

[68] "The Hidden Dependency: Installing a CUDA Keyring to Unlock 75% of a Model's Performance" — Message 10029, CUDA keyring installation.

[69] "The Missing Compiler: Installing CUDA Toolkit in a Container to Unlock GatedDeltaNet Fast Paths" — Message 10030, CUDA toolkit installation.

[70] "The Four-Minute Build: Installing causal-conv1d to Restore Fast Kernel Paths in a Multi-GPU Training Pipeline" — Message 10031, causal-conv1d build.

[71] "The Verification That Unlocked a Model's True Performance" — Message 10032, fast path verification.

[72] "The Checkpoint Message: When Two Bottlenecks Finally Break" — Message 10033, the checkpoint message marking both bottlenecks addressed.

[73] "The Deployment That Tied It Together: A Single File Copy in a Multi-Threaded ML Training Saga" — Message 10034, file deployment.

[74] "Verifying the Fix: Benchmarking GatedDeltaNet Fast Path in a Multi-GPU Training Pipeline" — Message 10035, benchmark verification.

[75] "The Bittersweet Benchmark: A Moment of Progress and Setback in DFlash Training" — Message 10036, the benchmark that confirmed the fast path but revealed the SDPA crash.