The Three-Word Optimization: "use 128 workers"

In the middle of a sprawling, multi-week machine learning engineering session — one that had already spanned GPU driver installation, flash-attn compilation battles, multi-node Ray debugging, and the design of a custom speculative decoding training pipeline — a single three-word user message arrives: "use 128 workers" ([msg 7724]). On its surface, it is a trivial parameter adjustment. In context, it is a microcosm of the entire engineering philosophy driving this project: identify the bottleneck, turn the dial to maximum, and let the hardware eat.

The Message in Full

The user's message, quoted exactly, is:

use 128 workers

That is the entirety of the communication. No explanation, no justification, no question mark. It is a command — concise, unambiguous, and backed by the full weight of the preceding conversation.

The Context That Gives It Meaning

To understand why this message was written, one must trace the chain of events that led to it. The session had been building toward training a DFlash speculative decoder — a lightweight drafter model that learns to predict multiple tokens per forward pass of a large target model. The training pipeline had been decomposed into three phases, and the team had just completed implementing all three scripts ([msg 7705]): a standalone DFlash model implementation (dflash_model.py), a tokenization script (tokenize_completions.py), and an online training script (train_dflash_online.py).

The tokenization phase — Phase 1 — was the immediate task. It involved downloading 1,805 JSONL completion files from S3 (totaling 902,087 samples, 1.64B output tokens), applying the Qwen3.6 chat template with thinking tokens, generating loss masks, and saving the result as a Hugging Face Arrow dataset for upload back to S3. The assistant had initially written the download loop as serial — one file at a time ([msg 7705]). The user immediately spotted this inefficiency: "Are we parallelising the download? 2k files ideally we can download 10-50 at a time" ([msg 7706]). The assistant complied, adding a 32-thread download pool ([msg 7708]).

Then came a broader directive: "Tokenize here and put tokenized in S3 too. Use high parallelism too" ([msg 7712]). The assistant took this to heart, rewriting the tokenization to use multiprocessing for the CPU-bound work and parallelizing the S3 upload ([msg 7717]). After fixing a missing import torch that crashed the first run ([msg 7721]), the assistant launched the script with --download-workers 32 --tokenize-workers 12 ([msg 7723]). The script was now running — but the user saw the --tokenize-workers 12 parameter and immediately intervened with "use 128 workers."

The Reasoning and Motivation

The user's motivation is straightforward: maximize throughput. The tokenization of 902,087 samples is a CPU-bound operation — each sample requires loading a JSON object, formatting messages through the Qwen3.6 chat template, running the tokenizer (a fast Rust-based tokenizer via the transformers library), computing loss masks, and serializing the result. With 1.87 billion tokens to process across nearly a million samples, every additional worker core directly reduces wall-clock time.

The number 128 is not arbitrary. The machine executing this tokenization — the same machine that had been the workhorse of the entire session — is a high-end server. Earlier in the session, the assistant had verified the CPU topology and memory configuration. A machine with 128+ logical cores (common for dual-socket AMD EPYC or Intel Xeon server configurations) can comfortably run 128 worker processes for a CPU-bound, embarrassingly parallel task like tokenization. The user knew this hardware capability and was demanding it be fully utilized.

There is also an implicit critique in the message: 12 workers was too conservative. The assistant had chosen 12 as a "safe" default — high enough to show parallelism, low enough to avoid resource contention. The user rejected this caution. When you have 128 cores and 902K samples to process, you use all of them. The message reflects a "go fast or go home" engineering culture where hardware is pushed to its limits and software is expected to scale accordingly.

Assumptions Embedded in the Message

The message carries several assumptions, most of them correct:

Hardware assumption: The machine has at least 128 CPU cores. This is a safe bet given the context — the server had been provisioned with 8× NVIDIA RTX PRO 6000 Blackwell GPUs, which implies a motherboard and CPU platform capable of supporting that many PCIe lanes. Such platforms (e.g., AMD EPYC 9004 series or Intel Xeon Scalable) routinely offer 96–192 cores.

Software assumption: The tokenization script's multiprocessing implementation can handle 128 workers without hitting diminishing returns or resource exhaustion. The script used multiprocessing.Pool with a chunked processing strategy, meaning each worker independently tokenizes a batch of samples and writes to a shared output queue. At 128 workers, the bottleneck shifts from CPU to I/O — the tokenizer itself is fast (Rust-based), but the overhead of process spawning, pickle serialization of results, and file writes could become significant. The user implicitly trusts that the implementation is efficient enough to benefit from this level of parallelism.

Network assumption: The S3 upload at the end of tokenization can handle 128 concurrent connections. The upload phase uses a separate thread pool, and 128 simultaneous PUT requests to S3 could saturate the network link or trigger S3 throttling. However, the user likely considers this acceptable — S3 is designed for high concurrency, and the local network (likely 25 GbE or faster in a data center environment) can handle the throughput.

No memory concern: 128 worker processes each holding a copy of the tokenizer could consume significant memory. The Qwen3.6 tokenizer vocabulary is modest (~152K tokens), and the tokenizer object itself is relatively lightweight, but 128 copies could add up to several GB of RSS. The user assumes this is within the machine's memory budget — again, a safe bet for a server with 512 GB or more of RAM.

What the Message Reveals About the Engineering Process

This message is a window into the collaboration dynamic. The assistant, left to its own devices, chose conservative parallelism (12 workers). The user, who owns the hardware and pays the cloud bills, pushed for aggressive parallelism (128 workers). This tension between "safe and correct" and "fast and efficient" recurs throughout the session.

Earlier, the same dynamic played out with flash-attn compilation: the assistant tried to build with MAX_JOBS=128 (using all cores), which caused memory exhaustion and build failures. The user had to intervene and reduce it to MAX_JOBS=20 to get a successful build. Now, for tokenization — a purely CPU-bound, non-memory-intensive task — the user is pushing the other direction: 12 is too few, go to 128. The user understands the difference between compilation (which is memory-intensive per job) and tokenization (which is CPU-bound with modest per-process memory), and adjusts expectations accordingly.

The Outcome

The assistant would go on to implement this instruction, likely editing the script's default or adjusting the command-line invocation. The tokenization would complete in approximately 6.5 minutes — a remarkable throughput of ~2,300 samples per second across 128 workers, producing 1.87 billion tokens across 47 Arrow shards ([chunk 44.1]). The 12-worker configuration the assistant initially chose would have taken roughly 70 minutes. The user's three-word intervention saved over an hour of wall-clock time.

Conclusion

"use 128 workers" is a message that contains no explanation because none is needed. It is the product of a user who knows their hardware, understands the workload characteristics, and trusts the assistant to execute. It is a reminder that in high-performance ML engineering, the difference between a 6-minute job and a 70-minute job is often not a smarter algorithm or a better model — it is simply turning the parallelism knob to the right setting. The message is brief, but the reasoning behind it encompasses hardware topology awareness, workload profiling, operational experience, and a willingness to push systems to their limits.