Parallelizing S3 Downloads with boto3: A Pragmatic Response to a Data Pipeline Bottleneck

Introduction

In the course of training a DFlash speculative decoding drafter for the Qwen3.6-27B language model, a critical data pipeline bottleneck emerged. The training required approximately 19 GB of tokenized completion data stored in an S3-compatible object store, but the initial download approach—using the aws s3 sync command-line tool—was proving painfully slow. After several minutes, only 10 out of 47 files had been transferred, representing roughly 4.5 GB of the total. The user, observing this sluggishness, asked a simple but pointed question: "For download can we parallelise more?" ([msg 7838]). This article analyzes the assistant's response to that question: a single message ([msg 7839]) that killed the existing download, assessed what had been completed, and replaced the single-threaded AWS CLI sync with a custom Python script using boto3 and a 32-worker ThreadPoolExecutor.

This message is a masterclass in pragmatic systems engineering under real-world constraints. It demonstrates how a deep understanding of tool internals, network I/O patterns, and Python concurrency primitives can be combined to solve a performance bottleneck in minutes, using nothing more than an inline script executed over SSH. The message also reveals important assumptions about network bandwidth, S3 compatibility, and the trade-offs between parallelism and complexity.

Context: The DFlash Training Pipeline

To understand the significance of this message, one must appreciate the broader context. The team was training a DFlash (Drafting with Flash Attention) speculative decoding drafter for the Qwen3.6-27B model on a freshly provisioned machine equipped with 4× NVIDIA RTX PRO 6000 Blackwell GPUs, 1 TB of RAM, and 962 GB of disk space ([msg 7829]). This was the third machine attempted in the session—the first had connectivity issues with GitHub, and the second lacked the aws CLI tool ([msg 7826]). The new machine had uv preinstalled and everything was being set up from scratch: a Python virtual environment, PyTorch 2.11.0 with CUDA 13.0 support, the FLA (Flash Linear Attention) library, the Qwen3.6-27B model weights (52 GB, downloaded in an impressive 29 seconds), and the tokenized training data from S3.

The data download was the slowest component. The initial command used aws s3 sync with the --no-progress flag, running in the background. When the assistant checked progress a few minutes later ([msg 7836]), only 10 files had arrived—about 4.5 GB out of an estimated 19 GB total. The user's question about parallelization was entirely reasonable: aws s3 sync is fundamentally single-threaded for individual file transfers. While it can handle multiple files, the throughput per-file is limited by a single TCP connection, and on high-bandwidth connections (the machine had 256 CPU cores and presumably excellent network), this leaves significant performance on the table.

The Message: Quoted and Analyzed

The assistant's response was a single bash command executed over SSH, containing an inline Python script. Here is the message, with credentials redacted:

ssh -o StrictHostKeyChecking=no -p 10638 root@154.59.156.41 '
# Kill slow single-threaded aws s3 sync
pkill -f "aws s3" 2>/dev/null
echo "Killed old sync"

# Check what we already have
echo "Already downloaded:"
ls /workspace/tokenized_completions/ | wc -l
echo ""

# Use Python with boto3 + ThreadPoolExecutor for parallel download
source /root/venv/bin/activate
python3 -c "
import boto3, os, time
from botocore.config import Config
from concurrent.futures import ThreadPoolExecutor, as_completed

endpoint = \"https://eu-west-1.s3.fil.one\"
bucket = \"train-dflash-qwen36-27b\"
prefix = \"tokenized-completions/\"
dest = \"/workspace/tokenized_completions/\"
os.makedirs(dest, exist_ok=True)

client = boto3.client(\"s3\", endpoint_url=endpoint,
    aws_access_key_id=\"[REDACTED]\",
    aws_secret_access_key=\"[REDACTED]\",
    config=Config(s3={\"addressing_style\": \"path\"}, max_pool_connections=64))

resp = client.list_objects_v2(Bucket=bucket, Prefix=prefix)
files = [(o[\"Key\"], o[\"Size\"]) for o in resp.get(\"Contents\", []) if o[\"Size\"] > 0]
print(f\"Total files: {len(files)}\")

# Skip already downloaded
to_download = []
for key, size in files:
    fname = os.path.basename(key)
    local = os.path.join(dest, fname)
    if os.path.exists(local) and os.path.getsize(local) == size:
        continue
    to_download.append((key, local, size))
print(f\"Need to download: {len(to_download)}\")

def dl(args):
    key, local, size = args
    t0 = time.time()
    client.download_file(bucket, key, local)
    return os.path.basename(key), time.time()-t0, size/1e6

t0 = time.time()
with ThreadPoolExecutor(max_workers=32) as pool:
    futs = [pool.submit(dl, a) for a in to_download]
    done = 0
    for f in as_completed(futs):
        name, dt, mb = f.result()
        done += 1
        if done % 10 == 0 or done == len(to_download):
            print(f\"  {done}/{len(to_download)} files, last: {name} ({mb:.0f}MB in {dt:.1f}s)\")

elapsed = time.time() - t0
total_mb = sum(s for _,_,s in to_download) / 1e6
print(f\"Done: {total_mb:.0f} MB in {elapsed:.1f}s ({total_mb/elapsed:.0f} MB/s)\")
" 2>&1'

The output was simply (no output), which is notable and will be discussed later.

WHY: The Reasoning and Motivation

The motivation for this message was straightforward: the existing download was too slow, and the user explicitly requested parallelization. But the reasoning behind how the assistant chose to address this reveals several layers of decision-making.

First, the assistant recognized that the bottleneck was not the network link itself but the single-threaded nature of aws s3 sync. The machine had 256 CPU cores and was connected to a high-bandwidth network (likely 10 Gbps or more). The S3 sync tool, while convenient, serializes file transfers and cannot take advantage of this parallelism. On a machine with this much headroom, saturating the network link requires concurrent transfers.

Second, the assistant chose to replace the AWS CLI entirely with a custom Python script rather than attempting to configure the CLI for higher parallelism. This was a deliberate architectural decision. The aws s3 sync command does not expose a simple "number of concurrent transfers" parameter in the way that, say, aws s3 cp with --recursive might. While one could run multiple aws s3 sync processes in parallel for different prefixes, the data was organized as a flat list of files under a single prefix. The cleanest solution was to bypass the CLI entirely and use the underlying boto3 SDK, which gives full control over concurrency.

Third, the assistant chose to kill the existing process first rather than letting it continue alongside the new download. This avoids duplicate work and potential file corruption from two processes writing to the same destination directory. It also simplifies the logic for tracking what has already been downloaded—the script checks for existing files and skips them, but only one writer at a time ensures consistency.

HOW: Decisions Made in the Implementation

The implementation reveals several deliberate design choices:

Concurrency level: The assistant set max_workers=32 in the ThreadPoolExecutor. This is an aggressive but reasonable number. With 256 CPU cores available, 32 concurrent threads are unlikely to cause CPU contention. The limiting factor will be the network bandwidth and the S3 endpoint's ability to handle concurrent requests. The max_pool_connections=64 in the boto3 config doubles the connection pool size relative to the worker count, ensuring that connection reuse doesn't become a bottleneck.

Idempotent download: The script checks if each file already exists locally with the correct size before downloading. This is critical because some files had already been transferred by the killed aws s3 sync process. Without this check, the script would waste time re-downloading 4.5 GB of data. The check is simple (file existence + size match) but sufficient for this use case—it doesn't use checksums, which would add overhead.

Progress reporting: The script prints progress every 10 files and at completion. This is a pragmatic choice: too frequent reporting would clutter the output, while too infrequent would leave the user wondering if the script is stuck. The 10-file cadence provides a good balance.

Error handling: Notably, there is no explicit error handling for individual download failures. If a single file fails to download, the ThreadPoolExecutor will re-raise the exception in the main thread when f.result() is called, which would crash the entire script. This is a trade-off: the assistant prioritized simplicity and speed of implementation over robustness. In a production pipeline, one would add retry logic and error reporting, but for a one-shot data sync, the risk was deemed acceptable.

S3 compatibility: The script uses endpoint_url to point to a non-AWS S3 endpoint (https://eu-west-1.s3.fil.one). This is a Filestack or similar S3-compatible storage service. The addressing_style: "path" configuration is necessary because some S3-compatible services don't support virtual-hosted-style URLs. This shows awareness of the nuances of S3 API compatibility.

Assumptions Made

The message makes several implicit assumptions:

Network bandwidth is the bottleneck, not CPU or memory: The assumption is that 32 concurrent downloads will saturate the network link without overwhelming the machine. This is reasonable given 256 cores and 1 TB of RAM, but it assumes the S3 endpoint can handle 32 concurrent requests without throttling.
The S3 endpoint supports concurrent downloads: Some S3-compatible storage services have rate limiting or connection limits per IP. The assistant assumes this endpoint can handle the parallelism.
File sizes in the listing are reliable for deduplication: The script uses file size to determine if a file has been fully downloaded. This assumes that a partially downloaded file (from the killed aws s3 sync) would have a different size than the complete file. This is generally true but could fail if a previous download was interrupted at exactly the right byte count with different content—a pathological edge case.
All files are under a single prefix: The script lists objects with a specific prefix and assumes no subdirectory nesting. The os.path.basename(key) extraction would flatten any directory structure, which could cause filename collisions if files in different "subdirectories" had the same name.
boto3 is already installed: The script imports boto3, which was installed earlier in the session ([msg 7830]) as part of the uv pip install command that included boto3 alongside torch, transformers, datasets, etc.
The credentials are valid for this endpoint: The AWS access key and secret key are hardcoded in the script. The assumption is that these credentials will work for the duration of the download and won't expire.

Mistakes and Incorrect Assumptions

The most notable issue is the output: (no output). The bash command was constructed to print progress messages from the Python script via 2>&1, which should have captured all stdout and stderr. The fact that no output appeared suggests one of several possibilities:

The command timed out: The SSH command may have been terminated by a timeout before the script completed or produced output. The previous message ([msg 7837]) had shown the S3 download was slow, and earlier commands in the session had been terminated by timeouts (e.g., [msg 7826] shows a 30-second timeout). If the script took more than the default SSH timeout to list objects and begin downloading, the connection might have been severed before any output was printed.
The Python script failed silently: If there was an import error (e.g., boto3 not installed correctly, or botocore missing), the script would fail before producing any output. However, boto3 was installed in [msg 7830] and the script uses standard library modules otherwise.
The SSH command structure had a quoting issue: The nested quotes (single quotes for the outer bash command, escaped double quotes for the Python string) are fragile. A subtle quoting error could cause the Python code to be malformed when executed remotely. The assistant's next message ([msg 7840]) simply checks the download progress again, finding still only 10 files and 4.7 GB. This suggests the parallel download either didn't run, didn't complete, or didn't produce the expected speedup. The user then follows up with additional guidance about using vens (likely referring to a tool or approach for faster downloads), indicating that the parallelization attempt may not have resolved the bottleneck. Another potential mistake: the script uses ThreadPoolExecutor for network I/O, which is appropriate (threads are fine for I/O-bound tasks in Python due to the GIL not being a bottleneck for network operations). However, boto3's download_file internally uses its own transfer manager with configurable concurrency. The script's thread-level parallelism operates above that layer, which is fine, but the interaction between the two concurrency layers could lead to suboptimal behavior if the internal transfer manager also spawns threads.

Input Knowledge Required

To understand and write this message, the assistant needed:

Knowledge of aws s3 sync limitations: Understanding that the CLI tool is single-threaded for file transfers and cannot be easily configured for higher parallelism.
boto3 API knowledge: Familiarity with boto3.client("s3"), list_objects_v2, download_file, and the botocore.config.Config options including max_pool_connections and addressing_style.
Python concurrency primitives: Understanding that ThreadPoolExecutor with 32 workers is appropriate for I/O-bound network downloads, and that as_completed provides a convenient way to process results as they finish.
S3-compatible storage nuances: Knowledge that non-AWS S3 endpoints may require path addressing style, and that the endpoint URL must be explicitly specified.
SSH and bash quoting: The ability to construct a complex nested command with proper escaping, including single quotes for the outer shell, escaped double quotes for the Python string, and proper handling of special characters.
Context about the data: Knowing the bucket name, prefix, destination directory, and that the data consists of flat files (no subdirectories) so os.path.basename is safe.

Output Knowledge Created

This message created several pieces of knowledge:

A reusable parallel S3 download script: While written inline for a specific task, the pattern of using boto3 with ThreadPoolExecutor is a general template that could be adapted for any S3-compatible storage.
Confirmation of the download state: The script would have reported the number of files already downloaded and the number remaining, providing visibility into the data pipeline status.
A benchmark of the S3 endpoint performance: The final print statement would have reported total throughput in MB/s, providing a baseline for the storage endpoint's performance under concurrent access.
A demonstration of the assistant's debugging methodology: The message shows a pattern of identifying a bottleneck, proposing a targeted fix, and implementing it quickly with available tools—a hallmark of effective systems engineering.

The Thinking Process

The assistant's reasoning, while not explicitly shown in a separate thinking block, is visible in the structure and content of the message:

Diagnose the problem: The user says "For download can we parallelise more?" The assistant immediately understands the issue: aws s3 sync is single-threaded.
Plan the intervention: Kill the old process first to avoid conflicts. Then assess what's already been downloaded. Then implement the parallel solution.
Choose the right tool: Rather than trying to configure the AWS CLI for parallelism (which would be fighting against its design), switch to the SDK layer where concurrency is controllable.
Design for idempotency: The script checks for existing files before downloading, ensuring that killing the old process doesn't cause data loss or unnecessary re-downloads.
Set appropriate parameters: 32 workers, 64 connection pool slots, path-style addressing—each parameter reflects knowledge of the specific environment and storage backend.
Provide visibility: Progress reporting every 10 files and a final summary give the user confidence that the download is proceeding correctly. The absence of error handling and the fragile quoting are evidence of the assistant's prioritization: speed of implementation over robustness. In a fast-moving debugging session where the user is actively waiting, getting a working solution deployed quickly is often more valuable than handling every edge case.

Conclusion

Message [msg 7839] is a compact but revealing artifact of real-world ML infrastructure engineering. It demonstrates how a seemingly simple question—"can we parallelise more?"—triggers a cascade of technical decisions: process management, tool selection, API design, concurrency modeling, and error handling strategy. The assistant's response shows deep knowledge of the AWS SDK, Python concurrency, and S3 compatibility, all deployed in service of a single goal: getting 19 GB of training data onto a machine as quickly as possible.

The message also illustrates the inherent tension between speed and robustness in interactive debugging. The script was written quickly, with minimal error handling and fragile quoting, because the priority was to get the download moving faster. Whether it succeeded is ambiguous from the available output, but the approach itself—replacing a single-threaded CLI tool with a custom parallel script—is a textbook solution to this class of problem.

In the broader narrative of the DFlash training pipeline, this message represents a minor but critical infrastructure moment. The training data had to be in place before any model training could begin, and every minute spent waiting for a slow download was a minute not spent training. The assistant's willingness to drop down to the SDK level and write a custom solution, rather than accepting the CLI's limitations, is characteristic of the systems-thinking mindset that defines effective ML engineering at scale.