Segment 54

In this sub-session, the user pivoted from architecture tuning to data-centric improvements. They set up SGLang on SM120 for high-throughput batch inference, generating 193K diverse prompts from Infinity-Instruct, WebInstruct, CodeFeedback, MetaMathQA, Hermes FC, and Agent Training datasets, producing 523M output tokens. These were tokenized and merged with the existing 902K dataset, yielding 1,095,082 samples totaling 2.411B tokens. Training was resumed from step 690 with the expanded dataset, but GPU 6 OOM'd during ramp-up. Attempts to restart with reduced memory parameters (token_budget 45056, max_batch_size 48) and a 6-target + 2-drafter configuration failed to recover the previous 20 Ktok/s throughput. The user rejected the degraded performance, and the assistant began reverting torch from cu130 back to cu128 to restore the original memory budget and stable 5t+3d configuration.

The 200MB That Broke a Training Pipeline: Data Expansion, Dependency Cascades, and the Fragility of GPU Memory Budgets on 8× Blackwell GPUs 4116 words

Chunks

• The Data Expansion Odyssey: From Training Halt to 193K Diverse Prompts on Blackwell GPUs
  2515 words
• The 200MB That Broke a Training Pipeline: A Case Study in Dependency Cascades and Memory Budget Crises
  2215 words
• The 200MB That Broke a Training Pipeline: A Case Study in Dependency-Induced Memory Regression
  2697 words