Enhancement - Fix preprocessing crash when num_workers=0 in Megatron GPT3 dataset generation#800

Open

polarG wants to merge 5 commits intomainfrom

dev/hongtaozhang/fix-num_workers-usage

Commits on Apr 1, 2026

Fix the num_workers usage.
Hongtao Zhang
committed

Commits on Apr 27, 2026

Address PR review: derive --output-prefix from data_prefix, add tests
Hongtao Zhang
committed
Tighten test substring matches with trailing space
Hongtao Zhang
committed

Commits on May 2, 2026

Address PR review: use normalize_command and valid code_base in dataset generate test
Hongtao Zhang
committed

Commits on May 3, 2026

Address review feedback for num_workers / data_prefix handling
Hongtao Zhang
committed