Skip to content

From NVIDIA Megatron-LM for visibility #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
RaymondLi0 wants to merge 6,246 commits into
base: multi-query-attention
Choose a base branch
from
Open

From NVIDIA Megatron-LM for visibility #18

RaymondLi0 wants to merge 6,246 commits into from

Conversation

@RaymondLi0
Copy link
Collaborator

@RaymondLi0 RaymondLi0 commented Jan 24, 2023

No description provided.

@RaymondLi0 RaymondLi0 changed the base branch from multi-query-attention to before-merge June 20, 2023 20:12
@RaymondLi0 RaymondLi0 changed the base branch from before-merge to multi-query-attention June 20, 2023 20:12
hexinw-nvidia and others added 28 commits January 14, 2026 18:29
@hexinw-nvidia @rhewett-nv
@yaoyu-33 @dimapihtar @Phlip79 @tdene @deepakn94
Various fixes to in-job restarter and better time accounting of start…
2dc8873 
…up operations (#2698)

Signed-off-by: dimapihtar <dpihtar@gmail.com>
Co-authored-by: Russell Hewett <rhewett@nvidia.com>
Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com>
Co-authored-by: Mcore Bot <mcore-bot@nvidia.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: dimapihtar <dpihtar@gmail.com>
Co-authored-by: Philip Petrakian <pgpetrak@gmail.com>
Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com>
Co-authored-by: Deepak Narayanan <2724038+deepakn94@users.noreply.github.com>
@BestJuly @shifangx @Wohox
feat(moe): Support placing MTP layers into standalone stages (#2136)
5247a1f 
Signed-off-by: Li Tao <lit@nvidia.com>
Signed-off-by: lit <lit@nvidia.com>
Co-authored-by: shifangx <shifangx@nvidia.com>
Co-authored-by: Pingtian Li <pingtianl@nvidia.com>
@lmcafee-nvidia
Inference | Evict overflow paused requests from context. (#2926)
fcb4d54
@mathemakitten @tdene
Enable training cudagraphs for RL (#2452)
a01c476 
Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com>
@Deepak-J0shi
Fix minor README wording and capitalization (#2928)
cb1e9a8
@ko3n1g
Revert "Various fixes to in-job restarter and better time accounting …
411c3d8 
…of startup operations (#2698)"

This reverts commit 2dc8873.
@ko3n1g
ci: Restore grpo tests (#2952)
463b428 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@github-actions
Update copy-pr-bot.yaml [skip ci]
a0ab5a1
@tdene @ko3n1g
Fix GitHub GRPO resharding functional test (#2927)
5542273 
Co-authored-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
gb200 & hybrid_static_inference_tp1_pp1_2B_logitsmatch
61c5839 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
cp: ci(fix): GB200 racecondition (2962) into main (#2963)
4bbcd72 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
Revert "cp: ci(fix): GB200 racecondition (2962) into main (#2963)"
c3253bd 
This reverts commit 4bbcd72.
@lhb8125
feat(moe): Fine-grained activation offloading (#1913)
b97184d 
Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
Signed-off-by: Hongbin Liu <hongbinl@cw-dfw-cs-001-login-02.cm.cluster>
Signed-off-by: root <root@eos0321.eos.clusters.nvidia.com>
Co-authored-by: Hongbin Liu <hongbinl@cw-dfw-cs-001-login-02.cm.cluster>
Co-authored-by: root <root@eos0321.eos.clusters.nvidia.com>
@Phlip79
Add out-of-SLA link (#2903)
4f1e720
@jon-barker
Fix broken mamba-moe unit test (#2970)
235da16
@ko3n1g
ci: Fix GB200 change (#2969)
4fa27a1 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@tdene
Update golden values for reshard test (#2971)
a0908fc
@ko3n1g
chore: Update golden values (#2973)
ad2ab2c 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci(hotfix): _TENSORBOARD_PATH
99ea287 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@jkamalu @nvda-bnorick
Enable phase transition iterations (#2938)
4da7da9 
Co-authored-by: Brandon Norick <bnorick@nvidia.com>
@ko3n1g
gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_no_mtp_no_a2a_ovlp_fine_grained_of…
710ff97 
…floading

Signed-off-by: oliver könig <okoenig@nvidia.com>
@jon-barker
add missing import in rl_utils.py (#2915)
048825c
@ChenhanYu
Pass through --trust-remote-code and add this to all Nemotron model c…
7f85843 
…onfigs (#2939)
@wdykas @ericharper
Cuda 13 UVM (#2957)
15ea236 
Co-authored-by: Eric Harper <eharper@nvidia.com>
@Wohox @Phlip79
[Main] Partial CUDA Graph support for EP Overlap (#2184)
1501c0f 
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
@duncanriach @ericharper @Phlip79
Add sequence packing support for hybrid model (#2913)
03c0727 
Co-authored-by: Eric Harper <eharper@nvidia.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
@xuwchen
docs(megatron-fsdp): add Megatron-FSDP user guide (#2396)
55103f9
@kunlunl @Phlip79
DeepSeek V3.2 support (#2440)
382eeea 
Signed-off-by: kunlunl <kunlunl@nvidia.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
Co-authored-by: Philip Petrakian <pgpetrak@gmail.com>
ko3n1g and others added 30 commits February 6, 2026 17:08
@ko3n1g
docs: Add changelog for 0.15.3 (#3286)
a116ce3 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@chtruong814
ci: Set throughput tests as flaky (#3301)
4376cc5 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@ko3n1g
chore: Move GB200 tests to nightly (#3302)
f92460b 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@nschank @ko3n1g @yashaswikarnati
Ensure type-checker understands use of Submodules in bert_model (#3256)
cfbe9b5 
Co-authored-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com>
@nschank @chrisgrimm
@ericharper @yaox12 @chtruong814
Override extra_repr instead of __repr__ (#3200)
a63d045 
Co-authored-by: Chris Grimm <chris@reflection.ai>
Co-authored-by: Eric Harper <eharper@nvidia.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
@nschank @yaox12 @chtruong814
Replace ModuleSpec with Protocols for LayerNorm submodules (#3090)
f68c7c1 
Co-authored-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
@ko3n1g
chore: Remove gpt_grpo_tp2tp1_pp4pp2_dp8_583m_throughputtest
2f99ee8 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@wdykas
Non colocated refit (#3213)
e3ae6e4 
Co-authored-by: root <root@gpu-h100-0348.cm.cluster>
Co-authored-by: root <root@gpu-h100-0193.cm.cluster>
Co-authored-by: root <root@gpu-h100-0082.cm.cluster>
Co-authored-by: root <root@gpu-h100-0495.cm.cluster>
Co-authored-by: William Dykas <wdykas@cw-pdx-cs-001-vscode-02.cm.cluster>
Co-authored-by: root <root@gpu-h100-0213.cm.cluster>
Co-authored-by: root <root@gpu-h100-0435.cm.cluster>
Co-authored-by: root <root@gpu-h100-0188.cm.cluster>
Co-authored-by: root <root@gpu-h100-0032.cm.cluster>
Co-authored-by: root <root@gpu-h100-0023.cm.cluster>
Co-authored-by: root <root@gpu-h100-0368.cm.cluster>
Co-authored-by: root <root@gpu-h100-0203.cm.cluster>
Co-authored-by: root <root@gpu-h100-0229.cm.cluster>
Co-authored-by: root <root@gpu-h100-0123.cm.cluster>
Co-authored-by: root <root@gpu-h100-0217.cm.cluster>
Co-authored-by: root <root@gpu-h100-0496.cm.cluster>
Co-authored-by: root <root@gpu-h100-0261.cm.cluster>
@xiaoxi-wangfj @yaox12
Fuse permute+pad and unpermute+unpad ops for FP8/FP4 training (#2763)
554ce49 
Signed-off-by: xiaoxi-wangfj <690912414@qq.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
@shjwudp @yaox12
Add check to prevent MFSDP from numeric issue in gradient accumulate …
7cbbba2 
…fusion (#2904)

Co-authored-by: Xin Yao <xiny@nvidia.com>
@c1lovez1 @yaox12
update get_embedding_ranks and get_position_embedding_ranks docstrings (
c99c962 
#3223)

Co-authored-by: Xin Yao <xiny@nvidia.com>
@chtruong814
ci: Add secrets detector (#3180)
6d81e3d 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@skydoorkai @BestJuly
@ericharper @Phlip79
Param offset in _ParamAndGradBucket should be aligned (#3007)
a3ec4b0 
Signed-off-by: skydoorkai <htsantaclara@163.com>
Co-authored-by: lit <lit@nvidia.com>
Co-authored-by: Eric Harper <eharper@nvidia.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
@yeyu-nvidia @dimapihtar
updates to support modelopt EAGLE training with CP (#3147)
916301a 
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
@nschank @chtruong814
Ensure type-checker understands use of Submodules in llava_model (#3257)
6103cb5 
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
@shjwudp @yaox12
M-FSDP: Remove redundant stream waits in HSDP to prevent CG fail (#2941)
4ff7686 
Co-authored-by: Xin Yao <xiny@nvidia.com>
@dimapihtar @yaox12
fully remove legacy tokenizer system (#2946)
3257093 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
@ahmadki @yaox12
General README and pyproject fixes (#2907)
3069591 
Co-authored-by: Xin Yao <xiny@nvidia.com>
@ko3n1g
chore: More aggressive checkpointing (#3315)
3bb539e 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: Pin down setuptools to lt 82 (#3313)
c072f89 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
fix: T5 dataset (#3307)
9ddbce3 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
fix: numpy overflow (#3306)
f14d161 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@chtruong814
ci: Revert "ci: Add secrets detector (#3180)" (#3330)
8d79987 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@ko3n1g
ci: Add more tests, run on merge-queue (#3317)
55c3e63 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@chtruong814
ci: Remove merge-gate environment check (#3331)
ba76934 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@kwyss-nvidia @yaox12 @ericharper
Use FP4 context for mamba (#2604)
ab5e277 
Co-authored-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: Eric Harper <eharper@nvidia.com>
@chtruong814
ci: Ensure we run all functional tests in merge group (#3332)
fc557ec 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@nschank
Replace ModuleSpec with Protocols for inputs to MLP (#3084)
55198ba
@chtruong814
ci: Fix merge queue functional tests (#3337)
5eb20b8 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@ko3n1g
ci: skip queue in merge-gate (#3343)
367f0b8 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.