Add Gemma 4 support#492

Open

jlamypoirier wants to merge 18 commits intomainfrom

worktree-gemma

Commits on May 5, 2026

Add embedding_scale and final_logit_softcap for Gemma 4 support

jlamypoirier
and
claude
committed
Add QK norm and post-mixer/MLP normalization for Gemma 4 support

jlamypoirier
and
claude
committed
Expand test_attention with MQA/MHA/rotary/norm coverage

jlamypoirier
and
claude
committed
Add ProportionalRotary and rewrite test_rotary as parametrized suite

jlamypoirier
and
claude
committed
Add value_norm, shared_key_value, and FixedRMSNorm (Gemma 4 attention)

jlamypoirier
and
claude
committed
Fix in-place rotary corruption of query_norm backward context

jlamypoirier
and
claude
committed
Add HybridMoEMLP and independent pre-norm controls for decoder block

jlamypoirier
and
claude
committed
Add Gemma 4 HuggingFace checkpoint converter

jlamypoirier
and
claude
committed
Improve Gemma4 converter: shared_key_value, MoE weight shapes, roundtrip test

jlamypoirier
and
claude
committed
Fix test_rotary and test_mlp failures

jlamypoirier
and
claude
committed
Add output_scale to DecoderBlock for Gemma 4 layer_scalar

jlamypoirier
and
claude
committed
Fold output_scale into bias_dropout_add; fix subprocess interpreter

jlamypoirier
and
claude
committed
Move pre/post norm to MLPBaseConfig; add MoE router preprocessing

jlamypoirier
and
claude
committed
Add learnable per-expert scale to MoE router
jlamypoirier
committed
Exercise HybridMoE in gemma4 test fixture
jlamypoirier
committed
Guard converters against unsupported features
jlamypoirier
committed
Address PR review feedback
jlamypoirier
committed
Address second-round PR review

jlamypoirier
and
claude
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Gemma 4 support#492

Add Gemma 4 support#492
jlamypoirier wants to merge 18 commits intomainfrom
worktree-gemma

Commits on May 5, 2026

Add embedding_scale and final_logit_softcap for Gemma 4 support

Add QK norm and post-mixer/MLP normalization for Gemma 4 support

Expand test_attention with MQA/MHA/rotary/norm coverage

Add ProportionalRotary and rewrite test_rotary as parametrized suite

Add value_norm, shared_key_value, and FixedRMSNorm (Gemma 4 attention)

Fix in-place rotary corruption of query_norm backward context

Add HybridMoEMLP and independent pre-norm controls for decoder block

Add Gemma 4 HuggingFace checkpoint converter

Improve Gemma4 converter: shared_key_value, MoE weight shapes, roundtrip test

Fix test_rotary and test_mlp failures

Add output_scale to DecoderBlock for Gemma 4 layer_scalar

Fold output_scale into bias_dropout_add; fix subprocess interpreter

Move pre/post norm to MLPBaseConfig; add MoE router preprocessing

Add learnable per-expert scale to MoE router

Exercise HybridMoE in gemma4 test fixture

Guard converters against unsupported features

Address PR review feedback

Address second-round PR review