Add Gemma 4 fine-tune recipes with FastVisionModel support#1
Open
webcoderz wants to merge 2 commits into
Open
Conversation
Adds 12 training recipes for all Gemma 4 models (E2B, E4B, 26B-A4B MoE, 31B): - 4 SFT text recipes with reasoning preservation (<|think|>, 75% mix) - 2 SFT vision recipes using FastVisionModel + UnslothVisionDataCollator - 4 GRPO RL recipes with Gemma 4 inference params (temp=1.0, top_p=0.95, top_k=64) - 1 DPO alignment recipe (E4B) - 1 HPO hyperparameter optimization recipe (E4B) Extends 08_train_sft_unsloth.py with full vision/multimodal training path: - FastVisionModel for model loading and LoRA (finetune_vision/language/attention/mlp) - UnslothVisionDataCollator for image/audio data processing - --vision CLI flag and vision: true recipe field - Handles train_on_completions via vision collator MoE (26B-A4B) recipes default to 16-bit (4-bit not recommended for MoE). All recipes use r=16/alpha=16 per Unsloth's Gemma 4 recommendation. https://claude.ai/code/session_01Sds4ZGRD4HnVrQ9LLHysgu
README.md: - Add Gemma 4 to SFT and GRPO recipe tables (all 12 recipes) - Add Gemma 4 to Model Selection Guide (multimodal, MoE) - Add Vision / Multimodal Fine-Tuning section with data format, recipe fields, and tips - Add Gemma 4 Reasoning Preservation section (<|think|>, 75% mix, MoE 4-bit warning) - Add --vision flag to Additional Training Features table - Add make targets to quick-start bash blocks Makefile: - Add 13 Gemma 4 convenience targets: train-sft-gemma4, train-sft-gemma4-4bit, train-sft-gemma4-e2b, train-sft-gemma4-31b, train-sft-gemma4-moe, train-sft-gemma4-vision, train-sft-gemma4-vision-31b, train-dpo-gemma4, train-grpo-gemma4, train-grpo-gemma4-e2b, train-grpo-gemma4-31b, train-grpo-gemma4-moe, hpo-gemma4 Recipes: - Vision recipes: default finetune_vision_layers to false (save VRAM, enable after text works) - MoE recipes: add moe_backend: grouped_mm comment (12x faster) https://claude.ai/code/session_01Sds4ZGRD4HnVrQ9LLHysgu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds 12 training recipes for all Gemma 4 models (E2B, E4B, 26B-A4B MoE, 31B):
Extends 08_train_sft_unsloth.py with full vision/multimodal training path:
MoE (26B-A4B) recipes default to 16-bit (4-bit not recommended for MoE).
All recipes use r=16/alpha=16 per Unsloth's Gemma 4 recommendation.
https://claude.ai/code/session_01Sds4ZGRD4HnVrQ9LLHysgu