Skip to content

[fix] BC for legacy configs with top-level rope_theta when rope_parameters is set via rope_scaling#45036

Open
matdou wants to merge 1 commit intohuggingface:mainfrom
matdou:fix-rope-theta-legacy
Open

[fix] BC for legacy configs with top-level rope_theta when rope_parameters is set via rope_scaling#45036
matdou wants to merge 1 commit intohuggingface:mainfrom
matdou:fix-rope-theta-legacy

Conversation

@matdou
Copy link

@matdou matdou commented Mar 26, 2026

What does this PR do?

Fixes #45030

Configs like tiny-random/glm-4v store rope_theta at the top level of config.json alongside a rope_scaling dict (legacy format). For config classes that don't declare rope_parameters as a dataclass field (e.g. Glm4vConfig), convert_rope_params_to_dict is skipped because hasattr(self, "rope_parameters") is False before the kwargs loop.

During initialization, the kwargs loop sets self.rope_parameters via the rope_scaling property setter — but without rope_theta — and assigns self.rope_theta as a separate attribute. When @strict's validate_rope runs after __init__, it expects rope_theta inside rope_parameters and raises a KeyError.

Fix

Add a normalization step in PreTrainedConfig.__post_init__ after the kwargs loop:

  • If rope_parameters is a dict
  • AND rope_theta is missing from it
  • AND self.rope_theta exists
  • AND the rope type is "default"

→ Inject rope_theta into rope_parameters.

This ensures backward compatibility with legacy configs while keeping behavior
unchanged for newer ones.

Why this is safe

  • New configs already:
    • declare rope_parameters as a field → go through
      convert_rope_params_to_dict, or
    • include rope_theta inside rope_parameters
  • The fix only applies to legacy edge cases
  • No change to validation logic (modeling_rope_utils.py untouched)

Test

python -c "from transformers import AutoConfig; AutoConfig.from_pretrained('tiny-random/glm-4v'); print('ok')"

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

  • I confirm that this is not a pure code agent PR.

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Tagging: @ArthurZucker @Cyrilvallez (model loading)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tiny-random glm4v configuration can't load due to config validation changes

1 participant