Clarify whether slow update should be validation-gated or force-injected

[luigi-sqwish]

Hi SkillOpt team, thanks for releasing the paper and code. I wanted to clarify the intended semantics of the slow-update path.

In the paper, Section 3.6 appears to describe slow update as still going through the validation/selection gate after the longitudinal guidance is injected. My reading is that the slow-update candidate should be accepted only if it passes the same held-out selection validation used for normal edits.

In current main, however, the implementation appears to force-inject slow-update guidance into both current_skill and best_skill without a selection gate:

SkillOpt/skillopt/engine/trainer.py

Lines 1580 to 1584 in 75b5c7f

	# Slow update field is force-updated into both
	# current_skill and best_skill unconditionally.
	# The epoch-level longitudinal guidance should always
	# persist — it must not be gated by step-level
	# selection scores.

The surrounding code records the action as force_accept:

SkillOpt/skillopt/engine/trainer.py

Lines 1597 to 1598 in 75b5c7f

	slow_result["action"] = "force_accept"
	current_origin = f"slow_update_epoch_{epoch:02d}"

Could you clarify which behavior is intended for reproducing/reporting SkillOpt results?

1. Should slow-update guidance be validation-gated, as described in the paper?
2. Or is the force-injection behavior in the released code the intended implementation?
3. If force-injection is intended, should exported best_skill.md be interpreted as the best validation-gated skill plus the latest slow-update field, rather than the exact best step candidate selected by the held-out gate?

Thanks!

[Yif-Yang]

Hi @luigi-sqwish — thanks for the careful read. This is a real discrepancy and your framing of it is exactly right. Let me lay out what's actually going on, since the situation is a bit more specific than "paper and code disagree on main".

TL;DR

• The paper version is gated. Section 3.6 describes slow update going through the held-out selection gate, and every experimental number reported in the paper was produced under that gated implementation.
• Our earlier code was also gated, matching the paper. The current main is newer than the paper version: a recent change switched slow update to force-inject (the force_accept path you cited), based on follow-up experiments we ran after submission.
• We did not ship the exact gated snapshot used for the paper to this GitHub repo — that's our oversight, and it's the reason your reading of main doesn't line up with the paper.
• We're going to (1) make this a config flag so both behaviors are reproducible from the same checkout, and (2) publish the post-submission force-inject experiments in the repo so the trade-off is on the record.

What the paper says vs. what main does

• Paper (Section 3.6 / sections/3_methods.tex): the slow-update candidate is built, then re-evaluated on the held-out selection split, and only accepted if it passes the same gate as a normal step-level edit. All numbers in the paper come from this gated implementation.
• main (the lines you cited, skillopt/engine/trainer.py:1580–1604): the slow update is not re-gated. Once the optimizer produces non-empty slow_update_content, we replace the protected SLOW_UPDATE field on both current_skill and best_skill, record action: "force_accept", and push the new hash into sel_cache with the current selection score so downstream steps measure deltas against the post-injection skill.

So both sides of your observation are correct — they really do disagree.

Why they disagree (the actual history)

The order of events matters here, because the paper is not wrong about its own numbers:

1. Submission-version code: gated. The codebase we used to produce every experimental result in the paper went through the held-out gate for slow update, exactly as Section 3.6 describes.
2. Post-submission experiments: force-inject. After submission we noticed that under the gated regime the slow-update field very often never landed — a longitudinal guidance block changes a fairly large region of the document, so even a qualitatively good block frequently produces a candidate that ties or marginally underperforms the current skill on the selection split, gets rejected, and is then not re-attempted the next epoch (the next epoch's slow update is computed against the un-injected skill). This made the slow-update mechanism silent on many runs, which is in tension with how we describe it in the paper as "longer-horizon momentum". The force_accept path on main is our follow-up fix.
3. Release mismatch. When we open-sourced the repo, the snapshot we pushed corresponded to the newer post-submission code, not the paper-version code. We did not also publish the gated snapshot, and we did not flag this difference in the README. That's the gap you've correctly identified.

So: the paper is internally consistent (paper text describes gated; paper numbers were run gated), and the current main is internally consistent (force-inject), but main is not the code that produced the paper's tables, and we owe readers a clearer statement of that.

To your three questions

1. Should slow-update guidance be validation-gated, as described in the paper?
   Yes for paper reproduction. The submission-version implementation was gated, and the reported numbers were obtained under that semantics. If you want to reproduce the arXiv tables exactly, gated is the reference behavior.

2. Or is the force-injection behavior in the released code the intended implementation?
   It is what main currently does and what we currently believe is the better design going forward, based on post-submission experiments. But it is not the implementation that produced the paper's numbers, and we should not have shipped it as "the SkillOpt implementation" without saying so.

3. If force-injection is intended, should exported best_skill.md be interpreted as "best validation-gated skill plus the latest slow-update field"?
   Yes — under the current main, best_skill.md is exactly that: the held-out-gate-winning skill from step-level edits, with its SLOW_UPDATE protected region overwritten by the latest epoch's slow-update content. It is not the exact best step candidate that the held-out gate selected.

How to reproduce, in the meantime

(a) For reproducing the paper: use the gated version. This is the submission-version implementation we ran for the paper.

(b) Honest caveat: we did not push the exact gated snapshot to this repo at release. That is an oversight on our side, not a deliberate choice. Until the config flag lands (see below), the easiest path is to locally patch skillopt/engine/trainer.py:1580–1604 to (i) call evaluate_gate on the slow-update candidate and (ii) only replace the SLOW_UPDATE field on current_skill / best_skill if the gate accepts. We will publish a clean version of this so nobody has to do it by hand.

(c) For running main as-is: the deployed best_skill.md should be interpreted as in (3) above — gate-selected skill with slow-update field force-overwritten.

What we're going to do

We're treating this as a real issue, not a documentation patch:

1. Make slow-update gating a config hyperparameter (tentatively slow_update_gate_with_selection: true|false), so reproducing the paper version and running the current main behavior are both available from the same checkout and differ by exactly one config line. Default value will be chosen after step 2.
2. Publish the post-submission force-inject experiments in the repo so the comparison is on the record — i.e. side-by-side numbers for gated vs. force-inject under matched settings on the same benchmarks. This makes clear which design corresponds to which set of numbers, and lets readers / reproducers decide for themselves.
3. Update the README with: the history above, the new config flag, the comparison table, and explicit guidance on which setting matches the arXiv version vs. the post-submission update.
4. After (1)–(3), the paper version of SkillOpt and the main version of SkillOpt will both be runnable from this repo, clearly labeled, with their respective numbers documented.

I'll link the follow-up PR and the README update to this issue when they land. Thanks again — this is exactly the kind of audit we want the repo to support, and you've helped us close a gap we should have closed at release.

[luigi-sqwish]

Perfect, thanks a lot! Looking forward to it