try-runtime 0.8.0 fails on on-runtime-upgrade while 0.7.0 works fine

[TomatoAres]

Description

I'm facing an issue when running the on-runtime-upgrade command with try-runtime. The command works as expected with try-runtime 0.7.0, but it fails with try-runtime 0.8.0.

Environment

Operating System: Ubuntu 24.04 MacOS

Commands and Outputs

try-runtime 0.8.0

try-runtime \
    --runtime bifrost_kusama_runtime.compact.compressed.wasm \
    on-runtime-upgrade --print-storage-diff \
    --blocktime 6000 --disable-idempotency-checks live \
    --uri wss://hk.bifrost-rpc.liebi.com:443/ws

[2025-03-18T08:54:02Z INFO  remote-ext] adding data for hashed prefix: , took 24.15s
[2025-03-18T08:54:02Z INFO  remote-ext] adding data for hashed key: 3a636f6465
[2025-03-18T08:54:02Z INFO  remote-ext] adding data for hashed key: 26aa394eea5630e07c48ae0c9558cef7f9cce9c888469bb1a0dceaa129672ef8
[2025-03-18T08:54:02Z INFO  remote-ext] adding data for hashed key: 26aa394eea5630e07c48ae0c9558cef702a5c1b19ab7a04f536c519aca4983ac
[2025-03-18T08:54:02Z INFO  remote-ext] 👩‍👦 scraping child-tree data from 23 top keys
[2025-03-18T08:54:03Z INFO  remote-ext] initialized state externalities with storage root 0xc6e670291ac9c70208bda135227190c399099402cb3007a84ceae6bbf280f7c7 and state_version V0
[2025-03-18T08:54:03Z INFO  try-runtime::cli] Original runtime [Name: RuntimeString::Owned("bifrost")] [Version: 17000] [Code hash: 0x6e49...83b7]
[2025-03-18T08:54:03Z INFO  try-runtime::cli] New runtime      [Name: RuntimeString::Owned("bifrost")] [Version: 18000] [Code hash: 0xb855...a43c]
[2025-03-18T08:54:10Z INFO  try_runtime_core::common::empty_block::production] Producing new empty block at height 8496154
[2025-03-18T08:54:10Z INFO  try_runtime_core::common::empty_block::production] Produced a new block (0x1c53bf5e685cad6f69fcdf361f2774baf9b4ee6e6c7c23483a1d0ed6ce502055)
[2025-03-18T08:54:11Z ERROR runtime] panicked at /root/.cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/de5b874/substrate/frame/executive/src/lib.rs:836:9:
    Storage root must match that calculated.
thread 'main' panicked at cli/main.rs:326:10:
called `Result::unwrap()` on an `Err` value: Input("failed to execute 'Core_execute_block': Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed\nWASM backtrace:\nerror while executing at wasm backtrace:\n    0: 0x7d610c - bifrost_kusama_runtime.wasm!rust_begin_unwind\n    1: 0x36ee - bifrost_kusama_runtime.wasm!core::panicking::panic_fmt::hd2903ad87c0415c7\n    2: 0xcb095 - bifrost_kusama_runtime.wasm!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h70694f215c6d4b0d\n    3: 0x6e1db9 - bifrost_kusama_runtime.wasm!Core_execute_block")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Build step 'Execute shell' marked build as failure
Finished: FAILURE

try-runtime 0.7.0

try-runtime-v0.7.0 \
    --runtime bifrost_kusama_runtime.compact.compressed.wasm \
    on-runtime-upgrade --print-storage-diff \
    --disable-idempotency-checks live \
    --uri wss://hk.bifrost-rpc.liebi.com:443/ws

[2025-03-19T04:10:52Z INFO  remote-ext] replacing ws:// in uri with http://: "http://172.19.64.31:29944" (ws is currently unstable for fetching remote storage, for more see https://github.com/paritytech/jsonrpsee/issues/1086)
[2025-03-19T04:10:52Z INFO  remote-ext] since no at is provided, setting it to latest finalized head, 0xe5c0ff19b3369a5b675237a643141e2b77cf13a61d31d3062f744085cb7fdd6f
[2025-03-19T04:10:52Z INFO  remote-ext] since no prefix is filtered, the data for all pallets will be downloaded
[2025-03-19T04:10:52Z INFO  remote-ext] scraping key-pairs from remote at block height 0xe5c0ff19b3369a5b675237a643141e2b77cf13a61d31d3062f744085cb7fdd6f
✅ Found 357363 keys (2.94s)
[00:00:18] ✅ Downloaded key values 19,392.8515/s [=======================================================================================================================================================================================================] 357363/357363 (0s)
✅ Inserted keys into DB (2.76s)
[2025-03-19T04:11:17Z INFO  remote-ext] adding data for hashed prefix: , took 24.73s
[2025-03-19T04:11:17Z INFO  remote-ext] adding data for hashed key: 3a636f6465
[2025-03-19T04:11:17Z INFO  remote-ext] adding data for hashed key: 26aa394eea5630e07c48ae0c9558cef7f9cce9c888469bb1a0dceaa129672ef8
[2025-03-19T04:11:17Z INFO  remote-ext] adding data for hashed key: 26aa394eea5630e07c48ae0c9558cef702a5c1b19ab7a04f536c519aca4983ac
[2025-03-19T04:11:17Z INFO  remote-ext] 👩‍👦 scraping child-tree data from 23 top keys
[2025-03-19T04:11:18Z INFO  remote-ext] initialized state externalities with storage root 0x6a2a38ad6dcaec84e9d31ea8a22828e8f0e073e0dfba5dda91cdf18f5d5ffdfb and state_version V0
[2025-03-19T04:11:18Z INFO  try-runtime::cli] Original runtime [Name: RuntimeString::Owned("bifrost")] [Version: 17000] [Code hash: 0x6e49...83b7]
[2025-03-19T04:11:19Z INFO  try-runtime::cli] New runtime      [Name: RuntimeString::Owned("bifrost")] [Version: 18000] [Code hash: 0xd646...472c]
[2025-03-19T04:11:19Z INFO  try-runtime::cli] 🚀 Speed up your workflow by using snapshots instead of live state. See `try-runtime create-snapshot --help`.
[2025-03-19T04:11:19Z INFO  try_runtime_core::common::misc_logging] ------------------------------------------------------------------


[2025-03-19T04:11:19Z INFO  try_runtime_core::common::misc_logging] 🔬 Running TryRuntime_on_runtime_upgrade with checks: PreAndPost


[2025-03-19T04:11:19Z INFO  try_runtime_core::common::misc_logging] ------------------------------------------------------------------


[2025-03-19T04:11:32Z INFO  try_runtime_core::common::misc_logging] ------------------------------------------------------------------------------------------------------


[2025-03-19T04:11:32Z INFO  try_runtime_core::common::misc_logging] 🔬 TryRuntime_on_runtime_upgrade succeeded! Running it again without checks for weight measurements.


[2025-03-19T04:11:32Z INFO  try_runtime_core::common::misc_logging] ------------------------------------------------------------------------------------------------------


[2025-03-19T04:11:32Z INFO  try_runtime_core::commands::on_runtime_upgrade] ℹ Skipping idempotency check
[2025-03-19T04:11:32Z INFO  try-runtime::cli] PoV size (zstd-compressed compact proof): 31.4 KB. For parachains, it's your responsibility to verify that a PoV of this size fits within any relaychain constraints.
[2025-03-19T04:11:32Z INFO  try-runtime::cli] Consumed ref_time: 0.0020081s (0.10% of max 2s)
[2025-03-19T04:11:32Z INFO  try-runtime::cli] ✅ No weight safety issues detected. Please note this does not guarantee a successful runtime upgrade. Always test your runtime upgrade with recent state, and ensure that the weight usage of your migrations will not drastically differ between testing and actual on-chain execution.

Expected Behavior

The on-runtime-upgrade command should work without errors in both try-runtime 0.7.0 and 0.8.0.

Steps to Reproduce

Install try-runtime 0.8.0 and 0.7.0, from github release.
Run the on-runtime-upgrade command with try-runtime 0.8.0 as shown above.
Observe the error.
Run the same command with try-runtime 0.7.0.
Observe that it works without errors.

[ggwpez]

Yes 0.8.0 has some regression as it was quickly released to support multi block migrations... cc @nkpar
Does --disable-mbm-checks change anything? Maybe upload the runtime here and give us a block number so that we can reproduce this later.