You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The bisection of last week and performance pointed to reconstructing the LargeSmtForest from the SQL as the by far leading cause, specifically fn add_lineage which effectively linearizes account additions to the forest (bad).
Run the benchmark RUST_LOG=info cargo bench --bench state_load -- --nocapture to see span based logs:
INFO state_load: message=Reusing cached data directory, num_accounts=100000
Benchmarking state_load/100000_accounts: Warming up for 3.0000 sINFO load: load: miden-store: message=Connected to the database, sqlite=/media/humpback/projects/miden/miden-node-bernhard-improve-node-startup/crates/store/../../target/bench-data/100000/miden-store.sqlite3
INFO miden_node_store::db::migrations: message=Applying migrations, target="miden-store", migrations=0
INFO verify_schema: miden-store: message=Database schema verification passed, objects=37
INFO verify_schema: miden-store: message="close", time.busy=1.31ms, time.idle=1.79µs
INFO load: load: miden-store: message="close", time.busy=196µs, time.idle=2.49ms
INFO load: load_mmr: miden-store: message="close", time.busy=17.6µs, time.idle=46.0µs
INFO load: load_account_tree: miden-store: message="close", time.busy=868ms, time.idle=2.30µs
INFO load: load_nullifier_tree: miden-store: message=RocksDB nullifier tree storage is empty, populating from SQLite
INFO load: load_nullifier_tree: miden-store: message="close", time.busy=406ms, time.idle=1.89ms
INFO load: verify_tree_consistency: miden-store: message="close", time.busy=11.4µs, time.idle=202µs
paging [=============================> ] 59000/100000 (73s remaining)
fetching [============================> ] 56000/100000 (85s remaining)
inserting [===========================> ] 54000/100000 (84s remaining)
A bunch of optimizations are already done:
Batch fetching from DB and processing - keep icaches warm while utilizing many more cores
Limited parallelism
Staged work
Backpressure
What is still required to land this:
0xMiden/crypto#910 which allows to batch apply lineages and avoids a lot of intermediate hash re-calc on addition.
If things are still looking somewhat bleak, having a begin() -> Operator; Operator::add_lineage; Operator::drop/end()-style approach, avoiding all intermediate node operations would be my next steps to take
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ref #1697
Adds a bench with progress bars and prints closing spans, which shows "the numbers" for
State::loadpieces.