Skip to content

Parallelize InMemoryIndex construction with bucket put loop.#5288

Open
dmkozh wants to merge 1 commit into
stellar:masterfrom
dmkozh:par_index
Open

Parallelize InMemoryIndex construction with bucket put loop.#5288
dmkozh wants to merge 1 commit into
stellar:masterfrom
dmkozh:par_index

Conversation

@dmkozh
Copy link
Copy Markdown
Contributor

@dmkozh dmkozh commented May 22, 2026

Description

Build the LiveBucketIndex on an async worker thread while the put loop in mergeInMemory runs on the main thread. Both only read mergedEntries as const, so they need no synchronization.

Checklist

  • Reviewed the contributing document
  • Rebased on top of master (no merge commits)
  • Ran clang-format v8.0.0 (via make format or the Visual Studio extension)
  • Compiles
  • Ran all tests
  • If change impacts performance, include supporting evidence per the performance document

Build the LiveBucketIndex on an async worker thread while the put loop in mergeInMemory runs on the main thread. Both only read mergedEntries as const, so they need no synchronization.
Copilot AI review requested due to automatic review settings May 22, 2026 00:45
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR parallelizes level-0 LiveBucket::mergeInMemory work by constructing the LiveBucketIndex on a worker thread while the main thread writes merged entries to the bucket file, then passes the pre-built index into BucketOutputIterator::getBucket() to avoid rebuilding it.

Changes:

  • Spawn LiveBucketIndex construction via std::async during LiveBucket::mergeInMemory while the merge output is written to disk.
  • Extend BucketOutputIterator::getBucket() to accept an optional pre-built index (preBuiltIndex) and prefer it when provided.
  • Wire the pre-built index from mergeInMemory into getBucket().

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
src/bucket/LiveBucket.cpp Starts async index construction during in-memory merge and forwards the built index into bucket adoption.
src/bucket/BucketOutputIterator.h Adds an optional preBuiltIndex parameter to getBucket() with a default nullptr.
src/bucket/BucketOutputIterator.cpp Uses preBuiltIndex when provided to avoid recomputing an index from in-memory state or disk.

Comment thread src/bucket/LiveBucket.cpp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants