Skip to content

impr(store_volatile): index group children in a separate ETS table#879

Closed
shyba wants to merge 2 commits into
edgefrom
impr/volatile_idx
Closed

impr(store_volatile): index group children in a separate ETS table#879
shyba wants to merge 2 commits into
edgefrom
impr/volatile_idx

Conversation

@shyba
Copy link
Copy Markdown

@shyba shyba commented Apr 21, 2026

What

Replaces the volatile store's in-band child set with a dedicated
children index table, adding an overwrite fast path to write/3 and
make_link/3.

Why

hb_store_volatile tracked group membership by storing a sets:set()
of children inside the main table's {group, Set} row. Every write
walked the parent chain and called sets:add_element/2, which copies
the ordset.

How

  • Second ETS table per store instance, bag-typed, keyed by parent
    path, each row {Parent, ChildName}. Bag semantics collapse
    duplicate rows automatically.
  • Group markers in the main table simplify to {group, true}. All
    membership lives in the index table.
  • list/2 on a group path becomes a single ets:lookup/2 on the
    index — O(family-size), no scan.
  • write/3 and make_link/3 branch once on the existing entry:
    • absent → run parent-index maintenance as before
    • existing {group, true} → delete the index rows for that path
      before overwriting, so the new raw/link value doesn't leave
      stale children visible on later recreate
    • existing raw/link → skip parent-index maintenance (already
      indexed when the key was first created)
  • reset/1 clears both tables.

No change to the store's external semantics. list/2, type/2,
make_group/2, read/2, resolve/2 behavior is preserved.

Measured impact

Direct writes (1 KiB value, single parent):

N before after
5 000 26k ops/s 1.12M ops/s
20 000 6.9k ops/s 995k ops/s
50 000 2.2k ops/s 1.13M ops/s

End-to-end hb_cache:write on volatile, 200 1 KiB committed
messages, 10-run median:

  • before: ~590 msgs/s
  • after: ~1714 msgs/s

For comparison, hb_store_lmdb on the same workload: ~1800 msgs/s.

Tests

New coverage:

  • list_test, list_dedup_test, list_with_link_test — group
    listing and dedup invariants.
  • overwrite_link_to_raw_test — raw value replacing a link.
  • overwrite_group_to_raw_test — raw value replacing a group
    marker; asserts index rows for the path are gone before recreate.
  • overwrite_group_to_link_test — link replacing a group marker;
    same cleanup invariant, verified through list/2 via link resolution.

@shyba shyba marked this pull request as ready for review April 22, 2026 13:27
@shyba shyba marked this pull request as draft April 22, 2026 18:10
@shyba
Copy link
Copy Markdown
Author

shyba commented Apr 22, 2026

Back to draft for 2 reasons:

  • there is a better way to do it also using two tables, like this one
  • another PR will show another way of redesigning it entirely, so it goes back to 1 table

@shyba
Copy link
Copy Markdown
Author

shyba commented Apr 29, 2026

Replaced by #887

@shyba shyba closed this Apr 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant