[phase-31 4/4] PostgreSQL metastore — migration + compaction columns by g-talbot · Pull Request #6245 · quickwit-oss/quickwit

g-talbot · 2026-03-30T17:22:06Z

Summary

Add compaction metadata to the PostgreSQL metastore with migration 27 and updated SQL queries (Phase 31 Metadata Foundation, PR 4 of 4).

Stacks on gtt/phase-31-writer-wiring (PR #6244).

What's included

Migration 27 (27_add-compaction-metadata.{up,down}.sql):

ALTER TABLE metrics_splits: adds window_start, window_duration_secs, sort_fields, num_merge_ops, row_keys, zonemap_regexes
Partial index idx_metrics_splits_compaction_scope on (index_uid, sort_fields, window_start) WHERE split_state = 'Published'

stage_metrics_splits:

INSERT extended from 15 to 21 bind parameters ($16-$21 for compaction columns)
ON CONFLICT SET clause updates all 6 compaction columns
zonemap_regexes cast from text to JSONB in SELECT

list_metrics_splits:

PgMetricsSplit construction includes compaction fields (defaults until migration runs; authoritative data from JSON blob via to_metadata())

Bug fixes (pre-existing on upstream-10b-parquet-actors):

Added missing StageMetricsSplitsRequestExt import
Fixed index_id vs index_uid type mismatches in publish, mark, and delete functions
Fixed IndexUid binding to sqlx (use .to_string())
Fixed ListMetricsSplitsResponseExt::try_from_splits trait disambiguation

Verification

cargo build -p quickwit-metastore --features quickwit-metastore/postgres ✅
Clippy: dependent on upstream-10b-parquet-actors test compilation (tests not gated)

Test plan

Compilation with postgres feature succeeds
Integration tests require Docker (Postgres) — run with make test-metrics-e2e
Migration SQL is syntactically valid and reversible

🤖 Generated with Claude Code

fulmicoton-dd · 2026-03-31T09:27:59Z

quickwit/quickwit-metastore/migrations/postgresql/27_add-compaction-metadata.up.sql

+
+-- Compaction scope index: supports the compaction planner's primary query pattern
+-- "give me all Published splits for a given (index_uid, sort_fields, window_start) triple."
+CREATE INDEX IF NOT EXISTS idx_metrics_splits_compaction_scope


On the log side, we have a column (maturity_timestamp) that is used (I think?) to filter further the scope of splits candidate for merge: they need to be published and immature.

I would prefer to keep the same logic, but it should work without it.

Generally speaking AI hurts the consistency of this project. I guess that's the price to pay for the increased productivity.

fulmicoton-dd · 2026-03-31T09:29:08Z

quickwit/quickwit-metastore/src/metastore/file_backed/file_backed_index/mod.rs

    }

+    // Filter by compaction scope
+    if let Some(ws) = query.window_start


can we avoid two letter variable name. This is just hurting readability here.

ws -> window_start

fulmicoton-dd · 2026-03-31T09:32:08Z

quickwit/quickwit-metastore/src/metastore/file_backed/file_backed_index/mod.rs

+    if let Some(ws) = query.window_start
+        && split.metadata.window_start != Some(ws)
+    {
+        return false;


So this checks for window_start being equal. What is ListMetricsSplitsQuery?
If it is for the read path, it should check if the window intersect, shoudn't it?

fulmicoton-dd · 2026-03-31T09:33:51Z

quickwit/quickwit-metastore/src/metastore/postgres/metastore.rs

        let mut size_bytes_list = Vec::with_capacity(splits_metadata.len());
        let mut split_metadata_jsons = Vec::with_capacity(splits_metadata.len());
+        let mut window_starts: Vec<Option<i64>> = Vec::with_capacity(splits_metadata.len());
+        let mut window_duration_secs_list: Vec<Option<i32>> =


I expected u32 here. You made it unsigned in the file backed metastore. Does postgres not allow this?

fulmicoton-dd · 2026-03-31T09:39:33Z

quickwit/quickwit-metastore/src/metastore/mod.rs

    pub tag_region: Option<String>,
    /// Host tag filter.
    pub tag_host: Option<String>,
+    /// Window start filter for compaction scope queries.


ah ok, so if I understand correctly, this query object exists both for compaction and for the read path.

For the read path we use range_start/range_end, for compaction we use window_start.
Do we ever change the window duration in reality?

Could we drop window_start and query for [time_range_start..time_range_start + 15mn) ?

…publish Add compaction metadata to the PostgreSQL metastore: Migration 27: - 6 new columns: window_start, window_duration_secs, sort_fields, num_merge_ops, row_keys, zonemap_regexes - Partial index idx_metrics_splits_compaction_scope on (index_uid, sort_fields, window_start) WHERE split_state = 'Published' stage_metrics_splits: - INSERT extended from 15 to 21 bind parameters for compaction columns - ON CONFLICT SET updates all compaction columns list_metrics_splits: - PgMetricsSplit construction includes compaction fields (defaults from JSON) Also fixes pre-existing compilation errors on upstream-10b-parquet-actors: - Missing StageMetricsSplitsRequestExt import - index_id vs index_uid type mismatches in publish/mark/delete - IndexUid binding (to_string() for sqlx) - ListMetricsSplitsResponseExt trait disambiguation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ublish validation Close critical gaps identified during port review: split_writer.rs: - Store table_config on ParquetSplitWriter (not just pass-through) - Compute window_start from batch time range using table_config.window_duration_secs - Populate sort_fields, window_duration_secs, parquet_files on metadata before write - Call write_to_file_with_metadata(Some(&metadata)) to embed KV metadata in Parquet - Update size_bytes after write completes metastore/mod.rs: - Add window_start and sort_fields fields to ListMetricsSplitsQuery - Add with_compaction_scope() builder method metastore/postgres/metastore.rs: - Add compaction scope filters (AND window_start = $N, AND sort_fields = $N) to list query - Add replaced_split_ids count verification in publish_metrics_splits - Bind compaction scope query parameters ingest/config.rs: - Add table_config: TableConfig field to ParquetIngestConfig Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ad code removal - file_backed_index/mod.rs: Add window_start and sort_fields filtering to metrics_split_matches_query() for compaction scope queries - writer.rs: Add test_meta07_self_describing_parquet_roundtrip test (writes compaction metadata to Parquet, reads back from cold file, verifies all fields roundtrip correctly) - fields.rs: Remove dead sort_order() method (replaced by TableConfig) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…regexes Gap 1: Change window_duration_secs from i32 to Option<i32> in both PgMetricsSplit and InsertableMetricsSplit. Pre-Phase-31 splits now correctly map 0 → NULL in PostgreSQL, enabling Phase 32 compaction queries to use `WHERE window_duration_secs IS NOT NULL` instead of the fragile `WHERE window_duration_secs > 0`. Gap 2: Change zonemap_regexes from String to serde_json::Value in both structs. This maps directly to JSONB in sqlx, avoiding ambiguity when PostgreSQL JSONB operators are used in Phase 34/35 zonemap pruning. Gap 3: Add two missing tests: - test_insertable_from_metadata_with_compaction_fields: verifies all 6 compaction fields round-trip through InsertableMetricsSplit - test_insertable_from_metadata_pre_phase31_defaults: verifies pre-Phase-31 metadata produces window_duration_secs: None, zonemap_regexes: json!({}) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

g-talbot requested review from fulmicoton-dd and mattmkim March 30, 2026 17:23

g-talbot mentioned this pull request Mar 30, 2026

feat(31): shared invariants module in quickwit-dst #6246

Open

8 tasks

fulmicoton-dd reviewed Mar 31, 2026

View reviewed changes

g-talbot force-pushed the gtt/phase-31-pg-metastore branch from c8836d6 to eccf657 Compare March 31, 2026 20:41

g-talbot force-pushed the gtt/phase-31-writer-wiring branch 2 times, most recently from 3bbfb71 to 95c3596 Compare March 31, 2026 20:55

g-talbot force-pushed the gtt/phase-31-pg-metastore branch 2 times, most recently from 6eeaecd to f2113e5 Compare March 31, 2026 21:03

g-talbot force-pushed the gtt/phase-31-writer-wiring branch 2 times, most recently from 179ccd2 to ed6d687 Compare March 31, 2026 21:08

g-talbot force-pushed the gtt/phase-31-pg-metastore branch from f2113e5 to 0d561b4 Compare March 31, 2026 21:08

g-talbot force-pushed the gtt/phase-31-writer-wiring branch from ed6d687 to a4d0d36 Compare March 31, 2026 21:26

g-talbot force-pushed the gtt/phase-31-pg-metastore branch 2 times, most recently from 8ba9201 to 0761d11 Compare March 31, 2026 21:29

g-talbot force-pushed the gtt/phase-31-writer-wiring branch from a4d0d36 to f05d4e7 Compare March 31, 2026 21:31

g-talbot force-pushed the gtt/phase-31-pg-metastore branch from 0761d11 to b551b13 Compare March 31, 2026 21:31

g-talbot force-pushed the gtt/phase-31-writer-wiring branch from f05d4e7 to 4a0507e Compare March 31, 2026 21:33

g-talbot force-pushed the gtt/phase-31-pg-metastore branch from b551b13 to 398cdf2 Compare March 31, 2026 21:33

g-talbot force-pushed the gtt/phase-31-writer-wiring branch from 4a0507e to bc9458d Compare March 31, 2026 21:40

g-talbot force-pushed the gtt/phase-31-pg-metastore branch 2 times, most recently from 3289f58 to dbbbd8b Compare March 31, 2026 21:50

g-talbot force-pushed the gtt/phase-31-writer-wiring branch 2 times, most recently from 2599f67 to 46903b3 Compare April 1, 2026 10:48

g-talbot force-pushed the gtt/phase-31-pg-metastore branch 2 times, most recently from 54a1f95 to 9696d99 Compare April 1, 2026 11:02

g-talbot force-pushed the gtt/phase-31-writer-wiring branch 2 times, most recently from 74bfd04 to de0f8c6 Compare April 1, 2026 11:10

g-talbot force-pushed the gtt/phase-31-pg-metastore branch from 9696d99 to bf23ab3 Compare April 1, 2026 11:10

g-talbot and others added 5 commits April 1, 2026 07:30

style: rustfmt

a60b361

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

g-talbot force-pushed the gtt/phase-31-writer-wiring branch from de0f8c6 to 2d9e6eb Compare April 1, 2026 11:30

g-talbot force-pushed the gtt/phase-31-pg-metastore branch from bf23ab3 to a60b361 Compare April 1, 2026 11:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[phase-31 4/4] PostgreSQL metastore — migration + compaction columns#6245

[phase-31 4/4] PostgreSQL metastore — migration + compaction columns#6245
g-talbot wants to merge 5 commits intogtt/phase-31-writer-wiringfrom
gtt/phase-31-pg-metastore

g-talbot commented Mar 30, 2026

Uh oh!

fulmicoton-dd Mar 31, 2026

Uh oh!

fulmicoton-dd Mar 31, 2026

Uh oh!

fulmicoton-dd Mar 31, 2026

Uh oh!

fulmicoton-dd Mar 31, 2026

Uh oh!

fulmicoton-dd Mar 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

g-talbot commented Mar 30, 2026

Summary

What's included

Verification

Test plan

Uh oh!

fulmicoton-dd Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

fulmicoton-dd Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

fulmicoton-dd Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

fulmicoton-dd Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

fulmicoton-dd Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants