feature: TableStrategy #5640

a10y · 2025-12-06T20:32:54Z

Extracting this out of my hack week project. This adds a new TableStrategy that is like the StructStrategy, except that it allows users to override the write strategy at particular field paths in the schema.

For example, if you have a schema like this:

request: struct?
|__ id: utf8
|__ duration_ms: i64
|__ body: struct?
	|__ bytes: binary?
    |__ json: utf8?

Before you'd be stuck with choosing a single strategy for all fields of the Struct.

TableStrategy allows you to override particular field paths anywhere in the field tree. For example, if you wanted to allow uncompressed struct validity, default BtrBlocks compression but ZSTD compression just for the request.body.bytes field, you'd do:

let btrblocks = Arc::new(CompressingStrategy::new_btrblocks(FlatLayoutStrategy::default()));
let zstd = Arc::new(CompressingStrategy::new_compact(FlatLayoutStrategy::default()));

let table_strategy = TableStrategy::new(
            FlatLayoutStrategy::default(), 
            btrblocks,
        )
        .with_leaf_strategy(field_path!(request.body.bytes), zstd);

I've replaced StructStrategy with this in the default WriteBuilder.

If we like this, we might want to consider marking StructStrategy as deprecated.

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

codecov · 2025-12-06T20:45:23Z

Codecov Report

❌ Patch coverage is 89.16667% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.39%. Comparing base (374882d) to head (11a8904).
⚠️ Report is 22 commits behind head on develop.

Files with missing lines	Patch %	Lines
vortex-layout/src/layouts/table.rs	87.23%	18 Missing ⚠️
vortex-file/src/strategy.rs	38.46%	8 Missing ⚠️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

- add integration test for vortex-file - fix bug in PathStrategy descend - return Writer back from the write pathway Signed-off-by: Andrew Duffy <andrew@a10y.dev>

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

gatesn

Yeah really nice idea. Definitely should just immediately deprecate the StructStrategy.

gatesn · 2025-12-10T10:19:43Z

vortex-layout/src/layouts/path.rs

+        field_path: impl Into<FieldPath>,
+        writer: Arc<dyn LayoutStrategy>,
+    ) -> Self {
+        self.leaf_writers.insert(field_path.into(), writer);


I think we should check that we don't have any overlapping prefixes?

should we panic on this or just ignore/maybe log warning?

my vote would be panic

i went with panic

gatesn · 2025-12-10T10:20:59Z

vortex-layout/src/layouts/table.rs

+    /// A set of leaf field overrides, e.g. to force one column to be compact-compressed.
+    leaf_writers: HashMap<FieldPath, Arc<dyn LayoutStrategy>>,
+    /// The writer for any validity arrays that may be present
+    validity: Arc<dyn LayoutStrategy>,


Is this only for non-leaf struct validity? Or all validity? Do we just need a special type of FieldPath that allows us to index into the nulls etc?

e.g. enum PathComponent { Field(FieldName), ListElements, Validity } maybe? Not sure... might be gross.

structs are the only case right now where validity is extracted out and compressed into its own child layout, all of the other strategies just store it alongside the array data

gatesn · 2025-12-10T10:21:21Z

vortex-layout/src/layouts/table.rs

+    ///
+    /// let strategy = PathStrategy::new(flat.clone(), flat.clone());
+    /// ```
+    pub fn new(validity: Arc<dyn LayoutStrategy>, fallback: Arc<dyn LayoutStrategy>) -> Self {


I think validity should just default to flat? Or maybe default everything and use builder pattern to override?

gatesn · 2025-12-10T10:22:46Z

vortex-layout/src/layouts/table.rs

+    ) -> VortexResult<LayoutRef> {
+        let dtype = stream.dtype().clone();
+
+        // Fallback: if the array is not a struct, fallback to writing a single array.


Yeah, I think you actually want to check the leaf_writers here for a root FieldPath, that would allow overriding intermediate struct strategies.

This is actually to maintain compatibility with behavior in StructStrategy, which lets you serialize non-struct flat arrays. Not sure why we let you do that but it's the current behavior and we have unit tests for it 🤷

vortex-file/src/strategy.rs

vortex-file/tests/test_write_table.rs

vortex-layout/src/layouts/table.rs

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

a10y added 4 commits December 6, 2025 14:59

port over code from geovortex branch

b439ec4

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

comments

332f981

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

add field_path!() macro

bb0fa8c

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

fix

c94aa82

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

a10y added 4 commits December 9, 2025 16:12

fix doctest

b7440bb

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

implement more

fd2543b

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

a few things

92a4e32

- add integration test for vortex-file - fix bug in PathStrategy descend - return Writer back from the write pathway Signed-off-by: Andrew Duffy <andrew@a10y.dev>

undo

2117c5e

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

a10y requested review from AdamGS and gatesn December 9, 2025 22:15

a10y added 2 commits December 9, 2025 17:19

fixup duckdb/python/java

d4acdd1

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

ok nvm then

157ee48

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

a10y force-pushed the aduffy/path-strategy branch from 518a2e6 to 157ee48 Compare December 9, 2025 22:34

a10y added the feature Release label indicating a new feature or request label Dec 9, 2025

fix test

5c676c9

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

gatesn requested changes Dec 10, 2025

View reviewed changes

AdamGS reviewed Dec 10, 2025

View reviewed changes

a10y added 3 commits December 10, 2025 08:52

inline hashmap

6fa4ff4

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

mark StructStrategy as deprecated

5640bae

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

validate paths before we insert

15b8c9e

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

a10y force-pushed the aduffy/path-strategy branch from c6dc373 to 15b8c9e Compare December 10, 2025 15:52

a10y added 2 commits December 10, 2025 11:02

rename PathStrategy -> TableStrategy

50f2a1c

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

add second field to test

b20d9e8

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

a10y changed the title ~~feature: PathStrategy~~ feature: TableStrategy Dec 10, 2025

a10y requested a review from gatesn December 10, 2025 16:04

a10y added 2 commits December 10, 2025 11:08

allow overriding default and validity strategy

25f0918

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

doc link

11a8904

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

feature: TableStrategy #5640

Are you sure you want to change the base?

feature: TableStrategy #5640

Uh oh!

Conversation

a10y commented Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gatesn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

a10y commented Dec 6, 2025 •

edited

Loading

codecov bot commented Dec 6, 2025 •

edited

Loading