Skip to content

compute: extend temporal bucketing to Reduce's input arrangement#36644

Open
ggevay wants to merge 1 commit into
MaterializeInc:mainfrom
ggevay:temporal-bucketing-reduce
Open

compute: extend temporal bucketing to Reduce's input arrangement#36644
ggevay wants to merge 1 commit into
MaterializeInc:mainfrom
ggevay:temporal-bucketing-reduce

Conversation

@ggevay
Copy link
Copy Markdown
Contributor

@ggevay ggevay commented May 20, 2026

fec2af8 introduced temporal bucketing on PlanNode::ArrangeBy: the
lowering picks an ArrangementStrategy from a has_future_updates
analysis bit, and ensure_collections inserts a bucketing operator in
front of the arrangement when the strategy is TemporalBucketing.
TopK, Threshold, and Negate get bucketing transitively, because
their lowering wraps the input in a synthetic ArrangeBy.

Reduce is left out of both hooks: its LIR node has no strategy field,
and its render path (render_reduce via KeyValPlan) bypasses
ensure_collections. As a result, a Temporal Filter -> GroupAggregate
pattern — where no ArrangeBy sits between the temporal MFP and the
reduce — is never bucketed today.

Add a temporal_bucketing_strategy: ArrangementStrategy field to the
Reduce LIR node and set it during lowering from the input's
has_future_updates flag. At render time, render_reduce applies
apply_bucketing_strategy to the (key, val) stream right before
render_reduce_plan arranges it internally.

The lowering also clears LoweredExpr::has_future_updates on bucketing
absorption, so a stack of temporal MFPs only buckets at the lowest
layer; a trailing temporal MFP fused above naturally re-arms the flag.
The render-side bucketed: bool safety net in ensure_collections is
still needed to prevent double-bucketing within a single call (the raw-
collection site vs. the collections.arranged loop, or across multiple
keys in that loop).

@ggevay ggevay added the A-CLUSTER Topics related to the CLUSTER layer label May 20, 2026
@ggevay ggevay force-pushed the temporal-bucketing-reduce branch 7 times, most recently from 528ffbe to 557bc76 Compare May 20, 2026 20:49
@ggevay ggevay marked this pull request as ready for review May 20, 2026 20:57
@ggevay ggevay requested a review from a team as a code owner May 20, 2026 20:57
@ggevay ggevay requested a review from antiguru May 20, 2026 20:57
fec2af8 introduced temporal bucketing on `PlanNode::ArrangeBy`: the
lowering picks an `ArrangementStrategy` from a `has_future_updates`
analysis bit, and `ensure_collections` inserts a bucketing operator in
front of the arrangement when the strategy is `TemporalBucketing`.
`TopK`, `Threshold`, and `Negate` get bucketing transitively, because
their lowering wraps the input in a synthetic `ArrangeBy`.

`Reduce` is left out of both hooks: its LIR node has no strategy field,
and its render path (`render_reduce` via `KeyValPlan`) bypasses
`ensure_collections`. As a result, a `Temporal Filter -> GroupAggregate`
pattern — where no `ArrangeBy` sits between the temporal MFP and the
reduce — is never bucketed today.

Add a `temporal_bucketing_strategy: ArrangementStrategy` field to the
`Reduce` LIR node and set it during lowering from the input's
`has_future_updates` flag. At render time, `render_reduce` applies
`apply_bucketing_strategy` to the `(key, val)` stream right before
`render_reduce_plan` arranges it internally.

The lowering also clears `LoweredExpr::has_future_updates` on bucketing
absorption, so a stack of temporal MFPs only buckets at the lowest
layer; a trailing temporal MFP fused above naturally re-arms the flag.
The render-side `bucketed: bool` safety net in `ensure_collections` is
still needed to prevent double-bucketing within a single call (the raw-
collection site vs. the `collections.arranged` loop, or across multiple
keys in that loop).
@ggevay ggevay force-pushed the temporal-bucketing-reduce branch from 557bc76 to b5f9752 Compare May 20, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-CLUSTER Topics related to the CLUSTER layer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant