Skip to content

Add partition filter pushdown for more efficient rolling windows#1854

Draft
shangyian wants to merge 3 commits intoDataJunction:mainfrom
shangyian:partition-filter-pushdown
Draft

Add partition filter pushdown for more efficient rolling windows#1854
shangyian wants to merge 3 commits intoDataJunction:mainfrom
shangyian:partition-filter-pushdown

Conversation

@shangyian
Copy link
Collaborator

@shangyian shangyian commented Mar 11, 2026

Summary

This PR improves the efficiency of generated SQL by pushing down temporal filters from the outer query into the upstream driving CTE, rather than applying them after the full query has been evaluated.

Previously, when include_temporal_filters=True was requested, DJ would inject the DJ_LOGICAL_TIMESTAMP() filter as a WHERE clause on the outermost grain-group query. For simple fact table queries this is fine, but for transforms that use a date spine to drive a range join (e.g. rolling window metrics), the filter arrived too late, and the entire join had already been evaluated across all dates before a single row was selected.

With this change, DJ detects when the parent transform's primary FROM table is itself an upstream transform that exposes the temporal FK column. In that case, the filter is injected directly into that upstream CTE's WHERE clause instead, so the join only runs for the target partition(s). If no eligible upstream transform is found (e.g. the parent reads directly from a source table), the behavior falls back to the original outer WHERE approach unchanged.

This is particularly important for rolling window patterns, where a date spine drives a range join against a large fact table. Filtering the date spine to the target date before the join means the fact table is only scanned for the rows that are actually needed, rather than for every date in history.

Test Plan

  • PR has an associated issue: #
  • make check passes
  • make test shows 100% unit test coverage

Deployment Plan

@netlify
Copy link

netlify bot commented Mar 11, 2026

Deploy Preview for thriving-cassata-78ae72 canceled.

Name Link
🔨 Latest commit 5ca8c4d
🔍 Latest deploy log https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/69b1ebce21461e000816ac52

@shangyian shangyian changed the title Add partition filter pushdown to support rolling window pattern Add partition filter pushdown for more efficient rolling window pattern Mar 11, 2026
@shangyian shangyian changed the title Add partition filter pushdown for more efficient rolling window pattern Add partition filter pushdown for more efficient rolling windows Mar 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant