DataFusion-Federation: Union Flattening Across Executors

[vikrantmehta123]

I was experimenting with union pushdown behavior in DataFusion Federation and noticed something I wanted to clarify.

Setup:

• Two independent SQL executors (separate DB containers)
• Executor 1 exposes tables t1, t2
• Executor 2 exposes table t3

The initial logical plan looks like:

Projection
  Union
    Union
      TableScan(t1)
      TableScan(t2)
    TableScan(t3)

With optimize_unions enabled, this is flattened into:

Projection
  Union
    TableScan(t1)
    TableScan(t2)
    TableScan(t3)

This results in three independent SELECTs being executed, with the UNION performed in-memory.

If I disable optimize_unions optimizer rule, Executor 1 instead receives a single pushed-down SQL query:

SELECT ... FROM t1
UNION ALL
SELECT ... FROM t2

while Executor 2 executes its own SELECT, followed by one final in-memory UNION.

---

Is flattening unions across executor boundaries here an intentional optimizer choice? Would pushing down UNIONs be better?