Describe the bug
In the awslabs benchmark, some scans in q25 fell back to Spark because the scans use DPP. This is expected behavior. However, the fallback somehow changed the Spark plan, removing DPP, causing the scan to read all partitions, resulting in dramatic regression.
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
Describe the bug
In the awslabs benchmark, some scans in q25 fell back to Spark because the scans use DPP. This is expected behavior. However, the fallback somehow changed the Spark plan, removing DPP, causing the scan to read all partitions, resulting in dramatic regression.
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response