Skip to content

fix: [branch-0.16] prevent native sort crash for Struct(Map(...)) keys#4265

Draft
andygrove wants to merge 1 commit intoapache:branch-0.16from
andygrove:backport-4157-to-0.16
Draft

fix: [branch-0.16] prevent native sort crash for Struct(Map(...)) keys#4265
andygrove wants to merge 1 commit intoapache:branch-0.16from
andygrove:backport-4157-to-0.16

Conversation

@andygrove
Copy link
Copy Markdown
Member

Which issue does this PR close?

Backport of #4157 (closes #4123) to branch-0.16.

Rationale for this change

Spark 4.1.1 SQL coverage exposed a case where Comet allows native aggregation for grouping keys that contain nested MapType (for example StructType(MapType(...))). Although planning succeeds, downstream native sort/row-format handling does not support this shape, so the query fails at execution time instead of falling back to Spark during planning.

This brings the fix to the 0.16 release line so the upcoming patch release does not crash on these grouping keys.

What changes are included in this PR?

Clean cherry-pick of commit `7cf04c8e39d557ec895a6489b5eefd6076d648ac` from `main`:

  • add a recursive `containsMapType` helper in `CometBaseAggregate`
  • make aggregate planning fall back when any grouping expression contains `MapType` directly or nested in `StructType`/`ArrayType`
  • add regression tests in `CometAggregateSuite` for grouping on `Struct(Map(...))` and `Array(Map(...))`

How are these changes tested?

Inherited from the original PR:

@andygrove andygrove changed the title fix: prevent native sort crash for Struct(Map(...)) keys (backport #4157 to branch-0.16) fix: branch-0.16] prevent native sort crash for Struct(Map(...)) keys (backport #4157 to branch-0.16) May 8, 2026
@andygrove andygrove changed the title fix: branch-0.16] prevent native sort crash for Struct(Map(...)) keys (backport #4157 to branch-0.16) fix: branch-0.16] prevent native sort crash for Struct(Map(...)) keys May 8, 2026
@andygrove andygrove changed the title fix: branch-0.16] prevent native sort crash for Struct(Map(...)) keys fix: [branch-0.16] prevent native sort crash for Struct(Map(...)) keys May 8, 2026
@andygrove andygrove marked this pull request as draft May 8, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants