Update `explain` to Include Singular, CROSS and User Collections by hadia206 · Pull Request #501 · bodo-ai/PyDough

hadia206 · 2026-03-09T21:46:40Z

Extends pydough.explain() so it can explain

Singular
CROSS and
user-generated collections (range_collection, dataframe_collection)

Extends pydough.expalin_term()

Singular
CROSS
UDF

john-sanchez31

LGTM!

knassre-bodo

A few things should be added in my opinion, but so far looks great @hadia206 :)

knassre-bodo · 2026-03-13T04:30:02Z

        if root is not None:
            qualified_node = qualify_node(node, session)
        else:
-            # If the root is None, it means that the node was an expression
-            # without information about its context.
-            lines.append(
-                f"Cannot call pydough.explain on {display_raw(node)}.\n"
-                "Did you mean to use pydough.explain_term?"
-            )
+            # No root in the tree (e.g. UnqualifiedGeneratedCollection, or a
+            # bare expression like LOWER(first_name + last_name)). Try to
+            # qualify anyway for generated collections. If it still fails,
+            # raise an exception.
+            try:
+                qualified_node = qualify_node(node, session)
+            except Exception:
+                lines.append(
+                    f"Cannot call pydough.explain on {display_raw(node)}.\n"
+                    "Did you mean to use pydough.explain_term?"
+                )
+                return "\n".join(lines)


What if we jus got rid of if root is not None and always did the try-except?

knassre-bodo · 2026-03-13T17:24:46Z

+                if isinstance(prop, CartesianProductMetadata):
+                    child_name = prop.child_collection.name
+                    left_desc = (
+                        qualified_node.preceding_context.to_string()
+                        if qualified_node.preceding_context is not None
+                        else collection_name
+                    )
+                    lines.append(
+                        "This node is a CROSS join: every row of the left "
+                        "collection is paired with every row of the right "
+                        "collection."
+                    )
+                    lines.append(f"Left (parent): {left_desc}")
+                    lines.append(f"Right (child): {child_name}")
+                    lines.append(
+                        f"Metadata: {collection_name}.{property_name} -> {child_name}. "
+                        f"Call pydough.explain(graph['{collection_name}']['{property_name}']) "
+                        "to learn more."
+                    )


I don't necessarily like this way of handling CartesianProductMetadata; what was wrong with using the same message as other sub-collection types, then relying on pydough.explain on the metadata to explain the distinction between it being a simple join vs general join vs cross join?

ok. I'll revert that. I started with wrong CROSS test and went rabbit hole on that.

knassre-bodo · 2026-03-13T17:27:47Z

-                "Did you mean to use pydough.explain_term?"
-            )
+            # No root in the tree (e.g. UnqualifiedGeneratedCollection, or a
+            # bare expression like LOWER(first_name + last_name)). Try to


Bare expressions shouldn't use explain, they should use explain_term since things like LOWER(first_name) can only be explained within a context.

I had to do that because I was hitting a problem with if isinstance(qualified_node, PyDoughExpressionQDAG): being an issue with collection.
I updated the code to handle it differently. See if that's okay now

…update_explain

knassre-bodo

A few more things I'd like iterated on, though some of these can potentially be in followups.

knassre-bodo · 2026-03-23T17:21:32Z

+def cross_impl() -> UnqualifiedNode:
+    return nations.CROSS(regions)
+
+
+def cross_nations_impl() -> tuple[UnqualifiedNode, UnqualifiedNode]:
+    return nations.CROSS(regions), nations


What about another one like this, where the term is a CROSS:

return customers.WHERE(market_segment == 'BUILDING'), CROSS(nations.WHERE(region.name == 'ASIA'))

And one where there is ancestry stuff going on:

return customers.CALCULATE(cust_nationkey).CROSS(nations), cust_nationkey

knassre-bodo · 2026-03-23T17:24:16Z

+  TPCH.nations.TPCH.regions.CALCULATE(COUNT(nations.comment))
+  TPCH.nations.TPCH.regions.WHERE(HAS(nations))
+  TPCH.nations.TPCH.regions.ORDER_BY(COUNT(nations).DESC())


I don't know if this is displaying correctly... it should be TPCH.nations.CROSS(TPCH.regions).CALCULATE(...)

Not 100% sure how to fix this... that may need to be a followup as part of a larger refactoring

CROSS collections render as TPCH.nations.TPCH.regions instead of TPCH.nations.CROSS(TPCH.regions)

Added a todo for now as I agree it's a larger refactoring effort.

knassre-bodo · 2026-03-23T17:28:41Z

+      ├─── SubCollection[customers]
+      └─── Where[RANKING(by=(account_balance.DESC(na_pos='last')), levels=1) == 1]
+
+This child is plural with regards to the collection, meaning its scalar terms can only be accessed by the collection if they are aggregated.


This is wrong... it shouldn't be plural. Some adjustments to the explain code may be required to get this to display the correct logic.

knassre-bodo · 2026-03-23T17:30:29Z

+
+The term is the following expression: RANKING(by=(name.ASC(na_pos='first')))
+
+This expression calls the window function 'RANKING' with the following arguments:


We should clean this up so it displasy nicely when there are no regular arguments, but it also explains any by / per arguments, as well as other ancillary arguments (e.g. cumulative, frame, default, n_buckets)

did some changes, let me know if this is okay now

unified both UDF and non-UDF branches with these improvements:

No positional args: says "This expression calls the window function 'NAME'." instead of "...with the following arguments:" followed by nothing

Ordering (by): new section lists each collation_arg when present

Partition levels (per): shown when levels is not None

Additional options: new section lists each kwarg (e.g. cumulative: True, allow_ties: False) when present

I think the behavior of specific known ones (cumulative, frame, default, n_buckets) can be explained in the string.

knassre-bodo · 2026-03-23T17:33:03Z

+This expression calls the user-defined window function 'NVAL' with the following arguments:
+  name
+  1


Same comment as earlier about other window function arguments

knassre-bodo · 2026-03-23T17:36:38Z

+  key
+
+Description: Returns true if the argument is greater than zero.
+This function is defined by the SQL macro: '{0} > 0'.


Perhaps for macro-like UDFs we can follow this up with an example by:

Count how many arguments it has (from the arguments to the call)

Creating that many dummy variables (?a, ?b, ?c)

Injecting those into the SQL text via Python's .format method

So for this text it would be:

Suppose this function were called on arguments that are translated to the following in SQL: '?a' Then the final SQL text for this function call would be: '?a > 0'

knassre-bodo

Great job @hadia206! Left a few final comments, but feel free to merge after

knassre-bodo · 2026-03-23T23:52:59Z

+        return [
+            f"This node accesses user-generated collection {self.name!r}.\n"
+            f"Columns: {', '.join(sorted(self.columns))}",
+            f"Unique columns: {', '.join(sorted(unique_terms))}",


I think this one should be in verbose, but the type of user generated collection should not be (add that in the individual overrides)

knassre-bodo · 2026-03-23T23:55:23Z

+                    # TODO: when the collection is a CROSS, qualified_node.to_string()
+                    # renders as e.g. "TPCH.nations.TPCH.regions" instead of the
+                    # friendlier "TPCH.nations.CROSS(TPCH.regions)". Fixing this
+                    # properly requires propagating CROSS identity through the QDAG


Or finding a way to render the unqualified node that looks nicer, like how QDAG looks

knassre-bodo · 2026-03-23T23:56:24Z

+
+The term is the following expression: RANKING(by=(name.ASC(na_pos='first')))
+
+This expression calls the window function 'RANKING' with the following arguments:


I think the behavior of specific known ones (cumulative, frame, default, n_buckets) can be explained in the string.

knassre-bodo · 2026-03-24T00:01:54Z

+  TPCH.regions.CALCULATE(nations.customers.WHERE(RANKING(by=(account_balance.DESC(na_pos='last')), levels=1) == 1).SINGULAR.account_balance)
+
+To learn more about this child, you can try calling pydough.explain on the following:
+  TPCH.regions.nations.customers.WHERE(RANKING(by=(account_balance.DESC(na_pos='last')), levels=1) == 1).SINGULAR


Oop, we should adjust the repr for SINGULAR in qdag to have the () at the end.

knassre-bodo · 2026-03-24T00:02:50Z

+Suppose this function were called on arguments that are translated to the following in SQL: '?a', '?b', '?c'
+Then the final SQL text for this function call would be: 'ABS(?b - ?a) <= ?c'


I think this part might a verbose-only thing

…update_explain

…te tests)

…nces[bad_plural_j]

[run CI] update explain

fa72c02

hadia206 changed the title ~~Update explain to include Singular/CROSS and user collections~~ Update explain to Include Singular/CROSS and User Collections Mar 9, 2026

hadia206 added 3 commits March 9, 2026 15:16

[run CI] fix generic error

ac3f74b

[run CI] fix generic error

535fa12

[run CI] update message

a27ded7

hadia206 marked this pull request as ready for review March 10, 2026 01:10

hadia206 requested review from a team, john-sanchez31, juankx-bodo and knassre-bodo and removed request for a team March 10, 2026 01:10

john-sanchez31 approved these changes Mar 10, 2026

View reviewed changes

[run CI] try CROSS

d3389ca

hadia206 changed the title ~~Update explain to Include Singular/CROSS and User Collections~~ Update explain to Include Singular, CROSS and User Collections Mar 12, 2026

[run CI] handle CROSS, update Singular test

27cd9f2

knassre-bodo reviewed Mar 13, 2026

View reviewed changes

hadia206 added 6 commits March 13, 2026 10:38

[run CI] comments and some cleanup

daef4cf

Merge branch 'main' of https://github.com/bodo-ai/PyDough into Hadia/…

e58d22b

…update_explain

address Kian's comment pt. 1

235f2a3

[run CI] add singular term information as Kian asked

7b7e7db

verbose show df and explain_term test with CROSS

5c5c7bf

[run CI] UDF

004cade

hadia206 requested a review from knassre-bodo March 16, 2026 21:10

knassre-bodo reviewed Mar 23, 2026

View reviewed changes

[run CI] address Kian's comments

ad5e427

hadia206 requested a review from knassre-bodo March 23, 2026 21:32

knassre-bodo approved these changes Mar 24, 2026

View reviewed changes

hadia206 added 3 commits March 24, 2026 11:52

Merge branch 'main' of https://github.com/bodo-ai/PyDough into Hadia/…

afbb050

…update_explain

[run CI] address Kian's comments (move some words to verbose and upda…

55c7b32

…te tests)

[run CI] update text to SINGULAR() in test_malformed_collection_seque…

77bc9c1

…nces[bad_plural_j]

hadia206 merged commit 158a0df into main Mar 24, 2026
15 checks passed

hadia206 deleted the Hadia/update_explain branch March 24, 2026 19:50


		The term is the following expression: RANKING(by=(name.ASC(na_pos='first')))

		This expression calls the window function 'RANKING' with the following arguments:

		Suppose this function were called on arguments that are translated to the following in SQL: '?a', '?b', '?c'
		Then the final SQL text for this function call would be: 'ABS(?b - ?a) <= ?c'

Conversation

hadia206 commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

john-sanchez31 left a comment

Choose a reason for hiding this comment

Uh oh!

knassre-bodo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hadia206 Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

knassre-bodo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knassre-bodo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hadia206 commented Mar 9, 2026 •

edited

Loading

hadia206 Mar 13, 2026 •

edited

Loading