Arrow Pushdowns fail when *any* column is a string_view

[paultiq]

What happens?

When any column is a string_view, Arrow pushdowns fail... even if the pushdown expression doesn't involve a string_view.

Related to #187, although Vortex actually doesn't fail the below case.

To Reproduce

import pyarrow as pa
import duckdb
print(f"{duckdb.__version__=}")
arrow_table = pa.table({"id": [1], "name": ["something"]}, schema=pa.schema([("id", pa.int64()), ("name", pa.string_view())]))
print(arrow_table.schema)
duckdb.execute("SELECT * FROM arrow_table WHERE id > 90").fetchall()

duckdb.version='1.4.4.dev11'
id: int64
value: string_view
...
_duckdb.InvalidInputException: Invalid Input Error: arrow_scan: get_next failed(): NotImplemented: Function 'array_filter' has no kernel matching input types (string_view, bool)

OS:

Linux

DuckDB Package Version:

1.4.4.dev11

Python Version:

3.13

Full Name:

Paul T

Affiliation:

Iqmo

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a nightly build

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration to reproduce the issue?

• Yes, I have

[paultiq]

This is an Arrow issue as shown below, but DuckDB could/should not attempt a pushdown if any column is a string_view (or perhaps any View).

apache/arrow#43010

import pyarrow as pa
import pyarrow.dataset as ds

table = pa.table({"id": [1, 2, 3], "value": ["a", "b", "c"]}).cast(pa.schema([("id", pa.int64()), ("value", pa.string_view())]))
scanner = ds.dataset(table).scanner(filter=ds.field("id") > 1)
scanner.to_table()

[Tishj]

This was merged upstream to fix the problem: duckdb/duckdb#20165

[paultiq]

That PR doesn't fix this issue. This issue was tested after that was merged.

That PR handles cases where the expression compares a string_view column.

This issue occurs when any column is a string_view.

Example:

import polars as pl
import duckdb
print(duckdb.__version__)

df = pl.DataFrame({"sv_col": ["abc"], "int_col": [1]})
arrow_table = df.to_arrow(compat_level=pl.CompatLevel.newest())


# Passes, because of duckdb/duckdb#20165
duckdb.execute("select * from arrow_table where sv_col = 'abc'").df()

# Still fails
duckdb.execute("select * from arrow_table where int_col = 1").df()

[paultiq]

In digging into this, and the related Vortex issue, there's a messy set of problems bigger than just this issue:

Arrow backends (Vortex, Arrow Compute, etc) support different predicates, but arrow.cpp:CanPushdown assumes coverage is universal:

• Arrow Compute's array_filter doesn't support tables that contain string_views (this issue), but Vortex DataSets do
• After Map view types when converting arrow_expressions to vortex expressions vortex-data/vortex#5765 is merged, Vortex DataSets will support equality / inequalities on string_views, but Arrow Compute will not
• Arrow Compute supports comparisons of timestamps, but Vortex DataSets do not Epic: vortex-compute compute functions vortex-data/vortex#5781. This is a new issue that raises an exception in duckdb.

In other words, duckdb makes the false assumption that all Arrow datasets support the same predicates / comparisons for pushdown.

Possible Solutions

• duckdb should be pessimistic, and deny pushdown when any backend doesn't support a particular predicate on a particular datatype
• duckdb should be optimistic, attempting to pushdown, but duckdb-python would gracefully handle an exception during pushdown - falling back to a no-pushdown case. Currently, duckdb-python will reraise the exception.
• (too complicated) duckdb would need to add rules to handle backend specific cases, handling (for instance) vortex datasets differently.
• somehow check at runtime which predicates are supported: this would involve some change in Arrow.