Add options to skip decoding Statistics and SizeStatistics in Parquet metadata

[etseidl]

Which issue does this PR close?

• Part of [EPIC] A collection of items to improve speed of parquet metadata encoding #5853.

Rationale for this change

Add ability to skip the decoding of more types of statistics contained in the Parquet column metadata. While this currently doesn't have a huge impact on decode time, it can reduce the amount of memory used by the ParquetMetaData.

What changes are included in this PR?

Adds more options and tests for those options. Also adds size statistics to the metadata bench.

Are these changes tested?

Yes

Are there any user-facing changes?

Only adds new options, no breaking changes.

[alamb]

Sorry I didn't see this one before. I'll try and review it shortly

[alamb]

run benchmark encoding metadata

[alamb-ghbot]

🤖 ./gh_compare_arrow.sh gh_compare_arrow.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing skip_column_stats (072ecf6) to c2bd7d9 diff
BENCH_NAME=encoding
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench encoding
BENCH_FILTER=
BENCH_BRANCH_NAME=skip_column_stats
Results will be posted here when complete

[alamb]

Makes sense to me -- thank you @etseidl

I launched a few more benchmarks off just to be sure this doesn't have some weird impact but I don't expect it to

[alamb-ghbot]

🤖: Benchmark completed

Details

group                                                                main                                   skip_column_stats
-----                                                                ----                                   -----------------
decoding: dtype=FixedLenByteArray(16), encoding=BYTE_STREAM_SPLIT    1.01    560.7±6.71µs        ? ?/sec    1.00    557.3±4.57µs        ? ?/sec
decoding: dtype=FixedLenByteArray(2), encoding=BYTE_STREAM_SPLIT     1.00    371.3±4.78µs        ? ?/sec    1.03   383.7±15.37µs        ? ?/sec
decoding: dtype=f32, encoding=BYTE_STREAM_SPLIT                      1.00     34.4±0.21µs        ? ?/sec    1.00     34.5±0.45µs        ? ?/sec
decoding: dtype=f64, encoding=BYTE_STREAM_SPLIT                      1.00    105.0±1.28µs        ? ?/sec    1.00    105.2±1.16µs        ? ?/sec
encoding: dtype=FixedLenByteArray(16), encoding=BYTE_STREAM_SPLIT    1.00    325.4±3.17µs        ? ?/sec    1.00    325.0±3.15µs        ? ?/sec
encoding: dtype=FixedLenByteArray(2), encoding=BYTE_STREAM_SPLIT     1.05     44.2±6.92µs        ? ?/sec    1.00     42.1±5.30µs        ? ?/sec
encoding: dtype=f32, encoding=BYTE_STREAM_SPLIT                      1.00     22.9±0.16µs        ? ?/sec    1.00     22.9±0.06µs        ? ?/sec
encoding: dtype=f64, encoding=BYTE_STREAM_SPLIT                      1.00     56.9±0.27µs        ? ?/sec    1.00     57.0±0.34µs        ? ?/sec

[alamb-ghbot]

🤖 ./gh_compare_arrow.sh gh_compare_arrow.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing skip_column_stats (072ecf6) to c2bd7d9 diff
BENCH_NAME=metadata
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench metadata
BENCH_FILTER=
BENCH_BRANCH_NAME=skip_column_stats
Results will be posted here when complete

[alamb-ghbot]

🤖: Benchmark completed

Details

group                                            main                                   skip_column_stats
-----                                            ----                                   -----------------
decode metadata (wide) with schema               1.02     40.1±0.31ms        ? ?/sec    1.00     39.5±0.42ms        ? ?/sec
decode metadata (wide) with skip PES             1.04     40.1±0.70ms        ? ?/sec    1.00     38.7±0.25ms        ? ?/sec
decode metadata (wide) with skip all stats                                              1.00     44.2±0.44ms        ? ?/sec
decode metadata (wide) with skip column stats                                           1.00     42.6±0.50ms        ? ?/sec
decode metadata (wide) with skip size stats                                             1.00     49.0±1.01ms        ? ?/sec
decode metadata (wide) with stats mask           1.02     39.8±0.33ms        ? ?/sec    1.00     38.9±0.40ms        ? ?/sec
decode metadata with schema                      1.01      5.7±0.08µs        ? ?/sec    1.00      5.6±0.03µs        ? ?/sec
decode metadata with skip PES                    1.02      8.9±0.06µs        ? ?/sec    1.00      8.7±0.07µs        ? ?/sec
decode metadata with skip column stats                                                  1.00      8.9±0.13µs        ? ?/sec
decode metadata with stats mask                  1.02      8.9±0.14µs        ? ?/sec    1.00      8.8±0.10µs        ? ?/sec
decode parquet metadata                          1.00      9.4±0.11µs        ? ?/sec    1.00      9.4±0.08µs        ? ?/sec
decode parquet metadata (wide)                   1.02     43.4±0.39ms        ? ?/sec    1.00     42.4±0.43ms        ? ?/sec
decode parquet metadata w/ size stats (wide)                                            1.00     55.8±0.65ms        ? ?/sec
open(default)                                    1.02     10.0±0.20µs        ? ?/sec    1.00      9.8±0.11µs        ? ?/sec
open(page index)                                 1.00    166.1±1.37µs        ? ?/sec    1.00    165.9±0.71µs        ? ?/sec

[scovich]

LGTM

[alamb]

I am working to prepare the arrow 57.2.0 release -- do we want to merge this one, or shall we wait for arrow 58?

[etseidl]

I am working to prepare the arrow 57.2.0 release -- do we want to merge this one, or shall we wait for arrow 58?

I'll defer to your judgement @alamb. I don't think there's any pressing need at the moment, but it also doesn't need any further changes AFAIK, and it's not a breaking change.

[alamb]

Thanks @etseidl -- let's wait then -- we already have quite a lot of good stuff queued up

[alamb]

This PR appears to have picked up some conflicts -- likely with

• feat: change default behavior for Parquet PageEncodingStats to bitmask #9051

[alamb]

main is now open for the next release so I think once this the conflicts are resolved we can merge this one in too

[etseidl]

@alamb I'm finally waking from my holiday hibernation. I fixed the conflicts, but also reverted some benchmark changes made in #9051 that I didn't have the opportunity to review. The non-new benches should now match their counterparts in 57.2.

[alamb]

@alamb I'm finally waking from my holiday hibernation.

Nice! That was me too this week (and there was quite a bit of backlog to get through!)

I spent some non trivial amount of time over the break studying and profiling the parquet reader so I expect to be doing a bunch of micro optimizations there (aka reducing allocations / reallocations).

Should be a fun January!

[alamb]

Thanks again @etseidl and @scovich