Question regarding Parquet Page Index: Why enable it during write if it's not utilized during read?

[Patzifist]

Describe the usage question you have. Please include as many useful details as possible.

Hi team,

I’m currently configuring parquet-go (v18) for high-performance data ingestion and I have a question regarding the utility of the Page Index (Column Index / Offset Index).

In my current setup, I see that PageIndexEnabled can be toggled in WriterProperties. However, after digging into the arrow-go reader and scanner implementations, I couldn't find clear evidence that the Page Index is being used to perform page-level skipping during queries.

Questions:

1. Read-side support: Does the current arrow-go Parquet reader or the higher-level Scanner API actually implement page-level pruning using the Page Index? Or is filtering still limited to Row Group boundaries?
2. Writing strategy: If the Go reader doesn't support it yet, is there any reason to enable it during the write phase other than compatibility with external engines (like Spark or Trino)?
3. Overhead: Are there any significant performance penalties when writing files with Page Index enabled in a Go-centric environment, given the extra metadata management?

I want to avoid including "dead weight" metadata in my files if it doesn't provide any performance benefits within the Go ecosystem.

Looking forward to your clarification.

Component(s)

Parquet

[zeroshade]

The ColumnChunkReader objects have SeekToRow methods which will utilize the PageIndex if it exists to do page-level pruning when seeking to a specific row index.

There's also a TODO note in the pageindex reader where we could potentially implement prefetching of pages into a page cache for quicker/faster reading.

Other than seeking to specific rows, we aren't yet automatically utilizing the Page, Column or Offset indexes in the reader. That said, if you're building something that processes queries on Parquet, you should be able to leverage the indexes to determine what rows to seek to (which will then leverage the index) programmatically.

There's definitely room for us to add more routes and utility into the API to better utilize the indexes if they exist. But I will absolutely admit that right now, if you're not doing row seeking, you're not leveraging other engines, and you aren't going to use the indexes yourself to perform the seeking, then writing the indexes probably won't benefit you much YET. Eventually I plan on trying to improve the APIs to allow better usage of the indexes, but you're correct that it doesn't benefit performance outside of these small cases yet.