Repeated columns error

[moaxcp]

Hello,

Great library! I am trying to optimize some data by splitting a string into a repeated field but I ran into an issue when using REPEATED. I wrote a small example.

$schema = Schema::with(FlatColumn::string('test'));
$writer = new Writer();
$writer->open('/path/test.parquet', $schema);
$writer->writeRow(['test' => ['one', 'two', 'three']]);
$writer->close();

This results in an error.

TypeError: Flow\Parquet\Dremel\ColumnData\FlatValue::__construct(): Argument #4 ($value) must be of type string|int|float|bool|null, array given

Shouldn't this work or am I using this feature wrong?

[norberttech]

Hey @moaxcp !
Yea I think this should be explained better covered in documentation.
Repetition::REPEATED - this is not how you can store ['one', 'two', 'three'] in test column.

['one', 'two', 'three'] - in parquet this is list<string>, as there is no such concept as array in parquet types.

I prepared for you working example here so you can play with it.

I'm closing it, as it's not really a bug, but feel free to use this issue in case of any further questions/issues

[moaxcp]

Thanks for the help!

I was able to use a NestedColumn with a list and it worked. I have also been looking at the parquet js library since I have to write data in php and read it in js. It appears to have both lists and arrays. Is it doing something wrong with arrays? The file looks wrong.

test.js

[norberttech]

let me take a look again!

[norberttech]

@moaxcp I'm afraid I dont have good news for you, so what this js parquet library is doing, is a hack, parquet like I said, as a binary format does not have anything like array.
In reality what you refer to as an array is a list and this is the only supported type by parquet specification.

The explanation why it can't be supported is because in binary formats like Parquet, there's an important distinction between the physical schema (how bytes are encoded) and the logical schema (what the data means). The repeated keyword you see in Parquet schemas is actually machinery for the Dremel encoding algorithm—it's not a data type, it's an instruction for how to serialize nested data.
The Parquet specification explicitly defines LIST as a 3-level structure:

required group my_list (LIST) {
  repeated group list {
    required string element;
  }
}

This structure might seem verbose, but it serves a critical purpose: it allows both the list itself and its elements to be nullable. A bare repeated string tags can't distinguish between "no list," "empty list," and "list with null element" - they all look the same in the encoding.

Files written with bare repeated fields cause real problems:

• AWS Athena: Throws errors like "Expected LIST column to only have one field"
• Hive/Presto: Misinterpret the schema or fail to read the data
• Spark: Has complex backward-compatibility rules but may still have issues

So my recommendation for you would be to just use list and also read it as a list.