_get_boundaries() fails with "munmap_chunk(): invalid pointer"

[pundumx]

@BoXHED @apakbin Applying preprocess() as in the tutorial to 2.5M rows of data, I get the error:

munmap_chunk(): invalid pointer
Fatal Python error: Aborted

Current thread 0x00007f7a55e27100 (most recent call first):
File "/jupyterhub/anaconda3/lib/python3.9/site-packages/boxhed_prep/boxhed_prep.py", line 174 in _get_boundaries
File "/jupyterhub/anaconda3/lib/python3.9/site-packages/boxhed_prep/boxhed_prep.py", line 343 in preprocess
File "/jupyterhub/anaconda3/lib/python3.9/site-packages/boxhed/boxhed.py", line 128 in preprocess
File "/jupyter-data/home/pundumx/model4_bh_250103.py", line 680 in
Aborted (core dumped)

[pundumx]

The issue is another data requirement not documented in the tutorial: every occurrence of any given value in the ID column must be consecutive.

[apakbin]

hi @pundumx, do you face the same issue with smaller datasets? If not the issue seems to be memory related, as there is not enough space on memory for the dataset you have. And thanks for your feedback on the data requirement. If by that you mean the ID's should look like: "1, 1, 1, 2, 2, 2, 2, 2, 3, 3, ..." you are right.

[pundumx]

@apakbin I tested a dataset with only 25 rows, IDs not consecutive, and got the same error.