When reading data xr.open_mfdataset , we get this warning:
UserWarning: The specified chunks separate the stored chunks along dimension "lon" starting at index 240. This could degrade performance. Instead, consider rechunking after loading.
Hi @SarahAlidoost, I did a bit further investigation. This userwarning is cause by the fact that we are calling chunk reading to netcdf files here and here. Since the original nc files are not chunked. This chunking probably causes multiple file reading opt in chunks, hence the warning.
If I remove the chunking args but chunk afterwards, the warnings are gone. However on HPC this causes memeory issue.
based on this, I think it still worth to use Zarr since it's natively chunked files. I am running another tiny experiments to see if switching to Zarr helps with improving CPU efficiency as in #44
Originally posted by @rogerkuou in #42
When reading data xr.open_mfdataset , we get this warning:
Originally posted by @rogerkuou in #42