[Tracking] Tracking issue for the new memory efficient dataset.

[Logiquo]

Starting from 2.0a11, PyHealth starts to use a disk-based memory efficient dataset to reduce the memory usage for large dataset such as MIMIC4.

This issues tracks any potential bugs or improvements required for new memory efficient dataset.

Improvements

• Better docs for the cache behaviour. #763
• Add option to cache transformed data from processors and skip pipeline entirely #774
  Add option to cache transformed data from processors and skip pipeline entirely #783
• .set_task gives write cache to the same directory for the same task with different configuration #764
  Update the default task cache path to include task parameter names and values #766
• Batched processing for task transformation to speed up.
  Furthur optimization on task transformation.  #750
• Support multi-worker for task transformation.
  Multiprocess task transformation #748
• Support configure n_worker for dask.
  Add num_workers to BaseDataset #743

Bugs

• Dataset cache name doesn't seem to include the tables used when cache_dir is set. #833
• Add a .clear_cache and .clear_task_cache method to avoid the need to manually delete the cache. #765
  Add clear_cache and clear_task_cache methods to BaseDataset #770
• The code will hang at set_task if any of the worker have 0 sample written. #782
  Fix the code will hang at set_task if any of the worker have 0 sample written #784
• Temporary folder for dataset is not proprely cleaned after dataset processing.
  Clean up tmpdir correctly, cache task transformation result, and better notebook support.  #753
• Cached data is not cleaned if the program crashed in the middle, which may lead to corrupted cache file.
  Clean up tmpdir correctly, cache task transformation result, and better notebook support.  #753
• Incorrect null handling for patient_id and timestamp.
  Fix incorrect null handling for patient_id and timestamp #746
• Time-series processor, process() doesn't seem to properly set the self.n_features or self.size() function properly #742
  Fix/processors fit process #744

[EricSchrock]

@jhnwu3 and @Logiquo while developing #751, I noticed a couple sharp edges with the cache_dir parameter (added in #739).

First, if you call set_task() multiple times on a dataset object, it will reuse the same samples cache. Then when you try to access the different sample datasets, you will get different samples than you expect. To reproduce this issue, comment out the following lines in test/core/test_mimic3_readmission_prediction.py.

cls.samples15days = dataset.set_task(
    task=ReadmissionPredictionMIMIC3(window=timedelta(days=15)),
    # cache_dir=cls.cache_dir1.name
)

cls.samples5days = dataset.set_task(
    task=ReadmissionPredictionMIMIC3(window=timedelta(days=5)),
    # cache_dir=cls.cache_dir2.name
)

cls.sampleswithminors = dataset.set_task(
    task=ReadmissionPredictionMIMIC3(exclude_minors=False),
    # cache_dir=cls.cache_dir3.name
)

Second, if you initialize a dataset, make a change to the input files, and then initialize the dataset again, it will not pick up your change to the input file unless you delete the cache. I noticed this issue because I had to make a change to test-resources/core/mimic3demo/ADMISSIONS.csv.gz to test the exclude_minors parameter of the ReadmissionPredictionMIMIC3 task.

Ideas

• Perhaps we could hash the __init__() and set_task() parameter values into the cache paths and use that to tell whether to use cached dataset data and/or samples?
• Perhaps we could also hash the contents of the input files into the cache paths and use that to tell whether the input files have changed and need to be re-read into the cache?
• Maybe we just need better documentation of how to use cache_dir with proper warnings and instructions for people iteratively developing their own dataset and task classes?
• Other?

[Logiquo]

• Perhaps we could hash the __init__() and set_task() parameter values into the cache paths and use that to tell whether to use cached dataset data and/or samples?
• Perhaps we could also hash the contents of the input files into the cache paths and use that to tell whether the input files have changed and need to be re-read into the cache?
• Maybe we just need better documentation of how to use cache_dir with proper warnings and instructions for people iteratively developing their own dataset and task classes?
• Other?

I agree with your on point 1 and 3. For 2, given the potential amount of data we are dealing with, hashing the entire input files is both slow and resource intensive. We do hash the root and dataset_name though, so if you have 2 version of the file and they are at different locations or with a different dataset_name, it should be at different location.

For the task, this is indeed a issue. We will need to figure out a way for task to expose a unique name for different configuration, as it's currently purely based on task.task_name.

In terms of documation, I agree that we need to better document the behaviours.

[jhnwu3]

Sorry for being late to the discussion, re: potential solutions to hashing tasks

Each task can have its own input and output schemas, it's probably possible to convert each into a string along with all parameters as effectively each schema is just dictionary object, so we can probably flatten and then hash in this case.

Something like task_name_input_schema_input_param1_out_schema -> hash