Implement train/test/validation strategy and hyper-parameter tuning.

[SarahAlidoost]

We want to decide which data to use for tasks of train, test and validation. One approach is mentioned in this comment which is selecting different years in the dataset for each task.

Also, we should come up with a best list of values for model parameters. how to do hyper parameter tuning?

[meiertgrootes]

We could consider RayTune (an extension of Ray for hyperparameter tuning) for our hyperparameter tuning. It could also (in principle) be used to distribute training (better) over (multiple) CPU/GPU.
It is widely adopted, in particular in the Python/PyTorch ecosystem.
remains to see which parameters we want to tune.

[SarahAlidoost]

We could consider RayTune (an extension of Ray for hyperparameter tuning) for our hyperparameter tuning. It could also (in principle) be used to distribute training (better) over (multiple) CPU/GPU. It is widely adopted, in particular in the Python/PyTorch ecosystem. remains to see which parameters we want to tune.

good idea, I am now working on #44, I'll also explore whether we can embed ray.dataset and ray.train in the current workflow.

[SarahAlidoost]

We could consider RayTune (an extension of Ray for hyperparameter tuning) for our hyperparameter tuning. It could also (in principle) be used to distribute training (better) over (multiple) CPU/GPU. It is widely adopted, in particular in the Python/PyTorch ecosystem. remains to see which parameters we want to tune.

@meiertgrootes I had a look at ray.dataset and ray.train. As you mentioned, Ray is really designed for bigger, distributed workloads. Even though it can use some PyTorch features under the hood, from what I see bringing Ray in would mean rewriting our dataset into a Ray pipeline and moving things into its APIs. Same goes for training, which would add extra overhead. Since our current use case can run on a single node, I don't think Ray really brings much benefit. In #45, I implemented a util function for handling cpu and gpu usage with built-in PyTorch functionality. I’d suggest using PyTorch for hyperparameter tuning too. Ray is definitely powerful, but it’s something that’s better considered from the start rather than introduced at this stag, unless you know a clear example showing we can integrate it without major changes to the current code of dataset, train and prediction.

[SarahAlidoost]

@meiertgrootes and @rogerkuou do you have any ideas/suggestions on how to split the data into train, test, and validation sets? See example in point 2 of this comment.

[meiertgrootes]

We could consider RayTune (an extension of Ray for hyperparameter tuning) for our hyperparameter tuning. It could also (in principle) be used to distribute training (better) over (multiple) CPU/GPU. It is widely adopted, in particular in the Python/PyTorch ecosystem. remains to see which parameters we want to tune.

@meiertgrootes I had a look at ray.dataset and ray.train. As you mentioned, Ray is really designed for bigger, distributed workloads. Even though it can use some PyTorch features under the hood, from what I see bringing Ray in would mean rewriting our dataset into a Ray pipeline and moving things into its APIs. Same goes for training, which would add extra overhead. Since our current use case can run on a single node, I don't think Ray really brings much benefit. In #45, I implemented a util function for handling cpu and gpu usage with built-in PyTorch functionality. I’d suggest using PyTorch for hyperparameter tuning too. Ray is definitely powerful, but it’s something that’s better considered from the start rather than introduced at this stag, unless you know a clear example showing we can integrate it without major changes to the current code of dataset, train and prediction.

Hi @SarahAlidoost, I may be mistaken, but what I had wanted to suggest was a limited use of raytune, which is specifically designed as a hyperparameter tuning wrapper. The documentation itself provides the following example (https://docs.ray.io/en/latest/tune/examples/tune-pytorch-cifar.html) applied to an existing pytorch model. The imports are very limited and as far as I could tell changes to the load, model, predict pytorch pipeline are minimal.

The alternative would indeed be to write our own gridsearch (or similar) using pytorch, but my guess would be that that is mire work and more error prone.

I fully agree that use of ray for its other capabilities is not strictly needed for our project and could/would cause unneccessary work/delay.

[SarahAlidoost]

We could consider RayTune (an extension of Ray for hyperparameter tuning) for our hyperparameter tuning. It could also (in principle) be used to distribute training (better) over (multiple) CPU/GPU. It is widely adopted, in particular in the Python/PyTorch ecosystem. remains to see which parameters we want to tune.

@meiertgrootes I had a look at ray.dataset and ray.train. As you mentioned, Ray is really designed for bigger, distributed workloads. Even though it can use some PyTorch features under the hood, from what I see bringing Ray in would mean rewriting our dataset into a Ray pipeline and moving things into its APIs. Same goes for training, which would add extra overhead. Since our current use case can run on a single node, I don't think Ray really brings much benefit. In #45, I implemented a util function for handling cpu and gpu usage with built-in PyTorch functionality. I’d suggest using PyTorch for hyperparameter tuning too. Ray is definitely powerful, but it’s something that’s better considered from the start rather than introduced at this stag, unless you know a clear example showing we can integrate it without major changes to the current code of dataset, train and prediction.

Hi @SarahAlidoost, I may be mistaken, but what I had wanted to suggest was a limited use of raytune, which is specifically designed as a hyperparameter tuning wrapper. The documentation itself provides the following example (https://docs.ray.io/en/latest/tune/examples/tune-pytorch-cifar.html) applied to an existing pytorch model. The imports are very limited and as far as I could tell changes to the load, model, predict pytorch pipeline are minimal.

The alternative would indeed be to write our own gridsearch (or similar) using pytorch, but my guess would be that that is mire work and more error prone.

I fully agree that use of ray for its other capabilities is not strictly needed for our project and could/would cause unneccessary work/delay.

@meiertgrootes thanks for explanations. Could you look into whether hyperparameter tuning with Ray might help address this issue? I’ve assigned it to you, let me know what you think.