Is your feature request related to a problem? Please describe.
Previously setting the container home cache directory to host /tmp directory is a bad design as the /tmp directory are not guaranteed to be cleared on a regular basis, the solution was to have a user-defined cache directory mounted to container home cache directory, and it's up to the user to constantly monitor the usage of that directory, this solves the issue but is still not ideal.
Describe the solution you'd like
SLURM_TMPDIR is a temp directory associated with the Slurm job and would be deleted with the job, we should use this directory instead of a user-defined cache directory to avoid asking users to be mindful of their storage space.