Significant bottleneck with text encoder outputs and latent caching to disk with .npz vs .safetensors

[gazingstars123]

Hello, I wanted to ask if there are plans to officially support .safetensors for latent and text encoder caching (as discussed in #1750) or if it's already implemented that I'm not aware of. However, skimming through the newer Musubi Tuner code, it seems like that isn't the case.

I've been testing on very large datasets (hundreds of thousands of images or millions of images) and found that the current .npz implementation is a significant bottleneck with huge additional file sizes and with both read and write operations, in many cases up to 40-50 times slower reading cache from disk compared to .safetensors.

[kohya-ss]

Indeed, the biggest problem right now is that bfloat16 cannot be saved to npz.

#1750 set the bar too high, so I think we should start by simply replacing .npz with .safetensors (which seems functionally sufficient).

[kohya-ss]

I've just opened a draft PR #2290