Consider adding a compressed extension

[twiddli]

I understand that bbf is intended to be a container format, like mkv, but it would be really nice if it could also reduce file size (beyond deduplication) using some kind of compression scheme.

In terms of usage, this could be an option like --compression [scheme] in the CLI tool (optionally coupled with a .bbz or similar extension for distinction).

I assume the format is primarily intended for image collections, and I know that most image files are already compressed, which is why you usually don’t gain much by applying conventional archive compression. However, unlike collection formats such as zip, which are designed for arbitrary file types, bbf (as it is intended to be used) has the advantage of targeting a much narrower range of formats, e.g., image formats.

For image collections, this means we could use methods that exploit cross-image redundancy.

Consider this example:
I have a folder containing 8 large grayscale PNG files totaling 51.1 MB.
Creating a 7z archive with these files, as expected, does not result in much size reduction, if any.
But what if we use a format that leverages cross-image redundancy, such as a video codec?
A command like ffmpeg -framerate 1 -pattern_type glob -i "*.png" -c:v libx265 -crf 28 output.mp4
produces a file that is only 7.46 MB.
Since the images were encoded as a 1 fps video, extracting them is straightforward: ffmpeg -i output.mp4 -vsync 0 frame_%04d.png
Reducing 51.1 MB to 7.46 MB is a significant improvement.

Incorporating such a compression scheme would make bbf valuable not only for metadata and hierarchical categorization, but also for meaningful storage reduction.

[ef1500]

Hello again!

For image collections, this means we could use methods that exploit cross-image redundancy.

This seems really interesting. This would probably take on a different spec than the typical .bbf file. Mainly because you could specify a compression scheme by using the flags for asset entries (see spec), but this is more complex. It would be optimal to use the pre-existing spec in some manner, but I don't know if that'd work. I'm trying to keep the spec's structs frozen from here on out.

If you wanted to implement a spec for .bbz (or if you just use the spec's structs in a certain manner) and get something working, I wouldn't mind a PR. This would be something really cool to see!

To maybe get you thinking, I will also share some of my other ideas I had for using this format.

I played around with the notion of .bvf (bound video format) which could be used for flipbooks and other things which aren't quite videos, but aren't quite comics or manga either. I sort of dropped the idea because it would just be easy enough to set flags in the page entry to tell whether or not it should be part of an animation sequence or not.

Slightly tangential to cross-image redundancy, it would be a perfectly legitimate use of the format (though a flag might have to be added) to have translation layers. Since you can just store the cleaned image once, and all the transparent PNGs containing the translations would be different assets. A reader would just have to overlay the translation layer on top of the cleaned asset, and it would probably save quite a bit of space.

A user several weeks ago opened up #5 talking about block deduplication, so I added adjustable alignment in the spec. I also added the variable ream size so smaller assets can get packed to a smaller boundary.