mipmap_cache: respect conf request to always use embedded thumbnail

[dtorop]

When conf plugins/lighttable/thumbnail_raw_min_level is never, always use the thumbnail embedded in a raw, even when it is lower resolution the requested mipmap.

This fixes a problem particularly visible on high dpi screens with fractional scaling: Due to limitations of GTK3 (which always returns a whole-number from gtk_widget_get_scale_factor), we generate mipmaps larger than the physical display resolution. These can easily exceed the resolution of the embedded JPEG. This problem is more visible now that darktable can run on Wayland, which allows fractional scaling.

Fixes #19944.

[wpferguson]

Hi @dtorop, it's nice to see your around again :-)

I think this may mess up culling (at least for me). I'll test and see what happens.

[dtorop]

@wpferguson: Thanks likewise!

RE culling: I'm testing now. I think there's a minor problem in _thumbs_prefetch() where it also doesn't take into account PPD, so it prefetches a lower-resolution mip, then on display fetches the correct resolution mip. It's quick to fix, I may add it to this PR as it is related. But please let me know if there's another problem which you are seeing.

[dtorop]

Added fix for respecting display scaling in thumbnail prefetch. This is another problem visible when using display scaling.

@wpferguson: Does this relate to the problem you are seeing in culling, though?

[wpferguson]

My situation

• I have use raw instead of jpeg from size set to never and it's been that way for years
• In Aug of 2021 I added a function to Lua to generate cache
• At that time I had a 1080p and a 2K screen
• When I generated cache, the thumbnail came from the embedded jpeg, but the full size preview was built from raw
• I now have an R7 and a 4K screen. The largest embedded preview is cache size 3.
• For the 4K screen I need size 3 for the thumbtable and size 6 for the full preview, so 3 is the embedded and 6 is built from raw

With this patch:

• the use raw instead of jpeg from size is actually respected
• now size 6 is scaled up from the jpeg preview
• size 8 is scaled up

My concern is that user's are used to this behavior, since it's been this way for years. Using a scaled up JPG for culling is difficult since you can't tell if it's in focus or not and doing a full size preview to check focus is useless since again it's scaled way up.

I can solve my problem by setting use raw instead of jpeg from size to 1080p (size 4) and everything works like it currently works in master and before.

CAVEAT: If your embedded JPG is larger than 4K, I guess it's a moot point.

So use raw instead of jpeg from size works for maybe the first time ever, but we probably need to do a lot of advertising before we release it or deal with some issues that arise because of it.

[ralfbrown]

Should we tag this with default-behavior-change?

[dtorop]

Should we tag this with default-behavior-change?

Probably. It makes the default behavior match the behavior as documented by the config option. But before dt would never upscale a thumb if it could downscale a raw instead, and I understand from @wpferguson that this is a behavior which some (many?) users would prefer.

I'm trying to see if dt is a bit more laggy with this PR, when on a 4K display and moving through image previews. I wonder if upscaling a mip6 via Cairo is faster than working with a mip8.

[dtorop]

Maybe good to think about ideal behavior rather than extant code vs. documentation. Some useful behaviors:

1. Small thumbs use JPEGs for speed, fullscreen previews generated from raw (or very hires embedded jpeg) to allow for checking focus.
2. All thumbs and fullscreen previews use embedded JPEG, for sake of speed and because SOC JPEG can be nicer than unmodified pixelpipe on some cameras.
3. All thumbs and fullscreen previews generated from pixelpipe, because user either has a nice Lua script to do this offline, or wants consistency between thumbs & previews and has a well set up default pixelpipe.

Are there other scenarios? The current use raw instead of jpeg from size config can accommodate all of these, but it's a bit fiddly. Would a simple config giving these three choices be more helpful, and dt behind the scenes would then, if necessary, determine the mipmap cutoff to handle thumbs vs previews? But what happens then when a user switches monitors? And are dt users so competent that the current system is fine? Regardless, this may be beyond scope of this PR.

[wpferguson]

But what happens then when a user switches monitors?

I had this when my laptop was 1080p and my external monitor was 2k. I ended up generating cache for both.

'm trying to see if dt is a bit more laggy with this PR

My script spits out timing so when set to never 1171 size 3 (pull the embedded jpeg) took 22 seconds, and it took 44 seconds to upscale the 1171 images to size 6. I did cull with them and sometimes displayed full size (upscaled from size 6) and I didn't notice any lag, certainly nothing like building from raw.

For reference, building size 6 cache from raw took 470 seconds for 1171 images.

EDIT:

that this is a behavior which some (many?) users would prefer

I'm not sure they prefer it, but I'm worried they might expect it and be surprised with the change.

EDIT 2:

Lua script to do this offline

Actually it's online and runs from a post-import-film trigger so that every time I import the thumbtable cache gets built quickly so I can scroll through the thumbtable and get an idea of what I have while the culling cache is being built. Once it's done I can cull at full speed. I don't zoom to full size enough to make worthwhile to build cache for it.

[dtorop]

I didn't notice any lag, certainly nothing like building from raw

Good to know about timings. I just tried this PR on a 1920x1080 screen, and it's quite responsive -- but so was dt without this PR on that setup.

I'm not sure they prefer it, but I'm worried they might expect it and be surprised with the change.

Yes, I'm not sure either. In my own case, I prefer using embedded JPEG when culling images, even at fullscreen. I can view images without waiting for pixelpipe, and the embedded JPEGs are decent quality. I believe that the switch to Wayland and its handling of fractional scaling caused a regression for me. On a 4K display, culling images became laggy, as dt was busily running the pixelpipe in the background.

It seems like there's a balance, where some users want low latency and some want highest resolution, and ideally dt could accommodate both (with a user-visible conf option).

I did notice that with this PR, using ctrl-scroll or ctrl-middle-click to zoom in an image has a change in behavior with default config. Before, dt would run the full pixelpipe at 100% and (after a lag) show 1:1 the detail in the raw file (albeit with the image looking different than a zoomed out view, due to running the pixelpipe). Now, dt will generate a mip8 from the embedded JPEG, and zoom in on that. For me, I can get decent sharpness information even from the embedded JPEG at 100% (or switch to darkroom and see the raw), and appreciate that it is faster not to run the pixelpipe. Others may have a different experience/preference!

[ralfbrown]

Yeah, the main divide is going to be whether the camera's largest embedded preview is anywhere near the monitor's resolution, e.g 4K monitor and a Sony camera from when they didn't embed anything bigger than 1920x1080 (some really old cameras from various manufacturers only had 640x480!)

Maybe we could add an "auto" option to "use raw from size", where Auto selects what amounts to the old behavior: a cutoff of the monitor's linear pixel size and no upscaling from mipmaps.

[dtorop]

To document the changed behavior for the conf setting use raw instead of jpeg from size aka plugins/lighttable/thumbnail_raw_min_level:

Prior Behavior

thumbnail_raw_min_level	requested thumbnail resolution	generate
never	> embedded JPEG	default pixelpipe
	< embedded JPEG	embedded JPEG
small...5K	> thumbnail_hq_min_level or > embedded JPEG	default pixelpipe
	otherwise	embedded JPEG
always	any	default pixelpipe

Behavior with this PR

Changed behavior is emphasized.

thumbnail_raw_min_level	requested thumbnail resolution	generate
never	> embedded JPEG	embedded JPEG, upscaled
	< embedded JPEG	embedded JPEG
small...5K	> thumbnail_hq_min_level or > embedded JPEG	default pixelpipe
	otherwise	embedded JPEG
always	any	default pixelpipe

Recommendation

If you previously used the never setting and prefer its prior behavior:

• Set use raw instead of jpeg from size to just smaller than the resolution of your screen to always run the pixelpipe for fullscreen previews.
• Set use raw instead of jpeg from size to just smaller than the resolution of the embedded JPEG of your camera(s) (could range from 1080p to 5K) to run the pixelpipe rather than upscaling JPEGs.

If you previously used the never setting and never want to see processed raw files of unaltered images, leave the setting at never. In this case, when you zoom an image to 100%, you will now see the embedded JPEG at 100%, rather than the pixelpipe processed image.

Query

Would it be helpful to add > embedded thumb to thumbnail_hq_min_level, and make this replicate the prior behavior of never?

[dtorop]

Maybe we could add an "auto" option to "use raw from size", where Auto selects what amounts to the old behavior: a cutoff of the monitor's linear pixel size and no upscaling from mipmaps.

Overlapping comments! But yes, that is exactly it! And auto is better then > embedded thumb. I will put in a commit to this PR to do this.

[dtorop]

Maybe we could add an "auto" option to "use raw from size", where Auto selects what amounts to the old behavior: a cutoff of the monitor's linear pixel size and no upscaling from mipmaps.

A place this gets a bit tricky is when the user changes the conf value from/to auto. In thumtable.c:_thumbs_ask_for_discard(), the code will need to throw away cached thumbnails based on the resolution of the embedded JPEG of the image. This seems finicky. A quick and ugly solution (assuming an auto setting to retain prior behavior is important) would be to throw away all cached thumbnails when switching to/from auto.

[jenshannoschwalm]

Just mentioning this, when using either the lua script or dt internal backgriund update thumbs we currently already have thumbs in requested size processed after a short while. If i add a day's shots i just have an espresso and all is good and fast for culling.

But "never is never" :-) Dan, nice having you around again !

[dtorop]

@jenshannoschwalm Thank you!

For my workflow, I tend to ignore the small thumbs and go through each image via fullscreen preview, and give each image a second or two to rate it. So I'm sensitive to when those are slow to load, hence a true never mode sounds good to me.

I'm trying to wrap my head around how to discard thumbs in auto vs. never vs. particular mipmap modes. Once I do, one more commit on the way... Looking at the code I also notice that DT_MIPMIP_8 is used as a synonym for highest-resolution 8 bit mipmap. There probably should be a different constant (set to be DT_MIPMAP_8) to express that, in case there are more mipmaps eventually.

[TurboGit]

@dtorop : Nice to see you again :) I also voice my approval for "never is never". I've not looked at the actual rendering when upscaling, but if a user want this then the option is there.

[dtorop]

@TurboGit: Thank you!

Yes "never is never" seems like the right formulation.

I added a commit to support an auto option. As mentioned in the commit notes, the updated code to discard mipmaps on pref change uses a heuristic which errs on discarding too many mipmaps when switching between auto/never. I figure this is better than altering the mipmap code to tag mipmaps with metadata to identify which to discard. Switching between auto/never is an edge case of a conf which is probably rarely changed.

Also added a release note to document this. And a small bugfix where the code skipped discarding cached mip8 files during a conf switch.