Trouble encoding a 16k (15360x7680) video

[mrlando]

8k videos are fine and fast (9 seconds for a 2 sec file) but I'm having trouble encoding a 16k ProRes file to spatial. I did some AI upscaling to get to 16k to see if it looks better.
It is very slow (like 23 minutes for a 2 sec 16k file). Often it errors too with: "Cannot finish writing"

There is an app that does the encode faster (like 25 sec for 2 sec file) so I know it is possible. It is called MVConverter.
https://mvconverter.app/

If you want I can send you a 16k sample to test.

I'm using the 1.2 version of the GUI app and version 0.6.2 of the spatial terminal app.

[mrlando]

Oh and when you doubleclick on the top op the window, the buttons dissapear.

[AndrewHazelden]

Could you do a quick performance comparison:

If you run Mike Swanson's Spatial CLI tool directly in a macOS terminal session, do you also get the same slow video encoding performance?

If not, then I will have to look at the CLI flags I am using and revise them. 🙂

[mrlando]

I tried the CLI tool and the behaviour looks the same. Very slow progress, it seems to slow down further the more frames it does.

[AndrewHazelden]

My understanding is the Spatial CLI is simply calling the macOS native built-in AV Foundation library from Apple.

There is a chance the other spatial video converter you are exploring is using a custom encoder toolchain stacked ontop of FFmpeg or LibAV. This is very likely, if there is significant differences in performance. On the downside, if this is the case, their might be legal issues on Fraunhofer patent license fees coverage for the MV-HEVC patent license if FFmpeg is used since the open-source project offloads that responsibility to the downstream user. Which is you.

[mrlando]

Mmmh, ok so you are saying that the issue lies in the AV Foundation library from Apple, so they would need to update that to fix this?

[AndrewHazelden]

There might be some subtle aspects that can be refined by the API user that relates to if the Apple Silicon CPU vs XPU (Hybrid CPU + GPU) encoding is fully tuned. That is not something I have explored since I only made a wrapper GUI for my part of spatial video encoding R&D project.

[mrlando]

Perhaps the resolution is too big to be handled by the GPU and it switches to CPU?

[AndrewHazelden]

The macOS "Activity Monitor" program has a GPU usage history view. That could clarify how much CPU vs GPU load happens at different video frame sizes.

[mrlando]

Yes there is a significant difference:

8k:

16k:

But most of the time this:

[hughred22]

Wow, 16K - what camera are you using?!

[mikeswanson]

@mrlando ...this sounds more like an issue with the command-line tool. Feel free to drop me a note. If you can share a sample clip, that would be useful. 16K x ?K frames are tough to encode, so it doesn't surprise me that they're causing issues.

[mrlando]

Wow, 16K - what camera are you using?!

I’m from the future and shot this on my iPhone 18 Pro Ultra Immersive! 😎

No just kidding! I shot on a rented Canon R5 with the dual fisheye lens at 8k and did an AI upscale.

Excellent meeting you here Hugh! Have watched a lot of your great videos!

[mrlando]

Hi Mike, I’ve sent you the sample clip yesterday!

Thanks for making your excellent tool!

[mrlando]

I've did some testing and it seems 15168x7568 is the biggest it will go without choking the spatial 0.6.2 terminal app. That is not too far off 15360x7680.

I did notice the GUI tool displays the bitrate to be 0 bps. If I use the terminal app directly it correctly shows the bitrate (see screenshots).

GUI:

Terminal:

[mrlando]

And another thing:

The GUI tool often seems to fail to remove the temp file (at least with the bigger frame sizes. The terminal tool doesn't create a temp file.