NVENC & QSV: RGB/YUV444 Texture Encoding

[derrod]

Description

This PR implements support for encoding RGB or YUV444 with texture sharing, avoiding the CPU conversion and system memory round-trip.

Encoder Support

Format / Encoder	NVENC H.264	NVENC HEVC	QSV HEVC
8-bit RGB	✅	✅	✅3
8-bit 4:4:4 YUV	✅	✅	✅3
10-bit RGB (SDR)	✅1, 2	✅2	✅3
10-bit RGB (HDR)	❌	✅2	✅3
10-bit 4:4:4 YUV (SDR)	✅1, 2	✅2	✅3
10-bit 4:4:4 YUV (HDR)	❌	✅2	✅3

¹ 10-bit H.264 is only supported starting with Blackwell generation (RTX 50-series) GPUs
² Windows-only as Y410 is not supported by CUDA/OpenGL. Linux 10-bit support may be possible via planar GBR/YUV with 3x16bpp planes, but would require adding additional formats to OBS.
³ Windows-only since I'm not familiar with Linux QSV, but support may be possible

• NVIDIA NVENC H.264 and HEVC support is available for Windows and Linux (except for 10-bit)
  • NVENC AV1 does not support 4:4:4
• Intel QuickSync HEVC support is Winodws only at the moment
  • QuickSync H.264 does not support 4:4:4
  • QuickSync AV1 should support 4:4:4 from Panther Lake onwards, but is not currently implemented
• AMD AMF does not support 4:4:4 encoding at all so it remains unsupported

Linux support for QuickSync should be possible (at least for 8-bit), but I'll leave that to people that are more familiar with the Linux side of things :)

---

Frontend changes

• Added R10l as a new 10-bit RGB video format (Windows only)
• • Added Y410 as a new 10-bit YUV 4:4:4 video format (Windows only)

libobs changes

• Added GS_AYUV and GS_Y410 texture formats
• Changed encoder GPU scaling to be automatically used for format conversion for texture encoders when available, even if scaling itself is disabled
  • obs_encoder_info.get_video_info() is now called for texture encoders to allow encoders to negotiate a format with libobs
  • Also fixed some edge cases and added some additional logging when setting up a GPU scaling mix
• Changed from old using_p010_tex /using_nv12_tex to using an enum storing the active GPU texture type (since there can only be one anyway)
• Added GPU conversion and shared texture support for existing formats VIDEO_FORMAT_AYUV and VIDEO_FORMAT_R10L
• Added VIDEO_FORMAT_GBRA, VIDEO_FORMAT_Y410, and VIDEO_FORMAT_GBR10

libobs-d3d11 changes

• Added GS_AYUV textures using DXGI_FORMAT_AYUV
• Added GS_Y410 textures using DXGI_FORMAT_Y410
• Updated DirectX feature level array to include 11.1 (techincally required for DXGI_FORMAT_AYUV)

libobs-opengl changes

• Added GS_AYUV textures using GL_RGBA
• Added GS_Y410 textures using GL_RGB10_A2
  • Currently not advertised as available as QSV/NVENC do not seem able to consume these on Linux

obs-nvenc

• Added get_video_info() for texture encoders to negotiate texture formats for best performance
• Added support for encoding YUV 4:4:4 or GBR via AYUV textures
  • Windows (D3D11) and Linux (OpenGL) are supported
• Added support for encoding 10-bit YUV 4:4:4 or GBR via Y410 textures
  • Windows (D3D11) only for now
• Some code cleanup related to CUDA and VUI parameters

obs-qsv11

• Fixed get_video_info() callback for texture encoders, where data is NULL
• Added support for encoding YUV444 or GBR data in AYUV textures
  • Windows-only for now, Linux support unclear
• Added support for encoding 10-bit YUV 4:4:4 or GBR via Y410 textures
  • Windows (D3D11) only for now

---

Inspired by and extended from, but also closes #13113

Also closes #8872

Motivation and Context

4:4:4 encoding is currently only available via CPU conversion (NVENC) or not at all (QSV). The primary use-case is for raw footage without chrome subsampling, for example, screen recordings that are intended for editing, where chroma subsampling results in undesireable artefacts around text.

The supported modes all have their pros and cons, hence why multiple are supported:

OBS Setting	Encoder Format	Pros	Cons
BGRA (8-bit)	GBRA	- No subsampling - No RGB->YUV conversion losses	Unsupported in some software, e.g. Davinci Resolve
I444 (8-bit)	AYUV	- Wider compatibility - No subsampling	Some color inaccuracies due to RGB->YUV conversion
R10l (10-bit)	GBR10	- No subsampling - No RGB->YUV conversion losses - HDR support	- Unsupported in some software, e.g. Davinci Resolve - Support for 10-bit video may be spotty
Y410 (10-bit)	Y410	- No subsampling - Reduced RGB->YUV conversion loss due to higher precision - HDR support	- Support for 10-bit video may be spotty

Despite the alpha channel technically being included, none support it in encoded form right now (NVENC only supports alpha with RGBA or NV12 input).

Examples:*

YUV 4:2:0 (OBS: NV12)	YUV 4:4:4 (OBS: I444)	RGB (OBS: BGRA)

How Has This Been Tested?

• NVENC
  • Windows with an RTX Pro 6000 Blackwell
  • Linux (Ubuntu) with an RTX 5080
• QuickSync
  • Windows with an Arc A380

Types of changes

• New feature (non-breaking change which adds functionality)
• Tweak (non-breaking change to improve existing functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[mikooomich]

#8872 can be closed by this PR

[derrod]

Updated with a couple fixes for thing I noticed while re-reviewing my own PR.

But more importantly, also added support for 10-bit 4:4:4 encoding with NVENC when OBS is set to P416. This is somewhat experimental, and while it works great for SDR, I'm not familiar enough with HDR to say whether this is fully correct. It looks the same as P010 PQ though.

This should also be possible with NVENC on Linux and with QuickSync on Windows, but I haven't tested that yet.

[zhangboyang]

Tested my specific setup (8bpc NVENC), it works great! 😀

[derrod]

Couple ideas that came up while chatting with some folks:

• Use a triple height single-channel DXGI_FORMAT_R16_UNORM texture to create a P410 input for NVENC
• Use DXGI_FORMAT_Y410 for Intel 10-bit with the RGB->YUV conversion being on the OBS side

These should, in theory, fix the full range issue with QSV and allow the same GBR+YUV feature as I have for 8-bit, but require a bit more work and more complex shaders. I'll move this to draft for now since it'll probably mean removing and reworking the last 6 or so commits. Should be done later this week though, so not for long.

[derrod]

Unfortunately my experiments with three-plane 10-bit YUV did not quite work out. I'm waiting to hear back from NVIDIA on this. For now, I've just fixed up the RGB16A2 variant based on review comments and some other issues I've found. It's probably good enough for now, though I'd love to have 10-bit GBR encoding in the future as well.

[derrod]

Unfortunately my experiments with three-plane 10-bit YUV did not quite work out. I'm waiting to hear back from NVIDIA on this. For now, I've just fixed up the RGB16A2 variant based on review comments and some other issues I've found. It's probably good enough for now, though I'd love to have 10-bit GBR encoding in the future as well.

I tested DXGI_FORMAT_Y410 today and that just work™ with both NVENC and QSV so we got 10-bit YUV and GBR encoding now!

Linux unfortunately will have to wait. I think QSV may be possible because VA-API seems to support Y410, but it seems more complex when dealing with DMABUF. NVIDIA on Linux should still work with a planar approach (triple-height GS_R16) and CUDA, but that requires additional work on the OBS side of things to support it, and I don't think it's particularly worth it right now.