libobs-d3d12: Added D3D12 rendering backend and NVIDIA D3D12 texture encoding

[wanhongqing123]

Description

D3D12 is a new api in windows10 and windows11, Direct3D 12 lets apps get closer to hardware than ever before. By being closer to hardware, Direct3D 12 is faster and more efficient.

• Implement all interfaces in graphics.
• Image、duplicator、video、GDI+ source have been tested.
• WGC Duplication Capture with D3D11 shared texture.

Motivation and Context

As Windows updates and iterates, D3D11 may become obsolete, therefore D3D12 will be supported here.

@Fenrirthviti disagrees with reusing D3D11. The current OBS architecture's rendering pipeline is single-threaded, and multi-threading cannot be implemented at the underlying level. Therefore, D3D12 and D3D11 share some code. I will later place the common methods of D3D11 and D3D12 in appropriate locations.

Furthermore, in the current D3D11 rendering architecture, if the output frame rate is set to 60fps or 120fps, the frame rate will drop when the mouse moves to the middle of the rendered object. With technological advancements, 120fps and 60fps will become necessary. Currently, D3D12 Render and D3D12 NVENC hardware encoding can effectively utilize the advantages of modern GPUs.

Furthermore, NVIDIA graphics cards now support D3D12 encoding mode, so obs-nvenc also needs some modifications.

How Has This Been Tested?

• Video, Image, Color, Duplicator, Game, Media source is teseted.
• Recording is tested.
• Color Range and Space is tested
• NVIDIA's encoder is tested.
• The WinRT Capture section reuses the renderer's device code; however, Microsoft's WinRT Capture may not currently support capture strategies for D3D12 devices. I also consulted some resources but couldn't find a WinRT D3D12 capture implementation. Therefore, this is implemented using a shared texture between D3D11 and D3D12.
• Duplicator Capture not supported by D3D12 Device, this is implemented using a shared texture between D3D11 and D3D12.
  you can use D3D12 by:

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[FerrariHD]

Hi, please dont give up, this feature sounds awesome for modern GPUs! cannot wait to see it in release version

[missingnoXXVI]

Was also too interested for my own good in trying this out and spent a frustrating amount of time getting set up only to get stuck with a NVENC exceeding height/width error that I don't get with my 5060 on the latest OBS Studio's NVENC encoder. Guess I will just have to hope that Wan shows up to continue at a later point.

[wanhongqing123]

Currently, there is a memory leak issue in the main process of NVIDIA's encoding module, and I am working on fixing it.

[missingnoXXVI]

Currently, there is a memory leak issue in the main process of NVIDIA's encoding module, and I am working on fixing it.

👍Glad to see you're still interested then. Was just a but of a mood killer to hit the streaming button only to stuck with the error. For as much as there are additions that I am looking forward to adding to my ease-of-streaming repertoire from the upcoming build of OBS Studio, this has my interest enough that I am fully prepared to stick with it for however long until merged (depending on the stability.)

[wanhongqing123]

This PR can now be used to experience D3D12 features.

[missingnoXXVI]

This PR can now be used to experience D3D12 features.

Haven't tried the latest pr action with the code cleaning, but 2nd to latest I was able to record with the NVENC HEVC D3D12 Encoder quite well for the better part of an hour test. Only thing is now the H.264 D3D12 doesn't show up — unless I am missing that it is currently not included — and it still errors out when I try with the default H.264.

From my log.
12:48:48.397: [obs-nvenc: 'advanced_video_stream'] get_tex_from_handle: OpenSharedResource failed: 0x80070057
12:48:48.397: Error encoding with encoder 'advanced_video_stream'
12:48:48.397: [rtmp stream: 'adv_stream'] Encoder error, disconnecting
12:48:48.397: [rtmp stream: 'adv_stream'] Socket send buffer is 65536 bytes

Still don't have this issue with latest release of OBS Studio or Release Candidate in portable mode. Also can't tell if it is because the newest release I tested fixed it or if recording before helps workaround the problem, but initially when I would try and test stream, after the error out it would also act like it was attempting to record as well but hang and then OBS Studio itself would hang when I would try and exit, forcing me to close it out with Task Manager.

[wanhongqing123]

This PR can now be used to experience D3D12 features.

Haven't tried the latest pr action with the code cleaning, but 2nd to latest I was able to record with the NVENC HEVC D3D12 Encoder quite well for the better part of an hour test. Only thing is now the H.264 D3D12 doesn't show up — unless I am missing that it is currently not included — and it still errors out when I try with the default H.264.

From my log. 12:48:48.397: [obs-nvenc: 'advanced_video_stream'] get_tex_from_handle: OpenSharedResource failed: 0x80070057 12:48:48.397: Error encoding with encoder 'advanced_video_stream' 12:48:48.397: [rtmp stream: 'adv_stream'] Encoder error, disconnecting 12:48:48.397: [rtmp stream: 'adv_stream'] Socket send buffer is 65536 bytes

Still don't have this issue with latest release of OBS Studio or Release Candidate in portable mode. Also can't tell if it is because the newest release I tested fixed it or if recording before helps workaround the problem, but initially when I would try and test stream, after the error out it would also act like it was attempting to record as well but hang and then OBS Studio itself would hang when I would try and exit, forcing me to close it out with Task Manager.

I'm fixing this issue, which is caused by your current rendering device being D3D12, but your encoding device is using D3D11.

[wanhongqing123]

NVIDIA NVENC H264, NVIDIA NVENC HEVC, and NVIDIA NVENC AV1 all currently support D3D12 and D3D11 device encoding. If the rendering device currently being used is a D3D11 device, then D3D11 encoding will be created when encoding is started; otherwise, D3D12 will be used.

[missingnoXXVI]

NVIDIA NVENC H264, NVIDIA NVENC HEVC, and NVIDIA NVENC AV1 all currently support D3D12 and D3D11 device encoding. If the rendering device currently being used is a D3D11 device, then D3D11 encoding will be created when encoding is started; otherwise, D3D12 will be used.

So, if I am reading you correctly, I shouldn't concern myself with distinct D3D12 encoders no longer loading up because renderer usage now should dictate the type of encoding used (D3D11/D3D12) by default?

Either way, I did a quick run through of streaming across Twitch and YouTube (so I could use NVENC AV1, NVENC HEVC, as well as NVENC H.264 again) with the latest attempt and was finally able to stably do so. I intend to do a much lengthier stream and then recording after later but performance seemed great. No hangs or crashes. Switched between Hybrid MP4 and Hybrid MOV to be able to record with the available NVENC selection and that was all 👍👍 as well switching between CBR and CQP. And general performance seems more like making what's already been good for me since I got my 5060 into "even better" is the best way I can describe it.

[wanhongqing123]

NVIDIA NVENC H264, NVIDIA NVENC HEVC, and NVIDIA NVENC AV1 all currently support D3D12 and D3D11 device encoding. If the rendering device currently being used is a D3D11 device, then D3D11 encoding will be created when encoding is started; otherwise, D3D12 will be used.

So, if I am reading you correctly, I shouldn't concern myself with distinct D3D12 encoders no longer loading up because renderer usage now should dictate the type of encoding used (D3D11/D3D12) by default?

Yes, you are right.

[wanhongqing123]

The development of the D3D12 rendering module has been completed. All modules have been tested on my machine and are now ready for code review.

[missingnoXXVI]

The development of the D3D12 rendering module has been completed. All modules have been tested on my machine and are now ready for code review.

No one else has commented on their own testing but I just finished using the last pr action build the same way as I did before and it will be my go to until officially merged or otherwise. Decided to try out the Multi RTMP plugin with it as well and it was smooth sailing.

[Chiitoo]

From a quick look-test-see, still builds and runs on Linux at least. :]

[FerrariHD]

What the chances to see it in Q2 in prod?

[Fenrirthviti]

What the chances to see it in Q2 in prod?

Zero.

[Chiitoo]

What the chances to see it in Q2 in prod?

Seems like something that will take a byte more review than possible to be done for that to happen perhaps.

Nothing stopping from building from source and applying the changes as a patch though!

[missingnoXXVI]

What the chances to see it in Q2 in prod?

Seems like something that will take a byte more review than possible to be done for that to happen perhaps.

Nothing stopping from building from source and applying the changes as a patch though!

I have no problem with the principle of making sure every last nook and cranny is taken care of over a period of time before public consumption.. but it's bad optics and harmfully stubborn to curmudgeonly hold on to the old if a genuine and stable-enough upgrade is within grasp. But, like I said for myself, this will be my go to until officially merged or otherwise. I don't have a reason not to.

[Fenrirthviti]

... but it's bad optics and harmfully stubborn to curmudgeonly hold on to the old if a genuine and stable-enough upgrade is within grasp....

This is not how software development works, and definitely not in a project as large and as complex as OBS.

First, this code hasn't even been reviewed. "It runs on my machine" is a far cry from a proper code review. Previous submissions of this change were not up to the standards of the project, took an approach that was unacceptable, and this was resubmitted as a draft PR despite our requests not to do so.

Second, a handful of users who have gone through limited test cases is a far cry from a fully tested and working "stable-enough" to be merged in to the project. This has nothing to do with being stubborn, this has everything to do with our commitment to providing a stable, well-tested, reliable application for the millions of users who use us every day.

As it stands, this PR seems to be taking a similar approach that we were not comfortable with in the first place, but we won't know until we have time to do a proper review.

Experts in rendering and D3D12 are welcome to provide their thoughts and comments on why this change is "better" than the current D3D11 implementation. Right now, the goal of this PR seems to be simply making OBS run on D3D12, rather than actually taking advantage of tangible benefits from doing so.

Past that, we will review this when the project has time, but asking us to blindly merge code that hasn't been properly vetted, reviewed, and tested, is not really acceptable.

[wanhongqing123]

We are currently developing on Vulkun in OBS. OBS will offer users three options: D3D12, Metal, and Vulkun. For users who are more determined to use D3D11 and OpenGL, these traditional renderers will also be available for them to choose from.

In addition, current commits are all within a complete libobs-d3d12 folder, where all the code is concentrated. Furthermore, if any issues arise during use, a complete switch to D3D11 is possible. I will continue using D3D12 until the official release, and I believe it will perform better on my RTX 5080 device.

240 FPS test results is
CPU time during D3D12 rendering:
render_video: min=0.002 ms, median=0.127 ms, max=10.017 ms, 99th percentile=0.771 ms

CPU time during D3D11 rendering:
render_video: min=0.001 ms, median=0.132 ms, max=12.904 ms, 99th percentile=0.936 ms

[wanhongqing123]

@DmitriyFrogo Please see this PR, You can now experience D3D12.

[DmitriyFrogo]

@DmitriyFrogo Please see this PR, You can now experience D3D12.

Hi! Initially, my problem was with VRR and MPO since Win11 24H2 release
Seems like, my problem was finally fixed by Microsoft in 25H2 build 26200.7840, and I can't reproduce my problem even on a DirectX 11 renderer
But I still wanted to test DirectX 12 renderer, and I encountered problem in one game today (Dungeons of DUSK demo from Steam Next Fest)
As I see, it's built with web technologies and uses Edge WebView (msedgewebview2.exe), and when I capturing window (Game Capture not working), I see color issues
For comparison 2 screenshots

DX11 OBS Renderer:

DX12 OBS Renderer:

[wanhongqing123]

@DmitriyFrogo Please see this PR, You can now experience D3D12.

Hi! Initially, my problem was with VRR and MPO since Win11 24H2 release Seems like, my problem was finally fixed by Microsoft in 25H2 build 26200.7840, and I can't reproduce my problem even on a DirectX 11 renderer But I still wanted to test DirectX 12 renderer, and I encountered problem in one game today (Dungeons of DUSK demo from Steam Next Fest) As I see, it's built with web technologies and uses Edge WebView (msedgewebview2.exe), and when I capturing window (Game Capture not working), I see color issues For comparison 2 screenshots

DX11 OBS Renderer:

DX12 OBS Renderer:

Could you please provide me with a copy of the OBS DX11 and DX12 logs? Based on current results, DX12 displays exhibit color loss.

[DmitriyFrogo]

Could you please provide me with a copy of the OBS DX11 and DX12 logs? Based on current results, DX12 displays exhibit color loss.

Can't see any serious differences in logs

DX11.txt
DX12.txt

[wanhongqing123]

Could you please provide me with a copy of the OBS DX11 and DX12 logs? Based on current results, DX12 displays exhibit color loss.

Can't see any serious differences in logs

DX11.txt DX12.txt

Did you use this game for testing?

Maybe SDR issue,I am trying to fix this problem.
Force SDR unchecked:

Force SDR checked:

[wanhongqing123]

Could you please provide me with a copy of the OBS DX11 and DX12 logs? Based on current results, DX12 displays exhibit color loss.

Can't see any serious differences in logs

DX11.txt DX12.txt

This issue has been fixed.