Fix DLSS on Windows: CUDA interop warm-up and NGX application identity #6

Open
sajittarius wants to merge 2 commits from sajittarius/the-beast-roofer-edition:dlss-cuda-warmup-and-ngx-identity into master
First-time contributor

What this fixes

DLSS was not working in Windows with an RTX 4090
(original was created on another machine with RTX 5090).

Two separate faults stacked on top of each other,
and each one hid the next.

Fault 1 — vkCreateDevice refused two extensions. VK_NVX_binary_import
and VK_NV_optical_flow were rejected with ErrorInitializationFailed, which
crashed the emulator at device creation whenever DLSS was enabled.

Fault 2 — NGX refused the application identity. Once the extensions were
accepted, DLSS still reported ErrorFeatureNotSupported.

Root cause 1: CUDA init ordering

Guest process creation inside Device.LoadNsp reserves the guest address space.
After that reservation exists, cuInit returns CUDA_ERROR_OUT_OF_MEMORY for
the rest of the process lifetime. Both refused extensions are backed by separate
NVIDIA runtime components (nvcuda.dll and nvofapi64.dll), so the driver
refuses them once CUDA cannot initialize. The four other DLSS extensions have no
such backing component, which is why they were always accepted.

CUDA state that initializes before the reservation stays valid for the whole
session.

Fix: the VulkanRenderer constructor runs before game load. When DLSS is
enabled, it creates and destroys a throwaway device requesting
VK_NVX_binary_import, forcing CUDA to initialize first.

Bisection that located this (probe at four points in the process lifetime):

Checkpoint VK_NVX_binary_import alone
Main start OK
After Initialize OK
After new Switch(...) OK
After Device.LoadNsp ErrorInitializationFailed

Root cause 2: no application identity at slInit

Streamline.Initialize set no ApplicationId and no ProjectId. Streamline's
own log named the problem directly: Please provide correct application id when calling slInit - NGX based features will be disabled.

Fix: pass the Streamline sample ApplicationId plus a ProjectId GUID and
EngineVersion, which selects NGX's project-id init path.

Also in this change

  • Staged device-creation fallback. Instead of crashing on
    ErrorInitializationFailed, CreateDevice retries without optical flow, then
    probes each DLSS device extension on its own and keeps the ones the driver
    accepts. This runs only in the failure path, so it costs nothing on a healthy
    driver. It is the safety net if a future driver refuses an extension again.
  • GPU selection. The fallback path now prefers a discrete GPU rather than
    the first compatible device in enumeration order, and logs why a preferred
    device was skipped. On a machine with both a discrete and an integrated GPU,
    the old code could silently pick the integrated one with no log line.
  • VkDiagProbe (new file): the raw P/Invoke helper the warm-up uses.

Verification

Tested on an RTX 4090, NVIDIA driver 616.56, Windows 11, plain launch with no
environment variables set:

  • All 30 device extensions accepted at stage 1. No fallback needed.
  • Dedicated optical-flow queue acquired (family 5).
  • DLSS: available. — first time on record for this build.
  • In gameplay (Super Mario Bros. Wonder): SetOptions preset=11
    (transformer / DLSS4), TryRun: using mode Dlaa, real depth capture.

Also verified after the diagnostic cleanup in the second commit, to confirm the
strip-down did not regress the fix.

What was ruled out along the way

Recorded here so nobody repeats the search:

  • Vulkan layers — three runs with VK_LOADER_LAYERS_DISABLE covering
    Rockstar Social Club, EOS Overlay, Steam Overlay, Steam Fossilize and NV
    Optimus all failed identically. ReShade's layer was already off.
  • Driver version — the refusals were identical on 610.88 and on a clean
    616.56 install.
  • Instance API version — raising the instance from Vulkan 1.2 to 1.3 changed
    nothing; probe results were byte-for-byte identical.
  • GPU selection — the RTX 4090 was always correctly chosen. The integrated
    GPU was never involved.

Scope

Four files, +406 / -24. Two commits:

  • 45d51ccf3 — the fix
  • b882b46e5 — removes the investigation's diagnostic checkpoints

All changes sit behind the existing DLSS enable check, so behaviour with DLSS
off is unchanged.

## What this fixes DLSS was not working in Windows with an RTX 4090 (original was created on another machine with RTX 5090). Two separate faults stacked on top of each other, and each one hid the next. **Fault 1 — `vkCreateDevice` refused two extensions.** `VK_NVX_binary_import` and `VK_NV_optical_flow` were rejected with `ErrorInitializationFailed`, which crashed the emulator at device creation whenever DLSS was enabled. **Fault 2 — NGX refused the application identity.** Once the extensions were accepted, DLSS still reported `ErrorFeatureNotSupported`. ## Root cause 1: CUDA init ordering Guest process creation inside `Device.LoadNsp` reserves the guest address space. After that reservation exists, `cuInit` returns `CUDA_ERROR_OUT_OF_MEMORY` for the rest of the process lifetime. Both refused extensions are backed by separate NVIDIA runtime components (`nvcuda.dll` and `nvofapi64.dll`), so the driver refuses them once CUDA cannot initialize. The four other DLSS extensions have no such backing component, which is why they were always accepted. CUDA state that initializes *before* the reservation stays valid for the whole session. **Fix:** the `VulkanRenderer` constructor runs before game load. When DLSS is enabled, it creates and destroys a throwaway device requesting `VK_NVX_binary_import`, forcing CUDA to initialize first. Bisection that located this (probe at four points in the process lifetime): | Checkpoint | `VK_NVX_binary_import` alone | |---|---| | Main start | OK | | After `Initialize` | OK | | After `new Switch(...)` | OK | | After `Device.LoadNsp` | **ErrorInitializationFailed** | ## Root cause 2: no application identity at `slInit` `Streamline.Initialize` set no `ApplicationId` and no `ProjectId`. Streamline's own log named the problem directly: `Please provide correct application id when calling slInit - NGX based features will be disabled`. **Fix:** pass the Streamline sample `ApplicationId` plus a `ProjectId` GUID and `EngineVersion`, which selects NGX's project-id init path. ## Also in this change - **Staged device-creation fallback.** Instead of crashing on `ErrorInitializationFailed`, `CreateDevice` retries without optical flow, then probes each DLSS device extension on its own and keeps the ones the driver accepts. This runs only in the failure path, so it costs nothing on a healthy driver. It is the safety net if a future driver refuses an extension again. - **GPU selection.** The fallback path now prefers a discrete GPU rather than the first compatible device in enumeration order, and logs why a preferred device was skipped. On a machine with both a discrete and an integrated GPU, the old code could silently pick the integrated one with no log line. - **`VkDiagProbe`** (new file): the raw P/Invoke helper the warm-up uses. ## Verification Tested on an RTX 4090, NVIDIA driver 616.56, Windows 11, plain launch with no environment variables set: - All 30 device extensions accepted at stage 1. No fallback needed. - Dedicated optical-flow queue acquired (family 5). - `DLSS: available.` — first time on record for this build. - In gameplay (Super Mario Bros. Wonder): `SetOptions preset=11` (transformer / DLSS4), `TryRun: using mode Dlaa`, real depth capture. Also verified after the diagnostic cleanup in the second commit, to confirm the strip-down did not regress the fix. ## What was ruled out along the way Recorded here so nobody repeats the search: - **Vulkan layers** — three runs with `VK_LOADER_LAYERS_DISABLE` covering Rockstar Social Club, EOS Overlay, Steam Overlay, Steam Fossilize and NV Optimus all failed identically. ReShade's layer was already off. - **Driver version** — the refusals were identical on 610.88 and on a clean 616.56 install. - **Instance API version** — raising the instance from Vulkan 1.2 to 1.3 changed nothing; probe results were byte-for-byte identical. - **GPU selection** — the RTX 4090 was always correctly chosen. The integrated GPU was never involved. ## Scope Four files, +406 / -24. Two commits: - `45d51ccf3` — the fix - `b882b46e5` — removes the investigation's diagnostic checkpoints All changes sit behind the existing DLSS enable check, so behaviour with DLSS off is unchanged.
sajittarius added 2 commits 2026-08-30 15:15:15 +00:00
DLSS never worked on Windows: vkCreateDevice refused VK_NVX_binary_import
and VK_NV_optical_flow with ErrorInitializationFailed, and NGX refused to
initialize even when the extensions were present.

Root cause 1: once guest process creation (Device.LoadNsp) reserves the
guest address space, cuInit fails with CUDA_ERROR_OUT_OF_MEMORY for the
rest of the process lifetime, so the driver refuses the two CUDA/OFA-
backed device extensions. CUDA state initialized before the reservation
stays valid all session. Fix: the VulkanRenderer constructor (which runs
before game load) creates and destroys a throwaway device requesting
VK_NVX_binary_import when DLSS is enabled, forcing CUDA init first.

Root cause 2: Streamline.Initialize passed no application identity, and
NGX disables all NGX-based features without one ("Please provide correct
application id when calling slInit" in sl.log). Fix: pass the Streamline
sample ApplicationId plus a ProjectId GUID and EngineVersion, selecting
the NGX project-id init path.

Also in this change:

- Staged device-creation fallback: retry without optical flow, then
  probe each DLSS device extension individually and keep the accepted
  ones, instead of crashing on ErrorInitializationFailed.
- GPU selection: prefer a discrete GPU in the fallback path and log why
  a preferred device was skipped.
- VkDiagProbe: raw P/Invoke probe used for the warm-up, plus lifecycle
  diagnostic checkpoints gated on RYUJINX_VK_DIAG (off by default),
  appending to vkdiag.txt next to the exe.

Verified end to end on an RTX 4090 (driver 616.56): all 30 device
extensions accepted, dedicated optical-flow queue acquired, DLSS
reports available, and DLSS4 transformer upscaling runs in-game.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drops the RYUJINX_VK_DIAG lifecycle probes (Program.cs, AppHost.cs,
VulkanInitialization.cs) and the failure-time feature-bisect loop in
CreateDevice; they existed only to locate the CUDA init failure. Strips
VkDiagProbe down to the WarmUpCudaInterop path that the fix uses.

The staged device-creation fallback and per-extension probe stay: they
are the runtime safety net when a driver refuses an extension.

Verified after cleanup: warm-up OK, optical-flow queue acquired, DLSS
available on a plain launch with no env vars.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
You are not authorized to merge this pull request.
This pull request can be merged automatically.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u https://thebeastroofer-edition.com/sajittarius/the-beast-roofer-edition dlss-cuda-warmup-and-ngx-identity:sajittarius-dlss-cuda-warmup-and-ngx-identity
git checkout sajittarius-dlss-cuda-warmup-and-ngx-identity
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rooferdev/the-beast-roofer-edition#6