Skip to content

fix: prevent nvidia-drm unload hangs after GPU failure#1252

Open
hurryman2212 wants to merge 1 commit into
NVIDIA:mainfrom
hurryman2212:fix/nvidia-drm-unload-after-gpu-hang
Open

fix: prevent nvidia-drm unload hangs after GPU failure#1252
hurryman2212 wants to merge 1 commit into
NVIDIA:mainfrom
hurryman2212:fix/nvidia-drm-unload-after-gpu-hang

Conversation

@hurryman2212

Copy link
Copy Markdown

Problem:

After a GPU becomes unresponsive, unloading nvidia_drm still
performs an atomic display shutdown and releases NVKMS modeset
ownership. Releasing modeset ownership attempts to restore the
framebuffer console, submitting display methods to a GPU that can
no longer process them.

The failure was preceded by:

  [drm] [nvidia-drm] [GPU ID 0x00000300] Unloading driver
  nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:0:0:0x00000062
  nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:2:0:0x00000062
  nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:4:0:0x00000062
  nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:6:0:0x00000062
  nvidia-modeset: WARNING: GPU:0: Lost display notification (0:0x00000000); continuing.
  nvidia-modeset: WARNING: GPU:0: Lost display notification (0:0x00000000); continuing.

Console restoration then entered nvEvoMakeRoom(), which repeatedly
reported:

  nvidia-modeset: ERROR: GPU:0: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:200:192

Its five-second timeout only prints the error and restarts the
timer; it does not leave the wait loop.

The unload remained stuck in:

  nv_drm_dev_unload
    -> ReleaseOwnership
    -> RestoreConsole
    -> nvEvoRestoreConsole
    -> nvSetDispModeEvo / nvShutDownApiHeads
    -> nvEvoMakeRoom

The kernel reported:

  INFO: task nvidia-modeset/:2535 blocked for more than 120 seconds.
  task:nvidia-modeset/ state:D
  INFO: task nvidia-modeset/:2535 blocked on a semaphore likely last held by task modprobe:629175

The NVIDIA modules could no longer be unloaded and reloaded,
leaving a reboot as the only way to restore the driver stack.

Fix:

Query NV2080_CTRL_CMD_GPU_GET_RECOVERY_ACTION before nvidia_drm
teardown. When RM reports that recovery is required, stop new
event handling, cancel pending hotplug work, and skip atomic
display shutdown and framebuffer console restoration.

Mark NVKMS for recovery teardown before releasing modeset
ownership, then avoid display method submission and GPU progress
waits while continuing to release software resources and RM
handles.

Keep the existing teardown path when RM reports no recovery action
or does not support the recovery-action query.

Detect RM recovery state before nvidia-drm teardown and skip console restoration, display method submission, and GPU progress waits while releasing NVKMS software resources.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant