refactor: offload DP/DRA management to KMM

[abyrne55]

This PR replaces the operator's direct DP/DRA DaemonSet management with a single KMM (Kernel Module Management) Module CR. Two controllers collapse into one, and the operator no longer needs to build DaemonSets, manage RBAC, handle pod readiness, or construct OpenShift SCCs at runtime.

	Before	After
Controllers	2 (DevicePluginReconciler, DRAReconciler)	1 (KMMReconciler)
K8s objects managed at runtime	DaemonSets, ServiceAccounts, ClusterRoles, ClusterRoleBindings, DeviceClasses, AdmissionPolicies, SCCs	1 Module CR
Embedded YAML + loaders	9 files	0

Existing ClusterPolicy specs work unchanged — KMM is used transparently under the hood.

Beyond simplifying what we already have, KMM also brings capabilities we'd otherwise need to build: OOT kernel module loading (see new CRD field below), kernel-version-aware rollouts, ordered upgrades, pre-flight validation, and build/sign integration. Ordered upgrades and pre-flight validation apply immediately for OOT driver modules; build/sign integration is available when needed.

How it works

A new KMMReconciler sub-controller replaces DevicePluginReconciler and DRAReconciler in the reconcile loop. It uses controllerutil.CreateOrPatch to manage the Module CR idempotently, with SetControllerReference so the Module is garbage-collected when the ClusterPolicy is deleted.

DRA RBAC (ServiceAccount, ClusterRole, ClusterRoleBinding), OpenShift SCCs, and ValidatingAdmissionPolicy resources have been moved out of operator-managed code and into deployment tooling (Helm templates and kustomize overlays).

KMM availability is detected at startup via API group discovery (kmm.sigs.x-k8s.io), same pattern as DRAEnable. If the CRD isn't installed, the operator starts normally and reports an error in ClusterPolicy status.

New CRD field: kernelModule

spec:
  kernelModule:            # optional; omit for in-tree driver
    moduleName: "xe"       # kernel module to load
    image: "registry.example.com/xe-driver:1.0"  # default/fallback image

When kernelModule is set, KMM loads the specified OOT driver module on each node. When omitted, the in-tree kernel driver is used.

Multi-kernel support

kernelMappings maps kernel version patterns to container images, enabling multi-kernel clusters:

spec:
  kernelModule:
    moduleName: xe
    kernelMappings:
    - regexp: "^5\\.14\\.0-.*\\.el9.*\\.x86_64$"
      containerImage: registry.example.com/xe-rhel9:1.0
    - regexp: "^6\\.12\\..*"
      containerImage: registry.example.com/xe-rhel10:1.0

When kernelMappings is empty, a single wildcard mapping (^.+$) is generated from image. When kernelMappings is set, image serves as a KMM-level fallback for mappings that omit containerImage.

In-cluster builds

Each kernel mapping can include a build spec for cluster-time driver image building via KMM:

    kernelMappings:
    - regexp: "^5\\.14\\..*"
      build:
        dockerfileConfigMap:
          name: xe-dockerfile
        buildArgs:
        - name: XE_TAG
          value: v1.0
        secrets:
        - name: private-repo

Additional fields

Field	Description
inTreeModulesToRemove	In-tree modules to unload before OOT insertion. moduleName always included implicitly.
modulesLoadingOrder	Softdep-style loading order for multi-module drivers (first = moduleName, >=2 entries).
firmwarePath	In-container path for firmware files (maps to KMM ModprobeSpec.FirmwarePath).
skipTLSVerify	Disable TLS verification for OOT image registry (maps to KMM RegistryTLS).

KMM status

ClusterPolicyStatus.kmmStatus reports module loader availability (available/desired), visible via kubectl get clusterpolicy as a new KMM column. Shows N/A when kernelModule is nil (in-tree mode).

Mapping to KMM Module CR

KernelModuleSpec	KMM Module CR
ModuleName	ModprobeSpec.ModuleName
Image	ModuleLoaderContainerSpec.ContainerImage (fallback)
KernelMappings[]	ModuleLoaderContainerSpec.KernelMappings[]
InTreeModulesToRemove	ModuleLoaderContainerSpec.InTreeModulesToRemove
ModulesLoadingOrder	ModprobeSpec.ModulesLoadingOrder
FirmwarePath	ModprobeSpec.FirmwarePath
SkipTLSVerify	RegistryTLS.InsecureSkipTLSVerify

Aligned with tkatila's DriverXeSpec API sketch. Genericized Xe-specific fields (e.g., XeTag/XeSHA → BuildArgs). Skipped fields already covered elsewhere (NodeSelector, PullSecret, Enable) and Xe-specific enums (XeOSTarget). No Sign spec yet.

Known limitations

• In-cluster build untested. Build spec support is implemented and unit-tested but hasn't been validated end-to-end.

Test results

Tested on OCP 4.22.2 (SNO) with two Intel Arc Pro B70 GPUs. KMM and NFD pre-installed.

• DP mode with in-tree driver: Module CR created, KMM spun up DP DaemonSet, GPU devices registered, test pod scheduled and saw /dev/dri devices
• DRA mode with in-tree driver: Module CR created, KMM spun up DRA DaemonSet, DeviceClasses created, GPU devices visible in ResourceSlices, test pod with DRA ResourceClaim scheduled and saw /dev/dri devices
• DRA mode with OOT driver (simple moduleName + image): KMM loaded OOT xe module, DRA ResourceSlices populated, test pod saw GPUs
• XPU Manager runs alongside KMM-managed DRA
• ClusterPolicy deletion cleans up Module CR (DaemonSet garbage-collected by KMM)
• make test passes (all specs including expanded KMM controller tests)

This PR was written in part with the assistance of generative AI.

[tkatila]

Hi @abyrne55 ! Thanks for the PoC!

• K8s object creation (serviceaccount, SCCs, etc.) is moved from operator runtime to operator deployment time, what is the reasoning for this move?
  • While it can have security benefits (less access rights in the operator), it limits what the operator can do.
  • It also forces to use the "is openshift" variable in the chart.
  • https://operatorframework.io/operator-capabilities/ would suggest that operator should be capable of creating objects, but I'm not sure if there is general consensus on it.
• Was there a specific KMM version required for this to work? I just applied KMM via the kubectl oneliner from the repo. Seems to be "latest" image. I'm getting this error with DRA:

E0722 06:33:59.300935       1 clusterpolicy_controller.go:234] Return sub-controller errorfailed to reconcile KMM Module gpu-policy-gpu for ClusterPolicy gpu-policy: admission webhook "vmodule.kb.io" denied the request: failed to validate DRA volumes: spec.dra.volumes[0]: hostPath "/etc/cdi" is not allowed; only /dev, /sys, /var, /opt and /run paths are permitted

• And this with DP (in KMM):

E0722 06:31:42.060239       1 controller.go:347] "Reconciler error" err="could handle device plugin: DaemonSet.apps \"g
pu-policy-gpu-device-plugin-l9t68\" is invalid: spec.template.spec.containers[0].volumeMounts[3].mountPath: Invalid value: \"/var/lib/kubelet/device-plugins\": must be unique" logger="kmm" controller="DevicePluginReconciler" controllerGroup="kmm.sigs.x-k8s.io" controllerKind="Module" Module="intel-gpu-base-operator/gpu-policy-gpu" namespace="intel-gpu-base-operator" name="gpu-policy-gpu" reconcileID="57b85f12-1653-490f-b814-05f0da238e78"

• I see KMM has received a helm chart (Adding helm deployment kubernetes-sigs/kernel-module-management#1309). Is there a timeline for the first release with it?

I'm still thinking if I'd like to have DP/DRA deployment also in the base-operator side. For anyone not wanting to deploy KMM to their cluster. Both methods could use the same building blocks so there wouldn't be too much overhead. More testing though.

[abyrne55]

@tkatila thanks for taking a look!

My latest push should resolve the hostPath and "duplicate mount" bugs: the /etc/cdi volume was removed since KMM sets CDI_ROOT=/var/run/cdi, and the duplicate /var/lib/kubelet/device-plugins mount was dropped since KMM auto-adds it.

Re. the KMM version, I've been using "bleeding edge" builds from rh-ecosystem-edge/kernel-module-management:main while we wait for the next KMM y-stream release (release-2.7 branch), which fixes a couple DRA-related bugs that impeded my earlier testing. If you're only testing with the device plugin, then you should be okay using the latest stable KMM version.

On the K8s object creation move: the main motivation was reducing the operator's RBAC footprint. Agreed it's a tradeoff: happy to discuss where to draw the line.

Re. keeping a non-KMM path: let's discuss. I think that might negate a lot of the maintainability benefits of this PR.