Container Service for Kubernetes:Configure and manage the ack-nvidia-device-plugin component

View the NVIDIA Device Plugin version

kubectl get pods -n kube-system -l component=nvidia-device-plugin \
  -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"\t"}{.spec.nodeName}{"\n"}{end}' \
  | awk -F'[:/]' '{split($NF, a, "-"); print a[1] "\t" $0}' \
  | sort -k1,1V \
  | cut -f2- \
  | awk -F'\t' '{split($1, img, ":"); print img[NF] "\t" $2}'

1. Upgrade the ack-nvidia-device-plugin component.

   For cluster versions 1.32 and later

   1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

   2. On the Clusters page, find the cluster you want and click its name. In the left-side pane, choose Operations > Add-ons.

   3. On the Component Management page, search for the ack-nvidia-device-plugin component and click Upgrade on its card.

   4. In the dialog box that appears, click OK.

   For cluster versions from 1.20 to 1.31

   1. On the Clusters page, click the name of the one you want to change. In the navigation pane on the left, choose Nodes > Nodes.

   2. Select the GPU nodes on which you want to perform batch operations. Below the node list, click Batch Operations. In the Batch Operations dialog box, select Execute Shell Command and click OK.

      Important

      First, select a small number of GPU nodes to upgrade. After you verify that the Device Plugin works as expected on these nodes, perform the operation on a larger batch of nodes.

   3. You are redirected to the OOS interface. Set Execution Mode to Pause On Failure, and then click Next: Set Parameters.

   4. On the Set Parameters page, select Run Shell Script and paste the following sample script.

      Note

      In the script, change the RUN_PKG_VERSION parameter to the major version of your cluster, such as 1.30. Do not enter a minor version, such as 1.30.1. Otherwise, the script fails.

      #!/bin/bash
      set -e
      
      RUN_PKG_VERSION=1.30
      
      function update_device_plugin() {
      	base_dir=/tmp/update_device_plugin
      	rm -rf $base_dir
      	mkdir -p $base_dir
      	cd $base_dir
      	region_id=$(curl -ssL 100.100.100.200/latest/meta-data/region-id  2> /dev/null || echo "")
      	if [[ $region_id == "" ]]; then
      		echo "Error: failed to get region id,region id is null"
      		exit 1
      	fi
      	PKG_URL=https://aliacs-k8s-${region_id}.oss-${region_id}.aliyuncs.com/public/pkg/run/run-${RUN_PKG_VERSION}.tar.gz
      	curl -sSL --retry 3 --retry-delay 2 -o run.tar.gz $PKG_URL
      	tar -xf run.tar.gz
      
      	local dir=pkg/run/$RUN_PKG_VERSION/module
      	sed -i "s@registry.cn-hangzhou.aliyuncs.com/acs@registry-${region_id}-vpc.ack.aliyuncs.com/acs@g" $dir/nvidia-device-plugin.yml
      	mkdir -p /etc/kubernetes/device-plugin-backup
      	mkdir -p /etc/kubernetes/manifests
      	mv  /etc/kubernetes/manifests/nvidia-device-plugin.yml /etc/kubernetes/device-plugin-backup/nvidia-device-plugin.yml.$(date +%s)
      	sleep 5
      	cp -a $dir/nvidia-device-plugin.yml /etc/kubernetes/manifests
      	echo "succeeded to update device plugin"
      }
      
      if [ -f /etc/kubernetes/manifests/nvidia-device-plugin.yml ]; then
      	update_device_plugin
      else
      	echo "skip to update device plugin"
      fi

   5. Click Next: Confirm. After you confirm the information, click Create.

      After the task is created, you are redirected to the Task Execution Management page where you can view the task status. If the Execution Output shows succeeded to update device plugin, the update was successful.

2. Check whether the component is running as expected.

   Run the following command to check whether the Device Plugin is working as expected on the GPU nodes.

   1. Connect to the cluster using kubectl in Workbench or CloudShell.

   2. Run the following command to check whether the NVIDIA Device Plugin has restarted:

      kubectl get po -n kube-system -l component=nvidia-device-plugin 

      Sample output:

      NAME                             READY   STATUS    RESTARTS      AGE
      nvidia-device-plugin-xxxx        1/1     Running   1             1m

   3. After all Pods are restarted, run the following script to check whether the nodes report GPU resources:

      #!/bin/bash
      
      # Get all eligible NVIDIA Device Plugin Pods and the nodes where they are located.
      PODS=$(kubectl get po -n kube-system -l component=nvidia-device-plugin -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeName}{"\n"}{end}')
      
      # Iterate over the node of each Pod.
      echo "$PODS" | while IFS=$'\t' read -r pod_name node_name; do
          # Get the value of nvidia.com/gpu resource allocation for the node.
          gpu_allocatable=$(kubectl get node "$node_name" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}' 2>/dev/null)
      
          # Check if the resource value is 0.
          if [ "$gpu_allocatable" == "0" ]; then
              echo "Error: node=$node_name, pod=$pod_name, resource(nvidia.com/gpu) is 0"
          fi
      done

      If a node reports 0 for the resource, see Restart the NVIDIA Device Plugin.