# Guided Exercise: Assess the Health of an OpenShift Cluster Verify the health of an OpenShift cluster by querying the status of its cluster operators, nodes, pods, and systemd services. Also verify cluster events and alerts. **Outcomes** * View the status and get information about cluster operators. * Retrieve information about cluster pods and nodes. * Retrieve the status of a node's systemd services. * View cluster events and alerts. * Retrieve debugging information for the cluster. As the `student` user on the `workstation` machine, use the `lab` command to prepare your system for this exercise. This command ensures that all resources are available for this exercise. ```plaintext [student@workstation ~]$ lab start cli-health ``` **Procedure 2.3. Instructions** 1. Retrieve the status and view information about cluster operators. 1. Log in to the OpenShift cluster as the `admin` user with the `redhatocp` password. ```plaintext [student@workstation ~]$ oc login -u admin -p redhatocp \ https://api.ocp4.example.com:6443 Login successful ...output omitted... ``` 2. List the operators that users installed in the OpenShift cluster. ```plaintext [student@workstation ~]$ oc get operators NAME AGE metallb-operator.metallb-system 27d odf-lvm-operator.openshift-storage 27d ``` 3. List the cluster operators that are installed by default in the OpenShift cluster. ```plaintext [student@workstation ~]$ oc get clusteroperators NAME VERSION AVAILABLE PROGRESSING DEGRADED ... authentication 4.12.0 True False False ... baremetal 4.12.0 True False False ... cloud-controller-manager 4.12.0 True False False ... cloud-credential 4.12.0 True False False ... cluster-autoscaler 4.12.0 True False False ... config-operator 4.12.0 True False False ... console 4.12.0 True False False ... control-plane-machine-set 4.12.0 True False False ... csi-snapshot-controller 4.12.0 True False False ... dns 4.12.0 True False False ... etcd 4.12.0 True False False ... ...output omitted... ``` 4. Use the `describe` command to view detailed information about the `openshift-apiserver` cluster operator, such as related objects, events, and version. ```plaintext [student@workstation ~]$ oc describe clusteroperators openshift-apiserver Name: openshift-apiserver Namespace: Labels: Annotations: exclude.release.openshift.io/internal-openshift-hosted: true include.release.openshift.io/self-managed-high-availability: true include.release.openshift.io/single-node-developer: true API Version: config.openshift.io/v1 Kind: ClusterOperator Metadata: ...output omitted... Spec: Status: Conditions: Last Transition Time: 2023-02-09T22:41:08Z Message: All is well Reason: AsExpected Status: False Type: Degraded ...output omitted... Extension: Related Objects: Group: operator.openshift.io Name: cluster Resource: openshiftapiservers Group: Name: openshift-config Resource: namespaces Group: Name: openshift-config-managed Resource: namespaces Group: Name: openshift-apiserver-operator Resource: namespaces Group: Name: openshift-apiserver Resource: namespaces ...output omitted... Versions: Name: operator Version: 4.12.0 Name: openshift-apiserver Version: 4.12.0 Events: ``` The `Related Objects` attribute includes information about the name, resource type, and groups for objects that are related to the operator. 5. List the pods in the `openshift-apiserver-operator` namespace. Then, view the detailed status of an `openshift-apiserver-operator` pod by using the JSON format and the `jq` command. Your pod names might differ. ```plaintext [student@workstation ~]$ oc get pods -n openshift-apiserver-operator NAME READY STATUS RESTARTS AGE openshift-apiserver-operator-7ddc8958fb-7m2kr 1/1 Running 11 27d ``` ```plaintext [student@workstation ~]$ oc get pod -n openshift-apiserver-operator \ openshift-apiserver-operator-7ddc8958fb-7m2kr \ -o json | jq .status { "conditions": [ ...output omitted... { "lastProbeTime": null, "lastTransitionTime": "2023-03-08T15:41:34Z", "status": "True", "type": "Ready" }, ...output omitted... ], "containerStatuses": [ { ...output omitted... "name": "openshift-apiserver-operator", "ready": true, "restartCount": 11, "started": true, "state": { "running": { "startedAt": "2023-03-08T15:41:34Z" } } } ], "hostIP": "192.168.50.10", "phase": "Running", "podIP": "10.8.0.5", ...output omitted... } ``` 2. Retrieve the status, resource consumption, and events of cluster pods. 1. List the memory and CPU usage of all pods in the cluster. Use the `--sum` option to print the sum of the resource usage. The resource usage on your system probably differs. ```plaintext [student@workstation ~]$ oc adm top pods -A --sum NAMESPACE NAME CPU(cores) MEMORY(bytes)metallb-system controller-5f6dfd8c4f-ddr8v 0m 39Mi metallb-system metallb-operator-controller-manager-... 1m 38Mi metallb-system metallb-operator-webhook-server-... 1m 18Mi metallb-system speaker-2dds4 10m 94Mi ...output omitted... 505m 8982Mi ``` 2. List the pods and their labels in the `openshift-etcd` namespace. ```plaintext [student@workstation ~]$ oc get pods -n openshift-etcd --show-labels NAME READY STATUS RESTARTS AGE LABELS etcd-master01 4/4 Running 40 27d app=etcd,etcd=true,k8s-app=etcd,revision=3 installer-2-master01 0/1 Completed 0 27d app=installer installer-3-master01 0/1 Completed 0 27d app=installer ``` 3. List the resource usage of the containers in the `etcd-master01` pod in the `openshift-etcd` namespace. The resource usage on your system probably differs. ```plaintext [student@workstation ~]$ oc adm top pods etcd-master01 \ -n openshift-etcd --containers POD NAME CPU(cores) MEMORY(bytes) etcd-master01 POD 0m 0Mi etcd-master01 etcd 57m 1096Mi etcd-master01 etcd-metrics 7m 20Mi etcd-master01 etcd-readyz 4m 40Mi etcd-master01 etcdctl 0m 0Mi ``` 4. Display a list of all resources, their status, and their types in the `openshift-monitoring` namespace. ```plaintext [student@workstation ~]$ oc get all -n openshift-monitoring --show-kind NAME READY STATUS ... pod/alertmanager-main-0 6/6 Running ... pod/cluster-monitoring-operator-56b769b58f-dtmqj 2/2 Running ... pod/kube-state-metrics-75455b796c-8q28d 3/3 Running ... ...output omitted... NAME TYPE CLUSTER-IP ... service/alertmanager-main ClusterIP 172.30.85.183 ... service/alertmanager-operated ClusterIP None ... service/cluster-monitoring-operator ClusterIP None ... service/kube-state-metrics ClusterIP None ... ...output omitted... ``` 5. View the logs of the `alertmanager-main-0` pod in the `openshift-monitoring` namespace. The logs might differ on your system. ```plaintext [student@workstation ~]$ oc logs alertmanager-main-0 -n openshift-monitoring ...output omitted... ts=2023-03-09T14:57:11.850Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml ts=2023-03-09T14:57:11.850Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml ``` 6. Retrieve the events for the `openshift-image-registry` namespace. ```plaintext [student@workstation ~]$ oc get events -n openshift-image-registry LAST SEEN TYPE REASON OBJECT MESSAGE 42m Normal Scheduled pod/image-pruner-27972000-dg8qt Successfully assigned openshift-image-registry/image-pruner-27972000-dg8qt to master01 42m Normal AddedInterface pod/image-pruner-27972000-dg8qt Add eth0 [10.8.0.96/23] from ovn-kubernetes 42m Normal Pulled pod/image-pruner-27972000-dg8qt Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1fc4...685b" already present on machine 42m Normal Created pod/image-pruner-27972000-dg8qt Created container image-pruner ...output omitted... ``` 3. Retrieve information about cluster nodes. 1. View the status of the nodes in the cluster. ```plaintext [student@workstation ~]$ oc get nodes NAME STATUS ROLES AGE VERSION master01 Ready control-plane,master,worker 27d v1.25.4+77bec7a ``` 2. Retrieve the resource consumption of the `master01` node. The resource usage on your system probably differs. ```plaintext [student@workstation ~]$ oc adm top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master01 781m 10% 11455Mi 60% ``` 3. Use a JSONPath filter to determine the capacity and allocatable CPU for the `master01` node. The values might differ on your system. ```plaintext [student@workstation ~]$ oc get node master01 -o jsonpath=\ 'Allocatable: {.status.allocatable.cpu}{"\n"}'\ 'Capacity: {.status.capacity.cpu}{"\n"}' Allocatable: 7500m Capacity: 8 ``` 4. Determine the number of allocatable pods for the node. ```plaintext [student@workstation ~]$ oc get node master01 -o jsonpath=\ '{.status.allocatable.pods}{"\n"}' 250 ``` 5. Use the `describe` command to view the events, resource requests, and resource limits for the node. The output might differ on your system. ```plaintext [student@workstation ~]$ oc describe node master01 ...output omitted... Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 3158m (42%) 980m (13%) memory 12667Mi (66%) 1250Mi (6%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Starting 106m kubelet Starting kubelet. Normal NodeHasSufficientMemory 106m (x9 over 106m) kubelet Node master01 status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 106m (x7 over 106m) kubelet Node master01 status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 106m (x7 over 106m) kubelet Node master01 status is now: NodeHasSufficientPID ...output omitted... ``` 4. Retrieve the logs and status of the systemd services on the `master01` node. 1. Display the logs of the node. Filter the logs to show the most recent log for the `crio` service. The logs might differ on your system. ```plaintext [student@workstation ~]$ oc adm node-logs master01 -u crio --tail 1 -- Logs begin at Thu 2023-02-09 21:19:09 UTC, end at Thu 2023-03-09 16:57:00 UTC. -- Mar 09 02:39:29.158989 master01 crio[3201]: time="2023-03-09 02:39:29.158737393Z" level=info msg="Image status: &ImageStatusResponse ...output omitted... ``` 2. Display the two most recent logs of the `kubelet` service on the node. The logs might differ on your system. ```plaintext [student@workstation ~]$ oc adm node-logs master01 -u kubelet --tail 2 -- Logs begin at Thu 2023-02-09 21:19:09 UTC, end at Thu 2023-03-09 16:59:16 UTC. -- Mar 09 02:40:57.466711 master01 systemd[1]: Stopped Kubernetes Kubelet. Mar 09 02:40:57.466835 master01 systemd[1]: kubelet.service: Consumed 1h 27min 8.069s CPU time -- Logs begin at Thu 2023-02-09 21:19:09 UTC, end at Thu 2023-03-09 16:59:16 UTC. -- Mar 09 16:58:52.133046 master01 kubenswrapper[3195]: I0309 16:58:52.132866 3195 kubelet_getters.go:182] "Pod status updated" pod="openshift-etcd/etcd-master01" status=Running Mar 09 16:58:52.133046 master01 kubenswrapper[3195]: I0309 16:58:52.132882 3195 kubelet_getters.go:182] "Pod status updated" pod="openshift-kube-apiserver/kube-apiserver-master01" status=Running ``` 3. Create a debug session for the node. The, use the `chroot /host` command to access the host binaries. ```plaintext [student@workstation ~]$ oc debug node/master01 Temporary namespace openshift-debug-kzz4c is created for debugging node... Starting pod/master01-debug ... To use host binaries, run `chroot /host` Pod IP: 192.168.50.10 If you don't see a command prompt, try pressing enter. sh-4.4# chroot /host sh-4.4# ``` 4. Verify the status of the `kubelet` service. ```plaintext sh-4.4# systemctl status kubelet ● kubelet.service - Kubernetes Kubelet Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/kubelet.service.d └─01-kubens.conf, 10-mco-default-madv.conf, 20-logging.conf, 20-nodenet.conf Active: active (running) since Thu 2023-03-09 14:54:51 UTC; 2h 8min ago Main PID: 3195 (kubelet) Tasks: 28 (limit: 127707) Memory: 540.7M CPU: 18min 32.117s ...output omitted... ``` Press **Ctrl**+**C** to quit the command. 5. Confirm that the `crio` service is active. ```plaintext sh-4.4# systemctl is-active crio active ``` 6. Exit the debug pod. ```plaintext sh-4.4# exit exit sh-4.4# exit exit Removing debug pod ... Temporary namespace openshift-debug-kzz4c was removed. ``` 5. Retrieve debugging information for the cluster. 1. Retrieve debugging information of the cluster by using the `oc adm must-gather` command. Specify the `/home/student/must-gather` directory as the destination directory. This command might take several minutes to complete. Then, confirm that the debugging information exists in the destination directory. ```plaintext [student@workstation ~]$ oc adm must-gather --dest-dir /home/student/must-gather [must-gather ] OUT Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:07d3...e94c ...output omitted... Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 94ff22c1-88a0-44cf-90f6-0b7b8b545434 ClusterVersion: Stable at "4.12.0" ClusterOperators: All healthy and stable [student@workstation ~]$ ls -la ~/must-gather/ total 688 drwxrwxr-x. 3 student student 174 Mar 9 12:23 . drwx------. 24 student student 4096 Mar 9 12:22 .. -rw-r--r--. 1 student student 691751 Mar 9 12:23 event-filter.html drwxrwxrwx. 13 student student 4096 Mar 9 12:23 quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-07d3...e94c -rw-r--r--. 1 student student 111 Mar 9 12:23 timestamp ``` 2. Generate debugging information for the `openshift-api` cluster operator. Specify the `/home/student/inspect` directory as the destination directory. Limit the debugging information to the last five minutes. Then, confirm that the debugging information exists in the destination directory. ```plaintext [student@workstation ~]$ oc adm inspect clusteroperator/kube-apiserver \ --dest-dir /home/student/inspect --since 5m Gathering data for ns/openshift-config... Gathering data for ns/openshift-config-managed... Gathering data for ns/openshift-kube-apiserver-operator... Gathering data for ns/openshift-kube-apiserver... Gathering data for ns/metallb-system... Gathering data for ns/openshift-monitoring... Gathering data for ns/openshift-machine-api... Gathering data for ns/openshift-multus... Gathering data for ns/openshift-cluster-node-tuning-operator... Gathering data for ns/openshift-cluster-storage-operator... Wrote inspect data to /home/student/inspect. [student@workstation ~]$ ls -la inspect/ total 208 drwxrwxr-x. 4 student student 98 Mar 9 12:33 . drwx------. 25 student student 4096 Mar 9 12:33 .. drwxrwxr-x. 8 student student 185 Mar 9 12:33 cluster-scoped-resources -rw-r--r--. 1 student student 198069 Mar 9 12:33 event-filter.html drwxrwxr-x. 12 student student 4096 Mar 9 12:33 namespaces -rw-r--r--. 1 student student 110 Mar 9 12:33 timestamp ``` 3. Delete the debugging information from your system. ```plaintext [student@workstation ~]$ rm -rf must-gather inspect ``` **Finish** On the `workstation` machine, use the `lab` command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises. ```plaintext [student@workstation ~]$ lab finish cli-health ```