# Lab: Troubleshoot and Scale Applications Navigate the OpenShift web console to identify CPU-consuming workloads. Troubleshoot and fix a failed MySQL pod. Manually scale an application. Configure health probes. **Outcomes** You should be able to troubleshoot malfunctioning workloads, configure deployments, and scale applications. As the `student` user on the `workstation` machine, use the `lab` command to prepare your system for this exercise. This command ensures that all resources are available for this exercise. The command also creates the `compreview-scale` project and deploys some applications in that project. The command creates the `/home/student/DO180/labs/compreview-scale/resources.txt` file. The `resources.txt` file contains the URLs of your OpenShift cluster and the name of the images that you use during the exercise. You can use the file to copy and paste these URLs and image names. ```plaintext [student@workstation ~]$ lab start compreview-scale ``` **Specifications** The API URL of your OpenShift cluster is [https://api.ocp4.example.com:6443](https://api.ocp4.example.com:6443), and the `oc` command is already installed on your `workstation` machine. The URL of the OpenShift web console is [https://console-openshift-console.apps.ocp4.example.com](https://console-openshift-console.apps.ocp4.example.com). When you access the web console, select **Red Hat Identity Management** as the authentication mechanism. Log in to the OpenShift cluster as the `developer` user with the `developer` password. The password for the `admin` user is `redhatocp`. Perform the following tasks to complete the comprehensive review: * A pod in the cluster is consuming excessive CPU and is interfering with other tasks. Identify the pod and remove its workload. * The `compreview-scale` project already includes a web application at [http://frontend-compreview-scale.apps.ocp4.example.com](http://frontend-compreview-scale.apps.ocp4.example.com). When you access this URL, the application returns a list of quotations from famous authors. The application is broken for now, and is missing some configuration to be ready for production. The application uses two Kubernetes `Deployment` objects. The `frontend` deployment provides the application web pages, and relies on the `quotesdb` deployment that runs a MySQL database. The `lab` command already created the services and routes that connect the application components and that make the application available from outside the cluster. Fix the application and make it ready for production: * The `quotesdb` deployment in the `compreview-scale` project starts a MySQL server, but the database is failing. Review the logs of the pod to identify and then fix the issue. Use the following parameters for the database: | Name | Value | | --- | --- | | Username | `operator1` | | Password | `redhat123` | | Database name | `quotes` | * You security team validated a new version of the MySQL container image that fixes a security issue. The new container image is [`registry.ocp4.example.com:8443/rhel9/mysql-80:1-237`](http://registry.ocp4.example.com:8443/rhel9/mysql-80:1-237). Update the `quotesdb` deployment to use this image. Ensure that the database redeploys. The classroom setup copied the image from the Red Hat Ecosystem Catalog. The original image is [`registry.redhat.io/rhel9/mysql-80:1-237`](http://registry.redhat.io/rhel9/mysql-80:1-237). * Add a probe to the `quotesdb` deployment so that OpenShift can detect when the database is ready to accept requests. Use the `mysqladmin ping` command for the probe. * Add a second probe that regularly verifies the status of the database. Use the `mysqladmin ping` command as well. * Configure CPU and memory usage for the `quotesdb` deployment. The deployment needs 200 millicores of CPU and 256 MiB of memory to run, and you must restrict its CPU usage to 500 millicores and its memory usage to 1 GiB. * Add a probe to the `frontend` deployment so that OpenShift can detect when the web application is ready to accept requests. The application is ready when an HTTP request on port 8000 to the `/status` path is successful. * Add a second probe that regularly verifies the status of the web front end. The front end works as expected when an HTTP request on port 8000 to the `/env` path is successful. * Configure CPU and memory usage for the `frontend` deployment. The deployment needs 200 millicores of CPU and 256 MiB of memory to run, and you must restrict its CPU usage to 500 millicores and its memory usage to 512 MiB. * Scale the `frontend` application to three pods to accommodate for the estimated production load. * To verify your work, access the [http://frontend-compreview-scale.apps.ocp4.example.com](http://frontend-compreview-scale.apps.ocp4.example.com) URL. The application returns a list of quotations from famous authors. 1. Use the OpenShift web console to identify and then delete the pod that consumes excessive CPU. 1. Use a web browser to access the [https://console-openshift-console.apps.ocp4.example.com](https://console-openshift-console.apps.ocp4.example.com) URL. 2. Select **Red Hat Identity Management**, and then log in as the `admin` user with the `redhatocp` password. Click **Skip tour** if the **Welcome to the Developer Perspective** message is displayed. 3. Switch to the **Administrator** perspective and then navigate to **Observe** → **Dashboards**. ![](https://rol.redhat.com/rol/static/static_file_cache/do180-4.12/compreview/scale/assets/dashboards.png align="center") 4. Select the **Kubernetes / Compute Resources / Cluster** dashboard, and then click **Inspect** in the **CPU Usage** graph. ![](https://rol.redhat.com/rol/static/static_file_cache/do180-4.12/compreview/scale/assets/cpu.png align="center") 5. Set the zoom to five minutes and then hover over the graph. Notice that the interface lists the `compreview-scale-load` namespace in the first position, which indicated that this namespace is the first CPU consumer. ![](https://rol.redhat.com/rol/static/static_file_cache/do180-4.12/compreview/scale/assets/consumer.png align="center") 6. Navigate to **Observe** → **Dashboards** and then select the **Kubernetes / Compute Resources / Namespace (Workloads)** dashboard. Select the `compreview-scale-load` namespace and then set the time range to the last five minutes. The `computeprime` deployment is the workload that consumes excessive CPU. ![](https://rol.redhat.com/rol/static/static_file_cache/do180-4.12/compreview/scale/assets/workload.png align="center") 7. Navigate to **Workloads** → **Deployments** and then select the `compreview-scale-load` project. Select the menu for the `computeprime` deployment and then click **Delete Deployment**. Click **Delete** to confirm the operation. ![](https://rol.redhat.com/rol/static/static_file_cache/do180-4.12/compreview/scale/assets/delete.png align="center") 2. Review the logs of the pod that is failing for the `quotesdb` deployment. Set the missing environment variables in the `quotesdb` deployment. 1. Log in to the OpenShift cluster from the command line. ```plaintext [student@workstation ~]$ oc login -u developer -p developer \ https://api.ocp4.example.com:6443 Login successful. ...output omitted... ``` 2. Set the `compreview-scale` project as the active project. ```plaintext [student@workstation ~]$ oc project compreview-scale ...output omitted... ``` 3. List the pods to identify the failing pod from the `quotesdb` deployment. The names of the pods on your system probably differ. ```plaintext [student@workstation ~]$ oc get pods NAME READY STATUS RESTARTS AGE frontend-5fb85b4c75-5s7xr 0/1 CrashLoopBackOff 14 (2m52s ago) 50m quotesdb-9b9776479-4z4g9 0/1 CrashLoopBackOff 14 (3m4s ago) 50m ``` 4. Retrieve the logs for the failing pod. Some environment variables are missing. ```plaintext [student@workstation ~]$ oc logs quotesdb-9b9776479-4z4g9 => sourcing 20-validate-variables.sh ... You must either specify the following environment variables: MYSQL_USER (regex: '^[a-zA-Z0-9_]+$') MYSQL_PASSWORD (regex: '[a-zA-Z0-9_~!@#$%&*()-=<>,.?;:|]+$') MYSQL_DATABASE (regex: '^[a-zA-Z0-9_]+$') ...output omitted... ``` 5. Add the missing environment variables to the `quotesdb` deployment. Ignore the warning message. ```plaintext [student@workstation ~]$ oc set env deployment/quotesdb \ MYSQL_USER=operator1 MYSQL_PASSWORD=redhat123 MYSQL_DATABASE=quotes Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... deployment.apps/quotesdb updated ``` 3. Update the MySQL container image for the `quotesdb` deployment. 1. Retrieve the name of the container that is running inside the pod. You need the container name to update its image. ```plaintext [student@workstation ~]$ oc get deployment/quotesdb -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS ... quotesdb 1/1 1 1 59m mysql-80 ... ``` 2. Set the image to [`registry.ocp4.example.com:8443/rhel9/mysql-80:1-237`](http://registry.ocp4.example.com:8443/rhel9/mysql-80:1-237). Ignore the warning message. ```plaintext [student@workstation ~]$ oc set image deployment/quotesdb \ mysql-80=registry.ocp4.example.com:8443/rhel9/mysql-80:1-237 Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... deployment.apps/quotesdb image updated ``` 3. Verify your work. ```plaintext [student@workstation ~]$ oc get deployment/quotesdb -o wide NAME ... CONTAINERS IMAGES quotesdb ... mysql-80 registry.ocp4.example.com:8443/rhel9/mysql-80:1-237 ``` 4. Wait for the deployment to roll out. You might have to rerun the command several times for the pod to report a `Running` status. The name of the pod on your system probably differs. ```plaintext [student@workstation ~]$ oc get pods NAME READY STATUS RESTARTS AGE frontend-5fb85b4c75-5s7xr 0/1 CrashLoopBackOff 15 (3m39s ago) 56m quotesdb-54d64749c4-chhq6 1/1 Running 0 106s ``` 4. Add a readiness and a liveness probe to the `quotesdb` deployment that runs the `mysqladmin ping` command. 1. Use the `oc set probe` command with the `--readiness` option to add the readiness probe. Ignore the warning message. ```plaintext [student@workstation ~]$ oc set probe deployment/quotesdb \ --readiness -- mysqladmin ping Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... deployment.apps/quotesdb probes updated ``` 2. Use the `oc set probe` command with the `--liveness` option to add the liveness probe. Ignore the warning message. ```plaintext [student@workstation ~]$ oc set probe deployment/quotesdb \ --liveness -- mysqladmin ping Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... deployment.apps/quotesdb probes updated ``` 5. Define resource limits for the `quotesdb` deployment. Set the CPU request to 200 millicores and the memory request to 256 MiB. Set the CPU limit to 500 millicores and the memory limit to 1 GiB. Ignore the warning message. ```plaintext [student@workstation ~]$ oc set resources deployment/quotesdb \ --requests cpu=200m,memory=256Mi --limits cpu=500m,memory=1Gi Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... deployment.apps/quotesdb resource requirements updated ``` 6. Add a readiness and a liveness probe to the `frontend` deployment. 1. Use the `oc set probe` command with the `--readiness` option to add the readiness probe that tests the `/status` path on HTTP port 8000. Ignore the warning message. ```plaintext [student@workstation ~]$ oc set probe deployment/frontend --readiness \ --get-url http://:8000/status Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... deployment.apps/frontend probes updated ``` 2. Use the `oc set probe` command with the `--liveness` option to add the liveness probe that tests the `/env` path on HTTP port 8000. Ignore the warning message. ```plaintext [student@workstation ~]$ oc set probe deployment/frontend --liveness \ --get-url http://:8000/env Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... deployment.apps/frontend probes updated ``` 7. Define resource limits for the `frontend` deployment. Set the CPU request to 200 millicores and the memory request to 256 MiB. Set the CPU limit to 500 millicores and the memory limit to 512 MiB. Ignore the warning message. ```plaintext [student@workstation ~]$ oc set resources deployment/frontend \ --requests cpu=200m,memory=256Mi --limits cpu=500m,memory=512Mi Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... deployment.apps/frontend resource requirements updated ``` 8. Scale the `frontend` deployment to three pods. 1. Scale the deployment. ```plaintext [student@workstation ~]$ oc scale deployment/frontend --replicas 3 deployment.apps/frontend scaled ``` 2. Wait for the deployment to scale up. You might have to rerun the command several times for the pods to report a `Running` status. The names of the pods on your system probably differ. ```plaintext [student@workstation ~]$ oc get pods NAME READY STATUS RESTARTS AGE frontend-86cdd7c7bf-8vrrs 1/1 Running 0 3m10s frontend-86cdd7c7bf-ds79w 1/1 Running 0 44s frontend-86cdd7c7bf-hpnwz 1/1 Running 0 44s quotesdb-66ff98b88c-fhwhs 1/1 Running 0 12m ``` 9. Verify that the application responds to web requests. 1. Retrieve the URL of the application. ```plaintext [student@workstation ~]$ oc get route NAME HOST/PORT PATH SERVICES ... frontend frontend-compreview-scale.apps.ocp4.example.com frontend ... ``` 2. Use the `curl` command to test the application. ```plaintext [student@workstation ~]$ curl \ http://frontend-compreview-scale.apps.ocp4.example.com Quotes

Quote List