# Lab: Configure Applications for Reliability Deploy and troubleshoot a reliable application that defines health probes, compute resource requests, and compute resource limits so it can run N instances per node; and configure a horizontal pod autoscaler that will scale to a maximum of N instances. **Outcomes** You should be able to add resource requests to a `Deployment` object, configure probes, and create a horizontal pod autoscaler resource. As the `student` user on the `workstation` machine, use the `lab` command to prepare your system for this exercise. This command ensures that all resources are available for this exercise. It also creates the `reliability-review` project and deploys the `longload` application in that project. ```plaintext [student@workstation ~]$ lab start reliability-review ``` **Procedure 6.6. Instructions** The API URL of your OpenShift cluster is [https://api.ocp4.example.com:6443](https://api.ocp4.example.com:6443), and the `oc` command is already installed on your `workstation` machine. Log in to the OpenShift cluster as the `developer` user with the `developer` password. Use the `reliability-review` project for your work. 1. The `longload` application in the `reliability-review` project fails to start. Diagnose and then fix the issue. The application needs 512 MiB of memory to work. After you fix the issue, you can confirm that the application works by running the `~/DO180/labs/reliability-review/curl_loop.sh` script that the `lab` command prepared. The script sends requests to the application in a loop. For each request, the script displays the pod name and the application status. Press **Ctrl**+**C** to quit the script. 1. Log in to the OpenShift cluster. ```plaintext [student@workstation ~]$ oc login -u developer -p developer \ https://api.ocp4.example.com:6443 Login successful. ...output omitted... ``` 2. Set the `reliability-review` project as the active project. ```plaintext [student@workstation ~]$ oc project reliability-review ...output omitted... ``` 3. List the pods in the project. The pod is in the `Pending` status. The name of the pod on your system probably differs. ```plaintext [student@workstation ~]$ oc get pods NAME READY STATUS RESTARTS AGE longload-64bf8dd776-b6rkz 0/1 Pending 0 8m1s ``` 4. Retrieve the events for the pod. No compute node has enough memory to accommodate the pod. ```plaintext [student@workstation ~]$ oc describe pod longload-64bf8dd776-b6rkz Name: longload-64bf8dd776-b6rkz Namespace: reliability-review ...output omitted... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 8m default-scheduler 0/1 nodes are available: 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. ``` 5. Review the resource requests for memory. The `longload` deployment requests 8 GiB of memory. ```plaintext [student@workstation ~]$ oc get deployment longload -o \ jsonpath='{.spec.template.spec.containers[0].resources.requests.memory}{"\n"}' 8Gi ``` 6. Set the memory requests to 512 MiB. Ignore the warning message. ```plaintext [student@workstation ~]$ oc set resources deployment/longload \ --requests memory=512Mi Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... deployment.apps/longload resource requirements updated ``` 7. Wait for the pod to start. You might have to rerun the command several times for the pod to report a `Running` status. The name of the pod on your system probably differs. ```plaintext [student@workstation ~]$ oc get pods NAME READY STATUS RESTARTS AGE longload-5897c9558f-cx4gt 1/1 Running 0 86s ``` 8. Run the `~/DO180/labs/reliability-review/curl_loop.sh` script to confirm that the application works. ```plaintext [student@workstation ~]$ ~/DO180/labs/reliability-review/curl_loop.sh 1 curl: (7) Failed to connect to master01.ocp4.example.com port 30372: Connection refused 2 longload-5897c9558f-cx4gt: app is still starting 3 longload-5897c9558f-cx4gt: app is still starting 4 longload-5897c9558f-cx4gt: app is still starting 5 longload-5897c9558f-cx4gt: Ok 6 longload-5897c9558f-cx4gt: Ok 7 longload-5897c9558f-cx4gt: Ok 8 longload-5897c9558f-cx4gt: Ok ...output omitted... ``` Press **Ctrl**+**C** to quit the script. [Hide Solution](https://rol.redhat.com/rol/app/#) 2. When the application scales up, your customers complain that some requests fail. To replicate the issue, manually scale up the `longload` application to three replicas, and run the `~/DO180/labs/reliability-review/curl_loop.sh` script at the same time. The application takes seven seconds to initialize. The application exposes the `/health` API endpoint on HTTP port 3000. Configure the `longload` deployment to use this endpoint, to ensure that the application is ready before serving client requests. 1. Open a new terminal window and run the `~/DO180/labs/reliability-review/curl_loop.sh` script. ```plaintext [student@workstation ~]$ ~/DO180/labs/reliability-review/curl_loop.sh 1 longload-5897c9558f-cx4gt: Ok 2 longload-5897c9558f-cx4gt: Ok 3 longload-5897c9558f-cx4gt: Ok 4 longload-5897c9558f-cx4gt: Ok ...output omitted... ``` Leave the script running and do not interrupt it. 2. Scale up the application to three replicas. ```plaintext [student@workstation ~]$ oc scale deployment/longload --replicas 3 deployment.apps/longload scaled ``` 3. Watch the output of the `curl_loop.sh` script in the second terminal. Some requests fail because OpenShift sends requests to the new pods before the application is ready. ```plaintext ...output omitted... 22 longload-5897c9558f-cx4gt: Ok 23 longload-5897c9558f-cx4gt: Ok 24 longload-5897c9558f-cx4gt: Ok 25 curl: (7) Failed to connect to master01.ocp4.example.com port 30372: Connection refused 26 curl: (7) Failed to connect to master01.ocp4.example.com port 30372: Connection refused 27 longload-5897c9558f-cx4gt: Ok 28 curl: (7) Failed to connect to master01.ocp4.example.com port 30372: Connection refused 29 longload-5897c9558f-cx4gt: Ok 30 curl: (7) Failed to connect to master01.ocp4.example.com port 30372: Connection refused 31 longload-5897c9558f-tpssf: app is still starting 32 longload-5897c9558f-kkvm5: app is still starting 33 longload-5897c9558f-cx4gt: Ok 34 longload-5897c9558f-tpssf: app is still starting 35 longload-5897c9558f-tpssf: app is still starting 36 longload-5897c9558f-tpssf: app is still starting 37 longload-5897c9558f-cx4gt: Ok 38 longload-5897c9558f-tpssf: app is still starting 39 longload-5897c9558f-cx4gt: Ok 40 longload-5897c9558f-cx4gt: Ok ...output omitted... ``` Leave the script running and do not interrupt it. 4. Add a readiness probe to the `longload` deployment. Ignore the warning message. ```plaintext [student@workstation ~]$ oc set probe deployment/longload --readiness \ --initial-delay-seconds 7 \ --get-url http://:3000/health Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... deployment.apps/longload probes updated ``` 5. Scale down the application back to one pod. ```plaintext [student@workstation ~]$ oc scale deployment/longload --replicas 1 deployment.apps/longload scaled ``` 6. To test your work, scale up the application to three replicas again. ```plaintext [student@workstation ~]$ oc scale deployment/longload --replicas 3 deployment.apps/longload scaled ``` 7. Watch the output of the `curl_loop.sh` script in the second terminal. No request fails. ```plaintext ...output omitted... 92 longload-7ddcc9b7fd-72dtm: Ok 93 longload-7ddcc9b7fd-72dtm: Ok 94 longload-7ddcc9b7fd-72dtm: Ok 95 longload-7ddcc9b7fd-qln95: Ok 96 longload-7ddcc9b7fd-wrxrb: Ok 97 longload-7ddcc9b7fd-qln95: Ok 98 longload-7ddcc9b7fd-wrxrb: Ok 99 longload-7ddcc9b7fd-72dtm: Ok ...output omitted... ``` Press **Ctrl**+**C** to quit the script. [Hide Solution](https://rol.redhat.com/rol/app/#) 3. Configure the application so that it automatically scales up when the average memory usage is above 60% of the memory requests value, and scales down when the usage is below this percentage. The minimum number of replicas must be one, and the maximum must be three. The resource that you create for scaling the application must be named `longload`. The `lab` command provides the `~/DO180/labs/reliability-review/hpa.yml` resource file as an example. Use the `oc explain` command to learn the valid parameters for the [`hpa.spec.metrics.resource.target`](http://hpa.spec.metrics.resource.target) attribute. Because the file is incomplete, you must update it first if you choose to use it. To test your work, use the `~/DO180/labs/reliability-review/allocate.sh` script that the `lab` command prepared. This script sends an HTTP request to the application `/leak` API endpoint. Each request consumes an additional 480 MiB of memory. To free this memory, you can use the `~/DO180/labs/reliability-review/free.sh` script. 1. Before you create the horizontal pod autoscaler resource, scale down the application to one pod. ```plaintext [student@workstation ~]$ oc scale deployment/longload --replicas 1 deployment.apps/longload scaled ``` 2. Edit the `~/DO180/labs/reliability-review/hpa.yml` resource file. You can retrieve the parameters for the `resource` attribute by using the `oc explain hpa.spec.metrics.resource` and `oc explain` [`hpa.spec.metrics.resource.target`](http://hpa.spec.metrics.resource.target) commands. ```plaintext apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: longload labels: app: longload spec: maxReplicas: 3 minReplicas: 1 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: longload metrics: - type: Resource resource: name: memory target: type: Utilization averageUtilization: 60 ``` 3. Use the `oc apply` command to deploy the horizontal pod autoscaler. ```plaintext [student@workstation ~]$ oc apply -f ~/DO180/labs/reliability-review/hpa.yml horizontalpodautoscaler.autoscaling/longload created ``` 4. In the second terminal, run the `watch` command to monitor the `oc get hpa longload` command. Wait for the `longload` horizontal pod autoscaler to report usage in the `TARGETS` column. The percentage on your system probably differs. ```plaintext [student@workstation ~]$ watch oc get hpa longload Every 2.0s: oc get hpa longload workstation: Fri Mar 10 05:15:34 2023 NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE longload Deployment/longload 13%/60% 1 3 1 75s ``` Leave the command running and do not interrupt it. 5. To test your work, run the `~/DO180/labs/reliability-review/allocate.sh` script in the first terminal for the application to allocate 480 MiB of memory. ```plaintext [student@workstation ~]$ ~/DO180/labs/reliability-review/allocate.sh longload-7ddcc9b7fd-72dtm: consuming memory! ``` 6. In the second terminal, after two minutes, the `oc get hpa longload` command shows the memory increase. The horizontal pod autoscaler scales up the application to more than one replica. The percentage on your system probably differs. ```plaintext Every 2.0s: oc get hpa longload workstation: Fri Mar 10 05:19:44 2023 NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE longload Deployment/longload 145%/60% 1 3 2 5m18s ``` Press **Ctrl**+**C** to quit the `watch` command. Close that second terminal when done. [Hide Solution](https://rol.redhat.com/rol/app/#) **Evaluation** As the `student` user on the `workstation` machine, use the `lab` command to grade your work. Correct any reported failures and rerun the command until successful. ```plaintext [student@workstation ~]$ lab grade reliability-review ``` **Finish** As the `student` user on the `workstation` machine, use the `lab` command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises. ```plaintext [student@workstation ~]$ lab finish reliability-review ```