# Guided Exercise: Application High Availability with Kubernetes Simulate different types of application failures and observe how Kubernetes handles them. **Outcomes** * Explore how the `restartPolicy` attribute affects crashing pods. * Observe the behavior of a slow-starting application that has no configured probes. * Use a deployment to scale the application, and observe the behavior of a broken pod. As the `student` user on the `workstation` machine, use the `lab` command to prepare your system for this exercise. This command ensures that the following conditions are true: * The `reliability-ha` project exists. * The resource files are available in the course directory. * The classroom registry has the `long-load` container image. The `long-load` container image contains an application with utility endpoints. These endpoints perform such tasks as crashing the process and toggling the server's health status. ```plaintext [student@workstation ~]$ lab start reliability-ha ``` **Procedure 6.1. Instructions** 1. As the `developer` user, create a pod from a YAML manifest in the `reliability-ha` project. 1. Log in as the `developer` user with the `developer` password. ```plaintext [student@workstation ~]$ oc login https://api.ocp4.example.com:6443 -u developer ...output omitted... ``` 2. Select the `reliability-ha` project. ```plaintext [student@workstation ~]$ oc project reliability-ha Now using project "reliability-ha" on server "https://api.ocp4.example.com:6443". ``` 3. Navigate to the lab materials directory and view the contents of the pod definition. In particular, `restartPolicy` is set to `Always`. ```plaintext [student@workstation ~]$ cd DO180/labs/reliability-ha ``` ```plaintext [student@workstation reliability-ha]$ cat long-load.yaml apiVersion: v1 kind: Pod metadata: name: long-load spec: containers: - image: registry.ocp4.example.com:8443/redhattraining/long-load:v1 name: long-load securityContext: allowPrivilegeEscalation: false restartPolicy: Always ``` 4. Create a pod by using the `oc apply` command. ```plaintext [student@workstation reliability-ha]$ oc apply -f long-load.yaml pod/long-load created ``` 5. Send a request to the pod to confirm that it is running and responding. ```plaintext [student@workstation reliability-ha]$ oc exec long-load -- \ curl -s localhost:3000/health Ok ``` 2. Trigger the pod to crash, and observe that the `restartPolicy` instructs the cluster to re-create the pod. 1. Observe that the pod is running and has not restarted. ```plaintext [student@workstation reliability-ha]$ oc get pods NAME READY STATUS RESTARTS AGE long-load 1/1 Running 0 1m ``` 2. Send a request to the `/destruct` endpoint in the application. This request triggers the process to crash. ```plaintext [student@workstation reliability-ha]$ oc exec long-load -- \ curl -s localhost:3000/destruct command terminated with exit code 52 ``` 3. Observe that the pod is running and restarted one time. ```plaintext [student@workstation reliability-ha]$ oc get pods NAME READY STATUS RESTARTS AGE long-load 1/1 Running 1 (34s ago) 4m16s ``` 4. Delete the `long-load` pod. ```plaintext [student@workstation reliability-ha]$ oc delete pod long-load pod "long-load" deleted ``` The pod is not re-created, because it was created manually, and not via a workload resource such as a deployment. 3. Use a restart policy of `Never` to create the pod, and observe that it is not re-created on crashing. 1. Modify the `long-load.yaml` file so that the `restartPolicy` is set to `Never`. ```plaintext ...output omitted... restartPolicy: Never ``` 2. Create the pod with the updated YAML file. ```plaintext [student@workstation reliability-ha]$ oc apply -f long-load.yaml pod/long-load created ``` 3. Send a request to the pod to confirm that the pod is running and that the application is responding. ```plaintext [student@workstation reliability-ha]$ oc exec long-load -- \ curl -s localhost:3000/health Ok ``` 4. Send a request to the `/destruct` endpoint in the application to crash it. ```plaintext [student@workstation reliability-ha]$ oc exec long-load -- \ curl -s localhost:3000/destruct command terminated with exit code 52 ``` 5. Observe that the pod is not restarted and is in an error state. ```plaintext [student@workstation reliability-ha]$ oc get pods NAME READY STATUS RESTARTS AGE long-load 0/1 Error 0 2m36s ``` 6. Delete the `long-load` pod. ```plaintext [student@workstation reliability-ha]$ oc delete pod long-load pod "long-load" deleted ``` 4. Because the cluster does not know when the application inside the pod is ready to receive requests, you must add a startup delay to the application. Adding this capability by using probes is covered in a later exercise. 1. Update the `long-load.yaml` file by adding a startup delay and use a restart policy of `Always`. Set the `START_DELAY` variable to 60,000 milliseconds (one minute) so that the file looks like the following excerpt: ```plaintext ...output omitted... spec: containers: - image: registry.ocp4.example.com:8443/redhattraining/long-load:v1 imagePullPolicy: Always securityContext: allowPrivilegeEscalation: false name: long-load env: - name: START_DELAY value: "60000" restartPolicy: Always ``` ### **NOTE** Although numbers are a valid YAML type, environment variables must be passed as strings. YAML syntax is also indentation-sensitive. For these reasons, ensure that your file appears *exactly* as the preceding example. 2. Apply the YAML file to create the pod and proceed within one minute to the next step. ```plaintext [student@workstation reliability-ha]$ oc apply -f long-load.yaml pod/long-load created ``` 3. Within a minute of pod creation, verify the status of the pod. The status shows as ready even though it is not. Try to send a request to the application, and observe that it fails. ```plaintext [student@workstation reliability-ha]$ oc get pods NAME READY STATUS RESTARTS AGE long-load 1/1 Running 0 16s ``` ```plaintext [student@workstation reliability-ha]$ oc exec long-load -- \ curl -s localhost:3000/health app is still starting ``` 4. After waiting a minute for the application to start, send another a request to the pod to confirm that it is running and responding. ```plaintext [student@workstation reliability-ha]$ oc exec long-load -- \ curl -s localhost:3000/health Ok ``` 5. Use a deployment to scale up the number of deployed pods. Observe that deleting the pods causes service outages, even though the deployment handles re-creating the pods. 1. Review the `long-load-deploy.yaml` file, which defines a deployment, service, and route. The deployment creates three replicas of the application pod. In each pod, a `START_DELAY` environment variable is set to 15,000 milliseconds (15 seconds). In each pod, the application responds that it is not ready until after the delay. ```plaintext [student@workstation reliability-ha]$ cat long-load-deploy.yaml ...output omitted... spec: containers: - image: registry.ocp4.example.com:8443/redhattraining/long-load:v1 imagePullPolicy: Always name: long-load env: - name: START_DELAY value: "15000" ...output omitted... ``` 2. Start the load test script, which sends a request to the `/health` API endpoint of the application every two seconds. Leave the script running in a visible terminal window. ```plaintext [student@workstation reliability-ha]$ ./load-test.sh ...output omitted... ``` 3. In a new terminal window, apply the `~/DO180/labs/reliability-ha/long-load-deploy.yaml` file. ```plaintext [student@workstation reliability-ha]$ oc apply -f \ ~/DO180/labs/reliability-ha/long-load-deploy.yaml deployment.apps/long-load created service/long-load created route.route.openshift.io/long-load created ``` 4. Watch the output of the load test script as the pods and the application instances start. After a delay, the requests succeed. ```plaintext ...output omitted... Ok Ok Ok ...output omitted... ``` 5. By using the `/togglesick` API endpoint of the application, put one of the three pods into a broken state. ```plaintext [student@workstation reliability-ha]$ curl \ long-load-reliability-ha.apps.ocp4.example.com/togglesick no output expected ``` 6. Watch the output of the load test script as some requests start failing. Because of the load balancer, the exact order of the output is random. ```plaintext ...output omitted... Ok app is unhealthy app is unhealthy Ok Ok ...output omitted... ``` Press **Ctrl**+**C** to end the load test script. 7. Return to the `/home/student/` directory. ```plaintext [student@workstation reliability-ha]$ cd /home/student/ [student@workstation ~]$ ``` **Finish** On the `workstation` machine, use the `lab` command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises. ```plaintext [student@workstation ~]$ lab finish reliability-ha ```