Syarif's Blog

Guided Exercise: Application High Availability with Kubernetes

Syarif's photo
Syarif
Jul 12, 2023·

6 min read

Simulate different types of application failures and observe how Kubernetes handles them.

Outcomes

  • Explore how the restartPolicy attribute affects crashing pods.

  • Observe the behavior of a slow-starting application that has no configured probes.

  • Use a deployment to scale the application, and observe the behavior of a broken pod.

As the student user on the workstation machine, use the lab command to prepare your system for this exercise.

This command ensures that the following conditions are true:

  • The reliability-ha project exists.

  • The resource files are available in the course directory.

  • The classroom registry has the long-load container image.

The long-load container image contains an application with utility endpoints. These endpoints perform such tasks as crashing the process and toggling the server's health status.

[student@workstation ~]$ lab start reliability-ha

Procedure 6.1. Instructions

  1. As the developer user, create a pod from a YAML manifest in the reliability-ha project.

    1. Log in as the developer user with the developer password.

       [student@workstation ~]$ oc login https://api.ocp4.example.com:6443 -u developer
 ...output omitted...

    2. Select the reliability-ha project.

       [student@workstation ~]$ oc project reliability-ha
 Now using project "reliability-ha" on server "https://api.ocp4.example.com:6443".

    3. Navigate to the lab materials directory and view the contents of the pod definition. In particular, restartPolicy is set to Always.

       [student@workstation ~]$ cd DO180/labs/reliability-ha

       [student@workstation reliability-ha]$ cat long-load.yaml
 apiVersion: v1
 kind: Pod
 metadata:
   name: long-load
 spec:
   containers:
   - image: registry.ocp4.example.com:8443/redhattraining/long-load:v1
     name: long-load
     securityContext:
       allowPrivilegeEscalation: false
   restartPolicy: Always

    4. Create a pod by using the oc apply command.

       [student@workstation reliability-ha]$ oc apply -f long-load.yaml
 pod/long-load created

    5. Send a request to the pod to confirm that it is running and responding.

       [student@workstation reliability-ha]$ oc exec long-load -- \
   curl -s localhost:3000/health
 Ok

  2. Trigger the pod to crash, and observe that the restartPolicy instructs the cluster to re-create the pod.

    1. Observe that the pod is running and has not restarted.

       [student@workstation reliability-ha]$ oc get pods

 NAME        READY   STATUS    RESTARTS   AGE
 long-load   1/1     Running   0          1m

    2. Send a request to the /destruct endpoint in the application. This request triggers the process to crash.

       [student@workstation reliability-ha]$ oc exec long-load -- \
   curl -s localhost:3000/destruct
 command terminated with exit code 52

    3. Observe that the pod is running and restarted one time.

       [student@workstation reliability-ha]$ oc get pods
 NAME        READY   STATUS    RESTARTS      AGE
 long-load   1/1     Running   1 (34s ago)   4m16s

    4. Delete the long-load pod.

       [student@workstation reliability-ha]$ oc delete pod long-load
 pod "long-load" deleted

      The pod is not re-created, because it was created manually, and not via a workload resource such as a deployment.

  3. Use a restart policy of Never to create the pod, and observe that it is not re-created on crashing.

    1. Modify the long-load.yaml file so that the restartPolicy is set to Never.

       ...output omitted...
   restartPolicy: Never

    2. Create the pod with the updated YAML file.

       [student@workstation reliability-ha]$ oc apply -f long-load.yaml
 pod/long-load created

    3. Send a request to the pod to confirm that the pod is running and that the application is responding.

       [student@workstation reliability-ha]$ oc exec long-load -- \
   curl -s localhost:3000/health
 Ok

    4. Send a request to the /destruct endpoint in the application to crash it.

       [student@workstation reliability-ha]$ oc exec long-load -- \
   curl -s localhost:3000/destruct
 command terminated with exit code 52

    5. Observe that the pod is not restarted and is in an error state.

       [student@workstation reliability-ha]$ oc get pods
 NAME        READY   STATUS   RESTARTS   AGE
 long-load   0/1     Error    0          2m36s

    6. Delete the long-load pod.

       [student@workstation reliability-ha]$ oc delete pod long-load
 pod "long-load" deleted

  4. Because the cluster does not know when the application inside the pod is ready to receive requests, you must add a startup delay to the application. Adding this capability by using probes is covered in a later exercise.

    1. Update the long-load.yaml file by adding a startup delay and use a restart policy of Always. Set the START_DELAY variable to 60,000 milliseconds (one minute) so that the file looks like the following excerpt:

       ...output omitted...
 spec:
   containers:
   - image: registry.ocp4.example.com:8443/redhattraining/long-load:v1
     imagePullPolicy: Always
     securityContext:
       allowPrivilegeEscalation: false
     name: long-load
     env:
       - name: START_DELAY
         value: "60000"
   restartPolicy: Always

      NOTE

      Although numbers are a valid YAML type, environment variables must be passed as strings. YAML syntax is also indentation-sensitive.

      For these reasons, ensure that your file appears exactly as the preceding example.

    2. Apply the YAML file to create the pod and proceed within one minute to the next step.

       [student@workstation reliability-ha]$ oc apply -f long-load.yaml
 pod/long-load created

    3. Within a minute of pod creation, verify the status of the pod. The status shows as ready even though it is not. Try to send a request to the application, and observe that it fails.

       [student@workstation reliability-ha]$ oc get pods
 NAME        READY   STATUS    RESTARTS   AGE
 long-load   1/1     Running   0          16s

       [student@workstation reliability-ha]$ oc exec long-load -- \
   curl -s localhost:3000/health
 app is still starting

    4. After waiting a minute for the application to start, send another a request to the pod to confirm that it is running and responding.

       [student@workstation reliability-ha]$ oc exec long-load -- \
   curl -s localhost:3000/health
 Ok

  5. Use a deployment to scale up the number of deployed pods. Observe that deleting the pods causes service outages, even though the deployment handles re-creating the pods.

    1. Review the long-load-deploy.yaml file, which defines a deployment, service, and route. The deployment creates three replicas of the application pod.

      In each pod, a START_DELAY environment variable is set to 15,000 milliseconds (15 seconds). In each pod, the application responds that it is not ready until after the delay.

       [student@workstation reliability-ha]$ cat long-load-deploy.yaml
 ...output omitted...
     spec:
       containers:
       - image: registry.ocp4.example.com:8443/redhattraining/long-load:v1
         imagePullPolicy: Always
         name: long-load
         env:
         - name: START_DELAY
           value: "15000"
 ...output omitted...

    2. Start the load test script, which sends a request to the /health API endpoint of the application every two seconds. Leave the script running in a visible terminal window.

       [student@workstation reliability-ha]$ ./load-test.sh
 ...output omitted...

    3. In a new terminal window, apply the ~/DO180/labs/reliability-ha/long-load-deploy.yaml file.

       [student@workstation reliability-ha]$ oc apply -f \
   ~/DO180/labs/reliability-ha/long-load-deploy.yaml
 deployment.apps/long-load created
 service/long-load created
 route.route.openshift.io/long-load created

    4. Watch the output of the load test script as the pods and the application instances start. After a delay, the requests succeed.

       ...output omitted...
 Ok
 Ok
 Ok
 ...output omitted...

    5. By using the /togglesick API endpoint of the application, put one of the three pods into a broken state.

       [student@workstation reliability-ha]$ curl \
   long-load-reliability-ha.apps.ocp4.example.com/togglesick
 no output expected

    6. Watch the output of the load test script as some requests start failing. Because of the load balancer, the exact order of the output is random.

       ...output omitted...
 Ok
 app is unhealthy
 app is unhealthy
 Ok
 Ok
 ...output omitted...

      Press Ctrl+C to end the load test script.

    7. Return to the /home/student/ directory.

       [student@workstation reliability-ha]$ cd /home/student/
 [student@workstation ~]$

Finish

On the workstation machine, use the lab command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises.

[student@workstation ~]$ lab finish reliability-ha
openshift