# Guided Exercise: Kubernetes Pod and Service Networks

Deploy a database server and access it through a Kubernetes service.

**Outcomes**

You should be able to deploy a database server, and access it indirectly through a Kubernetes service, and also directly pod-to-pod for troubleshooting.

As the `student` user on the `workstation` machine, use the `lab` command to prepare your system for this exercise.

This command ensures that all resources are available for this exercise. It also creates the deploy-services project and the `/home/student/DO180/labs/deploy-services/resources.txt` file. The `resources.txt` file contains some of the commands that you use during the exercise. You can use the file to copy and paste these commands.

```plaintext
[student@workstation ~]$ lab start deploy-services
```

### **NOTE**

It is safe to ignore pod security warnings for exercises in this course. OpenShift uses the Security Context Constraints controller to provide safe defaults for pod security.

**Procedure 4.3. Instructions**

1. Log in to the OpenShift cluster as the `developer` user with the `developer` password. Use the `deploy-services` project.
    
    1. Log in to the OpenShift cluster.
        
        ```plaintext
        [student@workstation ~]$ oc login -u developer -p developer \
          https://api.ocp4.example.com:6443
        Login successful.
        ...output omitted...
        ```
        
    2. Set the `deploy-services` project as the active project.
        
        ```plaintext
        [student@workstation ~]$ oc project deploy-services
        ...output omitted...
        ```
        
2. Use the [`registry.ocp4.example.com:8443/rhel8/mysql-80`](http://registry.ocp4.example.com:8443/rhel8/mysql-80) container image to create a MySQL deployment named `db-pod`. Add the missing environment variables for the pod to run.
    
    1. Create the `db-pod` deployment.
        
        ```plaintext
        [student@workstation ~]$ oc create deployment db-pod --port 3306 \
          --image registry.ocp4.example.com:8443/rhel8/mysql-80
        Warning: would violate PodSecurity "restricted:v1.24":
        ...output omitted...
        deployment.apps/db-pod created
        ```
        
    2. Add the environment variables.
        
        ```plaintext
        [student@workstation ~]$ oc set env deployment/db-pod \
          MYSQL_USER=user1 \
          MYSQL_PASSWORD=mypa55w0rd \
          MYSQL_DATABASE=items
        Warning: would violate PodSecurity "restricted:v1.24":
        ...output omitted...
        deployment.apps/db-pod updated
        ```
        
    3. Confirm that the pod is running.
        
        ```plaintext
        [student@workstation ~]$ oc get pods
        NAME                      READY   STATUS    RESTARTS   AGE
        db-pod-6ccc485cfc-vrc4r   1/1     Running   0          2m30s
        ```
        
        Your pod name might differ from the previous output.
        
3. Expose the `db-pod` deployment to create a ClusterIP service.
    
    1. View the deployment for the pod.
        
        ```plaintext
        [student@workstation ~]$ oc get deployment
        NAME     READY   UP-TO-DATE   AVAILABLE   AGE
        db-pod   1/1     1            1           3m36s
        ```
        
    2. Expose the `db-pod` deployment to create a service.
        
        ```plaintext
        [student@workstation ~]$ oc expose deployment/db-pod
        service/db-pod exposed
        ```
        
4. Validate the service. Confirm that the service selector matches the label on the pod. Then, confirm that the `db-pod` service endpoint matches the IP of the pod.
    
    1. Identify the selector for the `db-pod` service. Use the `oc get service command` with the `-o wide` option to retrieve the selector that the service uses.
        
        ```plaintext
        [student@workstation ~]$ oc get service db-pod -o wide
        NAME     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE    SELECTOR
        db-pod   ClusterIP   172.30.108.92   <none>        3306/TCP   108s   app=db-pod
        ```
        
        The selector shows an `app=db-pod` key:value pair.
        
    2. Capture the name of the pod in a variable.
        
        ```plaintext
        [student@workstation ~]$ PODNAME=$(oc get pods \
          -o jsonpath='{.items[0].metadata.name}')
        ```
        
    3. Query the label on the pod.
        
        ```plaintext
        [student@workstation ~]$ oc get pod $PODNAME --show-labels
        NAME                     READY   STATUS    RESTARTS   AGE     LABELS
        db-pod-6ccc485cfc-vrc4r  1/1     Running   0          6m50s   app=db-pod ...
        ```
        
        Notice that the label list includes the `app=db-pod` key-value pair, which is the selector for the `db-pod` service.
        
    4. Retrieve the endpoints for the `db-pod` service.
        
        ```plaintext
        [student@workstation ~]$ oc get endpoints
        NAME     ENDPOINTS        AGE
        db-pod   10.8.0.85:3306   4m38s
        ```
        
        Your endpoints values might differ from the previous output.
        
    5. Verify that the service endpoint matches the `db-pod` IP address. Use the `oc get pods` command with the `-o wide` option to view the pod IP address.
        
        ```plaintext
        [student@workstation ~]$ oc get pods -o wide
        NAME                      READY   STATUS    RESTARTS   AGE   IP        ...
        db-pod-6ccc485cfc-vrc4r   1/1     Running   0          54m   10.8.0.85 ...
        ```
        
        The service endpoint resolves to the IP address that is assigned to the pod.
        
5. Delete and then re-create the `db-pod` deployment. Confirm that the `db-pod` service endpoint automatically resolves to the IP address of the new pod.
    
    1. Delete the `db-pod` deployment.
        
        ```plaintext
        [student@workstation ~]$ oc delete deployment.apps/db-pod
        deployment.apps "db-pod" deleted
        ```
        
    2. Verify that the service still exists without the deployment.
        
        ```plaintext
        [student@workstation ~]$ oc get service
        NAME     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
        db-pod   ClusterIP   172.30.108.92  <none>        3306/TCP   9m53s
        ```
        
    3. The list of endpoints for the service is now empty.
        
        ```plaintext
        [student@workstation ~]$ oc get endpoints
        NAME     ENDPOINTS   AGE
        db-pod   <none>      12m
        ```
        
    4. Re-create the `db-pod` deployment.
        
        ```plaintext
        [student@workstation ~]$ oc create deployment db-pod --port 3306 \
          --image registry.ocp4.example.com:8443/rhel8/mysql-80
        Warning: would violate PodSecurity "restricted:v1.24":
        ...output omitted...
        deployment.apps/db-pod created
        ```
        
    5. Add the environment variables.
        
        ```plaintext
        [student@workstation ~]$ oc set env deployment/db-pod \
          MYSQL_USER=user1 \
          MYSQL_PASSWORD=mypa55w0rd \
          MYSQL_DATABASE=items
        Warning: would violate PodSecurity "restricted:v1.24":
        ...output omitted...
        deployment.apps/db-pod updated
        ```
        
    6. Confirm that the newly created pod has the `app=db-pod` selector.
        
        ```plaintext
        [student@workstation ~]$ oc get pods --selector app=db-pod -o wide
        NAME                    READY   STATUS    RESTARTS   AGE   IP        ...
        db-pod-6ccc485cfc-l2x   1/1     Running   0          32s   10.8.0.85 ...
        ```
        
        Notice the change in the pod name. The pod IP address might also change. Your pod name and IP address might differ from the previous output.
        
    7. Confirm that the endpoints for the `db-pod` service include the newly created pod.
        
        ```plaintext
        [student@workstation ~]$ oc get endpoints
        NAME     ENDPOINTS        AGE
        db-pod   10.8.0.85:3306   16m
        ```
        
6. Create a pod to identify the available DNS name assignments for the service.
    
    1. Create a pod named `shell` to use for troubleshooting. Use the `oc run` command and the [`registry.ocp4.example.com:8443/ubi8/ubi`](http://registry.ocp4.example.com:8443/ubi8/ubi) container image.
        
        ```plaintext
        [student@workstation ~]$ oc run shell -it \
          --image registry.ocp4.example.com:8443/ubi8/ubi
        If you don't see a command prompt, try pressing enter.
        bash-4.4$
        ```
        
    2. From the prompt inside the `shell` pod, view the `/etc/resolv.conf` file to identify the cluster-domain name.
        
        ```plaintext
        bash-4.4$ cat /etc/resolv.conf
        search deploy-services.svc.cluster.local svc.cluster.local ...
        nameserver 172.30.0.10
        options ndots:5
        ```
        
        The container uses the values from the `search` directive as suffix values on DNS searches. The container appends these values to a DNS query, in the written order, to resolve the search. The cluster-domain name is the last few components of these values that start after `svc`.
        
    3. Use the `timeout` command to test the available DNS names for the service.
        
        ### **NOTE**
        
        The `ping` utility is often used for this test. However, the `ping` utility is not available in the `shell` pod, because the `ubi8/ubi` container image is configured as a non-root container. Non-root containers cannot use the `ping` utility by default, because it requires elevated privileges to establish raw sockets. However, the widely available `timeout` command can test port connectivity.
        
        ```plaintext
        timeout <value> bash -c "/dev/tcp/<server>/<port>"
        ```
        
        The `<value>` object is the timeout value for the poll on the `<server>` target on the `<port>` port.
        
        ```plaintext
        bash-4.4$ timeout 5 bash -c \
          "</dev/tcp/db-pod.deploy-services.svc.cluster.local/3306" && \
          echo "connection success to db-pod.deploy-services.svc.cluster.local:3306" || \
          echo "connection failed"
        connection success to db-pod.deploy-services.svc.cluster.local:3306
        ```
        
        The long version of the DNS name is required when accessing the service from a different project. When the pod is in the same project, you can use a shorter version of the DNS name.
        
        ```plaintext
        bash-4.4$ timeout 5 bash -c \
          "</dev/tcp/db-pod.deploy-services/3306" && \
          echo "connection success to db-pod.deploy-services:3306" || \
          echo "connection failed"
        connection success to db-pod.deploy-services:3306
        ```
        
        The `search` directive in the `resolv.conf` file enables an even shorter form without the namespace component.
        
        ```plaintext
        bash-4.4$ timeout 5 bash -c \
          "</dev/tcp/db-pod/3306" && \
          echo "connection success to db-pod:3306" || \
          echo "connection failed"
        connection success to db-pod:3306
        ```
        
    4. Exit the interactive session.
        
        ```plaintext
        bash-4.4$ exit
        Session ended, resume using 'oc attach shell -c shell -i -t' command when the pod is running
        ```
        
    5. Delete the pod for the shell.
        
        ```plaintext
        [student@workstation ~]$ oc delete pod shell
        pod "shell" deleted
        ```
        
7. Use a new project to test pod communications across namespaces.
    
    1. Create a second namespace with the `oc new-project` command.
        
        ```plaintext
        [student@workstation ~]$ oc new-project deploy-services-2
        Now using project "deploy-services-2" on server "https://api.ocp4.example.com:6443".
        ...output omitted...
        ```
        
    2. Execute a `timeout` command from a pod to test the DNS name access to another namespace.
        
        ```plaintext
        [student@workstation ~]$ oc run shell --env POD_IP=$POD_IP -it --rm \
          --image registry.ocp4.example.com:8443/ubi8/ubi --restart Never \
          --command -- timeout 5 bash -c \
          "</dev/tcp/db-pod.deploy-services.svc.cluster.local/3306" && \
          echo "connection success to db-pod.deploy-services.svc.cluster.local:3306" \
          || echo "connection failed"
        ...output omitted...
        pod "shell" deleted
        connection success to db-pod.deploy-services.svc.cluster.local:3306
        ```
        
    3. Return to the `deploy-services` project.
        
        ```plaintext
        [student@workstation ~]$ oc project deploy-services
        Now using project "deploy-services" on server "https://api.ocp4.example.com:6443".
        ```
        
8. Use a Kubernetes job to add initialization data to the database.
    
    1. Create a job named `mysql-init` that uses the [`registry.ocp4.example.com:8443/redhattraining/do180-dbinit`](http://registry.ocp4.example.com:8443/redhattraining/do180-dbinit) container image. This image uses the `mysql-80` container image as a base image, and it includes a script that adds a few initial records to the database.
        
        ```plaintext
        [student@workstation ~]$ oc create job mysql-init \
          --image registry.ocp4.example.com:8443/redhattraining/do180-dbinit \
          -- /bin/bash -c "mysql -uuser1 -pmypa55w0rd --protocol tcp \
          -h db-pod -P3306 items </tmp/db-init.sql"
        Warning: would violate PodSecurity "restricted:v1.24":
        ...output omitted...
        job.batch/mysql-init created
        ```
        
        The `-h` option of the `mysql` command directs the command to communicate with the DNS short name of the `db-pod` service. The `db-pod` short name can be used here, because the pod for the job is created in the same namespace as the service.
        
        The double dash `--` before `/bin/bash` separates the `oc` command arguments from the command in the pod. The `-c` option of `/bin/bash` directs the command interpreter in the container to execute the command string. The `/tmp/db-init.sql` file is redirected as input for the command. The `db-init.sql` file is included in the image, and contains the following script.
        
        ```plaintext
        DROP TABLE IF EXISTS `Item`;
        CREATE TABLE `Item` (`id` BIGINT not null auto_increment primary key, `description` VARCHAR(100), `done` BIT);
        INSERT INTO `Item` (`id`,`description`,`done`) VALUES (1,'Pick up newspaper', 0);
        INSERT INTO `Item` (`id`,`description`,`done`) VALUES (2,'Buy groceries', 1);
        ```
        
    2. Confirm the status of the `mysql-init` job. Wait for the job to complete.
        
        ```plaintext
        [student@workstation ~]$ oc get job
        NAME         COMPLETIONS   DURATION   AGE
        mysql-init   1/1                      22m
        ```
        
    3. Retrieve the status of the `mysql-init` job pod, to confirm that the pod has a `Completed` status.
        
        ```plaintext
        [student@workstation ~]$ oc get pods
        NAME                      READY   STATUS      RESTARTS      AGE
        db-pod-6ccc485cfc-2lklx   1/1     Running     0             4h24m
        mysql-init-ln9cg          0/1     Completed   0             23m
        ```
        
    4. Delete the `mysql-init` job, because it is no longer needed.
        
        ```plaintext
        [student@workstation ~]$ oc delete job mysql-init
        job.batch "mysql-init" deleted
        ```
        
    5. Verify that the corresponding `mysql-init` pod is also deleted.
        
        ```plaintext
        [student@workstation ~]$ oc get pods
        NAME                      READY   STATUS    RESTARTS      AGE
        db-pod-6ccc485cfc-2lklx   1/1     Running   0             4h2
        ```
        
9. Create a `query-db` pod by using the `oc run` command and the [`registry.ocp4.example.com:8443/redhattraining/do180-dbinit`](http://registry.ocp4.example.com:8443/redhattraining/do180-dbinit) container image. Use the pod to execute a query against the database service.
    
    1. Create the `query-db` pod. Configure the pod to use the MySQL client to execute a query against the `db-pod` service. You can use the `db-pod` service short name, which provides a stable reference.
        
        ```plaintext
        [student@workstation ~]$ oc run query-db -it --rm \
          --image registry.ocp4.example.com:8443/redhattraining/do180-dbinit \
          --restart Never \
          --command -- /bin/bash -c "mysql -uuser1 -pmypa55w0rd --protocol tcp \
          -h db-pod -P3306 items -e 'select * from Item;'"
        Warning: would violate PodSecurity "restricted:v1.24":
        ...output omitted...
        +----+-------------------+------------+
        | id | description       | done       |
        +----+-------------------+------------+
        |  1 | Pick up newspaper | 0x00       |
        |  2 | Buy groceries     | 0x01       |
        +----+-------------------+------------+
        pod "query-db" deleted
        ```
        
10. As the cluster administrator, identify the pod and service subnet ranges.
    
    1. Log in as the `admin` user with the `redhatocp` password.
        
        ```plaintext
        [student@workstation ~]$ oc login -u admin -p redhatocp \
          https://api.ocp4.example.com:6443
        Login successful.
        ...output omitted...
        ```
        
    2. Observe the status of the `network-operator` deployment in the `openshift-network-operator` namespace.
        
        ```plaintext
        [student@workstation ~]$ oc get -n openshift-network-operator \
          deployment/network-operator
        NAME               READY   UP-TO-DATE   AVAILABLE   AGE
        network-operator   1/1     1            1           41d
        ```
        
        Administrator access is required to view the `openshift-network-operator` namespace.
        
    3. Retrieve the `cluster` network configuration for the cluster network operator. Compare the IP ranges that the pod subnet and the service subnet use.
        
        ```plaintext
        [student@workstation ~]$ oc get network cluster -o json | jq .spec
        {
          "clusterNetwork": [
            {
              "cidr": "10.8.0.0/14",
              "hostPrefix": 23
            }
          ],
          "externalIP": {
            "policy": {}
          },
          "networkType": "OVNKubernetes",
          "serviceNetwork": [
            "172.30.0.0/16"
          ]
        }
        ```
        
        All pods in this cluster are created in the 10.8.0.0/14 range, and all services are created in the 172.30.0.0/16 range.
        
11. It might be necessary to use pod-to-pod communications for troubleshooting. Use the `oc run` command to create a pod that executes a network test against the IP address of the database pod.
    
    1. Confirm the IP address of the MySQL database pod. Your pod IP address might differ from the output.
        
        ```plaintext
        [student@workstation ~]$ oc get pods -o wide
        NAME                     READY  STATUS     RESTARTS     AGE  IP         ...
        db-pod-6ccc485cfc-2lklx  1/1    Running    0            4h5  10.8.0.69  ...
        ```
        
    2. Capture the IP address in an environment variable.
        
        ```plaintext
        [student@workstation ~]$ POD_IP=$(oc get pod -l app=db-pod \
          -o jsonpath='{.items[0].status.podIP}')
        ```
        
    3. Create a test pod named `shell` with the `oc run` command. Execute the `timeout` command to test against the `$POD_IP` environment variable and the `3306` port for the database.
        
        ```plaintext
        [student@workstation ~]$ oc run shell --env POD_IP=$POD_IP -it --rm \
          --image registry.ocp4.example.com:8443/ubi8/ubi --restart Never \
          --command -- timeout 5 bash -c "</dev/tcp/$POD_IP/3306" && \
          echo "connection success to $POD_IP:3306" || echo "connection failed"
        Warning: would violate PodSecurity "restricted:v1.24":
        ...output omitted...
        pod "shell" deleted
        connection success to 10.8.0.69:3306
        ```
        

**Finish**

On the `workstation` machine, use the `lab` command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises.

```plaintext
[student@workstation ~]$ lab finish deploy-services
```