Syarif's Blog

Guided Exercise: Kubernetes Pod and Service Networks

Syarif's photo
Syarif
·

10 min read

Deploy a database server and access it through a Kubernetes service.

Outcomes

You should be able to deploy a database server, and access it indirectly through a Kubernetes service, and also directly pod-to-pod for troubleshooting.

As the student user on the workstation machine, use the lab command to prepare your system for this exercise.

This command ensures that all resources are available for this exercise. It also creates the deploy-services project and the /home/student/DO180/labs/deploy-services/resources.txt file. The resources.txt file contains some of the commands that you use during the exercise. You can use the file to copy and paste these commands.

[student@workstation ~]$ lab start deploy-services

NOTE

It is safe to ignore pod security warnings for exercises in this course. OpenShift uses the Security Context Constraints controller to provide safe defaults for pod security.

Procedure 4.3. Instructions

  1. Log in to the OpenShift cluster as the developer user with the developer password. Use the deploy-services project.

    1. Log in to the OpenShift cluster.

       [student@workstation ~]$ oc login -u developer -p developer \
   https://api.ocp4.example.com:6443
 Login successful.
 ...output omitted...

    2. Set the deploy-services project as the active project.

       [student@workstation ~]$ oc project deploy-services
 ...output omitted...

  2. Use the registry.ocp4.example.com:8443/rhel8/mysql-80 container image to create a MySQL deployment named db-pod. Add the missing environment variables for the pod to run.

    1. Create the db-pod deployment.

       [student@workstation ~]$ oc create deployment db-pod --port 3306 \
   --image registry.ocp4.example.com:8443/rhel8/mysql-80
 Warning: would violate PodSecurity "restricted:v1.24":
 ...output omitted...
 deployment.apps/db-pod created

    2. Add the environment variables.

       [student@workstation ~]$ oc set env deployment/db-pod \
   MYSQL_USER=user1 \
   MYSQL_PASSWORD=mypa55w0rd \
   MYSQL_DATABASE=items
 Warning: would violate PodSecurity "restricted:v1.24":
 ...output omitted...
 deployment.apps/db-pod updated

    3. Confirm that the pod is running.

       [student@workstation ~]$ oc get pods
 NAME                      READY   STATUS    RESTARTS   AGE
 db-pod-6ccc485cfc-vrc4r   1/1     Running   0          2m30s

      Your pod name might differ from the previous output.

  3. Expose the db-pod deployment to create a ClusterIP service.

    1. View the deployment for the pod.

       [student@workstation ~]$ oc get deployment
 NAME     READY   UP-TO-DATE   AVAILABLE   AGE
 db-pod   1/1     1            1           3m36s

    2. Expose the db-pod deployment to create a service.

       [student@workstation ~]$ oc expose deployment/db-pod
 service/db-pod exposed

  4. Validate the service. Confirm that the service selector matches the label on the pod. Then, confirm that the db-pod service endpoint matches the IP of the pod.

    1. Identify the selector for the db-pod service. Use the oc get service command with the -o wide option to retrieve the selector that the service uses.

       [student@workstation ~]$ oc get service db-pod -o wide
 NAME     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE    SELECTOR
 db-pod   ClusterIP   172.30.108.92   <none>        3306/TCP   108s   app=db-pod

      The selector shows an app=db-pod key:value pair.

    2. Capture the name of the pod in a variable.

       [student@workstation ~]$ PODNAME=$(oc get pods \
   -o jsonpath='{.items[0].metadata.name}')

    3. Query the label on the pod.

       [student@workstation ~]$ oc get pod $PODNAME --show-labels
 NAME                     READY   STATUS    RESTARTS   AGE     LABELS
 db-pod-6ccc485cfc-vrc4r  1/1     Running   0          6m50s   app=db-pod ...

      Notice that the label list includes the app=db-pod key-value pair, which is the selector for the db-pod service.

    4. Retrieve the endpoints for the db-pod service.

       [student@workstation ~]$ oc get endpoints
 NAME     ENDPOINTS        AGE
 db-pod   10.8.0.85:3306   4m38s

      Your endpoints values might differ from the previous output.

    5. Verify that the service endpoint matches the db-pod IP address. Use the oc get pods command with the -o wide option to view the pod IP address.

       [student@workstation ~]$ oc get pods -o wide
 NAME                      READY   STATUS    RESTARTS   AGE   IP        ...
 db-pod-6ccc485cfc-vrc4r   1/1     Running   0          54m   10.8.0.85 ...

      The service endpoint resolves to the IP address that is assigned to the pod.

  5. Delete and then re-create the db-pod deployment. Confirm that the db-pod service endpoint automatically resolves to the IP address of the new pod.

    1. Delete the db-pod deployment.

       [student@workstation ~]$ oc delete deployment.apps/db-pod
 deployment.apps "db-pod" deleted

    2. Verify that the service still exists without the deployment.

       [student@workstation ~]$ oc get service
 NAME     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
 db-pod   ClusterIP   172.30.108.92  <none>        3306/TCP   9m53s

    3. The list of endpoints for the service is now empty.

       [student@workstation ~]$ oc get endpoints
 NAME     ENDPOINTS   AGE
 db-pod   <none>      12m

    4. Re-create the db-pod deployment.

       [student@workstation ~]$ oc create deployment db-pod --port 3306 \
   --image registry.ocp4.example.com:8443/rhel8/mysql-80
 Warning: would violate PodSecurity "restricted:v1.24":
 ...output omitted...
 deployment.apps/db-pod created

    5. Add the environment variables.

       [student@workstation ~]$ oc set env deployment/db-pod \
   MYSQL_USER=user1 \
   MYSQL_PASSWORD=mypa55w0rd \
   MYSQL_DATABASE=items
 Warning: would violate PodSecurity "restricted:v1.24":
 ...output omitted...
 deployment.apps/db-pod updated

    6. Confirm that the newly created pod has the app=db-pod selector.

       [student@workstation ~]$ oc get pods --selector app=db-pod -o wide
 NAME                    READY   STATUS    RESTARTS   AGE   IP        ...
 db-pod-6ccc485cfc-l2x   1/1     Running   0          32s   10.8.0.85 ...

      Notice the change in the pod name. The pod IP address might also change. Your pod name and IP address might differ from the previous output.

    7. Confirm that the endpoints for the db-pod service include the newly created pod.

       [student@workstation ~]$ oc get endpoints
 NAME     ENDPOINTS        AGE
 db-pod   10.8.0.85:3306   16m

  6. Create a pod to identify the available DNS name assignments for the service.

    1. Create a pod named shell to use for troubleshooting. Use the oc run command and the registry.ocp4.example.com:8443/ubi8/ubi container image.

       [student@workstation ~]$ oc run shell -it \
   --image registry.ocp4.example.com:8443/ubi8/ubi
 If you don't see a command prompt, try pressing enter.
 bash-4.4$

    2. From the prompt inside the shell pod, view the /etc/resolv.conf file to identify the cluster-domain name.

       bash-4.4$ cat /etc/resolv.conf
 search deploy-services.svc.cluster.local svc.cluster.local ...
 nameserver 172.30.0.10
 options ndots:5

      The container uses the values from the search directive as suffix values on DNS searches. The container appends these values to a DNS query, in the written order, to resolve the search. The cluster-domain name is the last few components of these values that start after svc.

    3. Use the timeout command to test the available DNS names for the service.

      NOTE

      The ping utility is often used for this test. However, the ping utility is not available in the shell pod, because the ubi8/ubi container image is configured as a non-root container. Non-root containers cannot use the ping utility by default, because it requires elevated privileges to establish raw sockets. However, the widely available timeout command can test port connectivity.

       timeout <value> bash -c "/dev/tcp/<server>/<port>"

      The <value> object is the timeout value for the poll on the <server> target on the <port> port.

       bash-4.4$ timeout 5 bash -c \
   "</dev/tcp/db-pod.deploy-services.svc.cluster.local/3306" && \
   echo "connection success to db-pod.deploy-services.svc.cluster.local:3306" || \
   echo "connection failed"
 connection success to db-pod.deploy-services.svc.cluster.local:3306

      The long version of the DNS name is required when accessing the service from a different project. When the pod is in the same project, you can use a shorter version of the DNS name.

       bash-4.4$ timeout 5 bash -c \
   "</dev/tcp/db-pod.deploy-services/3306" && \
   echo "connection success to db-pod.deploy-services:3306" || \
   echo "connection failed"
 connection success to db-pod.deploy-services:3306

      The search directive in the resolv.conf file enables an even shorter form without the namespace component.

       bash-4.4$ timeout 5 bash -c \
   "</dev/tcp/db-pod/3306" && \
   echo "connection success to db-pod:3306" || \
   echo "connection failed"
 connection success to db-pod:3306

    4. Exit the interactive session.

       bash-4.4$ exit
 Session ended, resume using 'oc attach shell -c shell -i -t' command when the pod is running

    5. Delete the pod for the shell.

       [student@workstation ~]$ oc delete pod shell
 pod "shell" deleted

  7. Use a new project to test pod communications across namespaces.

    1. Create a second namespace with the oc new-project command.

       [student@workstation ~]$ oc new-project deploy-services-2
 Now using project "deploy-services-2" on server "https://api.ocp4.example.com:6443".
 ...output omitted...

    2. Execute a timeout command from a pod to test the DNS name access to another namespace.

       [student@workstation ~]$ oc run shell --env POD_IP=$POD_IP -it --rm \
   --image registry.ocp4.example.com:8443/ubi8/ubi --restart Never \
   --command -- timeout 5 bash -c \
   "</dev/tcp/db-pod.deploy-services.svc.cluster.local/3306" && \
   echo "connection success to db-pod.deploy-services.svc.cluster.local:3306" \
   || echo "connection failed"
 ...output omitted...
 pod "shell" deleted
 connection success to db-pod.deploy-services.svc.cluster.local:3306

    3. Return to the deploy-services project.

       [student@workstation ~]$ oc project deploy-services
 Now using project "deploy-services" on server "https://api.ocp4.example.com:6443".

  8. Use a Kubernetes job to add initialization data to the database.

    1. Create a job named mysql-init that uses the registry.ocp4.example.com:8443/redhattraining/do180-dbinit container image. This image uses the mysql-80 container image as a base image, and it includes a script that adds a few initial records to the database.

       [student@workstation ~]$ oc create job mysql-init \
   --image registry.ocp4.example.com:8443/redhattraining/do180-dbinit \
   -- /bin/bash -c "mysql -uuser1 -pmypa55w0rd --protocol tcp \
   -h db-pod -P3306 items </tmp/db-init.sql"
 Warning: would violate PodSecurity "restricted:v1.24":
 ...output omitted...
 job.batch/mysql-init created

      The -h option of the mysql command directs the command to communicate with the DNS short name of the db-pod service. The db-pod short name can be used here, because the pod for the job is created in the same namespace as the service.

      The double dash -- before /bin/bash separates the oc command arguments from the command in the pod. The -c option of /bin/bash directs the command interpreter in the container to execute the command string. The /tmp/db-init.sql file is redirected as input for the command. The db-init.sql file is included in the image, and contains the following script.

       DROP TABLE IF EXISTS `Item`;
 CREATE TABLE `Item` (`id` BIGINT not null auto_increment primary key, `description` VARCHAR(100), `done` BIT);
 INSERT INTO `Item` (`id`,`description`,`done`) VALUES (1,'Pick up newspaper', 0);
 INSERT INTO `Item` (`id`,`description`,`done`) VALUES (2,'Buy groceries', 1);

    2. Confirm the status of the mysql-init job. Wait for the job to complete.

       [student@workstation ~]$ oc get job
 NAME         COMPLETIONS   DURATION   AGE
 mysql-init   1/1                      22m

    3. Retrieve the status of the mysql-init job pod, to confirm that the pod has a Completed status.

       [student@workstation ~]$ oc get pods
 NAME                      READY   STATUS      RESTARTS      AGE
 db-pod-6ccc485cfc-2lklx   1/1     Running     0             4h24m
 mysql-init-ln9cg          0/1     Completed   0             23m

    4. Delete the mysql-init job, because it is no longer needed.

       [student@workstation ~]$ oc delete job mysql-init
 job.batch "mysql-init" deleted

    5. Verify that the corresponding mysql-init pod is also deleted.

       [student@workstation ~]$ oc get pods
 NAME                      READY   STATUS    RESTARTS      AGE
 db-pod-6ccc485cfc-2lklx   1/1     Running   0             4h2

  9. Create a query-db pod by using the oc run command and the registry.ocp4.example.com:8443/redhattraining/do180-dbinit container image. Use the pod to execute a query against the database service.

    1. Create the query-db pod. Configure the pod to use the MySQL client to execute a query against the db-pod service. You can use the db-pod service short name, which provides a stable reference.

       [student@workstation ~]$ oc run query-db -it --rm \
   --image registry.ocp4.example.com:8443/redhattraining/do180-dbinit \
   --restart Never \
   --command -- /bin/bash -c "mysql -uuser1 -pmypa55w0rd --protocol tcp \
   -h db-pod -P3306 items -e 'select * from Item;'"
 Warning: would violate PodSecurity "restricted:v1.24":
 ...output omitted...
 +----+-------------------+------------+
 | id | description       | done       |
 +----+-------------------+------------+
 |  1 | Pick up newspaper | 0x00       |
 |  2 | Buy groceries     | 0x01       |
 +----+-------------------+------------+
 pod "query-db" deleted

  10. As the cluster administrator, identify the pod and service subnet ranges.

    1. Log in as the admin user with the redhatocp password.

       [student@workstation ~]$ oc login -u admin -p redhatocp \
   https://api.ocp4.example.com:6443
 Login successful.
 ...output omitted...

    2. Observe the status of the network-operator deployment in the openshift-network-operator namespace.

       [student@workstation ~]$ oc get -n openshift-network-operator \
   deployment/network-operator
 NAME               READY   UP-TO-DATE   AVAILABLE   AGE
 network-operator   1/1     1            1           41d

      Administrator access is required to view the openshift-network-operator namespace.

    3. Retrieve the cluster network configuration for the cluster network operator. Compare the IP ranges that the pod subnet and the service subnet use.

       [student@workstation ~]$ oc get network cluster -o json | jq .spec
 {
   "clusterNetwork": [
     {
       "cidr": "10.8.0.0/14",
       "hostPrefix": 23
     }
   ],
   "externalIP": {
     "policy": {}
   },
   "networkType": "OVNKubernetes",
   "serviceNetwork": [
     "172.30.0.0/16"
   ]
 }

      All pods in this cluster are created in the 10.8.0.0/14 range, and all services are created in the 172.30.0.0/16 range.

  11. It might be necessary to use pod-to-pod communications for troubleshooting. Use the oc run command to create a pod that executes a network test against the IP address of the database pod.

    1. Confirm the IP address of the MySQL database pod. Your pod IP address might differ from the output.

       [student@workstation ~]$ oc get pods -o wide
 NAME                     READY  STATUS     RESTARTS     AGE  IP         ...
 db-pod-6ccc485cfc-2lklx  1/1    Running    0            4h5  10.8.0.69  ...

    2. Capture the IP address in an environment variable.

       [student@workstation ~]$ POD_IP=$(oc get pod -l app=db-pod \
   -o jsonpath='{.items[0].status.podIP}')

    3. Create a test pod named shell with the oc run command. Execute the timeout command to test against the $POD_IP environment variable and the 3306 port for the database.

       [student@workstation ~]$ oc run shell --env POD_IP=$POD_IP -it --rm \
   --image registry.ocp4.example.com:8443/ubi8/ubi --restart Never \
   --command -- timeout 5 bash -c "</dev/tcp/$POD_IP/3306" && \
   echo "connection success to $POD_IP:3306" || echo "connection failed"
 Warning: would violate PodSecurity "restricted:v1.24":
 ...output omitted...
 pod "shell" deleted
 connection success to 10.8.0.69:3306

Finish

On the workstation machine, use the lab command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises.

[student@workstation ~]$ lab finish deploy-services
openshift