Guided Exercise: Kubernetes Pod and Service Networks
Deploy a database server and access it through a Kubernetes service.
Outcomes
You should be able to deploy a database server, and access it indirectly through a Kubernetes service, and also directly pod-to-pod for troubleshooting.
As the student
user on the workstation
machine, use the lab
command to prepare your system for this exercise.
This command ensures that all resources are available for this exercise. It also creates the deploy-services project and the /home/student/DO180/labs/deploy-services/resources.txt
file. The resources.txt
file contains some of the commands that you use during the exercise. You can use the file to copy and paste these commands.
[student@workstation ~]$ lab start deploy-services
NOTE
It is safe to ignore pod security warnings for exercises in this course. OpenShift uses the Security Context Constraints controller to provide safe defaults for pod security.
Procedure 4.3. Instructions
Log in to the OpenShift cluster as the
developer
user with thedeveloper
password. Use thedeploy-services
project.Log in to the OpenShift cluster.
[student@workstation ~]$ oc login -u developer -p developer \ https://api.ocp4.example.com:6443 Login successful. ...output omitted...
Set the
deploy-services
project as the active project.[student@workstation ~]$ oc project deploy-services ...output omitted...
Use the
registry.ocp4.example.com:8443/rhel8/mysql-80
container image to create a MySQL deployment nameddb-pod
. Add the missing environment variables for the pod to run.Create the
db-pod
deployment.[student@workstation ~]$ oc create deployment db-pod --port 3306 \ --image registry.ocp4.example.com:8443/rhel8/mysql-80 Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... deployment.apps/db-pod created
Add the environment variables.
[student@workstation ~]$ oc set env deployment/db-pod \ MYSQL_USER=user1 \ MYSQL_PASSWORD=mypa55w0rd \ MYSQL_DATABASE=items Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... deployment.apps/db-pod updated
Confirm that the pod is running.
[student@workstation ~]$ oc get pods NAME READY STATUS RESTARTS AGE db-pod-6ccc485cfc-vrc4r 1/1 Running 0 2m30s
Your pod name might differ from the previous output.
Expose the
db-pod
deployment to create a ClusterIP service.View the deployment for the pod.
[student@workstation ~]$ oc get deployment NAME READY UP-TO-DATE AVAILABLE AGE db-pod 1/1 1 1 3m36s
Expose the
db-pod
deployment to create a service.[student@workstation ~]$ oc expose deployment/db-pod service/db-pod exposed
Validate the service. Confirm that the service selector matches the label on the pod. Then, confirm that the
db-pod
service endpoint matches the IP of the pod.Identify the selector for the
db-pod
service. Use theoc get service command
with the-o wide
option to retrieve the selector that the service uses.[student@workstation ~]$ oc get service db-pod -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR db-pod ClusterIP 172.30.108.92 <none> 3306/TCP 108s app=db-pod
The selector shows an
app=db-pod
key:value pair.Capture the name of the pod in a variable.
[student@workstation ~]$ PODNAME=$(oc get pods \ -o jsonpath='{.items[0].metadata.name}')
Query the label on the pod.
[student@workstation ~]$ oc get pod $PODNAME --show-labels NAME READY STATUS RESTARTS AGE LABELS db-pod-6ccc485cfc-vrc4r 1/1 Running 0 6m50s app=db-pod ...
Notice that the label list includes the
app=db-pod
key-value pair, which is the selector for thedb-pod
service.Retrieve the endpoints for the
db-pod
service.[student@workstation ~]$ oc get endpoints NAME ENDPOINTS AGE db-pod 10.8.0.85:3306 4m38s
Your endpoints values might differ from the previous output.
Verify that the service endpoint matches the
db-pod
IP address. Use theoc get pods
command with the-o wide
option to view the pod IP address.[student@workstation ~]$ oc get pods -o wide NAME READY STATUS RESTARTS AGE IP ... db-pod-6ccc485cfc-vrc4r 1/1 Running 0 54m 10.8.0.85 ...
The service endpoint resolves to the IP address that is assigned to the pod.
Delete and then re-create the
db-pod
deployment. Confirm that thedb-pod
service endpoint automatically resolves to the IP address of the new pod.Delete the
db-pod
deployment.[student@workstation ~]$ oc delete deployment.apps/db-pod deployment.apps "db-pod" deleted
Verify that the service still exists without the deployment.
[student@workstation ~]$ oc get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE db-pod ClusterIP 172.30.108.92 <none> 3306/TCP 9m53s
The list of endpoints for the service is now empty.
[student@workstation ~]$ oc get endpoints NAME ENDPOINTS AGE db-pod <none> 12m
Re-create the
db-pod
deployment.[student@workstation ~]$ oc create deployment db-pod --port 3306 \ --image registry.ocp4.example.com:8443/rhel8/mysql-80 Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... deployment.apps/db-pod created
Add the environment variables.
[student@workstation ~]$ oc set env deployment/db-pod \ MYSQL_USER=user1 \ MYSQL_PASSWORD=mypa55w0rd \ MYSQL_DATABASE=items Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... deployment.apps/db-pod updated
Confirm that the newly created pod has the
app=db-pod
selector.[student@workstation ~]$ oc get pods --selector app=db-pod -o wide NAME READY STATUS RESTARTS AGE IP ... db-pod-6ccc485cfc-l2x 1/1 Running 0 32s 10.8.0.85 ...
Notice the change in the pod name. The pod IP address might also change. Your pod name and IP address might differ from the previous output.
Confirm that the endpoints for the
db-pod
service include the newly created pod.[student@workstation ~]$ oc get endpoints NAME ENDPOINTS AGE db-pod 10.8.0.85:3306 16m
Create a pod to identify the available DNS name assignments for the service.
Create a pod named
shell
to use for troubleshooting. Use theoc run
command and theregistry.ocp4.example.com:8443/ubi8/ubi
container image.[student@workstation ~]$ oc run shell -it \ --image registry.ocp4.example.com:8443/ubi8/ubi If you don't see a command prompt, try pressing enter. bash-4.4$
From the prompt inside the
shell
pod, view the/etc/resolv.conf
file to identify the cluster-domain name.bash-4.4$ cat /etc/resolv.conf search deploy-services.svc.cluster.local svc.cluster.local ... nameserver 172.30.0.10 options ndots:5
The container uses the values from the
search
directive as suffix values on DNS searches. The container appends these values to a DNS query, in the written order, to resolve the search. The cluster-domain name is the last few components of these values that start aftersvc
.Use the
timeout
command to test the available DNS names for the service.NOTE
The
ping
utility is often used for this test. However, theping
utility is not available in theshell
pod, because theubi8/ubi
container image is configured as a non-root container. Non-root containers cannot use theping
utility by default, because it requires elevated privileges to establish raw sockets. However, the widely availabletimeout
command can test port connectivity.timeout <value> bash -c "/dev/tcp/<server>/<port>"
The
<value>
object is the timeout value for the poll on the<server>
target on the<port>
port.bash-4.4$ timeout 5 bash -c \ "</dev/tcp/db-pod.deploy-services.svc.cluster.local/3306" && \ echo "connection success to db-pod.deploy-services.svc.cluster.local:3306" || \ echo "connection failed" connection success to db-pod.deploy-services.svc.cluster.local:3306
The long version of the DNS name is required when accessing the service from a different project. When the pod is in the same project, you can use a shorter version of the DNS name.
bash-4.4$ timeout 5 bash -c \ "</dev/tcp/db-pod.deploy-services/3306" && \ echo "connection success to db-pod.deploy-services:3306" || \ echo "connection failed" connection success to db-pod.deploy-services:3306
The
search
directive in theresolv.conf
file enables an even shorter form without the namespace component.bash-4.4$ timeout 5 bash -c \ "</dev/tcp/db-pod/3306" && \ echo "connection success to db-pod:3306" || \ echo "connection failed" connection success to db-pod:3306
Exit the interactive session.
bash-4.4$ exit Session ended, resume using 'oc attach shell -c shell -i -t' command when the pod is running
Delete the pod for the shell.
[student@workstation ~]$ oc delete pod shell pod "shell" deleted
Use a new project to test pod communications across namespaces.
Create a second namespace with the
oc new-project
command.[student@workstation ~]$ oc new-project deploy-services-2 Now using project "deploy-services-2" on server "https://api.ocp4.example.com:6443". ...output omitted...
Execute a
timeout
command from a pod to test the DNS name access to another namespace.[student@workstation ~]$ oc run shell --env POD_IP=$POD_IP -it --rm \ --image registry.ocp4.example.com:8443/ubi8/ubi --restart Never \ --command -- timeout 5 bash -c \ "</dev/tcp/db-pod.deploy-services.svc.cluster.local/3306" && \ echo "connection success to db-pod.deploy-services.svc.cluster.local:3306" \ || echo "connection failed" ...output omitted... pod "shell" deleted connection success to db-pod.deploy-services.svc.cluster.local:3306
Return to the
deploy-services
project.[student@workstation ~]$ oc project deploy-services Now using project "deploy-services" on server "https://api.ocp4.example.com:6443".
Use a Kubernetes job to add initialization data to the database.
Create a job named
mysql-init
that uses theregistry.ocp4.example.com:8443/redhattraining/do180-dbinit
container image. This image uses themysql-80
container image as a base image, and it includes a script that adds a few initial records to the database.[student@workstation ~]$ oc create job mysql-init \ --image registry.ocp4.example.com:8443/redhattraining/do180-dbinit \ -- /bin/bash -c "mysql -uuser1 -pmypa55w0rd --protocol tcp \ -h db-pod -P3306 items </tmp/db-init.sql" Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... job.batch/mysql-init created
The
-h
option of themysql
command directs the command to communicate with the DNS short name of thedb-pod
service. Thedb-pod
short name can be used here, because the pod for the job is created in the same namespace as the service.The double dash
--
before/bin/bash
separates theoc
command arguments from the command in the pod. The-c
option of/bin/bash
directs the command interpreter in the container to execute the command string. The/tmp/db-init.sql
file is redirected as input for the command. Thedb-init.sql
file is included in the image, and contains the following script.DROP TABLE IF EXISTS `Item`; CREATE TABLE `Item` (`id` BIGINT not null auto_increment primary key, `description` VARCHAR(100), `done` BIT); INSERT INTO `Item` (`id`,`description`,`done`) VALUES (1,'Pick up newspaper', 0); INSERT INTO `Item` (`id`,`description`,`done`) VALUES (2,'Buy groceries', 1);
Confirm the status of the
mysql-init
job. Wait for the job to complete.[student@workstation ~]$ oc get job NAME COMPLETIONS DURATION AGE mysql-init 1/1 22m
Retrieve the status of the
mysql-init
job pod, to confirm that the pod has aCompleted
status.[student@workstation ~]$ oc get pods NAME READY STATUS RESTARTS AGE db-pod-6ccc485cfc-2lklx 1/1 Running 0 4h24m mysql-init-ln9cg 0/1 Completed 0 23m
Delete the
mysql-init
job, because it is no longer needed.[student@workstation ~]$ oc delete job mysql-init job.batch "mysql-init" deleted
Verify that the corresponding
mysql-init
pod is also deleted.[student@workstation ~]$ oc get pods NAME READY STATUS RESTARTS AGE db-pod-6ccc485cfc-2lklx 1/1 Running 0 4h2
Create a
query-db
pod by using theoc run
command and theregistry.ocp4.example.com:8443/redhattraining/do180-dbinit
container image. Use the pod to execute a query against the database service.Create the
query-db
pod. Configure the pod to use the MySQL client to execute a query against thedb-pod
service. You can use thedb-pod
service short name, which provides a stable reference.[student@workstation ~]$ oc run query-db -it --rm \ --image registry.ocp4.example.com:8443/redhattraining/do180-dbinit \ --restart Never \ --command -- /bin/bash -c "mysql -uuser1 -pmypa55w0rd --protocol tcp \ -h db-pod -P3306 items -e 'select * from Item;'" Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... +----+-------------------+------------+ | id | description | done | +----+-------------------+------------+ | 1 | Pick up newspaper | 0x00 | | 2 | Buy groceries | 0x01 | +----+-------------------+------------+ pod "query-db" deleted
As the cluster administrator, identify the pod and service subnet ranges.
Log in as the
admin
user with theredhatocp
password.[student@workstation ~]$ oc login -u admin -p redhatocp \ https://api.ocp4.example.com:6443 Login successful. ...output omitted...
Observe the status of the
network-operator
deployment in theopenshift-network-operator
namespace.[student@workstation ~]$ oc get -n openshift-network-operator \ deployment/network-operator NAME READY UP-TO-DATE AVAILABLE AGE network-operator 1/1 1 1 41d
Administrator access is required to view the
openshift-network-operator
namespace.Retrieve the
cluster
network configuration for the cluster network operator. Compare the IP ranges that the pod subnet and the service subnet use.[student@workstation ~]$ oc get network cluster -o json | jq .spec { "clusterNetwork": [ { "cidr": "10.8.0.0/14", "hostPrefix": 23 } ], "externalIP": { "policy": {} }, "networkType": "OVNKubernetes", "serviceNetwork": [ "172.30.0.0/16" ] }
All pods in this cluster are created in the 10.8.0.0/14 range, and all services are created in the 172.30.0.0/16 range.
It might be necessary to use pod-to-pod communications for troubleshooting. Use the
oc run
command to create a pod that executes a network test against the IP address of the database pod.Confirm the IP address of the MySQL database pod. Your pod IP address might differ from the output.
[student@workstation ~]$ oc get pods -o wide NAME READY STATUS RESTARTS AGE IP ... db-pod-6ccc485cfc-2lklx 1/1 Running 0 4h5 10.8.0.69 ...
Capture the IP address in an environment variable.
[student@workstation ~]$ POD_IP=$(oc get pod -l app=db-pod \ -o jsonpath='{.items[0].status.podIP}')
Create a test pod named
shell
with theoc run
command. Execute thetimeout
command to test against the$POD_IP
environment variable and the3306
port for the database.[student@workstation ~]$ oc run shell --env POD_IP=$POD_IP -it --rm \ --image registry.ocp4.example.com:8443/ubi8/ubi --restart Never \ --command -- timeout 5 bash -c "</dev/tcp/$POD_IP/3306" && \ echo "connection success to $POD_IP:3306" || echo "connection failed" Warning: would violate PodSecurity "restricted:v1.24": ...output omitted... pod "shell" deleted connection success to 10.8.0.69:3306
Finish
On the workstation
machine, use the lab
command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises.
[student@workstation ~]$ lab finish deploy-services