Guided Exercise: Troubleshoot Containers and Pods
Troubleshoot and fix a failed MySQL pod and manually initialize a database with test data.
Outcomes
Investigate errors with creating a pod.
View the status, logs, and events for a pod.
Copy files into a running pod.
Connect to a running pod by using port forwarding.
As the student
user on the workstation
machine, use the lab
command to prepare your system for this exercise.
This command ensures that the cluster and all exercise resources are available.
[student@workstation ~]$ lab start pods-troubleshooting
Procedure 3.3. Instructions
Log in to the OpenShift cluster and create the
pods-troubleshooting
project.Log in to the OpenShift cluster as the
developer
user with theoc
command.[student@workstation ~]$ oc login -u developer -p developer \ https://api.ocp4.example.com:6443 ...output omitted...
Create the
pods-troubleshooting
project.[student@workstation ~]$ oc new-project pods-troubleshooting ...output omitted...
Create a MySQL pod called
mysql-server
with theoc run
command. Use theregistry.ocp4.example.com:8443/rhel9/mysql-80:1228
container image for the pod. Specify the environment variables with the following values:| Variable | Value | | --- | --- | |
MYSQL_USER
|redhat
| |MYSQL_PASSWORD
|redhat123
| |MYSQL_DATABASE
|world
|Then, view the status of the pod with the
oc get
command.Create the
mysql-server
pod with theoc run
command. Specify the environment values with the--env
option.[student@workstation ~]$ oc run mysql-server \ --image registry.ocp4.example.com:8443/rhel9/mysql-80:1228 \ --env MYSQL_USER=redhat \ --env MYSQL_PASSWORD=redhat123 \ --env MYSQL_DATABASE=world pod/mysql created
After a few moments, retrieve the status of the pod. Execute the
oc get pods
command a few times to view the status updates for the pod.[student@workstation ~]$ oc get pods NAME READY STATUS RESTARTS AGE mysql-server 0/1 ErrImagePull 0 30s
[studet@workstation ~]$ oc get pods NAME READY STATUS RESTARTS AGE mysql-server 0/1 ImagePullBackoff 0 45s
The pod failed to start.
Identify the root cause of the pod's failure.
Retrieve the pod's logs with the
oc logs
command.[student@workstation ~]$ oc logs mysql-server Error from server (BadRequest): container "mysql-server" in pod "mysql-server" is waiting to start: trying and failing to pull image
The logs state that the pod cannot pull the container image.
Retrieve the events log with the
oc get events
command.student@workstation ~]$ oc get events LAST SEEN TYPE REASON OBJECT MESSAGE 33s Normal Scheduled pod/mysql-server Successfully assigned pods-troubleshooting/mysql-server to master01 by master01 31s Normal AddedInterface pod/mysql-server Add eth0 [10.8.0.68/23] from ovn-kubernetes 16s Normal Pulling pod/mysql-server Pulling image "registry.ocp4.example.com:8443/rhel9/mysql-80:1228" 16s Warning Failed pod/mysql-server Failed to pull image "registry.ocp4.example.com:8443/rhel9/mysql-80:1228": rpc error: code = Unknown desc = reading manifest 1228 in registry.ocp4.example.com:8443/rhel9/mysql-80: manifest unknown: manifest unknown 16s Warning Failed pod/mysql-server Error: ErrImagePull 4s Normal BackOff pod/mysql-server Back-off pulling image "registry.ocp4.example.com:8443/rhel9/mysql-80:1228" 4s Warning Failed pod/mysql-server Error: ImagePullBackOff
The output states that the image pull failed because the
1228
manifest is unknown. This failure could mean that the manifest, or image tag, does not exist in the repository.Inspect the available manifest in the
registry.ocp4.example.com:8443/rhel9/mysql-80
container repository. Authenticate to the container repository with theskopeo login
command as thedeveloper
user with thedeveloper
password. Then, use theskopeo inspect
command to retrieve the available manifests in theregistry.ocp4.example.com:8443/rhel9/mysql-80
repository.[student@workstation ~]$ skopeo login registry.ocp4.example.com:8443 Username: developer Password: developer Login Succeeded!
[student@workstation ~]$ skopeo inspect \ docker://registry.ocp4.example.com:8443/rhel9/mysql-80 ...output omitted... "Name": "registry.ocp4.example.com:8443/rhel9/mysql-80", "Digest": "sha256:d282...f38f", "RepoTags": [ "1-237", "1-228", "1-228-source", "1-224", "1-224-source", "1", "latest ], ...output omitted...
The
1228
manifest, or tag, is not available in the repository, which means that theregistry.ocp4.example.com:8443/rhel9/mysql-80:1228
image does not exist. However, the1-228
tag does exist.
The pod failed to start because of a typing error in the image tag. Update the pod's configuration to use the
registry.ocp4.example.com:8443/rhel9/mysql-80:1-228
container image. Confirm that the pod is re-created after editing the resource.Edit the pod's configuration with the
oc edit
command. Locate the.spec.containers.image
object. Update the value to theregistry.ocp4.example.com:8443/rhel9/mysql-80:1-228
container image and save the change.[student@workstation ~]$ oc edit pod/mysql-server
...output omitted... apiVersion: v1 kind: Pod metadata: ...output omitted... spec: containers: - image: registry.ocp4.example.com:8443/rhel9/mysql-80:1-228 ...output omitted...
Verify the status of the
mysql-server
pod with theoc get
command. The pod's status might take a few moments to update after the resource edit. Repeat theoc get
command until the pod's status changes.[student@workstation ~]$ oc get pods NAME READY STATUS RESTARTS AGE mysql-server 1/1 Running 0 10m
The
mysql-server
pod successfully pulled the image and created the container. The pod now shows aRunning
status.
Prepare the database on the
mysql-server
pod. Copy the~/DO180/labs/pods-troubleshooting/world_x.sql
file to themysql-server
pod. Then, connect to the pod and execute theworld_x.sql
file to populate theworld
database.Use the
oc cp
command to copy theworld_x.sql
file in the~/DO180/labs/pods-troubleshooting
directory to the/tmp/
directory on themysql-server
pod.[student@workstation ~]$ oc cp ~/DO180/labs/pods-troubleshooting/world_x.sql \ mysql-server:/tmp/
Confirm that the
world_x.sql
file is accessible within themysql-server
pod with theoc exec
command.[student@workstation ~]$ oc exec -it pod/mysql-server -- ls -la /tmp/ total 0 drwxrwxrwx. 1 root root 22 Nov 4 14:45 . dr-xr-xr-x. 1 root root 50 Nov 4 14:34 .. -rw-rw-r--. 1 1000680000 root 558791 Nov 4 14:45 world_x.sql
Connect to the
mysql-server
pod with theoc rsh
command. Then, log in to MySQL as theredhat
user with theredhat123
password.[student@workstation ~]$ oc rsh mysql-server
sh-5.1$ mysql -u redhat -p Enter password: redhat123 Welcome to the MySQL monitor. Commands end with ; or \g. ...output omitted... mysql>
From the MySQL prompt, select the
world
database. Source theworld_x.sql
script inside the pod to initialize and populate theworld
database.mysql> USE world; Database changed
mysql> SOURCE /tmp/world_x.sql; ...output omitted...
Execute the
SHOW TABLES;
command to confirm that the database now contains tables. Then, exit the database and the pod.mysql> SHOW TABLES; ----------------- | Tables_in_test | ----------------- | city | | country | | countryinfo | | countrylanguage | ----------------- 4 rows in set (0.00 sec)
mysql> exit; Bye sh-5.1$ exit exit [student@workstation ~]$
Configure port forwarding and then use the MySQL client on the
workstation
machine to connect to theworld
database on themysql-server
pod. Confirm that you can access data within theworld
database from theworkstation
machine.From the
workstation
machine, use theoc port-forward
command to forward the3306
local port to the3306
port on themysql-server
pod.[student@workstation ~]$ oc port-forward mysql-server 3306:3306 Forwarding from 127.0.0.1:3306 -> 3306 Forwarding from [::1]:3306 -> 3306
Open another terminal window on the
workstation
machine. Connect to theworld
database with the local MySQL client on theworkstation
machine. Log in as theredhat
user with theredhat123
password. Specify the host as the localhost127.0.0.1
IP address and use3306
as the port.[student@workstation ~]$ mysql -u redhat -p -h 127.0.0.1 -P 3306 Enter password: redhat123 Welcome to the MySQL monitor. Commands end with ; or \g. ...output omitted... mysql>
Select the
world
database and execute theSHOW TABLES;
command.mysql> USE world; Database changed
mysql> SHOW TABLES; ----------------- | Tables_in_test | ----------------- | city | | country | | countryinfo | | countrylanguage | ----------------- 4 rows in set (0.01 sec)
Confirm that you can retrieve data from the
country
table. Execute theSELECT COUNT(*) FROM country
command to retrieve the number of countries within thecountry
table.mysql> SELECT COUNT(*) FROM country; ---------- | COUNT(*) | ---------- | 239 | ---------- 1 row in set (0.01 sec)
Exit the database.
mysql> exit; Bye [student@workstation ~]$
Return to the terminal that is executing the
oc port-forward
command. Press Ctrl+C to end the connection.[student@workstation ~]$ oc port-forward mysql-server 3306:3306 Forwarding from 127.0.0.1:3306 -> 3306 Forwarding from [::1]:3306 -> 3306 Handling connection for 3306 Handling connection for 3306 ^C[student@workstation ~]$
Finish
On the workstation
machine, use the lab
command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises.
[student@workstation ~]$ lab finish pods-troubleshooting