Debugging Kubernetes with Kubectl

Introduction

Kubernetes is undeniably a powerful tool as a container infrastructure. However, it is unavoidable that you have to debug your application due to errors, service down, performance issues, or any unexpected behaviors. But don’t worry, `kubectl` can cover most of the situations.

In this tutorial, I will show you several ways to use `kubectl` when you need to debug your Pods or containers. In addition, I will provide some real-world use cases for an easier understanding of how you can apply them during daily operations. Finally, there will be some best practices and tips that give you a smoother debugging process.

Leveraging kubectl Logs

Log is always the most straightforward way of debugging that gives an insight into the issue. No matter whether it is access or error log, it could be a clue to the solution. You simply use the `kubectl logs` command that prints the recent logs. With providing option `-f`, it will print the logs in real-time mode.

In the above example, I have used the `kubectl logs` command to print the `kubernetes-dashboard-auth` Pod logs in both standard output and continuous streaming mode.

You can also combine the `kubectl logs` command with any Bash commands to truncate meaningless logs. For example, use the `grep` command to search if a specific API endpoint exists in the access log.

The following figures are some examples that you can use with `kubectl logs` that can benefit your debugging process.

Use the `grep` command to truncate the output.

Use `>` to store the logs in a text file.

Get the logs of a previous failed Pod.

There are many more patterns available with `kubectl logs` depending on your debugging needs. It is always recommended as the standard output cannot provide exactly what you want in most real-world cases.

Using kubectl Port Forwarding

The `kubectl` port-forwarding is a method to access Kubernetes resources from your local machine. By default, the Kubernetes cluster is an internal network. Without proper network configuration like Service or Ingress, you cannot access the Pods in the cluster from outside the cluster. It also raises security concerns if you simply open your cluster to the public. 

Therefore, this port-forwarding feature establishes a secure tunnel between the cluster and your local machine. With this secure tunnel, you will then be able to access a specific port of the Pod or Service through a designated port in your local machine.

Here are two examples the Jenkins Pod or Service was not accessible through my local machine. Then, I use the `kubectl port-forward` command to build a tunnel between port 8080 of the Pod and port 8123 of my local machine. As a result, I can `curl` the Jenkins Pod or Service using port 8123 locally.

A small tip here, you can run the port-forwarding in the background by adding `&` at the end of the command. You can simply use `Ctrl + C` to return to the command prompt while the port-forwarding is running in the background. To terminate it, just run `fg %1` to bring the process back to the foreground, and `Ctrl + C` to stop it.

kubectl port-forward [pod/service] [name] [local port]:[resource port] 
&fg %1

So, you may ask when I need to use port-forwarding. Here are some real-world use cases,

• Service is down. We need to check if the application is not running correctly or if there is a network misconfiguration that the public-facing routing has been affected.
• Access internal-only services, like a bastion host to reach database servers.
• Testing and debugging new features of deployed but not yet published applications.
• Local development with remote services while some services cannot be set up locally.
• Perform live debugging in the production environment.

To conclude, microservice architecture is more popular nowadays, where an application is composed of multiple services. Port forward becomes particularly useful in debugging inter-pod or inter-service communication.

Interactive Debugging with kubectl Exec

The `kubectl exec` is one of the most powerful among all `kubectl` commands, which allows you to execute commands directly inside a container. You can as simply execute a single command, or as complicated as execute a script file.

During the `kubectl exec` execution, it connects to the cluster API server and locates the containers and Pods, and then an `exec` session will be started. The communication between you and the target container goes through the session. Finally, it handles the commands or scripts you have provided for execution if the context running the command has appropriate privileges and permissions.

The most awesome part is that you can pass a `sh` or any valid interpreter as the command. You will then be in an interactive mode where you can interact with the container in the same terminal. To enable it, make sure the container image contains the necessary interpreter.

Here are some common scenarios of using the `kubectl exec` command.

• Log tailing.

kubectl exec nginx -- tail -f /var/log/nginx/access.log

• SQL command execution.

kubectl exec mysql -- mysql -h localhost:3306 -u user --password=password -e "select * from users;"

• Retrieve environment variables.

kubectl exec apache -- printenv

Using the `kubectl exec` command is more for investigation purposes, as the changes directly done inside the container are not permanent. There is also a use case in which you can make temporary changes to the application code as a quick verification of potential fixes. Once the pod is restarted or recreated, it will reset back to the default state. It should be powerful enough to debug most of the situations.

Diagnose Issues by kubectl Describe

The `kubectl describe` command is a strong command that can give you super detailed information about any Kubernetes resources. Normally, this command could be used in various situations, but we will only focus on its debugging power. If previous `kubectl` commands are for application-level debugging, this command is more for Kubernetes-level debugging.

We mainly focus on the Events section in the output. This section shows all the recent events about the described resources. You can see what actions have been done by which Kubernetes component, and then trace the root cause.

Here is an example of describing a Pod. In the Events section, you can see the Pod failed to schedule due to `ErrImagePull` and `ImagePullBackOff` errors, which were caused by the failure to pull the image. For sure, failing to pull an image could be related to many possibilities. However, the events give you a starting point for finding the root cause.

Best Practice and Tips

The `kubectl` commands have already saved us from having a headache about debugging on the Kubernetes clusters. However, we can still apply some better practices to make the whole process more effective, especially for real-world use cases.

Here I will give you some best practices and tips to apply while debugging on Kubernetes with `kubectl`.

1. Labeling.
   Make sure the resources like Pods, Deployments, and Services are properly labeled, which helps you to identify corresponding resources easily. 
2. Resources isolation by namespace
   Namespace is a logical grouping in Kubernetes. Organize your resources by isolating them into corresponding namespaces. You can debug the application more easily while all the related resources are grouped. 
3. Standardize the log format
   It is common for applications to have different log formats. But it could be confusing or misleading which slows down the debugging process. The better understandable format, the less chance to make mistakes.
4. Automated workflows
   Repeating errors and false positives are annoying during debugging. They could confuse you in finding the real issue. It is also time-consuming to handle those unrelated items repeatedly. You should implement an automated workflow to reduce manual intervention and repeat tasks.
5. Multiple environments
   Roll-out of undetected issues to the production environment is a popular mistake. With multiple environments and proper deployment methods, we can detect issues earlier and debug them in a non-production environment. It prevents us from having service downtime due to the debugging process. And don’t debug directly on the production environment unless necessary.
6. Security control
   The `kubectl` tool is powerful and it allows users to access the data and even modify them during their debugging process. It is convenient before any security issues. You should restrict user permissions using the RBAC system in Kubernetes according to their job functions. Make sure only authorized users can perform actions on the cluster. Also, review the security policies regularly.

Conclusion

In this article, we have covered the topic of debugging on the Kubernetes cluster. I have introduced three powerful `kubectl` commands to help with the debugging process. With the tips and the best practices section, you should have known more about an effective, secure, and reliable way to improve your current workflows. 

Remember, always be cautious while executing any commands.