Logging In Kubernetes

Read Time:8 Minute, 58 Second

Applications and systems logging output to some magical location that we end up finding on the operating system or on a system path has been the savior of every Sysadmin and Developer since the beginning of computers.

In Kubernetes, it’s no different.

Without proper logging and overall log consumption, you’ll never be able to troubleshoot what’s happening in your Kubernetes environment.

In this blog post, you’ll learn about what logging is, different logging methodologies in Kubernetes, and how you can implement them in production.

What Is Logging

Have you ever seen an error pop up on your screen when you were trying to install something or use a specific tool/platform?

Have you ever had to go on a computer and look at the output for a particular application?

Those are logs.

Logs contain information about the system, the operating system, the application, and many other aspects as long as wherever you’re viewing the logs support logging. What’s meant by that is every application has the ability to send out logs, but sometimes they don’t. It’s all up to the developer to implement the functionality to save/output logs for the app.

Logging has been a lifesaver for almost everyone in the technology space. Whether you’re a Sysadmin, a cloud engineer, a developer, or anything in between, it’s almost guaranteed that you’ve looked at logs at some point to solve a specific problem.

In short, logs are the events that occur on a computer or an app.

Logs In Kubernetes

In Kubernetes, you’re going to care about two different types of logs:

  • Cluster logs
  • Kubernetes resource/object logs

Cluster logs are in regard to how to cluster is performing. If the control plane is healthy if the worker nodes are healthy, and depending on how they’re installed, how the Kubernetes components are performing (Etcd, Scheduler, API Server, etc.). For example, in standard Kubeadm deployments, Etcd would run as a Pod, so it would be considered a “Kubernetes resource/object log”. However, if Etcd is configured across clusters running on its own server, it may not be bootstrapped as a Pod, and therefore would be considered a “cluster log”.

Kubernetes resource/object logs are any Kubernetes resource that’s running. You can get logs and events for various Kubernetes resources, but the bulk of what engineers usually look at are Pod and container logs as this is the primary location to give you information on the running application.

The quickest way to check logs for Pods is with the kubectl logs pod_name command. However, kubectl logs only work to check logs for Pods and no other Kubernetes resource.

Different Types Of Logging Methods In Kubernetes

When you’re thinking about your logging strategy for a Kubernetes environment, there are a few different ways that you can output logs:

  • Application Forwarding
  • Sidecar
  • Node Agent Forward

Let’s break these down.

Application Forwarding is done inside of the application code. For example, let’s say you’re writing a fronted app. Inside of the app, you can specify logic that says “send the logs from here to this system”. Although that may sound straightforward, it’s arguably the worst way to manage logs. Why? Because you’re putting a dependency on the code and the logging system. What if the logging system changes? Are you going to implement this for each and every app and piece of each app? This method should be avoided at all times unless you have some time of ridiculously compelling reason to you it. For example, a compelling reason to do this is if you have a legacy application that already has the log functionality embedded into it (although, you should be planning how to change this functionality for the future).

The sidecar method would be if you implement a log aggregator into a Kubernetes Pod. You have end up having two containers in the same Pod. One of the containers is the app itself and the other container is the logging system you’re using. This is an “alright” method, and should definitely be used instead of Application Forwarding, but there’s a better method – Node Agent Forwarding.

The Node Agent Forwarding method is a Pod that runs on each Kubernetes Worker Node. The job of the Pod is to read the log files of the containerized app and send them to whichever logging tool/platform you’re using. This is the best method for a few reason. 1) You aren’t implementing a sidecar which removes extra functionality for a Pod which should technically be doing one job. 2) Segregates the workload processes vs having dependencies and in turn you have one Kubernetes Resource (the Pod) that’s doing one job (sending logs).

Kubernetes Audit Logs

The thing about logs is that logs can be considered pretty much any output. Events, app logs, terminal output, exit events, and a lot of other pieces. Because of that, we can’t cover everything without this blog post turning into a whitepaper, so let’s stick with the most important.

To kick off the hands-on portion of this blog post, you’ll start with Kubernetes audit logs.

Kubernetes audit logs give you the ability to capture and view any output that the Kubernetes Resource creates. For example, when you create a Kubernetes Deployment, a lot is going on.

  • The container image is being pulled from a registry.
  • The Pods are being scheduled on worker nodes.
  • The Pods are either starting or failing and if they’re failing, there’s a reason as to why.
  • Scaling on the Pods is done.

Because of that detail, you may want to specify to collect certain logs or maybe you want all of them.

For example, below is the Policy Kubernetes Resource being used to create an audit policy which literally collects everything that it possibly can from every single Kubernetes Resource.

Enter fullscreen mode Exit fullscreen mode

First, let’s see how this is done in an on-prem Kubernetes cluster. For example, bootstrapped with Kubeadm as it’s quite different compared to cloud services. Then, you’ll see how auditing is configure for cloud services.


First, create a new YAML for the policy that you want to create.

Enter fullscreen mode Exit fullscreen mode

Next, specify a policy. For example, you can use the policy below which collects audit logs for Pods and anyone that’s authenticating to the API.

Enter fullscreen mode Exit fullscreen mode

Once saved, you need to update the API server. When bootstrapping a Kubernetes cluster with Kubeadm, all Manifests for the Pods that make up the control plane are under /etc/kubernetes/manifests.

Open up the kube-apiserver.yaml Manifest.

Enter fullscreen mode Exit fullscreen mode

Save the following lines under the kube-apiserver configuration which will specify where you want the logs to go and the policy that you created above.

Enter fullscreen mode Exit fullscreen mode

Logging In Kubernetes

Next, specify the audit policy and the log as a mount point.

Enter fullscreen mode Exit fullscreen mode

Image description

The last step is to specify the volumes host path for the mount paths that you created previously.

Enter fullscreen mode Exit fullscreen mode

Image description

Once complete, restart the Kubelet.

Enter fullscreen mode Exit fullscreen mode

You should now be able to see the logs at the following location.

Enter fullscreen mode Exit fullscreen mode

Azure Kubernetes Service

By default, auditing is enabled on AKS and an Audit Policy is in place to collect everything from all Kubernetes resources.

However, the functionality to view the audit logs isn’t on by default. To turn it on, do the following.

First, go to your AKS cluster and under Monitoring, click on Diagnostic settings. Then, click the blue + Add diagnostic setting button.

Image description

Under the Diagnostic settings, click on the Kubernetes Audit category. At this point, you’ll have an option on where you want to save the logs. For the purposes of this section, you can choose the Send to Log Analytics workspace option.

Image description

Now that the auditing is turned on, you can open up the analytics workspace and run a query to retrieve the events.

For example, the below query will query everything for Kubernetes audit logs.

Enter fullscreen mode Exit fullscreen mode

You can see an example of the output below.

Image description

AWS Elastic Kubernetes Service

For AWS, audit logging is not on by default. You can turn it on when you’re creating a Kubernetes cluster and then the audit logs will go to CloudWatch by default.

Image description

Kubernetes Events

In this section, you’re going to learn how to retrieve Kubernetes Events, which are, in a way, logs in themselces.

An event is a Kubernetes resource/object. that occurs when a change happens with another Kubernetes resource/object whether it’s Pods, Services, Nodes, etc…

You can run the kubectl get events command to see what’s happening inside of a cluster.

Enter fullscreen mode Exit fullscreen mode

However, there isn’t a whole lot going on because the kubectl get events command was ran on a new cluster. Let’s throw some traffic at it to generate some events.

Run the following YAML which will deploy an Nginx Kubernetes Deployment.

Enter fullscreen mode Exit fullscreen mode

If you run the kubectl get events command again, you’ll see a few more events as the Pods for the Nginx Deployment were created and the Nginx container image was pulled down for the Docker registry.

Enter fullscreen mode Exit fullscreen mode

As you can imagine, running kubectl get events will end up retrieving a ton of data depending on how new the cluster is.

To get a more specific event list, you can target a specific Kubernetes Resource.

In the example below, you can see that the describe command is used to see the events (and many other components) of the Kubernetes Deployment.

Enter fullscreen mode Exit fullscreen mode

If you scroll down on the output of the describe command, you’ll see an output similar to the output below which shows the events for the resource.

Enter fullscreen mode Exit fullscreen mode


Logging in Kubernetes is much like any other method of logging – collect events, view them, and utilize them to troubleshoot or understand what’s happening in a system and in an application. The thing to remember here is to ensure that you send the logs to a proper location that can be utilized by the entire team.

Source: https://dev.to/thenjdevopsguy/logging-in-kubernetes-2l7m

Tag Cloud

Java Java Logical Programs OTP Generation in Java python Recursion youtube video ASCII Upper and Lower Case blockchain javascript graph learn to code software development Successful Software Engineers breadth first search Java Array Programs Java Programs Uncategorized android ios programming kotlin web-development django data sql cybersecurity database swiftui serverless aws swift rust react background-position gradients loader mask grid nth-child pseudo elements indieweb WordPress Print Array without brackets C++ factorial Java String Programs Final Keyword Static Variable Axie Infinity Cryptokitties NFT games tool inserting MISC Tips Codes python code python projects python3 system info python project Bigginers How to Do Integrations Payment Gateways PHP checkout page in php Implement stripe payment gateway in Step by step in PHP integrate stripe gatway in php mysql payment gateway integration in php step by step payment gateway integration in php step by step with source code payment gateway integration in website PHP Integrate Stripe Payment Gateway Tutorial PHP shopping cart checkout code shopping cart in php stripe php checkout PHP/MySQL/JSON best international payment gateway does google pay accept international payments how to accept international payments in india paytm payment gateway razorpay codeigniter github razorpay custom checkout github razorpay get payment details razorpay integration in codeigniter github razorpay international payments Razorpay payment gateway integration in CodeIgniter razorpay payment gateway integration in php code Razorpay payment gateway integration with PHP and CodeIgniter Razorpay payment gateway setup in CodeIgniter Library & Frameworks Tips & Tricks UI/UX & Front-end coding birds online html code for google sign in login with google account in PHP login with google account using javascript login with google account using javascript codeigniter login with google account using php login with google account using php source code
🎄 New year open-source giveaway: 0 Amazon Gift Card + Full Swag Kit Previous post 🎄 New year open-source giveaway: $100 Amazon Gift Card + Full Swag Kit
Sentiment prediction with Python Next post Sentiment prediction with Python

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.