Kubernetes logging architecture with AWS EKS and Elastic Cloud — Elasticsearch, Fluent Bit and Kibana (EFK)

8 min readAug 3, 2021

Logging Architecture

Application logs can help you understand what is happening inside your application. The logs are particularly useful for debugging problems and monitoring cluster activity. Most modern applications have some kind of logging mechanism. Likewise, container engines are designed to support logging. The easiest and most adopted logging method for containerised applications is writing to standard output and standard error streams.

However, the native functionality provided by a container engine or runtime is usually not enough for a complete logging solution. For example, you may want access your application’s logs if a container crashes; a pod gets evicted; or a node dies. In a Kubernetes cluster, logs should have a separate storage and lifecycle independent of nodes, pods, or containers. This concept is called cluster-level logging.

Cluster-level logging architectures require a separate backend to store, analyse, and query logs. Kubernetes does not provide a native storage solution for log data. Instead, there are many logging solutions that integrate with Kubernetes. One popular centralised logging solution is the Elasticsearch, Fluent Bit and Kibana (EFK).

Cluster-level logging architectures

While Kubernetes does not provide a native solution for cluster-level logging, there are several common approaches you could consider. Here are some options:

Use a node-level logging agent that runs on every node
*Include a dedicated sidecar container for logging in an application pod
Push logs directly to a backend from within an application

Using a node-level logging agent

You can implement cluster-level logging by including a node-level logging agent on each Kubernetes cluster node. The logging agent is a dedicated tool that exposes logs or pushes logs to a backend. Commonly, the logging agent is a container that has access to a directory with log files from all of the application containers on that node. In a Kubernetes cluster, the directory should be the /var/log/containers/.

Because the logging agent must run on every cluster node, it is recommended to run the agent as a DaemonSet. Node-level logging creates only one agent per node and doesn’t require any changes to the applications running on the node. Containers write stdout and stderr, but with no agreed format. A node-level agent collects these logs and forwards them for aggregation.

* Based on my experience, sidecar pattern for logging should be applied for smaller-scale setups or in setups without the root access to the cluster nodes to read the logs. I would personally choose Fluent Bit as a DaemonSet because it’s more straightforward to configure, it has a small footprint and it requires less resources.

Elasticsearch, Fluent Bit and Kibana (EFK)

Elasticsearch is a real-time, distributed, and scalable search engine which allows for full-text and structured search, as well as analytics. It is commonly used to index and search through large volumes of logs.

Elasticsearch is commonly deployed alongside Kibana, a data visualisation frontend and dashboard for Elasticsearch. Kibana allows you to explore your Elasticsearch logs through a web interface, and build dashboards and queries to quickly answer questions and gain insight into your Kubernetes applications.

In this article, I will provide all the technical details/specifications for integrating and sending application logs from the AWS Elastic Kubernetes Service (EKS) to the Elastic Cloud https://www.elastic.co/cloud/ managed service (Elasticsearch, Kibana, APM & Fleet) using a node-level logging agent based on Fluent Bit.

Fluent Bit is an open source Log Processor and Forwarder which allows you to collect any data like metrics and logs from different sources, enrich them with filters and send them to multiple destinations. It’s the preferred choice for containerised environments like Kubernetes as it enriches the logs with Kubernetes metadata. Fluent Bit is designed with performance in mind: high throughput with low CPU and Memory usage. Fluent Bit is a CNCF (Cloud Native Computing Foundation) subproject under the umbrella of Fluentd.

Set up — Deploy Elastic Cloud managed service

Using Elasticsearch Service is a great way to take advantage of all the power that Elastic offers. You can start with a free trial account and you can find here https://www.elastic.co/guide/en/cloud/current/ec-getting-started.html all the details for the first steps. The 14-day free trial includes a deployment to explore Elastic solutions for Enterprise Search, Observability, Security, or the latest version of the Elastic Stack.

A deployment should be created first and you can find here https://www.elastic.co/guide/en/cloud/current/ec-create-deployment.html#ec-create-deployment all the details for the deployment configuration. A deployment helps you manage an Elasticsearch cluster and other Elastic products, like Kibana or APM instances, in one place.

You can pick a cloud platform where Elastic hosts your deployment among Amazon Web Services (AWS), Google Cloud Platform (GCP) and Microsoft Azure. Which platform you pick is a personal preference. Elasticsearch Service handles all hosting details for you, no additional cloud accounts required.

You can either customise a new deployment or customise an existing one. On the Create a deployment page, click Edit settings to change the cloud provider, region, hardware profile, and stack version; or click Advanced settings for more complex configuration settings.

You can enable at the beginning the Elasticsearch and Kibana services/nodes at their minimum sizing as you can see below. You could even select an Elasticsearch deployment within a single availability zone but you will be at risk of data loss with fault tolerance set to a single zone.

After creating the deployment you will receive the username/password credentials for this deployment. Please download and save them in your end. This is what you should expect after a successful deployment creation.

Please note the Elasticsearch endpoint (includes the port), the Cloud ID and the Username/Password credentials as you would need them in the Fluent Bit configuration.

e.g Elasticsearch endpoint: https://my-cluster-deployment.es.eu-central-1.aws.cloud.es.io

e.g Elasticsearch port: 9243

e.g Cloud ID: my-cluster-deployment:ZXUtY2VudHJhbC0xLmF3cy5jbG91ZC5lcy5pbyQ1NDkyNTJlOWZkNDY0YTNlOTQxMTc2YmIxMTMyZDE5MCQzNzQyNjNjZjIzNDQ0NGM2YWQ0MjI0NTcyMWNkOWYwOA==

Set up — Deploy Fluent Bit

At this section, you will find all the steps in order to set up Fluent Bit to collect logs from your containers. Using docker, the containers save their log files in the host’s directory /var/log/containers/ with a .log file extension. The log is formatted in JSON because Kubernetes nodes are configured with docker json logging driver. The container log file will have the format {“log”:”<CONTAINER_LOG>”,”stream”:”stdout”,”time”:”<TIMESTAMP>”}.

The following yaml resources are mostly based on the Kubernetes Logging with Fluent Bit Github official repository https://github.com/fluent/fluent-bit-kubernetes-logging but you will find also here https://github.com/gtsopour/fluent-bit-kubernetes-logging all the adaptations I have done on top of the default resources.

Please note that for the execution of all the following commands, the kubectl command line too should have been already configured and connected to your Kubernetes cluster.

Create Kubernetes namespace

Please create a new namespace with the name logging for the Fluent Bit deployment.

#Create a Kubernetes namespace
kubectl create namespace logging

Create RBAC resources for Fluent Bit

Fluent Bit must be deployed as a DaemonSet so that it will be available on every node of the Kubernetes cluster. To get started run the following commands to create the service account and role setup.

#Create ServiceAccount
kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml#Create ClusterRole
kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml#Create ClusterRoleBinding
kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml

Create Fluent Bit ConfigMap

The next step is to create a ConfigMap that will be used by the Fluent Bit DaemonSet.

#Create ConfigMap
kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-configmap.yaml

Regarding the Fluent Bit ConfigMap and as I am using the Elastic Cloud managed service, I had to adapt the es output configuration as below. The es output plugin, allows to ingest your records into an Elasticsearch database. Please you can find here https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch all the instructions about the es plugin configuration parameters. As I am using Cloud Elastic’s Elasticsearch Service, Cloud_ID is the id of the cluster running and Cloud_Auth specifies the credentials to use to connect to Elastic’s Elasticsearch Service.

From

output-elasticsearch.conf: |
 [OUTPUT]
     Name            es
     Match           *
     Host            ${FLUENT_ELASTICSEARCH_HOST}
     Port            ${FLUENT_ELASTICSEARCH_PORT}
     Logstash_Format On
     Replace_Dots    On
     Retry_Limit     False

output-elasticsearch.conf: |
 [OUTPUT]
     Name            es
     Match           *
     Host            ${CLOUD_ELASTICSEARCH_HOST}
     Port            ${CLOUD_ELASTICSEARCH_PORT}
     Cloud_ID        ${CLOUD_ELASTICSEARCH_ID}
     Cloud_Auth      ${CLOUD_ELASTICSEARCH_USER}:${CLOUD_ELASTICSEARCH_PASSWORD}
     Logstash_Format On
     Logstash_Prefix my-cluster
     Replace_Dots    On
     Retry_Limit     False
     tls             On
     tls.verify      Off

You can also find here the adapted fluent-bit-configmap.yaml https://raw.githubusercontent.com/gtsopour/fluent-bit-kubernetes-logging/master/fluent-bit-configmap.yaml and you can apply that using the following command.

#Create ConfigMap
kubectl create -f https://raw.githubusercontent.com/gtsopour/fluent-bit-kubernetes-logging/master/fluent-bit-configmap.yaml

Create Fluent Bit DaemonSet

The next step is to create the Fluent Bit DaemonSet ready to be used with the Elasticsearch on a normal Kubernetes Cluster.

#Create DaemonSet
kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds.yaml

Regarding the Fluent Bit DaemonSet, as I had adapted the ConfigMap before, I had to adapt the env variables section in the DaemonSet and drive their values properly from two new ConfigMap and Secret resources. Please make sure that you apply the values collected from the Elastic Cloud deployment to the resource placeholders. The ConfigMap and the Secret resources are pre-requisites for the DaemonSet.

<CLOUD_ELASTICSEARCH_ID> = https://my-cluster-deployment.es.eu-central-1.aws.cloud.es.io
<CLOUD_ELASTICSEARCH_HOST> = my-cluster-deployment:ZXUtY2VudHJhbC0xLmF3cy5jbG91ZC5lcy5pbyQ1NDkyNTJlOWZkNDY0YTNlOTQxMTc2YmIxMTMyZDE5MCQzNzQyNjNjZjIzNDQ0NGM2YWQ0MjI0NTcyMWNkOWYwOA==
<CLOUD_ELASTICSEARCH_DEPLOYMENT_USER> = …<CLOUD_ELASTICSEARCH_DEPLOYMENT_PASSWORD> = …

#Elasticsearch ConfigMap
kind: ConfigMap
apiVersion: v1
metadata:
 name: elasticsearch-configmap
 namespace: logging
data:
 CLOUD_ELASTICSEARCH_ID: <CLOUD_ELASTICSEARCH_ID>
 CLOUD_ELASTICSEARCH_HOST: <CLOUD_ELASTICSEARCH_HOST>
 CLOUD_ELASTICSEARCH_PORT: '9243'#Elasticsearch Secret
kind: Secret
apiVersion: v1
metadata:
 name: elasticsearch-secret
 namespace: logging
data:
 CLOUD_ELASTICSEARCH_PASSWORD: <CLOUD_ELASTICSEARCH_DEPLOYMENT_PASSWORD>
 CLOUD_ELASTICSEARCH_USER: <CLOUD_ELASTICSEARCH_DEPLOYMENT_USER>
type: Opaque

#DaemonSet (partial change)

From

env:
 - name: FLUENT_ELASTICSEARCH_HOST
   value: "elasticsearch"
 - name: FLUENT_ELASTICSEARCH_PORT
   value: "9200"

env:
 - name: CLOUD_ELASTICSEARCH_HOST
   valueFrom:
     configMapKeyRef:
       name: elasticsearch-configmap
       key: CLOUD_ELASTICSEARCH_HOST
 - name: CLOUD_ELASTICSEARCH_PORT
   valueFrom:
     configMapKeyRef:
       name: elasticsearch-configmap
       key: CLOUD_ELASTICSEARCH_PORT
 - name: CLOUD_ELASTICSEARCH_ID
   valueFrom:
     configMapKeyRef:
       name: elasticsearch-configmap
       key: CLOUD_ELASTICSEARCH_ID
 - name: CLOUD_ELASTICSEARCH_USER
   valueFrom:
     secretKeyRef:
       name: elasticsearch-secret
       key: CLOUD_ELASTICSEARCH_USER
 - name: CLOUD_ELASTICSEARCH_PASSWORD
   valueFrom:
     secretKeyRef:
       name: elasticsearch-secret
       key: CLOUD_ELASTICSEARCH_PASSWORD

You can also create all the above resources referring them directly from this https://github.com/gtsopour/fluent-bit-kubernetes-logging Github repository.

#Create Elasticsearch ConfigMap
kubectl create -f https://raw.githubusercontent.com/gtsopour/fluent-bit-kubernetes-logging/master/elasticsearch-configmap.yaml#Create Elasticsearch Secret
kubectl create -f https://raw.githubusercontent.com/gtsopour/fluent-bit-kubernetes-logging/master/elasticsearch-secret.yaml#Create DaemonSet
kubectl create -f https://raw.githubusercontent.com/gtsopour/fluent-bit-kubernetes-logging/master/fluent-bit-ds.yaml

If you create successfully the DaemonSet, you should get something like this below having a Fluent Bit pod for each of your Kubernetes nodes.

Kibana Index Management/Pattern

Kibana requires an index pattern to access the Elasticsearch data that you want to explore. An index pattern selects the data to use and allows you to define properties of the fields.

An index pattern can point to a specific index, for example, your log data from yesterday, or all indices that contain your data. Please find here https://www.elastic.co/guide/en/kibana/current/index-patterns.html all the details about the Kibana index patterns.