The cloud revolution has fundamentally reshaped how we manage and monitor applications. With services like Amazon EKS (Elastic Kubernetes Service), organizations can efficiently run Kubernetes without the complexities of managing control plane infrastructures. However, managing logs from the EKS control plane can still be a challenge. This guide will explain an effective way to stream EKS control plane logs directly to Amazon S3 using Fluentd, allowing for scalable storage and analysis.

Why Stream EKS Control Plane Logs?

Effective log management is critical for troubleshooting, monitoring, and compliance. EKS control plane logs provide valuable insights for diagnosing cluster issues, enhancing security, and maintaining operational excellence.

Here are a few reasons why you should stream EKS control plane logs to Amazon S3:

• Durability and Availability: Amazon S3 offers 99.999999999% durability of objects over a given year, ensuring that critical logs are securely stored and retrievable.
• Scalability: With Amazon S3, you don’t have to worry about storage limits, making it perfect for growing log files.
• Cost Efficiency: S3’s pay-as-you-go model allows for cost-effective storage, especially when coupled with tiered storage options.
• Centralized Storage: Storing logs in S3 centralizes them, facilitating integration with platforms like Amazon Athena for querying and AWS Lambda for automated processing.

What is Fluentd?

Fluentd is an open-source log collector that allows you to unify data collection and consumption for better analytics and monitoring. Fluentd enables you to collect logs from various data sources, transform them, and route them to different outputs, including Amazon S3.

Some of the key benefits of Fluentd include:

• Unified Logging Layer: Collect and manage all logs with a single platform.
• Data Transformation: Fluentd provides a flexible way to transform collected data before routing it to destinations.
• Extensibility: With a rich plugin ecosystem, Fluentd can easily integrate into any logging infrastructure.

Prerequisites

Before beginning the setup, ensure you have the following:

• An existing EKS Cluster with version 1.21 or higher.
• An IAM user/role with the necessary permissions to create and manage AWS resources like S3 buckets and IAM policies.
• Fluentd installed and configured on your logging host.
• A configured Amazon S3 Bucket for storing the logs.

Step 1: Configuring EKS Control Plane Logs

Start by enabling the EKS control plane logs.

• Open the Amazon EKS console.
• In the navigation pane, select Clusters.
• Select the cluster for which you want to enable logging.
• On the Logging tab, select the checkboxes for control plane logging types you want:
  • API Server
  • Audit
  • Authenticator
  • Controller Manager
  • Scheduler
• Click on Save.

Step 2: Setting Up Fluentd

Install Fluentd

For most environments, install Fluentd with the following commands:

# Install Fluentd (Ubuntu Example)
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent2.sh | sh

Configure Fluentd to Send Logs to S3

You need to configure Fluentd to use the S3 plugin to store logs. Assuming you have the `fluent-plugin-s3` installed, configure Fluentd as follows:

# td-agent configuration file (td-agent.conf)


  @type http
  port 9880



  @type s3
  aws_key_id YOUR_AWS_ACCESS_KEY
  aws_sec_key YOUR_AWS_SECRET_KEY
  s3_bucket YOUR_S3_BUCKET_NAME
  path eks/controlplane/
  
    @type file
    path /var/log/fluent/s3
    timekey 1h
    timekey_wait 10m
    timekey_use_utc true
  

Replace `YOUR_AWS_ACCESS_KEY`, `YOUR_AWS_SECRET_KEY`, and `YOUR_S3_BUCKET_NAME` with your actual AWS credentials and bucket name.

Step 3: Ensure IAM Permissions

To allow Fluentd to write logs to your S3 bucket, create an IAM policy with the following permissions:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": "s3:PutObject",
			"Resource": "arn:aws:s3:::YOUR_S3_BUCKET_NAME/*"
		}
	]
}

Attach this policy to the IAM user or role that you’ll be using for Fluentd.

Step 4: Verify Log Streaming

After configuring Fluentd, you can test the setup by triggering events in your EKS cluster and checking if they appear in your S3 bucket. Use the AWS S3 console or CLI to inspect the logs.

Troubleshooting Tips

If you encounter any issues, refer to the Fluentd and AWS logs for more information. Some common issues include:

• Incorrect IAM Permissions: Ensure the IAM role has the necessary AWS S3 permissions.
• Fluentd Configuration: Double-check the configuration files for any syntax errors.
• Network Issues: Ensure that the network setup allows traffic between Fluentd and Amazon S3.

Conclusion

Streaming EKS control plane logs to Amazon S3 using Fluentd provides a robust solution for log management. With this setup, you gain access to a scalable, reliable, and secure log storage solution that can integrate smoothly with other AWS services for further log analysis and processing.

For additional resources, consider exploring the EKS Control Plane Logs documentation or the Fluentd documentation. By leveraging the power of Fluentd and Amazon S3, your organization can achieve better observability and insights into your Kubernetes environment.