ChatOps on EKS using BotKube
ChatOps allows us to operate DevOps tools using chat clients. Implementing ChatOps with EKS is painless using BotKube. This tool lets us monitor and debug our K8S cluster. The app integrates with Slack, Teams, etc. So, for example, we could view our K8S deployments with a simple command on Slack or view logs in real-time. Also, give us recommendations for standard practice, such avoid using the latest containers image versions
BotKube is a messaging bot for monitoring and debugging K8S clusters in real-time. Some features are:
- Allow us to execute Kubectl commands on our chat clients.
- By Default uses a ReadOnly Service Account, thus is only possible to execute read commands.
- Can monitor any K8S resource, including Custom Resources (like certificate expiry or backup failure).
- Pretty easy to configure, less than 5 minutes.
- Let us do a quick debug using only our chat client mobile app.
- Deploy on any K8S Flavors (not only EKS)
Table of contents
Requirements
- AWS Account
- EKS Cluster
- Helm
- Slack
Configure BotKube on Slack
We need to install the BotKube Slack Application to our Slack workspace, using the following Link
After installing it, we need to copy the Bot Access Token, we’ll use it in the next step
Now we need to add the BotKube user to a Slack channel, for example, #chatops-eks. To add, it is only to write @BotKube
in the channel
Install BotKube on your EKS cluster
To install BotKube on EKS (or other K8S flavors), we use Helm, and it’s pretty easy. First, we need to configure some extra variables:
> export SLACK_API_BOT_TOKEN="xoxb-8064160117-xyz"
> export CLUSTER_NAME=eks-chatops
> export ALLOW_KUBECTL=true
> export SLACK_CHANNEL_NAME=chatops-eks
Now we can add the helm repo and deploy BotKube
> helm repo add botkube https://charts.botkube.io
> helm repo update
> helm install --version v0.13.0 botkube --namespace botkube --create-namespace \
--set communications.default-group.slack.enabled=true \
--set communications.default-group.slack.channels.default.name=${SLACK_CHANNEL_NAME} \
--set communications.default-group.slack.token=${SLACK_API_BOT_TOKEN} \
--set settings.clusterName=${CLUSTER_NAME} \
--set executors.kubectl-read-only.kubectl.enabled=${ALLOW_KUBECTL} \
botkube/botkube
Where:
- SLACK_CHANNEL_NAME is the channel name where @BotKube is added
- SLACK_API_BOT_TOKEN is the Token you received after installing BotKube app to your Slack workspace
- CLUSTER_NAME is the cluster name set in the incoming messages
- ALLOW_KUBECTL set true to allow kubectl command execution by BotKube on the cluster
BotKube will watch all and notify us for created/deleted resources and error events. Some examples:
Execute kubectl commands from Slack
First, we need to test the BotKube installation. Send @BotKube ping
on your Slack Channel, it will show something like:
pong from cluster 'eks-chatops'
K8s Server Version: v1.23.7-eks-4721010
BotKube version: v0.13.0
Finally, we could execute kubectl
commands using BotKube, for example @BotKube get pods
. We could check all allowed commands with @BotKube commands list
Disable notifications for a specified resource
Sometimes, we have a resource we don’t want to notify us, such as a pipeline that creates a new pod and deletes it after the tasks. To avoid this, we could use Kubernetes Annotations. So, we only need to add the Annotation botkube.io/disable=true
to our resource.
For example, I created a new Nginx pod and the BotKube notified me (also gave me an excellent recommendation to avoid latest
images)
But, when i created the same pod with the annotation. BotKube didn’t say anything.
> kubectl run testing-botkube-annotation --image nginx --annotation botkube.io/disable=true
Conclusion
BotKube is a fantastic tool and pretty easy to install and configure. Sometimes we are far away from our laptops, and we need to check something in our cluster and even do some quick troubleshooting. In this last case, my recommendation is to install also metric server to have real-time info about CPU and memory.
From now, I’ll install BotKube in all my K8S Clusters.