This POC demonstrates the integration of Amazon EKS with the Application Load Balancer (ALB) Ingress Controller. The primary goal is to expose multiple applications through a single ALB efficiently. This setup utilizes ExternalDNS in conjunction with Route53 to dynamically create DNS records using Ingress annotations within Kubernetes.

The architecture also includes SSL termination, leveraging the certificates generated and managed by AWS Certificate Manager (ACM). By employing the ALB Ingress Controller, ingress resources, ExternalDNS, and Route53, this setup streamlines the process of exposing services running on EKS clusters while ensuring secure and manageable access to these applications via a unified Application Load Balancer.

Table of contents

Requirements

• AWS Account
• EKS Cluster
• Eksctl
• Helm

Prerequisites

Set Variables

> export CLUSTER_NAME=maetadata-dev-poc-upgrade
> export REGION=us-east-1
> export AWS_ACCOUNT=571210339042

Deploy AWS Load Balancer Controller

This controller manages the AWS ELBs for K8S Clusters. Allows us to create two ELB types:

• ALB: When we creates an ingress
• NLB: When we creates a LoadBalancer service

1. Create an IAM Policy

Download an IAM policy to allows the AWS LB Controller to make AWS APIs calls

> curl -o iam_policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.6.2/docs/install/iam_policy.json
> aws iam create-policy \
--policy-name AWSLoadBalancerControllerIAMPolicy \
--policy-document file://iam_policy.json
{
    "Policy": {
        "PolicyName": "AWSLoadBalancerControllerIAMPolicy",
        "PolicyId": "ANPA5MP44VXIIJWEJH4FT",
        "Arn": "arn:aws:iam::920190234064:policy/AWSLoadBalancerControllerIAMPolicy",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 0,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "CreateDate": "2023-12-05T12:14:08+00:00",
        "UpdateDate": "2023-12-05T12:14:08+00:00"
    }
}

2. Create an IAM Role

Enable IAM Open ID

> eksctl utils associate-iam-oidc-provider --region=us-east-1 --cluster=maetadata-dev-poc-upgrade --approve
2023-12-05 13:15:27 [ℹ]  will create IAM Open ID Connect provider for cluster "maetadata-dev-poc-upgrade" in "us-east-1"
2023-12-05 13:15:28 [✔]  created IAM Open ID Connect provider for cluster "maetadata-dev-poc-upgrade" in "us-east-1"

Create a K8S Service Account and annotate with the name of the IAM Role

> eksctl create iamserviceaccount \
  --cluster=$CLUSTER_NAME \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name "AmazonEKSLoadBalancerControllerRole" \
  --attach-policy-arn=arn:aws:iam::$AWS_ACCOUNT:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve 

Install the AWS LB Controller using Helm

> helm repo add eks https://aws.github.io/eks-charts
> helm repo update
> helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=$CLUSTER_NAME \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller
  
NAME: aws-load-balancer-controller
LAST DEPLOYED: Tue Dec  5 13:18:03 2023
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
AWS Load Balancer controller installed!

Verify the controller installed

> kubectl get deployment -n kube-system aws-load-balancer-controller
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
aws-load-balancer-controller   2/2     2            2           46s

Setup External DNS

external-dns provisions DNS records based on the host information. This project will setup and manage records in Route 53 that point to controller deployed ALBs.

1. Set variables

• DOMAIN: The domain we configured on Route53
• TXT-ID: Is the zone id

Get the zone id, copy the command output

> aws route53 list-hosted-zones | jq -r '.HostedZones| .[] | .Id' | cut -f 3 -d /
Z05760333R4ZV2HW71X62

Configure variables

> export DOMAIN=nahuelhernandez.com
> export TXT_OWNER=Z05760333R4ZV2HW71X62

Get the OIDC and export in one variable

> aws eks describe-cluster --name $CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text --region us-east-1 | cut -f 5 -d /
D9BEEBAF076FCF7C5B1B1E8BF45C08DC
> export OIDC_ID=D9BEEBAF076FCF7C5B1B1E8BF45C08DC
> export AUD="$OIDC_ID":aud

Associate IAM OIDC Provider. In this case it’s already configured with Eksctl, but is useful to know if you are doing manually or with Terraform.

> eksctl utils associate-iam-oidc-provider --region=$REGION --cluster=$CLUSTER_NAME --approve

Create the trust policy

cat << EOF > trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::$AWS_ACCOUNT:oidc-provider/oidc.eks.$REGION.amazonaws.com/id/$OIDC_ID"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/$AUD": "sts.amazonaws.com"
        }
      }
    }
  ]
}
EOF

Checking the generated trust-policy file

> cat trust-policy.json 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::920190234064:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/D9BEEBAF076FCF7C5B1B1E8BF45C08DC"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/D9BEEBAF076FCF7C5B1B1E8BF45C08DC:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

Setup IAM permissions

Set up IAM permissions to give the ExternalDNS pod permissions to create, update, and delete Route 53 records in your AWS account.

Create the iam role.

> aws iam create-role --role-name eks-to-route53 --assume-role-policy-document file://trust-policy.json 

Attach the iam role, with the policy name

aws iam put-role-policy --role-name eks-to-route53 --policy-name eks-to-route53 --policy-document file://eks-to-route53.json

Checking the create IAM policy

> aws iam list-role-policies --role-name eks-to-route53 
{
    "PolicyNames": [
        "eks-to-route53"
    ]
}

References

• https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.6/how-it-works/
• https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html