EKS Hybride With Terraform: EC2 & Fargate
It has been a long time since I last wrote a new story. There have been a lot of changes, a lot of stuff to manage, and many problems to resolve ๐ .
But today, I am back with a new story where I will be talking about an EKS Hybrid Cluster with EC2 and Fargate, deployed with Terraform.
Context
When you arrive at a new project in a large company with a critical environment that requires maintaining a blueprint, you need to have a clear understanding of what has been previously done before implementing new features to the blueprint.
Furthermore, if you utilize the AWS EKS CLI, it will automatically install all necessary resources, and customization is unnecessary since all resources are deployed automatically ๐๐ผ.
The blueprint in question is an #EKS Blueprint with #EC2, and my task was to add a new feature to include the ability to use #EKS with #Fargate, keeping in mind that the Terraform EKS Module has not been implemented in the blueprint yet.
Prerequisites:
- A good understanding of EKS
- A good understanding of FargateProfile: https://docs.aws.amazon.com/eks/latest/userguide/fargate.html
- A good understanding of Terraform
- A good understanding of AWS resources such as Security Groups and Networking.
There are a lot of tutorials that explain how to deploy EKS with Terraform, such as https://engineering.finleap.com/posts/2020-02-27-eks-fargate-terraform/.
However, in this story, Iโll provide a SOLUTION to a significant problem that isnโt mentioned in these tutorials when using #NodeGroups with #AWSLaunchTemplate and #Fargate ๐ช๐ผ.
4 Steps to Add Fargate :
- Create a Fargate Profile
- Create a Role with principal service: eks-fargate-pods.amazonaws.com
- Create a policy attachment with this role and this policy
arn:aws:iam::aws:policy/AmazonEKSFargatePodExecutionRolePolicy
- Attache the role to the Fargate Profile
You can find all these steps in this Gist
The end ๐๐๐๐๐,
Iโm joking. Now, we will talk about the problem and the solution.
The Problem :
So, with this example, everything normally works fine, assuming that you already have your node groups working. When you deploy a new pod on Fargate, it should work fine ๐.
However, communication between pods on Fargate and pods on EC2 will not be resolved. ๐ฎ
For example, if you run this command from a pod deployed on Fargate, you will experience a DNS timeout
kubectl exec -it [podname] -n fargate-example -- nslookup kube-dns.kube-system.svc.cluster.local
If you run the same command from a pod deployed on an EC2 node to resolve the pod on Fargate, it will work ๐ฑ ๐ก.
Keep in mind that we already have a custom security group attached to the EKS control plane to allow communication between the control plane and nodes.
Solution:
After weeks of debugging and searching through AWS Documentation, we contacted AWS to see what the solution would be.๐
Be ready, itโs too simple but more difficult to find: When Terraform deploys the EKS Cluster, EKS will create a default security group, and we must attach this default security group to our managed nodes, especially when our node groups use a launch template to which we attach security groups.
To do so :
Conclusion :
โWhen you have a hybrid cluster with EC2 and Fargate, and you want to resolve DNS between pods deployed on Fargate and others on EC2, always have the reflex to attach the default security group of your cluster to your node groups, especially when you customize the security group attached to your launch template.โ
magic word ๐คช : aws_eks_cluster.eks_cluster.vpc_config.0.cluster_security_group_id
For more stories, please subscribe and interact. If you have another subject in mind that you would like me to explain, please feel free to put it in the comments.๐๐ผ
See you โ๐ผ