Karpenter Scalling

How to Set Up and Configure Karpenter in AWS EKS

master on your achievemnt eg scalling using karpeter

undertand how it works cons of karpenter compoentns alert when new node registered and other pros cons - not support other clouds

Nodepool defines when, under what conditions, nodes should be provisioned like constraints instance type - t3, m series, zone, capacity spot/on demand, architecture arm, AMD, taints, limit (max cpu, memory limit), disruption nodeclassref - which nodeclass to use

NodeClass → defines what type of node to provision define parameters like ami family, security group, subnet, user data, tags, EBS volume

Karpenter is an autoscaler for Kubernetes that provisions right-sized nodes dynamically to match the workload requirements. It eliminates the need for static node groups and works efficiently with AWS EC2 instances.

open-source project designed to enhance node lifecycle management within Kubernetes clusters. It automates provisioning and deprovisioning of nodes based on the specific scheduling needs of pods, allowing efficient scaling and cost optimization. Its main functions are:

• Monitor pods that the Kubernetes scheduler cannot schedule due to resource constraints.

• Evaluate the scheduling requirements (resource requests, node selectors, affinities, tolerations, etc.) of the unschedulable pods.

• Provision new nodes that meet the requirements of those pods.

• Remove nodes when they are no longer needed.

With Karpenter, you can define NodePools with constraints on node provisioning like taints, labels, requirements (instance types, zones, etc.), and limits on total provisioned resources. When deploying workloads, you can specify various scheduling constraints in the pod specifications like resource requests/limits, node selectors, node/pod affinities, tolerations, and topology spread constraints. Karpenter will then provision right sized nodes based on these specifications.

Reasons to use Karpenter

Before the launch of Karpenter, Kubernetes users relied primarily on Amazon EC2 Auto Scaling groups and the Kubernetes Cluster Autoscaler

(CAS) to dynamically adjust the compute capacity of their clusters. With Karpenter, you don’t need to create dozens of node groups to achieve the flexibility and diversity you get with Karpenter. Unlike CAS, Karpenter is not as tightly coupled to Kubernetes versions and doesn’t require you to jump between AWS and Kubernetes APIs.

Karpenter consolidates instance orchestration responsibilities within a single system, which is simpler, more stable and cluster-aware. Karpenter was designed to overcome some of the challenges presented by Cluster Autoscaler by providing simplified ways to:

• Provision nodes based on workload requirements.

• Create diverse node configurations by instance type, using flexible NodePool options. Instead of managing many specific custom node groups, Karpenter could let you manage diverse workload capacity with a single, flexible NodePool.

• Achieve improved pod scheduling at scale by quickly launching nodes and scheduling pods.

---

Step 1: Install Required CLI Tools

Ensure you have the following tools installed:

• AWS CLI

• kubectl

• eksctl

• Helm

You can install them using:

# Install eksctl
curl -sSL "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /usr/local/bin

# Install Helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

---

Step 2: Create an EKS Cluster (If Not Already Created)

If you don’t have an existing EKS cluster, create one using eksctl:

eksctl create cluster --name my-cluster --region us-east-1 --version 1.27 --without-nodegroup

Since Karpenter provisions nodes dynamically, we create an EKS cluster without a node group.

---

Step 3: Associate IAM OIDC Provider

Karpenter requires an OIDC identity provider for authentication. Run:

eksctl utils associate-iam-oidc-provider --region us-east-1 --cluster my-cluster --approve

Verify the OIDC provider:

aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text

---

Step 4: Create IAM Role for Karpenter

Karpenter needs an IAM role with permissions to manage EC2 instances.

1. Create IAM Policy for Karpenter:

curl -fsSL https://karpenter.sh/docs/aws/karpenter-policy.json | aws iam create-policy --policy-name KarpenterControllerPolicy --policy-document file://-

1. Create IAM Role and Attach the Policy:

eksctl create iamserviceaccount --cluster=my-cluster --namespace=karpenter --name=karpenter --attach-policy-arn=arn:aws:iam::$(aws sts get-caller-identity --query "Account" --output text):policy/KarpenterControllerPolicy --approve --role-only --role-name KarpenterControllerRole

---

Step 5: Install Karpenter on the EKS Cluster

1. Add the Helm repository for Karpenter:

helm repo add karpenter https://charts.karpenter.sh/
helm repo update

1. Install Karpenter using Helm:

helm install karpenter karpenter/karpenter --namespace karpenter --create-namespace \
  --set serviceAccount.create=false \
  --set serviceAccount.name=karpenter \
  --set controller.clusterName=my-cluster \
  --set controller.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile \
  --set controller.aws.interruptionQueueName=karpenter-interruption-queue

---

Step 6: Configure Karpenter Node Provisioning

Karpenter provisions nodes based on a Provisioner.

1. Create a provisioner manifest (provisioner.yaml):

apiVersion: karpenter.k8s.aws/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  provider:
    subnetSelector:
      karpenter.sh/discovery: my-cluster
    securityGroupSelector:
      karpenter.sh/discovery: my-cluster
  limits:
    resources:
      cpu: "100"
      memory: "500Gi"
  ttlSecondsAfterEmpty: 30
  labels:
    team: devops
  requirements:
    - key: "node.kubernetes.io/instance-type"
      operator: In
      values: ["t3.medium", "m5.large"]
    - key: "topology.kubernetes.io/zone"
      operator: In
      values: ["us-east-1a", "us-east-1b"]
---
apiVersion: karpenter.k8s.aws/v1alpha5
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  amiFamily: AL2
  subnetSelector:
    karpenter.sh/discovery: my-cluster
  securityGroupSelector:
    karpenter.sh/discovery: my-cluster
  instanceProfile: KarpenterNodeInstanceProfile

1. Apply the provisioner:

kubectl apply -f provisioner.yaml

---

Step 7: Test Karpenter by Deploying a Pod

Run a test deployment that requests resources:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
    spec:
      containers:
      - name: test-container
        image: public.ecr.aws/nginx/nginx:latest
        resources:
          requests:
            cpu: "0.5"
            memory: "512Mi"

Apply it:

kubectl apply -f test-deployment.yaml

Since no nodes exist, Karpenter will automatically provision an instance.

---

Step 8: Verify Karpenter is Working

1. Check Karpenter logs:

kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter

1. List Nodes Created by Karpenter:

kubectl get nodes

1. Describe the provisioner:

kubectl describe provisioner default

---

Common Issues Faced When Setting Up Karpenter in AWS EKS

❌ 1. No nodes are provisioned even when workloads are scheduled

✅ Fix:

• Check if the Provisioner is correctly configured. Run:

  kubectl get provisioners

• Ensure IAM Role for Karpenter has correct permissions.

• Verify that instance types are available in the region.

---

❌ 2. Karpenter fails with "Insufficient Capacity Error"

✅ Fix:

• Modify instance-type selection in the Provisioner to include more instance types.

• Check AWS EC2 limits in your region:

  aws service-quotas list-service-quotas --service-code ec2

---

❌ 3. Karpenter nodes get terminated immediately

✅ Fix:

• Ensure ttlSecondsAfterEmpty in the Provisioner is set high enough (e.g., 30s).

• Ensure the workload is properly scheduled and using resource requests.

---

❌ 4. Nodes are not joining the cluster

✅ Fix:

• Ensure that karpenter.sh/discovery tag is applied to subnets and security groups.

• Verify that the IAM instance profile has permissions for ec2:DescribeInstances and eks:DescribeCluster.

---

❌ 5. Karpenter Pods are in CrashLoopBackOff

✅ Fix:

• Check logs using:

  kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter

• Ensure karpenter namespace exists and is correctly configured.

• Verify that Helm values match your cluster settings.

---

Final Thoughts

Karpenter is an efficient alternative to Kubernetes Cluster Autoscaler. By following these steps, you can dynamically provision nodes in AWS EKS without manually managing node groups. 🚀

Would you like additional troubleshooting tips or a specific use case covered?