Stop One EKS Node Group From Blocking Scale-Down in Another

If idle EKS nodes keep running while other node groups scale up around them, Cluster Autoscaler's global scale-down cooldown is the likely cause, and --scale-down-delay-type-local=true is the fix. The flag makes Cluster Autoscaler track post-scale-up cooldowns per node group instead of cluster-wide, so a GPU pool adding capacity no longer resets the clock on an idle node in an unrelated memory pool. The default is false, which means every cluster running multiple ASGs has this behavior until someone turns it off.

We hit this with Dagster. Jobs used separate pools for general work, large CPU and memory tasks, and GPU tasks, each with min_size = 0, so we expected an unused pool to disappear on its own schedule. It didn't. A job starting in one pool delayed scale-down in every other pool, and with jobs starting and ending throughout the day, nodes that were no longer needed stayed alive far longer than they should have.

Why idle nodes stay running

Take two node groups, one for memory-heavy batch jobs and one for GPU jobs, both with min_size = 0. At noon the last memory-heavy job completes, its node goes empty, and Cluster Autoscaler starts the normal waiting period before deletion. Ten minutes later that node has waited long enough - and at almost the same moment a GPU job appears, its pod can't schedule, and Cluster Autoscaler grows the GPU Auto Scaling Group. You'd expect the memory node to disappear now: it belongs to another group, has no useful work, and has already passed its own waiting period. It stays.

Time	Memory group	GPU group	Result
12:00	Last job ends	Idle	Memory node becomes unneeded
12:10	Node has waited 10 minutes	New job triggers scale-up	Global cooldown starts
12:11	Node is eligible for deletion	New node is starting	Memory node cannot scale down
12:20	Node has been idle 20 minutes	Cooldown expires	Deletion can proceed

One extra ten-minute wait doesn't look serious. Bursty workloads turn it into a loop: a CPU pool grows, then a GPU pool grows, then a general pool grows, and each event postpones unrelated scale-down work. The problem is easy to miss because every node group looks correctly configured - minimum size is zero, jobs finish, pods disappear, and the autoscaler still reports nodes as unneeded. Yet the EC2 instances stay active, and nothing in your Terraform or your Helm values looks wrong. We didn't identify the shared cooldown from configuration alone; it took finding upstream issue #4872, which describes the same cross-group delay.

What Cluster Autoscaler actually tracks

Cluster Autoscaler treats an AWS Auto Scaling Group as a Kubernetes node group and decides both when to grow that group and when nodes can be safely removed from it. Scale-down has two separate waiting mechanisms that are easy to confuse: one governs each node's eligibility, the other protects the cluster after recent scaling actions.

Setting	Default	Meaning
--scale-down-unneeded-time	10m	How long a node must remain unneeded
--scale-down-delay-after-add	10m	How long scale-down waits after scale-up
--scale-down-delay-after-delete	0s	How long scale-down waits after deletion
--scale-down-delay-after-failure	3m	How long scale-down waits after failed scale-down
--scale-down-delay-type-local	false	Whether post-action delays are per node group

scale-down-unneeded-time answers "has this node been unnecessary for long enough?" - Cluster Autoscaler checks resource requests, movable pods, scheduling rules, PodDisruptionBudgets, and other safety conditions to decide. The scale-down-delay-after-* settings answer a completely different question: "is scale-down allowed at all right now, given recent autoscaler activity?" A node can pass its unneeded timer while a cooldown still blocks deletion, and that's exactly what happens here. Another node group scaling up does not erase the idle node's unneeded history - it adds a separate gate on top of it. Before 2024 that gate was always global, and with the default false it still is. Support for per-group cooldowns arrived through kubernetes/autoscaler#5729, which lets one process remember scaling activity per node group instead of treating every action as cluster-wide.

Enable per-node-group cooldowns

Keep one Deployment managing all node groups and set the flag. If you install Cluster Autoscaler with Helm, the change belongs in your values file:

cloudProvider: aws

autoDiscovery:
  clusterName: production-eks

awsRegion: eu-west-1

extraArgs:
  scale-down-delay-type-local: "true"
  scale-down-delay-after-add: 10m
  scale-down-unneeded-time: 10m

A GPU group scale-up now starts a cooldown for the GPU group only, and an eligible node in the memory group can be removed while it runs. Keep the delay values at their defaults unless you have measurements that justify changing them - new nodes need time to register, start DaemonSets, advertise GPUs, and receive pending pods, and local scope fixes the isolation problem without removing that protection. Setting scale-down-delay-after-add: 0 is not an equivalent shortcut. It does avoid cross-group delay, but it does so by removing the cooldown everywhere, including from the group that just added nodes, which is the one group that actually needs it.

Check your binary supports the flag

The feature merged into the main branch in January 2024 and was backported to the 1.29 release branch that March through #6484. The original 1.29.0 release does not include it, so do not assume every binary labeled 1.29 supports the flag. Check the actual image and the rendered command:

kubectl -n kube-system get deployment cluster-autoscaler \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

kubectl -n kube-system get deployment cluster-autoscaler \
  -o jsonpath='{range .spec.template.spec.containers[0].command[*]}{.}{"\n"}{end}'

If the logs contain unknown flag: --scale-down-delay-type-local, the image lacks the feature - upgrade the binary rather than quietly dropping the setting. That upgrade has a constraint of its own: Cluster Autoscaler simulates Kubernetes scheduling, so its minor version must match your cluster's Kubernetes minor version, and cross-version combinations are not supported. AWS repeats this in its EKS guidance. One more trap sits between you and the right binary: Helm chart version and Cluster Autoscaler application version are different numbers. Inspect the chart's appVersion or pin image.tag explicitly, so a chart that looks current doesn't deploy a binary that isn't.

Local cooldowns only isolate delays within a single Cluster Autoscaler process, which makes process ownership the next thing to confirm.

Make sure one autoscaler owns the ASGs

The guarantee is worthless if a second, unmanaged autoscaler also watches some of your ASGs, or if one process never discovers all of them. On AWS, auto-discovery selects ASGs by tags, and the standard Helm values look for two keys:

autoDiscovery:
  clusterName: production-eks
  tags:
    - k8s.io/cluster-autoscaler/enabled
    - k8s.io/cluster-autoscaler/{{ .Values.autoDiscovery.clusterName }}

Every ASG the autoscaler should own needs both keys attached before discovery picks it up. Tag values are ignored unless you include them in the discovery expression, but values like true and owned make ownership readable and give you something to write IAM conditions against.

Use separate node groups when capacity or scheduling actually differs, and give each one real labels, real taints, and scale limits that allow it to reach zero. With terraform-aws-modules/eks/aws (~> 21.0), the node group definition is the part that matters:

eks_managed_node_groups = {
  batch = {
    instance_types = ["m7i.2xlarge"]

    min_size     = 0   # allows the group to empty completely
    max_size     = 20
    desired_size = 0

    labels = {
      "node-role" = "batch"
    }

    taints = {
      dedicated = {
        key    = "dedicated"
        value  = "batch"
        effect = "NO_SCHEDULE"
      }
    }
  }
}

The ASG this creates is the resource your discovery tags must reach, and managed node groups generate ASG names you don't control. Module inputs and outputs also shift between versions, so verify the tags actually landed. Tags on launch templates, security groups, or EC2 instances do not count - auto-discovery reads ASG tags and nothing else:

aws autoscaling describe-auto-scaling-groups \
  --query 'AutoScalingGroups[].{Name:AutoScalingGroupName,Tags:Tags[?starts_with(Key, `k8s.io/cluster-autoscaler/`)].{Key:Key,Value:Value}}' \
  --output table

If your module doesn't attach the discovery tags, read the generated ASG name and attach them yourself:

locals {
  batch_asg_name = module.eks.eks_managed_node_groups["batch"].node_group_resources[0].autoscaling_groups[0].name

  autoscaler_tags = {
    "k8s.io/cluster-autoscaler/enabled"        = "true"
    "k8s.io/cluster-autoscaler/production-eks" = "owned"
  }
}

resource "aws_autoscaling_group_tag" "autoscaler" {
  for_each = local.autoscaler_tags

  autoscaling_group_name = local.batch_asg_name

  tag {
    key                 = each.key
    value               = each.value
    propagate_at_launch = false  # this tag is for the ASG itself, not the instances it launches
  }
}

Do not combine static --nodes arguments with auto-discovery. Mixing the two makes ownership hard to reason about and will eventually have you debugging the state of a node group the process was never managing. The AWS provider documentation recommends auto-discovery and warns against the combination. Discovery controls what the process can see; IAM controls what it can change. Use IRSA or EKS Pod Identity and constrain SetDesiredCapacity and TerminateInstanceInAutoScalingGroup to ASGs carrying the cluster tags - AWS publishes a tag-conditioned IAM example.

Scale node groups back up from zero

Scale-to-zero introduces a different failure mode with the same symptom: a pool that sits at the wrong size and won't move. When a node group is empty, Kubernetes has no live Node object advertising that group's labels, taints, or custom resources, and Cluster Autoscaler still has to decide whether a pending pod would fit there. This bites hardest on pools selected by labels, protected by taints, or backed by GPUs.

Cluster Autoscaler infers CPU, memory, and GPU capacity from AWS configuration, and for EKS managed node groups it can also read labels and taints through eks:DescribeNodegroup, including at zero nodes. If that covers everything your pod's scheduling rules reference, you may not need explicit node-template tags at all. When it doesn't, put the metadata on the ASG, where it describes the simulated node used for scale-from-zero decisions:

resource "aws_autoscaling_group_tag" "batch_label" {
  autoscaling_group_name = local.batch_asg_name

  tag {
    key                 = "k8s.io/cluster-autoscaler/node-template/label/node-role"
    value               = "batch"
    propagate_at_launch = false
  }
}

resource "aws_autoscaling_group_tag" "batch_taint" {
  autoscaling_group_name = local.batch_asg_name

  tag {
    key                 = "k8s.io/cluster-autoscaler/node-template/taint/dedicated"
    value               = "batch:NoSchedule"  # value and effect packed into one string
    propagate_at_launch = false
  }
}

Node-template tags do not configure real nodes. They describe a hypothetical node for the simulator, so the same labels and taints must also exist on the managed node group itself - the labels and taints blocks shown earlier - and the pending pod must request that same contract:

spec:
  nodeSelector:
    node-role: batch
  tolerations:
    - key: dedicated
      operator: Equal
      value: batch
      effect: NoSchedule
  containers:
    - name: job
      image: example.com/data-job:2026-07-28
      resources:
        requests:
          cpu: "2"
          memory: 6Gi

Break that three-way match and you get one of two outcomes, both of which look like a stuck queue. Either the pool never scales up, because Cluster Autoscaler can't tell the pending pod would fit, or it scales up a node that still can't schedule the pod, because the real labels don't match what the pod asked for. Whichever source the metadata comes from, simulated and real must agree.

Diagnose and verify

Start by confirming the nodes are genuinely idle from Cluster Autoscaler's perspective. Low observed CPU is not enough - Cluster Autoscaler reasons about pod resource requests and whether pods can move elsewhere, not utilization. Run it at verbosity 4 while investigating and follow the Deployment logs:

kubectl -n kube-system logs deployment/cluster-autoscaler --follow \
  | grep -E 'unneeded|scale down|scaled up recently|removing'

You're looking for evidence that Cluster Autoscaler already considers the node removable, followed by a cooldown message instead of a deletion. Exact wording varies by version, but the pattern holds: one line marks a node unneeded or eligible, a later line skips scale-down because something scaled up recently. Compare those timestamps against ASG scaling activity in the other group - a scale-up in group B shortly before an eligible node in group A fails to disappear is the signature:

aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name <group-b-asg-name> \
  --max-items 20

aws autoscaling describe-auto-scaling-groups \
  --query 'AutoScalingGroups[].{Name:AutoScalingGroupName,Desired:DesiredCapacity,Min:MinSize,Max:MaxSize}' \
  --output table

Rule out the ordinary blockers before blaming the cooldown. A node legitimately stays when it hosts local storage, a restrictive PodDisruptionBudget, an unreplicated pod, or safe-to-evict: "false". Scheduling constraints do the same: if the pods on that node can't fit anywhere else because of selectors, affinity, host ports, or resource requests, Cluster Autoscaler is keeping the node for good reason.

Once the flag is deployed, confirm that exactly one Deployment manages the intended ASGs and that the rendered command carries the setting:

kubectl -n kube-system get deployments | grep cluster-autoscaler

kubectl -n kube-system get deployment cluster-autoscaler \
  -o jsonpath='{range .spec.template.spec.containers[0].command[*]}{.}{"\n"}{end}' \
  | grep scale-down-delay

Then run a controlled two-group test. Let a node in group A become unneeded, wait for the logs to show it as a candidate, and create a pod only group B can run - here a batch pool labeled node-role=batch and tainted dedicated=batch:NoSchedule:

apiVersion: v1
kind: Pod
metadata:
  name: force-batch-scale-up
spec:
  restartPolicy: Never
  nodeSelector:
    node-role: batch
  tolerations:
    - key: dedicated
      operator: Equal
      value: batch
      effect: NoSchedule
  containers:
    - name: pause
      image: public.ecr.aws/eks-distro/kubernetes/pause:3.10
      resources:
        requests:
          cpu: "2"
          memory: 6Gi

Group B's scale-up should no longer postpone the eligible node in group A. Deletion still depends on scan interval, drain time, AWS API latency, and the node staying unneeded throughout, so give it a couple of scan cycles before declaring the test failed.

Legacy: one autoscaler per ASG

Before per-node-group cooldowns existed, sharding was the practical workaround: run one Cluster Autoscaler per ASG, each process watching a disjoint discovery tag. Each shard then kept its own cooldown state, so a GPU scale-up couldn't delay the process responsible for memory nodes.

It's heavier than it looks. Every shard needs its own ASG ownership boundary, leader election, service account identity, and IAM scope, and the operational surface grows linearly with node group count. Worse, shards watch the same unschedulable pods without coordinating, so two autoscalers can each decide their ASG can help and add duplicate capacity for a single pending pod. AWS recommends sharding only as a last resort, and the only case that still qualifies is a cluster where the binary matching your Kubernetes minor version predates the feature and the upgrade is blocked.

What's next

Run one Cluster Autoscaler with --scale-down-delay-type-local enabled, keep the default safety delays, and let them apply to the group that earned them. Verify three things afterward: one process owns every ASG you care about, the deployed binary actually accepts the flag, and any group that reaches zero has matching metadata across ASG node-template tags, real node configuration, and pod scheduling rules.

Cluster Autoscaler's behavior shifts between releases, and flags have been added, renamed, and defaulted differently over time. Check the AWS provider documentation against the source tag matching your image before you ship, then test it with real workloads rather than a single pause pod.

Need a second opinion on your autoscaling setup?

We can audit your Cluster Autoscaler deployment, IAM scope, and Terraform node group definitions to confirm one process safely owns every ASG it should.

Contact Us