Cluster Autoscaler is enabled, but underloaded nodes do not get removed
The cluster has nodes that are underloaded, but they are not automatically removed, even though Cluster Autoscaler is enabled. By default, Cluster Autoscaler must remove nodes with no running pods or nodes whose pods can be safely evicted.
There might be several possible reasons for this issue. Follow the instructions below one by one until the issue is resolved.
-
Connect to the cluster via
kubectl. -
Check whether the Cluster Autoscaler logs have warnings that contain
pdb blocking. If present, this indicates that one or more ongoing operations are causing pod disruptions, and the number of these disruptions has exceeded the allowable threshold defined in the Pod Disruption Budget (PDB).How PDB worksPDB is configured at the cluster level and controls the number of simultaneous disruptions allowed in a set of pods. PDB ensures that the cluster always has the minimum number of running pods necessary for continuous service operation. When configuring a PDB object, the following parameters are specified:
minAvailable— the minimum number of pods that must remain available and have thereadystatus at any given time.maxUnavailable— the maximum number of pods that can be disrupted simultaneously.
Kubernetes checks the PDB state when any operation that can lead to a pod disruption is performed, including automatic scaling of the nodes. When the maximum number of disruptions specified by the
maxUnavailableparameter is reached or exceeded, Kubernetes blocks this operation until sufficient resources are available for new disruptions. -
Increase the number of pods that can be disrupted simultaneously by increasing the value of the
maxUnavailableparameter:-
In your namespace, display the list of all deployed objects with the
PodDisruptionBudgettype:kubectl get pdb <NAMESPACE> -
Open the manifest of the required PDB object.
-
In the
specblock, increase the value of themaxUnavailableparameter. -
Apply the changes in the PDB manifest:
kubectl apply -f <PDB_MANIFEST_NAME>
-
For more details on disruptions and PDB, refer to the official Kubernetes documentation.
-
Connect to the cluster via
kubectl. -
Display information about the system pods of the cluster in the
kube-systemnamespace:kubectl get pods -n kube-system -o wide -
In the
kube-systemnamespace, display the list of all deployed objects with thePodDisruptionBudgettype:kubectl get pdb kube-systemIf PDB is not configured for these pods, the following message displays:
No resources found in kube-system namespace. -
Forward the information about this issue to technical support.
-
Connect to the cluster via
kubectl. -
Display the manifest of the affected pod using the command:
kubectl describe pod <POD_NAME> -n kube-system -
In the
OwnerorOwnerReferencesparameter, check if a workload controller is specified for the pod. If these parameters do not exist, or they are empty, the pod has no workload controller.This can happen if the pod is created manually, not as part of Deployment, StatefulSet and other controllers. Such pods need to be managed manually, which can lead to issues with availability, updates, automatic scaling of applications, as well as automatic recovery after failures, as all these processes are controlled by workload controllers.
-
Recreate the pod, specifying the required type of the workload controller in the
kindparameter in the pod manifest.
-
Connect to the cluster via
kubectl. -
Check if the affected node is marked with an annotation that prohibits Cluster Autoscaler to remove it. Run the command:
kubectl describe node <NODE_NAME> | grep scale-down-disabledThe command filters the node information by the
scale-down-disabledparameter and finds the annotation, if any:cluster-autoscaler.kubernetes.io/scale-down-disabled: trueThe annotation value can be set to
trueif this node must not be deleted during automatic scaling even if it is empty. These annotations can be used to indicate certain nodes that must not be evicted, for example, system components, critical services, or pods that must run only on specific nodes. If this is the case for the affected node, you do not need to do anything.If you still want to allow removing this node:
- Verify that removing it will not lead to issues in the cluster.
- Remove the annotation.
-
Connect to the cluster via
kubectl. -
Check whether the Cluster Autoscaler logs have warnings that contain the following line:
scale-down: node <NODE_NAME> is not marked as unneeded for deletionIf they do, the grace period set for the node has not yet expired, so you do not need to do anything.
In Managed Containers, the grace period is set to 5 minutes. If the node is empty for 5 minutes (all its pods are evicted), and no new pods appear, Cluster Autoscaler triggers the removal of this node.
If the issue persists, contact technical support.