When checking the cluster, we noticed an unusually high memory usage. We see that one node is fully utilized while the other shows little load. This often happens when a cluster with less than three nodes is upgraded at the nodes. Here we explain how it comes to it.
First, let’s see how many nodes are in the cluster and what is the current load of cpu/memory.
kubectl top node shows the current node utilization. In the example, there are two running nodes.
First the node is cordoned, so that no new pods are scheduled to this node.
Then the node is drained, so it is completely empty and the pods that used to run on the native node are distributed to all other nodes of the cluster. With only two nodes, everything is placed on exactly the other node, then we will have a high loaded node.
When the second node is back after the update, the pods will not be automatically distributed to both nodes. This leads to the initially observed imbalance.
- We recommend to use at least three nodes, because in that way loads could be better distributed during upgrades.
- The tool ‘popeye’ analyzes clusters and makes suggestions for improvement based on best practices
- Upgrade the nodes to the latest version and follow the final step in the guide: https://gecio.github.io/imke/clusterlifecycle/upgradingacluster/