Environment  环境
  • Rancher v2.6+  牧场主 v2.6+
  • A Rancher-provisioned RKE2 cluster
    一个由牧场者配置的 RKE2 集群
Situation  地理位置

There are a high number of restarts for cluster component Pods in the affected downstream RKE2 cluster:
受影响的下游 RKE2 集群中,集群组件 Pod 的重启次数较多:

NAMESPACE      NAME                                           READY   STATUS     RESTARTS            
cattle-fleet-system fleet-agent-cc8c97f97-bvx78               1/1    Running      185
cattle-system cattle-cluster-agent-b1460cbd-8ct5c             1/1    Running      115
cattle-system cattle-cluster-agent-b1460cbd-l2l8l             1/1    Running      168
kube-system kube-apiserver-cluster-suse-cp-f777105c-2qgvh     0/1    Running      314
kube-system kube-controller-manager-cluster-suse-cp-5c-2qgvh  1/1    Running      491
kube-system cloud-controller-manager-cluster-suse-cp-5c-2qgvh 1/1    Running      501

The kube-apiserver Pod flaps between a ready and not ready status:
kube-apiserver Pod 在准备状态和未准备好状态之间摇摆:

NAMESPACE              NAME                                 READY   STATUS     RESTARTS 

kube-system kube-apiserver-cluster-suse-cp-f777105c-2qgvh     0/1   Running    314

The kubelet logs register failing probes against the kube-apiserver.
kubelet 日志会对 kube-apiserver 进行检测失败。

Resolution  结局
  1. Enable kubelet debug logging
    启用 kubelet 调试日志
    1. Navigate to Cluster Management
      导航至集群管理
    2. Click Edit Config for the affected downstream RKE2 cluster
      点击 “编辑配置 ”以查看受影响的下游 RKE2 集群
    3. Click the Advanced tab in the Cluster Configuration form
      点击集群配置表单中的高级标签
    4. Under Additional Kubelet Args click Add Global Argument
      在 “额外 Kubelet Args”下点击添加全局参数
    5. In the new argument field enter v=9
      在新的参数字段中,输入 v=9
    6. Click Save  点击保存
  2. Replicate the liveness probe and check the kubelet logs
    复制活性探针并检查 kubelet 日志
    1. Open an SSH session to a master node in the affected RKE2 downstream cluster
      在受影响的 RKE2 下游集群中,向主节点开启 SSH 会话
    2. Check the kubelet log (tail -f /var/lib/rancher/rke2/agent/logs/kubelet.log | grep kube-apiserver) for failing kube-apiserver liveness probes 
      检查 kubelet 日志( tail -f /var/lib/rancher/rke2/agent/logs/kubelet.log | grep kube-apiserver ),查找失败的 kube-apiserver liveness probes 
    3. Execute the following command to simulate the liveness probe for the kube-apiserver Pod, which should fail, if encountering the issue:
      执行以下命令来模拟 kube-apiserver Pod 的活性探测,如果遇到问题,该探测应该会失败:
      /var/lib/rancher/rke2/bin/crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock exec $(/var/lib/rancher/rke2/bin/crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock ps | grep kube-apiserver | awk '{print $1}') kubectl get --server=https://localhost:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/livez
    4. Perform the simulated liveness probe for the kube-apiserver again, replacing localhost with 127.0.0.1, which should succeed:
      再次对 kube-apiserver 进行模拟的活体探测,将 localhost 替换为 127.0.0.1,应该能成功:
      /var/lib/rancher/rke2/bin/crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock exec $(/var/lib/rancher/rke2/bin/crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock ps | grep kube-apiserver | awk '{print $1}') kubectl get --server=https://127.0.0.1:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/livez
  3. Fix the host or host template, to ensure a valid /etc/hosts file is present, with an entry mapping localhost to 127.0.0.1, as expected.
    修复主机或主机模板,确保存在有效的 /etc/hosts 文件,并按预期将 localhost 映射到 127.0.0.1。
Cause  病因

The /etc/hosts file on the node was empty and did not contain any localhost references, causing DNS resolution failures for the kube-apiserver liveness probes to localhost.
节点上的 /etc/hosts 文件是空的,且不包含任何 localhost 引用,导致 kube-apiserver liveness 探测器向 localhost 的 DNS 解析失败。

Logo

开源鸿蒙跨平台开发社区汇聚开发者与厂商,共建“一次开发,多端部署”的开源生态,致力于降低跨端开发门槛,推动万物智联创新。

更多推荐