Install-Kubernete
Install-Kubernete

[TOC]

Kubernetes1.27版本安装记录过程

安装参考： https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

安装前准备

集群规划

  • 系统： Ubuntu22.04 
    root@k8s-master01:~# lsb_release -a 
No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:    22.04
Codename:    jammy
  • 安装工具：kubeadm
  • 集群分布： 3主节点 + 2个工作节点
  • 容器运行时： 选用Docker
    • 因为1.24版本在k8s主干分支中移除了 dockershim ，并将其给了docker维护，后将其改名为cni-dockerd,因此我们需要安装：
      • 安装Docker
      • 安装cni-dockerd

采用堆叠的方式创建高可用集群（etcd和控制平面其他组件在同一个节点上）：

高可用架构图

集群节点图：
k8s-cluster.svg

高可用参考： https://developer.aliyun.com/article/853054

前提准备

​

  • 节点之中不可以有重复的主机名、MAC 地址或 product_uuid。请参见这里了解更多详细信息。
    • 主机名验证 
      # 查看主机名
hostname

# 修改为k8s-master (替换成每个节点主机名)
hostnamectl hostname k8s-master
  • Mac地址验证 
    使用命令 ip link 或 ifconfig -a 来获取网络接口的 MAC 地址
  • Product_uuid验证 
    sudo cat /sys/class/dmi/id/product_uuid
  • 配置主机名映射

    ​ 配置文件：/etc/hosts

    192.168.0.18 k8s-master
192.168.0.19 k8s-master01
192.168.0.20 k8s-master02
192.168.0.21 k8s-master03
192.168.0.22 k8s-slave01
192.168.0.23 k8s-slave02

    192.168.0.18 k8s-master 这一条是为了配置高可用集群准备的。

  • 关闭防火墙 
    ufw disable
  • 转发 IPv4 并让 iptables 看到桥接流量

    执行下述指令：

    cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# 设置所需的 sysctl 参数，参数在重新启动后保持不变
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# 应用 sysctl 参数而不重新启动
sudo sysctl --system

    通过运行以下指令确认 br_netfilter 和 overlay 模块被加载

lsmod | grep br_netfilter
lsmod | grep overlay

通过运行以下指令确认 net.bridge.bridge-nf-call-iptables、net.bridge.bridge-nf-call-ip6tables 和 net.ipv4.ip_forward 系统变量在你的 sysctl 配置中被设置为 1：

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward
  • 一台兼容的 Linux 主机。Kubernetes 项目为基于 Debian 和 Red Hat 的 Linux 发行版以及一些不提供包管理器的发行版提供通用的指令。
  • 每台机器 2 GB 或更多的 RAM（如果少于这个数字将会影响你应用的运行内存）。
  • CPU 2 核心及以上。
  • 集群中的所有机器的网络彼此均能相互连接（公网和内网都可以）。
  • 开启机器上的某些端口。请参见这里了解更多详细信息。 
    nc 127.0.0.1 6443

    无输出，表示正常

  • 禁用交换分区。为了保证 kubelet 正常工作，你必须禁用交换分区。
      • 例如，sudo swapoff -a 将暂时禁用交换分区。要使此更改在重启后保持不变，请确保在如 /etc/fstab、systemd.swap 等配置文件中禁用交换分区，具体取决于你的系统如何配置。 
        #关闭swap
swapoff -a  
sed -ri 's/.*swap.*/#&/' /etc/fstab

    ​

  • 查看主机cgroup版本 
    stat -fc %T /sys/fs/cgroup/

    对于 cgroup v1，输出为 tmpfs。

    对于 cgroup v2，输出为 cgroup2fs。 cgroup v2 具有以下要求：

    • 操作系统发行版启用 cgroup v2
    • Linux 内核为 5.8 或更高版本
    • 容器运行时支持 cgroup v2。例如：
    • kubelet 和容器运行时被配置为使用 systemd cgroup 驱动

参考： https://kubernetes.io/zh-cn/docs/concepts/architecture/cgroups/

开始安装

1. 安装容器运行时（所有节点执行）

https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/#docker

安装前准备

更新apt 并 安装必要工具包
sudo apt-get remove docker docker-engine docker.io containerd runc

sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
添加Docker官方GPG key
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
设置apt库
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

安装Docker

安装24.0.0版本Docker

sudo apt-get update 

VERSION_STRING=5:24.0.0-1~ubuntu.22.04~jammy
sudo apt-get install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin -y
验证是否成功
root@k8s-master01:~# docker version
Client: Docker Engine - Community
 Version:           24.0.0
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        98fdcd7
 Built:             Mon May 15 18:49:22 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.0
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       1331b8c
  Built:            Mon May 15 18:49:22 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
手动设置Docker使用systemd cgroup驱动

修改： /etc/docker/daemon.json文件,添加如下内容 保存重启docker


cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://84bkfzte.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

systemctl daemon-reload && systemctl restart docker
启动并设置开机启动
 systemctl enable docker --now
查看Docker的cgroup驱动是systemd
root@k8s-master01:~# docker info | grep -i cgroup
 Cgroup Driver: systemd
 Cgroup Version: 2
  cgroupns

安装cri-dockerd

下载cri-docker版本：0.3.2

​ https://github.com/Mirantis/cri-dockerd/releases/tag/v0.3.2

安装deb软件包
sudo dpkg -i  ./cri-dockerd_0.3.2.3-0.ubuntu-jammy_amd64.deb
添加infra容器镜像配置

修改镜像地址为国内，否则kubelet拉取不了镜像导致启动失败

sudo sed -i 's|ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd://|ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9|' /usr/lib/systemd/system/cri-docker.service

修改以后内容如下：

cat /usr/lib/systemd/system/cri-docker.service

ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9
启动并设置开机启动
systemctl daemon-reload 
systemctl enable cri-docker --now

2. 安装 kubeadm、kubelet 和 kubectl （所有节点执行）

你需要在每台机器上安装以下的软件包：

  • kubeadm：用来初始化集群的指令。
  • kubelet：在集群中的每个节点上用来启动 Pod 和容器等。
  • kubectl：用来与集群通信的命令行工具。
  1. 更新 apt 包索引并安装使用 Kubernetes apt 仓库所需要的包， 并配置阿里apt源：

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/kubernetes-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
  1. 更新 apt 包索引，安装 kubelet、kubeadm 和 kubectl，并锁定其版本：
sudo apt-get update
sudo apt-get install -y kubelet=1.27.1-00 kubeadm=1.27.1-00 kubectl=1.27.1-00
sudo apt-mark hold kubelet kubeadm kubectl
  1. 查看对应版本
   root@k8s-master01:~# kubectl version -o yaml
   clientVersion:
     buildDate: "2023-04-14T13:21:19Z"
     compiler: gc
     gitCommit: 4c9411232e10168d7b050c49a1b59f6df9d7ea4b
     gitTreeState: clean
     gitVersion: v1.27.1
     goVersion: go1.20.3
     major: "1"
     minor: "27"
     platform: linux/amd64
   kustomizeVersion: v5.0.1
   
   The connection to the server localhost:8080 was refused - did you specify the right host or port?
   root@k8s-master01:~# kubelet --version
   Kubernetes v1.27.1
   
   root@k8s-master01:~# kubeadm version
   kubeadm version: &version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:20:04Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
   root@k8s-master01:~#

注意：kubelet 现在每隔几秒就会重启，因为它陷入了一个等待 kubeadm 指令的死循环（正常现象，初始化好主节点就好了）。

3. 配置cgoup驱动程序（所有节点执行）

配置配置kubelet cgroup驱动

https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#%E6%9B%B4%E6%96%B0%E6%89%80%E6%9C%89%E8%8A%82%E7%82%B9%E7%9A%84-cgroup-%E9%A9%B1%E5%8A%A8

警告：

你需要确保容器运行时和 kubelet 所使用的是相同的 cgroup 驱动，否则 kubelet 进程会失败。

相, 我们都是使用 systemd, 新建文件并写入内容：


root@k8s-master01:~# cat << EOF >> /etc/default/kubelet 
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"
EOF


root@k8s-master01:~# cat /etc/default/kubelet 
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"

root@k8s-master01:~# systemctl restart kubelet

4. 配置负载均衡（控制平面节点执行）

​ 参考页面：https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#options-for-software-load-balancing

说明

我们使用KeepAlive和HAProxy完成负载均衡高可用的配置。

安装KeepAlive（k8s静态Pod方式安装）

创建/etc/keepalived/keepalived.conf配置文件（主： k8s-master01节点）

sudo mkdir -p  /etc/keepalived

sudo cat << EOF > /etc/keepalived/keepalived.conf

! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script check_apiserver {
  # 指定验证KeepAlive是否存活脚本位置
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -2
  fall 10
  rise 2
}

vrrp_instance VI_1 {
    ! 指定MASTER 或者 BACKUP， 这里我们使用k8s-master01节点作为MASTER
    state MASTER
    ! 网卡名称
    interface enp0s3
    ! 指定router_id，集群该值必须相同，这里指定为：51
    virtual_router_id 51
    ! 优先级： MASTER节点优先级要高于BACKUP节点，我们MASTER节点配置：101， BACKUP节点设置：100
    priority 101
    authentication {
        auth_type PASS
        ! 验证密码： 集群该值必须相同，这里指定为：42
        auth_pass 42
    }
    virtual_ipaddress {
        ! 虚拟IP地址： 改地址将作为KeepAlive对外暴露的地址，指定的IP必须是你集群所在的网络里面没有被使用的IP地址，这里指定：192.168.0.18
        ! 同时改地址也是将要指定kubeadm init 命令 --control-plane-endpoint 参数中的，至于端口， 需要在HAProxy里面指定
        192.168.0.18
    }
    track_script {
        check_apiserver
    }
}
EOF

cat /etc/keepalived/keepalived.conf
创建/etc/keepalived/keepalived.conf配置文件（备： k8s-master02、k8s-master03节点）
sudo mkdir -p  /etc/keepalived

sudo cat << EOF > /etc/keepalived/keepalived.conf

! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script check_apiserver {
  # 指定验证KeepAlive是否存活脚本位置
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -2
  fall 10
  rise 2
}

vrrp_instance VI_1 {
    ! 指定MASTER 或者 BACKUP， 这里我们使用k8s-master01节点作为MASTER
    state BACKUP
    ! 网卡名称
    interface enp0s3
    ! 指定router_id，集群该值必须相同，这里指定为：51
    virtual_router_id 51
    ! 优先级： MASTER节点优先级要高于BACKUP节点，我们MASTER节点配置：101， BACKUP节点设置：100
    priority 100
    authentication {
        auth_type PASS
        ! 验证密码： 集群该值必须相同，这里指定为：42
        auth_pass 42
    }
    virtual_ipaddress {
        ! 虚拟IP地址： 改地址将作为KeepAlive对外暴露的地址，指定的IP必须是你集群所在的网络里面没有被使用的IP地址，这里指定：192.168.0.18
        ! 同时改地址也是将要指定kubeadm init 命令 --control-plane-endpoint 参数中的，至于端口， 需要在HAProxy里面指定
        192.168.0.18
    }
    track_script {
        check_apiserver
    }
}
EOF

cat /etc/keepalived/keepalived.conf
创建心跳检测文件（主备都执行： k8s-master01、k8s-master02、k8s-master03）
sudo mkdir -p  /etc/keepalived

sudo cat << EOF > /etc/keepalived/check_apiserver.sh

# /etc/keepalived/check_apiserver.sh
#!/bin/sh

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

curl --silent --max-time 2 --insecure https://localhost:8443/ -o /dev/null || errorExit "Error GET https://localhost:8443/"
if ip addr | grep -q 192.168.0.18; then
    curl --silent --max-time 2 --insecure https://192.168.0.18:8443/ -o /dev/null || errorExit "Error GET https://192.168.0.18:8443/"
fi
EOF

cat /etc/keepalived/check_apiserver.sh
创建Keepalive Pod yaml文件（主备都执行： k8s-master01、k8s-master02、k8s-master03）

文件名： /etc/kubernetes/manifests/keepalived.yaml

sudo cat << EOF > /etc/kubernetes/manifests/keepalived.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  name: keepalived
  namespace: kube-system
spec:
  containers:
  - image: osixia/keepalived:2.0.17
    name: keepalived
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        - NET_BROADCAST
        - NET_RAW
    volumeMounts:
    - mountPath: /usr/local/etc/keepalived/keepalived.conf
      name: config
    - mountPath: /etc/keepalived/check_apiserver.sh
      name: check
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/keepalived/keepalived.conf
    name: config
  - hostPath:
      path: /etc/keepalived/check_apiserver.sh
    name: check
status: {}

EOF

cat /etc/kubernetes/manifests/keepalived.yaml

安装HAProxy（k8s静态Pod方式安装）

说明：

​ 由于现在没有进行kubeadm init 操作，因此现在kubelet组件启动不了，因此想要看到效果需要等到kubeadm init 指定完成以后。

创建HAProxy配置文件 （k8s-master01、k8s-master02、k8s-master03执行）
sudo mkdir -p /etc/haproxy

sudo cat << EOF > /etc/haproxy/haproxy.cfg

# /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log /dev/log local0
    log /dev/log local1 notice
    daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          20s
    timeout server          20s
    timeout http-keep-alive 10s
    timeout check           10s

#---------------------------------------------------------------------
# apiserver frontend which proxys to the control plane nodes
#---------------------------------------------------------------------
frontend apiserver
    # 指定负载均衡绑定的地址和端口，这里的端口需要和/etc/keepalived/check_apiserver.sh文件中监控的端口相同
    bind *:8443
    mode tcp
    option tcplog
    default_backend apiserver

#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserver
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    option ssl-hello-chk
    balance     roundrobin
        # HAProxy负载均衡器代理的后端节点
        server k8s-master01 192.168.0.19:6443 check
        server k8s-master02 192.168.0.20:6443 check
        server k8s-master03 192.168.0.21:6443 check
        
        # [...]

EOF

cat /etc/haproxy/haproxy.cfg
创建HAProxy Pod 需要的yaml文件（k8s-master01、k8s-master02、k8s-master03执行）

sudo cat << EOF > /etc/kubernetes/manifests/haproxy.yaml

apiVersion: v1
kind: Pod
metadata:
  name: haproxy
  namespace: kube-system
spec:
  containers:
  - image: haproxy:2.1.4
    name: haproxy
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: localhost
        path: /healthz
        # 指定HAProxy代理的端口，该端口必须和/etc/haproxy/haproxy.cfg配置的端口相同
        port: 8443
        scheme: HTTPS
    volumeMounts:
    - mountPath: /usr/local/etc/haproxy/haproxy.cfg
      name: haproxyconf
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/haproxy/haproxy.cfg
      type: FileOrCreate
    name: haproxyconf
status: {}

EOF

cat /etc/kubernetes/manifests/haproxy.yaml
针对Keepalive和HAProxy的说明
  • 我们配置负载均衡使用8443端口而没有使用默认的6443端口是因为我们Keepalive和HAProxy都部署在主节点上，而主节点上也部署Kubernetes的api-server组件，而api-server主键已经占用了 6443端口，因此，我们这里配置了8443端口。
  • 当使用Keepalived和HAProxy组合作为高可用负载均衡时，可以构建一个可靠的架构来提供高可用性和负载均衡功能。下面是一个简化的架构图，以可视化方式展示Keepalived和HAProxy的组合： 
              +----------------------+
          |      Load Balancer    |
          |      (HAProxy)        |
          +----------------------+
                  |       |
                  |       |
         +--------+       +--------+
         |                        |
+--------------+          +--------------+
|  Backend 1  |          |  Backend 2  |
+--------------+          +--------------+

    在上面的架构图中，有以下组件：

    1. Load Balancer (HAProxy)：负责接收客户端请求并将其转发到后端服务器。HAProxy是一种高性能的负载均衡器，能够根据不同的负载均衡算法将请求分发到后端服务器，以实现负载均衡和高可用性。
    2. Backend 1 和 Backend 2：这些是真实的后端服务器，用于处理客户端请求。可以有多个后端服务器，以实现负载均衡和高可用性。这些后端服务器可以是应用服务器、数据库服务器等。
    3. Keepalived：用于实现高可用性的组件。Keepalived监测Load Balancer节点的可用性，并在主节点发生故障时将其切换到备份节点。Keepalived使用虚拟IP地址（VIP）来提供无缝的故障转移和高可用性。

    在这个架构中，客户端请求首先到达Load Balancer（HAProxy），然后根据负载均衡算法选择一个后端服务器进行处理。如果Load Balancer节点出现故障，Keepalived会自动检测到，并将主节点的VIP切换到备份节点，以确保服务的持续可用性。

  • 访问HAProxy的入口是Keepalived提供的虚拟IP（VIP）。Keepalived会将虚拟IP绑定到主节点上，以便客户端可以通过该虚拟IP与负载均衡器通信。

    在高可用负载均衡架构中，客户端不需要直接连接到单个负载均衡器节点。相反，客户端将请求发送到虚拟IP地址，该虚拟IP地址由Keepalived管理并绑定到当前的主节点上。通过这种方式，客户端可以无需关心主节点和备份节点之间的切换，而始终通过虚拟IP与负载均衡器通信。

    当Keepalived检测到主节点故障时，它会自动将虚拟IP迁移到备份节点上，以实现无缝的故障转移。这样，客户端可以继续使用相同的虚拟IP与负载均衡器通信，无需感知主备节点的切换。

    总结起来，访问HAProxy的入口就是通过Keepalived提供的虚拟IP。客户端可以使用该虚拟IP来连接负载均衡器，并由负载均衡器将请求转发到后端服务器。

5. 初始化Master节点（控制平面节点执行）

参考界面

kubeadm init 命令运行过程：

​ https://kubernetes.io/zh-cn/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images

kubeadm config print 打印kubeadm join 或者 kubeadm init 命令默认值：

​ https://kubernetes.io/zh-cn/docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-print

​

查看kubeadm init命令默认配置文件 （参考）

这里输出的yaml格式不正确： 正确格式参考：

​ https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta3/

root@k8s-master01:~# kubeadm config print init-defaults | tee kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: node
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: 1.27.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

下载镜像

手动安装1.27.1版本需要的镜像
# 注意： 这里 EOF 是使用单引号括起来的，不适用单引号，后面的脚本文件会执行，导致结果错误
sudo cat << 'EOF' > download_image.sh

#!/bin/bash
# Kubernetes 安装的版本
KUBERNETES_VERSION=$(kubeadm version | grep -oP 'GitVersion:"v\K[^"]+')
# 阿里Kubernetes官方镜像库
AILI_KUBERNETES_REGISTRY="registry.cn-hangzhou.aliyuncs.com/google_containers"

echo "KUBERNETES_VERSION => ${KUBERNETES_VERSION}"
echo "AILI_KUBERNETES_REGISTRY => ${AILI_KUBERNETES_REGISTRY}"

# 下载并重命名镜像
function download_and_tag_image() {
  # 官方镜像全称： registry.k8s.io/xxx/xxx:xxx
  # 比如： registry.k8s.io/kube-proxy:v1.27.1
  local full_official_image=$1
  local ali_image
  ali_image=$(echo "$full_official_image" | sed -E "s|(.*/)(.*)|$AILI_KUBERNETES_REGISTRY/\2|")
  echo "downloading image => $ali_image"
  echo "downloading image => $ali_image"
  sudo docker pull "$ali_image"

  # 重命名镜像
  echo "rename image $ali_image to $full_official_image"
  sudo docker tag "$ali_image" "$full_official_image"
}

# 官方镜像列表
OFFICIAL_IMAGE_LIST=$(kubeadm config images list --kubernetes-version "$KUBERNETES_VERSION" 2>/dev/null | grep "$OFFICIAL_KUBERNETES_REGISTRY")

for official_image in $OFFICIAL_IMAGE_LIST; do
  download_and_tag_image "$official_image"
done

EOF

cat download_image.sh


sudo chmod u+x ./download_image.sh && ./download_image.sh

启动初始化

root@k8s-master01:~#  kubeadm init \
  --apiserver-advertise-address=192.168.0.19 \
  --kubernetes-version v1.27.1 \
  --service-cidr=10.96.0.0/12 \
  --pod-network-cidr=10.244.0.0/16 \
  --cri-socket=unix:///var/run/cri-dockerd.sock \
  --control-plane-endpoint=k8s-master:8443 \
  --upload-certs
[init] Using Kubernetes version: v1.27.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W0520 20:12:40.038258   30744 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.1, falling back to the nearest etcd version (3.5.7-0)
W0520 20:12:40.293262   30744 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.19]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.0.19 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.0.19 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
W0520 20:12:42.465414   30744 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
W0520 20:12:42.555482   30744 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
W0520 20:12:42.934781   30744 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
W0520 20:12:43.058171   30744 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
W0520 20:12:43.628841   30744 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.1, falling back to the nearest etcd version (3.5.7-0)
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 9.011713 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
3636bc7d84515aeb36ca79597792b07cc64a888ebdea9221ab68a5bae93ac947
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: th5i1f.fnzc9v0yb6z3aok8
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
W0520 20:12:54.340208   30744 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join k8s-master:8443 --token th5i1f.fnzc9v0yb6z3aok8 \
    --discovery-token-ca-cert-hash sha256:25357bff7f44a787886222dc9439916ab271dc5af5d5bbef274288fdd8e245b4 \
    --control-plane --certificate-key 3636bc7d84515aeb36ca79597792b07cc64a888ebdea9221ab68a5bae93ac947

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s-master:8443 --token th5i1f.fnzc9v0yb6z3aok8 \
    --discovery-token-ca-cert-hash sha256:25357bff7f44a787886222dc9439916ab271dc5af5d5bbef274288fdd8e245b4
  • --apiserver-advertise-address=192.168.0.20 : 用于为控制平面节点的 API server 设置广播地址（必须指定master节点IP）
  • --cri-socket=unix:///var/run/cri-dockerd.sock : 指定CRI 套接字的路径，我们这里有两个运行时环境： containerd 和 cri-dockerd， 这里我们指定cri-dockerd。 注意： kubeadm join 命令的时候也需要指定。
  • --control-plane-endpoint=k8s-master:8443 : 用于为所有控制平面(高可用环境必须指定)节点设置共享端点，集群节点都要需要配置 /etc/hosts 将k8s-master 指定我们Keepalive中配置的虚拟IP地址, 8443这个端口是我们之前在HAProxy中配置的。

​ kubeadm 不支持将没有 --control-plane-endpoint 参数的单个控制平面集群转换为高可用性集群。

  • --service-cidr=10.96.0.0/12 : 指定service网段
  • --pod-network-cidr=10.244.0.0/16: 指定pod IP地址网段
  • --kubernetes-version 1.27.0 : 指定k8s版本
  • --upload-certs : 指定上传证书 ， 高可用集群建议指定。
    • 如果不指定，也可以手动指定: 参考如下部分证书分配手册。
    • 或者安装完成集群以后，重新上传证书： sudo kubeadm init phase upload-certs --upload-certs

这里有可能失败，如果失败就手动下载这个镜像： registry.k8s.io/pause:3.6

cat << 'EOF' > download.sh

#!/bin/bash

# 官方镜像列表
OFFICIAL_IMAGE_LIST=("$@")

# 阿里Kubernetes官方惊镜像库
AILI_KUBERNETES_REGISTRY="registry.cn-hangzhou.aliyuncs.com/google_containers"

echo "AILI_KUBERNETES_REGISTRY => ${AILI_KUBERNETES_REGISTRY}"

# 下载并重命名镜像
function download_and_tag_image() {
  # 官方镜像全称： registry.k8s.io/xxx/xxx:xxx
  # 比如： registry.k8s.io/kube-proxy:v1.27.1
  local full_official_image=$1
  local ali_image
  ali_image=$(echo "$full_official_image" | sed -E "s|(.*/)(.*)|$AILI_KUBERNETES_REGISTRY/\2|")
  echo "downloading image => $ali_image"
  sudo docker pull "$ali_image"

  # 重命名镜像
  echo "rename image $ali_image to $full_official_image"
  sudo docker tag "$ali_image" "$full_official_image"
}

for official_image in "${OFFICIAL_IMAGE_LIST[@]}"; do
  download_and_tag_image "$official_image"
done

EOF

sudo chmod u+x ./download.sh && ./download.sh registry.k8s.io/pause:3.6

也有可能报错：Nameserver limits exceeded

root@k8s-master01:/etc/kubernetes# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Tue 2023-05-30 00:34:59 CST; 3s ago
       Docs: https://kubernetes.io/docs/home/
   Main PID: 7672 (kubelet)
      Tasks: 13 (limit: 13832)
     Memory: 25.7M
        CPU: 588ms
     CGroup: /system.slice/kubelet.service
             └─7672 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --containe>

May 30 00:35:01 k8s-master01 kubelet[7672]: E0530 00:35:01.033186    7672 dns.go:158] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, >

解决：Nameserver limits exceeded

​ 参考： https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#known-issues

root@k8s-master01:~# cat /etc/resolv.conf 
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
## 注释： 这里只有三条解析，我们做修改
nameserver 127.0.0.53
options edns0 trust-ad
search .
root@k8s-master01:~# cat /etc/systemd/resolved.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the resolved.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
#
# See resolved.conf(5) for details.
## 注释： 这个文件没有解析，不做修改
[Resolve]
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
# Google:     8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
# Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
#DNS=
#FallbackDNS=
#Domains=
#DNSSEC=no
#DNSOverTLS=no
#MulticastDNS=no
#LLMNR=no
#Cache=no-negative
#CacheFromLocalhost=no
#DNSStubListener=yes
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no
root@k8s-master01:~# cat  /run/systemd/resolve/resolv.conf
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
# 这个文件有多条nameserver解析，我们注释其中的两条，因为我们不使用ip6,所以这里注释掉ip6解析
nameserver 192.168.0.1
nameserver 192.168.1.1
# 注释掉
# nameserver fe80::1%2
# Too many DNS servers configured, the following entries may be ignored.
# 注释掉
# nameserver 240c::6666
search .

root@k8s-master01:~# vim  /run/systemd/resolve/resolv.conf
# 重启
root@k8s-master01:~# systemctl restart kubelet 
# 重置kubeadm后,重新执行以上kubeadm init命令，即可解决
root@k8s-master01:~#     kubeadm reset \
  --cri-socket=unix:///var/run/cri-dockerd.sock 
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0530 00:45:44.483361   37269 reset.go:106] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get config map: configmaps "kubeadm-config" not found
W0530 00:45:44.483476   37269 preflight.go:56] [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W0530 00:45:46.625805   37269 removeetcdmember.go:106] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Deleted contents of the etcd data directory: /var/lib/etcd
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of directories: [/etc/kubernetes/manifests /var/lib/kubelet /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

重要的说明：

要重新配置一个已经创建的集群， 请参见重新配置一个 kubeadm 集群。

要再次运行 kubeadm init，你必须首先卸载集群。

配置用户可以使用kubectl 命令

非root用户执行（root用户也可以）：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

root用户（如果执行完上面的语句， 下面可以不执行）

export KUBECONFIG=/etc/kubernetes/admin.conf

验证kubectl 命令是否可用

root@k8s-master01:~# kubectl get nodes 
NAME         STATUS     ROLES           AGE   VERSION
k8s-master   NotReady   control-plane   21m   v1.27.1

安装容器网络（CNI）插件

为了保证pod之间能够正常访问，需要安装容器网络插件。

注意： 每个集群只能安装一个 Pod 网络（容器网络插件）。

你必须部署一个基于 Pod 网络插件的 容器网络接口 (CNI)，以便你的 Pod 可以相互通信。 在安装网络之前，集群 DNS (CoreDNS) 将不会启动。

这里有多种网络插件：https://kubernetes.io/zh-cn/docs/concepts/cluster-administration/addons/#networking-and-network-policy

我们选择： Calico

root@k8s-master01:~# kubectl apply -f https://projectcalico.docs.tigera.io/archive/v3.25/manifests/calico.yaml
poddisruptionbudget.policy/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
serviceaccount/calico-node created
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers created

查看是否安装完成：

root@k8s-master01:~# kubectl get pods -A 
NAMESPACE     NAME                                       READY   STATUS     RESTARTS   AGE
kube-system   calico-kube-controllers-6c99c8747f-dmwg7   0/1     Pending    0          118s
kube-system   calico-node-t5bc7                          0/1     Init:0/3   0          118s
kube-system   coredns-5d78c9869d-gm5vt                   0/1     Pending    0          33m
kube-system   coredns-5d78c9869d-xgkbj                   0/1     Pending    0          33m
kube-system   etcd-k8s-master                            1/1     Running    0          34m
kube-system   kube-apiserver-k8s-master                  1/1     Running    0          34m
kube-system   kube-controller-manager-k8s-master         1/1     Running    0          34m
kube-system   kube-proxy-d26m7                           1/1     Running    0          33m
kube-system   kube-scheduler-k8s-master                  1/1     Running    0          34m
root@k8s-master01:~# kubectl get pod -A  
NAMESPACE     NAME                                       READY   STATUS    RESTARTS      AGE
kube-system   calico-kube-controllers-6c99c8747f-cfqg9   1/1     Running   0             41s
kube-system   calico-node-rczss                          1/1     Running   0             41s
kube-system   coredns-5d78c9869d-gm5vt                   1/1     Running   0             80m
kube-system   coredns-5d78c9869d-xgkbj                   1/1     Running   0             80m
kube-system   etcd-k8s-master                            1/1     Running   1 (10m ago)   80m
kube-system   kube-apiserver-k8s-master                  1/1     Running   1 (10m ago)   80m
kube-system   kube-controller-manager-k8s-master         1/1     Running   1 (10m ago)   80m
kube-system   kube-proxy-d26m7                           1/1     Running   1 (10m ago)   80m
kube-system   kube-scheduler-k8s-master                  1/1     Running   1 (10m ago)   80m

需要等10多分钟。。。

最后 calico 相关的pod都 Running 状态， 并且都是 READY 1/1 ，表示安装完成， 可以看到 calico 安装完成以后， coredns 相关的pod也都是Running 状态了。

移除控制平面污点

默认控制平面会打上污点，pod是不会被调度到控制平面上的， 如果想要让pod调度到控制平面上，可以执行以下命令移除污点：

root@k8s-master01:~# kubectl taint nodes --all node-role.kubernetes.io/control-plane-
node/k8s-master untainted

7. Worker节点加入集群（所有Woker节点执行，必须使用root用户执行）

​ 我这里就是： k8s-slave-1 和 k8s-slave02 这两台集群执行：

root@k8s-slave01:~# kubeadm join k8s-master:8443 --token th5i1f.fnzc9v0yb6z3aok8 \
        --discovery-token-ca-cert-hash sha256:25357bff7f44a787886222dc9439916ab271dc5af5d5bbef274288fdd8e245b4 \
    --cri-socket=unix:///var/run/cri-dockerd.sock 
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

​ 如果在等一段时间还是node还是NotReady状态，则使用在master节点中执行以下命令：

root@k8s-master01:~# kubectl get node 
NAME           STATUS     ROLES           AGE     VERSION
k8s-master01   Ready      control-plane   27m     v1.27.1
k8s-master02   Ready      control-plane   10m     v1.27.1
k8s-master03   Ready      control-plane   9m45s   v1.27.1
k8s-slave01    NotReady   <none>          49s     v1.27.1
k8s-slave02    NotReady   <none>          44s     v1.27.1

root@k8s-master01:~# kubectl get pod -A  
NAMESPACE     NAME                                       READY   STATUS                  RESTARTS      AGE
kube-system   calico-kube-controllers-6c99c8747f-fslqk   1/1     Running                 0             23m
kube-system   calico-node-6wgk4                          0/1     Init:Error              4 (76s ago)   2m37s
kube-system   calico-node-grb97                          1/1     Running                 0             11m
kube-system   calico-node-ltczv                          0/1     Init:CrashLoopBackOff   4 (56s ago)   2m42s
kube-system   calico-node-pffcg                          1/1     Running                 0             12m
kube-system   calico-node-vtcqg                          1/1     Running                 0             23m
kube-system   coredns-5d78c9869d-m5zgd                   1/1     Running                 0             28m
kube-system   coredns-5d78c9869d-mnxzj                   1/1     Running                 0             28m
kube-system   etcd-k8s-master01                          1/1     Running                 0             29m
kube-system   etcd-k8s-master02                          1/1     Running                 0             12m
kube-system   etcd-k8s-master03                          1/1     Running                 0             11m
kube-system   haproxy-k8s-master01                       1/1     Running                 0             29m
kube-system   haproxy-k8s-master02                       1/1     Running                 0             12m
kube-system   haproxy-k8s-master03                       1/1     Running                 0             11m
kube-system   keepalived-k8s-master01                    1/1     Running                 0             29m
kube-system   keepalived-k8s-master02                    1/1     Running                 0             12m
kube-system   keepalived-k8s-master03                    1/1     Running                 0             10m
kube-system   kube-apiserver-k8s-master01                1/1     Running                 0             29m
kube-system   kube-apiserver-k8s-master02                1/1     Running                 0             12m
kube-system   kube-apiserver-k8s-master03                1/1     Running                 1 (11m ago)   11m
kube-system   kube-controller-manager-k8s-master01       1/1     Running                 1 (12m ago)   29m
kube-system   kube-controller-manager-k8s-master02       1/1     Running                 0             12m
kube-system   kube-controller-manager-k8s-master03       1/1     Running                 0             10m
kube-system   kube-proxy-lmw7g                           1/1     Running                 0             11m
kube-system   kube-proxy-mb8hx                           0/1     ErrImagePull            0             2m42s
kube-system   kube-proxy-nvx8b                           0/1     ImagePullBackOff        0             2m37s
kube-system   kube-proxy-phvcm                           1/1     Running                 0             28m
kube-system   kube-proxy-psst7                           1/1     Running                 0             12m
kube-system   kube-scheduler-k8s-master01                1/1     Running                 1 (12m ago)   29m
kube-system   kube-scheduler-k8s-master02                1/1     Running                 0             12m
kube-system   kube-scheduler-k8s-master03                1/1     Running                 0             10m

# 可以看到kube-proxy镜像可能现在失败，我们看一下Pod的详情
root@k8s-master01:~# kubectl describe pod kube-proxy-mb8hx -n kube-system
# 在Events里面可以看到是应为镜像没有pull下来，我们去两个node节点手动下载一下镜像
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  3m57s                default-scheduler  Successfully assigned kube-system/kube-proxy-mb8hx to k8s-slave01
  Warning  Failed     3m24s                kubelet            Failed to pull image "registry.k8s.io/kube-proxy:v1.27.1": rpc error: code = Unknown desc = Error response from daemon: Head "https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/kube-proxy/manifests/v1.27.1": dial tcp 64.233.187.82:443: i/o timeout
  Normal   Pulling    62s (x4 over 3m54s)  kubelet            Pulling image "registry.k8s.io/kube-proxy:v1.27.1"
  Warning  Failed     31s (x4 over 3m24s)  kubelet            Error: ErrImagePull
  Warning  Failed     31s (x3 over 2m42s)  kubelet            Failed to pull image "registry.k8s.io/kube-proxy:v1.27.1": rpc error: code = Unknown desc = Error response from daemon: Head "https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/kube-proxy/manifests/v1.27.1": dial tcp 64.233.188.82:443: i/o timeout
  Normal   BackOff    5s (x6 over 3m23s)   kubelet            Back-off pulling image "registry.k8s.io/kube-proxy:v1.27.1"
  Warning  Failed     5s (x6 over 3m23s)   kubelet            Error: ImagePullBackOff

# k8s-slave01节点下载镜像
root@k8s-slave01:~# bash download.sh registry.k8s.io/kube-proxy:v1.27.1
AILI_KUBERNETES_REGISTRY => registry.cn-hangzhou.aliyuncs.com/google_containers
downloading image => registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1
v1.27.1: Pulling from google_containers/kube-proxy
b6425c1785a5: Pull complete 
5730c7a042b6: Pull complete 
Digest: sha256:958ddb03a4d4d7a567d3563c759a05f3e95aa42ca8af2964aa76867aafc43610
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1
rename image registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1 to registry.k8s.io/kube-proxy:v1.27.1

# k8s-slave02节点下载镜像
root@k8s-slave02:~#  bash download.sh registry.k8s.io/kube-proxy:v1.27.1
AILI_KUBERNETES_REGISTRY => registry.cn-hangzhou.aliyuncs.com/google_containers
downloading image => registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1
v1.27.1: Pulling from google_containers/kube-proxy
b6425c1785a5: Pull complete 
5730c7a042b6: Pull complete 
Digest: sha256:958ddb03a4d4d7a567d3563c759a05f3e95aa42ca8af2964aa76867aafc43610
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1
rename image registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1 to registry.k8s.io/kube-proxy:v1.27.

# 看到kube-proxy已经Running并且READY 了
root@k8s-master01:~# kubectl get pod -A  
NAMESPACE     NAME                                       READY   STATUS                  RESTARTS      AGE
kube-system   calico-kube-controllers-6c99c8747f-fslqk   1/1     Running                 0             28m
kube-system   calico-node-6wgk4                          0/1     Init:CrashLoopBackOff   6 (57s ago)   7m38s
kube-system   calico-node-grb97                          1/1     Running                 0             16m
kube-system   calico-node-ltczv                          0/1     Init:CrashLoopBackOff   6 (79s ago)   7m43s
kube-system   calico-node-pffcg                          1/1     Running                 0             17m
kube-system   calico-node-vtcqg                          1/1     Running                 0             28m
kube-system   coredns-5d78c9869d-m5zgd                   1/1     Running                 0             33m
kube-system   coredns-5d78c9869d-mnxzj                   1/1     Running                 0             33m
kube-system   etcd-k8s-master01                          1/1     Running                 0             34m
kube-system   etcd-k8s-master02                          1/1     Running                 0             17m
kube-system   etcd-k8s-master03                          1/1     Running                 0             16m
kube-system   haproxy-k8s-master01                       1/1     Running                 0             34m
kube-system   haproxy-k8s-master02                       1/1     Running                 0             17m
kube-system   haproxy-k8s-master03                       1/1     Running                 0             16m
kube-system   keepalived-k8s-master01                    1/1     Running                 0             34m
kube-system   keepalived-k8s-master02                    1/1     Running                 0             17m
kube-system   keepalived-k8s-master03                    1/1     Running                 0             15m
kube-system   kube-apiserver-k8s-master01                1/1     Running                 0             34m
kube-system   kube-apiserver-k8s-master02                1/1     Running                 0             17m
kube-system   kube-apiserver-k8s-master03                1/1     Running                 1 (16m ago)   16m
kube-system   kube-controller-manager-k8s-master01       1/1     Running                 1 (17m ago)   34m
kube-system   kube-controller-manager-k8s-master02       1/1     Running                 0             17m
kube-system   kube-controller-manager-k8s-master03       1/1     Running                 0             15m
kube-system   kube-proxy-lmw7g                           1/1     Running                 0             16m
kube-system   kube-proxy-mb8hx                           1/1     Running                 0             7m43s
kube-system   kube-proxy-nvx8b                           1/1     Running                 0             7m38s
kube-system   kube-proxy-phvcm                           1/1     Running                 0             33m
kube-system   kube-proxy-psst7                           1/1     Running                 0             17m
kube-system   kube-scheduler-k8s-master01                1/1     Running                 1 (17m ago)   34m
kube-system   kube-scheduler-k8s-master02                1/1     Running                 0             17m
kube-system   kube-scheduler-k8s-master03                1/1     Running                 0             15m

# 但是现在还有Calico网络插件没有Running， 我们看一下原因
root@k8s-master01:~# kubectl describe pod calico-node-ltczv -n kube-system
# Events看到镜像已经下载成功了， 但是Back-off了，这里我们重新启动一下
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  9m38s                   default-scheduler  Successfully assigned kube-system/calico-node-ltczv to k8s-slave01
  Normal   Pulled     9m35s                   kubelet            Container image "docker.io/calico/cni:v3.25.0" already present on machine
  Normal   Created    9m35s                   kubelet            Created container upgrade-ipam
  Normal   Started    9m35s                   kubelet            Started container upgrade-ipam
  Normal   Pulled     7m55s (x5 over 9m35s)   kubelet            Container image "docker.io/calico/cni:v3.25.0" already present on machine
  Normal   Created    7m55s (x5 over 9m35s)   kubelet            Created container install-cni
  Normal   Started    7m54s (x5 over 9m34s)   kubelet            Started container install-cni
  Warning  BackOff    4m32s (x22 over 9m27s)  kubelet            Back-off restarting failed container install-cni in pod calico-node-ltczv_kube-system(c89e2e76-5045-4474-af93-9b839e1d2206)
# 重启一下Calico DaemonSet控制器控制的Pod
root@k8s-master01:~# kubectl -n kube-system rollout restart DaemonSet/calico-node
daemonset.apps/calico-node restarted
root@k8s-master01:~# kubectl get pod -A  
NAMESPACE     NAME                                       READY   STATUS     RESTARTS      AGE
kube-system   calico-kube-controllers-6c99c8747f-fslqk   1/1     Running    0             33m
kube-system   calico-node-bbfx2                          1/1     Running    0             15s
kube-system   calico-node-cf55q                          0/1     Init:2/3   0             4s
kube-system   calico-node-ltczv                          1/1     Running    0             12m
kube-system   calico-node-pffcg                          1/1     Running    0             22m
kube-system   calico-node-vtcqg                          1/1     Running    0             33m
kube-system   coredns-5d78c9869d-m5zgd                   1/1     Running    0             38m
kube-system   coredns-5d78c9869d-mnxzj                   1/1     Running    0             38m
kube-system   etcd-k8s-master01                          1/1     Running    0             39m
kube-system   etcd-k8s-master02                          1/1     Running    0             22m
kube-system   etcd-k8s-master03                          1/1     Running    0             21m
kube-system   haproxy-k8s-master01                       1/1     Running    0             39m
kube-system   haproxy-k8s-master02                       1/1     Running    0             22m
kube-system   haproxy-k8s-master03                       1/1     Running    0             21m
kube-system   keepalived-k8s-master01                    1/1     Running    0             39m
kube-system   keepalived-k8s-master02                    1/1     Running    0             22m
kube-system   keepalived-k8s-master03                    1/1     Running    0             20m
kube-system   kube-apiserver-k8s-master01                1/1     Running    0             39m
kube-system   kube-apiserver-k8s-master02                1/1     Running    0             22m
kube-system   kube-apiserver-k8s-master03                1/1     Running    1 (21m ago)   21m
kube-system   kube-controller-manager-k8s-master01       1/1     Running    1 (22m ago)   39m
kube-system   kube-controller-manager-k8s-master02       1/1     Running    0             22m
kube-system   kube-controller-manager-k8s-master03       1/1     Running    0             20m
kube-system   kube-proxy-lmw7g                           1/1     Running    0             21m
kube-system   kube-proxy-mb8hx                           1/1     Running    0             12m
kube-system   kube-proxy-nvx8b                           1/1     Running    0             12m
kube-system   kube-proxy-phvcm                           1/1     Running    0             38m
kube-system   kube-proxy-psst7                           1/1     Running    0             22m
kube-system   kube-scheduler-k8s-master01                1/1     Running    1 (22m ago)   39m
kube-system   kube-scheduler-k8s-master02                1/1     Running    0             22m
kube-system   kube-scheduler-k8s-master03                1/1     Running    0             20m
# 过一会就都启动了
root@k8s-master01:~# kubectl get pod -A  
NAMESPACE     NAME                                       READY   STATUS    RESTARTS      AGE
kube-system   calico-kube-controllers-6c99c8747f-fslqk   1/1     Running   0             34m
kube-system   calico-node-82c5z                          1/1     Running   0             48s
kube-system   calico-node-9vrzk                          1/1     Running   0             37s
kube-system   calico-node-bbfx2                          1/1     Running   0             69s
kube-system   calico-node-cf55q                          1/1     Running   0             58s
kube-system   calico-node-scrp4                          1/1     Running   0             26s
kube-system   coredns-5d78c9869d-m5zgd                   1/1     Running   0             39m
kube-system   coredns-5d78c9869d-mnxzj                   1/1     Running   0             39m
kube-system   etcd-k8s-master01                          1/1     Running   0             40m
kube-system   etcd-k8s-master02                          1/1     Running   0             23m
kube-system   etcd-k8s-master03                          1/1     Running   0             22m
kube-system   haproxy-k8s-master01                       1/1     Running   0             40m
kube-system   haproxy-k8s-master02                       1/1     Running   0             23m
kube-system   haproxy-k8s-master03                       1/1     Running   0             22m
kube-system   keepalived-k8s-master01                    1/1     Running   0             40m
kube-system   keepalived-k8s-master02                    1/1     Running   0             23m
kube-system   keepalived-k8s-master03                    1/1     Running   0             20m
kube-system   kube-apiserver-k8s-master01                1/1     Running   0             40m
kube-system   kube-apiserver-k8s-master02                1/1     Running   0             23m
kube-system   kube-apiserver-k8s-master03                1/1     Running   1 (22m ago)   22m
kube-system   kube-controller-manager-k8s-master01       1/1     Running   1 (23m ago)   40m
kube-system   kube-controller-manager-k8s-master02       1/1     Running   0             23m
kube-system   kube-controller-manager-k8s-master03       1/1     Running   0             21m
kube-system   kube-proxy-lmw7g                           1/1     Running   0             22m
kube-system   kube-proxy-mb8hx                           1/1     Running   0             13m
kube-system   kube-proxy-nvx8b                           1/1     Running   0             13m
kube-system   kube-proxy-phvcm                           1/1     Running   0             39m
kube-system   kube-proxy-psst7                           1/1     Running   0             23m
kube-system   kube-scheduler-k8s-master01                1/1     Running   1 (23m ago)   40m
kube-system   kube-scheduler-k8s-master02                1/1     Running   0             23m
kube-system   kube-scheduler-k8s-master03                1/1     Running   0             21m

# 节点都是 Ready 状态
root@k8s-master01:~# kubectl get node
NAME           STATUS   ROLES           AGE   VERSION
k8s-master01   Ready    control-plane   41m   v1.27.1
k8s-master02   Ready    control-plane   24m   v1.27.1
k8s-master03   Ready    control-plane   24m   v1.27.1
k8s-slave01    Ready    <none>          15m   v1.27.1
k8s-slave02    Ready    <none>          15m   v1.27.1

可能会出现镜像下载失败问题等等。

8. 高可用Master主节点加入集群（比如使用Root用户执行）

参考： https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/high-availability/

​ 如果使用kubeadm init时，没有指定--upload-certs 选项， 则可以现在重新配置上传证书阶段：

root@k8s-master01:~# kubeadm init phase upload-certs --upload-certs
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
c82900d92a026aa6f6498b41ea70c9602e052c88eaca3e019d99b297af43230e
  • 不要忘记默认情况下，--certificate-key 中的解密秘钥会在两个小时后过期。

​ 我们这里就是root@k8s-master02和 root@k8s-master03 这台机器， 使用上面命令生成的 certificate key: c82900d92a026aa6f6498b41ea70c9602e052c88eaca3e019d99b297af43230e：

root@k8s-master02:~#  kubeadm join k8s-master:8443 --token th5i1f.fnzc9v0yb6z3aok8 \
        --discovery-token-ca-cert-hash sha256:25357bff7f44a787886222dc9439916ab271dc5af5d5bbef274288fdd8e245b4 \
        --control-plane \
    --certificate-key 3636bc7d84515aeb36ca79597792b07cc64a888ebdea9221ab68a5bae93ac947 \
    --cri-socket=unix:///var/run/cri-dockerd.sock 
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W0520 20:29:27.062790   14892 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.1, falling back to the nearest etcd version (3.5.7-0)
W0520 20:29:27.337990   14892 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[download-certs] Saving the certificates to the folder: "/etc/kubernetes/pki"
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master k8s-master02 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.20]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master02 localhost] and IPs [192.168.0.20 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master02 localhost] and IPs [192.168.0.20 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
W0520 20:29:29.131836   14892 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
W0520 20:29:29.206366   14892 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
W0520 20:29:29.479200   14892 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[check-etcd] Checking that the etcd cluster is healthy
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Creating static Pod manifest for "etcd"
W0520 20:29:31.931154   14892 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.1, falling back to the nearest etcd version (3.5.7-0)
[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s
The 'update-status' phase is deprecated and will be removed in a future release. Currently it performs no operation
[mark-control-plane] Marking the node k8s-master02 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master02 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

配置节点kubectl命令

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

查看节点是否正常 （到此基本集群安装完成）

root@k8s-master03:~# kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
k8s-master01   Ready    control-plane   18m   v1.27.1
k8s-master02   Ready    control-plane   90s   v1.27.1
k8s-master03   Ready    control-plane   37s   v1.27.1

9. 可选操作

（可选）从控制平面节点以外的计算机控制集群

为了使 kubectl 在其他计算机（例如笔记本电脑）上与你的集群通信， 你需要将管理员 kubeconfig 文件从控制平面节点复制到工作站，如下所示：

scp root@<control-plane-host>:/etc/kubernetes/admin.conf .
kubectl --kubeconfig ./admin.conf get nodes

说明：

上面的示例假定为 root 用户启用了 SSH 访问。如果不是这种情况， 你可以使用 scp 将 admin.conf 文件复制给其他允许访问的用户。

admin.conf 文件为用户提供了对集群的超级用户特权。 该文件应谨慎使用。对于普通用户，建议生成一个你为其授予特权的唯一证书。 你可以使用 kubeadm alpha kubeconfig user --client-name <CN> 命令执行此操作。 该命令会将 KubeConfig 文件打印到 STDOUT，你应该将其保存到文件并分发给用户。 之后，使用 kubectl create (cluster)rolebinding 授予特权。

（可选）将 API 服务器代理到本地主机

如果要从集群外部连接到 API 服务器，则可以使用 kubectl proxy：

scp root@<control-plane-host>:/etc/kubernetes/admin.conf .
kubectl --kubeconfig ./admin.conf proxy

你现在可以在本地访问 API 服务器 http://localhost:8001/api/v1。

声明：本站所有文章，如无特殊说明或标注，均为本站原创发布。任何个人或组织，在未征得本站同意时，禁止复制、盗用、采集、发布本站内容到任何网站、书籍等各类媒体平台。如若本站内容侵犯了原著者的合法权益，可联系我们进行处理。