ubuntu部署k8s无VIP多Master部署方案

发布于2026-06-03 16:43:10 更新于2026-06-03 17:36:21
运维
浏览 27
shevechco
手机浏览
评论数 0

ubuntu部署k8s无VIP多Master部署方案，默认使用IPVS，这种方案的优势在于无需机房网络这块再多划分VIP地址段，不需要ARP生成VIP，所以也就不需要同二层广播域(同一个内网二层网段)，交换机ARP无拦截或者是无静态ARP绑定等这些需求，可以支持跨三层路由，也不会产生内网冲突的风险。另外有些场景下可能VIP漂移切换失败的问题也能避免。但无VIP多Master集群在k8多Master集群中还是属于非主流方案，不是没办法不建议使用这种方案。

我们租赁裸金属服务器会遇到服务商无法提供VIP的情况，虽然这种情况比较少见，但是还是有，所以今天记录下无VIP部署多Master的k8s方案。

1.安装基础软件包

sudo apt update -y
sudo apt install -y gcc gcc+ make apt-transport-https ca-certificates curl gnupg-agent gnupg lsb-release make software-properties-common net-tools git curl ntpdate haproxy

2.修改所有主机名

我们需要对所有主机名进行规范，命名规则就是GPU+内网IP，IP地址中的点换成横线

hostnamectl set-hostname master

修改hosts绑定内网IP和主机，ip和主机名根据情况修改为自己的

sudo tee -a /etc/hosts<<EOF
192.168.0.10 GPU-192-168-0-10
192.168.0.11 GPU-192-168-0-11
192.168.0.12 GPU-192-168-0-12
EOF

3.关闭swap和关闭防火墙

关闭swap

sudo sed -i '/swap/d' /etc/fstab
sudo swapoff -a
sudo systemctl stop swap.target
sudo systemctl disable swap.target

关闭防火墙

sudo systemctl stop ufw
sudo systemctl disable ufw

4.内核相关优化和加载相关内核模块

4.1.配置k8s相关需要的内核参数

sudo tee /etc/sysctl.d/k8s.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF

4.2.优化最大打开inotify实例数量

sudo tee -a /etc/sysctl.conf<<EOF
fs.inotify.max_user_instances=512
fs.inotify.max_user_watches=262144
EOF
sysctl --system

4.3.加载内核模块

sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF

手动打开和检查加载的内核模块

sudo modprobe br_netfilter
sudo modprobe overlay
lsmod | grep -iE 'br_netfilter|overlay'

4.4.开启内核转发

echo "1" > /proc/sys/net/ipv4/ip_forward

4.5.配置ipvs并加载到内核

sudo mkdir -p /etc/sysconfig/modules/
sudo tee /etc/sysconfig/modules/ipvs.modules<<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
sudo chmod 755 /etc/sysconfig/modules/ipvs.modules
sudo bash /etc/sysconfig/modules/ipvs.modules

4.6.禁止内核自动更新

sudo rm -f /etc/apt/apt.conf.d/50unattended-upgrades >/dev/null 2>&1

sudo sed -i '/Update-Package-Lists/s/1/0/' /etc/apt/apt.conf.d/10periodic
sudo sed -i '/Unattended-Upgrade/s/1/0/' /etc/apt/apt.conf.d/10periodic
sudo sed -i '/Update-Package-Lists/s/1/0/' /etc/apt/apt.conf.d/20auto-upgrades
sudo sed -i '/Unattended-Upgrade/s/1/0/' /etc/apt/apt.conf.d/20auto-upgrades

sudo systemctl stop unattended-upgrades.service
sudo systemctl disable unattended-upgrades.service
sudo systemctl stop apt-daily.timer apt-daily-upgrade.timer
sudo systemctl disable apt-daily.timer apt-daily-upgrade.timer

sudo systemctl stop unattended-upgrades.service
sudo systemctl disable unattended-upgrades.service
for i in `dpkg --list | grep -E 'linux-(headers|image|modules)-[0-9]' | awk '{print $2}'`
do sudo apt-mark hold $i
done

5.修改时区同步时间

sudo timedatectl set-timezone Asia/Shanghai
ntpdate time.windows.com

6.部署containerd容器

wget https://github.com/containerd/containerd/releases/download/v1.7.28/cri-containerd-1.7.28-linux-amd64.tar.gz
tar -zxvf cri-containerd-1.7.28-linux-amd64.tar.gz -C /
mkdir -p /etc/containerd
containerd config default >/etc/containerd/config.toml

修改config.toml配置

sudo sed -i "s@registry.k8s.io/pause:3.8@registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.10@g" /etc/containerd/config.toml
sudo sed -i '/SystemdCgroup/s/false/true/g' /etc/containerd/config.toml
sudo systemctl daemon-reload
sudo systemctl enable --now containerd

配置crictl工具

sudo tee /etc/crictl.yaml >/dev/null <<EOF
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: false
EOF

7.安装容器管理工具nerdctl

https://github.com/containerd/nerdctl/releases/download/v2.1.4/nerdctl-2.1.4-linux-amd64.tar.gz
tar -zxvf nerdctl-2.1.4-linux-amd64.tar.gz
sudo cp -a nerdctl /usr/local/bin/

8.部署配置k8s相关组件

8.1.安装k8s组件

K8S_VERSION=v1.33 #安装指定版本的k8s
sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/${K8S_VERSION}/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${K8S_VERSION}/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update -y
sudo apt install -y kubelet kubeadm kubectl

8.2.禁止k8s组件自动更新

sudo apt-mark hold kubelet kubectl
sudo systemctl enable kubelet

8.3.修改k8s cgroup配置

echo 'KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"' | sudo tee /etc/default/kubelet

8.4.指定k8s使用的内网IP

HOST_IP="192.168.0.10" #改为你当前部署机器内网IP
sudo sed -i "s/\$KUBELET_EXTRA_ARGS/\$KUBELET_EXTRA_ARGS --node-ip=${HOST_IP}/g" /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
sudo systemctl daemon-reload

9.配置haproxy代理k8s通信端口

我们需要先配置代理端口，因为加入集群都需要使用这个代理端口来加，每个节点都需要启动haproxy来代理，不管是Master节点还是Worker节点

cat /etc/haproxy/haproxy.cfg
global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        mode                    http
        log                     global
        option                  httplog
        option                  dontlognull
        option http-server-close
        option                  redispatch
        retries                 3
        timeout http-request    10s
        timeout queue           1m
        timeout connect         10s
        timeout client          1m
        timeout server          1m
        timeout http-keep-alive 10s
        timeout check           10s
        maxconn                 3000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

listen master
        bind 0.0.0.0:16443
        mode tcp
        option tcplog
        balance roundrobin
        server master1 192.168.0.10:6443 check inter 2000 fall 2 rise 2 weight 1
        server master2 192.168.0.11:6443 check inter 2000 fall 2 rise 2 weight 1
        server master3 192.168.0.12:6443 check inter 2000 fall 2 rise 2 weight 1

按照上述配置好以后重启下haproxy

sudo systemctl restart haproxy

然后检查下16443端口是否监听

netstat -ntpl | grep :16443

netstat -ntpl | grep :16443

10.初始化k8s配置

10.1.生成并修改k8s初始化配置

生成配置

kubeadm config print init-defaults | sudo tee /etc/kubernetes/default.yaml

修改配置如下，需要注意的是certSANs配置下是三个Master节点IP和127.0.0.1这个地址，另外控制平面使用127.0.0.1:16443

cat /etc/kubernetes/default.yaml
apiVersion: kubeadm.k8s.io/v1beta4
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.0.10
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  imagePullSerial: true
  name: GPU-192-168-0-10
  taints: null
timeouts:
  controlPlaneComponentHealthCheck: 4m0s
  discovery: 5m0s
  etcdAPICall: 2m0s
  kubeletHealthCheck: 4m0s
  kubernetesAPICall: 1m0s
  tlsBootstrap: 5m0s
  upgradeManifests: 5m0s
---
apiServer:
  timeoutForControlPlane: 4m0s
  certSANs:
  - 192.168.0.10
  - 192.168.0.11
  - 192.168.0.12
  - 127.0.0.1
apiVersion: kubeadm.k8s.io/v1beta4
caCertificateValidityPeriod: 87600h0m0s
certificateValidityPeriod: 8760h0m0s
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
encryptionAlgorithm: RSA-2048
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: 1.33.0
controlPlaneEndpoint: 127.0.0.1:16443
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.244.0.0/16
proxy: {}
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
  scheduler: "rr"
  minSyncPeriod: "1s"
  syncPeriod: "30s"
  strictARP: true

10.2.预拉取镜像

sudo kubeadm config images pull --config /etc/kubernetes/default.yaml

10.3.初始化k8s

sudo kubeadm init --config=/etc/kubernetes/default.yaml --upload-certs

初始化以后会打印一些加入集群命令，我们看到打印的命令都是join通过127.0.0.1:16443这个地址的，后续所有节点都需要通过这种方式加入集群，所以需要提前部署好haproxy。

10.4.管理配置

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

标签
k8s
ubuntu
vip
master

暂时没有上一篇

Ingress超时配置的几种方法

转载注明出处：https://www.sulao.cn/post/1176

今日天气

分类统计

博文归档

2篇

5篇

3篇

1篇

2篇

8篇

6篇

6篇

3篇

12篇

15篇

43篇

热门推荐

热门标签