1.k8s概述
1.1什么是k8s
参考链接（官网）:
https://kubernetes.io/zh-cn/
Kubernetes也称为K8s，是用于自动部署、扩缩和管理容器化应用程序的开源系统。
它将组成应用程序的容器组合成逻辑单元，以便于管理和服务发现。Kubernetes 源自 Google 15年生产环境的运维经验，同时凝聚了社区的最佳创意和实践。
Google每周运行数十亿个容器，Kubernetes基于与之相同的原则来设计，能够在不扩张运维团队的情况下进行规模扩展。
无论是本地测试，还是跨国公司，Kubernetes的灵活性都能让你在应对复杂系统时得心应手。
Kubernetes是开源系统，可以自由地部署在企业内部，私有云、混合云或公有云，让您轻松地做出合适的选择。1.2k8s架构
- master(主人)| control plane(控制面板)
- etcd
数据库，用于存储k8s集群数据。(并不是Google公司开发,只是用etcd去做存储)
- api-server：
k8s集群控制访问入口。
- scheduler:
调度器，负责调度的相关工作。
- controller manager
负责维护K8S集群状态,管理控制器。
- slave(奴隶)|worker(工作者)
- kubelet
负责Pod生命周期及worker节点状态监控并周期性上报给api-server组件。
- kube-proxy
负责代理Pod请求，实现集群内部或集群外部的负载均衡和服务发现。
- CNI(Container Network Interface):
负责k8s集群worker节点的Pod网络通信。2.k8s环境准备及基础优化
2.1环境准备
| 主机名 | IP地址 | 操作系统 | 硬件配置 |
| master231 | 10.0.0.231 | Ubuntu 22.04 LTS | 2c+，4G+，50G+ |
| worker232 | 10.0.0.232 | Ubuntu 22.04 LTS | 2c+，4G+，50G+ |
| worker233 | 10.0.0.233 | Ubuntu 22.04 LTS | 2c+，4G+，50G+ |
2.2Linux基础优化
1.关闭swap分区
swapoff -a && sysctl -w vm.swappiness=0 # 临时关闭，服务器重启后配置失效。
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab # 基于配置文件关闭，重启后依旧有效。
systemctl mask swap.target # ubt/debian系统，还需执行这个命令
2.确保各个节点MAC地址或product_uuid唯一
ifconfig ens33 | grep ether | awk '{print $2}'
cat /sys/class/dmi/id/product_uuid
温馨提示:
一般来讲，硬件设备会拥有唯一的地址，但是有些虚拟机的地址可能会重复。
Kubernetes使用这些值来唯一确定集群中的节点。 如果这些值在每个节点上不唯一，可能会导致安装失败。
3.检查网络节点是否互通
简而言之，就是检查你的k8s集群各节点是否互通，可以使用ping命令来测试。
ping baidu.com -c 10
4.允许iptable检查桥接流量
cat <<EOF | tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
5.检查端口是否被占用
参考: https://kubernetes.io/zh-cn/docs/reference/networking/ports-and-protocols/
6.Linux内核优化
cat > /etc/sysctl.d/k8s.conf <<'EOF'
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv6.conf.all.disable_ipv6 = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
sysctl --system2.3安装ipvsadm以实现kube-proxy的负载均衡
1 安装ipvsadm等相关工具
apt -y install ipvsadm ipset sysstat conntrack
2 所有节点创建要开机自动加载的模块配置文件
cat > /etc/modules-load.d/ipvs.conf << 'EOF'
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF2.4安装docker环境
1 安装docker环境
wget http://192.168.21.253/Resources/Docker/scripts/oldboyedu-autoinstall-docker-docker-compose.tar.gz
tar xf oldboyedu-autoinstall-docker-docker-compose.tar.gz
./install-docker.sh i
2 检查cgroup驱动是否是systemd
[root@master231 ~]# docker info | grep "Cgroup Driver:"
Cgroup Driver: systemd
[root@worker232 ~]# docker info | grep "Cgroup Driver:"
Cgroup Driver: systemd
[root@worker233 ~]# docker info | grep "Cgroup Driver:"
Cgroup Driver: systemd2.5所有节点安装kubeadm，kubelet，kubectl
| 软件包名称\描述 | 作用 |
| kubeadm | 用来初始化K8S集群的工具 |
| kubelet | 底层用到了静态Pod技术启动master组件及Pod生命周期管理。 |
| kubectl | 用来与K8S集群通信的命令行工具 |
1.软件包说明(如上表所示)
kubeadm不能帮你安装或者管理kubelet或kubectl，所以你需要确保它们与通过kubeadm安装的控制平面(master)的版本相匹配。
如果不这样做，则存在发生版本偏差的风险，可能会导致一些预料之外的错误和问题。
然而，控制平面与kubelet间的相差一个次要版本不一致是支持的，但kubelet的版本不可以超过"API SERVER"的版本。
例如，1.7.0版本的kubelet可以完全兼容1.8.0版本的"API SERVER"，反之则不可以。
2 K8S所有节点配置软件源
apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
apt-get update
3 查看一下当前环境支持的k8s版本
[root@master231 ~]# apt-cache madison kubeadm
kubeadm | 1.28.2-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
kubeadm | 1.28.1-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
kubeadm | 1.28.0-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
...
kubeadm | 1.23.17-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
kubeadm | 1.23.16-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
kubeadm | 1.23.15-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
kubeadm | 1.23.14-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
...
4 所有节点安装 kubelet kubeadm kubectl
apt-get -y install kubelet=1.23.17-00 kubeadm=1.23.17-00 kubectl=1.23.17-00
5 检查各组件版本
[root@master231 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:33:14Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
[root@master231 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:34:27Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?
[root@master231 ~]# kubelet --version
Kubernetes v1.23.17
[root@worker232 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:33:14Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
[root@worker232 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:34:27Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?
[root@worker232 ~]# kubelet --version
Kubernetes v1.23.17
[root@worker233 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:33:14Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
[root@worker233 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:34:27Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?
[root@worker233 ~]# kubelet --version
Kubernetes v1.23.17
2.6时区优化及快照
1.检查时区
[root@master231 ~]# ln -svf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
'/etc/localtime' -> '/usr/share/zoneinfo/Asia/Shanghai'
[root@master231 ~]# ll /etc/localtime
lrwxrwxrwx 1 root root 33 Feb 10 11:26 /etc/localtime -> /usr/share/zoneinfo/Asia/Shanghai
[root@master231 ~]# date -R
Tue, 06 Jan 2026 10:40:14 +0800
2.验证cpu核心数
[root@master231 ~]# lscpu | grep ^CPU\(s\)
CPU(s): 2
[root@worker232 ~]# lscpu | grep ^CPU\(s\)
CPU(s): 2
[root@worker233 ~]# lscpu | grep ^CPU\(s\)
CPU(s): 2
3.重启操作系统即可
reboot
4.验证加载的模块
lsmod | grep --color=auto -e ip_vs -e nf_conntrack
free -h
温馨提示:
Linux kernel 4.19+版本已经将之前的"nf_conntrack_ipv4"模块更名为"nf_conntrack"模块哟~
5.关机拍快照3.k8s集群部署
3.1部署master组件
1.提前导入镜像
[root@master231 ~]# wget http://192.168.21.253/Resources/Kubernetes/K8S%20Cluster/kubeadm/images/oldboyedu-master-1.23.17.tar.gz
[root@master231 ~]# docker load -i oldboyedu-master-1.23.17.tar.gz
[root@master231 ~]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-apiserver v1.23.17 62bc5d8258d6 23 months ago 130MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.23.17 1dab4fc7b6e0 23 months ago 120MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.23.17 bc6794cb54ac 23 months ago 51.9MB
registry.aliyuncs.com/google_containers/kube-proxy v1.23.17 f21c8d21558c 23 months ago 111MB
registry.aliyuncs.com/google_containers/etcd 3.5.6-0 fce326961ae2 2 years ago 299MB
registry.aliyuncs.com/google_containers/coredns v1.8.6 a4ca41631cc7 3 years ago 46.8MB
registry.aliyuncs.com/google_containers/pause 3.6 6270bb605e12 3 years ago 683kB
2.使用kubeadm初始化master节点
[root@master231 ~]# kubeadm init --kubernetes-version=v1.23.17 --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.100.0.0/16 --service-cidr=10.200.0.0/16 --service-dns-domain=oldboyedu.com
...
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.0.0.231:6443 --token dnc6v7.0c2a0yd101h24lzx \
--discovery-token-ca-cert-hash sha256:8dacfe663585bb1c36023aff0ceb991ceae9f833a50245fdd63001e26bd61013
相关参数说明:
--kubernetes-version:
指定K8S master组件的版本号。
--image-repository:
指定下载k8s master组件的镜像仓库地址。
--pod-network-cidr:
指定Pod的网段地址。
--service-cidr:
指定SVC的网段
--service-dns-domain:
指定service的域名。若不指定，默认为"cluster.local"。
使用kubeadm初始化集群时，可能会出现如下的输出信息:[init]
使用初始化的K8S版本。
[preflight]
主要是做安装K8S集群的前置工作，比如下载镜像，这个时间取决于你的网速。
[certs]
生成证书文件，默认存储在”/etc/kubernetes/pki”目录哟。
[kubeconfig]
生成K8S集群的默认配置文件，默认存储在”/etc/kubernetes”目录哟。
[kubelet-start]
启动kubelet， 环境变量默认写入:”/var/lib/kubelet/kubeadm-flags.env” 配置文件默认写入:”/var/lib/kubelet/config.yaml”
[control-plane]
使用静态的目录，默认的资源清单存放在:”/etc/kubernetes/manifests”。 此过程会创建静态Pod，包括”kube-apiserver”，”kube-controller-manager”和”kube-scheduler”
[etcd]
创建etcd的静态Pod，默认的资源清单存放在:”/etc/kubernetes/manifests”
[wait-control-plane]
等待kubelet从资源清单目录”/etc/kubernetes/manifests”启动静态Pod。
[apiclient]
等待所有的master组件正常运行。
[upload-config]
创建名为”kubeadm-config”的ConfigMap在”kube-system”名称空间中。
[kubelet]
创建名为”kubelet-config-1.22″的ConfigMap在”kube-system”名称空间中，其中包含集群中kubelet的配置
[upload-certs]
跳过此节点，详情请参考”–upload-certs”
[mark-control-plane]
标记控制面板，包括打标签和污点，目的是为了标记master节点。
[bootstrap-token]
创建token口令，例如:”kbkgsa.fc97518diw8bdqid”。 如下图所示，这个口令将来在加入集群节点时很有用，而且对于RBAC控制也很有用处哟。
[kubelet-finalize]
更新kubelet的证书文件信息
[addons]
3.2部署worker组件
1.提前导入镜像
[root@worker232 ~]# wget http://192.168.21.253/Resources/Kubernetes/K8S%20Cluster/kubeadm/images/oldboyedu-slave-1.23.17.tar.gz
[root@worker232 ~]# docker load -i oldboyedu-slave-1.23.17.tar.gz
[root@worker232 ~]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-proxy v1.23.17 f21c8d21558c 2 years ago 111MB
registry.aliyuncs.com/google_containers/coredns v1.8.6 a4ca41631cc7 3 years ago 46.8MB
registry.aliyuncs.com/google_containers/pause 3.6 6270bb605e12 3 years ago 683kB
[root@worker233 ~]# wget http://192.168.21.253/Resources/Kubernetes/K8S%20Cluster/kubeadm/images/oldboyedu-slave-1.23.17.tar.gz
[root@worker233 ~]# docker load -i oldboyedu-slave-1.23.17.tar.gz
[root@worker233 ~]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-proxy v1.23.17 f21c8d21558c 2 years ago 111MB
registry.aliyuncs.com/google_containers/coredns v1.8.6 a4ca41631cc7 3 years ago 46.8MB
registry.aliyuncs.com/google_containers/pause 3.6 6270bb605e12 3 years ago 683kB
2.将worker节点加入到master集群（注意，不要复制我的，而是根据你上一步master生成的token加入集群）
[root@worker232 ~]# kubeadm join 10.0.0.231:6443 --token dnc6v7.0c2a0yd101h24lzx \
--discovery-token-ca-cert-hash sha256:8dacfe663585bb1c36023aff0ceb991ceae9f833a50245fdd63001e26bd61013
[root@worker233 ~]# kubeadm join 10.0.0.231:6443 --token dnc6v7.0c2a0yd101h24lzx \
--discovery-token-ca-cert-hash sha256:8dacfe663585bb1c36023aff0ceb991ceae9f833a50245fdd63001e26bd61013
3.验证worker节点是否加入成功
[root@master231 ~]# kubectl get no
NAME STATUS ROLES AGE VERSION
master231 NotReady control-plane,master 8m1s v1.23.17
worker232 NotReady <none> 42s v1.23.17
worker233 NotReady <none> 38s v1.23.17
[root@master231 ~]# kubectl get no -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master231 NotReady control-plane,master 8m2s v1.23.17 10.0.0.231 <none> Ubuntu 22.04.4 LTS 5.15.0-119-generic docker://20.10.24
worker232 NotReady <none> 43s v1.23.17 10.0.0.232 <none> Ubuntu 22.04.4 LTS 5.15.0-119-generic docker://20.10.24
worker233 NotReady <none> 39s v1.23.17 10.0.0.233 <none> Ubuntu 22.04.4 LTS 5.15.0-119-generic docker://20.10.243.3k8s所有节点关机拍快照
可以删除无用的软件包后再关机拍快照，建议快照名称为: ‘k8s未安装CNI插件’。作者
2632782425@qq.com
相关文章
