这里二进制安装参考下面文章,文件路径都给予下面的文章,如果有不相同的地方可以自行修改即可。 有问题请在下方提问!
本次升级的环境版本为1.14升级到1.15版本
在每个大版本中,k8s组件的配置文件以及相关参数可能会发生变化,修改好对应的参数问题一般影响不大。
官方文档下载地址:https://github.com/kubernetes/kubernetes/releases
API版本变更适配
对于不同版本的Kubernetes,许多资源对象的API的版本可能会变更。可以通过下面的命令获取当前的API信息
[root@k8s-01 ~]# kubectl explain deployment.apiVersion KIND: Deployment VERSION: extensions/v1beta1 FIELD: apiVersion DESCRIPTION: APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources
升级kubectl
备份kubectl,在所有的master节点操作
cd /opt/k8s/bin [root@k8s-01 bin]# mv kubectl{,.bak_2020-04-13}
接下来我们下载最新的二进制包
mkdir -p /opt/new-client wget -P /opt/new-client http://down.i4t.com/k8s1.15/kubernetes-client-linux-amd64.tar.gz cd /opt/new-client tar xf kubernetes-client-linux-amd64.tar.gz
分发kubectl到所有节点
cd /opt/new-client source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" scp kubernetes/client/bin/kubectl root@${node_ip}:/opt/k8s/bin/ ssh root@${node_ip} "chmod +x /opt/k8s/bin/*" done
升级前
[root@k8s-01 bin]# kubectl version Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:23:09Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:14:56Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
升级后
因为apiserver还没有升级,所以在Server Version中显示为1.14版本
[root@k8s-01 new]# kubectl version Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:08:59Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:14:56Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
升级master组件
kubectl命令行工具已经升级完毕,接下来我们升级master (这里使用测试环境进行演示,线上环境可以一台一台的升级替换)
#在所有master节点执行 (升级线上环境可以不停止keeplived) systemctl stop keepalived # 先停掉本机keepalived,切走高可用VIP地址 systemctl stop kube-apiserver systemctl stop kube-scheduler systemctl stop kube-controller-manager
备份旧版本
for i in k8s-01 k8s-02 k8s-03 #这里只需要master节点,因为现在升级是master节点 do ssh root@$i mv /opt/k8s/bin/kube-apiserver{,.bak_2020-04-13} ssh root@$i mv /opt/k8s/bin/kube-controller-manager{,.bak_2020-04-13} ssh root@$i mv /opt/k8s/bin/kube-scheduler{,.bak_2020-04-13} ssh root@$i mv /opt/k8s/bin/kubeadm{,.bak_2020-04-13} done
升级拷贝新版本
mkdir /opt/new-server-k8s wget -P /opt/new-server-k8s/ down.i4t.com/k8s1.15/kubernetes-server-linux-amd64.tar.gz cd /opt/new-server-k8s tar xf kubernetes-server-linux-amd64.tar.gz #复制新版本客户端 cd /opt/new-server-k8s source /opt/k8s/bin/environment.sh for node_ip in ${MASTER_IPS[@]} do echo ">>> ${node_ip}" scp kubernetes/server/bin/kube-apiserver root@${node_ip}:/opt/k8s/bin/ scp kubernetes/server/bin/{kube-controller-manager,kube-scheduler,kubeadm} root@${node_ip}:/opt/k8s/bin/ ssh root@${node_ip} "chmod +x /opt/k8s/bin/*" done
接下来在所有master启动keepalived和apiserver
#所有master节点启动 systemctl start keepalived systemctl start kube-apiserver
启动时可以通过下面的命令查看服务是否有不兼容的问题
journalctl -fu kube-apiserver
可以查看到etcd中的数据说明kube-apiserver没有问题
#这里目前只有apiserver数据恢复,所以只要能出结果说明apiserver正常 [root@k8s-01 new-server-k8s]# kubectl get cs NAME STATUS MESSAGE ERROR scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused etcd-1 Healthy {"health":"true"} etcd-2 Healthy {"health":"true"} etcd-0 Healthy {"health":"true"}
并且查看客户端和服务端的版本都是v1.15.11,说明版本相同没有问题
[root@k8s-01 new-server-k8s]# kubectl version Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:08:59Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:00:06Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
接下来启动其他组件
systemctl start kube-controller-manager && systemctl start kube-scheduler #所有master节点执行
查看启动状态,此时kubernetes集群已经恢复
[root@k8s-01 ~]# kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-1 Healthy {"health":"true"} etcd-2 Healthy {"health":"true"} etcd-0 Healthy {"health":"true"}
升级node组件
现在需要停止服务,并且备份二进制文件
#在所有node节点执行 systemctl stop kubelet systemctl stop kube-proxy
所有node节点备份二进制文件
for i in k8s-01 k8s-02 k8s-03 k8s-04 k8s-05 do ssh root@$i mv /opt/k8s/bin/kubelet{,.bak_2020-04-13} ssh root@$i mv /opt/k8s/bin/kube-proxy{,.bak_2020-04-13} done
现在拷贝新的二进制文件
cd /opt/new-server-k8s source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" scp kubernetes/server/bin/{kubelet,kube-proxy} root@${node_ip}:/opt/k8s/bin/ ssh root@${node_ip} "chmod +x /opt/k8s/bin/*" done
在kubernetes 1.15版起kubelet需要删除下面的参数,因为新版本中这个参数已经过时。我们只需要在kubelet启动脚步中删除即可
参考https://github.com/microsoft/SDN/issues/379
#需要在kubelet启动文件删除相关参数 allow-privileged=true #这里需要在每台node节点执行! sed -i '/allow-privileged/d' /etc/systemd/system/kubelet.service systemctl daemon-reload && systemctl start kubelet 执行完命令后可以查看一下kubelet日志,检查是否有报错 journalctl -fu kubelet
kubelet成功启动后通过kubectl get node
所有节点的版本已经变为1.15.11
这里的过程可能比较慢,耐心等待一会就好了。 如果长时间处于NotReady就需要查看一下kubelet日志
[root@k8s-01 ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-01 Ready 3d10h v1.15.11 k8s-02 Ready 3d10h v1.15.11 k8s-03 Ready 3d10h v1.15.11 k8s-04 Ready 3d10h v1.15.11 k8s-05 Ready 3d10h v1.15.11
目前还没有结束,我们还需要启动kube-proxy
for i in k8s-01 k8s-02 k8s-03 k8s-04 k8s-05 do ssh root@$i systemctl start kube-proxy done
这里很多的命令都可以通过之前安装k8s的脚本进行获取,比如查看一下kube-proxy状态
source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh root@${node_ip} "systemctl status kube-proxy|grep Active" done
接下来我们可以验证一下集群的状态
[root@k8s-01 ~]# kubectl cluster-info Kubernetes master is running at https://192.168.31.105:8443 CoreDNS is running at https://192.168.31.105:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. [root@k8s-01 ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-01 Ready 3d11h v1.15.11 k8s-02 Ready 3d11h v1.15.11 k8s-03 Ready 3d11h v1.15.11 k8s-04 Ready 3d11h v1.15.11 k8s-05 Ready 3d11h v1.15.11 [root@k8s-01 ~]# kubectl get cs NAME STATUS MESSAGE ERROR etcd-0 Healthy {"health":"true"} etcd-2 Healthy {"health":"true"} etcd-1 Healthy {"health":"true"} scheduler Healthy ok controller-manager Healthy ok
由于之前我这里安装了coredns,所以我在检查一下coredns是否正常
#创建测试yaml cat<<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: busybox namespace: default spec: containers: - name: busybox image: busybox:1.28.3 command: - sleep - "3600" imagePullPolicy: IfNotPresent restartPolicy: Always EOF #使用nslookup查看是否能返回地址 [root@k8s-01 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE busybox 1/1 Running 0 26s [root@k8s-01 ~]# kubectl exec -ti busybox -- nslookup kubernetes Server: 10.254.0.2 Address 1: 10.254.0.2 kube-dns.kube-system.svc.cluster.local Name: kubernetes Address 1: 10.254.0.1 kubernetes.default.svc.cluster.local