Kubernetes 入门（3）- Kubernetes 安装后的服务管理

文章转载自： https://www.skynemo.cn/archives/02-k8s-after-install-configure

本文中所有操作均以 Kubernetes 1.23.5 为基准

配置 kubectl 命令补全

官方参考文档： kubectl的可选配置和插件｜Kubernetes

为方便使用，kubectl 为 Bash、Zsh、Fish 和 PowerShell 提供了自动补全功能，本文以 Bash 为例配置命令补全

补全脚本依赖于工具 bash-completion， 需要先安装该工具（可以用命令 type _init_completion 检查 bash-completion 是否已安装）。如果使用包管理器安装的 kubectl，默认已经安装 bash-completion 工具，并且已导入 SHELL，只需要启用补全功能即可

简介

kubectl 的 Bash 补全脚本可以用命令 kubectl completion bash 生成。 在 shell 中导入（Sourcing）补全脚本，就可以启用 kubectl 自动补全功能

安装 bash-completion

安装

很多包管理工具均支持 bash-completion（参见这里），可以直接使用包管理工具进行安装

# Debian 系
apt-get install bash-completion

# RedHat 系
yum install bash-completion

安装后将创建文件 /usr/share/bash-completion/bash_completion，它是 bash-completion 工具的主脚本，在工作时会导入 /etc/bash_completion.d 目录中的所有补全脚本。

导入脚本

依据包管理工具的实际情况，你需要在文件中手工导入ash-completion 工具的主脚本（包管理器安装一般不需要导入）

重新加载 shell（重开会话），并运行命令 type _init_completion。 如果命令执行成功，则设置完成，否则需要将运行函数（source /usr/share/bash-completion/bash_completion）加入到 SHELL 的 ~/.bashrc 配置文件中

$ vim ~/.bashrc
# 文件末尾新增以下导入脚本命令
source /usr/share/bash-completion/bash_completion

检查

root@kube-master-1:~# type _init_completion
_init_completion is a function
_init_completion ()
{
......

启动 kubectl 自动补全功能

两种方式的效果相同。重新加载 shell 后（重开会话），kubectl 自动补全功能即可生效（按 tab 键补全）

方式一：只对当前用户补全

root@kube-master-1:~# echo 'source <(kubectl completion bash)' >>~/.bashrc

方式二：对所有系统用户补全

root@kube-master-1:~# kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null

别名适配

如果 kubectl 有关联的别名，可以扩展 shell 补全来适配此别名

echo 'alias k=kubectl' >>~/.bashrc
echo 'complete -F __start_kubectl k' >>~/.bashrc

安装 Dashboard

Dashboard 的 Github地址：kubernetes/dashboard: General-purpose web UI for Kubernetes clusters (github.com)

Dashboard 可以用来简单的管理 Kubernetes 集群，包括查看集群各方面的统计状态、进入 pod 的控制台等等操作

版本选择

版本发布表： Releases · kubernetes/dashboard (github.com)

查看 Dashboard 版本发布列表，根据 Kubernetes 版本选择支持的 Dashboard 版本，此处选择安装 Dashboard v2.5.1

近期版本对应情况

Dashboard版本	Kubernetes版本
v2.4.0	1.20
v2.4.0	1.21
无完全支持版本	1.22
v2.5.1	1.23

安装

下载配置文件

root@kube-master-1:~# curl -o dashboard-v2.5.1-release.yaml \
        https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.1/aio/deploy/recommended.yaml

# 如果无法访问外网，或访问速度较慢，可以使用作者的配置备份
# curl -O https://resource-1258540788.file.myqcloud.com/99-external-resource/01-kubernetes/02-dashboard/dashboard-v2.5.1-release.yaml

修改配置

# 默认 dashboard 只能集群内部访问，修改 Service 为 NodePort 类型，暴露到外部
root@kube-master-1:~# vim dashboard-v2.5.1-release.yaml
......
---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  # 设置 type 为 NodePort
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
      # 设置 nodePort 为 30002
      nodePort: 30002
  selector:
    k8s-app: kubernetes-dashboard

---
......

应用部署

root@kube-master-1:~# kubectl apply -f dashboard-v2.5.1-release.yaml

查看 pod

可以看到 kubernetes-dashboard 部署在 kube-worker-2.skynemo.cn 节点上

root@kube-master-1:~# kubectl get pods -n kubernetes-dashboard -o wide
NAME                                         READY   STATUS    RESTARTS   AGE     IP           NODE                       NOMINATED NODE   READINESS GATES
dashboard-metrics-scraper-799d786dbf-bc7g4   1/1     Running   0          4m25s   10.100.3.5   kube-worker-1.skynemo.cn   <none>           <none>
kubernetes-dashboard-fb8648fd9-2rjw5         1/1     Running   0          4m13s   10.100.4.4   kube-worker-2.skynemo.cn   <none>           <none>

查看 service

可以看到端口以及映射到节点的 30002 端口

root@kube-master-1:~# kubectl get service -n kubernetes-dashboard
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.200.193.26    <none>        8000/TCP        49m
kubernetes-dashboard        NodePort    10.200.174.225   <none>        443:30002/TCP   49m

创建账号

参考 dashboard/creating-sample-user.md at master · kubernetes/dashboard (github.com)

编写配置文件

root@kube-master-1:~# vim admin-user.yaml
---
# 在名称空间 kubernetes-dashboard 中创建名为 admin-user 的服务帐户
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard

---
# 为 ServiceAccount 创建 ClusterRoleBinding
# kubeadm 工具配置集群之后，集群中已经存在 ClusterRole cluster admin，直接绑定
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard

应用配置

root@kube-master-1:~# kubectl apply -f admin-user.yaml
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created

获取 token

# 查看创建的用户
root@kube-master-1:~# kubectl -n kubernetes-dashboard get secrets | grep admin-user
admin-user-token-dsr99             kubernetes.io/service-account-token   3      2m58s

# 获取 token
root@kube-master-1:~# kubectl describe secrets admin-user-token-dsr99 -n kubernetes-dashboard
Name:         admin-user-token-dsr99
Namespace:    kubernetes-dashboard
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: admin-user
              kubernetes.io/service-account.uid: a142bea0-1123-49a2-b9e0-6ad863058d56

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1099 bytes
namespace:  20 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IlgwZXVpaWllTE1QUzF2dkE5dldjb1p3WjBTUGtvelRxRmRnaUQ1cXgwREkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLWRzcjk5Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJhMTQyYmVhMC0xMTIzLTQ5YTItYjllMC02YWQ4NjMwNThkNTYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.n5RLlvhckrEB2bEziUdCFiqXjPxv-6BlKdm9g2QwTHAiyqmlnsZuJEauW3CVYoeew8EEgpeLfXrYxJIWBQoVi_2rIcIGBOvA7QIuJnx0Wq_cuiHN9JR7kKhdO1zCrT-mEdHnSC0glx54qxZ1j9z4dgJ14PECM7-riozWZs5jA3FLAWCkEga01yAM-LJg8kXc1ZYoZZXOO0urgTE_XREYhjg_wKOqt8QKSbT5jFFPiQMH9s-pftNNAqXS1bTlWN__ex-iWyNzj1BF6NL41i0-7aIj9ycOr6TIYm5uxSqyu7Hd9oycW62Uuaihx-TfS8Mb-3mqpIclM9Cg2Sc-RWemGA

访问 Dashboard

浏览器输入 https://节点IP:映射端口 访问 Dashboard，本文为 https://kube-worker-2.skynemo.cn 节点 IP:30002

登录

登录成功

查看节点详细状态

删除账号

如果不再使用 Dashboard，可以删除账号

# 需要删除 ServiceAccount 和 ClusterRoleBinding.
kubectl -n kubernetes-dashboard delete serviceaccount admin-user
kubectl -n kubernetes-dashboard delete clusterrolebinding admin-user

Kubeadm 升级 Kubernetes

参考官方文档：

本文示例为小版本升级：kubernetes 1.23.5 —> kubernetes 1.23.6
建议先测试环境升级，没有问题后再升级生产环境

Kubeadm 不仅可以安装 Kubernetes，还可以对其进行升级，当然，升级仅适用于使用 Kubeadm 安装的 Kubernetes，不支持对其他方式安装的 Kubernetes进行升级

命令格式

root@kube-master-1:~# kubeadm upgrade --help
Upgrade your cluster smoothly to a newer version with this command

Usage:
  kubeadm upgrade [flags]
  kubeadm upgrade [command]

Available Commands:
  apply       Upgrade your Kubernetes cluster to the specified version
  diff        Show what differences would be applied to existing static pod manifests. See also: kubeadm upgrade apply --dry-run
  node        Upgrade commands for a node in the cluster
  plan        Check which versions are available to upgrade to and validate whether your current cluster is upgradeable. To skip the internet check, pass in the optional [version] parameter

Flags:
  -h, --help   help for upgrade

Global Flags:
      --add-dir-header           If true, adds the file directory to the header of the log messages
      --log-file string          If non-empty, use this log file
      --log-file-max-size uint   Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --one-output               If true, only write logs to their native severity level (vs also writing to each lower severity level)
      --rootfs string            [EXPERIMENTAL] The path to the 'real' host root filesystem.
      --skip-headers             If true, avoid header prefixes in the log messages
      --skip-log-headers         If true, avoid headers when opening log files
  -v, --v Level                  number for the log level verbosity

Use "kubeadm upgrade [command] --help" for more information about a command.

第一个控制面节点升级

升级 kubeadm

查看当前版本

root@kube-master-1:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:57:37Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}

选择要升级的版本，需要与要升级的 kubernetes 版本相对应

root@kube-master-1:~# apt-cache madison kubeadm | head -n5
   kubeadm |  1.24.0-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubeadm |  1.23.6-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubeadm |  1.23.5-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubeadm |  1.23.4-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubeadm |  1.23.3-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages

升级 kubeadm

# 解锁
root@kube-master-1:~# apt-mark unhold kubeadm
# 更新软件源信息
root@kube-master-1:~# apt-get update
# 安装升级
root@kube-master-1:~# apt-get install kubeadm=1.23.6-00
# 锁定版本
root@kube-master-1:~# apt-mark hold kubeadm

验证升级计划

查看升级计划可以检查是否具备升级条件，并确认升级的版本是否正确

root@kube-master-1:~# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.23.5
[upgrade/versions] kubeadm version: v1.23.6
I0513 01:21:05.764871  203520 version.go:255] remote version is much newer: v1.24.0; falling back to: stable-1.23
[upgrade/versions] Target version: v1.23.6
[upgrade/versions] Latest version in the v1.23 series: v1.23.6

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     6 x v1.23.5   v1.23.6

Upgrade to the latest version in the v1.23 series:

COMPONENT                 CURRENT   TARGET
kube-apiserver            v1.23.5   v1.23.6
kube-controller-manager   v1.23.5   v1.23.6
kube-scheduler            v1.23.5   v1.23.6
kube-proxy                v1.23.5   v1.23.6
CoreDNS                   v1.8.6    v1.8.6
etcd                      3.5.1-0   3.5.1-0

You can now apply the upgrade by executing the following command:

	kubeadm upgrade apply v1.23.6

_____________________________________________________________________

The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

升级部分组件

为防止由于镜像问题导致升级失败，建议先下载镜像

查看需要的镜像

root@kube-master-1:~# kubeadm config images list --kubernetes-version=1.23.6
k8s.gcr.io/kube-apiserver:v1.23.6
k8s.gcr.io/kube-controller-manager:v1.23.6
k8s.gcr.io/kube-scheduler:v1.23.6
k8s.gcr.io/kube-proxy:v1.23.6
k8s.gcr.io/pause:3.6
k8s.gcr.io/etcd:3.5.1-0
k8s.gcr.io/coredns/coredns:v1.8.6

下载镜像

可以连接外网的情况下使用 kubeadm config images pull 直接下载

root@kube-master-1:~# kubeadm config images pull --kubernetes-version=1.23.6
[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.6
[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.6
[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.6
[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.6
[config/images] Pulled k8s.gcr.io/pause:3.6
[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0
[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6

也可以使用作者在阿里云保存的镜像

root@kube-master-1:~# docker pull registry.cn-shanghai.aliyuncs.com/sync-k8s-images/kube-apiserver:v1.23.6
root@kube-master-1:~# docker pull registry.cn-shanghai.aliyuncs.com/sync-k8s-images/kube-controller-manager:v1.23.6
root@kube-master-1:~# docker pull registry.cn-shanghai.aliyuncs.com/sync-k8s-images/kube-scheduler:v1.23.6
root@kube-master-1:~# docker pull registry.cn-shanghai.aliyuncs.com/sync-k8s-images/kube-proxy:v1.23.6
root@kube-master-1:~# docker pull registry.cn-shanghai.aliyuncs.com/sync-k8s-images/pause:3.6
root@kube-master-1:~# docker pull registry.cn-shanghai.aliyuncs.com/sync-k8s-images/etcd:3.5.1-0
root@kube-master-1:~# docker pull registry.cn-shanghai.aliyuncs.com/sync-k8s-images/coredns:v1.8.6

升级

root@kube-master-1:~# kubeadm upgrade apply v1.23.6
# 需要输入一次确定升级，然后就是等待升级完成
.....
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.23.6". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
.....

升级 kubectl、kubelet

腾空节点

通过将节点标记为不可调度并腾空节点为节点作升级准备

# 将 <node-to-drain> 替换为你要腾空的控制面节点名称
root@kube-master-1:~# kubectl drain <node-to-drain> --ignore-daemonsets

查看 kubectl 以及 kubelet 版本

root@kube-master-1:~# apt-cache madison kubelet | head -n5
   kubelet |  1.24.0-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubelet |  1.23.6-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubelet |  1.23.5-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubelet |  1.23.4-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubelet |  1.23.3-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages

升级

# 解除锁定
root@kube-master-1:~# apt-mark unhold kubelet kubectl
# 更新软件源信息
root@kube-master-1:~# apt update
# 安装
root@kube-master-1:~# apt install kubelet=1.23.6-00 kubectl=1.23.6-00
# 锁定版本
root@kube-master-1:~# apt-mark hold kubelet kubectl

重启 kubelet

root@kube-master-1:~# systemctl daemon-reload
root@kube-master-1:~# systemctl restart kubelet

解除节点保护

通过将节点标记为可调度，让其重新上线

# 将 <node-to-drain> 替换为你的节点名称
root@kube-master-1:~# kubectl uncordon <node-to-drain>

CNI 网络插件升级

插件升级与安装操作一致，根据插件的 yaml 配置文件应用即可

验证版本信息

检查 kubelet 版本

root@kube-master-1:~# kubelet --version
Kubernetes v1.23.6

查看 kubernetes 版本

root@kube-master-1:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:49:13Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:43:11Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}

其他控制面节点升级

升级过程与第一个控制面节点相同

升级组件使用

sudo kubeadm upgrade node

而不是

sudo kubeadm upgrade apply

此外，不需要执行 kubeadm upgrade plan 和更新 CNI 插件的操作

工作节点升级

尽量不要全部节点一起升级，保证业务正常

升级 kubeadm

# 解锁
root@kube-worker-1:~# apt-mark unhold kubeadm
# 更新软件源信息
root@kube-worker-1:~# apt-get update
# 安装升级
root@kube-worker-1:~# apt install kubeadm=1.23.6-00
# 锁定版本
root@kube-worker-1:~# apt-mark hold kubeadm

升级本地 kubelet 配置

root@kube-worker-1:~# kubeadm upgrade node

升级 kubelet 和 kubectl

腾空节点

将节点标记为不可调度并驱逐所有负载，准备节点的维护：

# 将 <node-to-drain> 替换为你正在腾空的节点的名称
kubectl drain <node-to-drain> --ignore-daemonsets

升级

# 解锁
root@kube-worker-1:~# apt-mark unhold kubelet kubectl
# 更新软件源信息
root@kube-worker-1:~# apt-get update
# 安装升级
root@kube-worker-1:~# apt install kubelet=1.23.6-00 kubectl=1.23.6-00
# 锁定版本
root@kube-worker-1:~# apt-mark hold kubelet kubectl

重启 kubelet

root@kube-worker-1:~# systemctl daemon-reload
root@kube-worker-1:~# systemctl restart kubelet

取消节点保护

通过将节点标记为可调度，让节点重新上线:

# 将 <node-to-drain> 替换为当前节点的名称
kubectl uncordon <node-to-drain>

验证集群状态和版本

root@kube-master-1:~# kubectl get nodes
NAME                       STATUS   ROLES                  AGE     VERSION
kube-master-1.skynemo.cn   Ready    control-plane,master   6d      v1.23.6
kube-master-2.skynemo.cn   Ready    control-plane,master   6d      v1.23.6
kube-master-3.skynemo.cn   Ready    control-plane,master   6d      v1.23.6
kube-worker-1.skynemo.cn   Ready    <none>                 4d22h   v1.23.6
kube-worker-2.skynemo.cn   Ready    <none>                 4d22h   v1.23.6