在本文中，将探讨使用 k3s 的 kine 项目来替换掉 etcd，并通过实验使用 kubeadm 去 run 一个 k8s 集群，并用 k3s 的 kine 项目来替换掉 etcd。

为什么使用 kine

etcd 在 Kubernetes 之外基本上没有应用的场景，并且 etcd 迭代也比较慢，由于没有人愿意维护因此一直在衰退 [1]，并且，Kubernetes 集群中，etcd 也是一个影响集群规模的重大因素。并且 K3S 存在一个项目 Kine 可以使用关系型数据库运行，这样对集群维护者来说可以不需要维护复杂的 etcd 集群，由于关系型数据库有很多高可用方案，这将使得 k8s 集群规模变成了无限可能。

Kine 介绍

前文提到，kubernetes (kube-apiserver) 与 etcd 是耦合的，如果我们要使用 RDBMS 去替换 etcd 就需要实现 etcd 的接口，那么这个项目就是 Kine [2]。

Kine 是一个 etcdshim，处于 kube-apiserver 和 RDBMS 的中间层，它实现了 etcdAPI的子集（不是etcd的全部功能），Kine 在 RDBMS 数据库之上实现了简单的多版本并发控制；将所有信息存储在一个表中；每行存储此 key 的修订, key, 当前值, 先前值, 先前修订，以及表示该 Key 是已创建还是已删除的标记，通过这种机制可以作为 shim 层来替换 etcd。

简单提一句，shim 是计算机程序设计中的术语，表现为一个小型函数库，服务等，通过截取 API 调用，修改传入参数，来处理自行处理对应操作或者将操作交由其它地方执行。

总的来说 shim 是一种可以在新环境中支持老 API，也可以在老环境里支持新 API 辅助运行库或服务，在云原生场景中，我们经常看到 docker-shim，cri-shim 等。

前提条件

本文实验环境使用的软件版本如下

软件/硬件版本
操作系统Debian 11(bullseye) 2C/4G
Kubernetes版本v1.28.11(截至文章编写时间的最新版)
Kubernetes集群部署工具kubeadm
Kinev0.11.10 (截至文章编写时间的最新版)
MySQLDocker运行，镜像 mysql:5.7

使用 kubeadm 构建控制平面

为了展现 kine 的作用，首先我们需要准备一个 k8s 集群，这里简单使用 kubeadm + containerd 来构建一个 kuebrnetes 集群。

安装 containerd

载入内核依赖项

containerd 或 docker 的安装都需要内核支持 overlay 和 br_netfilter 模块，overlay 为 containerd 运行的文件系统，netfiler 用于维护容器内 (inter-container) 的网络。所以我们需要加载对应的内核模块。

bash
1
2
3
4

cat <<EOF | tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

手动执行下面命令

bash
1
2

modprobe overlay && \
modprobe br_netfilter

通过仓库 containerd

contanerd 是作为 docker-ce 的下层，所以很多 Linux 发行版都有对应的包管理工具的仓库，这里面维护了基本上比较新的版本，可以直接在对应操作系统下载

CentOS

bash
1
2
3

yum install yum-utils -y && \
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo && \
yum install containerd.io -y

Debian

Debian仓库中通常都有比较新版本的 containerd，可以直接安装

bash
1

apt list|grep containerd

安装

bash
1

apt -y install containerd

离线安装

如果需要离线环境安装的话，可以在手动下载 containerd 和 runc 后传入内网

下载 Containerd 的二进制包，这里下载containerd-<VERSION>-<OS>-<ARCH>.tar.gz 格式名称的发行版，后边在单独下载安装 runc

bash
1

wget https://github.com/containerd/containerd/releases/download/v1.7.3/containerd-1.7.3-linux-amd64.tar.gz

将其解压缩到 /usr/local 下:

bash
1

tar Cxzvf /usr/local containerd-1.7.3-linux-amd64.tar.gz

接下来从 runc 的 github 上下载安装 runc，该二进制文件是静态构建的，并且应该适用于任何Linux发行版。

bash
1
2

wget https://github.com/opencontainers/runc/releases/download/v1.1.9/runc.amd64
install -m 755 runc.amd64 /usr/local/sbin/runc

为了通过 systemd 管理 containerd，请还需要从仓库中下载 containerd.service 单元文件

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

cat << EOF > /usr/lib/systemd/system/containerd.service
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
#uncomment to enable the experimental sbservice (sandboxed) version of containerd/cri integration
#Environment="ENABLE_CRI_SANDBOXES=sandboxed"
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF

配置配置文件

bash
1
2

mkdir -p /etc/containerd && \
containerd config default | sudo tee /etc/containerd/config.toml

配置驱动为 systemd

将配置文件修改为实例所述

yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"
          runtime_engine = ""
          runtime_root = ""
          privileged_without_host_devices = false
          base_runtime_spec = ""

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
          SystemdCgroup = true

一键修改命令

bash
1

sed -i "s/SystemdCgroup = false/SystemdCgroup = true/g"  "${CONTAINDERD_CONFIG_PATH}"

启动服务

bash
1
2

systemctl enable --now containerd && \
systemctl restart containerd

使用kubeadm构建集群

加载内核依赖项

bash
1
2
3
4
5
6

cat > /etc/modules-load.d/kubernetes.conf <<EOF
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
EOF

执行以下命令使配置立即生效:

bash
1
2
3
4

modprobe ip_vs && \
modprobe ip_vs_rr && \
modprobe ip_vs_wrr && \
modprobe ip_vs_sh

安装kubeadm kubelet kubectl

安装 kubeadm 可以参考官网的步骤来 [3]

使用基于debian 包管理仓库

使用 Kubernetes apt 仓库

bash
1

apt-get install -y apt-transport-https ca-certificates curl gpg

下载公共签名key

bash
1

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

添加适合的 k8s 版本仓库，这里是 1.28

bash
1
2

# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

更新包索引

bash
1
2

apt update && \
apt install -y kubelet=1.28.11-1.1 kubeadm=1.28.11-1.1 kubectl=1.28.11-1.1

不使用包管理工具

下载 kubeadm, kubelet, kubectl 二进制文件

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# 这个文件内包含的是 kubernetes 最新稳定版的版本号，如果要安装最新版可以取消掉这行注释
# RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
RELEASE="v1.28.11"
ARCH="amd64"
DOWNLOAD_DIR="/usr/local/bin"
mkdir -p "$DOWNLOAD_DIR"

cd $DOWNLOAD_DIR
sudo curl -L --remote-name-all https://dl.k8s.io/release/${RELEASE}/bin/linux/${ARCH}/{kubeadm,kubelet}
sudo chmod +x {kubeadm,kubelet}

下载 kubelet 的 system单元文件 或手动添加所需的 systemd 单元文件

bash
1
2
3
4
5
6
7
8

# v0.16.2 是一个固定的版本号，不是 kubernetes 版本
RELEASE_VERSION="v0.16.2"
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/krel/templates/latest/kubelet/kubelet.service" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service

# kubelet.service 是一个单元文件
# systemd 的 service.d 目录是一个固定写法，这里表示可以使用 .conf 结尾的文件来覆盖这个服务的单元文件
mkdir -p /etc/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/krel/templates/latest/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

或者手动创建 kubelet.serivce 的 systemd 的单元文件

这个文件是将 rpm 或 dpkg 包的 kubelet.service 和上述 10-kubeadm.conf 融合为一起的，效果是相同的

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

cat << EOF > /usr/lib/systemd/system/kubelet.serivce
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
Wants=network-online.target
After=network-online.target

# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

离线环境镜像下载

列出所使用的镜像

bash
1
2
3
4
5
6
7
8

$ kubeadm config images list --kubernetes-version=1.28.8
registry.k8s.io/kube-apiserver:v1.28.8
registry.k8s.io/kube-controller-manager:v1.28.8
registry.k8s.io/kube-scheduler:v1.28.8
registry.k8s.io/kube-proxy:v1.28.8
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.12-0
registry.k8s.io/coredns/coredns:v1.10.1

下载对应镜像，并上传到私有仓库

bash
1
2
3
4
5

for n in `./kubeadm config images list --kubernetes-version=1.28.11`;
do
    docker pull $n; docker tag $n `echo $n | sed 's|registry.k8s.io|img.xxxx.com/system|'`
    docker push `echo $n | sed 's|registry.k8s.io|img.xxx.com/system|'`
done

生成配置文件

bash
1
2
3
4
5
6
7

./kubeadm config images list --image-repository img.xxx.com/system --kubernetes-version=v1.28.11

# 生成对应组件的的 kubeconfig
# kubelet
kubeadm config print init-defaults --component-configs KubeletConfiguration|grep -A 1000 'apiVersion: kubelet.config.k8s.io'|sed 's|0s|30s|g'
# kube-proxy
kubeadm config print init-defaults --component-configs KubeProxyConfiguration|grep -A 1000 'kubeproxy.config.k8s.io/'|sed 's|0s|30s|g'

使用配置文件安装

bash
1

kubeadm init --config kube.yaml  -v 10

使用命令初始化

bash
1
2
3
4
5
6
7
8
9

kubeadm init \
    --image-repository=img.xxx.com/system \
    --pod-network-cidr=10.10.0.0/16 \
    --service-cidr=10.11.0.0/24 \
    --kubernetes-version=v1.28.11 \
    --control-plane-endpoint=`hostname -I` \
    --apiserver-advertise-address=`hostname -I` \
    --apiserver-cert-extra-sans=`hostname -I` \
    --v=10

这个时候控制平面已经可以正常工作了

bash
1
2
3
4
5
6
7
8
9

$ kubectl --kubeconfig /etc/kubernetes/admin.conf  get pods -n kube-system
NAME                           READY   STATUS    RESTARTS   AGE
coredns-5dd5756b68-nvqwf       0/1     Pending   0          16h
coredns-5dd5756b68-t2tj5       0/1     Pending   0          16h
etcd-node                      1/1     Running   0          16h
kube-apiserver-node            1/1     Running   0          16h
kube-controller-manager-node   1/1     Running   0          16h
kube-proxy-g6fpc               1/1     Running   0          16h
kube-scheduler-node            1/1     Running   0          16h

使用 kine 来替换 etcd

查看官方示例

首先根据 kine 官方 example 来查看最小示例的来学习如何使用 kine [4]，通过文章得知，kine 运行有两种方式，kine 与数据库之间的使用 ssl 链接。

mysql

bash
1
2

kine --endpoint "mysql://root:$PASSWORD@tcp(localhost:3306)/kine"
--ca-file ca.crt --cert-file server.crt --key-file server.key

postgres

bash
1
2
3
4

kine --endpoint="postgres://$(POSTGRES_USERNAME):$(POSTGRES_PASSWORD)@localhost:5432/postgres"
      --ca-file=/var/lib/postgresql/ca.crt
      --cert-file=/var/lib/postgresql/server.crt
      --key-file=/var/lib/postgresql/server.key

这时我们需要查看一下 kine 的参数

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

GLOBAL OPTIONS:
   --listen-address value                     (default: "0.0.0.0:2379")
   --endpoint value                           Storage endpoint (default is sqlite)
   --ca-file value                            CA cert for DB connection
   --cert-file value                          Certificate for DB connection
   --server-cert-file value                   Certificate for etcd connection
   --server-key-file value                    Key file for etcd connection
   --datastore-max-idle-connections value     Maximum number of idle connections retained by datastore. If value = 0, the system default will be used. If value < 0, idle connections will not be reused. (default: 0)
   --datastore-max-open-connections value     Maximum number of open connections used by datastore. If value <= 0, then there is no limit (default: 0)
   --datastore-connection-max-lifetime value  Maximum amount of time a connection may be reused. If value <= 0, then there is no limit. (default: 0s)
   --key-file value                           Key file for DB connection
   --metrics-bind-address value               The address the metric endpoint binds to. Default :8080, set 0 to disable metrics serving. (default: ":8080")
   --slow-sql-threshold value                 The duration which SQL executed longer than will be logged. Default 1s, set <= 0 to disable slow SQL log. (default: 1s)
   --metrics-enable-profiling                 Enable net/http/pprof handlers on the metrics bind address. Default is false. (default: false)
   --watch-progress-notify-interval value     Interval between periodic watch progress notifications. Default is 10m. (default: 10m0s)
   --debug                                    (default: false)
   --help, -h                                 show help
   --version, -v                              print the version

通过参数得知，上面的除了官方给出的，kine 与数据库之间的连接也可以不使用 ssl，并通过 --server-cert-file 与 --server-key-file 来作为 kube-apiserver 连接 etcd 所使用的证书指定给 kine 就可以启动了。

编写静态文件

这里我们只需要删除 /etc/kubernetes/manifests/etcd.yaml 并将 etcd 使用的证书挂载到 kine pod 中，那么我们编写 /etc/kubernetes/manifests/kine.yaml 文件。

yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

cat /etc/kubernetes/manifests/kine.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kine
    tier: control-plane
  name: kine
  namespace: kube-system
spec:
  containers:
  - name: kine
    command: [ "/bin/sh", "-c", "--" ]
    args: [ 'kine --endpoint="mysql://root:111@tcp(10.0.0.1:3306)/kine"
 --server-cert-file=/etc/kubernetes/pki/etcd/server.crt
 --server-key-file=/etc/kubernetes/pki/etcd/server.key' ]
    image: docker.io/rancher/kine:v0.11.10-amd64
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    - mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
    name: etcd-certs
status: {}

kubuadm 生成的 kubelet 的 KubeletConfiguration 文件，中静态文件得路径参数 “staticPodPath”

yaml
1
2
3
4
5

$ cat /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
...
staticPodPath: /etc/kubernetes/manifests

这个时候可以启动 kubelet 服务，然后查看静态 Pod，此时可以看到， kube-system 名称空间 已经没有 etcd pod了

bash
1
2
3
4
5
6

$ kubectl --kubeconfig /etc/kubernetes/admin.conf get pod -n kube-system
NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
kube-system   kine-node                      1/1     Running   0          17s
kube-system   kube-apiserver-node            1/1     Running   19         7m15s
kube-system   kube-controller-manager-node   1/1     Running   6          7m5s
kube-system   kube-scheduler-node            1/1     Running   6          7m2s

此时就可以继续部署 k8s 的 worker 节点和 CNI 了

探索 kine

我们可以查看数据库表结构，来探索 kine 是如何实现的 etcdAPI 转换的，我们可以看到，kine 会在启动参数中配置的库名 创建对应的数据库，并且仅有一个表 kine

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| kine               |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.01 sec)

mysql> show tables;
+----------------+
| Tables_in_kine |
+----------------+
| kine           |
+----------------+

观察表结构

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

mysql> desc kine;
+-----------------+---------------------+------+-----+---------+----------------+
| Field           | Type                | Null | Key | Default | Extra          |
+-----------------+---------------------+------+-----+---------+----------------+
| id              | bigint(20) unsigned | NO   | PRI | NULL    | auto_increment |
| name            | varchar(630)        | YES  | MUL | NULL    |                |
| created         | int(11)             | YES  |     | NULL    |                |
| deleted         | int(11)             | YES  |     | NULL    |                |
| create_revision | bigint(20) unsigned | YES  |     | NULL    |                |
| prev_revision   | bigint(20) unsigned | YES  | MUL | NULL    |                |
| lease           | int(11)             | YES  |     | NULL    |                |
| value           | mediumblob          | YES  |     | NULL    |                |
| old_value       | mediumblob          | YES  |     | NULL    |                |
+-----------------+---------------------+------+-----+---------+----------------+
9 rows in set (0.00 sec)

查看数据是如何存储的

bash
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277

mysql> select count(id),name from kine group by name;
+-----------+---------------------------------------------------------------------------------------------+
| count(id) | name                                                                                        |
+-----------+---------------------------------------------------------------------------------------------+
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.                                            |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io                |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io                        |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.apps                                        |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io                       |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io                        |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.autoscaling                                 |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.batch                                       |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io                         |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io                         |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io                            |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.events.k8s.io                               |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.networking.k8s.io                           |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.node.k8s.io                                 |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.policy                                      |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io                   |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io                           |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1.storage.k8s.io                              |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1beta2.flowcontrol.apiserver.k8s.io           |
|         1 | /registry/apiregistration.k8s.io/apiservices/v1beta3.flowcontrol.apiserver.k8s.io           |
|         1 | /registry/apiregistration.k8s.io/apiservices/v2.autoscaling                                 |
|         1 | /registry/clusterrolebindings/cluster-admin                                                 |
|         1 | /registry/clusterrolebindings/system:basic-user                                             |
|         1 | /registry/clusterrolebindings/system:controller:attachdetach-controller                     |
|         1 | /registry/clusterrolebindings/system:controller:certificate-controller                      |
|         1 | /registry/clusterrolebindings/system:controller:clusterrole-aggregation-controller          |
|         1 | /registry/clusterrolebindings/system:controller:cronjob-controller                          |
|         1 | /registry/clusterrolebindings/system:controller:daemon-set-controller                       |
|         1 | /registry/clusterrolebindings/system:controller:deployment-controller                       |
|         1 | /registry/clusterrolebindings/system:controller:disruption-controller                       |
|         1 | /registry/clusterrolebindings/system:controller:endpoint-controller                         |
|         1 | /registry/clusterrolebindings/system:controller:endpointslice-controller                    |
|         1 | /registry/clusterrolebindings/system:controller:endpointslicemirroring-controller           |
|         1 | /registry/clusterrolebindings/system:controller:ephemeral-volume-controller                 |
|         1 | /registry/clusterrolebindings/system:controller:expand-controller                           |
|         1 | /registry/clusterrolebindings/system:controller:generic-garbage-collector                   |
|         1 | /registry/clusterrolebindings/system:controller:horizontal-pod-autoscaler                   |
|         1 | /registry/clusterrolebindings/system:controller:job-controller                              |
|         1 | /registry/clusterrolebindings/system:controller:namespace-controller                        |
|         1 | /registry/clusterrolebindings/system:controller:node-controller                             |
|         1 | /registry/clusterrolebindings/system:controller:persistent-volume-binder                    |
|         1 | /registry/clusterrolebindings/system:controller:pod-garbage-collector                       |
|         1 | /registry/clusterrolebindings/system:controller:pv-protection-controller                    |
|         1 | /registry/clusterrolebindings/system:controller:pvc-protection-controller                   |
|         1 | /registry/clusterrolebindings/system:controller:replicaset-controller                       |
|         1 | /registry/clusterrolebindings/system:controller:replication-controller                      |
|         1 | /registry/clusterrolebindings/system:controller:resourcequota-controller                    |
|         1 | /registry/clusterrolebindings/system:controller:root-ca-cert-publisher                      |
|         1 | /registry/clusterrolebindings/system:controller:route-controller                            |
|         1 | /registry/clusterrolebindings/system:controller:service-account-controller                  |
|         1 | /registry/clusterrolebindings/system:controller:service-controller                          |
|         1 | /registry/clusterrolebindings/system:controller:statefulset-controller                      |
|         1 | /registry/clusterrolebindings/system:controller:ttl-after-finished-controller               |
|         1 | /registry/clusterrolebindings/system:controller:ttl-controller                              |
|         1 | /registry/clusterrolebindings/system:discovery                                              |
|         1 | /registry/clusterrolebindings/system:kube-controller-manager                                |
|         1 | /registry/clusterrolebindings/system:kube-dns                                               |
|         1 | /registry/clusterrolebindings/system:kube-scheduler                                         |
|         1 | /registry/clusterrolebindings/system:monitoring                                             |
|         1 | /registry/clusterrolebindings/system:node                                                   |
|         1 | /registry/clusterrolebindings/system:node-proxier                                           |
|         1 | /registry/clusterrolebindings/system:public-info-viewer                                     |
|         1 | /registry/clusterrolebindings/system:service-account-issuer-discovery                       |
|         1 | /registry/clusterrolebindings/system:volume-scheduler                                       |
|         1 | /registry/clusterroles/admin                                                                |
|         1 | /registry/clusterroles/cluster-admin                                                        |
|         1 | /registry/clusterroles/edit                                                                 |
|         1 | /registry/clusterroles/system:aggregate-to-admin                                            |
|         1 | /registry/clusterroles/system:aggregate-to-edit                                             |
|         1 | /registry/clusterroles/system:aggregate-to-view                                             |
|         1 | /registry/clusterroles/system:auth-delegator                                                |
|         1 | /registry/clusterroles/system:basic-user                                                    |
|         1 | /registry/clusterroles/system:certificates.k8s.io:certificatesigningrequests:nodeclient     |
|         1 | /registry/clusterroles/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient |
|         1 | /registry/clusterroles/system:certificates.k8s.io:kube-apiserver-client-approver            |
|         1 | /registry/clusterroles/system:certificates.k8s.io:kube-apiserver-client-kubelet-approver    |
|         1 | /registry/clusterroles/system:certificates.k8s.io:kubelet-serving-approver                  |
|         1 | /registry/clusterroles/system:certificates.k8s.io:legacy-unknown-approver                   |
|         1 | /registry/clusterroles/system:controller:attachdetach-controller                            |
|         1 | /registry/clusterroles/system:controller:certificate-controller                             |
|         1 | /registry/clusterroles/system:controller:clusterrole-aggregation-controller                 |
|         1 | /registry/clusterroles/system:controller:cronjob-controller                                 |
|         1 | /registry/clusterroles/system:controller:daemon-set-controller                              |
|         1 | /registry/clusterroles/system:controller:deployment-controller                              |
|         1 | /registry/clusterroles/system:controller:disruption-controller                              |
|         1 | /registry/clusterroles/system:controller:endpoint-controller                                |
|         1 | /registry/clusterroles/system:controller:endpointslice-controller                           |
|         1 | /registry/clusterroles/system:controller:endpointslicemirroring-controller                  |
|         1 | /registry/clusterroles/system:controller:ephemeral-volume-controller                        |
|         1 | /registry/clusterroles/system:controller:expand-controller                                  |
|         1 | /registry/clusterroles/system:controller:generic-garbage-collector                          |
|         1 | /registry/clusterroles/system:controller:horizontal-pod-autoscaler                          |
|         1 | /registry/clusterroles/system:controller:job-controller                                     |
|         1 | /registry/clusterroles/system:controller:namespace-controller                               |
|         1 | /registry/clusterroles/system:controller:node-controller                                    |
|         1 | /registry/clusterroles/system:controller:persistent-volume-binder                           |
|         1 | /registry/clusterroles/system:controller:pod-garbage-collector                              |
|         1 | /registry/clusterroles/system:controller:pv-protection-controller                           |
|         1 | /registry/clusterroles/system:controller:pvc-protection-controller                          |
|         1 | /registry/clusterroles/system:controller:replicaset-controller                              |
|         1 | /registry/clusterroles/system:controller:replication-controller                             |
|         1 | /registry/clusterroles/system:controller:resourcequota-controller                           |
|         1 | /registry/clusterroles/system:controller:root-ca-cert-publisher                             |
|         1 | /registry/clusterroles/system:controller:route-controller                                   |
|         1 | /registry/clusterroles/system:controller:service-account-controller                         |
|         1 | /registry/clusterroles/system:controller:service-controller                                 |
|         1 | /registry/clusterroles/system:controller:statefulset-controller                             |
|         1 | /registry/clusterroles/system:controller:ttl-after-finished-controller                      |
|         1 | /registry/clusterroles/system:controller:ttl-controller                                     |
|         1 | /registry/clusterroles/system:discovery                                                     |
|         1 | /registry/clusterroles/system:heapster                                                      |
|         1 | /registry/clusterroles/system:kube-aggregator                                               |
|         1 | /registry/clusterroles/system:kube-controller-manager                                       |
|         1 | /registry/clusterroles/system:kube-dns                                                      |
|         1 | /registry/clusterroles/system:kube-scheduler                                                |
|         1 | /registry/clusterroles/system:kubelet-api-admin                                             |
|         1 | /registry/clusterroles/system:monitoring                                                    |
|         1 | /registry/clusterroles/system:node                                                          |
|         1 | /registry/clusterroles/system:node-bootstrapper                                             |
|         1 | /registry/clusterroles/system:node-problem-detector                                         |
|         1 | /registry/clusterroles/system:node-proxier                                                  |
|         1 | /registry/clusterroles/system:persistent-volume-provisioner                                 |
|         1 | /registry/clusterroles/system:public-info-viewer                                            |
|         1 | /registry/clusterroles/system:service-account-issuer-discovery                              |
|         1 | /registry/clusterroles/system:volume-scheduler                                              |
|         1 | /registry/clusterroles/view                                                                 |
|         1 | /registry/configmaps/default/kube-root-ca.crt                                               |
|         1 | /registry/configmaps/kube-node-lease/kube-root-ca.crt                                       |
|         1 | /registry/configmaps/kube-public/kube-root-ca.crt                                           |
|         1 | /registry/configmaps/kube-system/extension-apiserver-authentication                         |
|         1 | /registry/configmaps/kube-system/kube-apiserver-legacy-service-account-token-tracking       |
|         1 | /registry/configmaps/kube-system/kube-root-ca.crt                                           |
|         1 | /registry/csinodes/node                                                                     |
|         1 | /registry/endpointslices/default/kubernetes                                                 |
|         1 | /registry/events/default/node.17de1624f3c1624f                                              |
|         1 | /registry/events/default/node.17de1624f3c1e6bb                                              |
|         1 | /registry/events/default/node.17de1624f3c25c4f                                              |
|         1 | /registry/events/default/node.17de1624f5b37dfb                                              |
|         1 | /registry/events/default/node.17de1639e7890c71                                              |
|         1 | /registry/events/default/node.17de168dce4cdb68                                              |
|         1 | /registry/events/default/node.17de16a194521b80                                              |
|         1 | /registry/events/kube-system/kine-node.17de162525650d9b                                     |
|         1 | /registry/events/kube-system/kine-node.17de1625275ca2d7                                     |
|         1 | /registry/events/kube-system/kine-node.17de16252f773864                                     |
|         1 | /registry/events/kube-system/kine-node.17de1625a5af90c0                                     |
|         1 | /registry/events/kube-system/kine-node.17de169d120062cc                                     |
|         1 | /registry/events/kube-system/kine-node.17de169d1361dab8                                     |
|         1 | /registry/events/kube-system/kine-node.17de169d1855aee6                                     |
|         1 | /registry/events/kube-system/kine-node.17de16a1969b1ed6                                     |
|         1 | /registry/events/kube-system/kube-apiserver-node.17de162513417e64                           |
|         1 | /registry/events/kube-system/kube-apiserver-node.17de1625158e863e                           |
|         1 | /registry/events/kube-system/kube-apiserver-node.17de162525e8ebd2                           |
|         1 | /registry/events/kube-system/kube-apiserver-node.17de1629c37f0b35                           |
|         1 | /registry/events/kube-system/kube-apiserver-node.17de1629f6bc718f                           |
|         1 | /registry/events/kube-system/kube-apiserver-node.17de162ecf004a1d                           |
|         1 | /registry/events/kube-system/kube-apiserver-node.17de162eff4060dd                           |
|         1 | /registry/events/kube-system/kube-apiserver-node.17de1637f005507c                           |
|         1 | /registry/events/kube-system/kube-apiserver-node.17de1661f1bd6879                           |
|         1 | /registry/events/kube-system/kube-apiserver-node.17de16620f441326                           |
|         1 | /registry/events/kube-system/kube-apiserver-node.17de16a1985f63bd                           |
|         1 | /registry/events/kube-system/kube-controller-manager-node.17de162511a4b3be                  |
|         1 | /registry/events/kube-system/kube-controller-manager-node.17de162512f837ca                  |
|         1 | /registry/events/kube-system/kube-controller-manager-node.17de16251d8b658b                  |
|         1 | /registry/events/kube-system/kube-controller-manager-node.17de169b0537b6d0                  |
|         1 | /registry/events/kube-system/kube-controller-manager-node.17de16a1971f3999                  |
|         1 | /registry/events/kube-system/kube-controller-manager.17de1638e6568ffc                       |
|         1 | /registry/events/kube-system/kube-controller-manager.17de168d8ed8b9cc                       |
|         1 | /registry/events/kube-system/kube-controller-manager.17de16a150704739                       |
|         1 | /registry/events/kube-system/kube-scheduler-node.17de162512917b00                           |
|         1 | /registry/events/kube-system/kube-scheduler-node.17de16251515909b                           |
|         1 | /registry/events/kube-system/kube-scheduler-node.17de16252295ae29                           |
|         1 | /registry/events/kube-system/kube-scheduler-node.17de162a7ee366d4                           |
|         1 | /registry/events/kube-system/kube-scheduler-node.17de169c038ba9cc                           |
|         1 | /registry/events/kube-system/kube-scheduler-node.17de169cf8755bf3                           |
|         1 | /registry/events/kube-system/kube-scheduler-node.17de16a19797a620                           |
|         1 | /registry/events/kube-system/kube-scheduler.17de1643dd024555                                |
|         1 | /registry/events/kube-system/kube-scheduler.17de168dbd6f19b1                                |
|         1 | /registry/events/kube-system/kube-scheduler.17de16a24a03d6c8                                |
|         1 | /registry/flowschemas/catch-all                                                             |
|         1 | /registry/flowschemas/endpoint-controller                                                   |
|         1 | /registry/flowschemas/exempt                                                                |
|         1 | /registry/flowschemas/global-default                                                        |
|         1 | /registry/flowschemas/kube-controller-manager                                               |
|         1 | /registry/flowschemas/kube-scheduler                                                        |
|         1 | /registry/flowschemas/kube-system-service-accounts                                          |
|         1 | /registry/flowschemas/probes                                                                |
|         1 | /registry/flowschemas/service-accounts                                                      |
|         1 | /registry/flowschemas/system-leader-election                                                |
|         1 | /registry/flowschemas/system-node-high                                                      |
|         1 | /registry/flowschemas/system-nodes                                                          |
|         1 | /registry/flowschemas/workload-leader-election                                              |
|         1 | /registry/health                                                                            |
|        97 | /registry/leases/kube-node-lease/node                                                       |
|        97 | /registry/leases/kube-system/apiserver-6cazmjvz5glfjbabvahmi5cwfy                           |
|       484 | /registry/leases/kube-system/kube-controller-manager                                        |
|       485 | /registry/leases/kube-system/kube-scheduler                                                 |
|        99 | /registry/masterleases/10.0.0.14                                                            |
|        33 | /registry/minions/node                                                                      |
|         1 | /registry/namespaces/default                                                                |
|         1 | /registry/namespaces/kube-node-lease                                                        |
|         1 | /registry/namespaces/kube-public                                                            |
|         1 | /registry/namespaces/kube-system                                                            |
|         1 | /registry/pods/kube-system/kine-node                                                        |
|         1 | /registry/pods/kube-system/kube-apiserver-node                                              |
|         1 | /registry/pods/kube-system/kube-controller-manager-node                                     |
|         1 | /registry/pods/kube-system/kube-scheduler-node                                              |
|         1 | /registry/priorityclasses/system-cluster-critical                                           |
|         1 | /registry/priorityclasses/system-node-critical                                              |
|         1 | /registry/prioritylevelconfigurations/catch-all                                             |
|         1 | /registry/prioritylevelconfigurations/exempt                                                |
|         1 | /registry/prioritylevelconfigurations/global-default                                        |
|         1 | /registry/prioritylevelconfigurations/leader-election                                       |
|         1 | /registry/prioritylevelconfigurations/node-high                                             |
|         1 | /registry/prioritylevelconfigurations/system                                                |
|         1 | /registry/prioritylevelconfigurations/workload-high                                         |
|         1 | /registry/prioritylevelconfigurations/workload-low                                          |
|         1 | /registry/ranges/serviceips                                                                 |
|         1 | /registry/ranges/servicenodeports                                                           |
|         1 | /registry/rolebindings/kube-public/system:controller:bootstrap-signer                       |
|         1 | /registry/rolebindings/kube-system/system::extension-apiserver-authentication-reader        |
|         1 | /registry/rolebindings/kube-system/system::leader-locking-kube-controller-manager           |
|         1 | /registry/rolebindings/kube-system/system::leader-locking-kube-scheduler                    |
|         1 | /registry/rolebindings/kube-system/system:controller:bootstrap-signer                       |
|         1 | /registry/rolebindings/kube-system/system:controller:cloud-provider                         |
|         1 | /registry/rolebindings/kube-system/system:controller:token-cleaner                          |
|         1 | /registry/roles/kube-public/system:controller:bootstrap-signer                              |
|         1 | /registry/roles/kube-system/extension-apiserver-authentication-reader                       |
|         1 | /registry/roles/kube-system/system::leader-locking-kube-controller-manager                  |
|         1 | /registry/roles/kube-system/system::leader-locking-kube-scheduler                           |
|         1 | /registry/roles/kube-system/system:controller:bootstrap-signer                              |
|         1 | /registry/roles/kube-system/system:controller:cloud-provider                                |
|         1 | /registry/roles/kube-system/system:controller:token-cleaner                                 |
|         1 | /registry/serviceaccounts/default/default                                                   |
|         1 | /registry/serviceaccounts/kube-node-lease/default                                           |
|         1 | /registry/serviceaccounts/kube-public/default                                               |
|         1 | /registry/serviceaccounts/kube-system/attachdetach-controller                               |
|         1 | /registry/serviceaccounts/kube-system/bootstrap-signer                                      |
|         1 | /registry/serviceaccounts/kube-system/certificate-controller                                |
|         1 | /registry/serviceaccounts/kube-system/clusterrole-aggregation-controller                    |
|         1 | /registry/serviceaccounts/kube-system/cronjob-controller                                    |
|         1 | /registry/serviceaccounts/kube-system/daemon-set-controller                                 |
|         1 | /registry/serviceaccounts/kube-system/default                                               |
|         1 | /registry/serviceaccounts/kube-system/deployment-controller                                 |
|         1 | /registry/serviceaccounts/kube-system/disruption-controller                                 |
|         1 | /registry/serviceaccounts/kube-system/endpoint-controller                                   |
|         1 | /registry/serviceaccounts/kube-system/endpointslice-controller                              |
|         1 | /registry/serviceaccounts/kube-system/endpointslicemirroring-controller                     |
|         1 | /registry/serviceaccounts/kube-system/ephemeral-volume-controller                           |
|         1 | /registry/serviceaccounts/kube-system/expand-controller                                     |
|         1 | /registry/serviceaccounts/kube-system/generic-garbage-collector                             |
|         1 | /registry/serviceaccounts/kube-system/horizontal-pod-autoscaler                             |
|         1 | /registry/serviceaccounts/kube-system/job-controller                                        |
|         1 | /registry/serviceaccounts/kube-system/namespace-controller                                  |
|         1 | /registry/serviceaccounts/kube-system/node-controller                                       |
|         1 | /registry/serviceaccounts/kube-system/persistent-volume-binder                              |
|         1 | /registry/serviceaccounts/kube-system/pod-garbage-collector                                 |
|         1 | /registry/serviceaccounts/kube-system/pv-protection-controller                              |
|         1 | /registry/serviceaccounts/kube-system/pvc-protection-controller                             |
|         1 | /registry/serviceaccounts/kube-system/replicaset-controller                                 |
|         1 | /registry/serviceaccounts/kube-system/replication-controller                                |
|         1 | /registry/serviceaccounts/kube-system/resourcequota-controller                              |
|         1 | /registry/serviceaccounts/kube-system/root-ca-cert-publisher                                |
|         1 | /registry/serviceaccounts/kube-system/service-account-controller                            |
|         1 | /registry/serviceaccounts/kube-system/service-controller                                    |
|         1 | /registry/serviceaccounts/kube-system/statefulset-controller                                |
|         1 | /registry/serviceaccounts/kube-system/token-cleaner                                         |
|         1 | /registry/serviceaccounts/kube-system/ttl-after-finished-controller                         |
|         1 | /registry/serviceaccounts/kube-system/ttl-controller                                        |
|         1 | /registry/services/endpoints/default/kubernetes                                             |
|         1 | /registry/services/specs/default/kubernetes                                                 |
|         1 | compact_rev_key                                                                             |
+-----------+---------------------------------------------------------------------------------------------+
271 rows in set (0.00 sec)

如上所示，有一个名为的表 “kine”包含所有数据。Kine 使用数据库作为日志结构存储，因此来自 API 服务器的每次写入都会创建一个新行来存储已创建或更新的 Kubernetes 对象，“name” 列使用与 etcd 相同的存储结构 “/registry/RESOURCE_TYPE/NAMESPACE/NAME” 表示集群中对象。

k3s 资源分析

k3s 官方提供了 Resource Profiling [5] 来对比了 RDBMS 与 etcd 的性能对比。

总结

因为 RDBMS 大家都很熟悉，并且更高性能的分布式解决方案也有很多，例如 YugabyteDB (PostgreSQL兼容的分布式数据库)，也可以预创建 kine 表，通过分区形式将不同数据存储到不同的分区内。而且 k8s 对象的历史数据也是可以根据一定的规则进行删除，因为 kubernetes 中的对象都是实时协调的，所以也不怕误删除，这样就会使得 kubernetes 规模有更大扩展的可能。

Reference