本文发布于Cylon的收藏册，转载请著名原文链接~
在本文中，将探讨使用 k3s 的 kine 项目来替换掉 etcd，并通过实验使用 kubeadm 去 run 一个 k8s 集群，并用 k3s 的 kine 项目来替换掉 etcd。
为什么使用 kine
etcd 在 Kubernetes 之外基本上没有应用的场景，并且 etcd 迭代也比较慢，由于没有人愿意维护因此一直在衰退 [1]，并且，Kubernetes 集群中，etcd 也是一个影响集群规模的重大因素。并且 K3S 存在一个项目 Kine 可以使用关系型数据库运行，这样对集群维护者来说可以不需要维护复杂的 etcd 集群，由于关系型数据库有很多高可用方案，这将使得 k8s 集群规模变成了无限可能。
Kine 介绍
前文提到，kubernetes (kube-apiserver) 与 etcd 是耦合的，如果我们要使用 RDBMS 去替换 etcd 就需要实现 etcd 的接口，那么这个项目就是 Kine [2]。
Kine 是一个 etcdshim，处于 kube-apiserver 和 RDBMS 的中间层，它实现了 etcdAPI的子集（不是etcd的全部功能），Kine 在 RDBMS 数据库之上实现了简单的多版本并发控制；将所有信息存储在一个表中；每行存储此 key 的修订, key, 当前值, 先前值, 先前修订，以及表示该 Key 是已创建还是已删除的标记，通过这种机制可以作为 shim 层来替换 etcd。
简单提一句，shim 是计算机程序设计中的术语，表现为一个小型函数库，服务等，通过截取 API 调用，修改传入参数，来处理自行处理对应操作或者将操作交由其它地方执行。
总的来说 shim 是一种可以在新环境中支持老 API，也可以在老环境里支持新 API 辅助运行库或服务，在云原生场景中，我们经常看到 docker-shim，cri-shim 等。
前提条件
本文实验环境使用的软件版本如下
| 软件/硬件 | 版本 |
|---|---|
| 操作系统 | Debian 11(bullseye) 2C/4G |
| Kubernetes版本 | v1.28.11(截至文章编写时间的最新版) |
| Kubernetes集群部署工具 | kubeadm |
| Kine | v0.11.10 (截至文章编写时间的最新版) |
| MySQL | Docker运行，镜像 mysql:5.7 |
使用 kubeadm 构建控制平面
为了展现 kine 的作用，首先我们需要准备一个 k8s 集群，这里简单使用 kubeadm + containerd 来构建一个 kuebrnetes 集群。
安装 containerd
载入内核依赖项
containerd 或 docker 的安装都需要内核支持 overlay 和 br_netfilter 模块，overlay 为 containerd 运行的文件系统，netfiler 用于维护容器内 (inter-container) 的网络。所以我们需要加载对应的内核模块。
cat <<EOF | tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
手动执行下面命令
modprobe overlay && \
modprobe br_netfilter
通过仓库 containerd
contanerd 是作为 docker-ce 的下层，所以很多 Linux 发行版都有对应的包管理工具的仓库，这里面维护了基本上比较新的版本，可以直接在对应操作系统下载
CentOS
yum install yum-utils -y && \
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo && \
yum install containerd.io -y
Debian
Debian仓库中通常都有比较新版本的 containerd，可以直接安装
apt list|grep containerd
安装
apt -y install containerd
离线安装
如果需要离线环境安装的话，可以在手动下载 containerd 和 runc 后传入内网
下载 Containerd 的二进制包，这里下载containerd-<VERSION>-<OS>-<ARCH>.tar.gz 格式名称的发行版，后边在单独下载安装 runc
wget https://github.com/containerd/containerd/releases/download/v1.7.3/containerd-1.7.3-linux-amd64.tar.gz
将其解压缩到 /usr/local 下:
tar Cxzvf /usr/local containerd-1.7.3-linux-amd64.tar.gz
接下来从 runc 的 github 上下载安装 runc，该二进制文件是静态构建的，并且应该适用于任何Linux发行版。
wget https://github.com/opencontainers/runc/releases/download/v1.1.9/runc.amd64
install -m 755 runc.amd64 /usr/local/sbin/runc
为了通过 systemd 管理 containerd，请还需要从仓库中下载 containerd.service 单元文件
cat << EOF > /usr/lib/systemd/system/containerd.service
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target
[Service]
#uncomment to enable the experimental sbservice (sandboxed) version of containerd/cri integration
#Environment="ENABLE_CRI_SANDBOXES=sandboxed"
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target
EOF
配置配置文件
mkdir -p /etc/containerd && \
containerd config default | sudo tee /etc/containerd/config.toml
配置驱动为 systemd
将配置文件修改为实例所述
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
runtime_engine = ""
runtime_root = ""
privileged_without_host_devices = false
base_runtime_spec = ""
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
一键修改命令
sed -i "s/SystemdCgroup = false/SystemdCgroup = true/g" "${CONTAINDERD_CONFIG_PATH}"
启动服务
systemctl enable --now containerd && \
systemctl restart containerd
使用kubeadm构建集群
加载内核依赖项
cat > /etc/modules-load.d/kubernetes.conf <<EOF
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
EOF
执行以下命令使配置立即生效:
modprobe ip_vs && \
modprobe ip_vs_rr && \
modprobe ip_vs_wrr && \
modprobe ip_vs_sh
安装kubeadm kubelet kubectl
安装 kubeadm 可以参考官网的步骤来 [3]
使用基于debian 包管理仓库
使用 Kubernetes apt 仓库
apt-get install -y apt-transport-https ca-certificates curl gpg
下载公共签名key
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
添加适合的 k8s 版本仓库，这里是 1.28
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
更新包索引
apt update && \
apt install -y kubelet=1.28.11-1.1 kubeadm=1.28.11-1.1 kubectl=1.28.11-1.1
不使用包管理工具
下载 kubeadm, kubelet, kubectl 二进制文件
# 这个文件内包含的是 kubernetes 最新稳定版的版本号，如果要安装最新版可以取消掉这行注释
# RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
RELEASE="v1.28.11"
ARCH="amd64"
DOWNLOAD_DIR="/usr/local/bin"
mkdir -p "$DOWNLOAD_DIR"
cd $DOWNLOAD_DIR
sudo curl -L --remote-name-all https://dl.k8s.io/release/${RELEASE}/bin/linux/${ARCH}/{kubeadm,kubelet}
sudo chmod +x {kubeadm,kubelet}
下载 kubelet 的 system单元文件 或手动添加所需的 systemd 单元文件
# v0.16.2 是一个固定的版本号，不是 kubernetes 版本
RELEASE_VERSION="v0.16.2"
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/krel/templates/latest/kubelet/kubelet.service" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service
# kubelet.service 是一个单元文件
# systemd 的 service.d 目录是一个固定写法，这里表示可以使用 .conf 结尾的文件来覆盖这个服务的单元文件
mkdir -p /etc/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/krel/templates/latest/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
或者手动创建 kubelet.serivce 的 systemd 的单元文件
这个文件是将 rpm 或 dpkg 包的 kubelet.service 和上述 10-kubeadm.conf 融合为一起的，效果是相同的
cat << EOF > /usr/lib/systemd/system/kubelet.serivce
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
Wants=network-online.target
After=network-online.target
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
离线环境镜像下载
列出所使用的镜像
$ kubeadm config images list --kubernetes-version=1.28.8
registry.k8s.io/kube-apiserver:v1.28.8
registry.k8s.io/kube-controller-manager:v1.28.8
registry.k8s.io/kube-scheduler:v1.28.8
registry.k8s.io/kube-proxy:v1.28.8
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.12-0
registry.k8s.io/coredns/coredns:v1.10.1
下载对应镜像，并上传到私有仓库
for n in `./kubeadm config images list --kubernetes-version=1.28.11`;
do
docker pull $n; docker tag $n `echo $n | sed 's|registry.k8s.io|img.xxxx.com/system|'`
docker push `echo $n | sed 's|registry.k8s.io|img.xxx.com/system|'`
done
生成配置文件
./kubeadm config images list --image-repository img.xxx.com/system --kubernetes-version=v1.28.11
# 生成对应组件的的 kubeconfig
# kubelet
kubeadm config print init-defaults --component-configs KubeletConfiguration|grep -A 1000 'apiVersion: kubelet.config.k8s.io'|sed 's|0s|30s|g'
# kube-proxy
kubeadm config print init-defaults --component-configs KubeProxyConfiguration|grep -A 1000 'kubeproxy.config.k8s.io/'|sed 's|0s|30s|g'
使用配置文件安装
kubeadm init --config kube.yaml -v 10
使用命令初始化
kubeadm init \
--image-repository=img.xxx.com/system \
--pod-network-cidr=10.10.0.0/16 \
--service-cidr=10.11.0.0/24 \
--kubernetes-version=v1.28.11 \
--control-plane-endpoint=`hostname -I` \
--apiserver-advertise-address=`hostname -I` \
--apiserver-cert-extra-sans=`hostname -I` \
--v=10
这个时候控制平面已经可以正常工作了
$ kubectl --kubeconfig /etc/kubernetes/admin.conf get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-5dd5756b68-nvqwf 0/1 Pending 0 16h
coredns-5dd5756b68-t2tj5 0/1 Pending 0 16h
etcd-node 1/1 Running 0 16h
kube-apiserver-node 1/1 Running 0 16h
kube-controller-manager-node 1/1 Running 0 16h
kube-proxy-g6fpc 1/1 Running 0 16h
kube-scheduler-node 1/1 Running 0 16h
使用 kine 来替换 etcd
查看官方示例
首先根据 kine 官方 example 来查看最小示例的来学习如何使用 kine [4]，通过文章得知，kine 运行有两种方式，kine 与数据库之间的使用 ssl 链接。
mysql
kine --endpoint "mysql://root:$PASSWORD@tcp(localhost:3306)/kine"
--ca-file ca.crt --cert-file server.crt --key-file server.key
postgres
kine --endpoint="postgres://$(POSTGRES_USERNAME):$(POSTGRES_PASSWORD)@localhost:5432/postgres"
--ca-file=/var/lib/postgresql/ca.crt
--cert-file=/var/lib/postgresql/server.crt
--key-file=/var/lib/postgresql/server.key
这时我们需要查看一下 kine 的参数
GLOBAL OPTIONS:
--listen-address value (default: "0.0.0.0:2379")
--endpoint value Storage endpoint (default is sqlite)
--ca-file value CA cert for DB connection
--cert-file value Certificate for DB connection
--server-cert-file value Certificate for etcd connection
--server-key-file value Key file for etcd connection
--datastore-max-idle-connections value Maximum number of idle connections retained by datastore. If value = 0, the system default will be used. If value < 0, idle connections will not be reused. (default: 0)
--datastore-max-open-connections value Maximum number of open connections used by datastore. If value <= 0, then there is no limit (default: 0)
--datastore-connection-max-lifetime value Maximum amount of time a connection may be reused. If value <= 0, then there is no limit. (default: 0s)
--key-file value Key file for DB connection
--metrics-bind-address value The address the metric endpoint binds to. Default :8080, set 0 to disable metrics serving. (default: ":8080")
--slow-sql-threshold value The duration which SQL executed longer than will be logged. Default 1s, set <= 0 to disable slow SQL log. (default: 1s)
--metrics-enable-profiling Enable net/http/pprof handlers on the metrics bind address. Default is false. (default: false)
--watch-progress-notify-interval value Interval between periodic watch progress notifications. Default is 10m. (default: 10m0s)
--debug (default: false)
--help, -h show help
--version, -v print the version
通过参数得知，上面的除了官方给出的，kine 与数据库之间的连接也可以不使用 ssl，并通过 --server-cert-file 与 --server-key-file 来作为 kube-apiserver 连接 etcd 所使用的证书指定给 kine 就可以启动了。
编写静态文件
这里我们只需要删除 /etc/kubernetes/manifests/etcd.yaml 并将 etcd 使用的证书挂载到 kine pod 中，那么我们编写 /etc/kubernetes/manifests/kine.yaml 文件。
cat /etc/kubernetes/manifests/kine.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kine
tier: control-plane
name: kine
namespace: kube-system
spec:
containers:
- name: kine
command: [ "/bin/sh", "-c", "--" ]
args: [ 'kine --endpoint="mysql://root:111@tcp(10.0.0.1:3306)/kine"
--server-cert-file=/etc/kubernetes/pki/etcd/server.crt
--server-key-file=/etc/kubernetes/pki/etcd/server.key' ]
image: docker.io/rancher/kine:v0.11.10-amd64
imagePullPolicy: IfNotPresent
resources:
requests:
cpu: 250m
volumeMounts:
- mountPath: /etc/kubernetes/pki/etcd
name: etcd-certs
hostNetwork: true
volumes:
- hostPath:
path: /etc/kubernetes/pki/etcd
type: DirectoryOrCreate
name: etcd-certs
status: {}
kubuadm 生成的 kubelet 的 KubeletConfiguration 文件，中静态文件得路径参数 “staticPodPath”
$ cat /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
...
staticPodPath: /etc/kubernetes/manifests
这个时候可以启动 kubelet 服务，然后查看静态 Pod，此时可以看到， kube-system 名称空间 已经没有 etcd pod了
$ kubectl --kubeconfig /etc/kubernetes/admin.conf get pod -n kube-system
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kine-node 1/1 Running 0 17s
kube-system kube-apiserver-node 1/1 Running 19 7m15s
kube-system kube-controller-manager-node 1/1 Running 6 7m5s
kube-system kube-scheduler-node 1/1 Running 6 7m2s
此时就可以继续部署 k8s 的 worker 节点和 CNI 了
探索 kine
我们可以查看数据库表结构，来探索 kine 是如何实现的 etcdAPI 转换的，我们可以看到，kine 会在启动参数中配置的库名 创建对应的数据库，并且仅有一个表 kine
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| kine |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.01 sec)
mysql> show tables;
+----------------+
| Tables_in_kine |
+----------------+
| kine |
+----------------+
观察表结构
mysql> desc kine;
+-----------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
| name | varchar(630) | YES | MUL | NULL | |
| created | int(11) | YES | | NULL | |
| deleted | int(11) | YES | | NULL | |
| create_revision | bigint(20) unsigned | YES | | NULL | |
| prev_revision | bigint(20) unsigned | YES | MUL | NULL | |
| lease | int(11) | YES | | NULL | |
| value | mediumblob | YES | | NULL | |
| old_value | mediumblob | YES | | NULL | |
+-----------------+---------------------+------+-----+---------+----------------+
9 rows in set (0.00 sec)
查看数据是如何存储的
mysql> select count(id),name from kine group by name;
+-----------+---------------------------------------------------------------------------------------------+
| count(id) | name |
+-----------+---------------------------------------------------------------------------------------------+
| 1 | /registry/apiregistration.k8s.io/apiservices/v1. |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.apps |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.autoscaling |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.batch |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.events.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.networking.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.node.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.policy |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1.storage.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1beta2.flowcontrol.apiserver.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v1beta3.flowcontrol.apiserver.k8s.io |
| 1 | /registry/apiregistration.k8s.io/apiservices/v2.autoscaling |
| 1 | /registry/clusterrolebindings/cluster-admin |
| 1 | /registry/clusterrolebindings/system:basic-user |
| 1 | /registry/clusterrolebindings/system:controller:attachdetach-controller |
| 1 | /registry/clusterrolebindings/system:controller:certificate-controller |
| 1 | /registry/clusterrolebindings/system:controller:clusterrole-aggregation-controller |
| 1 | /registry/clusterrolebindings/system:controller:cronjob-controller |
| 1 | /registry/clusterrolebindings/system:controller:daemon-set-controller |
| 1 | /registry/clusterrolebindings/system:controller:deployment-controller |
| 1 | /registry/clusterrolebindings/system:controller:disruption-controller |
| 1 | /registry/clusterrolebindings/system:controller:endpoint-controller |
| 1 | /registry/clusterrolebindings/system:controller:endpointslice-controller |
| 1 | /registry/clusterrolebindings/system:controller:endpointslicemirroring-controller |
| 1 | /registry/clusterrolebindings/system:controller:ephemeral-volume-controller |
| 1 | /registry/clusterrolebindings/system:controller:expand-controller |
| 1 | /registry/clusterrolebindings/system:controller:generic-garbage-collector |
| 1 | /registry/clusterrolebindings/system:controller:horizontal-pod-autoscaler |
| 1 | /registry/clusterrolebindings/system:controller:job-controller |
| 1 | /registry/clusterrolebindings/system:controller:namespace-controller |
| 1 | /registry/clusterrolebindings/system:controller:node-controller |
| 1 | /registry/clusterrolebindings/system:controller:persistent-volume-binder |
| 1 | /registry/clusterrolebindings/system:controller:pod-garbage-collector |
| 1 | /registry/clusterrolebindings/system:controller:pv-protection-controller |
| 1 | /registry/clusterrolebindings/system:controller:pvc-protection-controller |
| 1 | /registry/clusterrolebindings/system:controller:replicaset-controller |
| 1 | /registry/clusterrolebindings/system:controller:replication-controller |
| 1 | /registry/clusterrolebindings/system:controller:resourcequota-controller |
| 1 | /registry/clusterrolebindings/system:controller:root-ca-cert-publisher |
| 1 | /registry/clusterrolebindings/system:controller:route-controller |
| 1 | /registry/clusterrolebindings/system:controller:service-account-controller |
| 1 | /registry/clusterrolebindings/system:controller:service-controller |
| 1 | /registry/clusterrolebindings/system:controller:statefulset-controller |
| 1 | /registry/clusterrolebindings/system:controller:ttl-after-finished-controller |
| 1 | /registry/clusterrolebindings/system:controller:ttl-controller |
| 1 | /registry/clusterrolebindings/system:discovery |
| 1 | /registry/clusterrolebindings/system:kube-controller-manager |
| 1 | /registry/clusterrolebindings/system:kube-dns |
| 1 | /registry/clusterrolebindings/system:kube-scheduler |
| 1 | /registry/clusterrolebindings/system:monitoring |
| 1 | /registry/clusterrolebindings/system:node |
| 1 | /registry/clusterrolebindings/system:node-proxier |
| 1 | /registry/clusterrolebindings/system:public-info-viewer |
| 1 | /registry/clusterrolebindings/system:service-account-issuer-discovery |
| 1 | /registry/clusterrolebindings/system:volume-scheduler |
| 1 | /registry/clusterroles/admin |
| 1 | /registry/clusterroles/cluster-admin |
| 1 | /registry/clusterroles/edit |
| 1 | /registry/clusterroles/system:aggregate-to-admin |
| 1 | /registry/clusterroles/system:aggregate-to-edit |
| 1 | /registry/clusterroles/system:aggregate-to-view |
| 1 | /registry/clusterroles/system:auth-delegator |
| 1 | /registry/clusterroles/system:basic-user |
| 1 | /registry/clusterroles/system:certificates.k8s.io:certificatesigningrequests:nodeclient |
| 1 | /registry/clusterroles/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient |
| 1 | /registry/clusterroles/system:certificates.k8s.io:kube-apiserver-client-approver |
| 1 | /registry/clusterroles/system:certificates.k8s.io:kube-apiserver-client-kubelet-approver |
| 1 | /registry/clusterroles/system:certificates.k8s.io:kubelet-serving-approver |
| 1 | /registry/clusterroles/system:certificates.k8s.io:legacy-unknown-approver |
| 1 | /registry/clusterroles/system:controller:attachdetach-controller |
| 1 | /registry/clusterroles/system:controller:certificate-controller |
| 1 | /registry/clusterroles/system:controller:clusterrole-aggregation-controller |
| 1 | /registry/clusterroles/system:controller:cronjob-controller |
| 1 | /registry/clusterroles/system:controller:daemon-set-controller |
| 1 | /registry/clusterroles/system:controller:deployment-controller |
| 1 | /registry/clusterroles/system:controller:disruption-controller |
| 1 | /registry/clusterroles/system:controller:endpoint-controller |
| 1 | /registry/clusterroles/system:controller:endpointslice-controller |
| 1 | /registry/clusterroles/system:controller:endpointslicemirroring-controller |
| 1 | /registry/clusterroles/system:controller:ephemeral-volume-controller |
| 1 | /registry/clusterroles/system:controller:expand-controller |
| 1 | /registry/clusterroles/system:controller:generic-garbage-collector |
| 1 | /registry/clusterroles/system:controller:horizontal-pod-autoscaler |
| 1 | /registry/clusterroles/system:controller:job-controller |
| 1 | /registry/clusterroles/system:controller:namespace-controller |
| 1 | /registry/clusterroles/system:controller:node-controller |
| 1 | /registry/clusterroles/system:controller:persistent-volume-binder |
| 1 | /registry/clusterroles/system:controller:pod-garbage-collector |
| 1 | /registry/clusterroles/system:controller:pv-protection-controller |
| 1 | /registry/clusterroles/system:controller:pvc-protection-controller |
| 1 | /registry/clusterroles/system:controller:replicaset-controller |
| 1 | /registry/clusterroles/system:controller:replication-controller |
| 1 | /registry/clusterroles/system:controller:resourcequota-controller |
| 1 | /registry/clusterroles/system:controller:root-ca-cert-publisher |
| 1 | /registry/clusterroles/system:controller:route-controller |
| 1 | /registry/clusterroles/system:controller:service-account-controller |
| 1 | /registry/clusterroles/system:controller:service-controller |
| 1 | /registry/clusterroles/system:controller:statefulset-controller |
| 1 | /registry/clusterroles/system:controller:ttl-after-finished-controller |
| 1 | /registry/clusterroles/system:controller:ttl-controller |
| 1 | /registry/clusterroles/system:discovery |
| 1 | /registry/clusterroles/system:heapster |
| 1 | /registry/clusterroles/system:kube-aggregator |
| 1 | /registry/clusterroles/system:kube-controller-manager |
| 1 | /registry/clusterroles/system:kube-dns |
| 1 | /registry/clusterroles/system:kube-scheduler |
| 1 | /registry/clusterroles/system:kubelet-api-admin |
| 1 | /registry/clusterroles/system:monitoring |
| 1 | /registry/clusterroles/system:node |
| 1 | /registry/clusterroles/system:node-bootstrapper |
| 1 | /registry/clusterroles/system:node-problem-detector |
| 1 | /registry/clusterroles/system:node-proxier |
| 1 | /registry/clusterroles/system:persistent-volume-provisioner |
| 1 | /registry/clusterroles/system:public-info-viewer |
| 1 | /registry/clusterroles/system:service-account-issuer-discovery |
| 1 | /registry/clusterroles/system:volume-scheduler |
| 1 | /registry/clusterroles/view |
| 1 | /registry/configmaps/default/kube-root-ca.crt |
| 1 | /registry/configmaps/kube-node-lease/kube-root-ca.crt |
| 1 | /registry/configmaps/kube-public/kube-root-ca.crt |
| 1 | /registry/configmaps/kube-system/extension-apiserver-authentication |
| 1 | /registry/configmaps/kube-system/kube-apiserver-legacy-service-account-token-tracking |
| 1 | /registry/configmaps/kube-system/kube-root-ca.crt |
| 1 | /registry/csinodes/node |
| 1 | /registry/endpointslices/default/kubernetes |
| 1 | /registry/events/default/node.17de1624f3c1624f |
| 1 | /registry/events/default/node.17de1624f3c1e6bb |
| 1 | /registry/events/default/node.17de1624f3c25c4f |
| 1 | /registry/events/default/node.17de1624f5b37dfb |
| 1 | /registry/events/default/node.17de1639e7890c71 |
| 1 | /registry/events/default/node.17de168dce4cdb68 |
| 1 | /registry/events/default/node.17de16a194521b80 |
| 1 | /registry/events/kube-system/kine-node.17de162525650d9b |
| 1 | /registry/events/kube-system/kine-node.17de1625275ca2d7 |
| 1 | /registry/events/kube-system/kine-node.17de16252f773864 |
| 1 | /registry/events/kube-system/kine-node.17de1625a5af90c0 |
| 1 | /registry/events/kube-system/kine-node.17de169d120062cc |
| 1 | /registry/events/kube-system/kine-node.17de169d1361dab8 |
| 1 | /registry/events/kube-system/kine-node.17de169d1855aee6 |
| 1 | /registry/events/kube-system/kine-node.17de16a1969b1ed6 |
| 1 | /registry/events/kube-system/kube-apiserver-node.17de162513417e64 |
| 1 | /registry/events/kube-system/kube-apiserver-node.17de1625158e863e |
| 1 | /registry/events/kube-system/kube-apiserver-node.17de162525e8ebd2 |
| 1 | /registry/events/kube-system/kube-apiserver-node.17de1629c37f0b35 |
| 1 | /registry/events/kube-system/kube-apiserver-node.17de1629f6bc718f |
| 1 | /registry/events/kube-system/kube-apiserver-node.17de162ecf004a1d |
| 1 | /registry/events/kube-system/kube-apiserver-node.17de162eff4060dd |
| 1 | /registry/events/kube-system/kube-apiserver-node.17de1637f005507c |
| 1 | /registry/events/kube-system/kube-apiserver-node.17de1661f1bd6879 |
| 1 | /registry/events/kube-system/kube-apiserver-node.17de16620f441326 |
| 1 | /registry/events/kube-system/kube-apiserver-node.17de16a1985f63bd |
| 1 | /registry/events/kube-system/kube-controller-manager-node.17de162511a4b3be |
| 1 | /registry/events/kube-system/kube-controller-manager-node.17de162512f837ca |
| 1 | /registry/events/kube-system/kube-controller-manager-node.17de16251d8b658b |
| 1 | /registry/events/kube-system/kube-controller-manager-node.17de169b0537b6d0 |
| 1 | /registry/events/kube-system/kube-controller-manager-node.17de16a1971f3999 |
| 1 | /registry/events/kube-system/kube-controller-manager.17de1638e6568ffc |
| 1 | /registry/events/kube-system/kube-controller-manager.17de168d8ed8b9cc |
| 1 | /registry/events/kube-system/kube-controller-manager.17de16a150704739 |
| 1 | /registry/events/kube-system/kube-scheduler-node.17de162512917b00 |
| 1 | /registry/events/kube-system/kube-scheduler-node.17de16251515909b |
| 1 | /registry/events/kube-system/kube-scheduler-node.17de16252295ae29 |
| 1 | /registry/events/kube-system/kube-scheduler-node.17de162a7ee366d4 |
| 1 | /registry/events/kube-system/kube-scheduler-node.17de169c038ba9cc |
| 1 | /registry/events/kube-system/kube-scheduler-node.17de169cf8755bf3 |
| 1 | /registry/events/kube-system/kube-scheduler-node.17de16a19797a620 |
| 1 | /registry/events/kube-system/kube-scheduler.17de1643dd024555 |
| 1 | /registry/events/kube-system/kube-scheduler.17de168dbd6f19b1 |
| 1 | /registry/events/kube-system/kube-scheduler.17de16a24a03d6c8 |
| 1 | /registry/flowschemas/catch-all |
| 1 | /registry/flowschemas/endpoint-controller |
| 1 | /registry/flowschemas/exempt |
| 1 | /registry/flowschemas/global-default |
| 1 | /registry/flowschemas/kube-controller-manager |
| 1 | /registry/flowschemas/kube-scheduler |
| 1 | /registry/flowschemas/kube-system-service-accounts |
| 1 | /registry/flowschemas/probes |
| 1 | /registry/flowschemas/service-accounts |
| 1 | /registry/flowschemas/system-leader-election |
| 1 | /registry/flowschemas/system-node-high |
| 1 | /registry/flowschemas/system-nodes |
| 1 | /registry/flowschemas/workload-leader-election |
| 1 | /registry/health |
| 97 | /registry/leases/kube-node-lease/node |
| 97 | /registry/leases/kube-system/apiserver-6cazmjvz5glfjbabvahmi5cwfy |
| 484 | /registry/leases/kube-system/kube-controller-manager |
| 485 | /registry/leases/kube-system/kube-scheduler |
| 99 | /registry/masterleases/10.0.0.14 |
| 33 | /registry/minions/node |
| 1 | /registry/namespaces/default |
| 1 | /registry/namespaces/kube-node-lease |
| 1 | /registry/namespaces/kube-public |
| 1 | /registry/namespaces/kube-system |
| 1 | /registry/pods/kube-system/kine-node |
| 1 | /registry/pods/kube-system/kube-apiserver-node |
| 1 | /registry/pods/kube-system/kube-controller-manager-node |
| 1 | /registry/pods/kube-system/kube-scheduler-node |
| 1 | /registry/priorityclasses/system-cluster-critical |
| 1 | /registry/priorityclasses/system-node-critical |
| 1 | /registry/prioritylevelconfigurations/catch-all |
| 1 | /registry/prioritylevelconfigurations/exempt |
| 1 | /registry/prioritylevelconfigurations/global-default |
| 1 | /registry/prioritylevelconfigurations/leader-election |
| 1 | /registry/prioritylevelconfigurations/node-high |
| 1 | /registry/prioritylevelconfigurations/system |
| 1 | /registry/prioritylevelconfigurations/workload-high |
| 1 | /registry/prioritylevelconfigurations/workload-low |
| 1 | /registry/ranges/serviceips |
| 1 | /registry/ranges/servicenodeports |
| 1 | /registry/rolebindings/kube-public/system:controller:bootstrap-signer |
| 1 | /registry/rolebindings/kube-system/system::extension-apiserver-authentication-reader |
| 1 | /registry/rolebindings/kube-system/system::leader-locking-kube-controller-manager |
| 1 | /registry/rolebindings/kube-system/system::leader-locking-kube-scheduler |
| 1 | /registry/rolebindings/kube-system/system:controller:bootstrap-signer |
| 1 | /registry/rolebindings/kube-system/system:controller:cloud-provider |
| 1 | /registry/rolebindings/kube-system/system:controller:token-cleaner |
| 1 | /registry/roles/kube-public/system:controller:bootstrap-signer |
| 1 | /registry/roles/kube-system/extension-apiserver-authentication-reader |
| 1 | /registry/roles/kube-system/system::leader-locking-kube-controller-manager |
| 1 | /registry/roles/kube-system/system::leader-locking-kube-scheduler |
| 1 | /registry/roles/kube-system/system:controller:bootstrap-signer |
| 1 | /registry/roles/kube-system/system:controller:cloud-provider |
| 1 | /registry/roles/kube-system/system:controller:token-cleaner |
| 1 | /registry/serviceaccounts/default/default |
| 1 | /registry/serviceaccounts/kube-node-lease/default |
| 1 | /registry/serviceaccounts/kube-public/default |
| 1 | /registry/serviceaccounts/kube-system/attachdetach-controller |
| 1 | /registry/serviceaccounts/kube-system/bootstrap-signer |
| 1 | /registry/serviceaccounts/kube-system/certificate-controller |
| 1 | /registry/serviceaccounts/kube-system/clusterrole-aggregation-controller |
| 1 | /registry/serviceaccounts/kube-system/cronjob-controller |
| 1 | /registry/serviceaccounts/kube-system/daemon-set-controller |
| 1 | /registry/serviceaccounts/kube-system/default |
| 1 | /registry/serviceaccounts/kube-system/deployment-controller |
| 1 | /registry/serviceaccounts/kube-system/disruption-controller |
| 1 | /registry/serviceaccounts/kube-system/endpoint-controller |
| 1 | /registry/serviceaccounts/kube-system/endpointslice-controller |
| 1 | /registry/serviceaccounts/kube-system/endpointslicemirroring-controller |
| 1 | /registry/serviceaccounts/kube-system/ephemeral-volume-controller |
| 1 | /registry/serviceaccounts/kube-system/expand-controller |
| 1 | /registry/serviceaccounts/kube-system/generic-garbage-collector |
| 1 | /registry/serviceaccounts/kube-system/horizontal-pod-autoscaler |
| 1 | /registry/serviceaccounts/kube-system/job-controller |
| 1 | /registry/serviceaccounts/kube-system/namespace-controller |
| 1 | /registry/serviceaccounts/kube-system/node-controller |
| 1 | /registry/serviceaccounts/kube-system/persistent-volume-binder |
| 1 | /registry/serviceaccounts/kube-system/pod-garbage-collector |
| 1 | /registry/serviceaccounts/kube-system/pv-protection-controller |
| 1 | /registry/serviceaccounts/kube-system/pvc-protection-controller |
| 1 | /registry/serviceaccounts/kube-system/replicaset-controller |
| 1 | /registry/serviceaccounts/kube-system/replication-controller |
| 1 | /registry/serviceaccounts/kube-system/resourcequota-controller |
| 1 | /registry/serviceaccounts/kube-system/root-ca-cert-publisher |
| 1 | /registry/serviceaccounts/kube-system/service-account-controller |
| 1 | /registry/serviceaccounts/kube-system/service-controller |
| 1 | /registry/serviceaccounts/kube-system/statefulset-controller |
| 1 | /registry/serviceaccounts/kube-system/token-cleaner |
| 1 | /registry/serviceaccounts/kube-system/ttl-after-finished-controller |
| 1 | /registry/serviceaccounts/kube-system/ttl-controller |
| 1 | /registry/services/endpoints/default/kubernetes |
| 1 | /registry/services/specs/default/kubernetes |
| 1 | compact_rev_key |
+-----------+---------------------------------------------------------------------------------------------+
271 rows in set (0.00 sec)
如上所示，有一个名为的表 “kine”包含所有数据。Kine 使用数据库作为日志结构存储，因此来自 API 服务器的每次写入都会创建一个新行来存储已创建或更新的 Kubernetes 对象，“name” 列使用与 etcd 相同的存储结构 “/registry/RESOURCE_TYPE/NAMESPACE/NAME” 表示集群中对象。
k3s 资源分析
k3s 官方提供了 Resource Profiling [5] 来对比了 RDBMS 与 etcd 的性能对比。
总结
因为 RDBMS 大家都很熟悉，并且更高性能的分布式解决方案也有很多，例如 YugabyteDB (PostgreSQL兼容的分布式数据库)，也可以预创建 kine 表，通过分区形式将不同数据存储到不同的分区内。而且 k8s 对象的历史数据也是可以根据一定的规则进行删除，因为 kubernetes 中的对象都是实时协调的，所以也不怕误删除，这样就会使得 kubernetes 规模有更大扩展的可能。
Reference
[1] Worrying state of Etcd community
[3] Installing kubeadm, kubelet and kubectl
[4] Minimal example of using kine
[6] Goodbye etcd, Hello PostgreSQL: Running Kubernetes with an SQL Database
本文发布于Cylon的收藏册，转载请著名原文链接~
链接：https://www.oomkill.com/2024/06/kubernetes-without-etcd-step-by-step/
版权：本作品采用「署名-非商业性使用-相同方式共享 4.0 国际」 许可协议进行许可。