本文是关于Kubernetes 4A解析的第5章

所有关于Kubernetes 4A部分代码上传至仓库 github.com/cylonchau/hello-k8s-4A

在 kubernetes 集群中，所有的通讯都是用 TLS 进行加密和认证，本文使用一次老集群（二进制部署集群）证书更换作为记录，通过这种方式深入对 kubernetes 的认证方式来了解更换证书的步骤，以及一次模拟老集群的更换步骤。

本文使用证书生成工具为 “kubernetes-generator” [1] 专用于 k8s 二进制部署生成证书和安装包的工具。

note
本文中的引用文档使用 web.archive 进行保存，避免官方版本更新，其概念与文档中的概念有变动导致，资料引用失败。

Kubernetes认证方式

为了了解证书更换需要做那些步骤，所以必须了解 k8s 的认证方式，这样才能更好的在更换证书时对集群上部署的业务系统的影响降低到最低。

X509 证书

kube-apiserver 的启动参数 --client-ca-file，可以使用客户端办法机构，英文代号就是熟悉的 “CA” (Certificate authority)，当这个证书传递后，可以验证 kube-apiserver 用于验证向 kube-apiserver 提供的客户端证书，这里包含 k8s 中提供的用户种类的两种：

  • 用户名：对应证书的 CN (Common Name)
  • 用户组：对应证书的O (organization)，用于对一个用户组进行授权，例如 “system:masters” 表示一个组 [1]，允许不受限制地访问 kube-apiserver

静态token

kube-apiserver 的启动参数 --token-auth-file，是以文件提供给 kube-apiserver，该 Token 会长期有效，并且如果不重启服务 Token 是不会更新。

tip
通常情况下，应避免使用静态 token 功能。

令牌文件的定义必须符合 [a-z0-9]{6}\.[a-z0-9]{16} 的格式，以 “.” 分割，第一部分是 “Token ID”； 第二部分是“令牌秘密（Token Secret” ；其后是这个用户的 Group，Group 可以包含多个，如果包含多个，则对应的列必须用双引号括起来。

bash
1
2

token,user,uid,group
token,user,uid,"group1,group2,group3"

例如

bash
1

123456.1234567890abcdef,dev,1,system:masters
note
system:masters 除非有必要，否则，应避免向该组分发证书。该组用户无法通过 ClusterRole 和 clusterRoleBinding 来控制权限，即使删除了所有集群角色，或者从来就没有分配过集群角色。该组的用户，仍然可以直接请求 kube-apiserver 执行任何操作。

使用时可以这样传入

bash
1
2

curl -k -X GET https://<k8s-apiserver>:6443/api/v1/nodes \
-H "Authorization: Bearer 3c9984.b947d02c14fe0a7f"

Bootstrap Tokens

Bootstrap Tokens 和 “静态Token”是属于相同类型，并存在于kube-system 名称空间中 Secret, 他的类型是 bootstrap.kubernetes.io/token ，==这种设计是为了能够支持 kubeadm==。 的情况下启动集群。[3]

Bootstrap Tokens 和 静态 token 是相同的格式类型，这里就不多做阐述。

这里我们看一下我们生成的 kubelet 相关证书与配置文件

kubelet
$ cat kubelet
# kubernetes kubelet config
#
# You can add your configuration own!
KUBELET_ARGS="--v=0 \
    --logtostderr=true \
    --network-plugin=cni \
    --config=/etc/kubernetes/kubelet-config.yaml \
    --kubeconfig=/etc/kubernetes/auth/kubelet.conf \
    --cgroup-driver=cgroupfs \
    --bootstrap-kubeconfig=/etc/kubernetes/auth/bootstrap.conf"

再查看一下 /etc/kubernetes/auth/bootstrap.conf 的配置文件

yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNxRENDQVpBQ0UwYlFJV1BEazYwK3dib05ZczJnZzA0bkM1QXdEUVlKS29aSWh2Y05BUUVMQlFBd0VURVAKTUEwR0ExVUVBd3dHYXpoekxXTmhNQjRYRFRJME1URXlNekF4TVRnMU9Gb1hEVEkwTVRFeU5EQXhNVGcxT0ZvdwpFVEVQTUEwR0ExVUVBd3dHYXpoekxXTmhNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDCkFRRUFuZ1hXRzBpK3N0WXpwRjYyMzJZNlpEZGZjOVZIR3M0THRWbU5sT1RSWnVWeGplZ2JySnhnTGVwLzNvalcKYWNDWWJpejZ1WmdTcEZHWmwrV3RKWXZMaURlOU5iampuaTgzWUYyYWY1MDVBT2gvNjMwc1Z5L2pIdWhvVGNQeApkOVc3YW42TlFiYlF0eVhlaVNHMldDdDlnbmlmdlpnN3VPbzdZYUhqc2pMYWtRbUt6WUhhKy9KaGFHR09LOEFtCkVxay9IV3grR3FSMnFpVDBXbndkTXhWdnluRFNwYTBrTGt6dWZsbFNtTktPL3VxbTQ2azBpWWNta1h4TWF2SlEKQjNQTVF5aFdtOFRJRVdGUWcvTWE0ZytMdDdJcjh2dmIwRWlyemtOZVFtRFJyaHN4K1hieHlUdnpBRVE2aE9MMgpvMldNR09CZGpIdVlSc0NzQW9abE1JdFFoUUlEQVFBQk1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQTN4NlpWCjJZNjlhTWtKYmJpQnA0RmM3YkRaektadk9xOWp2aUgwTjIvNERId2tPQ3FzenlOSkRkN3hVTUV4NE1BSGp0Vm4KSWVOSit4Sk4xM010eW1lYlhkbEw1b2pDTHArTkZSeEdibXdqM2tSUFBUQ3JZa3NEcGdReGlXVjZ5U2ZXMUNsdAo3UmowOU5jTEpqaU1aMjQyVW9qMzh0dmpEdkw3OFdod0FZb2VSaFBMcHptUndPc0plem9ON2FXZ0FXMDJsM0ZxCm9xVFFDUkJ2SkNDSjhMM09IVmZOYW9vNmI0YzNDbVM2RVdXdThuQmYxNHJVaTJNWCszSGJpbVg2R0w4ZTVnbTYKTXp4Q2VKcGxkbndYeDVRekdPWnpzQ1VQZ2xMRFdhMkxBRzg4V0VFVlRqWkVXbDdZZXVrdTBLV2tVSCs0VjR3WQpsbmFnNFM3ZWw0cFMwRVgrCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://10.0.0.4:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:bootstrapper
  name: system:bootstrapper@kubernetes
current-context: system:bootstrapper@kubernetes
kind: Config
preferences: {}
users:
- name: system:bootstrapper
  user:
    token: 3c9984.b947d02c14fe0a7f

可以用命令去求证 certificate-authority-data 是否与 CA 一致。

bash
1

openssl x509 -noout -text -in <(grep 'certificate-authority-data:' auth/bootstrap.conf | awk '{print $2}'|base64 -d)

使用命令可以查询到这个 bootstrap token 可以操作的权限

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

$ kubectl --token 3c9984.b947d02c14fe0a7f auth can-i --kubeconfig /dev/null --server=https://localhost:6443 --insecure-skip-tls-verify --list
Resources                                       Non-Resource URLs   Resource Names   Verbs
*.*                                             []                  []               [*]
                                                [*]                 []               [*]
selfsubjectaccessreviews.authorization.k8s.io   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                  []               [create]
                                                [/api/*]            []               [get]
                                                [/api]              []               [get]
                                                [/apis/*]           []               [get]
                                                [/apis]             []               [get]
                                                [/healthz]          []               [get]
                                                [/healthz]          []               [get]
                                                [/livez]            []               [get]
                                                [/livez]            []               [get]
                                                [/openapi/*]        []               [get]
                                                [/openapi]          []               [get]
                                                [/readyz]           []               [get]
                                                [/readyz]           []               [get]
                                                [/version/]         []               [get]
                                                [/version/]         []               [get]
                                                [/version]          []               [get]
                                                [/version]          []               [get]

因为我们在创建集群时，手动执行过这条命令

bash
1

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers

这是我们在证书生成脚本中的定义的用户与组

bash
1

echo "$BOOTSTRAP_TOKEN,\"system:bootstrapper\",10001,\"system:bootstrappers\"" > /tmp/token.csv
tip
在使用 bootstrap token 认证时，system:bootstrappers 组会被默认添加

例如 kubeadm 是使用了 controller-manager 中的 CSRApprovingController 自动对 bootstrap token 进行的签发。当接收到 CSR 时，CSRApprovingController 会验证他的 CSR 的合法性。可以从代码中看出。

go
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

func ValidateKubeletClientCSR(req *x509.CertificateRequest, usages sets.String) error {
  if !reflect.DeepEqual([]string{"system:nodes"}, req.Subject.Organization) {
    return organizationNotSystemNodesErr
  }

  if len(req.DNSNames) > 0 {
    return dnsSANNotAllowedErr
  }
  if len(req.EmailAddresses) > 0 {
    return emailSANNotAllowedErr
  }
  if len(req.IPAddresses) > 0 {
    return ipSANNotAllowedErr
  }
  if len(req.URIs) > 0 {
    return uriSANNotAllowedErr
  }

  if !strings.HasPrefix(req.Subject.CommonName, "system:node:") {
    return commonNameNotSystemNode
  }

  if !kubeletClientRequiredUsages.Equal(usages) && !kubeletClientRequiredUsagesNoRSA.Equal(usages) {
    return fmt.Errorf("usages did not match %v", kubeletClientRequiredUsages.List())
  }

  return nil
}

上述代码中表明了 CSR 的签发需满足的条件：

  • SAN Email, IP, URI 必须不提供。
  • 证书的 Organization 字段必须为 system:nodes，例如: O = system:nodes
  • 证书的 commonName 字段必须为 sytem:node: 开头，通常格式定义为 sytem:node:hostname，例如: CN = system:node:debian-demo。
  • 证书 Usage 必须配置 client auth 和 digital signature， 这部分在生成脚本中有配置。

这里做一个小实验，创建一个 匹配上述条件的 CSR，并使用 CertificateSigningRequest 去申请一个证书。

使用 cfssl 工具准备 CSR

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

$ cat << EOF | cfssl genkey - | cfssljson -bare demo
{
  "CN": "system:node:bootstap-demo",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "system:nodes"
    }
  ]
}
EOF

使用 openssl 工具准备 CSR

bash
1
2
3
4

openssl req -new -newkey rsa:2048 -nodes \
-keyout demo-key.pem \
-out demo.csr \
-subj "/CN=system:node:bootstap-demo/O=system:nodes"

查看 demo.csr 的详情

bash
1
2
3
4
5
6
7
8
9

$ openssl req -noout -text -in demo.csr 
Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: CN = system:node:bootstap-demo, O = system:nodes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:

使用 bootstrap token吧 CSR 发送给 apiserver 去申请证书, 需要注意的是 csr 资源的版本 旧版本certificates.k8s.io/v1beta1； certificates.k8s.io/v1 v1.19 。

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

$ cat << EOF | kubectl --token 3c9984.b947d02c14fe0a7f --insecure-skip-tls-verify --server=https://localhost:6443 apply -f - 
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: demo-bootstrap-token-csr
spec:
  signerName: kubernetes.io/kube-apiserver-client
  request: $(openssl req -new -newkey rsa:2048 -nodes -keyout demo-key.pem  -subj "/CN=system:node:bootstap-demo/O=system:nodes" 2>/dev/null| base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - client auth
EOF

在新版本中（v1.19）kube-controller-manager 中的它会自动审批 signerName 为 kubernetes.io/kube-apiserver-client-kubelet 的 CSR [4]。

而旧版本的 CertificateSigningRequest 资源不包含 spec.CertificateSigningRequest

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

$ cat << EOF | kubectl --token 3c9984.b947d02c14fe0a7f --insecure-skip-tls-verify --server=https://localhost:6443 apply -f - 
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: demo-bootstrap-token-csr
spec:
  request: $(openssl req -new -newkey rsa:2048 -nodes -keyout demo-key.pem  -subj "/CN=system:node:bootstap-demo/O=system:nodes" 2>/dev/null| base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - client auth
EOF

查看

bash
1
2

$ kubectl get csr
demo-bootstrap-token-csr   5s      system:bootstrapper           Pending

可以使用下面命令查看被签发的证书

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

$ openssl x509 -noout -text -in <(kubectl get csr demo-bootstrap-token-csr -o jsonpath='{.status.certificate}' | base64 -d)
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            32:f8:19:58:a5:4c:d0:24:82:6a:09:c4:0b:8a:2d:6c:7a:26:f3:7d
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = k8s-ca
        Validity
            Not Before: Nov 23 22:59:00 2024 GMT
            Not After : Nov 24 01:14:01 2024 GMT
        Subject: O = system:nodes, CN = system:node:bootstap-demo
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:97:ec:7c:55:f5:e4:1b:68:55:a1:74:28:2f:6d:
                    27:14:72:35:97:dd:b6:a2:ea:e4:29:8e:59:ca:0c:
                    fe:2b:9e:a8:5f:25:b8:b3:d7:3b:b7:26:b5:3a:27:
                    8f:8d:cc:b4:46:fc:88:b9:0f:55:6f:d1:ca:20:15:
                    16:2d:58:aa:1f:c9:46:01:b3:a7:10:45:39:5d:fc:
                    96:c6:ae:59:e0:a5:f0:61:13:f6:49:69:f6:fc:46:
                    56:5f:db:ac:3b:3f:c8:70:fb:bb:d8:4d:3c:c4:e8:
                    d3:43:12:0f:c8:a2:db:af:62:91:5e:37:2d:ce:5a:
                    04:a6:d9:66:6c:37:2a:b0:d0:78:2e:a7:02:87:f6:
                    78:ae:7a:b7:7e:52:19:d1:24:7e:94:c4:b2:ee:32:
                    e7:c3:90:b7:b3:fc:b8:5b:c8:44:af:5d:72:de:81:
                    b8:b5:86:a4:35:80:8f:97:36:77:bb:16:8d:5a:f2:
                    ef:b7:26:67:03:c7:2f:c7:f4:b1:6e:47:fa:a7:ad:
                    2c:f1:45:e6:3e:19:38:a1:81:52:20:5e:42:85:1c:
                    1a:30:c6:7b:04:c0:b6:e2:13:82:0b:d8:76:8a:a8:
                    c4:12:9a:66:1e:8a:e6:5f:a4:f2:7a:d9:28:af:d5:
                    8a:37:89:26:b8:b3:38:98:6d:a9:33:e6:54:3d:87:
                    8a:81
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                B2:15:FC:C5:FA:2A:53:1A:49:B7:B3:51:32:E4:29:29:2C:76:1E:10

kubelet 配置 bootstrap token

如果使用 bootstrap token 需要启用对应的配置，需要具备下列条件：

  1. kube-apiserver 相关配置

    • kube-apiserver 启用参数 --enable-bootstrap-token-auth=true

  2. kubelet 相关配置

    • 一个用来存储所生成的密钥和证书的路径

    • 一个用来指向尚不存在的 kubeconfig 文件的路径；kubelet 会将 bootstrap token 配置文件放到这个位置。

    • 一个指向 bootstrap token 的 kubeconfig 文件的路径，用来提供 API 服务器的 URL 和启动 bootstrap token。

tip
在启动 kubelet 时，如果参数 –kubeconfig 所指定的文件并不存在，会使用通过参数 –bootstrap-kubeconfig 所指定的启动引导 kubeconfig 配置来向 apiserver 请求客户端证书。 在证书请求被批复并被 kubelet 收回时，一个引用所生成的密钥和所获得证书的 kubeconfig 文件会被写入到通过 –kubeconfig 所指定的文件路径下。 证书和密钥文件会被放到 –cert-dir 所指定的目录中 [5]。

其他认证方式

其他认证方式指的是 serviceaccount, openid, webhook token等方式，因为这些不属于 kubernetes 集群中的基础认证方式，而是属于扩展认证方式，故这里不会在提及。

实验 - 证书更换

构建 kubernetes 1.16.10 实验环境

实验环境使用 kubernetes 1.16.10 来模拟常见的老旧集群（通过二进制方式进行部署）证书过期进行更换的问题。通过上面了解到的“基础认证”方式可以快速做到对集群进行证书更换的问题。

生成证书列表如下

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

├── apiserver-etcd.crt # kube-apiserver访问etcd使用的客户端证书
├── apiserver-etcd.key # kube-apiserver访问etcd使用的客户端证书
├── apiserver-kubelet-client.crt # kube-apiserver访问kubelet使用的证书
├── apiserver-kubelet-client.key # kube-apiserver访问kubelet使用的证书
├── ca.crt # k8s 集群使用的ca
├── ca.key # k8s 集群使用的ca
├── front-proxy-ca.crt # kube-aggregation使用ca是独立的一套
├── front-proxy-ca.key # kube-aggregation使用ca是独立的一套
├── front-proxy-client.crt # kube-aggregation使用
├── front-proxy-client.key # kube-aggregation使用
├── kube-apiserver.crt # kube-apiserver使用
├── kube-apiserver.key # kube-apiserver使用
├── kube-controller-manager.crt # kube-controller-manager使用
├── kube-controller-manager.key # kube-controller-manager使用
├── kube-proxy.crt # kube-proxy使用
├── kube-proxy.key # kube-proxy使用
├── kube-scheduler.crt # kube-scheduler使用
├── kube-scheduler.key # kube-scheduler使用
├── sa.key # serviceaccount 使用
└── sa.pub # serviceaccount 使用

etcd 证书更换

etcd 证书更换部分主要围绕 etcd 所使用的证书进行规划，例如：

  1. ca
  2. server
  3. peer
  4. client

这里使用脚本套件会根据 kubernetes 官方给的一个 kubernetes 集群所需要的一套证书的标准进行生成。手动更换即可。更换完成后检查集群状态。

tip
注意 etcd API 版本，3.5可以直接执行，3.5 之下要使用 V3。
bash
1
2
3
4
5

etcdctl --key=/etc/kubernetes/ssl/etcd-cluster-key.pem\
--cert=/etc/kubernetes/ssl/etcd-cluster.pem \
--cacert=/etc/kubernetes/ssl/ca.pem  \
--endpoints="https://10.0.0.221:2379" \
endpoint health

遇到的问题

text
1
2
3

etcd: {"level":"info","ts":"2024-11-24T12:00:18.546+0800","caller":"embed/etcd.go:465","msg":"starting with peer TLS","tls-info":"cert = /etc/etcd/ssl/etcd-cluster.pem, key = /etc/etcd/ssl/etcd-cluster-key.pem, trusted-ca = /etc/etcd/ssl/ca.pem, client-cert-auth = true, crl-file = ","cipher-suites":[]}

etcd: {"level":"info","ts":"2024-11-24T12:00:18.546+0800","caller":"embed/etcd.go:360","msg":"closing etcd server","name":"cert-expired-1","data-dir":"/var/lib/etcd/{etcd-cluster-name}","advertise-peer-urls":["https://10.0.0.138:2380"],"advertise-client-urls":["https://10.0.0.138:2379"]}

启动后立马停止，这里检查 etcd 证书是否正确。

kube-apiserver 证书更换

kube-apiserver 所使用的配置参数

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

KUBE_APISERVER_ARGS="\
--advertise-address=10.0.0.221 \
--allow-privileged=true \
--anonymous-auth=false \
--apiserver-count=3 \
--audit-log-maxage=7 \
--audit-log-maxbackup=10 \
--audit-log-maxsize=100 \
--audit-policy-file=/etc/kubernetes/config/audit-policy.yaml \
--audit-log-path=/var/log/kubernetes/audit/kubernetes-audit.log \
--authorization-mode=Node,RBAC \
--bind-address=10.0.0.221 \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--enable-aggregator-routing=true \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeClaimResize,TaintNodesByCondition,PodNodeSelector,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,NodeRestriction,Priority,ResourceQuota,PodSecurityPolicy \
--enable-swagger-ui=true \
--etcd-cafile=/etc/etcd/ssl/ca.pem \
--etcd-certfile=/etc/kubernetes/ssl/etcd-cluster.pem \
--etcd-keyfile=/etc/kubernetes/ssl/etcd-cluster-key.pem \
--etcd-servers=https://10.0.0.221:2379,https://10.0.0.138:2379,https://10.0.0.131:2379 \
--event-ttl=1h \
--feature-gates=ExpandInUsePersistentVolumes=true,ExpandPersistentVolumes=true,RotateKubeletServerCertificate=true \
--encryption-provider-config=/etc/kubernetes/pki/encryption-config.yaml \
--insecure-port=0 \
--kubelet-certificate-authority=/etc/kubernetes/ssl/ca.pem \
--kubelet-client-certificate=/etc/kubernetes/ssl/admin.pem \
--kubelet-client-key=/etc/kubernetes/ssl/admin-key.pem \
--kubelet-https=true \
--requestheader-client-ca-filel=/etc/kubernetes/ssl/ca.pem \
--requestheader-allowed-names=aggregator \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--profiling=false \
--proxy-client-cert-file=/etc/kubernetes/ssl/kube-aggregration.pem \
--proxy-client-key-file=/etc/kubernetes/ssl/kube-aggregration-key.pem \
--runtime-config=api/all \
--secure-port=6443 \
--service-account-lookup=true \
--service-account-key-file=/etc/kubernetes/ssl/service-account.pem \
--service-cluster-ip-range=10.191.196.0/24 \
--service-node-port-range=30000-52767 \
--tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \
--tls-min-version=VersionTLS12 \
--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 \
--tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem"

注意需要替换的证书部分

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

## etcd 部分
# etcd 的ca, 套件生成的 etcd 和 k8s 使用的是不同 ca
--etcd-cafile=/etc/etcd/ssl/ca.pem \
# etcd 客户端证书的 key 和 crt
--etcd-certfile=/etc/kubernetes/ssl/etcd-cluster.pem \
--etcd-keyfile=/etc/kubernetes/ssl/etcd-cluster-key.pem \

## kube-apiserver 和 kubelet 通讯部分
# k8s 集群中使用的ca
--kubelet-certificate-authority=/etc/kubernetes/ssl/ca.pem \
# kube-apiserver 请求到 kubelet 使用的证书
# 例如在 kubectl logs / kubectl exec 时，kube-apiserver会直接请求 kubelet 这里需要证书正确
--kubelet-client-certificate=/etc/kubernetes/ssl/admin.pem \
--kubelet-client-key=/etc/kubernetes/ssl/admin-key.pem \

## front-proxy部分，这部分主要适用于 kube-apiserver 与外部 apiserver通讯使用
--proxy-client-cert-file=/etc/kubernetes/ssl/kube-aggregration.pem \
--proxy-client-key-file=/etc/kubernetes/ssl/kube-aggregration-key.pem \

# serviceaccount
--service-account-key-file=/etc/kubernetes/ssl/service-account.pem \

遇到的问题

text
1
2
3

clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://10.0.0.221:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"Kubernetes\")". Reconnecting...

clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://10.0.0.221:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"Kubernetes\")". Reconnecting...

报错提示握手认证失败，手动使用 cert 和 key 访问是正常

bash
1
2

$ etcdctl --key=/etc/kubernetes/ssl/etcd-cluster-key.pem --cert=/etc/kubernetes/ssl/etcd-cluster.pem --cacert=/etc/etcd/ssl/ca.pem  --endpoints="https://10.0.0.221:2379" endpoint health
https://10.0.0.221:2379 is healthy: successfully committed proposal: took = 11.661676ms

原因：注意 etcd ca 是否正确。要确认下面的均为正确配置才可以

bash
1
2
3

--etcd-cafile=/etc/etcd/ssl/ca.pem \
--etcd-certfile=/etc/kubernetes/ssl/etcd-cluster.pem \
--etcd-keyfile=/etc/kubernetes/ssl/etcd-cluster-key.pem \

无法查看 pod 日志 You must be logged in to the server

bash
1

error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log xxx-xxx-xxx))

这个就是上面提到的，要保证 kube-apiserver 和 kubelet 使用的证书是否一致

bash
1
2

--kubelet-client-certificate
--kubelet-client-key

kube-controller-manager 证书更换

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

KUBE_CONTROLLER_MANAGER_ARGS="\
--allocate-node-cidrs=false \
--bind-address=10.0.0.221 \
--cluster-cidr=10.96.192.0/22 \
--authorization-always-allow-paths=/healthz,/metrics \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
--controllers=*,bootstrapsigner,tokencleaner \
--feature-gates=ExpandInUsePersistentVolumes=true,ExpandPersistentVolumes=true,RotateKubeletServerCertificate=true \
--kubeconfig=/etc/kubernetes/config/kube-controller-manager.kubeconfig \
--leader-elect=true \
--profiling=false \
--port=0 \
--node-monitor-grace-period=40s \
--node-monitor-period=5s \
--pod-eviction-timeout=2m \
--root-ca-file=/etc/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/etc/kubernetes/ssl/service-account-key.pem \
--service-cluster-ip-range=10.96.0.0/24 \
--secure-port=10257 \
--terminated-pod-gc-threshold=250 \
--tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \
--tls-min-version=VersionTLS12 \
--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 \
--tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \
--use-service-account-credentials=true"

除了上面提到的，ca 外，只需要替换 serviceaccount 所需的 key 即可

bash
1
2
3
4

--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
# 这个参数和 kube-apiserver 的是一对
--tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \

kube-scheduler 证书更换

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

KUBE_SCHEDULER_ARGS="\
--bind-address=10.0.0.221 \
--config=/etc/kubernetes/config/kube-scheduler.yaml \
--profiling=false \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--port=0 \
--secure-port=10259 \
--feature-gates=ExpandInUsePersistentVolumes=true,ExpandPersistentVolumes=true,RotateKubeletServerCertificate=true \
--tls-cert-file=/etc/kubernetes/ssl/kube-scheduler.pem \
--tls-min-version=VersionTLS12 \
--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 \
--tls-private-key-file=/etc/kubernetes/ssl/kube-scheduler-key.pem"
tip
需要注意的是，除了 kube-apiserver 服务外，其他的组件需要手动更换 kubeconfig 文件。

工作节点证书更换

如果节点使用 bootstrap 证书进行部署的，通常证书不正确 kubelet 会从新自动申请 csr 获取新证书，这部分根据 kubernetes 集群版本不同需要手动进行批准 (kubectl certificate approve)。

另外一种方式就是，手动重新给每一个节点的 kubelet 重新签发证书，并且替换每一个节点的 kube-proxy 的 kubeconfig 文件 (==这里使用的套件是 bootstrap token==)。生成证书的要求按照 bootstrap token 部分介绍的：

  1. SAN Email, IP, URI 必须不提供。
  2. 证书的 Organization 字段必须为 system:nodes，例如: O = system:nodes
  3. 证书的 commonName 字段必须为 sytem:node: 开头，通常格式定义为 sytem:node:hostname，例如: CN = system:node:debian-demo。
  4. 证书 Usage 必须配置 client auth 和 digital signature， 这部分在生成脚本中有配置。

更换cni使用证书

例如 calico 使用了etcd时，etcd为外部etcd时，这里的证书也要替换，否则 calico-controller 会无法连接etcd，下图是对应的日志。

image-20250119225856768

图：calico-controller无法连接etcd

更换serviceaccount

在 kubernetes 官方的 “手动轮换 CA 证书” [6] 一文中有提到，在更新了 CA 之后，集群内 Pod 需要重启以获得新的 Secret 数据。下图是当 Pod 没有重启时，在启动新 Pod 的报错

image-20250119231958618

图：network: Unauthorized

network: Unauthorized 原因如下：

  1. cni node 没有启动
  2. 旧的 default secret 还是旧的，没办法请求到 apiserver 去创建网络空间

当更换完成证书后，检查 node cni 服务全部没问题，但是新建的Pod无法创建网络，提示 network: Unauthorized；重建了所有的 secret后重启 Pod 就恢复了。这时 kubelet 出现日志如下：

text
kubelet报错日志1
1
2
3
4
5
6

W0113 12:11:06.526394   11195 docker_sandbox.go:394] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "a01-im-web-5f6f9b4459-p6gnt_a01": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "c8145349eebe397d4c7f30cb326f20338b668b67ebdfddf5e51be14992a18894"
W0113 12:11:06.528734   11195 docker_sandbox.go:394] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "a01-im-web-5f6f9b4459-p6gnt_a01": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "1823c082be12441a1df17875eb9cc9efdb19d7b37d71653c5182b23b7fcab694"
W0113 12:11:06.532607   11195 docker_sandbox.go:394] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "a01-im-web-5f6f9b4459-p6gnt_a01": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "1bae78ef6fa9dbb48002984d96b2d6222ddc14caf2e9bf04a9abb8fce7ad4a18"
W0113 12:11:06.534956   11195 docker_sandbox.go:394] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "a01-im-web-5f6f9b4459-p6gnt_a01": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "528754d0e7ca862dc6723a573c032f53326d53aff3e768694a11b01090cdf7d1"
W0113 12:11:06.537408   11195 docker_sandbox.go:394] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "a01-im-web-5f6f9b4459-p6gnt_a01": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "39e65a5a7250bd14305412a041fce9873f9dee88ecfc85b9b8c21c5db472a353"
W0113 12:11:06.539590   11195 docker_sandbox.go:394] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "a01-im-web-5f6f9b4459-p6gnt_a01": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "d52e93aa1163041e20ab0b5e8b200f14a70581252c2424c943d53478e8656a79"
text
kubelet报错日志2
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

6300 kubelet_volumes.go:154] orphaned pod "a3f45f45-b6e5-4fb8-9d32-d691f966173a" found, but volume subpaths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
6300 cni.go:364] Error adding push-web-api-6bcc765c4-8r242/0ed92e1b512ecc25414e0540a3bc06060cf4d4ea09ebd0fb9dbe20f1c9e0389e to network calico/k8s-pod-network: Unauthorized
6300 dns.go:135] Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.190.16.41 10.190.16.42 10.240.16.41
6300 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to set up sandbox container "0ed92e1b512ecc25414e0540a3bc06060cf4d4ea09ebd0fb9dbe20f1c9e0389e" network for pod "push-web-api-6bcc765c4-8r242": networkPlugin cni failed to set up pod "push-web-api-6bcc765c4-8r242_prd_user_32as1" network: Unauthorized
6300 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "push-web-api-6bcc765c4-8r242_prd_user_32as1(c87c4346-def7-4326-9406-48de0c641a94)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "0ed92e1b512ecc25414e0540a3bc06060cf4d4ea09ebd0fb9dbe20f1c9e0389e" network for pod "push-web-api-6bcc765c4-8r242": networkPlugin cni failed to set up pod "push-web-api-6bcc765c4-8r242_prd_user_32as1" network: Unauthorized
./kubelet.ph190svr015012.root.log.ERROR.20250113-171637.6300:E0113 17:32:27.121142    6300 kuberuntime_manager.go:710] createPodSandbox for pod "push-web-api-6bcc765c4-8r242_prd_user_32as1(c87c4346-def7-4326-9406-48de0c641a94)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "0ed92e1b512ecc25414e0540a3bc06060cf4d4ea09ebd0fb9dbe20f1c9e0389e" network for pod "push-web-api-6bcc765c4-8r242": networkPlugin cni failed to set up pod "push-web-api-6bcc765c4-8r242_prd_user_32as1" network: Unauthorized
6300 pod_workers.go:191] Error syncing pod c87c4346-def7-4326-9406-48de0c641a94 ("push-web-api-6bcc765c4-8r242_prd_user_32as1(c87c4346-def7-4326-9406-48de0c641a94)"), skipping: failed to "CreatePodSandbox" for "push-web-api-6bcc765c4-8r242_prd_user_32as1(c87c4346-def7-4326-9406-48de0c641a94)" with CreatePodSandboxError: "CreatePodSandbox for pod \"push-web-api-6bcc765c4-8r242_prd_user_32as1(c87c4346-def7-4326-9406-48de0c641a94)\" failed: rpc error: code = Unknown desc = failed to set up sandbox container \"0ed92e1b512ecc25414e0540a3bc06060cf4d4ea09ebd0fb9dbe20f1c9e0389e\" network for pod \"push-web-api-6bcc765c4-8r242\": networkPlugin cni failed to set up pod \"push-web-api-6bcc765c4-8r242_prd_user_32as1\" network: Unauthorized"
6300 cni.go:385] Error deleting logging_filebeat-filebeat-68525/131e91d83d51925a5298be7c21f739c5341db9ecde26bf58509c60278caae46e from network calico/k8s-pod-network: context deadline exceeded
6300 remote_runtime.go:128] StopPodSandbox "131e91d83d51925a5298be7c21f739c5341db9ecde26bf58509c60278caae46e" from runtime service failed: rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod "filebeat-filebeat-68525_logging" network: context deadline exceeded
6300 kuberuntime_manager.go:878] Failed to stop sandbox {"docker" "131e91d83d51925a5298be7c21f739c5341db9ecde26bf58509c60278caae46e"}
6300 cni.go:385] Error deleting public_public-gameinterface-office-74fc5bdcc-sfp5d/8a049c280b256e6afc5832b2f77d1e90b7b0ae8590e0403b5efedb391c4a1a7f from network calico/k8s-pod-network: context deadline exceeded
6300 remote_runtime.go:128] StopPodSandbox "8a049c280b256e6afc5832b2f77d1e90b7b0ae8590e0403b5efedb391c4a1a7f" from runtime service failed: rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod "public-gameinterface-office-74fc5bdcc-sfp5d_public" network: context deadline exceeded

遇到的问题

tls: bad certificate 问题原因，更新完成证书后，Pod 中 CA 会更新，但是 secret token 不会自动更新，容器内请求 API 使用 serviceaccount token，当更换 CA 后必须重启 Pod 才能重新在 Pod 内建立新的 token。

text
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

I0114 11:25:25.925418    1063 log.go:172] http: TLS handshake error from 10.190.15.40:3418: remote error: tls: bad certificate
I0114 11:25:25.931262    1063 log.go:172] http: TLS handshake error from 10.190.15.40:6413: remote error: tls: bad certificate
I0114 11:25:25.931643    1063 log.go:172] http: TLS handshake error from 10.190.15.40:61463: remote error: tls: bad certificate
I0114 11:25:25.937855    1063 log.go:172] http: TLS handshake error from 10.190.15.40:20758: remote error: tls: bad certificate
I0114 11:25:25.939317    1063 log.go:172] http: TLS handshake error from 10.190.15.40:49564: remote error: tls: bad certificate
I0114 11:25:25.945868    1063 log.go:172] http: TLS handshake error from 10.190.15.40:54670: remote error: tls: bad certificate
I0114 11:25:25.945937    1063 log.go:172] http: TLS handshake error from 10.190.15.40:10809: remote error: tls: bad certificate
I0114 11:25:25.954650    1063 log.go:172] http: TLS handshake error from 10.190.15.40:19277: remote error: tls: bad certificate
I0114 11:25:25.955125    1063 log.go:172] http: TLS handshake error from 10.190.15.40:12765: remote error: tls: bad certificate
I0114 11:25:25.963410    1063 log.go:172] http: TLS handshake error from 10.190.15.40:61484: remote error: tls: bad certificate
I0114 11:25:25.963924    1063 log.go:172] http: TLS handshake error from 10.190.15.40:21905: remote error: tls: bad certificate
I0114 11:25:25.972193    1063 log.go:172] http: TLS handshake error from 10.190.15.40:47752: remote error: tls: bad certificate
I0114 11:25:25.972608    1063 log.go:172] http: TLS handshake error from 10.190.15.40:12960: remote error: tls: bad certificate
I0114 11:25:25.972863    1063 log.go:172] http: TLS handshake error from 10.190.15.40:1635: remote error: tls: bad certificate
I0114 11:25:25.975030    1063 log.go:172] http: TLS handshake error from 10.190.15.40:23673: remote error: tls: bad certificate
I0114 11:25:25.975148    1063 log.go:172] http: TLS handshake error from 10.190.15.40:57676: remote error: tls: bad certificate
I0114 11:25:25.982164    1063 log.go:172] http: TLS handshake error from 10.190.15.40:40683: remote error: tls: bad certificate
I0114 11:25:25.982192    1063 log.go:172] http: TLS handshake error from 10.190.15.40:11049: remote error: tls: bad certificate
I0114 11:25:25.990024    1063 log.go:172] http: TLS handshake error from 10.190.15.40:25287: remote error: tls: bad certificate
I0114 11:25:25.990288    1063 log.go:172] http: TLS handshake error from 10.190.15.40:30061: remote error: tls: bad certificate
I0114 11:25:25.992142    1063 log.go:172] http: TLS handshake error from 10.190.15.40:38174: remote error: tls: bad certificate
I0114 11:25:26.000204    1063 log.go:172] http: TLS handshake error from 10.190.15.40:63942: remote error: tls: bad certificate
I0114 11:25:26.002349    1063 log.go:172] http: TLS handshake error from 10.190.15.40:55625: remote error: tls: bad certificate
I0114 11:25:26.027144    1063 log.go:172] http: TLS handshake error from 10.190.15.40:46402: remote error: tls: bad certificate
I0114 11:25:26.027458    1063 log.go:172] http: TLS handshake error from 10.190.15.40:2822: remote error: tls: bad certificate
I0114 11:25:26.036703    1063 log.go:172] http: TLS handshake error from 10.190.15.40:57254: remote error: tls: bad certificate
I0114 11:25:26.037100    1063 log.go:172] http: TLS handshake error from 10.190.15.40:53104: remote error: tls: bad certificate

验证上述问题

使用 SA token 访问 API

bash
1
2
3

curl -k \
 -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
 https://kubernetes.default.svc/api/v1/pods

未重启过的Pod

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

$ ls /var/run/secrets/kubernetes.io/serviceaccount/token
/var/run/secrets/kubernetes.io/serviceaccount/token

$ curl -k \un/secrets/kubernetes.io/serviceaccount/token
  -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  https://kubernetes.default.svc/api/v1/podsecrets/kubernetes.io/serviceaccount/t
{ https://kubernetes.default.svc/api/v1/pods
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

重启更换过 Token 的 Pod

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

curl -k \
  -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  https://kubernetes.default.svc/api/v1/pods
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:netbox:default\" cannot list resource \"pods\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}

总结

  • 了解了 k8s 集群认证模式（TLS Everywhere）。
  • 通过基础知识知道了换证书需要更换那些地方。
  • 没有尝试官方提到的滚动更新CA的方式。

Reference