跟着炎炎盐实践k8s---Kubernetes 1.16.10 二进制高可用集群部署之ETCD部署

xiunai

2020-07-04

这一节我们在三个主节点部署高可用的etcd集群，官方建议5节点或者7节点，我们三节点也是够用的。我们开始吧！

一、环境准备
```
10.13.33.29  etcd-0
  10.13.33.40  etcd-1
  10.13.33.38  etcd-2
```
etcd 数据目录：ETCD_DATA_DIR="/data/k8s/etcd/data"
etcd WAL 目录：ETCD_WAL_DIR="/data/k8s/etcd/wal"
（建议是 SSD 磁盘分区，或者和 ETCD_DATA_DIR 不同的磁盘分区）
二、部署安装ETCD

1、创建etcd相关证书

cd /opt/k8s/work
cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "10.13.33.38",
    "10.13.33.29",
    "10.13.33.40"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF
###host字段指定授权使用该证书的etcd节点IP或子网列表，需要将etcd集群的3个节点都添加其中

###生成证书和私钥

cfssl gencert -ca=/opt/k8s/work/ca.pem     -ca-key=/opt/k8s/work/ca-key.pem     -config=/opt/k8s/work/ca-config.json     -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
ls etcd*pem

###分发到etcd节点

mkdir -p /etc/etcd/cert
cp etcd*.pem /etc/etcd/cert/

2、安装etcd

wget https://github.com/etcd-io/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz         ##下载etcd组件
tar -xvf etcd-v3.3.13-linux-amd64.tar.gz                                                                                                  ##解压 
cp etcd-v3.3.13-linux-amd64/etcd* /opt/k8s/bin/                                                                                     ##复制etcd命令到指定目录
chmod +x /opt/k8s/bin/*                                                                                                                          ##赋权
mkdir -p /data/k8s/etcd/data /data/k8s/etcd/wal                                                                                     ##创建etcd数据目录

3、创建etcd.service

cat > etcd.service.template <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/data/k8s/etcd/data
ExecStart=/opt/k8s/bin/etcd   --data-dir ‘/data/k8s/etcd/data‘   --wal-dir ‘/data/k8s/etcd/wal‘   --name ‘master-01‘ \                                                                                           ###根据etcd节点命名
  --cert-file ‘/etc/etcd/cert/etcd.pem‘   --key-file ‘/etc/etcd/cert/etcd-key.pem‘   --trusted-ca-file ‘/etc/kubernetes/cert/ca.pem‘   --peer-cert-file ‘/etc/etcd/cert/etcd.pem‘   --peer-key-file ‘/etc/etcd/cert/etcd-key.pem‘   --peer-trusted-ca-file ‘/etc/kubernetes/cert/ca.pem‘   --peer-client-cert-auth    --client-cert-auth    --listen-peer-urls ‘https://10.13.33.38:2380‘   --initial-advertise-peer-urls ‘https://10.13.33.38:2380‘   --listen-client-urls ‘https://10.13.33.38:2379,http://127.0.0.1:2379‘   --advertise-client-urls ‘https://10.13.33.38:2379‘   --initial-cluster-token ‘etcd-cluster-0‘   --initial-cluster ‘master-03=https://10.13.33.40:2380,master-01=https://10.13.33.38:2380,master-02=https://10.13.33.29:2380‘ \                         ###各个etcd节点hostname及地址
  --initial-cluster-state ‘new‘   --auto-compaction-mode ‘periodic‘   --auto-compaction-retention ‘1‘   --max-request-bytes ‘33554432‘   --quota-backend-bytes ‘6442450944‘   --heartbeat-interval ‘250‘   --election-timeout ‘2000‘
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

3、启动etcd

systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd && systemctl status etcd|grep Active
curl -L 127.0.0.1:2379/health          ###检查启动状态

4、确认集群健康

[-01 work]# curl -L 127.0.0.1:2379/health   
{"health":"true"}
[-01 work]# ETCDCTL_API=3 /opt/k8s/bin/etcdctl   -w table --cacert=/etc/kubernetes/cert/ca.pem   --cert=/etc/etcd/cert/etcd.pem   --key=/etc/etcd/cert/etcd-key.pem   --endpoints=https://10.13.33.29:2379,https://10.13.33.38:2379,https://10.13.33.40:2379  endpoint status
+--------------------------+------------------+---------+---------+-----------+-----------+------------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+--------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://10.13.33.29:2379 | 3d57cc849d2a16** |  3.3.13 |   33 MB |      true |     36971 |    6201481 |
| https://10.13.33.38:2379 | bff9394ca77d77** |  3.3.13 |   33 MB |     false |     36971 |    6201481 |
| https://10.13.33.40:2379 | 2ce7325ad513f9** |  3.3.13 |   33 MB |     false |     36971 |    6201481 |
+--------------------------+------------------+---------+---------+-----------+-----------+------------+
[-01 work]# ETCDCTL_API=3 /opt/k8s/bin/etcdctl   -w table --cacert=/etc/kubernetes/cert/ca.pem   --cert=/etc/etcd/cert/etcd.pem   --key=/etc/etcd/cert/etcd-key.pem   --endpoints=https://10.13.33.29:2379,https://10.13.33.38:2379,https://10.13.33.40:2379  member list
+------------------+---------+-----------+--------------------------+--------------------------+
|        ID        | STATUS  |   NAME    |        PEER ADDRS        |       CLIENT ADDRS       |
+------------------+---------+-----------+--------------------------+--------------------------+
| 2ce7325ad513f9** | started | master-03 | https://10.13.33.40:2380 | https://10.13.33.40:2379 |
| 3d57cc849d2a16** | started | master-02 | https://10.13.33.29:2380 | https://10.13.33.29:2379 |
| bff9394ca77d77** | started | master-01 | https://10.13.33.38:2380 | https://10.13.33.38:2379 |
+------------------+---------+-----------+--------------------------+--------------------------+
{"health":"true"}[_API=3 /opt/k8s/bin/etcdctl   -w table --cacert=/etc/kubernetes/cert/ca.pem   --cert=/etc/etcd/cert/etcd.pem   --key=/etc/etcd/cert/etcd-key.pem   --endpoints=https://10.13.33.29:2379,https://10.13.33.38:2379,https://10.13.33.40:2379  endpoint health
https://10.13.33.38:2379 is healthy: successfully committed proposal: took = 2.174634ms
https://10.13.33.29:2379 is healthy: successfully committed proposal: took = 1.465878ms
https://10.13.33.40:2379 is healthy: successfully committed proposal: took = 2.36525ms

安装kubectl后可以通过kubectl查看
[-01 ~]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}

5、向etcd写入POD网段信息

etcdctl   --endpoints=https://10.13.33.29:2379,https://10.13.33.38:2379,https://10.13.33.40:2379   --ca-file=/opt/k8s/work/ca.pem   --cert-file=/opt/k8s/work/flanneld.pem   --key-file=/opt/k8s/work/flanneld-key.pem   mk /kubernetes/network/config ‘{"Network":"‘172.30.0.0/16‘", "SubnetLen": 21, "Backend": {"Type": "vxlan"}}‘

6、重启flannel、docker

systemctl restart flanneld docker
systemctl status flanneld docker | grep Active
ip addr show| grep flannel.1

7、检查flannel、docker是否获取到正确的子网

[-01 work]# ip addr show|grep flannel.1
5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    inet 172.30.48.0/32 scope global flannel.1
[-01 work]# ip addr show|grep docker 
6: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    inet 172.30.48.1/21 brd 172.30.55.255 scope global docker0

etcd k8s kubernetes