ETCD 集群部署
目录
下面的操作依托于上一篇文章
部署ETCD集群
这里使用的ETCD为三节点高可用集群，步骤如下
- 下载和分发etcd二进制文件
- 创建etcd集群各节点的x509证书，用于加密客户端(如kubectl)与etcd集群、etcd集群之间的数据流
- 创建etcd的system unit文件，配置服务参数
- 检查集群工作状态
注意: 没有特殊说明都在node01节点操作
Etcd 解析
本次使用etcd单独的域名解析
方法一 修改hosts文件
在所有机器上操作， /etc/hosts 文件最后增加下面内容，也可以替换成自己的IP
10.0.20.11 etcd01 etcd01.k8s.com 10.0.20.12 etcd02 etcd02.k8s.com 10.0.20.13 etcd03 etcd03.k8s.com
方法二 增加bind解析
如果使用的内网 DNS bind 做内网解析增加下面解析
etcd01 IN A 10.0.20.11 etcd02 IN A 10.0.20.12 etcd03 IN A 10.0.20.13
下载和分发etcd二进制文件
二进制文件在 部署前期准备工作 文章中已经下载好，直接使用；
分发二进制文件到ETCD集群节点
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
do
echo ">>> ${node_ip}"
scp etcd-v3.3.13-linux-amd64/etcd* ${node_ip}:/opt/k8s/bin
ssh ${node_ip} "chmod +x /opt/k8s/bin/*"
done创建etcd证书和私钥
cd /opt/k8s/work
cat > etcd-csr.json <<EOF
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"10.0.20.11",
"10.0.20.12",
"10.0.20.13",
"etcd01.k8s.com",
"etcd02.k8s.com",
"etcd03.k8s.com"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "4Paradigm"
}
]
}
EOF
#host字段指定授权使用该证书的etcd节点IP或域名列表，需要将etcd集群的3个节点都添加其中生成证书和私钥
cd /opt/k8s/work cfssl gencert -ca=/opt/k8s/work/ca.pem -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd ls etcd*pem
分发证书和私钥到etcd各个节点
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
do
echo ">>> ${node_ip}"
ssh ${node_ip} "mkdir -p /etc/etcd/cert"
scp etcd*.pem ${node_ip}:/etc/etcd/cert/
done创建etcd的启动文件
这里相对应的etcd 的配置，就保存在启动文件中
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > etcd.service.template <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=${ETCD_DATA_DIR}
ExecStart=/opt/k8s/bin/etcd \ --data-dir=${ETCD_DATA_DIR} \ --wal-dir=${ETCD_WAL_DIR} \ --name=##NODE_NAME## \ --cert-file=/etc/etcd/cert/etcd.pem \ --key-file=/etc/etcd/cert/etcd-key.pem \ --trusted-ca-file=/etc/kubernetes/cert/ca.pem \ --peer-cert-file=/etc/etcd/cert/etcd.pem \ --peer-key-file=/etc/etcd/cert/etcd-key.pem \ --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \ --peer-client-cert-auth \ --client-cert-auth \ --listen-peer-urls=https://##NODE_IP##:2380 \ --initial-advertise-peer-urls=https://##NODE_IP##:2380 \ --listen-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \ --advertise-client-urls=https://##NODE_IP##:2379 \ --initial-cluster-token=etcd-cluster-0 \ --initial-cluster=${ETCD_NODES} \ --initial-cluster-state=new \ --auto-compaction-mode=periodic \ --auto-compaction-retention=1 \ --max-request-bytes=33554432 \ --quota-backend-bytes=6442450944 \ --heartbeat-interval=250 \ --election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF配置说明 (此处不需要修改任何配置)
- WorkDirectory、–data-dir 指定etcd工作目录和数据存储为${ETCD_DATA_DIR},需要在启动前创建这个目录 (后面跟着我操作就可以，会有创建步骤)
- –wal-dir 指定wal目录，为了提高性能，一般使用SSD和–data-dir不同的盘
- –name 指定节点名称，当–initial-cluster-state值为new时，–name的参数值必须位于–initial-cluster列表中
- –cert-file、–key-file ETCD server与client通信时使用的证书和私钥
- –trusted-ca-file 签名client证书的CA证书，用于验证client证书
- –peer-cert-file、–peer-key-file ETCD与peer通信使用的证书和私钥
- –peer-trusted-ca-file 签名peer证书的CA证书，用于验证peer证书
拆分三个配置文件，并修改信息
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))
do
sed -e "s/##NODE_NAME##/${ETCD_NAMES[i]}/" -e "s/##NODE_IP##/${ETCD_IPS[i]}/" etcd.service.template > etcd-${ETCD_IPS[i]}.service
done
etcd*.service分发生成的etcd启动文件到对应的服务器
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
do
echo ">>> ${node_ip}"
scp etcd-${node_ip}.service ${node_ip}:/etc/systemd/system/etcd.service
done启动etcd服务
etcd首次进程启动会等待其他节点加入etcd集群，执行启动命令会卡顿一会，为正常现象
远程创建对应 ETCD 的数据目录等
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
do
echo ">>> ${node_ip}"
ssh ${node_ip} "mkdir -p ${ETCD_DATA_DIR} ${ETCD_WAL_DIR}"
ssh ${node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd " &
done测试 ETCD 集群状态
检查启动结果
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}
do
echo ">>> ${node_ip}"
ssh ${node_ip} "systemctl status etcd|grep Active"
done输出结果:
[ work]# for node_ip in ${MASTER_IPS[@]}
> do
> echo ">>> ${node_ip}"
> ETCDCTL_API=3 /opt/k8s/bin/etcdctl > --endpoints=https://${node_ip}:2379 > --cacert=/etc/kubernetes/cert/ca.pem > --cert=/etc/etcd/cert/etcd.pem > --key=/etc/etcd/cert/etcd-key.pem endpoint health
> done
>>> 10.0.20.11
https://10.0.20.11:2379 is healthy: successfully committed proposal: took = 1.609991ms
>>> 10.0.20.12
https://10.0.20.12:2379 is healthy: successfully committed proposal: took = 1.117871ms
>>> 10.0.20.13
https://10.0.20.13:2379 is healthy: successfully committed proposal: took = 1.49139ms通过下面命令查看当前etcd集群leader
source /opt/k8s/bin/environment.sh
ETCDCTL_API=3 /opt/k8s/bin/etcdctl -w table --cacert=/etc/kubernetes/cert/ca.pem --cert=/etc/etcd/cert/etcd.pem --key=/etc/etcd/cert/etcd-key.pem --endpoints=${ETCD_ENDPOINTS} endpoint status输出结果如下：
[ work]# source /opt/k8s/bin/environment.sh
[ work]# ETCDCTL_API=3 /opt/k8s/bin/etcdctl > -w table --cacert=/etc/kubernetes/cert/ca.pem > --cert=/etc/etcd/cert/etcd.pem > --key=/etc/etcd/cert/etcd-key.pem > --endpoints=${ETCD_ENDPOINTS} endpoint status
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://etcd01.k8s.com:2379 | 6330dc0a28f62066 | 3.3.13 | 16 kB | false | 35 | 14 |
| https://etcd02.k8s.com:2379 | 77bc4da10f4c40bb | 3.3.13 | 16 kB | true | 35 | 14 |
| https://etcd03.k8s.com:2379 | d2573d5cc998d0f0 | 3.3.13 | 16 kB | false | 35 | 14 |
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+---
如果对ETCD集群安装不熟悉的，可以参考文章 CentOS 7 ETCD集群配置大全