二进制部署Kubernetes V1.18.X（etcd集群篇）

1.概述

etcd 是基于 Raft 的分布式 KV 存储系统，由 CoreOS 开发，常用于服务发现、共享配置以及并发控制（如 leader 选举、分布式锁等）。kubernetes 使用 etcd 集群持久化存储所有 API 对象、运行数据，集群节点为奇数（3、5、7等）节点，3个集群节点可以容忍1个节点故障；

2.集群规划

3.ETCD集群部署

3.1自签TLS证书

• 创建SSL证书存放目录（/xdd/soft/tls）:

[root@k8s-master01 ~]# mkdir -p /xdd/soft/tls

• 下载ssl证书生成工具：cfssl

cfssl是一个开源的证书管理工具，使用json文件生成证书，相比openssl更方便使用。下载地址：https://github.com/cloudflare/cfssl/releases；在etcd01节点下执行：

[root@k8s-master01 ~]# cd  /xdd/soft/tls
[root@k8s-master01 tls]# wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl_1.5.0_linux_amd64
[root@k8s-master01 tls]# wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssljson_1.5.0_linux_amd64
[root@k8s-master01 tls]# wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl-certinfo_1.5.0_linux_amd64
[root@k8s-master01 tls]# wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl_1.5.0_checksums.txt

• 校验下载文件完整性

[root@k8s-master01 tls]#sed -i '/windows/d' cfssl_1.5.0_checksums.txt
[root@k8s-master01 tls]# sha256sum -c cfssl_1.5.0_checksums.txt |grep -w OK

• 添加cfssl执行权限

[root@k8s-master01 tls]# chmod +x cfssl_1.5.0_linux_amd64  cfssl-certinfo_1.5.0_linux_amd64  cfssljson_1.5.0_linux_amd64
[root@k8s-master01 tls]# mv cfssl_1.5.0_linux_amd64 /usr/local/bin/cfssl
[root@k8s-master01 tls]# mv cfssljson_1.5.0_linux_amd64 /usr/local/bin/cfssljson
[root@k8s-master01 tls]# mv cfssl-certinfo_1.5.0_linux_amd64 /usr/bin/cfssl-certinfo

• 创建CA证书

证书有效期设置：10年

cat > /xdd/soft/tls/etcd/ca-etcd-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "etcd": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF
cat > /xdd/soft/tls/etcd/ca-etcd-csr.json << EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Shenzheng",
            "ST": "Shenzheng"
        }
    ]
}
EOF

生成证书：[root@k8s-master01 tls]# cfssl gencert -initca ca-etcd-csr.json | cfssljson -bare ca-etcd -

• 使用自签CA签发Etcd HTTPS证书

创建证书生产脚本：touch certificate.sh

cat > /xdd/soft/tls/etcd/server-etcd-csr.json <<EOF
{
    "CN": "etcd",
    "hosts": [
        "127.0.0.1",
        "172.30.103.73",
        "172.30.103.64",
        "172.30.103.92",
        "172.30.103.86",
        "172.30.103.203",
        "172.30.103.11",
        "172.30.103.137",
        "172.30.103.105",
        "172.30.103.44",
        "172.30.103.237"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [{
        "C": "CN",
        "L": "Shenzheng",
        "ST": "Shenzheng",
        "O": "k8s",
        "OU": "System"
    }]
}
EOF

注：上述文件hosts字段中IP为所有etcd节点的集群内部通信IP，一个都不能少； 为了方便后期扩容可以多写几个预留的IP

[root@k8s-master01 tls]# cfssl gencert -ca=ca-etcd.pem -ca-key=ca-etcd-key.pem -config=ca-etcd-config.json -profile=etcd server-etcd-csr.json | cfssljson -bare server-etcd

3.2下载etcd二进制安装包

etcd官方下载地址：https://github.com/etcd-io/etcd/releases

下载指定版本：v3.4.16；创建安装目录：/xdd/soft/etcd

[root@k8s-master01 soft]# mkdir -p /xdd/soft/etcd
[root@k8s-master01 soft]# mkdir -p /xdd/package/etcd && cd /xdd/package/etcd
[root@k8s-master01 etcd]# wget https://github.com/etcd-io/etcd/releases/download/v3.4.16/SHA256SUMS
[root@k8s-master01 etcd]# wget  https://github.com/etcd-io/etcd/releases/download/v3.4.16/etcd-v3.4.16-linux-amd64.tar.gz
[root@k8s-master01 etcd]# sha256sum -c SHA256SUMS |grep OK

[root@k8s-master01 etcd]# mkdir /xdd/soft/etcd/{bin,cfg,ssl} -p
[root@k8s-master01 etcd]# tar -zxvf etcd-v3.4.16-linux-amd64.tar.gz -C /xdd/soft/etcd  --strip-components=1
[root@k8s-master01 etcd]# cd /xdd/soft/etcd
[root@k8s-master01 etcd]# ls -lht .
删除一些注解文件：Documentation、README-etcdctl.md  README.md  READMEv2-etcdctl.md
[root@k8s-master01 etcd]# rm Documentation、README-etcdctl.md  README.md  READMEv2-etcdctl.md -rf
[root@k8s-master01 etcd]# mv etcd etcdctl bin/

3.3 创建etcd服务环境变量

cat > /etc/profile.d/etcd.sh <<EOF
export ETCD_PATH=/xdd/soft/etcd
export ETCD_BIN=\$ETCD_PATH/bin
export PATH=\$ETCD_PATH/bin:\$PATH
export ETCD_PATH ETCD_BIN
EOF
[root@k8s-master01 etcd]# source /etc/profile
[root@k8s-master01 etcd]# which etcd
/xdd/soft/etcd/bin/etcd
[root@k8s-master01 etcd]# which etcdctl
/xdd/soft/etcd/bin/etcdctl

3.4 创建etcd配置文件

[root@k8s-master01 etcd]# mkdir -p /xdd/data/etcd
cat > /xdd/soft/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/xdd/data/etcd"
ETCD_LISTEN_PEER_URLS="https://172.30.103.73:2380,https://127.0.0.1:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.30.103.73:2379,https://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.30.103.73:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.30.103.73:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://172.30.103.73:2380,etcd-2=https://172.30.103.92:2380,etcd-3=https://172.30.103.64:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
# [security]
ETCD_CERT_FILE="/xdd/soft/etcd/ssl/server-etcd.pem"
ETCD_KEY_FILE="/xdd/soft/etcd/ssl/server-etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/xdd/soft/etcd/ssl/ca-etcd.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/xdd/soft/etcd/ssl/server-etcd.pem"
ETCD_PEER_KEY_FILE="/xdd/soft/etcd/ssl/server-etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/xdd/soft/etcd/ssl/ca-etcd.pem"
ETCD_PEER_AUTO_TLS="true"
# [logging]
ETCD_DEBUG="false"
# examples for -log-package-levels etcdserver=WARNING,security=DEBUG
ETCD_LOG_PACKAGE_LEVELS="etcdserver=WARNING,security=INFO"
ETCD_LOGGER="zap"
ETCD_LOG_OUTPUTS="stderr"
EOF

注解：

ETCD_NAME：节点名称，集群中唯一

ETCD_DATA_DIR：数据目录

ETCD_LISTEN_PEER_URLS：集群通信监听地址

ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址

ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址

ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址

ETCD_INITIAL_CLUSTER：集群节点地址

ETCD_INITIAL_CLUSTER_TOKEN：集群Token

ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

具体参数变量含义可参考etcd官方中文文档：https://doczhcn.gitbook.io/etcd/index/index-1/configuration

3.5 创建 etcd 的 systemd unit 模板文件

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
WorkingDirectory=/xdd/soft/etcd
Environment=ETCD_DATA_DIR
EnvironmentFile=-/xdd/soft/etcd/cfg/etcd.conf
ExecStart=/xdd/soft/etcd/bin/etcd
Restart=on-failure
RestartSec=5s
LimitNOFILE=65536


[Install]
WantedBy=multi-user.target
EOF

[root@k8s-master01 etcd]# systemctl daemon-reload
[root@k8s-master01 etcd]# systemctl enable etcd

将TLS证书拷贝到etcd/ssl目录下：

[root@k8s-master01 etcd]# cp /xdd/soft/tls/etcd/*.pem /xdd/soft/etcd/ssl/

3.5 部署etcd集群其它节点服务

• 拷贝以下文件到etcd-2、etcd-3

/etc/profile.d/etcd.sh（环境变量）
[root@k8s-master01 etcd]# scp -r /etc/profile.d/etcd.sh root@172.30.103.92:/etc/profile.d/
[root@k8s-master01 etcd]# scp -r /etc/profile.d/etcd.sh root@172.30.103.64:/etc/profile.d/
/usr/lib/systemd/system/etcd.service(systemd启动文件）
[root@k8s-master01 etcd]# scp /usr/lib/systemd/system/etcd.service root@172.30.103.92:/usr/lib/systemd/system/
[root@k8s-master01 etcd]# scp /usr/lib/systemd/system/etcd.service root@172.30.103.64:/usr/lib/systemd/system/
/xdd/soft/etcd（配置安装包）
[root@k8s-master01 etcd]# scp -r /xdd/soft/etcd root@172.30.103.92:/xdd/soft/
[root@k8s-master01 etcd]# scp -r /xdd/soft/etcd root@172.30.103.64:/xdd/soft/

• 创建数据目录和加载环境变量（etcd-2|etcd-3）

[root@k8s-slave01 soft]# mkdir /xdd/data/etcd -p
[root@k8s-slave01 soft]# source /etc/profile
[root@k8s-slave01 soft]# systemctl daemon-reload
[root@k8s-slave01 soft]# systemctl enable etcd
[root@k8s-slave02 soft]# mkdir /xdd/data/etcd -p
[root@k8s-slave02 soft]# source /etc/profile
[root@k8s-slave02 soft]# systemctl daemon-reload
[root@k8s-slave02 soft]# systemctl enable etcd

重点：权限必须是700

[root@k8s-slave02 soft]#chmod -R 700 /xdd/data/etcd
 /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-2"   # 修改此处，节点2改为etcd-2，节点3改为etcd-3
ETCD_LISTEN_PEER_URLS="https://172.30.103.92:2380"   # 修改此处为当前服务器IP
ETCD_LISTEN_CLIENT_URLS="https://172.30.103.92:2379"    # 修改此处为当前服务器IP
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.30.103.92:2380" # 修改此处为当前服务器IP
ETCD_ADVERTISE_CLIENT_URLS="https://172.30.103.92:2379" # 修改此处为当前服务器IP

依次启动，初次启动etcd-1节点比较慢，需要等待所有节点一起；

[root@k8s-master01 etcd]# systemctl start etcd
[root@k8s-slave01 cfg]# systemctl start etcd
[root@k8s-slave02 cfg]# systemctl start etcd

3.6验证集群状态

创建etcd集群监控检测脚本：etcd-checout

cat > /usr/local/bin/etcd-checout << EOF
#!/bin/bash
set -e
#加载环境变量
source /etc/profile
#定义etcd API接口版本
ETCDCTL_API=3
#定义etcd集群节点地址
ETCD_CLUSTER_NODES="https://172.30.103.73:2379,https://172.30.103.92:2379,https://172.30.103.64:2379"
#定义etcd 部署目录
ETCD_PATH="/xdd/soft/etcd"
#定义etcd环境变量
ETCDCRL="\$ETCD_PATH/bin/etcdctl"
#定义etcd ssl证书目录
ETCD_SSL_PATH="\$ETCD_PATH/ssl"
CACERT="\$ETCD_SSL_PATH/ca-etcd.pem"
CERT="\$ETCD_SSL_PATH/server-etcd.pem"
KEY="\$ETCD_SSL_PATH/server-etcd-key.pem"
#查看etcd集群状态
etcd_status()
{
\${ETCDCRL} --write-out=table --cacert=\${CACERT} --cert=\${CERT} --key=\${KEY} --endpoints="\${ETCD_CLUSTER_NODES}" endpoint status
}
#查看etcd集群健康状态
etcd_health()
{
\${ETCDCRL} --write-out=table --cacert=\${CACERT} --cert=\${CERT} --key=\${KEY}  --endpoints="\${ETCD_CLUSTER_NODES}" endpoint health
}
#查看etcd集群列表信息
etcd_member_list()
{
\${ETCDCRL} --write-out=table --cacert=\${CACERT} --cert=\${CERT} --key=\${KEY}  --endpoints="\${ETCD_CLUSTER_NODES}" member list
}
case \$1 in
    "status"| "-s" )
     etcd_status
     ;;
     "health"| "-h")
     etcd_health
     ;;
     "etcd_member_list"| "-l")
     etcd_member_list
     ;;
     *)
     echo "Usage: server.sh {[-s,status]|[-h,health]|}[-l,list]"
     ;;
esac
EOF

[root@k8s-master01 ~]# chmod +x /usr/local/bin/etcd-checout
[root@k8s-master01 ~]# etcd-checout -h

root@k8s-master01 ~]# etcd-checout -s

[root@k8s-node2 cfg]# etcd-checout -l

到此etcd集群部署完成！