Red Hat Ceph Storage 4安装指南

piaotiejun

2020-04-24

由于搞不到红帽官网的软件包，只能使用CentOS的软件仓库，环境信息如下：
Red Hat Ceph Storage 4安装指南

本文主要介绍服务端的安装配置，客户端的配置以及使用将在下一篇描述。

1、配置yum软件仓库

[ ~]# vi /etc/yum.repos.d/CentOS-Base.repo 
[Storage]
name=CentOS-$releasever - Storage
baseurl=https://mirrors.huaweicloud.com/centos/7.7.1908/storage/x86_64/ceph-nautilus/
gpgcheck=0
enabled=1
[ ~]# yum repolist

Red Hat Ceph Storage 4安装指南

2、配置节点互信操作(ommit)

这里就是配置ssh互信操作，可以参考其他文章。

3、管理节点安装ceph-ansible

这里使用ceph01作为ansible的管理节点。

[ ~]# yum install ceph-ansible

4、创建yml配置文件

切换目录至/usr/share/ceph-ansible文件夹，分别创建all.yml、osds.yml以及site.yml。

[ ceph-ansible]# cd group_vars/
[ group_vars]# cp all.yml.sample all.yml
[ group_vars]# vi all.yml
dummy:
fetch_directory: ~/ceph-ansible-keys
ceph_origin: distro
ceph_repository: dummy
cephx: true
monitor_interface: eth0
journal_size: 5120
public_network: 192.168.120.0/24
dashboard_admin_user: admin
dashboard_admin_password: redhat
grafana_admin_user: admin
grafana_admin_password: redhat
[ group_vars]# cp osds.yml.sample osds.yml
[ group_vars]# vi osds.yml
dummy:
devices:
  - /dev/sdb
osd_auto_discovery: true
[ group_vars]# cd ../
[ ceph-ansible]# cp site.yml.sample site.yml

5、定义Ansible inventory文件

此文件就是/etc/ansible/hosts文件，加入以下内容：

[ ceph-ansible]# vi /etc/ansible/hosts
[mons]
ceph01
ceph02
ceph03

[osds]
ceph01
ceph02
ceph03
ceph04

[mgrs]
ceph01
ceph02

[grafana-server]
ceph03

6、检查各节点是否能正常通信

[ ceph-ansible]# ansible all -m ping
[WARNING]: log file at /root/ansible/ansible.log is not writeable and we cannot create it, aborting

[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by 
default, this will change, but still be user configurable on deprecation. This feature will be removed in version 2.10. 
Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

ceph03 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}
ceph04 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}
ceph02 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}
ceph01 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}

7、开始安装配置

在执行之前，一定要切换到all.yml和site.yml文件所在的目录，否则会报错的。

[ ~]# cd /usr/share/ceph-ansible
[ ceph-ansible]# ansible-playbook site.yml
......
PLAY RECAP ***************************************************************************************************************
ceph01                     : ok=353  changed=46   unreachable=0    failed=0    skipped=509  rescued=0    ignored=0   
ceph02                     : ok=299  changed=42   unreachable=0    failed=0    skipped=451  rescued=0    ignored=0   
ceph03                     : ok=260  changed=53   unreachable=0    failed=0    skipped=385  rescued=0    ignored=0   
ceph04                     : ok=141  changed=23   unreachable=0    failed=0    skipped=264  rescued=0    ignored=0   

INSTALLER STATUS *********************************************************************************************************
Install Ceph Monitor           : Complete (0:01:21)
Install Ceph Manager           : Complete (0:02:19)
Install Ceph OSD               : Complete (0:01:36)
Install Ceph Dashboard         : Complete (0:01:11)
Install Ceph Grafana           : Complete (0:03:12)
Install Ceph Node Exporter     : Complete (0:01:54)

Friday 24 April 2020  12:20:47 +0800 (0:00:00.123)       0:38:03.733 ********** 
=============================================================================== 
ceph-common : install redhat ceph packages --------------------------------------------------------------------- 1267.52s
ceph-infra : install firewalld python binding ------------------------------------------------------------------- 131.81s
ceph-grafana : wait for grafana to start ------------------------------------------------------------------------ 106.77s
ceph-mgr : install ceph-mgr packages on RedHat or SUSE ----------------------------------------------------------- 62.62s
ceph-container-engine : install container packages --------------------------------------------------------------- 60.26s
check for python ------------------------------------------------------------------------------------------------- 21.11s
ceph-common : install centos dependencies ------------------------------------------------------------------------ 19.93s
ceph-grafana : install ceph-grafana-dashboards package on RedHat or SUSE ----------------------------------------- 19.47s
gather and delegate facts ---------------------------------------------------------------------------------------- 12.02s
ceph-osd : use ceph-volume lvm batch to create bluestore osds ---------------------------------------------------- 10.15s
ceph-facts : generate cluster fsid -------------------------------------------------------------------------------- 9.09s
ceph-mgr : wait for all mgr to be up ------------------------------------------------------------------------------ 8.02s
ceph-mon : fetch ceph initial keys -------------------------------------------------------------------------------- 7.40s
ceph-dashboard : set or update dashboard admin username and password ---------------------------------------------- 6.92s
ceph-facts : check if it is atomic host --------------------------------------------------------------------------- 6.21s
ceph-container-engine : start container service ------------------------------------------------------------------- 5.94s
ceph-mgr : create ceph mgr keyring(s) on a mon node --------------------------------------------------------------- 4.98s
ceph-mgr : add modules to ceph-mgr -------------------------------------------------------------------------------- 4.95s
ceph-facts : check for a ceph mon socket -------------------------------------------------------------------------- 4.39s
ceph-mgr : disable ceph mgr enabled modules ----------------------------------------------------------------------- 4.34s

配置完成后，使用下面的目录验证集群状态：

[ ceph-ansible]# ceph -s
  cluster:
    id:     ba2c9862-473f-4ca9-ae28-e907c1cbda45
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 14m)
    mgr: ceph01(active, since 4m), standbys: ceph02
    osd: 4 osds: 4 up (since 10m), 4 in (since 10m)

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   4.0 GiB used, 1.9 TiB / 1.9 TiB avail
    pgs:

8、配置Metadata servers服务

[ ceph-ansible]# vi /etc/ansible/hosts
[mdss]
ceph01
ceph02
ceph03
ceph04
[ ceph-ansible]# cp group_vars/mdss.yml.sample group_vars/mdss.yml
[ ceph-ansible]# ansible-playbook site.yml --limit mdss
......
PLAY RECAP ***************************************************************************************************************
ceph01                     : ok=419  changed=20   unreachable=0    failed=0    skipped=561  rescued=0    ignored=0   
ceph02                     : ok=344  changed=16   unreachable=0    failed=0    skipped=498  rescued=0    ignored=0   
ceph03                     : ok=301  changed=16   unreachable=0    failed=0    skipped=435  rescued=0    ignored=0   
ceph04                     : ok=186  changed=11   unreachable=0    failed=0    skipped=325  rescued=0    ignored=0   

INSTALLER STATUS *********************************************************************************************************
Install Ceph Monitor           : Complete (0:01:09)
Install Ceph Manager           : Complete (0:00:59)
Install Ceph OSD               : Complete (0:01:34)
Install Ceph MDS               : Complete (0:02:01)
Install Ceph Dashboard         : Complete (0:01:24)
Install Ceph Grafana           : Complete (0:01:10)
Install Ceph Node Exporter     : Complete (0:00:56)

Friday 24 April 2020  15:02:58 +0800 (0:00:00.121)       0:11:36.111 ********** 
=============================================================================== 
ceph-grafana : wait for grafana to start ------------------------------------------------------------------------- 35.44s
ceph-mds : install ceph-mds package on redhat or suse ------------------------------------------------------------ 16.13s
ceph-infra : install firewalld python binding -------------------------------------------------------------------- 15.14s
ceph-dashboard : set or update dashboard admin username and password ---------------------------------------------- 7.62s
gather and delegate facts ----------------------------------------------------------------------------------------- 6.37s
ceph-mds : create filesystem pools -------------------------------------------------------------------------------- 6.11s
ceph-mds : assign application to cephfs pools --------------------------------------------------------------------- 5.12s
ceph-mds : set pg_autoscale_mode value on pool(s) ----------------------------------------------------------------- 4.83s
ceph-mds : customize pool size ------------------------------------------------------------------------------------ 4.17s
ceph-mds : get keys from monitors --------------------------------------------------------------------------------- 3.40s
ceph-config : look up for ceph-volume rejected devices ------------------------------------------------------------ 3.34s
ceph-osd : get keys from monitors --------------------------------------------------------------------------------- 3.08s
ceph-mgr : create ceph mgr keyring(s) on a mon node --------------------------------------------------------------- 3.01s
ceph-config : look up for ceph-volume rejected devices ------------------------------------------------------------ 2.80s
ceph-config : look up for ceph-volume rejected devices ------------------------------------------------------------ 2.71s
check for python -------------------------------------------------------------------------------------------------- 2.70s
ceph-osd : apply operating system tuning -------------------------------------------------------------------------- 2.67s
ceph-osd : copy ceph key(s) if needed ----------------------------------------------------------------------------- 2.56s
ceph-mds : copy ceph key(s) if needed ----------------------------------------------------------------------------- 2.44s
ceph-osd : set noup flag ------------------------------------------------------------------------------------------ 2.40s