手动方式部署ceph集群以及添加osd

admin

一、准备

前一篇点击打开链接只部署了一个单机集群。在这一篇里，手动部署一个多机集群：mycluster。我们有三台机器nod1,node2和node3；其中node1可以免密ssh/scp任意其他两台机器。我们的所有工作都在node1上完成。

准备工作包括在各个机器上安装ceph rpm包（见前一篇第1节点击打开链接），并在各个机器上修改下列文件：

/usr/lib/systemd/system/ceph-mon@.service /usr/lib/systemd/system/ceph-osd@.service /usr/lib/systemd/system/ceph-mds@.service /usr/lib/systemd/system/ceph-mgr@.service /usr/lib/systemd/system/ceph-radosgw@.service

修改：

Environment=CLUSTER=ceph <--- 改成CLUSTER=mycluster ExecStart=/usr/bin/... --id %i --setuser ceph --setgroup ceph <--- 删掉--setuser ceph --setgroup ceph二、创建工作目录

在node1创建一个工作目录，后续所有工作都在node1上的这个工作目录中完成；

mkdir /tmp/mk-ceph-cluster cd /tmp/mk-ceph-cluster三、创建配置文件vim mycluster.conf [global] cluster = mycluster fsid = 116d4de8-fd14-491f-811f-c1bdd8fac141 public network = 192.168.100.0/24 cluster network = 192.168.73.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 128 osd pool default pgp num = 128 osd pool default crush rule = 0 osd crush chooseleaf type = 1 admin socket = /var/run/ceph/$cluster-$name.asock pid file = /var/run/ceph/$cluster-$name.pid log file = /var/log/ceph/$cluster-$name.log log to syslog = false max open files = 131072 ms bind ipv6 = false [mon] mon initial members = node1,node2,node3 mon host = 192.168.100.131:6789,192.168.100.132:6789,192.168.100.133:6789 ;Yuanguo: the default value of {mon data} is /var/lib/ceph/mon/$cluster-$id, ; we overwrite it. mon data = /var/lib/ceph/mon/$cluster-$name mon clock drift allowed = 10 mon clock drift warn backoff = 30 mon osd full ratio = .95 mon osd nearfull ratio = .85 mon osd down out interval = 600 mon osd report timeout = 300 debug ms = 20 debug mon = 20 debug paxos = 20 debug auth = 20 [mon.node1] host = node1 mon addr = 192.168.100.131:6789 [mon.node2] host = node2 mon addr = 192.168.100.132:6789 [mon.node3] host = node3 mon addr = 192.168.100.133:6789 [mgr] ;Yuanguo: the default value of {mgr data} is /var/lib/ceph/mgr/$cluster-$id, ; we overwrite it. mgr data = /var/lib/ceph/mgr/$cluster-$name [osd] ;Yuanguo: we wish to overwrite {osd data}, but it seems that 'ceph-disk' forces ; to use the default value, so keep the default now; maybe in later versions ; of ceph the limitation will be eliminated. osd data = /var/lib/ceph/osd/$cluster-$id osd recovery max active = 3 osd max backfills = 5 osd max scrubs = 2 osd mkfs type = xfs osd mkfs options xfs = -f -i size=1024 osd mount options xfs = rw,noatime,inode64,logbsize=256k,delaylog filestore max sync interval = 5 osd op threads = 2 debug ms = 100 debug osd = 100

需要说明的是，在这个配置文件中，我们覆盖了一些默认值，比如：{mon data}和{mgr data}，但是没有覆盖{osd data}，因为ceph-disk貌似强制使用默认值。另外，pid, sock文件被放置在/var/run/ceph/中，以$cluster-$name命名；log文件放置在/var/log/ceph/中，也是以$cluster-$name命名。这些都可以覆盖。

四、生成keyring

在单机部署中点击打开链接，我们说过，有两种操作集群中user及其权限的方式，这里我们使用第一种：先生成keyring文件，然后在创建集群时带入使之生效。

ceph-authtool --create-keyring mycluster.keyring --gen-key -n mon. --cap mon 'allow *'
ceph-authtool --create-keyring mycluster.client.admin.keyring --gen-key -n client.admin --set-uallow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
ceph-authtool --create-keyring mycluster.client.bootstrap-osd.keyring --gen-key -n client.bootstrap-osd --cap mon 'allow profile bootstrap-osd'
( e5 D$ c; Q. V6 g, t6 nceph-authtool --create-keyring mycluster.mgr.node1.keyring --gen-key -n mgr.node1 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
) O1 t) D% K  Z- m8 lceph-authtool --create-keyring mycluster.mgr.node2.keyring --gen-key -n mgr.node2 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
ceph-authtool --create-keyring mycluster.mgr.node3.keyring --gen-key -n mgr.node3 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
ceph-authtool mycluster.keyring --import-keyring mycluster.client.admin.keyring
! ^" d! a& G% f, k( uceph-authtool mycluster.keyring --import-keyring mycluster.client.bootstrap-osd.keyring
ceph-authtool mycluster.keyring --import-keyring mycluster.mgr.node1.keyring
) p5 ^; d, K' J$ f8 r. `9 xceph-authtool mycluster.keyring --import-keyring mycluster.mgr.node2.keyring
ceph-authtool mycluster.keyring --import-keyring mycluster.mgr.node3.keyring
* ~: v0 }2 B! J/ qcat mycluster.keyring [mon.] key = AQA525NZsY73ERAAIM1J6wSxglBNma3XAdEcVg== caps mon = "allow *" [client.admin] key = AQBJ25NZznIpEBAAlCdCy+OyUIvxtNq+1DSLqg== auid = 0 caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [client.bootstrap-osd] key = AQBW25NZtl/RBxAACGWafYy1gPWEmx9geCLi6w== caps mon = "allow profile bootstrap-osd" [mgr.node1] key = AQBb25NZ1mIeFhAA/PmRHFY6OgnAMXL1/8pSxw== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *" [mgr.node2] key = AQBg25NZJ6jyHxAAf2GfBAG5tuNwf9YjkhhEWA== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *" [mgr.node3] key = AQBl25NZ7h6CJRAAaFiea7hiTrQNVoZysA7n/g== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *"五、生成monmap

生成monmap并添加3个monitor

monmaptool --create --add node1 192.168.100.131:6789 --add node2 192.168.100.132:6789 --add node3 192.168.100.133:6789 --fsid 116d4de8-fd14-491f-811f-c1bdd8fac141
monmap [plain] view plain copy monmaptool --print monmap monmaptool: monmap file monmap epoch 0 fsid 116d4de8-fd14-491f-811f-c1bdd8fac141 last_changed 2017-08-16 05:45:37.851899 created 2017-08-16 05:45:37.851899 0: 192.168.100.131:6789/0 mon.node1 1: 192.168.100.132:6789/0 mon.node2 2: 192.168.100.133:6789/0 mon.node3六、分发配置文件

keyring和monmap

把第2、3和4步中生成的配置文件，keyring，monmap分发到各个机器。由于mycluster.mgr.nodeX.keyring暂时使用不到，先不分发它们(见第8节)。

cp mycluster.client.admin.keyring mycluster.client.bootstrap-osd.keyring mycluster.keyring mycluster.conf monmap /etc/ceph scp mycluster.client.admin.keyring mycluster.client.bootstrap-osd.keyring mycluster.keyring mycluster.conf monmap node2:/etc/ceph scp mycluster.client.admin.keyring mycluster.client.bootstrap-osd.keyring mycluster.keyring mycluster.conf monmap node3:/etc/ceph七、创建集群1、创建{mon data}目录mkdir /var/lib/ceph/mon/mycluster-mon.node1
ssh node2 mkdir /var/lib/ceph/mon/mycluster-mon.node2
8 Q2 A  d: ]( I2 a, C* X7 e9 A7 |ssh node3 mkdir /var/lib/ceph/mon/mycluster-mon.node3

注意，在配置文件mycluster.conf中，我们把{mon data}设置为/var/lib/ceph/mon/$cluster-$name，而不是默认的/var/lib/ceph/mon/$cluster-$id；
$cluster-$name展开为mycluster-mon.node1(23)；
默认的$cluster-$id展开为mycluster-node1(23)；

2、初始化monitorceph-mon --cluster mycluster --mkfs -i node1 --monmap /etc/ceph/monmap --keyring /etc/ceph/mycluster.keyring
ssh node2 ceph-mon --cluster mycluster --mkfs -i node2 --monmap /etc/ceph/monmap --keyring /etc/ceph/mycluster.keyring
0 ~; C3 q. E( o7 K) w( Gssh node3 ceph-mon --cluster mycluster --mkfs -i node3 --monmap /etc/ceph/monmap --keyring /etc/ceph/mycluster.keyring
$ x# j8 N2 G7 u! _0 _6 R

注意，在配置文件mycluster.conf，我们把{mon data}设置为/var/lib/ceph/mon/$cluster-$name，展开为/var/lib/ceph/mon/mycluster-mon.node1(23)。ceph-mon会
根据–cluster mycluster找到配置文件mycluster.conf，并解析出{mon data}，然后在那个目录下进行初始化。

3、touch donetouch /var/lib/ceph/mon/mycluster-mon.node1/done
+ `9 J1 V) U, y( B7 w6 d7 yssh node2 touch /var/lib/ceph/mon/mycluster-mon.node2/done
! m- V% }' E+ {ssh node3 touch /var/lib/ceph/mon/mycluster-mon.node3/done4、启动monitorssystemctl start ceph-mon@node1
( h' Q4 B) j. ?9 p3 E! L) l ssh node2 systemctl start [url=mailto:ceph-mon@node2]ceph-mon@node2
  y5 T  j! g1 ?9 p, C2 \ssh node3 systemctl start ceph-mon@node3[/url]
5、检查机器状态ceph --cluster mycluster -s
; o0 n: F; N: S! Q* fcluster: id: 116d4de8-fd14-491f-811f-c1bdd8fac141 health: HEALTH_OK services: mon: 3 daemons, quorum node1,node2,node3 mgr: no daemons active osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 0 kB used, 0 kB / 0 kB avail pgs:八、添加osd

每台集群都有一个/dev/sdb，我们把它们作为osd。

1、删除它们的分区2、prepareceph-disk prepare --cluster mycluster --cluster-uuid 116d4de8-fd14-491f-811f-c1bdd8fac141 --bluestore --block.db /dev/sdb --block.wal /dev/sdb /dev/sdb ssh node2 ceph-disk prepare --cluster mycluster --cluster-uuid 116d4de8-fd14-491f-811f-c1bdd8fac141 --bluestore --block.db /dev/sdb --block.wal /dev/sdb /dev/sdb ssh node3 ceph-disk prepare --cluster mycluster --cluster-uuid 116d4de8-fd14-491f-811f-c1bdd8fac141 /dev/sdb 注意：prepare node3:/dev/sdb时，我们没有加选项：--bluestore --block.db /dev/sdb --block.wal /dev/sdb；后面我们会看它和其他两个有什么不同。3、activateceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring ssh node2 ceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring ssh node3 ceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring

注意：ceph-disk好像有两个问题：

• 前面说过，它不使用自定义的{osd data}，而强制使用默认值 /var/lib/ceph/osd/$cluster-$id
  
• 好像不能为一个磁盘指定osd id，而只能依赖它自动生成。虽然ceph-disk prepare有一个选项–osd-id，但是ceph-disk activate并不使用它而是自己生成。当不匹配时，会出现 如下错误：
  
  

# ceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring command_with_stdin: Error EEXIST: entity osd.0 exists but key does not match mount_activate: Failed to activate '['ceph', '--cluster', 'mycluster', '--name', 'client.bootstrap-osd', '--keyring', '/etc/ceph/mycluster.client.bootstrap-osd.keyring', '-i', '-', 'osd', 'new', u'ca8aac6a-b442-4b07-8fa6-62ac93b7cd29']' failed with status code 17

从 ‘-i’, ‘-‘可以看出，它只能自动生成osd id；

4、检查osd

在ceph-disk prepare时，node1:/dev/sdb和node2:/dev/sdb一样，都有–bluestore –block.db /dev/sdb –block.wal选项；node3:/dev/sdb不同，没有加这些选项。我们看看有什么不同。

4.1 node1

mount | grep sdb /dev/sdb1 on /var/lib/ceph/osd/mycluster-0 type xfs (rw,noatime,seclabel,attr2,inode64,noquota) ls /var/lib/ceph/osd/mycluster-0/ activate.monmap block block.db_uuid block.wal bluefs fsid kv_backend mkfs_done systemd whoami active block.db block_uuid block.wal_uuid ceph_fsid keyring magic ready type ls -l /var/lib/ceph/osd/mycluster-0/block lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:52 /var/lib/ceph/osd/mycluster-0/block -> /dev/disk/by-partuuid/a12dd642-b64c-4fef-b9e6-0b45cff40fa9 ls -l /dev/disk/by-partuuid/a12dd642-b64c-4fef-b9e6-0b45cff40fa9 lrwxrwxrwx. 1 root root 10 Aug 16 05:55 /dev/disk/by-partuuid/a12dd642-b64c-4fef-b9e6-0b45cff40fa9 -> ../../sdb2 blkid /dev/sdb2 /dev/sdb2: PARTLABEL="ceph block" PARTUU cat /var/lib/ceph/osd/mycluster-0/block_uuid a12dd642-b64c-4fef-b9e6-0b45cff40fa9 ls -l /var/lib/ceph/osd/mycluster-0/block.db lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:52 /var/lib/ceph/osd/mycluster-0/block.db -> /dev/disk/by-partuuid/1c107775-45e6-4b79-8a2f-1592f5cb03f2 ls -l /dev/disk/by-partuuid/1c107775-45e6-4b79-8a2f-1592f5cb03f2 lrwxrwxrwx. 1 root root 10 Aug 16 05:55 /dev/disk/by-partuuid/1c107775-45e6-4b79-8a2f-1592f5cb03f2 -> ../../sdb3 blkid /dev/sdb3 /dev/sdb3: PARTLABEL="ceph block.db" PARTUU cat /var/lib/ceph/osd/mycluster-0/block.db_uuid 1c107775-45e6-4b79-8a2f-1592f5cb03f2 ls -l /var/lib/ceph/osd/mycluster-0/block.wal lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:52 /var/lib/ceph/osd/mycluster-0/block.wal -> /dev/disk/by-partuuid/76055101-b892-4da9-b80a-c1920f24183f ls -l /dev/disk/by-partuuid/76055101-b892-4da9-b80a-c1920f24183f lrwxrwxrwx. 1 root root 10 Aug 16 05:55 /dev/disk/by-partuuid/76055101-b892-4da9-b80a-c1920f24183f -> ../../sdb4 blkid /dev/sdb4 /dev/sdb4: PARTLABEL="ceph block.wal" PARTUU cat /var/lib/ceph/osd/mycluster-0/block.wal_uuid 76055101-b892-4da9-b80a-c1920f24183f

可见，node1(node2)上，/dev/sdb被分为4个分区：

• /dev/sdb1: metadata
• /dev/sdb2：the main block device
• /dev/sdb3: db
• /dev/sdb4: wal
  

具体见：ceph-disk prepare –help

4.2 node3

mount | grep sdb /dev/sdb1 on /var/lib/ceph/osd/mycluster-2 type xfs (rw,noatime,seclabel,attr2,inode64,noquota) ls /var/lib/ceph/osd/mycluster-2 activate.monmap active block block_uuid bluefs ceph_fsid fsid keyring kv_backend magic mkfs_done ready systemd type whoami ls -l /var/lib/ceph/osd/mycluster-2/block lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:54 /var/lib/ceph/osd/mycluster-2/block -> /dev/disk/by-partuuid/0a70b661-43f5-4562-83e0-cbe6bdbd31fb ls -l /dev/disk/by-partuuid/0a70b661-43f5-4562-83e0-cbe6bdbd31fb lrwxrwxrwx. 1 root root 10 Aug 16 05:56 /dev/disk/by-partuuid/0a70b661-43f5-4562-83e0-cbe6bdbd31fb -> ../../sdb2 blkid /dev/sdb2 /dev/sdb2: PARTLABEL="ceph block" PARTUU cat /var/lib/ceph/osd/mycluster-2/block_uuid 0a70b661-43f5-4562-83e0-cbe6bdbd31fb

可见，在node3上，/dev/sdb被分为2个分区：

• /dev/sdb1：metadata
• /dev/sdb2：the main block device；db和wal也在这个分区上。
  6 x$ O2 b5 o; _  ]0 A

具体见：ceph-disk prepare –help

5、检查集群状态ceph --cluster mycluster -s cluster: id: 116d4de8-fd14-491f-811f-c1bdd8fac141 health: HEALTH_WARN no active mgr services: mon: 3 daemons, quorum node1,node2,node3 mgr: no daemons active osd: 3 osds: 3 up, 3 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 0 kB used, 0 kB / 0 kB avail pgs:

由于没有添加mgr，集群处于WARN状态。

九、添加mgr1、创建{mgr data}目录mkdir /var/lib/ceph/mgr/mycluster-mgr.node1 ssh node2 mkdir /var/lib/ceph/mgr/mycluster-mgr.node2 ssh node3 mkdir /var/lib/ceph/mgr/mycluster-mgr.node3

注意，和{mon data}类似，在配置文件mycluster.conf中，我们把{mgr data}设置为/var/lib/ceph/mgr/$cluster-$name，而不是默认的/var/lib/ceph/mgr/$cluster-$id。

2、分发mgr的keyringcp mycluster.mgr.node1.keyring /var/lib/ceph/mgr/mycluster-mgr.node1/keyring scp mycluster.mgr.node2.keyring node2:/var/lib/ceph/mgr/mycluster-mgr.node2/keyring scp mycluster.mgr.node3.keyring node3:/var/lib/ceph/mgr/mycluster-mgr.node3/keyring3、启动mgrsystemctl start ceph-mgr@node1 ssh node2 systemctl start ceph-mgr@node2 ssh node3 systemctl start ceph-mgr@node34、检查集群状态ceph --cluster mycluster -s cluster: id: 116d4de8-fd14-491f-811f-c1bdd8fac141 health: HEALTH_OK services: mon: 3 daemons, quorum node1,node2,node3 mgr: node1(active), standbys: node3, node2 osd: 3 osds: 3 up, 3 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 5158 MB used, 113 GB / 118 GB avail pgs:

可见，添加mgr之后，集群处于OK状态。

admin

部署Ceph mon服务
安装Ceph-mon服务程序（所有设备执行）
& N, M/ k; u, y  Cyum install -y ceph-mon
8 C+ u) A/ n3 O" r! f, E; V1
& a2 I6 D1 `$ o3 X6 e6 f初始化Mon服务（Ceph 01执行）
& _& q$ a4 E% m" B! k( Q- g生成uuid
uuidgen
> 9bf24809-220b-4910-b384-c1f06ea80728
. C, z7 n2 }7 k$ j* {1
2
创建Ceph配置文件
' Q1 w- R" W% m) ^cat >> /etc/ceph/ceph.conf <<EOF
[global]
fsid = 9bf24809-220b-4910-b384-c1f06ea80728
5 Q0 g( G5 H3 p: Z# C  Z/ H) Nmon_initial_members = ceph01,ceph02,ceph03
8 I) G4 L0 m) k" b5 `mon_host = 10.40.65.156,10.40.65.175,10.40.65.129
( b. _0 m0 _6 @- o- {public_network = 10.40.65.0/24
. Y6 Z& l9 O9 [3 q6 v) p( W" d3 _7 m5 h2 _auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
- j3 Z+ Y1 O9 oosd_journal_size = 1024
osd_pool_default_size = 3
7 r+ d7 r! c/ z# rosd_pool_default_min_size = 2
osd_pool_default_pg_num = 64
osd_pool_default_pgp_num = 64
) V0 M. }0 t' K4 s# {* I7 fosd_crush_chooseleaf_type = 1
EOF
1
$ v6 A8 M' M1 u( _2
3
1 N; r3 E* V' z! j. j4
, O2 J5 Y0 @+ ?& `6 Z% J- V5
$ M- j/ X  G0 |6
7 N: f. A. j* p7
8
9
10
11
' k: N; \! V# F$ B% t0 r) g12
! r& P  K* D, d% d* m: C6 j13
14
15
16
; A8 x' i/ q/ c' d/ ]( N% h创建集群Monitor密钥。
) T4 u& N; O* M8 aceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
8 K5 t- O  T1 m1
4 A0 f& \6 i4 m  y" b0 C创建client.admin用户、client.bootstrap-osd用户密钥，添加到集群密钥中。
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
1
2
ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
1
% n  J/ O3 c% V, A& e$ t2
使用主机名、主机IP地址、FSID生成monitor map。
monmaptool --create --add ceph01 10.40.65.156 --add ceph02 10.40.65.175 --add ceph03 10.40.65.129 --fsid 9bf24809-220b-4910-b384-c1f06ea80728 /tmp/monmap
1
8 k. G) h5 `1 U0 f" f  d4 n初始化并启动monitor服务
4 h! m" _* y+ }6 usudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph01
1 a7 D+ i# J" G1 b" ochown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
* b; z5 u: p& p8 _  qsudo -u ceph ceph-mon --mkfs -i ceph01 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
3 x+ Q( e. y% a# U" Y) |ls /var/lib/ceph/mon/ceph-ceph01/
( c' [# M: E7 S$ v1 }* i1
$ _' ?8 R4 b) g$ ?: p2
3
4
systemctl start ceph-mon@ceph01
systemctl enable ceph-mon@ceph01
* ~6 R) @# |+ ~' ^1 osystemctl status ceph-mon@ceph01
* V9 v8 R# i- `% A1
9 H' o% T/ k! \$ L# T2
3
同步配置文件、密钥、monmap到其他节点中（Ceph 01执行）
6 |1 B/ P- s9 L0 [6 L  _复制ceph.client.admin.keyring、client.bootstrap-osd key、ceph.mon.keyring、monitor map、ceph.conf到另外2个节点
6 t  y9 y! a. T; J" ~scp /etc/ceph/ceph.client.admin.keyring root@ceph02:/etc/ceph/
scp /etc/ceph/ceph.client.admin.keyring root@ceph03:/etc/ceph/
1
2
scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph02:/var/lib/ceph/bootstrap-osd/
scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph03:/var/lib/ceph/bootstrap-osd/
4 f# _. s2 L: B2 U1 ~* ]7 G1
2
scp /tmp/ceph.mon.keyring root@ceph02:/tmp/
scp /tmp/ceph.mon.keyring root@ceph03:/tmp/
2 n$ q: _& G% W! p9 S* X$ _1
2
$ {: u" V  S  {1 w7 Xscp /tmp/monmap root@ceph02:/tmp/
5 t) U" M& W$ s- n, r+ b$ }% O) Gscp /tmp/monmap root@ceph03:/tmp/
1
2
scp /etc/ceph/ceph.conf root@ceph02:/etc/ceph/
scp /etc/ceph/ceph.conf root@ceph03:/etc/ceph/
1
2
+ v- F: B, _; t% G3 ]/ H启动其他节点的monitor服务（Ceph 02执行）
2 m7 K4 z/ e4 |9 W! o% msudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph02
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
) O* {; |1 c4 nsudo -u ceph ceph-mon --mkfs -i ceph02 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
ls /var/lib/ceph/mon/ceph-ceph02/
8 N9 V* A3 ~9 Y8 |- G1
2
3
4
systemctl start ceph-mon@ceph02
0 o) ?, @: \5 dsystemctl enable ceph-mon@ceph02
systemctl status ceph-mon@ceph02
* w" @  l/ a( P3 g1
% D& X3 Z- |8 ~2
% f- b% i& ?; K3
启动其他节点的monitor服务（Ceph 03执行）
4 S& ]3 E( b" _9 asudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph03
, r. B: X, [3 V& cchown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
) m# B. d$ ?" T5 {% w( C/ [sudo -u ceph ceph-mon --mkfs -i ceph03 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
1 I7 I) U% e" T: uls /var/lib/ceph/mon/ceph-ceph03/
# N/ \6 m2 g* u- W/ H5 m$ p2 v1
+ ?: c( F2 `$ O6 t, T# @2
3
" [: }2 C4 g5 ?1 r- e- c6 Y4
systemctl start ceph-mon@ceph03
- y# _7 S3 m; w7 D/ nsystemctl enable ceph-mon@ceph03
systemctl status ceph-mon@ceph03
% Z, O" X8 o: i# W0 n1
" C4 ?; @+ u' x7 A& c- X2 Y2
3
& u. `. t8 ~9 ]) u! s  \查看当前集群状态（任意节点执行）
; n% P  y+ s/ I1 }6 w通过ceph -s命令查询集群状态，可见集群services中3个mon服务已启动。
7 o' f4 N7 ]+ c3 ?: ?8 D( _ceph -s
> cluster:
8 d# I* P% l& I1 ^3 S>   id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a  
0 `! K$ F3 r; ?) _. I9 g2 i3 F>   health: HEALTH_OK
>
> services:  
+ n4 N' @9 W$ w' v>   mon:  3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)  
  I: T. ?0 K9 \4 ?" J>   mgr:   
% {3 X( u7 O/ o/ P" A1 j# h>   osd:
>
  ]7 c* e9 n8 s, g6 L> data:
4 B3 @) P3 T! H! `9 K- u8 `>    pools:   
! l1 v( E6 d& C+ h>   objects:
>   usage:   
>   pgs:
1
% I7 [7 i8 ?+ Q2
6 z9 g3 R0 m# s) y8 x! r3
/ o) u! m+ [2 o& d' d. }- R' n4
1 C' B/ v$ O' b" x7 _5
) Y! c9 B5 A- t* N& U, [* k6
7
8
9
10
11
12
( r, w" Q  Z5 i( e# `6 W+ }$ w13
14
5 W- `- Y- b$ c+ ~15
" j( e9 y: `1 w. S5 T' Y部署Ceph mon服务（ceph-volume 自动化创建）
安装Ceph-osd服务程序（所有设备执行）
, W( B3 l" F: N- n9 dyum install -y ceph-osd
8 V, E7 c, d) F7 z' `. Q+ G1
9 o# B, V; W0 H. W: X初始化osd服务（所有设备执行）
  V. [) R$ q( h9 q通过fdisk等工具查看磁盘盘符，然后利用ceph-volume工具自动化创建osd服务。
ceph-volume lvm create --data /dev/sda
2 P' E  M" F7 @9 V, b; `6 G% Y, uceph-volume lvm create --data /dev/sdb
5 I$ L* k2 k8 U" _ceph-volume lvm create --data /dev/sdc
1 m9 V3 W; j: L' x  f. s1
0 I& w$ d  |! t+ z2
3
查看当前集群状态（任意节点执行）
通过ceph osd tree命令查询集群状态，可见集群services中所有osd服务已启动。
$ Y; u% ]1 y! q$ V+ ]ceph osd tree
> ID CLASS WEIGHT   TYPE NAME       STATUS REWEIGHT PRI-AFF
> -1       16.36908 root default
> -3        5.45636     host ceph01
>  0   hdd  1.81879         osd.0       up  1.00000 1.00000
: _/ G: N$ M* E* `6 |>  1   hdd  1.81879         osd.1       up  1.00000 1.00000
( K* a. ?6 A2 h9 c% B. D! S* F>  2   hdd  1.81879         osd.2       up  1.00000 1.00000
> -5        5.45636     host ceph02
  O9 i+ p; ]# v; q8 Y, ^>  3   hdd  1.81879         osd.3       up  1.00000 1.00000
>  4   hdd  1.81879         osd.4       up  1.00000 1.00000
>  5   hdd  1.81879         osd.5       up  1.00000 1.00000
> -7        5.45636     host ceph03
" K" i" h, g- @5 X& U>  6   hdd  1.81879         osd.6       up  1.00000 1.00000
! J6 ~, n4 `# f8 `>  7   hdd  1.81879         osd.7       up  1.00000 1.00000
>  8   hdd  1.81879         osd.8       up  1.00000 1.00000
7 T8 E/ F2 X1 V3 Y( d0 w1
2
. q1 E1 c& D0 o8 L( U5 a3
1 l8 Z/ T8 @6 ^% v% b6 r  N( ]) g3 b$ w4
5
6
7
" F6 w, }* g4 {7 R; p- C8
* J3 v# H2 T6 w& e6 R9
10
11
" {/ w4 W# n% a; J  {8 `12
13
14
15
3 N7 A& h' r  O; j: G* ?部署Ceph mgr服务并开启Dashboard
3 d0 C) `8 V* W" |. ~. V( l! L安装Ceph-mgr服务程序（所有设备执行）
yum install -y ceph-mgr
1
. H0 s. O% x7 @初始化并启动主MGR服务（Ceph01执行）
) D8 L9 j3 z5 w- W( Amkdir -p /var/lib/ceph/mgr/ceph-ceph01
chown ceph.ceph -R /var/lib/ceph
- l, A# N; Q- [ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph01.keyring --gen-key -n mgr.ceph01 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
/ @+ ^7 m4 B4 V" `% t5 l1 T  kceph auth import -i /etc/ceph/ceph.mgr.ceph01.keyring
' X1 z% U1 y4 ?) ]4 s4 \; sceph auth get-or-create mgr.ceph01 -o /var/lib/ceph/mgr/ceph-ceph01/keyring
5 p) {1 y9 C$ F" H' Y1
7 A) g4 j0 g  Q3 C4 R' v2
3
3 ^% @4 q  ]/ N: x4
5
( _  h" K# n# c! C/ [systemctl start ceph-mgr@ceph01
systemctl enable ceph-mgr@ceph01
systemctl status ceph-mgr@ceph01
1
2
5 ~: z2 V/ w& h- V9 n3
/ n4 d. {' j$ a* w( k: [0 P' k初始化并启动从MGR服务（Ceph02执行）
( c0 d. M$ p/ S1 o0 xmkdir -p /var/lib/ceph/mgr/ceph-ceph02
! r: N# W# K% _chown ceph.ceph -R /var/lib/ceph
' b+ \6 ]0 D0 `) Gceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph02.keyring --gen-key -n mgr.ceph02 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
ceph auth import -i /etc/ceph/ceph.mgr.ceph02.keyring
ceph auth get-or-create mgr.ceph02 -o /var/lib/ceph/mgr/ceph-ceph02/keyring
1
2
3
4
5
systemctl start ceph-mgr@ceph02
systemctl enable ceph-mgr@ceph02
systemctl status ceph-mgr@ceph02
& H% U7 s  D; Z. D1
# o: \" l$ L. g/ ~# M2
, ^! s+ B0 g8 ~$ O3 n8 \3
! C4 }1 ^+ y- @& {初始化并启动从MGR服务（Ceph03执行）
mkdir -p /var/lib/ceph/mgr/ceph-ceph03
chown ceph.ceph -R /var/lib/ceph
ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph03.keyring --gen-key -n mgr.ceph03 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
ceph auth import -i /etc/ceph/ceph.mgr.ceph03.keyring
ceph auth get-or-create mgr.ceph03 -o /var/lib/ceph/mgr/ceph-ceph03/keyring
( _& S- K( |' f8 D) o1
2
% j! K0 l0 {( L. I# p3
4
5
, r. p, h. p  G4 }& c8 ^# z  R4 Msystemctl start ceph-mgr@ceph03
systemctl enable ceph-mgr@ceph03
systemctl status ceph-mgr@ceph03
1
8 F( F. @; M) ?, Z8 ?2
, J/ |( O, h& c3 D" [+ b3
+ G9 Z! m7 }5 g查看当前集群状态（任意节点执行）
0 n/ ?" U9 A& p, m3 b7 U; T( H6 Y通过ceph -s命令查询集群状态，可见集群services中3个mgr服务已启动。
ceph -s
. }$ O% _6 z4 d1 V; ]> cluster:
>   id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a  
. c+ g7 z" V- ]) g/ I, F>   health: HEALTH_OK
! o% l3 g5 A' b8 F; c. s% A>
; Y  u5 |& v. A) q> services:  
>   mon:  3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)  
>   mgr: ceph01(active, since 3w), standbys: ceph01、ceph02   
& Z1 W1 n& _7 F" U) d4 e* z' H>   osd:
>
> data:
  X5 A& a. e5 h! g7 J>    pools:   
>   objects:
2 W# y: F! v7 ]8 t& {) |6 l7 {/ ~  G>   usage:   
1 r7 \" L! O7 Q  z, u/ E) J* G>   pgs:
/ E5 c8 S# w! s& Z$ O3 Q5 `1
2
5 ?6 E# {# _9 R# q3
) L, y) C% _  [, {8 \; X4
5
6
4 H3 L5 ~8 u, O0 m7
' t# K3 n3 ^# `( Q8 ]- W7 U* S7 n  Z8
& G1 K2 m, I" O$ a9
7 i9 d8 Q6 e' L$ J1 u10
- @; H, B1 i! l  ?6 }( ^11
& G) \4 F2 _0 f$ Y- @12
13
& Z- i+ ~* w$ W. Q  _14
+ M+ _/ ^1 D- P7 C& X; R) q% Y15
7 s) D9 l: H- n" m4 `. e使能Dashboard访问功能（任意节点执行）
+ L% b" i0 j1 G9 C开启mgr dashboard功能
ceph mgr module enable dashboard
; b. D5 I9 b* z5 D  |1
+ F3 a) b' x+ O% R* \0 Q% z! d生成并安装自签名的证书
' \% v0 y" ^0 P& A% E9 zceph dashboard create-self-signed-cert
1
  `1 _. g* J2 d配置dashboard
* |' ~1 A% g& Q, ~5 uceph config set mgr mgr/dashboard/server_addr 10.40.65.148
ceph config set mgr mgr/dashboard/server_port 8080
$ v$ G2 b/ {+ Qceph config set mgr mgr/dashboard/ssl_server_port 8443
! H5 I5 y5 }. r1 _( p* F$ n2 f1
2
3
创建一个dashboard登录用户名密码
echo '123456' > password.txt
9 @! t8 ~" k/ o: {ceph dashboard ac-user-create admin  administrator -i password.txt
# N# b6 [& y" o/ c1
2
查看服务访问方式
( p0 p% w. U3 f, g% G  K* Gceph mgr services
1
通过web访问Ceph Dashboard，用户名密码为admin/123456
7 y7 P! Q2 N: S; t0 p  n+ B; e+ dhttps://10.40.65.148:8443

# U# Y8 G4 e& S9 J

admin

部署Ceph mon服务
安装Ceph-mon服务程序（所有设备执行）
8 [. k& L6 s& f( e: s
6 Q# a+ \! V7 r  X3 w8 J: P- B8 r2 iyum install -y ceph-mon
1
' k8 @, i$ ^3 t/ O! i+ a; K2 p% e& {初始化Mon服务（Ceph 01执行）
; N& P+ l( i3 J4 D
$ a) T  i" e- N; _8 ^! r. x( B* U生成uuid
  s' a: ]! _" t  {
. {, `8 o0 Y4 Zuuidgen
> 9bf24809-220b-4910-b384-c1f06ea80728
' _6 M1 ^3 ^6 P. s" X" S1
2
创建Ceph配置文件

cat >> /etc/ceph/ceph.conf <<EOF
' i$ U/ o2 g$ F( I" q  y8 M[global]
7 e) L( b# M( U+ z) _. Ufsid = 9bf24809-220b-4910-b384-c1f06ea80728
mon_initial_members = ceph01,ceph02,ceph03
  d4 ?6 ~) z" T; H- \mon_host = 10.40.65.156,10.40.65.175,10.40.65.129
; |& F- l, {. i6 t; K4 Q$ Lpublic_network = 10.40.65.0/24
7 ~. U# Y  P) b1 I$ E2 rauth_cluster_required = cephx
/ {8 _2 B- Y; p" o% mauth_service_required = cephx
2 `5 X. u. ^# Z0 }% k2 P3 L& jauth_client_required = cephx
osd_journal_size = 1024
. K, B: v2 d- W" A; \+ B& Bosd_pool_default_size = 3
& ~# ~5 D- W& E1 z; w( ~: uosd_pool_default_min_size = 2
osd_pool_default_pg_num = 64
osd_pool_default_pgp_num = 64
/ Z9 L. Y& L8 J+ S' x8 Nosd_crush_chooseleaf_type = 1
EOF
. a3 |8 l) w, ?+ Q3 Q1
) ~% L9 h2 |6 t8 C; F2
3
* U7 C% t8 L. l( _4
( j6 L# C6 _$ r. @8 k* C# p3 F5
, B1 s1 D- f4 p. u2 ^' d6
7
* z5 W( ]" H; D! F" H8
9
  F3 E. h: w- ^' [! u& G10
% q8 o5 `/ u" x, u! N* x( a1 E11
; a3 d9 ~! J5 O; ?0 W; c" h12
13
& C! S" X# B6 p7 c6 n" J14
3 d+ P5 ^3 T4 C9 b8 Y15
! A& L! h) W: S( l16
, p% t5 y% |+ L/ T' U创建集群Monitor密钥。

6 U$ D; N% r  M) I$ Kceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
8 J2 a6 n0 v+ C1 [: K% Y9 r1
3 b- R$ p9 `1 A. ^& s# W创建client.admin用户、client.bootstrap-osd用户密钥，添加到集群密钥中。
8 s* H! x7 Y) l1 e9 c
# f- g/ N$ K% P7 q0 b! n6 k2 j: {$ m6 Wceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
' t6 k' i, \0 W( `ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
/ \- r( K$ W. A& ]1
2
ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
1
. B" a) G& x3 b9 F% g& Q) j; b2
/ a8 v, T6 K* I3 d使用主机名、主机IP地址、FSID生成monitor map。
) U  S$ s, @. j! x! F$ c5 B, v
4 S8 }+ r/ d* k0 V+ T- I( s" smonmaptool --create --add ceph01 10.40.65.156 --add ceph02 10.40.65.175 --add ceph03 10.40.65.129 --fsid 9bf24809-220b-4910-b384-c1f06ea80728 /tmp/monmap
1
初始化并启动monitor服务

% f, ^- Y' i( F' @& }2 O: f/ ~sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph01
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
) m' Y. R. N6 c5 P# M) f+ B. Ysudo -u ceph ceph-mon --mkfs -i ceph01 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
: F$ z5 _$ F; z: Y- j; ^; jls /var/lib/ceph/mon/ceph-ceph01/
1
/ A7 Y* ?: v1 ^2 }$ x3 j: ~2
3
4
$ `" p5 H) c& g! s1 G) @& Esystemctl start ceph-mon@ceph01
systemctl enable ceph-mon@ceph01
" x( a* a* p8 Asystemctl status ceph-mon@ceph01
& J0 k3 b% E+ Q6 t# O1
2
3
# S  l) N  P+ Z  |; s; P! Y$ S% b同步配置文件、密钥、monmap到其他节点中（Ceph 01执行）
) B( l: e* n: z& r
& w  F- s, z; M* }* g% a复制ceph.client.admin.keyring、client.bootstrap-osd key、ceph.mon.keyring、monitor map、ceph.conf到另外2个节点
% C6 l( J2 W" m& d
" |9 t/ V0 \* M, d; q- E# O) wscp /etc/ceph/ceph.client.admin.keyring root@ceph02:/etc/ceph/
1 v! S+ |1 o+ {" W9 f' R) K# |scp /etc/ceph/ceph.client.admin.keyring root@ceph03:/etc/ceph/
+ l' y1 s2 v! I1
  x& r7 M, M6 A% x* Y2
! r* q- w: T  C: g) Rscp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph02:/var/lib/ceph/bootstrap-osd/
scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph03:/var/lib/ceph/bootstrap-osd/
1
2
scp /tmp/ceph.mon.keyring root@ceph02:/tmp/
( u' w( @8 K3 b4 E4 z3 {scp /tmp/ceph.mon.keyring root@ceph03:/tmp/
- J) g) g" e2 f( H5 C1
2
scp /tmp/monmap root@ceph02:/tmp/
scp /tmp/monmap root@ceph03:/tmp/
1
1 R' ]) ]4 m- w6 ~" q5 W* \9 ~# N2
" [- c# V' j0 gscp /etc/ceph/ceph.conf root@ceph02:/etc/ceph/
scp /etc/ceph/ceph.conf root@ceph03:/etc/ceph/
1
2
( h1 q% g' F8 i, t" t2 z启动其他节点的monitor服务（Ceph 02执行）

sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph02
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
/ [% e+ H# y( Isudo -u ceph ceph-mon --mkfs -i ceph02 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
3 ^9 |1 G! f2 x1 G# k8 z! }# als /var/lib/ceph/mon/ceph-ceph02/
1
2
% v, ~6 }' b" r1 s& f; W: b3
# z; |6 H, g* Q4
systemctl start ceph-mon@ceph02
systemctl enable ceph-mon@ceph02
7 x% O; s$ |, {! h; L! E7 `* q6 tsystemctl status ceph-mon@ceph02
1 {3 A7 ?, k' _1 ?; m# a1
2
3
" }0 A# v5 b. P2 m0 s启动其他节点的monitor服务（Ceph 03执行）
' h+ o2 H, v9 @8 M2 V0 p
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph03
3 q+ W. Z; o& b1 Mchown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
sudo -u ceph ceph-mon --mkfs -i ceph03 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
ls /var/lib/ceph/mon/ceph-ceph03/
# N) o& `0 b* H) F+ Z1
2
3
4
7 ?' M" _8 P; l' i+ `, }2 \5 rsystemctl start ceph-mon@ceph03
systemctl enable ceph-mon@ceph03
systemctl status ceph-mon@ceph03
& j7 `3 F7 Z0 ^/ N1
2
3
+ k; b; Z' `) f) p8 C1 `查看当前集群状态（任意节点执行）
通过ceph -s命令查询集群状态，可见集群services中3个mon服务已启动。
4 N# O) a6 \% o; ~
' w! Y6 m% K8 |# v& v  `ceph -s
> cluster:
9 n$ L' _& d( C>                 id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a        
>                 health: HEALTH_OK
>
; e6 o( h1 T3 I! z' b* l9 q> services:        
>                 mon:         3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)        
) O8 n, j6 L* h$ t. w>                 mgr:                               
>                 osd:
- ~4 z  Z5 g+ r1 m4 B>
> data:
* Z$ c* p' v: e6 k# u2 {* J# a9 Z>                  pools:   
1 M+ {( f- E% B8 d/ v9 D& ~* i. v>                 objects:
>                 usage:   
$ P7 q7 s: @$ k: V0 y( S>                 pgs:
1
2
$ }4 m5 P- c/ r$ {) ~; {8 ~3
4
+ t7 x9 u8 W+ w% e5
: D' ?; y$ `. R9 w% J. a6
& m5 Y$ D# i2 a$ U! U! F1 z+ W7
8
1 s8 y6 j) W' t1 Z  p: U9
10
11
$ g: l/ X1 x1 {% X$ C' F- V12
13
" T; l. c& O9 Q! R3 e/ [! F14
1 F6 |2 k3 F  |# x15
* l4 v) t1 _# s" }: V7 r部署Ceph mon服务（ceph-volume 自动化创建）
安装Ceph-osd服务程序（所有设备执行）
6 F, Q2 }& O5 \0 J; oyum install -y ceph-osd
1
: d" {0 m. l& D, T$ z初始化osd服务（所有设备执行）
通过fdisk等工具查看磁盘盘符，然后利用ceph-volume工具自动化创建osd服务。
$ f9 X) O' c, \" K: Kceph-volume lvm create --data /dev/sda
ceph-volume lvm create --data /dev/sdb
* R4 N5 ]0 K7 D3 Eceph-volume lvm create --data /dev/sdc
1
2
3
查看当前集群状态（任意节点执行）
' ?0 n; g* R, J- {' f通过ceph osd tree命令查询集群状态，可见集群services中所有osd服务已启动。
ceph osd tree
> ID CLASS WEIGHT   TYPE NAME       STATUS REWEIGHT PRI-AFF
+ b" x' F- A1 D$ F> -1       16.36908 root default
2 E# u8 W- H2 `# s! \8 J; ?> -3        5.45636     host ceph01
>  0   hdd  1.81879         osd.0       up  1.00000 1.00000
>  1   hdd  1.81879         osd.1       up  1.00000 1.00000
* U# a9 v) C% z9 P>  2   hdd  1.81879         osd.2       up  1.00000 1.00000
> -5        5.45636     host ceph02
>  3   hdd  1.81879         osd.3       up  1.00000 1.00000
& L* a$ Z5 A, n5 f( B% S>  4   hdd  1.81879         osd.4       up  1.00000 1.00000
- K% U5 Q, U6 a* e% I( T* c>  5   hdd  1.81879         osd.5       up  1.00000 1.00000
0 Q  [: d' t; S> -7        5.45636     host ceph03
>  6   hdd  1.81879         osd.6       up  1.00000 1.00000
>  7   hdd  1.81879         osd.7       up  1.00000 1.00000
>  8   hdd  1.81879         osd.8       up  1.00000 1.00000
1
( F2 L$ N" W, a  @6 Y2
% y$ i2 E6 K/ ~/ Y2 K3
4
4 E1 w4 O7 T. A, m5
  E. L5 Q. ], Y! m" Y6
7
8
9
10
11
" [2 ]1 [3 ^' ~2 |! ^% j12
13
7 ~# U' R/ t/ b5 v/ g14
) c- A/ Z" q, @, I) t4 a15
+ J/ p9 ~  r# A# z& c6 ~' K9 T( W% s部署Ceph mgr服务并开启Dashboard
安装Ceph-mgr服务程序（所有设备执行）
yum install -y ceph-mgr
1
初始化并启动主MGR服务（Ceph01执行）
7 w6 R0 i+ y; cmkdir -p /var/lib/ceph/mgr/ceph-ceph01
chown ceph.ceph -R /var/lib/ceph
* t" Z4 Y! Q& Yceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph01.keyring --gen-key -n mgr.ceph01 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
ceph auth import -i /etc/ceph/ceph.mgr.ceph01.keyring
ceph auth get-or-create mgr.ceph01 -o /var/lib/ceph/mgr/ceph-ceph01/keyring
4 r& q. ], f! g; S# }2 d2 L: ?1
2
3
' ~0 i# r3 f# D& i4
# B, s& s4 {+ ^" K: T/ v6 g, W5
systemctl start ceph-mgr@ceph01
+ B; ^9 r1 D. C* p: F8 ^; x# F4 dsystemctl enable ceph-mgr@ceph01
systemctl status ceph-mgr@ceph01
( B/ D/ C$ u8 h; B" E1
2
3
1 e% [8 y/ \: Y" s: }0 Z初始化并启动从MGR服务（Ceph02执行）
mkdir -p /var/lib/ceph/mgr/ceph-ceph02
! I& K0 m* v- S; W6 w7 T) wchown ceph.ceph -R /var/lib/ceph
ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph02.keyring --gen-key -n mgr.ceph02 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
ceph auth import -i /etc/ceph/ceph.mgr.ceph02.keyring
# ?6 W9 O2 |8 \& W  eceph auth get-or-create mgr.ceph02 -o /var/lib/ceph/mgr/ceph-ceph02/keyring
1
: B' Z9 a9 s* r5 G; v/ S- c/ o7 F2
' }8 t! U: X* z3
4
5
4 Y3 s+ g# q. P3 W) W% F$ i1 Psystemctl start ceph-mgr@ceph02
systemctl enable ceph-mgr@ceph02
6 F# U9 L" }! y9 i/ y4 tsystemctl status ceph-mgr@ceph02
9 e) C' m) P" Z# f1 }1
2
3
+ Z/ A. v& L6 a; b6 \! \; J! W' f初始化并启动从MGR服务（Ceph03执行）
mkdir -p /var/lib/ceph/mgr/ceph-ceph03
chown ceph.ceph -R /var/lib/ceph
ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph03.keyring --gen-key -n mgr.ceph03 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
, v! u  L/ @! e" ?! ]! O$ aceph auth import -i /etc/ceph/ceph.mgr.ceph03.keyring
ceph auth get-or-create mgr.ceph03 -o /var/lib/ceph/mgr/ceph-ceph03/keyring
1
2
+ v# _4 l$ ]( l4 x3
9 ^: D$ }4 I2 y5 n6 U- l6 ?4
5
/ {0 {% t3 m1 y4 ]systemctl start ceph-mgr@ceph03
systemctl enable ceph-mgr@ceph03
systemctl status ceph-mgr@ceph03
1
5 ^* n8 B1 }$ z' R& B+ r; @2
9 s+ Q! G! ^; `% }* y& f3 T3
查看当前集群状态（任意节点执行）
通过ceph -s命令查询集群状态，可见集群services中3个mgr服务已启动。
* ?9 ^' n5 @- o+ y( `ceph -s
- r8 n3 P4 K$ O/ f! ^> cluster:
6 M( s3 q! L0 ^/ U# S>                 id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a        
>                 health: HEALTH_OK
>
! ^- A* n3 ]- I6 r> services:        
>                 mon:         3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)        
>                 mgr:        ceph01(active, since 3w), standbys: ceph01、ceph02                       
: w1 B& R4 l( T! h0 c* D6 C! t$ g>                 osd:
>
' Z. s' S( H4 z* p6 h8 X1 _> data:
>                  pools:   
>                 objects:
0 t7 p0 t$ a8 `- h' ]* {>                 usage:   
>                 pgs:
1
2
+ B) y- i5 v( [$ ~# f  J! S4 r3
4
5
6
7
8 [( `( j3 N  ^6 h8
3 R9 {5 @; Z1 J6 n( q; T- D8 L9
, x$ F% q" U( M$ [10
11
12
5 Y2 ?/ }& I: r, f$ ?  X13
14
' O* f/ Y9 C$ \! M15
使能Dashboard访问功能（任意节点执行）
5 j9 ]) V: d3 T  ^- ]开启mgr dashboard功能
" I- w% ~2 f( s; J4 o- c! nceph mgr module enable dashboard
1
生成并安装自签名的证书
ceph dashboard create-self-signed-cert
. S5 i# z3 B. e2 w% D2 d& L1
配置dashboard
' T7 _. T. q) J, K/ O7 v% U! yceph config set mgr mgr/dashboard/server_addr 10.40.65.148
6 M, d4 [) P) X+ C- o5 r1 B. V& }ceph config set mgr mgr/dashboard/server_port 8080
* {: Q! _- l! \8 }( I* N9 V% \ceph config set mgr mgr/dashboard/ssl_server_port 8443
1
2
3
3 m2 c1 L: v/ I1 g5 b创建一个dashboard登录用户名密码
/ ~+ B0 K3 w' i$ \* C; I5 i: Jecho '123456' > password.txt
. Q" i2 d& F1 G6 m8 ?0 L, `6 u8 gceph dashboard ac-user-create admin  administrator -i password.txt
& A; I9 P5 z  m( O! q# I% N' D  t5 V1
! a* ~, [+ J- N; b2
0 k. Z" h$ a/ v& c查看服务访问方式
ceph mgr services
1
8 f" A3 Q2 T3 u3 q- x8 p通过web访问Ceph Dashboard，用户名密码为admin/123456
+ f: n1 t  n. Z  P- y$ vhttps://10.40.65.148:8443