找回密码
 注册
查看: 843|回复: 2

手动方式部署ceph集群以及添加osd

[复制链接]

0

主题

0

回帖

9

积分

管理员

积分
9
QQ
发表于 2022-7-19 11:20:00 | 显示全部楼层 |阅读模式
一、准备
前一篇点击打开链接只部署了一个单机集群。在这一篇里,手动部署一个多机集群:mycluster。我们有三台机器nod1,node2和node3;其中node1可以免密ssh/scp任意其他两台机器。我们的所有工作都在node1上完成。
准备工作包括在各个机器上安装ceph rpm包(见前一篇第1节点击打开链接),并在各个机器上修改下列文件:
/usr/lib/systemd/system/ceph-mon@.service /usr/lib/systemd/system/ceph-osd@.service /usr/lib/systemd/system/ceph-mds@.service /usr/lib/systemd/system/ceph-mgr@.service /usr/lib/systemd/system/ceph-radosgw@.service
修改:
Environment=CLUSTER=ceph <--- 改成CLUSTER=mycluster ExecStart=/usr/bin/... --id %i --setuser ceph --setgroup ceph <--- 删掉--setuser ceph --setgroup ceph二、创建工作目录
在node1创建一个工作目录,后续所有工作都在node1上的这个工作目录中完成;
mkdir /tmp/mk-ceph-cluster cd /tmp/mk-ceph-cluster三、创建配置文件vim mycluster.conf [global] cluster = mycluster fsid = 116d4de8-fd14-491f-811f-c1bdd8fac141 public network = 192.168.100.0/24 cluster network = 192.168.73.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 128 osd pool default pgp num = 128 osd pool default crush rule = 0 osd crush chooseleaf type = 1 admin socket = /var/run/ceph/$cluster-$name.asock pid file = /var/run/ceph/$cluster-$name.pid log file = /var/log/ceph/$cluster-$name.log log to syslog = false max open files = 131072 ms bind ipv6 = false [mon] mon initial members = node1,node2,node3 mon host = 192.168.100.131:6789,192.168.100.132:6789,192.168.100.133:6789 ;Yuanguo: the default value of {mon data} is /var/lib/ceph/mon/$cluster-$id, ; we overwrite it. mon data = /var/lib/ceph/mon/$cluster-$name mon clock drift allowed = 10 mon clock drift warn backoff = 30 mon osd full ratio = .95 mon osd nearfull ratio = .85 mon osd down out interval = 600 mon osd report timeout = 300 debug ms = 20 debug mon = 20 debug paxos = 20 debug auth = 20 [mon.node1] host = node1 mon addr = 192.168.100.131:6789 [mon.node2] host = node2 mon addr = 192.168.100.132:6789 [mon.node3] host = node3 mon addr = 192.168.100.133:6789 [mgr] ;Yuanguo: the default value of {mgr data} is /var/lib/ceph/mgr/$cluster-$id, ; we overwrite it. mgr data = /var/lib/ceph/mgr/$cluster-$name [osd] ;Yuanguo: we wish to overwrite {osd data}, but it seems that 'ceph-disk' forces ; to use the default value, so keep the default now; maybe in later versions ; of ceph the limitation will be eliminated. osd data = /var/lib/ceph/osd/$cluster-$id osd recovery max active = 3 osd max backfills = 5 osd max scrubs = 2 osd mkfs type = xfs osd mkfs options xfs = -f -i size=1024 osd mount options xfs = rw,noatime,inode64,logbsize=256k,delaylog filestore max sync interval = 5 osd op threads = 2 debug ms = 100 debug osd = 100
需要说明的是,在这个配置文件中,我们覆盖了一些默认值,比如:{mon data}和{mgr data},但是没有覆盖{osd data},因为ceph-disk貌似强制使用默认值。另外,pid, sock文件被放置在/var/run/ceph/中,以$cluster-$name命名;log文件放置在/var/log/ceph/中,也是以$cluster-$name命名。这些都可以覆盖。
四、生成keyring
在单机部署中点击打开链接,我们说过,有两种操作集群中user及其权限的方式,这里我们使用第一种:先生成keyring文件,然后在创建集群时带入使之生效。
ceph-authtool --create-keyring mycluster.keyring --gen-key -n mon. --cap mon 'allow *' 6 R( h  s/ S6 P4 D' f* O# z9 g+ L
ceph-authtool --create-keyring mycluster.client.admin.keyring --gen-key -n client.admin --set-uallow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
4 K) G, o6 r, ]! l0 X; {; cceph-authtool --create-keyring mycluster.client.bootstrap-osd.keyring --gen-key -n client.bootstrap-osd --cap mon 'allow profile bootstrap-osd'
8 _( v- m' l( h4 d. Oceph-authtool --create-keyring mycluster.mgr.node1.keyring --gen-key -n mgr.node1 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
2 o, R* @0 b0 s) Y6 S7 N  F; sceph-authtool --create-keyring mycluster.mgr.node2.keyring --gen-key -n mgr.node2 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
8 k2 f4 W( G+ n2 y4 Eceph-authtool --create-keyring mycluster.mgr.node3.keyring --gen-key -n mgr.node3 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
# S9 D3 G% z0 y/ D4 Q. Uceph-authtool mycluster.keyring --import-keyring mycluster.client.admin.keyring
0 E; b, V8 I9 oceph-authtool mycluster.keyring --import-keyring mycluster.client.bootstrap-osd.keyring
- @' ?# X7 P# {/ G5 |4 U2 t' g  m2 eceph-authtool mycluster.keyring --import-keyring mycluster.mgr.node1.keyring 8 ]/ C1 F* o, X* A* z5 E& f
ceph-authtool mycluster.keyring --import-keyring mycluster.mgr.node2.keyring
. ^. s5 h: g2 k8 x" [- J& e& \ceph-authtool mycluster.keyring --import-keyring mycluster.mgr.node3.keyring
7 T6 p5 z1 X: Y: S4 F7 _5 L7 C7 @cat mycluster.keyring [mon.] key = AQA525NZsY73ERAAIM1J6wSxglBNma3XAdEcVg== caps mon = "allow *" [client.admin] key = AQBJ25NZznIpEBAAlCdCy+OyUIvxtNq+1DSLqg== auid = 0 caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [client.bootstrap-osd] key = AQBW25NZtl/RBxAACGWafYy1gPWEmx9geCLi6w== caps mon = "allow profile bootstrap-osd" [mgr.node1] key = AQBb25NZ1mIeFhAA/PmRHFY6OgnAMXL1/8pSxw== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *" [mgr.node2] key = AQBg25NZJ6jyHxAAf2GfBAG5tuNwf9YjkhhEWA== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *" [mgr.node3] key = AQBl25NZ7h6CJRAAaFiea7hiTrQNVoZysA7n/g== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *"五、生成monmap
生成monmap并添加3个monitor
monmaptool --create --add node1 192.168.100.131:6789 --add node2 192.168.100.132:6789 --add node3 192.168.100.133:6789 --fsid 116d4de8-fd14-491f-811f-c1bdd8fac141 , f* L2 O! m, E; s0 Q8 y. j: r
monmap [plain] view plain copy monmaptool --print monmap monmaptool: monmap file monmap epoch 0 fsid 116d4de8-fd14-491f-811f-c1bdd8fac141 last_changed 2017-08-16 05:45:37.851899 created 2017-08-16 05:45:37.851899 0: 192.168.100.131:6789/0 mon.node1 1: 192.168.100.132:6789/0 mon.node2 2: 192.168.100.133:6789/0 mon.node3六、分发配置文件
keyring和monmap
把第2、3和4步中生成的配置文件,keyring,monmap分发到各个机器。由于mycluster.mgr.nodeX.keyring暂时使用不到,先不分发它们(见第8节)。
cp mycluster.client.admin.keyring mycluster.client.bootstrap-osd.keyring mycluster.keyring mycluster.conf monmap /etc/ceph scp mycluster.client.admin.keyring mycluster.client.bootstrap-osd.keyring mycluster.keyring mycluster.conf monmap node2:/etc/ceph scp mycluster.client.admin.keyring mycluster.client.bootstrap-osd.keyring mycluster.keyring mycluster.conf monmap node3:/etc/ceph七、创建集群1、创建{mon data}目录mkdir /var/lib/ceph/mon/mycluster-mon.node1
, d4 q. w7 I2 ~2 l9 R+ @0 gssh node2 mkdir /var/lib/ceph/mon/mycluster-mon.node2
# e, a+ Z: ~3 F7 L0 p! q! Issh node3 mkdir /var/lib/ceph/mon/mycluster-mon.node3
注意,在配置文件mycluster.conf中,我们把{mon data}设置为/var/lib/ceph/mon/$cluster-$name,而不是默认的/var/lib/ceph/mon/$cluster-$id;
* O* p: x. H  X3 T% J$cluster-$name展开为mycluster-mon.node1(23);% P6 d  G" k7 u1 v+ J
默认的$cluster-$id展开为mycluster-node1(23);
2、初始化monitorceph-mon --cluster mycluster --mkfs -i node1 --monmap /etc/ceph/monmap --keyring /etc/ceph/mycluster.keyring
& k, E2 s4 Z  r% Rssh node2 ceph-mon --cluster mycluster --mkfs -i node2 --monmap /etc/ceph/monmap --keyring /etc/ceph/mycluster.keyring
" H6 m4 o+ ^; z$ dssh node3 ceph-mon --cluster mycluster --mkfs -i node3 --monmap /etc/ceph/monmap --keyring /etc/ceph/mycluster.keyring
  q9 U+ c  Y, S: j; Q( @0 W
注意,在配置文件mycluster.conf,我们把{mon data}设置为/var/lib/ceph/mon/$cluster-$name,展开为/var/lib/ceph/mon/mycluster-mon.node1(23)。ceph-mon会5 q3 ?6 X6 \8 u4 y0 d4 ]2 l
根据–cluster mycluster找到配置文件mycluster.conf,并解析出{mon data},然后在那个目录下进行初始化。
3、touch donetouch /var/lib/ceph/mon/mycluster-mon.node1/done # B  L& P- L/ U  O  ^
ssh node2 touch /var/lib/ceph/mon/mycluster-mon.node2/done 0 u9 L) ]4 W1 n5 b/ S
ssh node3 touch /var/lib/ceph/mon/mycluster-mon.node3/done4、启动monitorssystemctl start ceph-mon@node1
8 P; n- c. L& J  H ssh node2 systemctl start [url=mailto:ceph-mon@node2]ceph-mon@node2
6 u& a/ [! s+ o3 g
ssh node3 systemctl start ceph-mon@node3[/url]
: K( t' d" H5 k5、检查机器状态ceph --cluster mycluster -s
9 W  v* B* o) Y2 ?7 J) K3 ycluster: id: 116d4de8-fd14-491f-811f-c1bdd8fac141 health: HEALTH_OK services: mon: 3 daemons, quorum node1,node2,node3 mgr: no daemons active osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 0 kB used, 0 kB / 0 kB avail pgs:八、添加osd
每台集群都有一个/dev/sdb,我们把它们作为osd。
1、删除它们的分区2、prepareceph-disk prepare --cluster mycluster --cluster-uuid 116d4de8-fd14-491f-811f-c1bdd8fac141 --bluestore --block.db /dev/sdb --block.wal /dev/sdb /dev/sdb ssh node2 ceph-disk prepare --cluster mycluster --cluster-uuid 116d4de8-fd14-491f-811f-c1bdd8fac141 --bluestore --block.db /dev/sdb --block.wal /dev/sdb /dev/sdb ssh node3 ceph-disk prepare --cluster mycluster --cluster-uuid 116d4de8-fd14-491f-811f-c1bdd8fac141 /dev/sdb 注意:prepare node3:/dev/sdb时,我们没有加选项:--bluestore --block.db /dev/sdb --block.wal /dev/sdb;后面我们会看它和其他两个有什么不同。3、activateceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring ssh node2 ceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring ssh node3 ceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring
注意:ceph-disk好像有两个问题:
  • 前面说过,它不使用自定义的{osd data},而强制使用默认值 /var/lib/ceph/osd/$cluster-$id
    # R4 i: U/ k  F1 y: V
  • 好像不能为一个磁盘指定osd id,而只能依赖它自动生成。虽然ceph-disk prepare有一个选项–osd-id,但是ceph-disk activate并不使用它而是自己生成。当不匹配时,会出现 如下错误:
    8 q4 \( s- H1 M. m- W
# ceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring command_with_stdin: Error EEXIST: entity osd.0 exists but key does not match mount_activate: Failed to activate '['ceph', '--cluster', 'mycluster', '--name', 'client.bootstrap-osd', '--keyring', '/etc/ceph/mycluster.client.bootstrap-osd.keyring', '-i', '-', 'osd', 'new', u'ca8aac6a-b442-4b07-8fa6-62ac93b7cd29']' failed with status code 17
从 ‘-i’, ‘-‘可以看出,它只能自动生成osd id;
4、检查osd
在ceph-disk prepare时,node1:/dev/sdb和node2:/dev/sdb一样,都有–bluestore –block.db /dev/sdb –block.wal选项;node3:/dev/sdb不同,没有加这些选项。我们看看有什么不同。
4.1 node1
mount | grep sdb /dev/sdb1 on /var/lib/ceph/osd/mycluster-0 type xfs (rw,noatime,seclabel,attr2,inode64,noquota) ls /var/lib/ceph/osd/mycluster-0/ activate.monmap block block.db_uuid block.wal bluefs fsid kv_backend mkfs_done systemd whoami active block.db block_uuid block.wal_uuid ceph_fsid keyring magic ready type ls -l /var/lib/ceph/osd/mycluster-0/block lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:52 /var/lib/ceph/osd/mycluster-0/block -> /dev/disk/by-partuuid/a12dd642-b64c-4fef-b9e6-0b45cff40fa9 ls -l /dev/disk/by-partuuid/a12dd642-b64c-4fef-b9e6-0b45cff40fa9 lrwxrwxrwx. 1 root root 10 Aug 16 05:55 /dev/disk/by-partuuid/a12dd642-b64c-4fef-b9e6-0b45cff40fa9 -> ../../sdb2 blkid /dev/sdb2 /dev/sdb2: PARTLABEL="ceph block" PARTUU cat /var/lib/ceph/osd/mycluster-0/block_uuid a12dd642-b64c-4fef-b9e6-0b45cff40fa9 ls -l /var/lib/ceph/osd/mycluster-0/block.db lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:52 /var/lib/ceph/osd/mycluster-0/block.db -> /dev/disk/by-partuuid/1c107775-45e6-4b79-8a2f-1592f5cb03f2 ls -l /dev/disk/by-partuuid/1c107775-45e6-4b79-8a2f-1592f5cb03f2 lrwxrwxrwx. 1 root root 10 Aug 16 05:55 /dev/disk/by-partuuid/1c107775-45e6-4b79-8a2f-1592f5cb03f2 -> ../../sdb3 blkid /dev/sdb3 /dev/sdb3: PARTLABEL="ceph block.db" PARTUU cat /var/lib/ceph/osd/mycluster-0/block.db_uuid 1c107775-45e6-4b79-8a2f-1592f5cb03f2 ls -l /var/lib/ceph/osd/mycluster-0/block.wal lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:52 /var/lib/ceph/osd/mycluster-0/block.wal -> /dev/disk/by-partuuid/76055101-b892-4da9-b80a-c1920f24183f ls -l /dev/disk/by-partuuid/76055101-b892-4da9-b80a-c1920f24183f lrwxrwxrwx. 1 root root 10 Aug 16 05:55 /dev/disk/by-partuuid/76055101-b892-4da9-b80a-c1920f24183f -> ../../sdb4 blkid /dev/sdb4 /dev/sdb4: PARTLABEL="ceph block.wal" PARTUU cat /var/lib/ceph/osd/mycluster-0/block.wal_uuid 76055101-b892-4da9-b80a-c1920f24183f
可见,node1(node2)上,/dev/sdb被分为4个分区:
  • /dev/sdb1: metadata
  • /dev/sdb2:the main block device
  • /dev/sdb3: db
  • /dev/sdb4: wal: P' Z, |3 U$ S) U4 ~' i# B, V
具体见:ceph-disk prepare –help
4.2 node3
mount | grep sdb /dev/sdb1 on /var/lib/ceph/osd/mycluster-2 type xfs (rw,noatime,seclabel,attr2,inode64,noquota) ls /var/lib/ceph/osd/mycluster-2 activate.monmap active block block_uuid bluefs ceph_fsid fsid keyring kv_backend magic mkfs_done ready systemd type whoami ls -l /var/lib/ceph/osd/mycluster-2/block lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:54 /var/lib/ceph/osd/mycluster-2/block -> /dev/disk/by-partuuid/0a70b661-43f5-4562-83e0-cbe6bdbd31fb ls -l /dev/disk/by-partuuid/0a70b661-43f5-4562-83e0-cbe6bdbd31fb lrwxrwxrwx. 1 root root 10 Aug 16 05:56 /dev/disk/by-partuuid/0a70b661-43f5-4562-83e0-cbe6bdbd31fb -> ../../sdb2 blkid /dev/sdb2 /dev/sdb2: PARTLABEL="ceph block" PARTUU cat /var/lib/ceph/osd/mycluster-2/block_uuid 0a70b661-43f5-4562-83e0-cbe6bdbd31fb
可见,在node3上,/dev/sdb被分为2个分区:
  • /dev/sdb1:metadata
  • /dev/sdb2:the main block device;db和wal也在这个分区上。
    0 h2 Z2 i# l0 M) p) ]
具体见:ceph-disk prepare –help
5、检查集群状态ceph --cluster mycluster -s cluster: id: 116d4de8-fd14-491f-811f-c1bdd8fac141 health: HEALTH_WARN no active mgr services: mon: 3 daemons, quorum node1,node2,node3 mgr: no daemons active osd: 3 osds: 3 up, 3 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 0 kB used, 0 kB / 0 kB avail pgs:
由于没有添加mgr,集群处于WARN状态。
九、添加mgr1、创建{mgr data}目录mkdir /var/lib/ceph/mgr/mycluster-mgr.node1 ssh node2 mkdir /var/lib/ceph/mgr/mycluster-mgr.node2 ssh node3 mkdir /var/lib/ceph/mgr/mycluster-mgr.node3
注意,和{mon data}类似,在配置文件mycluster.conf中,我们把{mgr data}设置为/var/lib/ceph/mgr/$cluster-$name,而不是默认的/var/lib/ceph/mgr/$cluster-$id。
2、分发mgr的keyringcp mycluster.mgr.node1.keyring /var/lib/ceph/mgr/mycluster-mgr.node1/keyring scp mycluster.mgr.node2.keyring node2:/var/lib/ceph/mgr/mycluster-mgr.node2/keyring scp mycluster.mgr.node3.keyring node3:/var/lib/ceph/mgr/mycluster-mgr.node3/keyring3、启动mgrsystemctl start ceph-mgr@node1 ssh node2 systemctl start ceph-mgr@node2 ssh node3 systemctl start ceph-mgr@node34、检查集群状态ceph --cluster mycluster -s cluster: id: 116d4de8-fd14-491f-811f-c1bdd8fac141 health: HEALTH_OK services: mon: 3 daemons, quorum node1,node2,node3 mgr: node1(active), standbys: node3, node2 osd: 3 osds: 3 up, 3 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 5158 MB used, 113 GB / 118 GB avail pgs:
可见,添加mgr之后,集群处于OK状态。

0

主题

0

回帖

9

积分

管理员

积分
9
QQ
 楼主| 发表于 2022-7-20 13:41:38 | 显示全部楼层
部署Ceph mon服务: O3 i- I+ \* C, w. M+ ]
安装Ceph-mon服务程序(所有设备执行)
) }* ~3 _" C- kyum install -y ceph-mon
# g* z3 x( c% E# ?/ F! K3 `' N1
0 p: g! r4 k8 {" \. h  i初始化Mon服务(Ceph 01执行)
# d' f; h" \6 p* k生成uuid' n* T0 E6 c8 z$ a
uuidgen
5 S/ G6 z6 S& m9 x> 9bf24809-220b-4910-b384-c1f06ea80728
* w, ~- j; r) `* a  D4 b5 V1
* k: f' Q  H: u, P, l; b2
; Z4 b3 J7 h/ {# H; K创建Ceph配置文件
4 |  ~9 D1 y; e: T0 J2 M# L8 Dcat >> /etc/ceph/ceph.conf <<EOF
8 f; b2 w5 k& d' p) E[global]
7 t( f) s" T# T5 }fsid = 9bf24809-220b-4910-b384-c1f06ea80728% U" {2 r+ I2 }! F& t, e
mon_initial_members = ceph01,ceph02,ceph03
5 ^6 e' l2 g! a$ h6 n* k/ x. ]mon_host = 10.40.65.156,10.40.65.175,10.40.65.1298 P, [, p3 _- g3 X; j
public_network = 10.40.65.0/24
+ q; v( D4 ~! a" O+ bauth_cluster_required = cephx
: Q7 x6 p* y+ b5 b1 i, X7 kauth_service_required = cephx/ @( Y" U8 K; h9 Q: d
auth_client_required = cephx
- a; s, C! U: J& Yosd_journal_size = 1024
9 Q& H- K; W/ m, F6 P$ Rosd_pool_default_size = 3; o5 x& e' L- n
osd_pool_default_min_size = 2
( @& N) G+ E' p) T; a7 cosd_pool_default_pg_num = 64
" E3 V$ q. q# [! tosd_pool_default_pgp_num = 64
; n  f+ d' X5 f) K' t6 Yosd_crush_chooseleaf_type = 1
5 I' e: p/ M' l7 g% U0 v/ A. fEOF
/ i: X- ]2 D: P0 I1+ A( L4 H: A9 }& R9 Y0 M6 ~
2) w, K. d+ G2 a. r0 C
38 U# B/ Y7 v# c4 j! K3 X
46 P- D7 \- p6 i
5
3 D( U. f5 T9 x, G3 o6
, Y8 M# b, t- G1 F4 x75 h- X4 v3 k3 R" m
8! ]) \1 N& d+ B  |/ M5 B
9+ X! A' s- E, C: z6 o/ q* Q, H( H* H/ U
10
& M' r) N) l5 K  H/ C11
, P5 z0 P: x; U  c' Y2 U# r12
3 F! y- R. ~; t13$ `1 @( A0 m, c# i
14
0 d# w* i' {) c3 }$ ]15# M% F6 v5 h( D5 U% ~
164 T% x( I$ q0 c5 R2 A) A- d
创建集群Monitor密钥。
- q0 R; R" }0 d/ F0 [: i8 N9 B* J/ [ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *', l+ X& i+ F! e3 F
11 G7 p0 ]: x3 V  M0 A. {
创建client.admin用户、client.bootstrap-osd用户密钥,添加到集群密钥中。$ x5 w) d0 A! C: v. b
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'7 r# V% `& X3 q" F6 y5 H& `
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
2 r+ N  ?. \. d" g; v+ D7 n1! h" W& K: A1 N5 T7 ?7 n$ K
2$ ^; b! c3 _8 d, h
ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'7 p$ \8 v1 c# \8 I/ W
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
$ Q2 X9 e0 F6 c0 L8 Q. L3 [; ~, O1( @+ I2 w9 v" |1 ^4 C. }* y" h, ^, ~
2
' F6 q" A9 @$ @. E: D* B使用主机名、主机IP地址、FSID生成monitor map。; n; Z. W' c5 G% k
monmaptool --create --add ceph01 10.40.65.156 --add ceph02 10.40.65.175 --add ceph03 10.40.65.129 --fsid 9bf24809-220b-4910-b384-c1f06ea80728 /tmp/monmap/ v/ k! n8 X7 s5 L
1
) ^; o+ H# w1 ^/ X初始化并启动monitor服务6 t$ I  E( b: S
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph01
" G& T, j& X  Z1 Fchown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
2 {9 T' r7 P) D! h2 z. Osudo -u ceph ceph-mon --mkfs -i ceph01 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring! A3 ]' d6 ?& g7 Z, q1 L) f8 h/ u
ls /var/lib/ceph/mon/ceph-ceph01/: b. _$ x( L0 v- I$ v5 q
1
. @9 n$ S( a7 C! p2: `( X' W  P) e" X. A& i
3& e& z2 H" h- F. C5 w4 M
4) L% C* B, a% q9 x) K* v
systemctl start ceph-mon@ceph01
) X( n9 a/ ~) Z2 h( Y  V! Ksystemctl enable ceph-mon@ceph01
1 H6 e1 J  n  q# M! Ysystemctl status ceph-mon@ceph01
5 y7 i: Q' D  k* X1
# z/ k; A+ j3 i$ f/ `6 Z2
) G, L# T( d+ v6 J7 g- _3
7 p+ T* d& |/ C, y  V* e同步配置文件、密钥、monmap到其他节点中(Ceph 01执行)7 T1 E0 J! N& k  R) C5 t; Y
复制ceph.client.admin.keyring、client.bootstrap-osd key、ceph.mon.keyring、monitor map、ceph.conf到另外2个节点
$ G# S# g! Y8 {. P7 O' sscp /etc/ceph/ceph.client.admin.keyring root@ceph02:/etc/ceph/3 r) g1 _7 a1 L; o8 W2 k6 j! S
scp /etc/ceph/ceph.client.admin.keyring root@ceph03:/etc/ceph/
* u* c8 y- W6 Y1 J+ N' a1
1 O) \, w8 z: z/ l2
# r  n. y; v/ D' ascp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph02:/var/lib/ceph/bootstrap-osd/3 c! Q/ q& k( ^
scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph03:/var/lib/ceph/bootstrap-osd/4 F8 W' _: s7 K4 g
1  ~4 ]# c3 F9 G1 L: m2 ?9 Y! `
2
2 S7 X  L/ I: n$ j* Z% ^# [scp /tmp/ceph.mon.keyring root@ceph02:/tmp/
: N  C% o* V+ S' L) Q1 \8 Iscp /tmp/ceph.mon.keyring root@ceph03:/tmp/
% S# S  c( i: f- s1
# {4 ]* a  t" L- J- l) ?26 j$ n; p8 E2 o3 K2 s
scp /tmp/monmap root@ceph02:/tmp/8 j/ \" `4 s7 x# U, V/ j; I" e
scp /tmp/monmap root@ceph03:/tmp/
- k2 D' O8 c% z2 }3 ]- |* k13 R, I9 g( B- B5 g3 y7 |0 ~% q
2
: `7 S! w% n8 ^$ L: K& ~scp /etc/ceph/ceph.conf root@ceph02:/etc/ceph/% n" A3 w: G, n5 q
scp /etc/ceph/ceph.conf root@ceph03:/etc/ceph/
4 }% ?2 e! \- K& m1
' `# H6 t; S) H9 c, F% r" k/ E! H21 c7 x3 m/ G* Q! {) D
启动其他节点的monitor服务(Ceph 02执行), u. Z& q1 M& G: l( B& m
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph02+ Y3 z' d/ X% M
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap) Q! Q& C1 E- t. T- u9 U8 ]2 G
sudo -u ceph ceph-mon --mkfs -i ceph02 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
( _. K/ _: y( W3 Rls /var/lib/ceph/mon/ceph-ceph02/. Z& j( _* [+ l0 v. d
1
! U/ F. K) T! P7 }2# c2 p$ Y/ Z0 \% k) ]
34 }  ~+ i9 {* Z4 n9 J* Q
4
% T' H/ [6 b3 o& z  esystemctl start ceph-mon@ceph02
- ]/ T6 R! p# }systemctl enable ceph-mon@ceph02
: O/ n. I; z& F- l9 Rsystemctl status ceph-mon@ceph02' S# Y4 w/ m$ Y  F! {
1
/ v  B4 m/ X- f0 H; r+ u  v2( p' |, C% M& @7 L! {8 N
3
3 n  _7 X  w5 q) {& ~. L启动其他节点的monitor服务(Ceph 03执行)
+ e, |0 ?0 M9 r! tsudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph03) `8 X7 q8 D' f$ j
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap4 d( F) a* u4 j1 |
sudo -u ceph ceph-mon --mkfs -i ceph03 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring! z! P# ^6 v; U& v/ n. K! ~# y
ls /var/lib/ceph/mon/ceph-ceph03/( R) u9 g: K+ P+ s4 d# H
1
3 F; g8 [$ B% X$ J% A5 S2
5 {6 T; |/ a4 e9 T8 ~4 c; y3
/ }# A( {4 r. A6 Z5 p7 N44 @' _" r3 }+ h
systemctl start ceph-mon@ceph033 r& Q1 F# \3 ^1 C
systemctl enable ceph-mon@ceph03
5 ^& S* `4 t. F9 G+ ~: x* }systemctl status ceph-mon@ceph03
' I( v1 \0 z  D; X1
  U" P% V3 P$ ]* O9 [2
; x/ P8 }( {$ l+ l6 Q33 }; n* s( f. a- k, b/ j
查看当前集群状态(任意节点执行)/ c5 v* _. y7 Y$ e
通过ceph -s命令查询集群状态,可见集群services中3个mon服务已启动。
; ?9 ?1 R/ P! u4 |ceph -s% E* j/ p( Q! r4 Y# \& K
> cluster:
; p; w4 ~8 ^. _& B9 }5 y  j>   id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a  * X  D" u: M; h, Q3 @
>   health: HEALTH_OK9 Q( m) [* ^1 {# c# _# s+ C
>
, V" r+ k1 X/ Y5 S> services:  0 p  C1 A5 J6 J) f
>   mon:  3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)  
8 ~% V3 E1 E( }5 S; P4 N, Y>   mgr:    ; B: {* R4 b7 A4 P
>   osd:
- I1 d& d- l. u) S& L) W( Z>
, A* x6 ?1 y5 n# _4 I- r9 X" n> data:
: {0 X0 ?- j4 j1 F7 f7 R>    pools:   2 c( y; A' I  q
>   objects:
; L, C3 ?  o# z( W$ b9 g( k>   usage:   2 B0 A# n0 [( {3 m
>   pgs:0 l5 _* M' m7 t5 e; t
12 L* K, p/ R+ M
2
# Q/ l5 ?) A/ }) r$ s, o6 W3
! f( T2 c- L1 }3 T; l5 d. v3 o4 {8 v* a4
4 q2 t! r! w' h/ g5* N* {6 `: C; X
6
% F& @& E, G' _. n. [3 b9 u7' {, ^* _7 X9 f2 F& ~/ @0 I
8
+ Q% P3 N  N1 l: I/ G& p: L4 Z9
. M1 G4 y5 q  U, D! n0 k* }10
* e4 U$ ]& d4 m. v% x- a  L6 u11# F2 U0 L( L( L! }7 y$ U5 |6 b
12( f; X+ [8 D& |3 `
13
, q4 n' O% R2 y14
% x. b9 l* b# f: r# G15% @# u- f& T2 Y# x! j% s
部署Ceph mon服务(ceph-volume 自动化创建)
0 ^0 V* }, [/ [7 H+ o( Z4 a安装Ceph-osd服务程序(所有设备执行)! n% x! v. t! t! }6 X
yum install -y ceph-osd3 x4 h4 ^3 }+ P/ r6 z
1+ z1 G" }% ^' f4 {/ P
初始化osd服务(所有设备执行)* d) Y0 l7 W& v+ ?- A
通过fdisk等工具查看磁盘盘符,然后利用ceph-volume工具自动化创建osd服务。# D) @& ^" X1 C0 M; z
ceph-volume lvm create --data /dev/sda
2 v3 m: L$ x/ i: [& B- C4 }# jceph-volume lvm create --data /dev/sdb
. o( u3 ^% M5 \+ S4 @9 n' e5 oceph-volume lvm create --data /dev/sdc6 c$ W8 K! R) ^' a3 [+ f0 a
1# f  p3 H# A9 E2 B3 v
2/ a, S" I" l0 [* U8 p
3' ~0 C. q% {. v' e% ~
查看当前集群状态(任意节点执行); [4 l' E+ e  L6 G6 ~7 q
通过ceph osd tree命令查询集群状态,可见集群services中所有osd服务已启动。% l, `$ K- z' p
ceph osd tree3 K6 G" U. K% D9 V6 W% ~& J+ `
> ID CLASS WEIGHT   TYPE NAME       STATUS REWEIGHT PRI-AFF
  t+ k& M+ t4 Y; P) P5 B' S$ K> -1       16.36908 root default
+ h; s  M! t7 ^0 u) R3 ^9 l' ]> -3        5.45636     host ceph01
1 }$ m6 }' W2 j1 |>  0   hdd  1.81879         osd.0       up  1.00000 1.00000* F2 h6 J$ E9 x. u
>  1   hdd  1.81879         osd.1       up  1.00000 1.00000
/ `7 ~5 C9 j/ m7 }) c>  2   hdd  1.81879         osd.2       up  1.00000 1.000003 K- I7 R, C# I! I- v0 R# G, E
> -5        5.45636     host ceph02
" `* W, d4 _/ \9 n! I) A( _2 h>  3   hdd  1.81879         osd.3       up  1.00000 1.000003 |: n% f+ ?  o! i% K: D- `! @+ M, _) }
>  4   hdd  1.81879         osd.4       up  1.00000 1.00000
  ^9 _0 F: g* h9 a>  5   hdd  1.81879         osd.5       up  1.00000 1.00000
/ ?9 x* @% l; a  O- W> -7        5.45636     host ceph03
* o8 _7 R1 u! A>  6   hdd  1.81879         osd.6       up  1.00000 1.00000
1 [* ?  f) q( }* k. L>  7   hdd  1.81879         osd.7       up  1.00000 1.00000- q9 ]7 J' M- s4 ?3 ?2 q( `
>  8   hdd  1.81879         osd.8       up  1.00000 1.00000
+ `& p& e2 {! F" G; J0 S1
5 f, F. p! C& u' S' i' D: q5 h28 w8 q5 [8 J( V( u6 ?& M
3
9 @% E* y1 V2 p4 S" L: p4( q& }0 m" j+ y9 L9 O- o
55 R" W8 ]1 T- z% F! j  y3 l& }( o
65 E" a7 _8 v0 ]; I
7( }- r% l  S; p/ T; H
8: J* l5 z. M  J$ L
9( Y8 E  n  A5 P. G' |
10
6 }4 f0 i# {9 R8 m# R% J11* g8 z" p5 v3 C0 a  ]. W
127 z: J; @) g- @8 I; B: \2 _7 g/ N
13
) Q6 q  U3 U4 v' b14
( z) r, l+ l5 e15% }6 `! t+ {- J) A+ R& X% L
部署Ceph mgr服务并开启Dashboard
4 X6 S9 _8 u) d2 o安装Ceph-mgr服务程序(所有设备执行): I8 U8 a7 R9 ?' U
yum install -y ceph-mgr
4 ?# c# j2 j. X3 `1
$ H$ H  g& Q8 X初始化并启动主MGR服务(Ceph01执行)4 O7 V+ w: U6 u: K1 A9 |, N% `
mkdir -p /var/lib/ceph/mgr/ceph-ceph019 `( V+ o  X, p; O7 q2 o
chown ceph.ceph -R /var/lib/ceph' I; d/ e% ^# A' x2 l
ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph01.keyring --gen-key -n mgr.ceph01 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
5 `8 N3 ^' f/ Y; t2 R" yceph auth import -i /etc/ceph/ceph.mgr.ceph01.keyring
; R2 C7 U  v8 `" l+ Dceph auth get-or-create mgr.ceph01 -o /var/lib/ceph/mgr/ceph-ceph01/keyring
& `& `6 Y* |( z& y: z/ H1+ Y8 Y$ o) k& ~+ x
2  I% W3 ?% P1 m! h8 J+ I" Z: {* [" [6 Z
3
. w( ?/ A( U7 x, v4
) {- j) }1 ?7 r& J! s5
+ U" h* i* Y5 u1 [: jsystemctl start ceph-mgr@ceph01
" z8 ~+ `: D" b3 @systemctl enable ceph-mgr@ceph01
$ r2 P, @' k3 Osystemctl status ceph-mgr@ceph01+ w( a& E8 w) w. Q2 A) P
1
5 i! j( m5 U& l. B! l+ c6 f% _2
' x" h/ B2 V) k8 M31 \/ a5 i' e1 K, o( V
初始化并启动从MGR服务(Ceph02执行)
0 N7 t$ Q/ O* q. tmkdir -p /var/lib/ceph/mgr/ceph-ceph02" x- {" W* f  V/ @! q" E
chown ceph.ceph -R /var/lib/ceph! x1 O# Q) R  d1 D
ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph02.keyring --gen-key -n mgr.ceph02 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'+ q" o! q% m# b
ceph auth import -i /etc/ceph/ceph.mgr.ceph02.keyring
- ~$ U$ f8 f  `, [9 w, sceph auth get-or-create mgr.ceph02 -o /var/lib/ceph/mgr/ceph-ceph02/keyring
6 J9 h8 m6 D6 T+ H/ n1
0 o5 r- b: M3 t2
! {& @. h& l7 m8 b2 G* V8 e3# G* V% [3 J% L+ D# z
4. E) G4 Q- c6 f" G5 H
5( W0 h) \% U  V0 D' ?9 P+ Q% K3 q( E
systemctl start ceph-mgr@ceph02+ m9 q/ E/ k# y( L, Q; u( z
systemctl enable ceph-mgr@ceph02$ w1 J( D1 e, T: P- f* d
systemctl status ceph-mgr@ceph02" M, `; o% o7 e8 ~
1
/ D" `# u; P* I/ R2
6 }+ D7 [% w0 t! {/ G3
& P  B( U9 Q" `8 }初始化并启动从MGR服务(Ceph03执行)
& g9 @. Q# N% Amkdir -p /var/lib/ceph/mgr/ceph-ceph03$ t* F& H5 }6 i* F( D
chown ceph.ceph -R /var/lib/ceph
% ]; n" O" U4 R# J/ B7 `ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph03.keyring --gen-key -n mgr.ceph03 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
0 ~4 I2 a5 G  M. _; E6 l2 ]! rceph auth import -i /etc/ceph/ceph.mgr.ceph03.keyring
& d& F. D# J7 |3 g2 zceph auth get-or-create mgr.ceph03 -o /var/lib/ceph/mgr/ceph-ceph03/keyring& f& f! j/ [+ K* V
18 N* c7 y9 j6 @( ^
2. Y6 i% y0 [' C5 A0 |
3
' O% N( k2 E5 `3 i* a49 R8 D  ^, O+ y( |2 T6 K
5. T$ X) l- }3 O+ w9 q: \
systemctl start ceph-mgr@ceph03( M2 x. j8 [1 b2 ]' E( v' P9 E$ D
systemctl enable ceph-mgr@ceph03, ]- k$ o0 C; R" u, o6 h' n  i! q
systemctl status ceph-mgr@ceph032 N, K# y- V/ R! R8 l
1
( e1 ^8 ]' z/ i9 }29 v2 k5 Q( _4 j% `: w
3
& W. @" \* ?' E' _查看当前集群状态(任意节点执行)
1 j4 p8 o7 T5 M! q+ q) ]通过ceph -s命令查询集群状态,可见集群services中3个mgr服务已启动。* k( O# C+ p  h* I& F4 H
ceph -s* n. f, l, f: i1 `3 d4 S
> cluster:
+ @2 ^5 v7 _5 i9 @>   id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a  . V7 {9 v: M( K+ k" a! B
>   health: HEALTH_OK
3 j; y: y2 q, |1 W1 b: z+ Y% G# \; @> 5 ^( C' \) L2 G: J+ P8 K
> services:  / r; \/ b/ G8 ]; d1 k! T
>   mon:  3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)  
% V' u5 d6 X2 p6 J1 J: ^* X* B>   mgr: ceph01(active, since 3w), standbys: ceph01、ceph02   
- O% D3 s8 }# }: Y! e>   osd: 4 b' D3 z" J& d5 x" I, J
>
1 x$ l  g' t2 S; p- X; ~0 X> data:9 B3 w6 x  L- S- z
>    pools:   " a( Z. A( v- L) x
>   objects:
( Y8 D  y! I; m5 @5 L>   usage:   
( g3 q6 v- |, o3 P6 m! H8 Z2 h>   pgs:
! \1 u2 f9 p& L& n  D& r% l13 X, O3 N6 W& q2 M9 ?( Q) J
2
' U4 P0 `2 D9 A9 C3/ S. j& w9 S: V$ A4 m! ?
4
: l& N- A0 m* ]( T5; B" k0 ]4 C" [# V
69 ^) `  _/ |. {% H, {( x  Z
70 |6 R" U' C$ f4 `* v: P
8, R5 f' L( K4 P
90 t' o6 X* o; r& O# D! k5 U
10
  z4 `3 p6 D4 N+ `" t  c1 z) j11
# p) A7 i% M' q0 i2 L' h12
' a4 y8 o" ^6 a138 v1 Y, n' }( L5 Z' d
14
! B4 @& }- L3 ^8 Q2 D0 N) }( `9 z15
5 x! E2 _) E: U# [使能Dashboard访问功能(任意节点执行)6 @, d  J) u/ Q+ s
开启mgr dashboard功能
, R& _  Q4 j  o" tceph mgr module enable dashboard
. h6 S. o  J5 n4 Y  W1; s! Z) ~' O' j: U1 z
生成并安装自签名的证书
/ D  M! [* I4 j; U5 b1 sceph dashboard create-self-signed-cert9 q2 b! F" H5 m+ S
1' a' D4 N( u- M4 S# Z
配置dashboard
$ ~/ w5 i' p* sceph config set mgr mgr/dashboard/server_addr 10.40.65.148
. f) g1 s3 Z% f# z4 H; nceph config set mgr mgr/dashboard/server_port 8080
% u+ c* x& ?( B0 m# @$ U' U' i% }7 Sceph config set mgr mgr/dashboard/ssl_server_port 8443
+ H8 m$ d- {: S" j2 ^% C1
! H5 ]" L9 A" a, \2 B* Y4 a1 s2& L/ a, y+ k% M8 e6 C+ w+ e
3
" A$ h; k+ |# v* J  L6 z; V4 D创建一个dashboard登录用户名密码
8 R* s6 d  }3 e4 X  Decho '123456' > password.txt
. b; x2 t$ x/ {( q1 Mceph dashboard ac-user-create admin  administrator -i password.txt4 H$ u! Q; _% k0 c8 j- A8 A
1
+ t: \' b: `! {3 H* I6 S0 U* N9 N2
& a& F) I  f. v2 _查看服务访问方式/ I/ ^' j8 l, D5 ~+ L, f+ Y  K. F
ceph mgr services
8 S/ O- Z0 a6 G3 Y3 M1 k; [1
& N) b5 x; z& t通过web访问Ceph Dashboard,用户名密码为admin/123456
: L( D9 k6 ~2 Jhttps://10.40.65.148:84432 _/ R( \1 }' Y

+ V. [+ h0 D9 W7 S* s

0

主题

0

回帖

9

积分

管理员

积分
9
QQ
 楼主| 发表于 2022-7-20 13:47:56 | 显示全部楼层
部署Ceph mon服务
' i% F; \9 F. M1 P1 a/ h安装Ceph-mon服务程序(所有设备执行)
0 {8 B" l4 e6 q* S7 B* g3 ?$ [$ h: s
6 G" D" ]8 ?7 V, Ayum install -y ceph-mon3 |1 z6 ^' }& P$ M
1
9 |- P2 \4 K3 L0 P初始化Mon服务(Ceph 01执行)
1 l5 Y; u9 ~& Y' q1 r: H3 j! [( A9 l( ]
生成uuid& ]+ Z  k1 U5 @1 F+ _- P' a/ {
0 h" [- f+ x8 f: Q; c
uuidgen
3 P* k9 i3 A3 e) U8 A> 9bf24809-220b-4910-b384-c1f06ea80728* z# S6 E2 E  r" f6 `1 f# E
1
3 y/ d) V; E! p$ G3 C2
" N$ {6 Z6 y" N; Y; S- M创建Ceph配置文件) f6 _1 k) `, t9 _+ Z

0 p- P) {* d; L% ~8 ^( U! }" Fcat >> /etc/ceph/ceph.conf <<EOF5 N& b5 G, U1 [4 m
[global]
/ j' [' h3 W0 z9 z* q' Zfsid = 9bf24809-220b-4910-b384-c1f06ea80728* ~" K, ^9 r! B* f$ e' }( r1 o
mon_initial_members = ceph01,ceph02,ceph039 U+ Z1 Q* f) g1 e: N# u4 d* @. o6 }6 z! @
mon_host = 10.40.65.156,10.40.65.175,10.40.65.1299 L/ k" ~+ w  n4 [# f
public_network = 10.40.65.0/24
5 N. v5 f  A9 G, Yauth_cluster_required = cephx+ z3 \& G; ]+ g- B. @9 l
auth_service_required = cephx
* E! R/ R5 `+ S3 I1 z4 {/ w( ~auth_client_required = cephx) w' x$ c5 ~  X$ M5 {( Q
osd_journal_size = 1024
3 j( E9 {1 y* t8 M+ D% Sosd_pool_default_size = 3
. A- a5 I1 u4 E1 y: aosd_pool_default_min_size = 2
0 p4 z1 b9 R# |- _1 w0 ?osd_pool_default_pg_num = 64% C, l- p- O% f* x, w
osd_pool_default_pgp_num = 64+ |, u1 t5 c. e4 i$ m0 K
osd_crush_chooseleaf_type = 1
4 D! q4 W; b1 J1 N6 B4 D0 h9 HEOF; W9 H- y% i# ]% a$ I
1
' a# p( a/ |8 n3 N* E/ T- j2
7 R/ b% D# A; `3 z+ E. U, ?. M% E3
% L) v( d2 O; G& p3 S4: `, J, a/ I5 G* P3 z, Z
5/ B# t$ k8 m, _2 |+ D9 a
6
" u2 V5 L4 Q$ v1 V1 |70 q. b! `9 w9 @. u6 Q( |7 H
8
7 F( T* ~2 T" K9 Y0 b. H$ e( z9
$ |3 [' W& X5 c7 G106 O: u( z1 c5 j* w+ w! K
11  v/ c: u+ E( U( w% E# {) j; v5 Z
12% j8 {  }" j/ s
13# \% X- a) _% {; }9 n4 `
14
% N# m4 w- X8 ]15
# K" e0 v2 Q5 Q9 ~7 W: Q" i16
! i/ h6 \* Q2 M/ E% g8 K* Y# y创建集群Monitor密钥。
" w0 o( C& l/ @9 L! u8 v6 N
. V1 Q2 L: D8 N! ^0 {3 tceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'# M: c6 Q% F5 A; ~' L+ A$ F" A
1; G7 k0 _9 \' `
创建client.admin用户、client.bootstrap-osd用户密钥,添加到集群密钥中。
4 l+ k: P  b7 `% {& s2 @0 N% y  Y
: W' G4 M, p1 S9 {ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
( d% E2 @, U. N: Uceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
9 p# M5 S9 f* ^1
% i" ]  ^- @. D0 Z" ?2
6 `- p  P4 A1 V' P- `ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
" h# [2 v5 M2 T4 V, I) T1 Eceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring" \8 V5 o: ]4 v9 a
1$ s) L8 k8 B" Q) M2 r: u/ B
2  k  r( B9 i1 S& \( R0 g
使用主机名、主机IP地址、FSID生成monitor map。
' n7 o3 B  k3 }6 l6 T/ l( Q2 W) d  O1 E* g* N  R  K4 r& V* y
monmaptool --create --add ceph01 10.40.65.156 --add ceph02 10.40.65.175 --add ceph03 10.40.65.129 --fsid 9bf24809-220b-4910-b384-c1f06ea80728 /tmp/monmap
. R- B' N& v8 r5 h% q* F1 j1
( |( ~9 D1 k  f初始化并启动monitor服务
9 U! A; r" Z- d3 J' ^/ m; L7 V/ [" k6 y/ j8 c
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph01* S; x% @3 s9 m7 H; Y4 i
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap) _" {6 ]1 s" }: w7 m- e0 q
sudo -u ceph ceph-mon --mkfs -i ceph01 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring% a2 E' e) d% M4 h7 s
ls /var/lib/ceph/mon/ceph-ceph01/( C" E( j: k4 u8 ~
17 d6 ?" P2 H# n# _
20 O7 a& N$ f  F- Y& S0 W- U
3
6 i; O- C! L) R2 P) v% u( o) g0 {4! z, m' F( ^. I0 X0 Z3 _& t
systemctl start ceph-mon@ceph01
& \& x  e! Q; u8 ?systemctl enable ceph-mon@ceph01
8 k* M" i% `# }, J5 `systemctl status ceph-mon@ceph01
" T! U& U: C4 ^1
& S* L6 m1 Y+ _8 l2 X21 A* N9 r* D3 k
3
1 e; S- R; h+ M: k同步配置文件、密钥、monmap到其他节点中(Ceph 01执行)2 m: j" S$ K" N
: w9 n$ A/ G% E" V3 @/ o5 _
复制ceph.client.admin.keyring、client.bootstrap-osd key、ceph.mon.keyring、monitor map、ceph.conf到另外2个节点
+ i! j+ H& Y) q/ b7 K2 W
, o$ h% q. m+ I5 D& m# I1 q! f  Wscp /etc/ceph/ceph.client.admin.keyring root@ceph02:/etc/ceph/6 m% P8 j/ n9 T& L: k# t
scp /etc/ceph/ceph.client.admin.keyring root@ceph03:/etc/ceph/! q9 f0 A9 R3 S7 g  |- {
1
, v( A- r: C# a: ^21 y& b( ?7 I1 V9 p
scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph02:/var/lib/ceph/bootstrap-osd/
+ I. G% N, z# m# B3 `scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph03:/var/lib/ceph/bootstrap-osd/" T1 n9 \! H5 S4 e6 }5 W- q& E% ^
11 y! H& U2 Q5 L' u7 F
2. g1 ~) K: t0 T6 O' B& h1 t  u
scp /tmp/ceph.mon.keyring root@ceph02:/tmp/: N, y& q$ [) m) e6 m0 S0 r9 L1 `
scp /tmp/ceph.mon.keyring root@ceph03:/tmp/
) h0 h' p) F3 p+ |- V: X9 z' ^1
# `# A  Z3 a' j7 M2$ O5 E  b. J/ n9 F, s2 P
scp /tmp/monmap root@ceph02:/tmp/( K( u0 F( }% `* t; H% q( J
scp /tmp/monmap root@ceph03:/tmp/6 R" z3 D$ F* L
1" }2 }9 X+ y4 Y, L' }' I1 V3 a2 ]
2# C8 o9 L& B6 b" s' V! }$ s& P
scp /etc/ceph/ceph.conf root@ceph02:/etc/ceph/* o8 h# s5 U. K6 e( ~% z
scp /etc/ceph/ceph.conf root@ceph03:/etc/ceph/
+ ?' y5 H( {1 Q3 X- a, B1, R6 }' @  ?/ {, l
2) w) D7 h3 K8 C( J" ~/ r6 \" y. n
启动其他节点的monitor服务(Ceph 02执行)
0 ~5 R& b% g/ X- p5 z9 B
7 ^7 W! f7 L1 E* l- c" \( }sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph02$ ^( P. u3 d/ e1 I/ V* u
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
- y: L/ q) [" u1 x& Usudo -u ceph ceph-mon --mkfs -i ceph02 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
& j: ?- \0 O6 y; ^$ n0 D1 Y- qls /var/lib/ceph/mon/ceph-ceph02/
4 E7 i  I9 z& {- `, C! E! w1
) o8 l% k/ `+ v; C8 J7 s  N2# T! Q0 _# |' P; c
3; g9 c+ S: ^$ G- p, E
4/ h0 f4 E% ^& M% I
systemctl start ceph-mon@ceph02
* _& t7 Q  D; w: ^4 P6 esystemctl enable ceph-mon@ceph026 Y( e: K/ ^' L% u# W2 K: w1 s/ p
systemctl status ceph-mon@ceph02
, a/ V7 H1 x, N6 `12 p$ x/ Y& l& E& q7 m6 S6 o; u% ^
2
: R6 p6 L* w/ J5 r4 X3
4 Z  Y( I2 o0 u( F1 m6 ~启动其他节点的monitor服务(Ceph 03执行)
7 u) f: B" z0 t( d
2 K3 F# ], t* V: E% `$ n7 ksudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph03/ _! K% R( y% @  u% e% q: j7 _+ j
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
& P3 e7 e: p* N0 K" _+ h; lsudo -u ceph ceph-mon --mkfs -i ceph03 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
/ z$ L1 M+ }, O( _5 y% ^ls /var/lib/ceph/mon/ceph-ceph03/
$ j8 y7 K" G  }& [7 e$ d$ n1
# c% Z# F! _! \) r27 k6 a6 z1 U- f( W$ d% w
3; k7 s& F  v  N( d8 U
4
" j) Q- V/ D; |6 c/ a# X: W; Vsystemctl start ceph-mon@ceph03
& i' a# V) X; E8 @/ ~! \) ]# psystemctl enable ceph-mon@ceph03
/ h- j/ e, v9 j  v4 n3 ^7 ^systemctl status ceph-mon@ceph03
3 I& Q% h1 [' C  x: Z$ h18 C' B2 M  D% X, t2 F# X4 h
2( ]3 ~6 i% s: r7 B9 P
3  b9 L/ M& X" c; X% D/ _; F7 }
查看当前集群状态(任意节点执行)
. n8 w# z& y7 D1 p/ b' x# z通过ceph -s命令查询集群状态,可见集群services中3个mon服务已启动。
5 m7 v' F* J/ {9 T2 r/ C! ^, t! Y1 ?0 W8 H: l, f% ]
ceph -s. c( M! [. R! f, C# Z; S/ H2 o! e; ~, P
> cluster: $ l2 W' S/ F4 P' s# e$ V" L
>                 id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a        
: p9 x9 D3 L, C  V) T0 o" @) z8 S>                 health: HEALTH_OK
4 \( k& ?% @& |1 H! J! ~  I' [> . i/ `  ?% V5 ]+ a# n  D- [
> services:         7 q& k- ]8 s$ g$ X' I9 d" e7 {
>                 mon:         3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)         7 P/ c$ J# A2 b, h$ p
>                 mgr:                                , c0 d1 a( z, F* d3 t" D) h/ p
>                 osd:
; \5 Z, y2 o5 m" a>
% H) @& d: P! p% p> data:
7 u% z8 L% _! Z$ z  `( `$ p>                  pools:   6 g  J% U' h5 t. @
>                 objects: 4 G) _. p- |6 Z2 b& e" m: k
>                 usage:   * w# X& L( m# O* t8 n; x/ E
>                 pgs:* q: k2 _5 ^/ y) _# W% u
1
" k% c+ T& m: \  b- Y; R2" K8 m) B/ @. k( e
3
3 h3 g& y7 t7 r4 b  z4
; N8 j# z' w) i5& c8 A  Y& i, X: o  j4 x
6
0 r1 @+ ?4 M) N7
6 D! K& l8 n1 `7 ?84 Q! \+ f; ^+ d4 F, _' b* d
94 }- N! D6 W; z2 w0 Y4 z
10
3 b& t8 ?( u1 Z+ ]11
' R1 w$ `+ t1 j' c0 J7 D12) Z$ E3 P5 d' Q7 {. F# k, y5 F
139 u8 \6 x+ `8 C; {6 N9 ~+ x8 l
14: T8 B: R3 U1 [0 z& J
15
4 a! y# e: n4 Y8 j! p! K部署Ceph mon服务(ceph-volume 自动化创建)2 C: K" i6 ]" }. n" _/ j
安装Ceph-osd服务程序(所有设备执行)
2 ?' _" r$ k9 g- Y+ U( Tyum install -y ceph-osd
/ y7 ]- K. w1 N: m1! ?: w7 Z# c/ m4 \
初始化osd服务(所有设备执行). \0 _5 \' h- q  h1 x( Q+ t$ l
通过fdisk等工具查看磁盘盘符,然后利用ceph-volume工具自动化创建osd服务。
3 {/ `" f, T4 [3 p- t' lceph-volume lvm create --data /dev/sda9 Z% f: N" h# s& z. K0 a4 }3 w
ceph-volume lvm create --data /dev/sdb" d! a" X; P6 n
ceph-volume lvm create --data /dev/sdc. ^7 h  w3 i8 e& Z. h
1
1 e; D: G" @& |( I* ]' \2) G; d8 q* t' T4 U! M
3
. m' r7 P$ M& V! N+ e) H查看当前集群状态(任意节点执行)- f- X! `6 f+ q3 x
通过ceph osd tree命令查询集群状态,可见集群services中所有osd服务已启动。
5 t  {5 K* U. V$ {0 T% cceph osd tree1 |( I5 n; ?, G. t
> ID CLASS WEIGHT   TYPE NAME       STATUS REWEIGHT PRI-AFF$ V& L) e8 v3 n
> -1       16.36908 root default4 }+ U- ]6 \& n
> -3        5.45636     host ceph01
- P: m5 @2 @) N6 Z; M5 a>  0   hdd  1.81879         osd.0       up  1.00000 1.000003 I' _& B6 Q" C4 |- s
>  1   hdd  1.81879         osd.1       up  1.00000 1.00000
! X  i+ H& e' y$ d, l( L>  2   hdd  1.81879         osd.2       up  1.00000 1.00000
  v  w6 ?8 d* B7 j9 |> -5        5.45636     host ceph02/ U0 Q% o5 j& d' D  k
>  3   hdd  1.81879         osd.3       up  1.00000 1.00000! t% {9 e% g5 [7 E: d
>  4   hdd  1.81879         osd.4       up  1.00000 1.00000
# Q: i' X  h* E: C>  5   hdd  1.81879         osd.5       up  1.00000 1.00000/ H( W7 w! T- |' c, n& c
> -7        5.45636     host ceph035 [! ?6 Z( `3 E+ x1 o
>  6   hdd  1.81879         osd.6       up  1.00000 1.00000
6 l/ ~$ l8 H  F: V8 g+ A) V>  7   hdd  1.81879         osd.7       up  1.00000 1.00000
; D% g& s; ~8 L' _" w>  8   hdd  1.81879         osd.8       up  1.00000 1.00000
- j5 T8 e2 T& h/ ?) a2 g1* u" H5 x. C; |
29 V# V6 H: V0 Q3 R/ z
3  F4 T, W- N% b; G% G. z
4& i% p$ e- l0 E4 E9 B1 h3 \& P
5  ~4 R% V$ C3 D$ ?# X' U
6
3 W0 H% t8 z$ r2 l* u6 c7, l- e5 x9 T6 w  @' R
8
" M/ ]$ n* `( T6 n: |# s" a3 z" |9* s' ^; L6 O* I/ g/ c+ ^
10* l- T, d8 I! |/ E% a* }
11
6 Z7 p* D% l, x/ t8 l# g12
) }  S4 Y; C2 e4 J. T6 @7 j3 e: y5 p13$ r2 }' `$ S+ X
14. Y* U4 F9 f4 F. Q; N5 Z
15' k  ~; K$ J+ m- G" U* b
部署Ceph mgr服务并开启Dashboard, g1 H. n0 y  ~) [. _( M: M
安装Ceph-mgr服务程序(所有设备执行)
. U2 W5 o- k/ l* G! F. d2 Nyum install -y ceph-mgr9 W( X; m1 T! T! w6 W4 g
1( l: u2 N* F! ^5 d. T
初始化并启动主MGR服务(Ceph01执行)
. Q$ Z2 K6 ~7 {0 g2 }mkdir -p /var/lib/ceph/mgr/ceph-ceph01
, P) S+ W, k0 Ochown ceph.ceph -R /var/lib/ceph9 x8 t1 v; D2 [0 ^
ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph01.keyring --gen-key -n mgr.ceph01 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'0 {* }* A6 W9 L# I( f
ceph auth import -i /etc/ceph/ceph.mgr.ceph01.keyring1 {4 s) h% t; y6 w/ R; W
ceph auth get-or-create mgr.ceph01 -o /var/lib/ceph/mgr/ceph-ceph01/keyring
1 U$ F7 O0 O" N! q1
; b4 O. P* k% t+ e4 M; g9 R2  p, x( T7 P0 L5 G' z0 _0 _% Q  B
3* Q8 V. n' d: i- D- N: I4 k
42 p* L4 q5 [% e0 z
5* j1 t2 y% T) {0 H5 g: H
systemctl start ceph-mgr@ceph01& F6 j; L4 I+ o5 u: [9 k; D
systemctl enable ceph-mgr@ceph01- A3 h+ f9 u) c
systemctl status ceph-mgr@ceph01
  @. O/ a  M0 [3 b: I  B1
% X$ T' x; c/ y; _# |% X  x1 _2
& U+ X2 g3 X2 H, V! }$ r" Q3
/ z9 K6 I" t( a9 n/ J初始化并启动从MGR服务(Ceph02执行)- U8 E2 S9 _7 V0 ^* M6 }3 N
mkdir -p /var/lib/ceph/mgr/ceph-ceph020 c3 D5 ?4 P1 L& u; g1 u$ I
chown ceph.ceph -R /var/lib/ceph
* |, q1 _4 [; F: k8 Cceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph02.keyring --gen-key -n mgr.ceph02 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
, I5 p' U% ^4 i! v3 v0 ^! uceph auth import -i /etc/ceph/ceph.mgr.ceph02.keyring
  n! U8 ~# V+ ~ceph auth get-or-create mgr.ceph02 -o /var/lib/ceph/mgr/ceph-ceph02/keyring; @- S, B. m9 w) X! d
1
% }7 K8 W6 b( O7 _2% ^2 Z3 w& v$ B. Z7 N
3' m) E9 ?7 o: h9 B
4
! c3 ]% Z. a5 z5
8 O1 f. R& J' R5 [( L! csystemctl start ceph-mgr@ceph02
- h' O' v) v# j* m; Csystemctl enable ceph-mgr@ceph02
$ ]/ B% W( r( x/ B; Asystemctl status ceph-mgr@ceph02# j% |3 f; W2 q- E+ o4 V+ T
18 t' R8 `' e& I% h' j
23 r  E/ t- o/ \' p' c: M" G
3
% ~! a. u: J: g* l. T, n7 y; [2 ?初始化并启动从MGR服务(Ceph03执行)$ @4 T3 L" X/ ~/ U& Y, {9 @
mkdir -p /var/lib/ceph/mgr/ceph-ceph03
2 Z' V: H* u. U3 E% {- pchown ceph.ceph -R /var/lib/ceph
" i$ J7 n) i4 n5 [ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph03.keyring --gen-key -n mgr.ceph03 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'9 d1 V! x: @, {- G8 K
ceph auth import -i /etc/ceph/ceph.mgr.ceph03.keyring% E6 ]5 Y* n" h. }+ ~
ceph auth get-or-create mgr.ceph03 -o /var/lib/ceph/mgr/ceph-ceph03/keyring: w  }! }+ |, @! q+ O+ I
1
4 p# \" v0 ?, o( L! A, p) Q! q6 R2* u# S5 [6 ~: ~; S
3
+ {+ h7 y- X; @# @" N+ l8 m0 j; Q4
5 R8 x4 P# @: u% i. _& F9 W" E  M5
& }- v  `; T7 f6 H) ?8 v+ I2 {8 |6 S2 ksystemctl start ceph-mgr@ceph03) \+ ~2 U4 N; I) W7 w! t4 [
systemctl enable ceph-mgr@ceph03
' M& b% P. t- l( b2 Rsystemctl status ceph-mgr@ceph03
3 b4 S# y5 _  o% i4 w- S5 W2 J6 R16 g* E) S8 K' M* h
2
$ M6 H: V! P, d. o  h3; C9 m& X9 R, i0 \! s+ v1 l6 \
查看当前集群状态(任意节点执行)
: B! m) Z8 e* j7 j) J0 Q通过ceph -s命令查询集群状态,可见集群services中3个mgr服务已启动。2 t6 Q% I: M3 s& }. N! d. r/ L
ceph -s# b8 i' ?9 _5 ?' X% n
> cluster:
. Q( Q* Z7 e7 w; p6 G$ c% h>                 id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a         8 n% d! n! {# P2 j' |
>                 health: HEALTH_OK1 t5 R2 i; A. r# [" c& q9 n0 v
> ! p% ]# {1 g7 s# V0 b# b
> services:         8 j/ W' b. L/ `; e- X# y9 d
>                 mon:         3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)        
. M9 T9 N0 U/ m! P>                 mgr:        ceph01(active, since 3w), standbys: ceph01、ceph02                        8 }0 w) s# ]6 v9 ?
>                 osd:
8 p/ h- d8 O+ O# u>
3 F5 E. |  K4 y' y" d) _6 c" J0 ?> data:  A; F' t9 `" G5 B. S! {  t2 Z
>                  pools:   
) i* P% }. G2 w! Z1 L9 O9 x; r>                 objects:
. I: _1 U- ^) h3 I% g# x+ s>                 usage:   # t8 |; z2 {# ^! o' K
>                 pgs:* e. s2 N) @+ V% m, f! j( S
18 M5 h9 P. w3 _1 F8 X
2
2 I2 n/ f: ^) g% J3 a+ ~39 a1 W* e5 V1 w2 a. n( q
4& f  I9 ?8 _0 S/ ]4 v
5- T% X+ i/ d6 j8 s3 p. g. k
6* s, [4 f6 {$ A) s
7
5 K$ q) F+ I- J" v: X; B0 G6 E8
4 u. L$ s5 |* v" B0 _: L93 m7 g5 V  ?( T" @9 v# `
10
( Y8 G, M/ _9 K- H11, [9 C7 j4 u$ n8 y1 x" ~. f% _. G
12
* d: I1 Z1 K) |& J13& s9 V+ M6 t% [
14) F( F; |7 x* Z7 j+ a+ I
15) [' n  Y+ }4 i
使能Dashboard访问功能(任意节点执行)5 Y/ u2 p/ T( P; K" E) c
开启mgr dashboard功能
, G1 C; d1 n$ _+ Z1 \ceph mgr module enable dashboard+ P# ], |" G, L8 |5 p- W# o
1+ @* g5 o6 r$ N( ~& c9 Q6 q
生成并安装自签名的证书
. p' V9 s0 f9 t; I% Lceph dashboard create-self-signed-cert. T$ R4 z7 _2 G8 |
1( I0 h* N" A- i! a, U
配置dashboard( q9 a3 ?9 d# U& t
ceph config set mgr mgr/dashboard/server_addr 10.40.65.148
- W" u0 L: E+ Q$ L) V# O3 Fceph config set mgr mgr/dashboard/server_port 8080! k9 [) k& Q- e( M
ceph config set mgr mgr/dashboard/ssl_server_port 8443' S. ]2 y7 }3 }: [% ^8 o
1$ y1 Z/ T/ ]" J4 n
2
7 r) W5 G0 K: j8 ]& k3
0 [8 v( n  Z, f0 b* _  E$ L( M2 u创建一个dashboard登录用户名密码
3 _, }2 B. X" h$ u' D; D9 ]echo '123456' > password.txt
# e! C: V: @# N- ^: bceph dashboard ac-user-create admin  administrator -i password.txt- V2 j0 o* E. }7 V
1
% b/ }! w2 ]3 h) d" C0 u% Q2
  T. A. S+ q( a  B+ C查看服务访问方式
3 M  O5 d4 |6 p! t5 B; i; K/ }ceph mgr services( ^( O# m6 Z& x+ i3 n# k. Q
1
7 o" |; G. |1 A通过web访问Ceph Dashboard,用户名密码为admin/123456$ o& k3 z8 I! C& W: A
https://10.40.65.148:8443
您需要登录后才可以回帖 登录 | 注册

本版积分规则

返回首页|Archiver|手机版|小黑屋|易陆发现技术论坛 ( 蜀ICP备2026014127号-1 )

GMT+8, 2026-6-11 22:56 , Processed in 0.017912 second(s), 23 queries .

Powered by Discuz! X5.0

© 2001-2026 Discuz! Team.

快速回复 返回顶部 返回列表