找回密码
 注册
查看: 845|回复: 2

手动方式部署ceph集群以及添加osd

[复制链接]

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
发表于 2022-7-19 11:20:00 | 显示全部楼层 |阅读模式
一、准备
前一篇点击打开链接只部署了一个单机集群。在这一篇里,手动部署一个多机集群:mycluster。我们有三台机器nod1,node2和node3;其中node1可以免密ssh/scp任意其他两台机器。我们的所有工作都在node1上完成。
准备工作包括在各个机器上安装ceph rpm包(见前一篇第1节点击打开链接),并在各个机器上修改下列文件:
/usr/lib/systemd/system/ceph-mon@.service /usr/lib/systemd/system/ceph-osd@.service /usr/lib/systemd/system/ceph-mds@.service /usr/lib/systemd/system/ceph-mgr@.service /usr/lib/systemd/system/ceph-radosgw@.service
修改:
Environment=CLUSTER=ceph <--- 改成CLUSTER=mycluster ExecStart=/usr/bin/... --id %i --setuser ceph --setgroup ceph <--- 删掉--setuser ceph --setgroup ceph二、创建工作目录
在node1创建一个工作目录,后续所有工作都在node1上的这个工作目录中完成;
mkdir /tmp/mk-ceph-cluster cd /tmp/mk-ceph-cluster三、创建配置文件vim mycluster.conf [global] cluster = mycluster fsid = 116d4de8-fd14-491f-811f-c1bdd8fac141 public network = 192.168.100.0/24 cluster network = 192.168.73.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 128 osd pool default pgp num = 128 osd pool default crush rule = 0 osd crush chooseleaf type = 1 admin socket = /var/run/ceph/$cluster-$name.asock pid file = /var/run/ceph/$cluster-$name.pid log file = /var/log/ceph/$cluster-$name.log log to syslog = false max open files = 131072 ms bind ipv6 = false [mon] mon initial members = node1,node2,node3 mon host = 192.168.100.131:6789,192.168.100.132:6789,192.168.100.133:6789 ;Yuanguo: the default value of {mon data} is /var/lib/ceph/mon/$cluster-$id, ; we overwrite it. mon data = /var/lib/ceph/mon/$cluster-$name mon clock drift allowed = 10 mon clock drift warn backoff = 30 mon osd full ratio = .95 mon osd nearfull ratio = .85 mon osd down out interval = 600 mon osd report timeout = 300 debug ms = 20 debug mon = 20 debug paxos = 20 debug auth = 20 [mon.node1] host = node1 mon addr = 192.168.100.131:6789 [mon.node2] host = node2 mon addr = 192.168.100.132:6789 [mon.node3] host = node3 mon addr = 192.168.100.133:6789 [mgr] ;Yuanguo: the default value of {mgr data} is /var/lib/ceph/mgr/$cluster-$id, ; we overwrite it. mgr data = /var/lib/ceph/mgr/$cluster-$name [osd] ;Yuanguo: we wish to overwrite {osd data}, but it seems that 'ceph-disk' forces ; to use the default value, so keep the default now; maybe in later versions ; of ceph the limitation will be eliminated. osd data = /var/lib/ceph/osd/$cluster-$id osd recovery max active = 3 osd max backfills = 5 osd max scrubs = 2 osd mkfs type = xfs osd mkfs options xfs = -f -i size=1024 osd mount options xfs = rw,noatime,inode64,logbsize=256k,delaylog filestore max sync interval = 5 osd op threads = 2 debug ms = 100 debug osd = 100
需要说明的是,在这个配置文件中,我们覆盖了一些默认值,比如:{mon data}和{mgr data},但是没有覆盖{osd data},因为ceph-disk貌似强制使用默认值。另外,pid, sock文件被放置在/var/run/ceph/中,以$cluster-$name命名;log文件放置在/var/log/ceph/中,也是以$cluster-$name命名。这些都可以覆盖。
四、生成keyring
在单机部署中点击打开链接,我们说过,有两种操作集群中user及其权限的方式,这里我们使用第一种:先生成keyring文件,然后在创建集群时带入使之生效。
ceph-authtool --create-keyring mycluster.keyring --gen-key -n mon. --cap mon 'allow *' ) F+ e, ^' _* w; Y1 l
ceph-authtool --create-keyring mycluster.client.admin.keyring --gen-key -n client.admin --set-uallow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' 0 W  E( V9 T( \! d
ceph-authtool --create-keyring mycluster.client.bootstrap-osd.keyring --gen-key -n client.bootstrap-osd --cap mon 'allow profile bootstrap-osd' ) G; W, s( z) Z- q# V5 A
ceph-authtool --create-keyring mycluster.mgr.node1.keyring --gen-key -n mgr.node1 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
3 u& C  L9 K! U$ I0 g! Iceph-authtool --create-keyring mycluster.mgr.node2.keyring --gen-key -n mgr.node2 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
; f% u4 ]6 s" X; d/ B  C, o/ `' ^2 hceph-authtool --create-keyring mycluster.mgr.node3.keyring --gen-key -n mgr.node3 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
% ?2 _' L) h3 G9 w5 Aceph-authtool mycluster.keyring --import-keyring mycluster.client.admin.keyring 9 F( u! I1 l3 Z) X  g+ q) Z. O* \7 {
ceph-authtool mycluster.keyring --import-keyring mycluster.client.bootstrap-osd.keyring & ^8 h7 v5 Q8 P7 g3 o
ceph-authtool mycluster.keyring --import-keyring mycluster.mgr.node1.keyring
1 t8 G3 J$ B& u; a+ w- \ceph-authtool mycluster.keyring --import-keyring mycluster.mgr.node2.keyring
* g  X* H% T( T7 h$ L' Nceph-authtool mycluster.keyring --import-keyring mycluster.mgr.node3.keyring7 T" [; d0 P; p
cat mycluster.keyring [mon.] key = AQA525NZsY73ERAAIM1J6wSxglBNma3XAdEcVg== caps mon = "allow *" [client.admin] key = AQBJ25NZznIpEBAAlCdCy+OyUIvxtNq+1DSLqg== auid = 0 caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [client.bootstrap-osd] key = AQBW25NZtl/RBxAACGWafYy1gPWEmx9geCLi6w== caps mon = "allow profile bootstrap-osd" [mgr.node1] key = AQBb25NZ1mIeFhAA/PmRHFY6OgnAMXL1/8pSxw== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *" [mgr.node2] key = AQBg25NZJ6jyHxAAf2GfBAG5tuNwf9YjkhhEWA== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *" [mgr.node3] key = AQBl25NZ7h6CJRAAaFiea7hiTrQNVoZysA7n/g== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *"五、生成monmap
生成monmap并添加3个monitor
monmaptool --create --add node1 192.168.100.131:6789 --add node2 192.168.100.132:6789 --add node3 192.168.100.133:6789 --fsid 116d4de8-fd14-491f-811f-c1bdd8fac141
; c- C! [2 g( X3 {& Qmonmap [plain] view plain copy monmaptool --print monmap monmaptool: monmap file monmap epoch 0 fsid 116d4de8-fd14-491f-811f-c1bdd8fac141 last_changed 2017-08-16 05:45:37.851899 created 2017-08-16 05:45:37.851899 0: 192.168.100.131:6789/0 mon.node1 1: 192.168.100.132:6789/0 mon.node2 2: 192.168.100.133:6789/0 mon.node3六、分发配置文件
keyring和monmap
把第2、3和4步中生成的配置文件,keyring,monmap分发到各个机器。由于mycluster.mgr.nodeX.keyring暂时使用不到,先不分发它们(见第8节)。
cp mycluster.client.admin.keyring mycluster.client.bootstrap-osd.keyring mycluster.keyring mycluster.conf monmap /etc/ceph scp mycluster.client.admin.keyring mycluster.client.bootstrap-osd.keyring mycluster.keyring mycluster.conf monmap node2:/etc/ceph scp mycluster.client.admin.keyring mycluster.client.bootstrap-osd.keyring mycluster.keyring mycluster.conf monmap node3:/etc/ceph七、创建集群1、创建{mon data}目录mkdir /var/lib/ceph/mon/mycluster-mon.node1 - o* V. _7 j$ `7 p* M; w
ssh node2 mkdir /var/lib/ceph/mon/mycluster-mon.node2   c; T5 C) }/ w1 |2 L5 ~
ssh node3 mkdir /var/lib/ceph/mon/mycluster-mon.node3
注意,在配置文件mycluster.conf中,我们把{mon data}设置为/var/lib/ceph/mon/$cluster-$name,而不是默认的/var/lib/ceph/mon/$cluster-$id;
/ Z% d8 P( B" J! y$ j: V7 D7 R$cluster-$name展开为mycluster-mon.node1(23);" }4 s; b' J' O5 A8 V$ r; o
默认的$cluster-$id展开为mycluster-node1(23);
2、初始化monitorceph-mon --cluster mycluster --mkfs -i node1 --monmap /etc/ceph/monmap --keyring /etc/ceph/mycluster.keyring
: m3 |6 ~, J/ r3 ?) ?ssh node2 ceph-mon --cluster mycluster --mkfs -i node2 --monmap /etc/ceph/monmap --keyring /etc/ceph/mycluster.keyring 3 X$ B7 J. {. Q1 @3 q1 ~
ssh node3 ceph-mon --cluster mycluster --mkfs -i node3 --monmap /etc/ceph/monmap --keyring /etc/ceph/mycluster.keyring2 k3 w3 \7 _  Q6 p% I) N# R
注意,在配置文件mycluster.conf,我们把{mon data}设置为/var/lib/ceph/mon/$cluster-$name,展开为/var/lib/ceph/mon/mycluster-mon.node1(23)。ceph-mon会
, e/ ^; c  b5 h3 u) v1 Y根据–cluster mycluster找到配置文件mycluster.conf,并解析出{mon data},然后在那个目录下进行初始化。
3、touch donetouch /var/lib/ceph/mon/mycluster-mon.node1/done 8 E7 c. m4 k( A
ssh node2 touch /var/lib/ceph/mon/mycluster-mon.node2/done 2 k2 ?; D5 S% A+ u
ssh node3 touch /var/lib/ceph/mon/mycluster-mon.node3/done4、启动monitorssystemctl start ceph-mon@node16 r: M# I; J" x- J( u3 ^1 R
ssh node2 systemctl start [url=mailto:ceph-mon@node2]ceph-mon@node2

5 |+ O4 C' |: V( b( ]ssh node3 systemctl start ceph-mon@node3[/url]# h, }# t& _8 p" U( |5 Q- u
5、检查机器状态ceph --cluster mycluster -s   T* X, W% y4 f# w, h$ \5 {
cluster: id: 116d4de8-fd14-491f-811f-c1bdd8fac141 health: HEALTH_OK services: mon: 3 daemons, quorum node1,node2,node3 mgr: no daemons active osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 0 kB used, 0 kB / 0 kB avail pgs:八、添加osd
每台集群都有一个/dev/sdb,我们把它们作为osd。
1、删除它们的分区2、prepareceph-disk prepare --cluster mycluster --cluster-uuid 116d4de8-fd14-491f-811f-c1bdd8fac141 --bluestore --block.db /dev/sdb --block.wal /dev/sdb /dev/sdb ssh node2 ceph-disk prepare --cluster mycluster --cluster-uuid 116d4de8-fd14-491f-811f-c1bdd8fac141 --bluestore --block.db /dev/sdb --block.wal /dev/sdb /dev/sdb ssh node3 ceph-disk prepare --cluster mycluster --cluster-uuid 116d4de8-fd14-491f-811f-c1bdd8fac141 /dev/sdb 注意:prepare node3:/dev/sdb时,我们没有加选项:--bluestore --block.db /dev/sdb --block.wal /dev/sdb;后面我们会看它和其他两个有什么不同。3、activateceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring ssh node2 ceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring ssh node3 ceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring
注意:ceph-disk好像有两个问题:
  • 前面说过,它不使用自定义的{osd data},而强制使用默认值 /var/lib/ceph/osd/$cluster-$id
    7 r" n3 w4 O0 a8 p* v7 _# v
  • 好像不能为一个磁盘指定osd id,而只能依赖它自动生成。虽然ceph-disk prepare有一个选项–osd-id,但是ceph-disk activate并不使用它而是自己生成。当不匹配时,会出现 如下错误:
    4 k' _4 a) m+ X7 q! D4 _" V+ R
# ceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring command_with_stdin: Error EEXIST: entity osd.0 exists but key does not match mount_activate: Failed to activate '['ceph', '--cluster', 'mycluster', '--name', 'client.bootstrap-osd', '--keyring', '/etc/ceph/mycluster.client.bootstrap-osd.keyring', '-i', '-', 'osd', 'new', u'ca8aac6a-b442-4b07-8fa6-62ac93b7cd29']' failed with status code 17
从 ‘-i’, ‘-‘可以看出,它只能自动生成osd id;
4、检查osd
在ceph-disk prepare时,node1:/dev/sdb和node2:/dev/sdb一样,都有–bluestore –block.db /dev/sdb –block.wal选项;node3:/dev/sdb不同,没有加这些选项。我们看看有什么不同。
4.1 node1
mount | grep sdb /dev/sdb1 on /var/lib/ceph/osd/mycluster-0 type xfs (rw,noatime,seclabel,attr2,inode64,noquota) ls /var/lib/ceph/osd/mycluster-0/ activate.monmap block block.db_uuid block.wal bluefs fsid kv_backend mkfs_done systemd whoami active block.db block_uuid block.wal_uuid ceph_fsid keyring magic ready type ls -l /var/lib/ceph/osd/mycluster-0/block lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:52 /var/lib/ceph/osd/mycluster-0/block -> /dev/disk/by-partuuid/a12dd642-b64c-4fef-b9e6-0b45cff40fa9 ls -l /dev/disk/by-partuuid/a12dd642-b64c-4fef-b9e6-0b45cff40fa9 lrwxrwxrwx. 1 root root 10 Aug 16 05:55 /dev/disk/by-partuuid/a12dd642-b64c-4fef-b9e6-0b45cff40fa9 -> ../../sdb2 blkid /dev/sdb2 /dev/sdb2: PARTLABEL="ceph block" PARTUU cat /var/lib/ceph/osd/mycluster-0/block_uuid a12dd642-b64c-4fef-b9e6-0b45cff40fa9 ls -l /var/lib/ceph/osd/mycluster-0/block.db lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:52 /var/lib/ceph/osd/mycluster-0/block.db -> /dev/disk/by-partuuid/1c107775-45e6-4b79-8a2f-1592f5cb03f2 ls -l /dev/disk/by-partuuid/1c107775-45e6-4b79-8a2f-1592f5cb03f2 lrwxrwxrwx. 1 root root 10 Aug 16 05:55 /dev/disk/by-partuuid/1c107775-45e6-4b79-8a2f-1592f5cb03f2 -> ../../sdb3 blkid /dev/sdb3 /dev/sdb3: PARTLABEL="ceph block.db" PARTUU cat /var/lib/ceph/osd/mycluster-0/block.db_uuid 1c107775-45e6-4b79-8a2f-1592f5cb03f2 ls -l /var/lib/ceph/osd/mycluster-0/block.wal lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:52 /var/lib/ceph/osd/mycluster-0/block.wal -> /dev/disk/by-partuuid/76055101-b892-4da9-b80a-c1920f24183f ls -l /dev/disk/by-partuuid/76055101-b892-4da9-b80a-c1920f24183f lrwxrwxrwx. 1 root root 10 Aug 16 05:55 /dev/disk/by-partuuid/76055101-b892-4da9-b80a-c1920f24183f -> ../../sdb4 blkid /dev/sdb4 /dev/sdb4: PARTLABEL="ceph block.wal" PARTUU cat /var/lib/ceph/osd/mycluster-0/block.wal_uuid 76055101-b892-4da9-b80a-c1920f24183f
可见,node1(node2)上,/dev/sdb被分为4个分区:
  • /dev/sdb1: metadata
  • /dev/sdb2:the main block device
  • /dev/sdb3: db
  • /dev/sdb4: wal: u% {/ P- ^/ H1 c. x
具体见:ceph-disk prepare –help
4.2 node3
mount | grep sdb /dev/sdb1 on /var/lib/ceph/osd/mycluster-2 type xfs (rw,noatime,seclabel,attr2,inode64,noquota) ls /var/lib/ceph/osd/mycluster-2 activate.monmap active block block_uuid bluefs ceph_fsid fsid keyring kv_backend magic mkfs_done ready systemd type whoami ls -l /var/lib/ceph/osd/mycluster-2/block lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:54 /var/lib/ceph/osd/mycluster-2/block -> /dev/disk/by-partuuid/0a70b661-43f5-4562-83e0-cbe6bdbd31fb ls -l /dev/disk/by-partuuid/0a70b661-43f5-4562-83e0-cbe6bdbd31fb lrwxrwxrwx. 1 root root 10 Aug 16 05:56 /dev/disk/by-partuuid/0a70b661-43f5-4562-83e0-cbe6bdbd31fb -> ../../sdb2 blkid /dev/sdb2 /dev/sdb2: PARTLABEL="ceph block" PARTUU cat /var/lib/ceph/osd/mycluster-2/block_uuid 0a70b661-43f5-4562-83e0-cbe6bdbd31fb
可见,在node3上,/dev/sdb被分为2个分区:
  • /dev/sdb1:metadata
  • /dev/sdb2:the main block device;db和wal也在这个分区上。
    : k7 \" ?" f2 @) S' Q
具体见:ceph-disk prepare –help
5、检查集群状态ceph --cluster mycluster -s cluster: id: 116d4de8-fd14-491f-811f-c1bdd8fac141 health: HEALTH_WARN no active mgr services: mon: 3 daemons, quorum node1,node2,node3 mgr: no daemons active osd: 3 osds: 3 up, 3 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 0 kB used, 0 kB / 0 kB avail pgs:
由于没有添加mgr,集群处于WARN状态。
九、添加mgr1、创建{mgr data}目录mkdir /var/lib/ceph/mgr/mycluster-mgr.node1 ssh node2 mkdir /var/lib/ceph/mgr/mycluster-mgr.node2 ssh node3 mkdir /var/lib/ceph/mgr/mycluster-mgr.node3
注意,和{mon data}类似,在配置文件mycluster.conf中,我们把{mgr data}设置为/var/lib/ceph/mgr/$cluster-$name,而不是默认的/var/lib/ceph/mgr/$cluster-$id。
2、分发mgr的keyringcp mycluster.mgr.node1.keyring /var/lib/ceph/mgr/mycluster-mgr.node1/keyring scp mycluster.mgr.node2.keyring node2:/var/lib/ceph/mgr/mycluster-mgr.node2/keyring scp mycluster.mgr.node3.keyring node3:/var/lib/ceph/mgr/mycluster-mgr.node3/keyring3、启动mgrsystemctl start ceph-mgr@node1 ssh node2 systemctl start ceph-mgr@node2 ssh node3 systemctl start ceph-mgr@node34、检查集群状态ceph --cluster mycluster -s cluster: id: 116d4de8-fd14-491f-811f-c1bdd8fac141 health: HEALTH_OK services: mon: 3 daemons, quorum node1,node2,node3 mgr: node1(active), standbys: node3, node2 osd: 3 osds: 3 up, 3 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 5158 MB used, 113 GB / 118 GB avail pgs:
可见,添加mgr之后,集群处于OK状态。

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2022-7-20 13:41:38 | 显示全部楼层
部署Ceph mon服务6 ?9 S* b, m( }4 h- t# {+ r; ?
安装Ceph-mon服务程序(所有设备执行)4 Y" ~+ b9 Q7 d0 ?6 s$ C7 x, Z5 p
yum install -y ceph-mon
8 v6 q6 P$ |2 I9 @1 v10 W1 l* e: P3 O( e" d! F1 o& W. V
初始化Mon服务(Ceph 01执行)
1 c; q: v. y3 X生成uuid: R; K8 f- x. F" H3 _7 Z
uuidgen6 H6 x1 i5 {0 X4 J; I
> 9bf24809-220b-4910-b384-c1f06ea80728+ q4 f7 L; |9 P) ^( |! ?- O3 ?/ \
1
0 s6 I; i4 d( c4 S3 v/ v2
$ o5 o4 M' j. r2 G创建Ceph配置文件
: x$ X& _  d: m1 }cat >> /etc/ceph/ceph.conf <<EOF
. O' d2 g/ O: E: C$ ?) j7 `' n' y[global]
% e7 U! g9 `, |, K  {fsid = 9bf24809-220b-4910-b384-c1f06ea80728
3 ?. n5 R2 |$ Fmon_initial_members = ceph01,ceph02,ceph03
# j9 ^, A% B6 f/ Z2 Z+ P$ _- Tmon_host = 10.40.65.156,10.40.65.175,10.40.65.129
' L4 ^6 |7 p5 {- a3 zpublic_network = 10.40.65.0/24
0 g& T5 r* n1 r- d, \! lauth_cluster_required = cephx
: w4 y" L) L( {2 r3 g( i4 I, K! ?  cauth_service_required = cephx
" y3 \; ^1 n% Q4 \3 k! i0 r8 Eauth_client_required = cephx! D. ]6 X+ |7 W, ~; P3 u
osd_journal_size = 1024
6 y4 M/ Z. }4 C+ g' Gosd_pool_default_size = 3- r* N6 y$ K! w# i1 u
osd_pool_default_min_size = 2
/ R5 {8 ]+ ]  G- F# Z, nosd_pool_default_pg_num = 64
# a4 i  J7 B9 [* w% z0 N/ m5 Rosd_pool_default_pgp_num = 64
6 ~" r8 F' b" N9 nosd_crush_chooseleaf_type = 1
6 W( a+ {9 b, zEOF
0 x$ h4 Z* i, V6 U  x1
0 Z! o4 V' h1 }7 ?6 z28 T. Q7 L7 a  z. v& r
32 S5 y1 x! s5 ^
4, f+ t" t! O; G5 p9 f, b9 r
51 D$ o# t/ q6 G/ y
6
4 T. H. j6 a. u% e6 s7
, O1 ?+ q9 e7 i7 G# A8
* m- l9 O$ v. I5 I97 r/ \: t( v6 `9 S9 C$ |! `; `) P+ n
105 h% h: C8 D0 D# a, I# H
11
- }/ ~8 `* Q1 ]  P; b12
& X5 A3 T0 ~2 P6 F) f- ]13
- g5 j6 u4 u, R14
; R2 o2 r  ~2 y- r% O3 e* S4 @15
9 ?( k$ E/ h: }16
0 U* ~2 j' _# ~& I" F' ^  K创建集群Monitor密钥。* ?: Z0 E9 Y& f% x! |. U. \. [1 f
ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
; D0 b% v( W2 r1+ }" A# u) g. C) A) R9 z2 ]" @
创建client.admin用户、client.bootstrap-osd用户密钥,添加到集群密钥中。3 G7 j! `# D8 S3 z2 W$ x
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'- W; k/ x3 y# X% D7 [& ~4 s. z% I
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
4 @/ M& U  n) G/ G1
! @; P' n: g6 U5 S7 ?/ k; Y* Y8 T4 I+ x2
, D" `5 T* C( X, Y$ u& [# Tceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
4 K( L  ~9 b- _0 ?( W; K7 Z: iceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring0 \" I3 l; I: w
1/ W$ W5 N" @/ G- @1 i
2/ O3 E: O* m2 s& N/ V! }
使用主机名、主机IP地址、FSID生成monitor map。6 Y3 f3 D+ d- h
monmaptool --create --add ceph01 10.40.65.156 --add ceph02 10.40.65.175 --add ceph03 10.40.65.129 --fsid 9bf24809-220b-4910-b384-c1f06ea80728 /tmp/monmap& U5 I# A0 J3 m0 Q
1
) R/ }3 G$ J# w: k! L初始化并启动monitor服务8 p" V0 o- o9 s5 h- A& h( q3 y5 T7 A1 S
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph017 z; J6 }3 Y1 ~3 y4 \
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
( H1 {0 u5 V; @sudo -u ceph ceph-mon --mkfs -i ceph01 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
! q6 `  f! z# @: els /var/lib/ceph/mon/ceph-ceph01/1 G2 M& K+ v/ R) H
1
8 A0 F$ O+ J6 _4 s+ D1 P2- A0 s1 t1 t! p& I
3
! h5 c1 F  ^$ ~4 y; @; \8 t: R( @4
$ {3 w2 Q/ L/ ysystemctl start ceph-mon@ceph01% J7 B$ G* q9 ?% e
systemctl enable ceph-mon@ceph01; X9 d. ?3 |1 b/ }0 F: t
systemctl status ceph-mon@ceph01
& F4 ^; G2 B; A" k1
' Q- ^' h( a- T) c2( Y4 i( x( g; s7 x! o
30 v' R; U# l4 R  _" A3 J$ Q
同步配置文件、密钥、monmap到其他节点中(Ceph 01执行)' v3 B. X# J1 ^9 U
复制ceph.client.admin.keyring、client.bootstrap-osd key、ceph.mon.keyring、monitor map、ceph.conf到另外2个节点6 f" g1 U! h5 l# G# i
scp /etc/ceph/ceph.client.admin.keyring root@ceph02:/etc/ceph/( I! c( }7 b: s. p" T
scp /etc/ceph/ceph.client.admin.keyring root@ceph03:/etc/ceph/
- w6 R* X( b( y6 I# [. B1
3 K' W; w+ \/ U* x( W5 H2
* o! L  b5 T; ]! Zscp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph02:/var/lib/ceph/bootstrap-osd/
5 s3 s- Q* z6 `  q1 `scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph03:/var/lib/ceph/bootstrap-osd/
8 p6 t" t2 ^- ~8 p9 X1
4 F" k2 H/ I* v" Q9 P" q- v2  n; J7 a( L; r0 q% b
scp /tmp/ceph.mon.keyring root@ceph02:/tmp/
3 f7 ^4 i6 |' V/ ~: f0 N* Kscp /tmp/ceph.mon.keyring root@ceph03:/tmp/2 l+ H+ n% @7 W0 @3 V
1" w5 x+ t$ G9 t/ A/ O" [. \
2/ ^# c# f0 B1 U# l
scp /tmp/monmap root@ceph02:/tmp/
: [- M" x$ ?* K8 J; Z) X1 cscp /tmp/monmap root@ceph03:/tmp/% V+ L4 R; ~4 d4 x* F
1) B* p, F5 _7 g/ l
2
- H% _4 f) {- g0 t% Hscp /etc/ceph/ceph.conf root@ceph02:/etc/ceph/
1 S6 k) A9 B$ ~* v$ L% h* uscp /etc/ceph/ceph.conf root@ceph03:/etc/ceph/
% y9 T/ \2 H1 [3 W$ K( J1
, T( \5 b) N! D2
0 \) x9 \% }9 H启动其他节点的monitor服务(Ceph 02执行)
# x/ S; i1 e; Jsudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph02
, X6 b& r3 ^0 c+ \4 kchown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap" z- l; U4 ^# N6 Q! m& u# |
sudo -u ceph ceph-mon --mkfs -i ceph02 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
* O9 v( P  Z( `# `. i& J) @ls /var/lib/ceph/mon/ceph-ceph02/
( W" s# Z. S2 P. p1
( k. d+ l: h$ c2' n9 ^9 H3 l3 ?1 w6 E7 I7 [
35 L$ O; m/ ^: H! A, c0 K+ o3 F
4. I* W$ h5 K% A$ H
systemctl start ceph-mon@ceph02
% p4 a9 F8 o: x) M  V& Bsystemctl enable ceph-mon@ceph02" r% S" a- C! E  s
systemctl status ceph-mon@ceph02
* y/ j8 [9 ]$ C; B, }9 H1
0 H2 g4 B9 j1 e7 h9 S! ~5 a29 |1 c6 B0 _" @8 O
3
; \) w( e4 `* u6 `启动其他节点的monitor服务(Ceph 03执行)/ t6 I! ]; b7 L3 b" H7 u1 l/ W! m
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph031 h/ \- I6 A: {8 H; ?, z
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
, b- _- h# S# X7 ysudo -u ceph ceph-mon --mkfs -i ceph03 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring/ k4 B' \9 X2 w6 m
ls /var/lib/ceph/mon/ceph-ceph03/0 p7 t& B# b) @( d
1
5 H+ z1 L4 ?8 w2 X2) O8 U+ g2 H4 ^7 g1 q# I' q1 w8 I
36 E; x* i: A& I) z; G& e' r3 i2 c# E
4/ i, E2 ^3 y$ K, \3 j9 c8 D
systemctl start ceph-mon@ceph03) y  ]: w" d) f; M0 E! z- _7 d
systemctl enable ceph-mon@ceph03
" a: D2 I5 ], Q4 G0 fsystemctl status ceph-mon@ceph03( q8 R: Q1 a# z5 h. e
1
" K  V5 _; u& Z( R7 L$ u8 g0 [2
6 C# P( n) r% z. J: ]$ B7 I3  J2 H$ z- n/ e3 X) _1 Z% r, w/ k) c
查看当前集群状态(任意节点执行)
$ v2 Y8 ]4 H6 O! {( s: X通过ceph -s命令查询集群状态,可见集群services中3个mon服务已启动。5 d& |1 ?. {8 h, p/ j4 e
ceph -s
% ^9 D  @% e2 H; ?' N+ z> cluster: # l1 R0 G' M  i$ h! n5 p4 G
>   id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a  
/ b* b# ^* C* n9 f9 k>   health: HEALTH_OK
+ ]# |0 b( j' T' X/ u1 L. c> - \2 s' g- s' ^/ d% N% R0 t! b4 ?# H3 E
> services:  
2 e2 y4 a( K# S0 C! {4 U>   mon:  3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)  
! z2 a- G/ s9 S& F$ A5 i/ j>   mgr:   
% X0 T3 a: E2 S0 W: |4 S>   osd: $ h' @( m' W8 w! i" w5 u5 k
>
( i# D" J$ d% i9 @7 f> data:
: V6 g* I) d& n8 r- A>    pools:   ( f: V* L2 a0 z" r. `
>   objects: , c/ h8 j3 u3 d  a/ `* H1 t/ ~
>   usage:   
8 C  h5 b; s' t0 J2 Y3 x* C>   pgs:
& ^" `! z( M- ~0 d$ P$ f. _2 V10 z& n% ?' |7 U# x$ x! W6 ?4 k5 O
29 q0 |4 A# F- N! \$ y7 o. P, U
35 y: @6 |; X& q4 k1 q/ |+ m$ C
4. }2 p! {! m$ T6 |* M# V9 V4 {% ]
5
1 H$ X$ ?* F) E: V# K% U0 O6" R/ l0 t! Z/ H  Z* @1 i  |
7# {# k7 d, {/ G2 A& u. G4 H
8
7 o0 z/ m% x- c8 n9
! L# B$ U; P5 {9 o10+ F, _) u) u! [( p
11
6 ?, ^4 \" [3 U; H- p2 A" X1 T3 D& u12. u4 Q0 _5 l9 I& S* d
13
( F$ j- I- |3 d; ~5 H5 \! t" P8 G14
) U5 X* K0 }9 d) A( e8 X- q157 o( @' H9 N$ L+ }; D
部署Ceph mon服务(ceph-volume 自动化创建)
" B6 _, N% w' ]. Z& i6 w+ E1 `安装Ceph-osd服务程序(所有设备执行)+ s9 i" `6 {( v3 _3 E
yum install -y ceph-osd. ]9 E: X4 o7 H+ n+ w, R8 o5 Y
1
! ~6 j2 N4 ^' c$ n4 D9 t初始化osd服务(所有设备执行)6 [! f8 [, V- E) e; C0 V
通过fdisk等工具查看磁盘盘符,然后利用ceph-volume工具自动化创建osd服务。" _+ ~* ]* n, w  s9 z7 G1 l
ceph-volume lvm create --data /dev/sda
" A+ H, _# ~  g6 mceph-volume lvm create --data /dev/sdb
, G; y1 V4 W0 D5 Cceph-volume lvm create --data /dev/sdc0 U0 {1 _8 b2 k! U$ H3 o% }0 |8 w
1) E/ f& S+ }5 u: V+ u( E, H! g
2
. F( h; R8 e0 R( h) n6 {- X3' P, [* T5 j- ?
查看当前集群状态(任意节点执行)
4 W. {4 b) ]' {2 ^) }0 \5 r: `- i通过ceph osd tree命令查询集群状态,可见集群services中所有osd服务已启动。
3 ?" f& u5 x/ A# Y) I4 |ceph osd tree
8 o3 ]8 E/ X% }- ^; O. z! j. C> ID CLASS WEIGHT   TYPE NAME       STATUS REWEIGHT PRI-AFF  T( J$ ~& Z3 A: J. u: F
> -1       16.36908 root default# v3 ?& `* l  ~5 }: e
> -3        5.45636     host ceph01
8 G  x6 W; @4 b6 e4 `>  0   hdd  1.81879         osd.0       up  1.00000 1.000004 m4 i& l( g8 B
>  1   hdd  1.81879         osd.1       up  1.00000 1.000007 c% [/ _  k8 V5 |
>  2   hdd  1.81879         osd.2       up  1.00000 1.00000' I6 Z' g. K5 ]2 ~9 Y( Z# `' C
> -5        5.45636     host ceph02. ]- c: P+ @8 b( s- r* f
>  3   hdd  1.81879         osd.3       up  1.00000 1.00000
) L5 R- k0 d. Y>  4   hdd  1.81879         osd.4       up  1.00000 1.00000
$ ^% `: d* x- t>  5   hdd  1.81879         osd.5       up  1.00000 1.00000
0 a' e  Z. n$ a5 l1 z4 ^3 y> -7        5.45636     host ceph03$ ?, H$ Q5 h. ?; k& R- K
>  6   hdd  1.81879         osd.6       up  1.00000 1.00000
; A+ [" L: N( q>  7   hdd  1.81879         osd.7       up  1.00000 1.00000" h, f' E7 x, ^) e, t
>  8   hdd  1.81879         osd.8       up  1.00000 1.00000
8 A. I. l8 Y& j1& a+ O* r3 D9 C! g8 l
28 V  H: ]  h2 y# T6 O* t+ F5 w
3
* B' H! ?2 {# X7 {2 F' j# l4
. q9 l: e: z! t5 v6 j% ?3 }5
' s/ \5 a6 q( {+ r( C6 `6
# r% k/ T% L6 J+ c7
  R; F' i9 f* r; x% P  q2 w  u+ d80 x4 ]) g- y( ^' ], V
9
/ o  w% B: n  B7 x10) k- v: N# s) s- c
113 E# O. _8 V6 r9 ?3 Q; d: a1 _7 }
12
: _6 s: V% K  _. l6 }1 P* A: o13- d3 Z+ r( U, w
14& ]' J* G0 N! M, ^
15
& `- w6 u8 v) r2 i4 a, Q部署Ceph mgr服务并开启Dashboard8 e' f7 j' m" D" a) d$ m
安装Ceph-mgr服务程序(所有设备执行)
; U+ t2 @# \+ z, q) K* l6 r4 t; ?yum install -y ceph-mgr
- x, V2 n; e- m1 Y1! O1 ?$ G" p0 w; l4 G7 l9 e
初始化并启动主MGR服务(Ceph01执行)
5 M7 b+ [9 [5 }8 {& }7 B" Qmkdir -p /var/lib/ceph/mgr/ceph-ceph01. w9 V$ c! v8 U+ J/ U
chown ceph.ceph -R /var/lib/ceph1 m3 O! h5 m% v# e
ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph01.keyring --gen-key -n mgr.ceph01 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'. g( \9 U1 q. f
ceph auth import -i /etc/ceph/ceph.mgr.ceph01.keyring
) o9 Y' i7 [) tceph auth get-or-create mgr.ceph01 -o /var/lib/ceph/mgr/ceph-ceph01/keyring4 E4 Z/ E/ I! h$ k( J
1
- k6 i" n( V- A% U* q9 C$ m7 o29 I9 q0 Z3 F& Y: a0 @% o
3* }& X  c7 J; v' s5 M( G
40 P0 P+ I5 p- r* d3 |
59 W: u8 V1 J7 E2 ~4 ]# Y0 L
systemctl start ceph-mgr@ceph01# K9 J+ l6 e  L; f! }
systemctl enable ceph-mgr@ceph01
' N  W3 d) {/ b! Bsystemctl status ceph-mgr@ceph01
3 [3 t+ p0 `$ T' L0 u1
/ P- p' m/ _% |. s* m% t! O2
( U1 k. G9 i4 H+ u3# X7 P2 t+ @! R/ \& ]' g
初始化并启动从MGR服务(Ceph02执行)
8 L8 U3 [+ q8 F2 L) x2 dmkdir -p /var/lib/ceph/mgr/ceph-ceph02
4 j( m5 V$ \( R# u* H% _chown ceph.ceph -R /var/lib/ceph
$ h$ D1 i% i) yceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph02.keyring --gen-key -n mgr.ceph02 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'. l2 G( F2 R* d
ceph auth import -i /etc/ceph/ceph.mgr.ceph02.keyring; q& ^) d6 G1 w  C7 z  J
ceph auth get-or-create mgr.ceph02 -o /var/lib/ceph/mgr/ceph-ceph02/keyring) j0 N" F2 S3 L' c+ E2 e/ d5 i% \
1& m+ |7 k4 N& d5 B* k' x
28 h. L  Q" D( I, p7 x/ v
3
  k3 a9 x( ~% c8 r  n$ Y49 {) b. ]: B6 L. J8 {6 Q, T
5
* f  H( `+ e7 G& Bsystemctl start ceph-mgr@ceph02
; P$ M" J% e: _9 R0 R$ m4 R+ w: `systemctl enable ceph-mgr@ceph02
; N9 T# N2 s" k" nsystemctl status ceph-mgr@ceph02  ^8 W' f5 {- r/ e* T$ ^# K
1
, }' k4 H$ F) o, \, J; F/ y5 s27 W. c$ i5 H; h( o& u1 i
3, K4 B  {3 i: K% y5 D- n
初始化并启动从MGR服务(Ceph03执行)/ m9 A# `' h- \/ d7 I
mkdir -p /var/lib/ceph/mgr/ceph-ceph036 `- G) l& r0 B; X8 C! v' O& j# \7 Y
chown ceph.ceph -R /var/lib/ceph, B( g: j" {/ ?
ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph03.keyring --gen-key -n mgr.ceph03 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'1 k$ K4 Z; e$ c, h) d3 Z: K
ceph auth import -i /etc/ceph/ceph.mgr.ceph03.keyring( s- K; X! g( L( `
ceph auth get-or-create mgr.ceph03 -o /var/lib/ceph/mgr/ceph-ceph03/keyring
! i2 F, }; |2 W* r1 F) m1
3 b0 ?+ `2 H  R- `" `( M23 ]! Y9 r  N1 |/ ?5 v% Q; l9 K
37 U$ h+ `9 d* q  w/ q
4
" K. |; D" `8 z" r5! E" z' I# Y9 `! S) U/ f/ D
systemctl start ceph-mgr@ceph03* C& |' Q. ~2 `. k  z% V% f% Q
systemctl enable ceph-mgr@ceph03
- N* \* I0 X1 T0 x, i' msystemctl status ceph-mgr@ceph03$ c$ c8 e3 L* j# t) X4 w
17 p$ A, E& E; R0 Q
24 b9 c1 d' p5 H
3) B- W% Q# h. U( u
查看当前集群状态(任意节点执行)
- c1 e5 S* O7 K6 Q通过ceph -s命令查询集群状态,可见集群services中3个mgr服务已启动。  ]7 n; \* Q, u3 |8 l9 c
ceph -s
" \" N6 b2 d* N5 J4 x> cluster:
1 `/ W: K, m2 }2 t) P>   id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a  7 h) T; G+ ~" @# A/ F
>   health: HEALTH_OK
) u. n- c( x' Z8 l/ K: a> ) A! L8 O. z: i
> services:  ; _" \& y$ H5 [# z' D
>   mon:  3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)  , N* ^' R, V, x( S/ p- d' E2 x6 y
>   mgr: ceph01(active, since 3w), standbys: ceph01、ceph02   
: j' E2 D8 {: \3 g& J3 u>   osd:
  Q( K& s  @& o4 |( l# t>
# n2 g" r/ }6 C> data:+ p0 w* s( |* E7 w' z* e
>    pools:   + i. n# ~, B) k# _
>   objects:
; i& z. @. Y% P' f1 f>   usage:   + z9 G7 ~# [3 @
>   pgs:+ l' P8 |& j1 r6 O$ q
1/ b9 z/ m* _) t1 o+ R$ b# y6 D
2% M1 G* p( e5 {7 [
3
5 |. @# ^9 n4 f  h$ I6 U4 W4& ~3 k3 i* V, C, I: B" }6 C; J
58 t- R. f# H/ K! a
6
% l: u# s! _, t& _7 A7
0 l1 e4 g0 {* v, @( n8
9 R+ Q! p/ Y# s* D" t) q( L9
# i, w7 J& ?6 p" \+ J4 \* x# P10
) G0 A% @4 p! e" Z  {0 X4 ~; p/ b4 |11
* r2 R# x7 ~1 ]4 O% ^12
9 G3 A* r, U/ L( p) G* q, k13, N  Q8 x2 n+ G: p# U$ w( Y
14- N7 y# u- J9 N3 z( u
15& z" }/ z: j' v6 N$ q& l
使能Dashboard访问功能(任意节点执行)
4 C  A0 K  V" z  O5 O开启mgr dashboard功能( [6 ?% B2 I5 h! I: {( R
ceph mgr module enable dashboard
. ]+ e2 @# D. G# w0 c$ X1
8 j8 O" N8 t9 Q( d1 Y8 g  e: e4 d生成并安装自签名的证书
! H. g& f& k" l, o& x/ sceph dashboard create-self-signed-cert3 w+ z' z7 L* z5 A6 J) E
16 ?1 M1 L0 z* j1 u: M* ]" c7 [
配置dashboard+ P" {& j- Q" A9 G7 [
ceph config set mgr mgr/dashboard/server_addr 10.40.65.148
- y  J6 D1 p- g. Iceph config set mgr mgr/dashboard/server_port 8080
3 R) ?, R( j" q! S+ ^, N9 a" X* L" lceph config set mgr mgr/dashboard/ssl_server_port 8443
. e$ n9 R& q" U) ?# c1/ U% S1 e8 w* j5 s+ ^3 f4 S- g1 ?
2) i; M; q6 l( }7 N
3' c. w% B8 |. A0 K, V/ J$ N
创建一个dashboard登录用户名密码! }5 I- m  c& E& V
echo '123456' > password.txt
3 F7 m$ ^( Y4 J) G) d; n: i% A0 Cceph dashboard ac-user-create admin  administrator -i password.txt
( f, b" v( B- J% _1- I; \6 m8 u1 X( g' [
2% @. ~0 S! _  U' v& Y8 |
查看服务访问方式
# L. z' Z3 Q: r& ]3 _; Gceph mgr services5 T: j/ K7 o# I4 v4 t
1, @" I2 |" a+ q% U
通过web访问Ceph Dashboard,用户名密码为admin/123456
+ ?" N5 r' f3 B0 K, H. u$ T4 Dhttps://10.40.65.148:8443
: w1 }. g; K3 j8 p9 |# V& O8 `. L2 l. L9 k- b

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2022-7-20 13:47:56 | 显示全部楼层
部署Ceph mon服务
" g0 A) s1 O0 N4 T, E1 e, G安装Ceph-mon服务程序(所有设备执行)
2 L5 Y9 H! k% ~
8 y" C9 e3 T- m2 {: _, y! M' uyum install -y ceph-mon' _; G: x3 Z/ |5 E' j% ~
1) ?. B0 B' g. n* R7 D  J3 q
初始化Mon服务(Ceph 01执行)4 H- i6 I7 W: {2 _1 Z/ k

: m) [, I0 J/ O( j. C8 P生成uuid
1 y8 z8 A' I; U% a) s  U, `; M$ o' r
uuidgen2 h1 Z( }) |1 L' X/ @& V  v% K$ n* U
> 9bf24809-220b-4910-b384-c1f06ea807282 z1 ?) E' o; X
1, r% }$ l* L9 U  D
27 s9 O+ L3 e" }. z7 _: ?9 E2 @
创建Ceph配置文件2 b% t' Y" k$ L; B1 |+ Q
2 l) S" r: G% p; L; n6 Z: ?
cat >> /etc/ceph/ceph.conf <<EOF6 E/ V4 L0 D, A, t7 Y, o
[global]4 S$ c4 S; U# F6 b8 c2 G1 E
fsid = 9bf24809-220b-4910-b384-c1f06ea80728( Q) ]9 _7 y: Z8 E7 t
mon_initial_members = ceph01,ceph02,ceph03
9 a5 G1 k, h- r8 C$ vmon_host = 10.40.65.156,10.40.65.175,10.40.65.129
7 C% k9 y! x! e: u' [& y6 c( @. ?public_network = 10.40.65.0/24
0 l8 [' C% j( R' Hauth_cluster_required = cephx
  \2 m" y+ |0 lauth_service_required = cephx
  p. S; }: P1 W& rauth_client_required = cephx. J( C8 w' E( y+ j/ ]( P
osd_journal_size = 10241 @( I; q4 V! E) i
osd_pool_default_size = 36 L7 B3 e. ^" J, v) L2 o( j1 y6 d
osd_pool_default_min_size = 27 S* d( V" n: F# u; H2 u
osd_pool_default_pg_num = 64
; y# B/ ^  h# _( Aosd_pool_default_pgp_num = 642 E7 }9 \3 j! v3 |. Y
osd_crush_chooseleaf_type = 1
" p; F8 l% w: gEOF
+ b3 U" o0 x1 D- P- |$ H1& D9 J3 D! o9 K( h# k
2
! r; D6 f! V" c  o# N' I: S/ D3
8 H- E  s8 w7 ?+ J4
6 U. W# v' D6 s5
) Z4 x! b6 |4 N, n! F, z6
' {! ~5 o) F. ^7* Z5 c; c# f. I* d# ~7 a
8  p: S5 Y' Y& B
9
' p, K0 C2 a8 C- H/ g9 |105 k% A' ?0 f; k* [7 D7 C
11. D3 D# u8 l4 Y, i" R
12
- D2 N$ j" S; m* i- ]0 C7 _$ ~$ y13, W& q9 D$ ]" |: R
14
4 W' c6 g5 [0 C15# E. b" m3 u* m7 b
16- W; b# y. S- O3 r7 h: c- ~
创建集群Monitor密钥。
! `+ j2 G) v) V& t  e+ X" h* u2 \1 T) }* Q+ F
ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'3 I# u& ^) ~' k& T; }9 A! X
1
6 e, e- B3 _8 e创建client.admin用户、client.bootstrap-osd用户密钥,添加到集群密钥中。
# K+ G# x4 T5 @6 l; y% S# K; s& f5 s, d7 L' p8 X
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'* C, e8 S' @3 n. N) T
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
# U0 M7 K3 q- x3 R- e1# D2 l* r5 [" ~$ j, P4 a3 b
2* n  K1 T" s9 A* O
ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
1 M% i" p; S* m: D3 i9 f) C/ w$ xceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
1 L! Q3 B0 _& N/ _$ O" b$ N1) }, W1 l! S+ f# X
2! i; A4 {7 Q5 a1 U. k# D
使用主机名、主机IP地址、FSID生成monitor map。- J: N% j# P' M$ _0 ?3 R
" w3 c: J6 Z1 s! y
monmaptool --create --add ceph01 10.40.65.156 --add ceph02 10.40.65.175 --add ceph03 10.40.65.129 --fsid 9bf24809-220b-4910-b384-c1f06ea80728 /tmp/monmap$ e* F8 w- a; ]  x$ e/ k
1
' w8 g  ]* A$ }% a. U初始化并启动monitor服务
8 t* ?2 b: s2 W; [& c' e* E5 S' y# H3 s* ?' h$ v/ U& d- r8 ^
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph01
- s+ T- N( r7 J. b# v( H- ?chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
" I: d! e9 C) }- r/ |2 N. fsudo -u ceph ceph-mon --mkfs -i ceph01 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring8 |2 z7 M; w; }9 K/ h* c1 T* h
ls /var/lib/ceph/mon/ceph-ceph01/7 V6 \8 Z: Y% I- f# m7 W
1
1 n- I" V) N, m0 }# U2) C3 M$ o; d8 t! _, j% g
3
2 e/ B) P6 m% ~+ R' l* m4
% Q3 t; g: }) q  v9 ~& Dsystemctl start ceph-mon@ceph01) w1 [8 K6 b8 f# G
systemctl enable ceph-mon@ceph014 s' @+ [! [* c, |) |
systemctl status ceph-mon@ceph01, b( L/ z% j3 N- G( M* h
1
: ^9 X: v- u( {/ T* G2
# d1 a: r1 b3 i  j3 a5 D2 N  |$ h7 e3: O$ ^7 B. k4 |- a0 Q1 G
同步配置文件、密钥、monmap到其他节点中(Ceph 01执行)
# A, h6 G( J$ S2 C7 [1 x8 R' o( L6 C( ]
复制ceph.client.admin.keyring、client.bootstrap-osd key、ceph.mon.keyring、monitor map、ceph.conf到另外2个节点9 O3 j/ H0 |5 u# @( r$ A

" v  \3 d+ K8 q( C% Iscp /etc/ceph/ceph.client.admin.keyring root@ceph02:/etc/ceph/
9 T/ S3 ^- L6 k* R% Escp /etc/ceph/ceph.client.admin.keyring root@ceph03:/etc/ceph/" Y& }" ?0 }$ ^2 p; i5 M
1) i9 O# F$ o) c( b
2
/ G$ c! F- L6 p, ?; N3 U& Ascp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph02:/var/lib/ceph/bootstrap-osd/) m% X( `. R" _
scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph03:/var/lib/ceph/bootstrap-osd/
1 m; `9 V: i: l# z1
" k( u' \4 U/ ^2( g! [6 R9 J$ q$ f" i
scp /tmp/ceph.mon.keyring root@ceph02:/tmp/! Y5 D' C$ T4 L4 ?6 A
scp /tmp/ceph.mon.keyring root@ceph03:/tmp/( x6 z6 m+ o" X8 P  ]
1
$ ]6 h5 ?, F, T) U9 y( d  j2
) |$ Z6 H. S( ]scp /tmp/monmap root@ceph02:/tmp/
: [- O, E, ~" a( W% `scp /tmp/monmap root@ceph03:/tmp/& b) K9 u' y4 f. O+ n
1
7 `0 a- T, m; M  v5 M7 V5 I* s* A8 x2
7 z7 r$ p) b, h6 ]4 f* Y. p+ Nscp /etc/ceph/ceph.conf root@ceph02:/etc/ceph/
2 N. Y4 ?" ?, L- ~! h# }scp /etc/ceph/ceph.conf root@ceph03:/etc/ceph/
, S% P+ H% E1 l1
9 t% R9 r: Q: c0 P; f2" K, X6 A' p% ~% ?) k
启动其他节点的monitor服务(Ceph 02执行)1 ^8 d7 L# n5 t) g! i- x
+ W  D& [9 d* d& f8 J" G
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph02( J8 C+ \" w5 C7 I
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap7 f" g' s+ \$ S- Z5 `
sudo -u ceph ceph-mon --mkfs -i ceph02 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
! o. `! l) ^" \8 e2 `6 G% E  Cls /var/lib/ceph/mon/ceph-ceph02/
2 x/ O8 t+ V& V) Q3 C( H/ y1" |; D9 }0 E) l/ q( v+ K
2! H9 ~6 W% p# M  x3 t' S) s
3
: J& c- |0 |% s8 `. h43 A. W* N6 I! U& u0 n1 k. j
systemctl start ceph-mon@ceph02
5 g4 `- b" I) b  |* t" msystemctl enable ceph-mon@ceph02
, ^. [, b) d% I( Asystemctl status ceph-mon@ceph02
! ]+ V9 E) b& F  ?/ `1
! S! {" {2 Z- U$ a" ]7 E* S$ s: u2. B" Y" d$ M% D& c
3' \3 ?; a& f) w6 b
启动其他节点的monitor服务(Ceph 03执行)% W1 {. b4 p# N% b% `; `
5 {  h  Z: \4 R
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph03$ @) d: _$ ]7 `  G4 I/ ?, ~% E
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
* |( b% @. U  P" {sudo -u ceph ceph-mon --mkfs -i ceph03 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
& c4 ^$ R! p6 K1 t$ Q7 Uls /var/lib/ceph/mon/ceph-ceph03/
5 U9 u# r8 E: l9 `# k1 g1
/ I, @- I$ X, |4 i  T2
& f. c# W$ M0 A" u6 \38 s* p7 K: p9 ^: C/ @* F' |- }
4
) v# U" Z9 N; h+ e* Zsystemctl start ceph-mon@ceph03
8 b. r3 M4 `9 [) X& J. Msystemctl enable ceph-mon@ceph03
: m8 p) q; S6 k6 tsystemctl status ceph-mon@ceph03
4 m4 k$ y" S9 {& f' m% C3 K9 `1
" c; }& o( ]8 O2$ X& j* ^  e- z1 M  E& ]/ I
34 w( u2 f4 K5 K* J# V
查看当前集群状态(任意节点执行). M- ?5 k+ ~9 g  Y& v
通过ceph -s命令查询集群状态,可见集群services中3个mon服务已启动。
* O5 l' g) W  [7 V
  T9 M: ^% y6 Z$ U6 W$ N1 Oceph -s
* B) p% z1 L+ Z# d. |) {> cluster:
% u" E, |  t2 A. o" F>                 id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a         * r( T/ F  s/ _  J: g' X
>                 health: HEALTH_OK  p1 H0 a- `! f* b. J, V# d, c' Z
>
: K* w! N, G* |2 ]7 G7 k" e> services:        
4 d) m- j" Z, ~% j>                 mon:         3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)         2 F7 {4 G2 e3 u# l) S! g% J  W3 ]
>                 mgr:                               
- _1 Q9 O7 P5 h0 V>                 osd: 4 z! V6 E5 Z4 b/ \% j) X- ^
> . i$ w& N! c* E* R
> data:3 W+ o" s% b7 d( M" y1 d. z
>                  pools:   # r# y3 h5 B# Q
>                 objects:
1 I3 i5 k5 y: T* P7 K/ ^% T  D>                 usage:   
$ x% d$ c& U) B: X9 c>                 pgs:
: j& w0 T) Q5 ~14 R  \( _) Y4 Y* C7 d: O# J
2
2 ~" g* _8 U% i6 k+ t$ f+ F. J& B3- ?4 E* U3 Y2 _, S
47 e( A7 b( B9 _
5) x$ a. i4 k+ d# Y
60 t* Q& b6 w% h' _- G; s
7
1 n$ o+ P0 M  q) d  c. b8
* I0 \0 k0 U, D: K* F9
) [1 v. H8 r8 z3 _* A106 }: l% I9 F7 D0 S7 {
11
0 y9 T8 |5 q. G2 O% o" R$ _$ t12& _1 Z! X0 ^3 r9 n0 n; q/ D
13: R' R8 k5 C! o% I& ]( b
14* M+ O8 U6 c6 ^
15. q9 f- ^5 f: b5 m8 l6 W" t
部署Ceph mon服务(ceph-volume 自动化创建)
) g9 T9 p* ^4 l7 |  F安装Ceph-osd服务程序(所有设备执行)
- U7 f' G# c! M( ?4 Z/ \8 Ryum install -y ceph-osd
0 i( q/ P3 C; Q2 T& b+ t# @1
0 }1 u0 w8 l2 l1 V  |5 [初始化osd服务(所有设备执行)
- q5 ~2 Q% y$ L9 m) v( B& E通过fdisk等工具查看磁盘盘符,然后利用ceph-volume工具自动化创建osd服务。! m0 G3 j, s( L! B- |4 ?, c( ?0 ?6 Y1 \
ceph-volume lvm create --data /dev/sda# a( |  s) }( |6 ~
ceph-volume lvm create --data /dev/sdb7 E6 v/ h' b& M+ H. K
ceph-volume lvm create --data /dev/sdc
2 x: p4 U5 X' k# t4 {1( b  ^' A1 l" k" E
2
4 Q: k. t  u& |, e4 J$ D$ ?) k3
2 h% `' t# y- \9 K查看当前集群状态(任意节点执行)
* [& k/ R7 T2 v1 `; l0 D- Q3 ~# P5 _通过ceph osd tree命令查询集群状态,可见集群services中所有osd服务已启动。
$ C2 v9 ?# e- c: @* O% Bceph osd tree# P0 r. l9 v' f' I  z
> ID CLASS WEIGHT   TYPE NAME       STATUS REWEIGHT PRI-AFF! H; C0 c) i6 j$ k( q/ c
> -1       16.36908 root default
; A0 Z8 @" U* B' _* @; O2 c> -3        5.45636     host ceph01
( v) _0 F- W: z4 H7 p>  0   hdd  1.81879         osd.0       up  1.00000 1.00000
8 c$ b$ z" c& f* O: W% m8 u3 m! X( E# `" a>  1   hdd  1.81879         osd.1       up  1.00000 1.00000; s2 Q' @9 e  L% \& F& U
>  2   hdd  1.81879         osd.2       up  1.00000 1.00000' p# c& T6 }* c# r) U8 n* e8 W( [
> -5        5.45636     host ceph020 z6 g8 O+ r- [0 d" ~/ m3 S
>  3   hdd  1.81879         osd.3       up  1.00000 1.00000
/ L7 A  J& H* n. }; [  k>  4   hdd  1.81879         osd.4       up  1.00000 1.00000- D% O, p/ H$ z; v
>  5   hdd  1.81879         osd.5       up  1.00000 1.00000% u! Z% s* Z& I/ {0 l' O
> -7        5.45636     host ceph03' \9 K' [% [) c$ R3 x2 I- p
>  6   hdd  1.81879         osd.6       up  1.00000 1.00000, E0 Y  z" y) \4 C
>  7   hdd  1.81879         osd.7       up  1.00000 1.00000
# v8 ?8 p" a6 F: g+ }% X>  8   hdd  1.81879         osd.8       up  1.00000 1.00000& E6 W* S+ v( W' b/ _
1, f, E# S/ f: f0 e
2
$ ?& y# q8 Y0 J5 T% a" D3
! X, i$ M) H/ K' K* [1 m4
! D+ W$ i. E* `" @/ S5% B6 a1 p; X' o  q+ a" @
6
0 O& C- F& @- N3 S+ q* A1 R+ \5 A7
1 D) ~9 H8 D7 |: T; r- q8
2 G4 \% o- X2 m0 Q- k/ w: b9, u# S! [5 h7 h3 Y  z4 p# k
10  k% A$ E1 ]# |6 F
11
% K$ @5 Y8 u1 r' m12* @8 z5 x# D9 ?+ a% \, t0 _4 H
13
% F* Z! B) M3 @, p; w0 @# _146 r+ @8 b3 O0 ]& [/ B5 M, h, w
15
+ J. J1 T( _$ k/ ~" G部署Ceph mgr服务并开启Dashboard' a0 a" a! d$ k+ H
安装Ceph-mgr服务程序(所有设备执行)5 h5 b2 w& S8 \! H
yum install -y ceph-mgr" c4 N$ T( J) `0 @/ O. i4 V
1
! T% _# Z+ B' N7 {4 p+ Q' j初始化并启动主MGR服务(Ceph01执行)
2 v! Y  E* B. m  m6 [) M5 Mmkdir -p /var/lib/ceph/mgr/ceph-ceph01
6 ~! K1 U. V2 W8 J* f- kchown ceph.ceph -R /var/lib/ceph
2 z6 w, P* k/ d2 a. v6 |ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph01.keyring --gen-key -n mgr.ceph01 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
, R" `- c' _( B7 u. o. Eceph auth import -i /etc/ceph/ceph.mgr.ceph01.keyring5 u8 O- G4 r. g# ]) a) W) n: Q
ceph auth get-or-create mgr.ceph01 -o /var/lib/ceph/mgr/ceph-ceph01/keyring
2 o: [" p7 @. W1
3 T  [1 k9 l3 r: o0 Y; m  l( g/ F8 Z2! K4 I0 Y! x5 O
3: n" Y( D$ z2 L/ ?: j" B" a
4
- b/ z8 g5 O2 j! C* I- M. C+ m) s8 b# P50 Z  V2 ~9 w; J3 p0 Q; e6 `
systemctl start ceph-mgr@ceph01/ k7 [- z% R0 y9 o+ Y" q* [
systemctl enable ceph-mgr@ceph01
5 E2 E6 c- D6 D) Ksystemctl status ceph-mgr@ceph01+ ~# H9 Q1 m. D6 @. G
16 I0 y! Q5 I( e
2
/ U5 R( |. Z$ ^9 F% P4 p3* D3 n2 }5 F5 M5 `& z% |& d! s' d
初始化并启动从MGR服务(Ceph02执行)
- U  A0 n, r2 [mkdir -p /var/lib/ceph/mgr/ceph-ceph02/ g. B: p' ~2 A
chown ceph.ceph -R /var/lib/ceph- p9 g1 a6 i, y; M- k4 X
ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph02.keyring --gen-key -n mgr.ceph02 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'( w) s) r; m. A; Y7 |) D. R
ceph auth import -i /etc/ceph/ceph.mgr.ceph02.keyring" F3 d( R# S6 {
ceph auth get-or-create mgr.ceph02 -o /var/lib/ceph/mgr/ceph-ceph02/keyring. }6 g0 _2 V5 p/ [8 _$ ^' Q" d# ]( K
1
9 }6 E5 B( E+ |8 D$ Z( q+ ~0 C4 U! ~2; ?$ _# B* K9 r2 k2 x& z
3" F/ o# U3 Q/ i' O
4
/ a2 J" K3 h7 m4 _5; i  f$ ~6 g% {, J( ^' W
systemctl start ceph-mgr@ceph02% S5 E, s) {. q, f+ S8 z# q
systemctl enable ceph-mgr@ceph02: h- h9 [6 j. g+ N2 i- P3 O
systemctl status ceph-mgr@ceph02
( Y  s7 k4 }) `" _# T1
4 o- e  R' u* D+ R. w. G- V+ @2
. d. {3 ^9 F; M1 k& [7 A( D  X3/ {2 A7 V* g  {$ q6 K0 W1 a
初始化并启动从MGR服务(Ceph03执行)
5 q, H, M$ \0 \4 bmkdir -p /var/lib/ceph/mgr/ceph-ceph03
& a: ?, B% [1 r3 n4 t$ Schown ceph.ceph -R /var/lib/ceph. D" ]7 E' c: c3 t6 c
ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph03.keyring --gen-key -n mgr.ceph03 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'" p( Z1 ^8 w; s) I+ D* e, @
ceph auth import -i /etc/ceph/ceph.mgr.ceph03.keyring( b/ d, K! @  X5 c& V# {- j
ceph auth get-or-create mgr.ceph03 -o /var/lib/ceph/mgr/ceph-ceph03/keyring& q5 m' o& I& R4 T6 t6 A% e
1& s6 ?# {7 Q. p$ t, }
2
  \7 z/ Q# ]0 s3# w* d1 f( e; c! K$ b7 s/ e0 a
4
/ K2 P  e7 @/ @- m) ~5
5 w" a& B0 u! l! L& `; Tsystemctl start ceph-mgr@ceph03* W$ @. X1 L( w, o  s: e
systemctl enable ceph-mgr@ceph03  J. v1 D5 R+ F: H
systemctl status ceph-mgr@ceph037 L) M% W+ K, m( U5 [
1" H! F7 ~6 ^' }' ~1 }3 I$ z
2
7 _- E$ _5 c7 {, k$ S; R$ ]3
4 P. G- ^. M5 Y* R# A! `6 [6 p查看当前集群状态(任意节点执行)0 N1 r% ~% M2 {, S6 p
通过ceph -s命令查询集群状态,可见集群services中3个mgr服务已启动。
  y6 y4 E/ p: p. q/ J, c" O$ o4 kceph -s
6 |+ G5 v; N9 }# u7 q> cluster: / @( b  g4 l0 b( Y8 e$ x$ ~* P  |
>                 id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a        
$ k) {3 g3 r7 j( y0 {>                 health: HEALTH_OK
4 k9 O! T. s! s( A0 {& G7 o9 j> 3 V5 u) ~, f8 c
> services:        
9 }, ^% \: Y% U( u$ @>                 mon:         3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)         4 ?: {: E/ I8 J8 R# z- f
>                 mgr:        ceph01(active, since 3w), standbys: ceph01、ceph02                       
5 o- i+ ]$ Z7 S; ?1 N+ E; W1 j>                 osd: $ [( U9 c6 A2 C2 j! s6 @: y9 i! b/ ~
> , U2 e" A! Q9 e$ j$ v6 I
> data:
) Q! W0 C* m0 ^! B& t4 r>                  pools:   
$ C2 b6 \# i( q8 V. v7 r. Y& ~>                 objects:
1 O6 x- d+ t" A9 K. m>                 usage:     X% Q  U7 p: Y: ^% z# D
>                 pgs:' E# ]) ]6 S2 A4 \6 o
10 ~8 r9 K0 j# x
2
7 A5 b1 b& m3 V3 g1 {3( N- n# G$ a. q2 J  J0 Y7 t  {" z1 k7 R
4
+ q& L! t8 C& e6 k" e/ F5
8 h! j# }7 r: S$ T6 G* D8 J63 ]5 X$ L3 r* y
7& Z7 h3 }5 ~7 T  r  D
83 A2 y% T* ^6 ^
9
+ ?5 x4 X9 O! w% q105 \% ?0 u' A" n& ^
11/ R- V  F# y7 ^4 T8 j  m
12
9 ^) }: t) S) b0 F134 ]9 q$ f& g3 |, a" Q2 f& t
14% r. x! ?. p/ Y4 X8 F! ~
15) m5 N: T/ E9 T
使能Dashboard访问功能(任意节点执行)* P6 f) c9 T7 l$ i
开启mgr dashboard功能
. G5 w- o( G1 j/ t/ R9 }ceph mgr module enable dashboard2 w0 ^& x3 W  h6 C2 c) O
1& X' Y" e+ ?( p  d! H
生成并安装自签名的证书
# |" d# a/ A" x4 o' l' fceph dashboard create-self-signed-cert! p, x9 v% c+ a/ L
1
* [' c9 n  l2 G" Y" P配置dashboard
6 {7 `' k7 [  T  u1 U/ Gceph config set mgr mgr/dashboard/server_addr 10.40.65.1484 o+ ~9 l( Q9 \0 D. ?
ceph config set mgr mgr/dashboard/server_port 8080" r- R' \8 x7 }
ceph config set mgr mgr/dashboard/ssl_server_port 8443/ y* }# c  q8 Q
1
& z2 @8 q- G# f% M  b2, h8 D7 Z9 Z( C" g
3
) G2 L' |" R+ H- t' v. w( m9 v6 C创建一个dashboard登录用户名密码
. H' N7 }! A* L: N# S8 Qecho '123456' > password.txt
0 f% R  l, j, yceph dashboard ac-user-create admin  administrator -i password.txt
( F! [* l2 X: p0 c9 |+ [: e% I1, {4 O6 F3 u# D/ W# O# O
2- w- W( G7 Z; m3 z( E( u
查看服务访问方式
" G9 t9 i2 x& l3 E7 L1 kceph mgr services5 P0 |5 o2 I: Q* F2 {6 z0 V+ _9 w
1
; B, n; `: U8 R! H' p, S7 ?: m# Z通过web访问Ceph Dashboard,用户名密码为admin/123456
( K, R5 x! s; f& _5 Uhttps://10.40.65.148:8443
您需要登录后才可以回帖 登录 | 注册

本版积分规则

返回首页|Archiver|手机版|小黑屋|易陆发现技术论坛 ( 蜀ICP备2026014127号-1 )

GMT+8, 2026-6-11 23:58 , Processed in 0.018235 second(s), 23 queries .

Powered by Discuz! X5.0

© 2001-2026 Discuz! Team.

快速回复 返回顶部 返回列表