|
|
楼主 |
发表于 2023-5-26 10:34:17
|
显示全部楼层
: z+ X2 k& `5 l& R$ g
1.环境
4 ]6 y6 l3 ~0 r5 M 版本:! Q* E; G7 k8 ?, \4 C
centos 7.4: w1 [+ r2 @/ _3 D5 n, E! S
ceph 10.2.9 z& W& u3 _. ]
oracle 11.2.0.40 K; }: P S; \' v* @# a
J0 m& v) c" k# |* J$ z
机器构成:: l; q' V, h1 C! c# ?$ l
ceph 集群的构成:
; Z1 G5 I+ T: h5 V. G1 m3 w7 k5 ~ ceph-mon1 管理节点和监控节点 192.168.0.170
U" H, T' A/ L+ C- L ceph-osd1 存储节点 192.168.0.171 + U2 l/ m7 b9 `& M/ z! w
ceph-osd2 存储节点 192.168.0.172 3 [4 r/ }* @! a6 ?) C" a
ceph-osd3 存储节点 192.168.0.173 ~, s/ a8 @6 `* J
* X' S) ~; u6 z; c w1 A1 F
规划的每台机器贡献出来的存储空间
) @5 D: j7 b8 K0 R: H ceph-osd1 20g
, m/ I8 V2 H( c) D6 R3 u2 T" S8 `) \ ceph-osd2 30g) O8 L' G' W3 R# S
ceph-osd3 40g
: R$ J! L( m, o& F+ R/ o1 u" J; [2 p' W+ g4 l4 `
我这里是测试,正常规划,应该大小上接近,因为默认池的副本为3.5 M, q* p# U9 d4 V# f
. N3 w; ?! V' U" j! h
数据库环境:- c( R4 v6 c4 m! I& Z2 p% b
ceph-rac1 数据库节点 192.168.0.175$ O1 A1 y5 [# F. m/ i4 c
ceph-rac2 数据库节点 192.168.0.176
$ [3 Y2 ~7 y9 Y; D0 {
; Q+ {( ^# [& u7 u3 x9 K rac ip/ v6 R; Q* _2 D; i' [' n: i
192.168.0.175 ceph-rac1
; G3 ?8 Q0 `) }- v2 f# }0 y 192.168.0.176 ceph-rac2 : ?3 y" f) {* }+ M9 I+ r5 p7 ~
! q, I* D3 Q5 v; E
192.168.0.177 ceph-rac1-vip# |& \9 K* N" T8 ]6 |( U
192.168.0.178 ceph-rac2-vip3 {0 G8 P5 a ?" U2 {
1 k' O9 A( y$ E4 g2 b
192.168.0.179 ceph-scanip
* \4 E8 Z7 }4 L# p2.安装ceph 集群9 `% K3 `: m g) I/ T6 {' J
在 ceph-mon1,ceph-osd1 ,ceph-osd2,ceph-osd3上安装ceph集群$ J7 }2 e' w5 b
具体安装见另外的帖子.
- D; z0 T0 J4 [% ~3 X
* x' K, x0 r( M+ p4 R3.规划磁盘
! b2 G0 T+ i, E A- G% G7 T1 J4 T. U1 j e9 t, |$ X
规划的每台机器贡献出来的存储空间
! y O. d' ^: _. R& Q ceph-osd1 20g 分区 /dev/sdb
1 w d# b; Q! a ceph-osd2 30g 分区 /dev/sdb; l$ x- ]" p! L a+ ~
ceph-osd3 40g 分区 /dev/sdb
. `8 C" h# [4 h4 c1 D6 b9 S% [ 3.1 列出每个节点上的磁盘* B; U3 i6 ~5 a/ y& w3 G
9 \9 A, @1 w+ u: l l) [ 查看一下 Ceph 存储节点的硬盘情况：- S W$ S! ?$ s( d
# ceph-deploy disk list ceph-osd1
2 h: k+ W; E5 Q" \ # ceph-deploy disk list ceph-osd2
7 n0 x0 H$ j1 P) V* F0 [0 T # ceph-deploy disk list ceph-osd3
/ r( t/ S; B9 Y: w& _1 C8 j- b: E9 ?5 {- A: @: x e( g
3.2 擦出盘所有数据
5 f" P5 k+ x+ _0 ` # ceph-deploy disk zap ceph-osd1:/dev/sdb ceph-osd2:/dev/sdb ceph-osd3:/dev/sdb" [$ h: L) j* g* J' z2 Q
7 [. L8 I: m- |
3.3 格式化盘1 h$ ^0 N, y1 s0 L0 X, n
# ceph-deploy osd prepare ceph-osd1:/dev/sdb ceph-osd2:/dev/sdb ceph-osd3:/dev/sdb ; k, c4 M d9 i" q7 V
! w1 o4 D( A7 q+ K) H% r 3.4 激活盘/ U1 V i, q8 Z
#ceph-deploy osd activate ceph-osd1:/dev/sdb1 ceph-osd2:/dev/sdb1 ceph-osd3:/dev/sdb1 + z' l4 I' A4 H1 @1 z
; l' ]) g; h' l5 g4 F
[root@ceph-mon1 ceph-cluster]# ceph df
9 s8 m0 F/ R2 F7 L% B* W5 C; W) ~) WGLOBAL:
8 p! U7 ^% ^2 k0 c! ?: S% F* n SIZE AVAIL RAW USED %RAW USED
6 |! @( C7 r0 d 76756M 71317M 5439M 7.09
7 z$ y5 a C+ Z" e0 j9 s, bPOOLS:
" a& K7 `9 c- F NAME ID USED %USED MAX AVAIL OBJECTS6 h8 d$ a% B8 L. m
rbd 0
8 @9 A4 p# ?* K4 m6 H. ]
) Z) G+ X2 e* m0 o' I; ^ 默认有一个池 ,rbd8 ], M( V5 a7 w1 i* ^$ f
5 ]; M ` b- ^! A# L, E 3.5 创建新的池
5 h0 L* Q5 v g2 W 3.5.1 删除原来的池: E1 l- w9 x1 n+ m6 W2 t
rados rmpool rbd rbd --yes-i-really-really-mean-it
( X6 v9 Z% X: N4 ]/ O, e 3.5.2 新增加池0 M9 X( R5 E/ y4 j; m% Y( S
这里的池,就有点像我们lvm中的卷组% d# x/ k- c) f$ \4 \1 T
考虑到我们要安装rac,因此,把vote,ocr盘 ,与数据盘分开,因此,我们建立两个池.$ N7 r1 ^+ u( n% S
$ d! _1 [0 z( d0 P5 A5 K% w ceph osd pool create votpool 128 y" H$ t+ l# A
ceph osd pool create asmpool 120
# O; m0 ]; _3 u) t
+ z' d* [$ W! F! t. }# j ~# n 注意:以上在ceph-mon1机器上操作.
) n! A9 d6 c1 W' x* m# g8 f& f7 G( P* Y( X6 U: L d' m7 k
4. rac节点安装ceph 软件+ M3 O4 i# V! o4 B3 V) R/ N' q
在ceph-rac1,ceph-rac2上安装ceph软件,让他们可以使用ceph存储' [. U6 B2 ^2 m! [# P3 B9 D S
( g$ F N5 K" v8 h# G6 ]8 ^; e% D
具体安装安装简略如下:" A5 V- Q9 ]- t* Z; J
4.1 安装软件
1 a/ v0 b! k# h0 j) {, i1 p+ k yum install ceph ceph-radosgw rdate -y
* C) Y+ v( j7 k. ]9 |+ t8 p6 d7 L3 a( `! ?
注:这个操作在ceph-rac1,ceph-rac2上操作0 B9 X$ t) ^5 h& @4 K" e+ x9 x: s; |- z9 N
) E( w% q9 T7 Q1 _ 4.2 把配置文件和密钥拷贝到client节点ceph-rac1,ceph-rac2) W- N" b, }5 y1 Z# b' c! X2 V
[root@ceph-mon1 ~]# cd /root/ceph-cluster/
. ~2 u% E( N0 d1 I6 q6 t [root@ceph-mon1 ceph-cluster]# ll
8 q: i J7 K. u# k4 Q total 220
* K% F, z. S l5 O -rw------- 1 root root 113 Oct 26 04:21 ceph.bootstrap-mds.keyring( f8 h/ s c: p. c, q7 D- V
-rw------- 1 root root 113 Oct 26 04:21 ceph.bootstrap-osd.keyring
- o( |* _0 v9 e* }" v7 b6 |( ]- Q" S -rw------- 1 root root 113 Oct 26 04:21 ceph.bootstrap-rgw.keyring
) Q: ]6 U, e D9 c" D -rw------- 1 root root 129 Oct 26 04:21 ceph.client.admin.keyring N# _! u4 ^+ r* k2 h
-rw-r--r-- 1 root root 258 Oct 26 04:20 ceph.conf
3 P: {3 h& H5 a" q7 \& B# I# [ -rw-r--r-- 1 root root 200577 Oct 27 07:21 ceph-deploy-ceph.log9 T8 _+ b) x- y6 `
-rw------- 1 root root 73 Oct 26 04:16 ceph.mon.keyring
p% ^0 j0 J6 D3 l" u 看到有很多密钥，mds,osd,rgw,client密钥，需要拷贝的就是ceph.client.admin.keyring和 ceph.conf! g, q: B, S8 z" A* A% Z
可以用ceph-deploy工具拷贝# S7 g' {% Q' k" r# z' P* D, W
# [# f) M2 _" Z. Z" ~6 u
[root@ceph-mon1 ceph-cluster]# ceph-deploy admin ceph-rac1 ceph-rac2
0 \, a4 u! C6 v5 U- a
& |- a% c. H% v2 }7 D ceph-deploy 工具会把密钥环复制到 /etc/ceph 目录，要确保此密钥环文件有读权限
3 `7 @! D5 ~" j9 R8 P$ f1 |3 P+ q (如 chmod +r /etc/ceph/ceph.client.admin.keyring)。
' Y. q( G" [" C( o% F( E0 O5 t) S
注意:以上在ceph-mon1机器上操作. ; y6 J3 j2 H, N! }, T' ?& h. t
6 j9 g1 g: o! A, ~* q3 f
5.创建具体的image4 @# @3 ~; V! W
2 i" u7 }+ L' d1 i; V: e7 R 在 ceph-rac1 节点上创建一个块设备 image 。
* O! I& Q. ?' ?: k
# N3 E5 j8 u# p3 }- _3 h --创建3个1G 的vote盘 Image
# D0 p3 }- ]; s9 Z [root@ceph-oracle ceph]#
- N( g9 O8 N9 g/ z3 D' B' F' { rbd create votpool/img_vot1 --size 1G --image-format 2 --image-feature layering
; }3 Z; S. {- Q8 h6 y rbd create votpool/img_vot2 --size 1G --image-format 2 --image-feature layering) X; K; p6 L/ y
; _' s6 x; z+ |; D- ?% C2 L rbd create votpool/img_vot3 --size 1G --image-format 2 --image-feature layering
1 B- l: J( h# B
+ B5 y S) n8 G) R7 u- m --创建5个2G 的数据文件盘 Image
3 ?0 _2 }+ T s, g
/ d& a8 o; r# Q3 u rbd create asmpool/img_asm1 --size 2G --image-format 2 --image-feature layering) M" r. y+ r Z" P X' A$ q
rbd create asmpool/img_asm2 --size 2G --image-format 2 --image-feature layering
7 h, j% p, u. ]5 o* b6 e: z: p$ h% N* e rbd create asmpool/img_asm3 --size 2G --image-format 2 --image-feature layering4 ?0 X( P- a% S+ G( ^
rbd create asmpool/img_asm4 --size 2G --image-format 2 --image-feature layering* U' E5 y3 H; l
rbd create asmpool/img_asm5 --size 2G --image-format 2 --image-feature layering
' |/ E* T" ?5 r( G/ v- z$ m4 Z9 n9 x# G9 i/ G
--查看rbd池中的块设备
6 e: _! O8 `; m5 |$ l( ^ [root@ceph-mon1 ceph-cluster]# rbd ls --pool votpool
5 P: g+ M* i: a5 a- i img_vot1
# f9 G: C7 _0 |2 I" N. p img_vot2
h3 C5 E- A, {% z t w. y img_vot3 j7 W7 l+ {2 p, g2 G% a* C
$ {1 ^3 p" O- @7 y$ }8 N: B [root@ceph-mon1 ceph-cluster]# rbd ls --pool asmpool# O5 [4 z! X6 f8 O* u: ]
img_asm1! u8 F% v4 N; ~# x
img_asm2( @4 A# W, H* F6 ~$ U6 h
img_asm3
5 g3 x5 T! N. C4 ? img_asm4
1 `; |+ r# K; U+ P) J I img_asm5
2 p6 d6 C9 k9 V- M( c, v# G* @% i8 o6 B, a
6.映射块设备
, [1 Y2 Y p0 R& N1 V: X 这个必须在客户端节点上执行,比如ceph-rac1,ceph-rac2上执行.
9 }6 H+ q0 s: F1 l! v( L0 }8 Y$ I2 h3 h3 V4 d% \. J
[root@ceph-rac1 ceph]#4 N2 l. p3 u3 V; ]+ O6 y
- o; [. H H1 X; m/ y6 b
rbd map votpool/img_vot1
- @( ^* i: O) x" C rbd map votpool/img_vot2 ! l& Q6 b1 |$ u" {
rbd map votpool/img_vot3
. T: i9 l# Z1 O rbd map asmpool/img_asm1
$ j- l( B0 c9 P6 [% G$ T rbd map asmpool/img_asm25 w4 ~- x O" W! a
rbd map asmpool/img_asm3
1 p% S3 ] O- |5 r" ~ rbd map asmpool/img_asm4! T9 e x- j! ]: }- h" M4 B
rbd map asmpool/img_asm5
% I! n+ B$ k$ |
/ q5 z5 k4 g1 T% F( x5 c+ v 这个时候,我们通过lsblk上,可以看到新的块设备了:
, g8 b. M; c$ y# W( _4 L. U2 b [root@ceph-rac1 ~]# lsblk
2 T; I2 ?/ O+ s0 t0 f' I4 u NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT* w9 S" A* M; Q; W* A% T. Q+ R. W
sda 8:0 0 100G 0 disk; } j. ^8 e/ h- z- t
├─sda1 8:1 0 1G 0 part /boot
7 C9 ^ c# z6 u8 k) Q └─sda2 8:2 0 99G 0 part
# E) M2 {: c6 h+ A/ j ├─centos-root 253:0 0 50G 0 lvm /; R0 S$ A1 x4 s3 E0 I6 t4 [% c0 j5 J
├─centos-swap 253:1 0 2G 0 lvm [SWAP]7 G$ D' Y9 Y3 p. |* F- J' K" y1 t! E4 W
└─centos-home 253:2 0 47G 0 lvm /home: t" F4 x" g& `8 p/ H2 J0 ~& ^
2 o, j6 ?# n4 M/ S# N rbd0 252:0 0 1G 0 disk9 _- B+ P& W! W( @0 S2 f% X
rbd1 252:16 0 1G 0 disk
1 Q X9 Z9 R s rbd2 252:32 0 1G 0 disk9 G# {1 s# ]/ @$ a/ r
rbd3 252:48 0 2G 0 disk
# E9 H/ V3 s: b8 G8 K rbd4 252:64 0 2G 0 disk
$ s7 Z6 H) [ P0 ^: o9 t* W rbd5 252:80 0 2G 0 disk
1 t# K+ g& }# F( j% U1 T w; X+ T rbd6 252:96 0 2G 0 disk
' x( U8 T7 U7 o; n9 W rbd7 252:112 0 2G 0 disk
' H1 Y* k: U! F. ], ~7 s5 v& e+ `% o+ m0 L& v$ D! g2 O& m
--要查看这个对应关系:' L0 X7 M4 s$ U
; V! Y5 t; |' Y" _- ]) c# w$ n
[root@ceph-oracle ~]# rbd showmapped
* z/ m. v0 Z* i, z. R id pool image snap device * v6 D0 ~% t+ [; k
0 votpool img_vot1 - /dev/rbd0) F3 }6 J- ]( a: D9 y/ D/ B" q
1 votpool img_vot2 - /dev/rbd1! `' W% L% M9 g5 w$ O% B& T
2 votpool img_vot3 - /dev/rbd28 {7 j$ f+ h, y
3 asmpool img_asm1 - /dev/rbd36 r- j, x* W0 k) e0 s) G
4 asmpool img_asm2 - /dev/rbd4
0 l- N- ^; l7 N, O7 q5 A9 v 5 asmpool img_asm3 - /dev/rbd5
7 t% X% m; F1 o: M" \( p4 i: _& d 6 asmpool img_asm4 - /dev/rbd61 @, ?5 ]: x1 n2 r" w: S; s2 i
7 asmpool img_asm5 - /dev/rbd7
. u Y5 S* J0 Q/ y
2 C1 ]6 ^9 u3 {+ }/ k7.修改权限2 |6 O; h o* m r: \
这个块设备,每次重起,权限都会变回root:root
0 e8 L3 X h) S B* [' y 而且,这个对照也必须手工重起映射.& @ e3 P! ?+ K
因此,我们可以在 /etc/rc.local中,对应权限和映射,都写在里面
6 r# o6 V+ m1 L" g5 L4 p6 A" B. I
) }. i' b* ~5 T) R [root@ceph-rac1 ~]# cat /etc/rc.local
- J- ~) Y# h2 e# ], q# E/ k6 W# j#!/bin/bash
! p' Q6 O' p% M# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
4 ?; @; g) i6 ^( R- G6 q5 G# j3 |! j* Z" l( b" A
# It is highly advisable to create own systemd services or udev rules5 ]$ E, Z. C& m! B5 A- H2 z; N
# to run scripts during boot instead of using this file.- w4 P1 A" H m- A
#* P, E4 a, y. W) p& b8 y
# In contrast to previous versions due to parallel execution during boot
1 k3 y0 x8 m9 V1 K! N# P# this script will NOT be run after all other services.5 y) E5 k2 H7 h: O. ^! }
#- S% L) g1 |+ M& K' U
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
^& b% S2 F, b7 j, F' \# that this script will be executed during boot.
" Z% r- r( r9 ?) k9 o! C6 Ntouch /var/lock/subsys/local& x: b3 n; Y% u
rbd map votpool/img_vot1
0 A3 \! O" J; Urbd map votpool/img_vot2 1 T4 S- R. ], E2 M, b- w. U
rbd map votpool/img_vot3
' I% `! W- N% E% z( x9 O1 Brbd map asmpool/img_asm1
9 D9 \# P$ ^: u. T% O* m* h Urbd map asmpool/img_asm22 Y) K7 h. U8 ?& v0 j
rbd map asmpool/img_asm3
: {! c9 Q/ B5 K; V- yrbd map asmpool/img_asm4
! ~+ }$ k! x4 l- irbd map asmpool/img_asm5
, z* i1 Y3 S% C9 T1 a, r/ ^6 X" Bchown oracle:dba /dev/rbd*0 M0 s( a o% a( X" t9 Z
chmod 660 /dev/rbd*! F* `- o- y/ T/ m# j
: K3 U2 i; z. h% I7 r
5 s; v% J/ ^! C) W+ [
编辑完后,执行以下语句,否则,系统重起,不会执行/etc/rc.local
4 F1 [2 B7 _- n! G- Wchmod +x /etc/rc.d/rc.local0 [0 e3 m" O' r- P/ s2 V
# R9 P- x3 n8 n5 S+ K$ V/ O, O) }之后,每次启动后,映射和权限对应都会自动修改.
' P2 I% f* ]7 \' R$ F) ]( |( |5 n2 R( Y" c6 P
8.安装rac( B! o! k1 n l" Q4 U7 _, S
具体安装步骤略/ v/ f* z& | \
安装过程和普通的rac安装没有任何差别.
% F) m% l0 t. U$ M( m: {. d$ j8 b1 D% @
磁盘的对应关系:+ o4 _" |! E }$ B2 d; D' {& x
/dev/rbd0 dgocr9 @) l/ W& C) P9 v0 G" y
/dev/rbd1 dgocr2 F. i3 C8 F; G$ _. |6 U
/dev/rbd2 dgocr5 H7 \& Z0 W! j: L
2 @. W' n2 j* @* w: o& ^* z /dev/rbd3 dgdata& ]: C5 V& ?1 x% ]& x
/dev/rbd4 dgdata9 l9 r2 P& U+ l8 G; W8 r
/dev/rbd5 dgdata
: A1 `: h# ?3 C# t& J( z+ B /dev/rbd6 dgdata! R" z/ H1 J/ I) P8 f
/dev/rbd7 dgdata
, k8 ]; b' C6 K: O, U4 Y0 s' Z! c' U9 X5 L9 j
9.安装完成:- i; W7 M$ @7 b/ B) ^' u
1* select name,path,os_mb from v$asm_disk
+ b4 K4 O; }& f6 s1 G& I7 oNAME PATH OS_MB
) L# f- L, g# R" `7 F------------------------------ ------------------------------ ----------( V7 c/ E; `7 z
DGDATA_0004 /dev/rbd7 2048: K2 t1 F. Q+ \( |$ {3 h) A
DGDATA_0003 /dev/rbd6 2048
) ^0 t' |/ ~1 S w! _0 n9 ~2 rDGDATA_0002 /dev/rbd5 20487 q/ e. v, y( i9 H8 C3 \
DGDATA_0001 /dev/rbd4 2048 ~ A( x5 B( j% ]4 m
DGDATA_0000 /dev/rbd3 2048
- z7 n2 Y, \% H9 eDGOCR_0002 /dev/rbd2 1024
8 g: A; t- O# |1 ~8 y. I9 ]DGOCR_0001 /dev/rbd1 1024
( S% ^ ]. e, A% X+ ZDGOCR_0000 /dev/rbd0 10249 F0 ?- ]9 L0 I0 I
4 h" o7 I3 Z* j, Q8 \: J$ x+ E资源状态:) _2 b+ c7 _% ]
[oracle@ceph-rac1 ~]$ crsctl stat res -t
" N t) x% n) t! C/ q# ]6 P* O--------------------------------------------------------------------------------7 c0 c$ I0 Y: v
NAME TARGET STATE SERVER STATE_DETAILS
/ F# l7 N( O" B% O--------------------------------------------------------------------------------
) K6 x" ]! B* b8 tLocal Resources' j2 I1 m* i6 J
--------------------------------------------------------------------------------" K/ B. S" ]! }+ y! f& G, f H1 B% C
ora.DGDATA.dg4 J' O) O( Y G( d7 y8 u
ONLINE ONLINE ceph-rac1 4 S6 n" a) U- F9 b! \& }
ONLINE ONLINE ceph-rac2
- M7 _2 g2 q: eora.DGOCR.dg# P+ Y* v$ I/ K, k% Q" e/ S
ONLINE ONLINE ceph-rac1
, T4 K! G2 P# A ONLINE ONLINE ceph-rac2
6 G; v' C1 _* ?ora.LISTENER.lsnr( B) g# H9 w* [& J2 z" W2 l, }
ONLINE ONLINE ceph-rac1 * O" W* F% a4 Y8 q' x! |
ONLINE ONLINE ceph-rac2 2 P0 K9 l0 j+ f- U: h, a. G
ora.asm
! m8 E4 b" J5 w ONLINE ONLINE ceph-rac1 Started
: r0 q, I6 C1 U7 F# I( } ONLINE ONLINE ceph-rac2 Started % L9 H# L1 D: ]: d
ora.gsd2 |% }& V; ]* w% R" [; O$ w3 W
OFFLINE OFFLINE ceph-rac1
2 u2 }9 V& b" u' O& P/ ^" {# G OFFLINE OFFLINE ceph-rac2 . m& p t% W: o; I7 Y
ora.net1.network' G: M' [. K$ x1 g* s3 V
ONLINE ONLINE ceph-rac1
! y7 }3 y9 y5 C# R: n0 D0 m% \3 A% d+ p ONLINE ONLINE ceph-rac2 * {+ C5 }+ b. Q. z
ora.ons
1 ?- F' I# K9 m! d) R ONLINE ONLINE ceph-rac1
7 H$ J2 _3 |" t% Y; V7 H, o- v* _ ONLINE ONLINE ceph-rac2
* }8 b2 G5 n6 M# a+ {--------------------------------------------------------------------------------
3 Z" U- P3 s# f" y! t. l, HCluster Resources' R+ W/ k. L( U$ e: E5 V' @7 E
--------------------------------------------------------------------------------
; y; O4 P8 j/ s9 S% Gora.LISTENER_SCAN1.lsnr+ z# i5 }- D0 q( e6 p
1 ONLINE ONLINE ceph-rac1 ( O$ x f2 [" B
ora.ceph-rac1.vip+ p. a5 w1 c/ v# O; `
1 ONLINE ONLINE ceph-rac1
% p! C5 B# y# r' g& s+ ]ora.ceph-rac2.vip
# n, `, d0 V, z. ? 1 ONLINE ONLINE ceph-rac2
/ Q' i4 v0 v; K& o) M! }9 {ora.cvu, J2 \+ ~) h- k3 X+ Y
1 ONLINE ONLINE ceph-rac1 4 i) V6 a8 }( S+ w
ora.oc4j
- v& G( y6 j/ X+ R 1 ONLINE ONLINE ceph-rac1 % ^2 S, d9 V3 x
ora.scan1.vip7 y1 p( z; i: p S ^+ M, I
1 ONLINE ONLINE ceph-rac1
( j' G" {9 | |( \5 T0 `. ?: tora.sztech.db9 b* k. T, z0 x" n# O! H, R
1 ONLINE ONLINE ceph-rac1 Open * W& `# e/ l) o% z
2 ONLINE ONLINE ceph-rac2 Open |
|
发表于 2021-7-1 11:13:00