|
|
软件版本:
; S4 ]3 J! B& H4 Q; K: n& W9 ]* w' Y3 S
9 [5 f0 V) ^0 d6 Z; `. X: R9 S, j
! P% M2 u4 H( p- x& U! G! _8 i! l
B5 N( t/ P9 h9 b% I. o9 t$ W, M* Q
# O# }6 ?4 a7 m7 ?# _
# f, t7 _7 \) Y) H/ ~. ~5 r' L6 `; y0 I( c" c7 @+ z
! @% |" ]6 N$ [0 O0 t) f; b: w8 c
9 v% n5 K9 C1 G9 o1 R$ X
& @: h& D1 {' t. |4 i2 k7 T e2 F0 i! Q y* h
$ _$ k- b/ {7 o
1 p2 L$ f& c8 `' f# f/ a
% g, y" i8 \& M/ J% f" R' d0 d
! C5 S% x: v. ]: V3 G7 d6 q. H9 j0 \6 d! n4 K: w- [$ M
软件版本cephoctopus(15.2.16)centos7.9
* D- v6 B4 K* iceph官网版本:. A4 ?8 ^) k, |
docs.ceph.com/en/latest/r…
! W9 e& { k b1 I机器列表:1 x& i$ g0 T" B0 |8 \+ w
3 O \) ?, y2 ^6 `4 X
- t, d! ]4 y2 V8 g: \
. Q% K2 O" {% g* M* q6 e4 T7 n t! p. z. V+ O. _ \7 ~0 p: i7 C' O
1 Z: b6 A6 E" r# R8 i
; ^3 W. n) @7 F. p0 j& D
8 t6 P: k( c j. X
! @/ p* ~7 ]. B0 _6 E
. O3 {5 o3 }8 V
. D1 y: _. M. d/ S* q
2 A, @6 O" S" V3 n5 K; @$ I# m9 l3 C5 J
}6 m8 l- } S6 F
2 z1 z9 ~+ }# z( S% W, j: U
5 U; h0 C+ f2 X D( K
, u9 N2 o! x+ ~9 n0 s, D
; R3 f# `! J9 u+ F9 E
5 Z3 Y# B; Y7 I- d- t3 N
% T7 H$ o h5 e- j- v( w' c; X
# E+ W# B4 U8 N1 T4 W
* x4 |. y6 c1 X
: Z4 C X' \" `& [2 r
0 z) P4 {8 e" M$ J, F2 G% C% V& r8 M9 }7 w' W0 V0 R; i1 i& d
1 e; t: J4 s$ q, \. l& ~机器名称ip块设备. u' X+ Q X% N, g/ g
master0 12.70.10.161/dev/vdb 和
1 N2 j& v8 S9 L T) N! e0 V+ T3 V: ^/dev/vdcmaster1 12.70.10.162/dev/vdb 和- |8 D/ z9 Z h0 F4 y
/dev/vdcmaster2 12.70.10.163/dev/vdb 和 /dev/vdc) \( y& m4 K& n7 e
准备工作. f7 L1 x1 \8 _3 |4 b& x# H9 j7 J
开始使用ansible做些准备工作,实际开始搭建后,不用ansible,因为cephadm本身就是一个集群管理工具。
; t' n- F9 K$ y. \$ I) E/ ?6 z如下ansible剧本均在集群外的任意一台机器上面执行,需要安装下ansible,不明白的可以参照ansible使用方法。8 _* y6 Z/ ]( V' i, B
ansible的hosts配置如下(因为之前这些机器用来搭建k8s,所以名称没有改变):! L% D7 ^, C- q- Z1 D& b" {' R8 \
bash 体验AI代码助手 代码解读复制代码* v5 ~& G! Z% c
master0 ansible_host=12.70.10.161 ansible_port=22 ansible_user=root ansible_password=*** host_name=master0" s! o( F. H7 k; s- N! ] ]2 t
master1 ansible_host=12.70.10.162 ansible_port=22 ansible_user=root ansible_password=*** host_name=master1
$ q) X6 ] ]. n4 m6 @" pmaster2 ansible_host=12.70.10.163 ansible_port=22 ansible_user=root ansible_password=*** host_name=master2. q8 _% e& Q( c2 k) P* h
. z" _) X( g( A) O6 ~: U& w
[all]
7 ?$ g6 f1 G( u l4 Y; o9 qmaster0
7 q% s G# n" |: Emaster1
# Z0 y* Z! B7 g$ e' @9 Ymaster25 N2 b) c! t* O: g' w9 F7 U; o( A
# J6 Y$ u' S' I' P, _7 v5 `
升级系统内核& }& T! B; V! a) s$ q- T1 n1 s/ B+ C
升级系统内核到5.177 h, Z9 z( x. R; V& ?& D
ansible脚本如下(1.kenel.yaml):- W. M2 L5 p1 q0 [; ^
yaml 体验AI代码助手 代码解读复制代码- name: update kernel
6 {$ V: E: A$ Q4 F: v# p" }3 Y v hosts: all
4 P- [. ]" ^% ^' s0 N7 V0 J gather_facts: True
$ {/ \8 q0 h O/ _ k0 P: R6 L% x vars:. H% E) c# o- g, U; k# @8 E* E
tasks:# i; y- [3 ? ]: k
- name: create workspace6 Z' x6 `$ Z% W+ c
file:1 T0 f3 o$ Y5 ?6 |
path: /tmp/workspace/8 z3 W& ~/ ?* d* }7 a h
state: directory$ D7 r* D; z( \( b
tags: workspace
6 w) _1 Y7 z3 f" K" E9 O( {" C - name: yum install elrepo; Y1 ^. ~, c0 I2 g* q5 H
copy:1 j) q" w. R8 G- E* p
src: templates/rpm.sh.j2 d, o* L2 v) l, E7 i& { Z
dest: /tmp/workspace/rpm.sh
B# i9 x2 p5 [) Y mode: 755
7 J3 e; ?5 }* K tags:( B/ y1 h4 u/ s
- copy-rmp
+ ^. q8 Q$ f4 h - name: sh rpm.sh
$ q# Q& b9 p8 F1 S shell: sh /tmp/workspace/rpm.sh' k1 G& t& {0 w6 k# [% ~" ], \2 m
tags:4 U% x# z5 z3 c# t6 K+ K" A; ?* r
- sh-rmp2 w; m4 q" Q( \5 ?/ j( Y! m/ G
- name: 列出可用的内核相关包
5 x% M) L2 u. A' {: ?/ Q yum:
1 K6 o8 P" ~0 B$ H& q$ \" g3 z) C list: available3 W; ^" W& X2 q; I% ^: p& a" [3 K
disablerepo: "*"/ V$ ^ e( H% U( k: _
enablerepo: "elrepo-kernel"6 m& M' R3 m% b9 U6 r7 u
- name: 安装内核
8 e- a3 E$ R$ L# y( D' d yum:
+ O; N. r5 a; ~! p& w3 [& s name:
' [/ P5 U8 I0 |0 S. g# H - kernel-ml9 O8 @0 c/ [0 M# z+ K. t: [
#- kernel-lt.x86_64
) n+ d! Q- }( o #- kernel-lt-devel.x86_64' k( H3 _0 a( G8 ~, y
enablerepo: elrepo-kernel
) H o4 M0 H" ^/ z9 A - name: 查看内核版本默认启动顺序" X/ I* e0 N+ ?; _( t
shell: awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg
1 m8 N, E: ]/ q) A1 Q9 ~7 P - name: 修改grub中默认版本启动顺序
& t4 d6 c6 ~4 t lineinfile:
5 R, R) F0 A) s! ~; | path: /etc/default/grub$ N" b) n/ Y3 H' R' D
regexp: "^GRUB_DEFAULT"" ]1 A0 u3 _- ]9 P' m" Y1 @
line: "GRUB_0=saved"+ w4 V5 i j$ l% i3 _- t9 N2 U
backrefs: yes
% ^" M5 V u l+ Z8 P state: present
, l" L6 t3 p5 x tags:0 @# Q1 c% c3 w' \7 k, ^) J. _
- grub2 I) Q! B: _- s0 w z$ b( |! E
- name: 重新创建内核配置( q0 u" G1 ?: `
shell: grub2-mkconfig -o /boot/grub2/grub.cfg
1 ^7 R7 }) J# q: m+ U - name: Reboot the machine
1 x) ^ e1 R- Z reboot:' ]6 K& F8 ^/ v8 q# H! A7 I
reboot_timeout: 300. J0 q# @( z; R5 {5 g3 o7 i
- name: uname -r' `) Z7 O3 h4 O! w( b
shell: uname -r2 z) k4 \3 e: I
register: version
2 P8 D6 c5 t: e4 ~2 C9 |/ Q$ U$ m3 I - name: debug3 |) _$ X R7 D4 I+ ^
debug:9 r B" S' J% H/ P6 M! C) q2 E* e
msg: "{{ version }}"6 d5 g/ `+ ?1 c% L! Z$ G/ X
0 l+ f$ C- O, s+ Y% F0 v) J
升级rpm脚本在ansible剧本同目录下的templates/rpm.sh.j2% m/ A- }. j* F% h( R
内容是:' x& j: s& k+ v' G R8 y8 U
bash 体验AI代码助手 代码解读复制代码#!/bin/bash0 i6 U& {0 N) W" L L. e3 e
rpm -import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org$ f7 H; Q D4 V! ?: X( K" C
#rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm" }/ q, Y" E R) p& X! q
rpm -Uvh https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm
8 @3 X& [5 \7 V2 @echo 0, d1 n4 c/ m2 K$ e4 Y! \
/ t5 h+ s) N# q1 ]$ _执行ansible-playbook -v 1.kenel.yaml完成集群中系统内核升级6 s5 r" v: R& d; h* u8 T; D
安装必要工具和关闭相关配置
) D( k, I$ A4 z8 N! L2 w m3 y D8 L
脚本2.config.yaml如下:
: l# [) m: a, g. \- a' G& o* ]yaml 体验AI代码助手 代码解读复制代码- name: set config! D: q7 {8 p/ Z+ M
hosts: all
! e- r: S9 B' c7 q/ l% R. @ gather_facts: True
/ e/ j& \2 o+ g ^& i" s6 x6 R' o vars:' ?6 F* B- A% H$ F# q9 l8 F
handlers:
: e5 i9 T! \1 D8 i8 k8 X - name: update_yum6 T. q2 D: S p8 J
shell: |
& W- E! t1 Q! m. ^$ w sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo
. v+ j& c" K7 R1 o" p; ^% y$ J0 _ yum clean all( p, y/ V. R8 D6 z) I8 ]; d
yum makecache -y* m/ w3 E2 h7 J# s
- name: noswap_service #禁用swap
# e- c! d. Q3 ?: { systemd:, z2 U3 ^9 z# B$ D$ t
name: noswap- k0 c( z. }& G' \% k
state: started #指定服务状态,其值可以为stopped停止、started启动、reloaded、restarted、running
, I, o- S* s- V* d `& P/ G enabled: yes #指定服务是否为开机启动,yes为启动,no为不启动; C( O- X2 ^9 h% T% I
daemon_reload: yes #yes 重启systemd服务,让unit文件生效9 f' T% ~- F/ z+ o! |
tasks:: `0 g! h0 g' K2 D: p9 o" |
- name: back up repo0 W# n N& V6 O8 A) x# ?
shell: |
) O2 d$ t5 J: D" e2 g5 G( }) ` mv /etc/yum.repos.d/epel.repo /etc/yum.repos.d/epel.repo.backup
7 _6 ~( c. D" ~) ` mv /etc/yum.repos.d/epel-testing.repo /etc/yum.repos.d/epel-testing.repo.backup4 c, y: T9 `; b$ h
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
5 S4 H6 t6 ]2 E* N2 w+ m2 | - name: delete /etc/systemd/system/noswap.service #禁用swap2 m, K+ `6 E: j
file:0 V& V9 _) t+ `0 N4 I
path: /etc/systemd/system/noswap.service9 X2 O. h$ K. w
state: absent" h+ V$ |' z2 P+ s) p3 ^% N
- name: download repo #下载yum源
% ]% z: H2 @6 j. R$ w% B2 d: g get_url:
8 I1 {' Q8 H, Y; V# B3 p# ~% r url: "{{ item.url }}"# v" w% H* h0 t6 [! ], H
dest: "{{item.dest}}"
3 v/ D" X2 e( i# { q force: yes
5 L9 R1 O: S" [; B$ ]; H+ B with_items:
8 b. j. L" k' R) x- o- X - {url: "https://mirrors.aliyun.com/repo/Centos-7.repo", dest: "/etc/yum.repos.d/CentOS-Base.repo"} t$ l" r1 S- B) v) j8 g w
- {url: "http://mirrors.aliyun.com/repo/epel-7.repo", dest: "/etc/yum.repos.d/epel.repo"}
@+ o; M _% h6 Y) R( y - {url: "http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo", dest: "/etc/yum.repos.d/docker-ce.repo"}
8 d5 w4 T. f% d) l9 s7 _ notify: update_yum #更新yum(见上面的handlers)! _% `/ G% _0 }% v
- name: install net-tools #安装必要的工具4 P- r8 Q' e3 X$ U. [: k7 C
yum:
$ u4 M) S" d, ?- Y, g, ~ name:
8 S0 \( Q; B h4 G - net-tools
! y) ^! m6 C* ]. ^4 i( K' f( {2 Y, p - vim
5 ]9 [1 Z( q { - rsync
8 I% {4 S5 x! C) h! ^ - chrony: y, k3 p' m6 e4 S2 c3 S, F$ l
state: present! l; _+ N$ l( q* q# R4 G
update_cache: true; x1 y; l. p [- ~# W, Y& ?" C
tags: tools" m2 p J3 I9 v
- name: chrony_service# u5 n* @6 ^7 ]- `8 W) j
systemd:* [' a5 J# K8 s ^% u+ b
name: chronyd% T) V2 K6 r# ]9 a
state: started #指定服务状态,其值可以为stopped停止、started启动、reloaded、restarted、running
1 T5 c# s; X+ w9 O+ A; u enabled: yes #指定服务是否为开机启动,yes为启动,no为不启动% X: z0 H3 I# F7 [( G& C/ u6 X; [7 r. n% x
daemon_reload: yes #yes 重启systemd服务,让unit文件生效
! t7 _: Q) t; [3 |2 ^2 O tags: chronyd, Y, C4 R7 N; x6 ]6 [$ G/ k
- name: config /etc/hosts #关键 生成hosts文件9 k; [/ t7 X) K% I0 u- i6 c
template:
3 @1 I9 a! \/ j4 u/ i src: hosts.j2
# E% J* R" k1 B3 }4 n& R% p dest: /etc/hosts- u) w( n1 J! t8 F; G$ s
mode: 0644# p. u* H3 \, G/ s
backup: false I4 ]3 y' J, j0 T& N/ ^
tags:, Y0 H5 e$ C$ y% S* E" Z
- hosts
8 w# X' ]) }( n# t( I9 \ - name: set hostname #设置主机名+ K$ B! X1 w9 c) J
hostname:! s G. e, m/ m+ ^
name={{host_name}}* R# S& [: S* t+ K3 ~- p
- name: set timezone to Asia-Shanghai #设置时区9 Q8 h. G: M! i& R* g5 o, I
shell: |
( I+ I% Z$ w3 o# r# p, A9 N /usr/bin/timedatectl set-timezone Asia/Shanghai; ^& `; k' }7 k9 V8 }9 r/ ^
chronyc -a makestep1 S K. _* ^. \; p1 @
tags:
' N/ R5 |+ N$ v; t A+ w* b - set_timezone
" q# {5 b/ w/ ?# U' t- z - name: stop firewalld service #关闭防火墙/ L2 ?1 d* N, x2 W3 j6 w' ?
service :. z- l6 [5 `0 p: b! V' ?
name: firewalld.service
e2 G2 a3 J' \2 e8 X6 L a state: stopped
+ ]8 A1 n* b9 H$ b5 s- a enabled: no
$ t4 g' h4 a3 U$ G! U register: firewalld_service_result
( s' A' a' q8 Y" L failed_when: "firewalld_service_result is failed and 'Could not find the requested service' not in firewalld_service_result.msg"& P: b7 L: A9 X
tags: stop-firewall
: K" `5 V" d4 |: C% \& u* y - name: Write noswap systemd service config file
. A9 I0 b. S1 @% k, W% N' { template:* e) `- P: ^; U: ]0 E9 L- h7 M
src: noswap.service.j2
' \7 S" \* s% N( t# a dest: /etc/systemd/system/noswap.service
9 r$ ?4 g8 C% `; n6 z3 E owner: root T$ i) t" G) @* l
group: root
7 ~: |6 `' E6 w L mode: 0644
8 Z2 h9 W& |7 u# o$ h: D9 k notify: noswap_service( I$ o$ L; Z5 ^# u: N) I" _
- name: Disabling SELinux state #关闭suse
6 v/ H( Y5 Y# U- Z( m; Q selinux:9 F, K X4 H; E0 w, s( G
state: disabled
8 W6 A- X' r# _2 v9 J - name: Reboot the machine #重启机器& ]- q4 @7 J3 R; f+ U ~
reboot:
- d: m, x1 A& Q2 J0 D reboot_timeout: 300/ O* H: D5 _' X7 g( F* D
$ A2 g$ s0 p' C5 O. R7 x* j
templates/hosts.j2内容:
' s& u6 r4 ]* J1 _+ ybash 体验AI代码助手 代码解读复制代码& o5 W; r* {6 z" L& w% V2 f- M
; w: Y4 [" @3 @' V1 l127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4, w2 G8 M5 B0 m, i
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
. Y, z0 Y% Z4 R: }
% u" p) a k+ m5 T6 I/ V" v+ }% w7 h/ R' s P. k
{% for h in play_hosts %}. M' M, M4 a1 R7 N, r
{{hostvars[h]['ansible_default_ipv4']['address']}} {{hostvars[h]['inventory_hostname']}}
5 t% o" M' C/ U1 `, g# j/ o{% endfor -%}
/ V7 M! H& u" Y7 z' `+ e& ^' [
4 D6 u. G% Z! c% ]/ Q9 ^执行ansible-playbook -v 2.config.yaml完成相关配置工作0 V" P$ m Y9 C
安装docker& o, ] h: U" J. i# O3 b
按照官网说法,可以使用docker或者podman,本文选用docker。" z9 F! `" q8 d7 O' w5 U3 f
剧本3.docker.yaml内容如下:
' |. O# ?- ^5 Q, X" z; Byaml 体验AI代码助手 代码解读复制代码- name: install docker9 U, c& ^) |: P( \+ T" t) T2 @
hosts: all
8 l. U/ [' r: j5 r! | gather_facts: True# w& X( d/ H6 Q* V6 D# w. K1 A+ m/ a
vars:
# T9 G# X% k( O5 T) f+ z DOCKERHUB_URL: registry.aliyuncs.com/google_containers
* `+ ~0 t7 Z) B' \8 J7 P$ v handlers:/ m8 _4 F, _3 w# v; J1 P4 f
tasks:6 x5 `) r" v9 z4 _
- name: mkdir -p /etc/docker/2 P6 O; n8 v+ q; P! Q
file:
0 T: a5 i3 u1 b Q6 n path: /etc/docker/, V6 s* }1 r" j5 J
state: directory! G) y; P) K' i' R
- name: install docker/ Z7 Z; B: }3 ~6 O" R& q
yum: name=docker-ce
* W( Z4 f$ |7 c* P& s5 K, D - name: change config of docker
# [, k, y: @) w; L7 } shell: |& C o3 f) I1 I
cat > /etc/docker/daemon.json <<EOF
2 d* j* H6 [/ ]# d5 H& C/ h {"exec-opts": ["native.cgroupdriver=systemd"],
0 \9 h6 O; ~; d( H& o "registry-mirrors": ["https://registry.aliyuncs.com","https://registry.cn-beijing.aliyuncs.com"]
1 R% s0 A) G {5 B) V `- o* _ } v# k# T& o4 U6 r
EOF
" N9 Q' H8 O9 R: U tags:
0 a5 J! O' s- c% | - config
4 E7 Y0 u0 D& Y1 S7 t - name: add systemctl
; @7 m5 E$ b) o. F systemd:
4 N$ l- a% g2 Z name: docker.service6 l1 h& E# n6 w9 F
state: started #指定服务状态,其值可以为stopped、started、reloaded、restarted、running
8 R, C+ g) }: U1 l/ y enabled: yes #指定服务是否为开机启动,yes为启动,no为不启动* i7 s; K$ m9 q$ ]; T& g
daemon_reload: yes #yes 重启systemd服务,让unit文件生效; d L; r+ j8 L
tags:
; i8 Q1 J* E; r; W4 G: x - docker-daemon
4 X3 |4 l7 }" s. _7 s' y - name: docker login #在阿里云镜像仓库开了个账号,用于同步墙外面的镜像(https://cr.console.aliyun.com/cn-beijing/instance/credentials); K5 ?3 u/ \7 b
shell: docker login --username cyxinda@163.com --password *** registry.cn-beijing.aliyuncs.com
- d! @# N; L& F5 v* S. [ tags: login8 D1 q4 i/ y# o: o* ?5 d
8 W$ e5 S* n& O- E) ~: u执行ansible-playbook -v 3.docker.yaml完成安装docker的工作
8 y$ P2 y7 h5 ^+ h开始创建ceph集群/ q" t9 n: z. H/ r0 W! M+ |# P# t
安装cephadm
( @ z0 Q: J f, N9 ~0 J- s# v0 B继续使用ansible为集群中每台机器安装ceph4 y3 w% [& p7 m$ {8 `: _
剧本cephadm.yaml内容如下:" e: W$ s9 U3 L) d( u
yaml 体验AI代码助手 代码解读复制代码- name: download ceph
# m7 E, g7 w% ~2 O' v hosts: 127.0.0.1
% C$ y$ q, a, J. p% _" T connection: local
2 d- ~. O6 P& G0 v; o' ^/ P gather_facts: yes
5 h4 C% B7 H/ C, d7 D; V- m5 Q tasks:
6 I$ r% ]/ }( { - name: download cephadm: F a# Q E2 m
get_url:
1 V$ r3 S7 U! ?, }6 N, N) [ url: https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
+ S, \* g) w; g7 ? dest: /tmp/ceph/cephadm% t; _2 L. G) o8 T
force: yes
$ K3 G( O3 m3 Z3 {' j% r) Q* V mode: 755" \ G6 o9 b3 y' _; i$ j
timeout: 600
6 \, S$ z; \- g) L$ K- name: download cephadm and install
2 ]& e, l0 e Z$ m hosts: all
# Z: c' `4 Z5 b# X/ B gather_facts: True. p+ o5 R4 G' z J) R4 v
tasks:8 O3 m, F$ x0 L1 r) ~" l- W
- name: mkdir workdir
5 x: {$ y2 ?6 A: o6 L" c) L file:/ {, t; p& }5 U8 }* {" G6 v# T4 t O
path: /tmp/ceph) V2 R! E E; u. Z
state: directory0 Y5 b. I! T& p2 `' ^, K: f \
- name: 分发cephadm安装文件
7 ^- G- F1 ]8 L& n- S6 G' t copy:
3 D$ ?3 G' c- A, { src: "{{ item.src }}"9 P \: @3 _+ {2 e7 |2 q
dest: "{{ item.dest }}"
: Z$ o. a' }7 {) } owner: root0 z) @% }6 v* x* d, s
group: root) ~( v. g. l: P( `
mode: "{{item.mode}}"
) }/ R% H+ _5 m4 O w1 z0 ? with_items:7 e4 ]2 e. W' g( G
- {src: "/tmp/ceph/cephadm", dest: "/tmp/ceph/cephadm",mode: "755" }1 J l' S1 Z0 o2 c
tags: cp-cephadm1 v( g- X O$ i. N# f, }6 y# b* d5 r) m9 |
- name: install python3
! Y* f, M- V/ | yum:
7 {1 O/ }. S1 P; s: ?0 _, o$ U d name:
6 \( i* k5 F z3 n, t - python3
' [7 j; W* @. S9 F5 U - chrony, \& P% @4 e. K+ U
state: present b l- L& n( T8 A+ H2 Q
- name: add systemd* y4 S2 u( W+ t, j! O n4 m2 {
systemd:
: Q# t8 G- P) _, W name: chronyd
+ K$ S$ V4 h3 Q) x) T# K0 i state: started #指定服务状态,其值可以为stopped、started、reloaded、restarted、running
' N6 i# i% G. l- e& f' t enabled: yes #指定服务是否为开机启动,yes为启动,no为不启动
" H, e* e: H, W8 Y6 y% O- b2 F& v( P7 G daemon_reload: yes #yes 重启systemd服务,让unit文件生效
1 m+ L1 O9 ^$ a, D - name: add ceph release
4 u" P# E" m" }1 n! T shell: /tmp/ceph/cephadm add-repo --release octopus
3 I0 L' ^& y9 J8 h/ r4 G - name: install cephadm
3 |: I0 ^8 {5 H. ] shell: /tmp/ceph/cephadm install
# Y. q0 M% ] n9 }5 b, T$ J* g - name: which cephadm: {0 P( v/ W# M; o" B+ [2 J
shell: which cephadm6 h% j( B: m2 C5 U z, j" W
register: which_cephadm; b8 q# k" B! x0 P/ I
tags: which-ceph- Z1 o: H* V6 i# E/ o# w8 N6 g) J
- name: show
0 M: c. N1 W6 A/ Z$ D+ }8 O debug: var=which_cephadm verbosity=0 #check.stdout 显示出的信息会看的更清晰点
% D0 b0 Y- T% A' h+ ] tags: show-result0 y$ [: @: M' w$ O& _
0 G6 m c: h2 h
执行ansible-playbook -v cephadm.yaml完成ceph的安装3 Z6 V( f7 n) P, E q9 {
# n4 K$ k8 z( q1 Y1 m9 P引导集群0 h2 ^7 t1 O& \ J: @
将master0作为引导主机,在其上面执行如下引导命令:7 R" x4 L7 {4 J- Q( g, K5 r) [4 }
bash 体验AI代码助手 代码解读复制代码8 N' [$ I( D/ {! \( m6 E
2 b5 C3 y! }9 L$ G* F$ N3 L[root@master0 ~]# mkdir -p /etc/ceph$ n/ c0 G! E8 h
[root@master0 ~]# cephadm bootstrap --mon-ip 12.70.10.161
, f! W5 r6 x, g4 oVerifying podman|docker is present...
2 v. E# y( q) Q4 n; rVerifying lvm2 is present..." K: K! W2 @) q; ^
Verifying time synchronization is in place...
* n6 i, ^+ z( n E6 ^6 j0 QUnit chronyd.service is enabled and running @& {/ P1 h; H. \
Repeating the final host check...% e: ]3 d Y* Z- j/ a; Q- @
podman|docker (/usr/bin/docker) is present3 w( f9 X; C8 X }6 z
systemctl is present! c |4 F/ b" \# g/ c; s3 j
lvcreate is present
. B: G0 g W: g4 ?2 E+ r: K5 Z1 K2 FUnit chronyd.service is enabled and running" D. I# M, u5 F \4 c: C
Host looks OK
2 r3 y" w6 H! @: ~# a: [, P. eCluster fsid: e3386564-bb02-11ec-af56-525400299ff7
I' d7 C, \; L- eVerifying IP 172.70.10.161 port 3300 ...+ K5 P$ ]6 @' ?( f' b
Verifying IP 172.70.10.161 port 6789 ..." H0 s* O3 s7 @; I" y5 E0 G! N# {
Mon IP 172.70.10.161 is in CIDR network 172.70.10.0/24
- o& D' ~' d& j aPulling container image quay.io/ceph/ceph:v15.... z" \1 p' ]+ k& m+ m6 B8 ^
Extracting ceph user uid/gid from container image...; C! f0 v* |. u. t$ }5 R/ V- I
Creating initial keys...
. e2 D5 d+ F& Z" bCreating initial monmap...6 C" L8 w* {% g% U
Creating mon...
* o: D6 E, l, S# xWaiting for mon to start...
" Q! S8 e0 f n) c3 fWaiting for mon...8 @1 k8 v. f2 a7 U: [7 q
mon is available7 U2 n4 w1 w2 f, `
Assimilating anything we can from ceph.conf...7 c5 B4 K6 M5 \" c7 D! ?
Generating new minimal ceph.conf...: I" z+ h; `/ o$ q7 [
Restarting the monitor...% n8 _/ }' y+ u) L" F( o4 \2 p4 S
Setting mon public_network...# Y: B. m- `5 ^6 [
Creating mgr...
, @$ |% e% B8 {5 R; PVerifying port 9283 ...
) O, W4 l* P _% IWrote keyring to /etc/ceph/ceph.client.admin.keyring+ x }/ U) Z. Q( c L+ k
Wrote config to /etc/ceph/ceph.conf
: Y+ l6 g2 E& Z8 t$ b( W6 [9 jWaiting for mgr to start...- H* P- o6 I1 ^9 {8 e
Waiting for mgr...6 y& U# K! u5 G4 m9 k$ M! s
mgr not available, waiting (1/10)..." q+ b+ K S6 N" [
mgr not available, waiting (2/10)...
' P5 ?, T1 ]# w+ ~# G' Amgr not available, waiting (3/10)...
. X" {" D% z; a/ x w* H- pmgr is available$ V+ R- n$ j2 |9 Z
Enabling cephadm module...
! f+ }6 p D* rWaiting for the mgr to restart...
# l( Y2 U L: U; yWaiting for Mgr epoch 5...
. ? i: L9 M; H$ v" S) M5 HMgr epoch 5 is available+ Y) c/ B# w) p7 t& i* S
Setting orchestrator backend to cephadm...
% X5 S% d4 ?4 C# X/ m5 gGenerating ssh key...
( [3 }1 ~; r, [; K1 UWrote public SSH key to to /etc/ceph/ceph.pub& h% G- I2 }* `. q( Y) A3 X8 p6 ^" o
Adding key to root@localhost's authorized_keys... l& ?# Q2 X& O5 V. `3 `$ m; ~
Adding host master0...! V3 \3 a, L2 s$ o
Deploying mon service with default placement...
6 S, ?: o1 ?4 n8 NDeploying mgr service with default placement...' R5 X: l4 J2 ^, M
Deploying crash service with default placement...
8 `: `- x/ [; g/ m$ P I2 Y: j: ZEnabling mgr prometheus module...
3 x& M0 I$ n4 w' k* S% wDeploying prometheus service with default placement...9 O8 R' v& y1 J& L8 D. R
Deploying grafana service with default placement...
& ~) a2 l; `: l6 C' j \Deploying node-exporter service with default placement...
) d7 |3 `; O9 J6 LDeploying alertmanager service with default placement...
& B$ W9 U+ V) ZEnabling the dashboard module...; _" k9 \3 D! M( m& ?. o
Waiting for the mgr to restart...
/ Z* g( ~6 o: O5 x" ?6 G: c2 n) qWaiting for Mgr epoch 13.... }3 Z7 Q! i5 t& g5 U0 Z
Mgr epoch 13 is available
: ^: c9 l5 U1 U/ \8 }+ @% }Generating a dashboard self-signed certificate...
" E7 r2 k7 X a7 lCreating initial admin user...2 \; ?: F( v8 \( q
Fetching dashboard port number...! `, \) N' O: T
Ceph Dashboard is now available at:
! A; Q# J4 ?- S" D ?' u0 ?7 G3 R/ L P6 `8 E& ^
URL: https://master0:8443/
3 O! X2 Z( m. y. R5 r# l User: admin; v4 D$ W' r: C* U/ b
Password: vym1bdeajd, ?8 F5 q& [* r7 t& @4 t$ C
) K9 V3 T5 a+ T FYou can access the Ceph CLI with:
; c* n+ @6 V f m; R- z# K& Q/ e, E) Z3 o/ n1 }
sudo /usr/sbin/cephadm shell --fsid e338664-bb02-11ec-af6-525400299ff7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring0 U% `# ]& z$ }' W& t; g/ w0 I" v
s2 W6 j4 a9 m1 ~
Please consider enabling telemetry to help improve Ceph:) T) \ B2 Q2 g% F
, N* v0 T/ w4 [' \% }4 R L+ i ceph telemetry on
$ G! ~. Y1 g0 g- D; _ U' _5 R: V
% W7 S2 U; w& l, x+ \% RFor more information see:* b, U( h0 A+ ?
# C k2 [: H3 U! q https://docs.ceph.com/docs/master/mgr/telemetry/: R1 i9 r+ H) U/ m' A, H' O
3 d& I! y# n r" C1 @; h
Bootstrap complete.
! t& R% j& A& l3 @
' q8 N+ _: o3 b6 n' B; I- y. m- ~
在命令执行过程中,有如下提示:
4 R7 u9 r% R! t; J8 _0 lbash 体验AI代码助手 代码解读复制代码Ceph Dashboard is now available at : 2 Y$ l/ I: u; Q4 }# w" z) \: V
URL: https://master0:8443/
P) p. o' O4 F. P7 c9 t; ^2 ~2 e User: admin. v- L6 v; m5 u, o7 d6 b
Password: vymdeajd
* U1 n: D# _, \% E7 k
3 H- w; ]" s# b7 Y( w& i按照提示,在浏览器上面可以访问:
0 ?; w6 g/ T5 d, T/ y Z' r0 q' P" y! k6 `
进入到管理页面后,如下:& s- H j' r3 D4 h! K: E$ z' N
& c# P" d4 ]3 M) b7 H% x
参照安装文档,该命令将:
: r$ N( P1 g8 g- L/ z k在本地主机上为新集群创建一个监视器和管理器守护程序。; g# k1 X( I7 q2 z0 D
2 X) R$ k* I4 ^& e+ }
为 Ceph 集群生成一个新的 SSH 密钥并将其添加到 root 用户的/root/.ssh/authorized_keys文件中。
, R K% H, V% C: G* p$ J# Q; a% ]将与新集群通信所需的最小配置文件写入/etc/ceph/ceph.conf.
$ B& `' [; c- C3 H) g: x# d; F' T' C
client.admin将管理(特权!)密钥的副本写入/etc/ceph/ceph.client.admin.keyring.
7 b q8 ]$ b# J' @1 [: a: o/ ~0 P将公钥的副本写入 /etc/ceph/ceph.pub.
9 l# ~1 G' t1 S! g A$ y/ W' ], ~/ K启用 CEPH CLI(必须)
6 X: b4 k# u! ?继续在引导机器上面执行如下命令,即可开启ceph shell client$ J/ i+ i( d4 z$ ]& @ s2 A _0 ]& F
注意:后面的ceph命令,均需在ceph shell环境下执行
( y O1 N" u! L% g* Pbash 体验AI代码助手 代码解读复制代码* S0 a& R0 h9 g4 u6 u
( i6 E x) q& ]& k: {0 n. S
[root@master0 ~]# cephadm shell
6 M" S: S% _! x1 ]1 mInferring fsid e3386564-bb02-11ec-af56-525400299ff7
& l, x. X9 k# t1 G3 B/ W3 xInferring config /var/lib/ceph/e3386564-bb02-11ec-af56-525400299ff7/mon.master0/config
" U0 s# l9 }% E( [3 Z0 ~+ G/ ZUsing recent ceph image quay.io/ceph/ceph@sha256:1b0ceef23cbd6a1af6ba0cbde344ebe6bde4ae183f545c1ded9c7c684239947f2 S) T# C! `5 u* P
$ x3 ^/ Q$ O, t- V" K' G# V[ceph: root@master0 /]# ceph -v
' }0 `" b2 |4 K1 ?4 i" Rceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable); y5 t/ t% m8 p9 @! |% N. w$ \. P5 ]/ d
. P2 j/ N6 N( T* H
# z0 G; ?) O# K6 L[ceph: root@master0 /]# ceph -s% m! r* x" R7 M
cluster:& Y. S. K9 c% ?- A8 q4 E3 T) q2 w8 O
id: e3386564-bb02-11ec-af56-525400299ff78 g5 C5 S; ?3 z$ L* k8 V+ o
health: HEALTH_WARN
5 n3 O+ S4 C, q! x5 e. j( F. ? OSD count 0 < osd_pool_default_size 3" w! }* w# S, [! {1 x' R
! w B: ?/ h' ?( S( @* C4 S' A
services:
' g& C4 A. `; T i0 A$ c4 u mon: 1 daemons, quorum master0 (age 48m)* J: {( |6 Z d8 A6 A" \9 I4 @. J s7 Q
mgr: master0.ojikws(active, since 47m)
7 r; Z% s4 ~ C# _; u osd: 0 osds: 0 up, 0 in
- P; |1 y6 W- A+ _5 v$ ]+ N0 y
2 h2 s: O2 Q) ^; Y data:
0 I) s& n- F/ [& A pools: 0 pools, 0 pgs
* B7 q" ]( h/ J$ S objects: 0 objects, 0 B' @7 }9 R( k# w; z4 G& t
usage: 0 B used, 0 B / 0 B avail6 Q. _: q/ P4 E
pgs:
/ L6 H+ k0 ~) F% B2 F. ]6 k4 ^7 A: H$ |% h, z8 @7 M9 R, ?1 h
将主机添加到集群
% k+ K6 Y0 N: t3 W' p0 }0 D" ^3 t$ F0 Z4 { E0 V/ Q
在新主机的 root 用户 authorized_keys文件中安装集群的公共 SSH 密钥
3 M3 W" J* o- K3 Z; F$ c u7 N Y, ]! c* g K. [/ I0 F
bash 体验AI代码助手 代码解读复制代码/ w7 M5 q" f! K
& [- E- u+ k( [
[root@master0 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@master18 x- l$ E0 ^) R; Q: e/ w
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/etc/ceph/ceph.pub"
7 q- g, m+ B, ?. uThe authenticity of host 'master1 (12.70.10.162)' can't be established.
" m6 l5 \+ {& g& J: |: AECDSA key fingerprint is SHA256:J40vT3JXLYRku40nj9oOq1XQMbnkTXZ2Qc5IDFAy4xc.
1 v" s: q+ n9 v6 R. yECDSA key fingerprint is MD5:8d:ef:46:df:ce:06:7d:86:05:e9:04:ad:68:12:40:8c.8 r H* M# \0 ?7 U
Are you sure you want to continue connecting (yes/no)? yes
5 |% P6 w4 ^. o# u1 |root@master1's password:
* {* Y5 o+ p" X s7 K4 e
2 U& Z% q- p( Q4 [& N U, [, Z; vNumber of key(s) added: 1
& j0 H' U, R3 x% F+ {% y d( S& a" y# U# }3 L
Now try logging into the machine, with: "ssh 'root@master1'"
9 _8 n" H+ ]9 ?0 Z$ z5 k( Y. b: Cand check to make sure that only the key(s) you wanted were added. A, z* B. x- C: B" j
; I' d; x3 C b7 \9 d; ^! m[root@master0 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@master22 v$ ^, X1 E3 C
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/etc/ceph/ceph.pub"& @% a5 x7 v1 d; K" J U
The authenticity of host 'master2 (12.70.10.163)' can't be established.7 ~3 z1 }7 u8 H6 B4 I
ECDSA key fingerprint is SHA256:J40vT3JXLYRku40nj9oOq1XQMbnkTXZ2Qc5IDFAy4xc.
, N+ m% i! M) Y; V0 t5 o7 E8 p. j) UECDSA key fingerprint is MD5:8d:ef:46:df:ce:06:7d:86:05:e9:04:ad:68:12:40:8c.1 D9 F7 D9 }/ y7 h" Z
Are you sure you want to continue connecting (yes/no)? yes
4 V$ g3 j# \: ]- H- V5 D. |( droot@master2's password:
3 ^- J* G. ]4 ]" ^" p3 S& o
7 E/ r" J2 i) B9 q3 C9 R( }; eNumber of key(s) added: 1
- L i0 f& \4 S+ \! l8 F6 H
; c" x5 C) v4 Q* Y& S! |Now try logging into the machine, with: "ssh 'root@master2'"
, L3 b/ P$ c2 Q' o" [9 i) p; H* y- band check to make sure that only the key(s) you wanted were added. u2 v8 @& ?9 l% ^0 j; p9 z
Y+ B" i# |- C$ x& o0 G5 P" Z这部分本来也可以用ansible做的,但是懒得写脚本了,就两台机器,也就罢了% A: {0 ?. Q" E _) S. B
添加机器到集群:
: }! S ]. |- L- X, W, lbash 体验AI代码助手 代码解读复制代码
( J% G' n3 j* P/ T! r$ {9 }1 P$ k9 S7 E8 N: d( h5 g
[ceph: root@ceph1 /]# ceph orch host add ceph2 12.70.10.1625 z, @+ N! B2 O, _+ V
Added host 'ceph2'
4 X( L; t+ m3 `[ceph: root@ceph1 /]# ceph orch host add ceph3 12.70.10.163
( ] I+ h4 Z! X. xAdded host 'ceph3'
; g$ W) {% Z7 D v- P[ceph: root@ceph1 /]# ceph orch host add ceph4 12.70.10.1643 _( B! J! A5 g, k
Added host 'ceph4'
5 e8 X% ^3 H: \% g) v3 E[ceph: root@ceph1 /]# ceph orch host add ceph5 12.70.10.165% W' m9 G5 i3 A3 L
Added host 'ceph5'
$ K# q K# J! I" P- J5 d$ g[ceph: root@ceph1 /]# ceph orch host add ceph6 12.70.10.166
( |, N, S7 r+ g+ e- A% BAdded host 'ceph6'
, i ^) V" [% `% O& s6 j7 P' y( M. z/ l
& n: _ `; [" L9 E添加多个监控器. Z$ C4 @3 T+ V% `. D7 _
配置监视器子网:
u5 s4 m% k$ E$ ubash 体验AI代码助手 代码解读复制代码- H0 v" c% G/ H6 _ ^
[ceph: root@master0 /]# ceph config set mon public_network 12.70.10.0/24
* B; G3 B4 @' U# l[ceph: root@master0 /]# ceph config set mon public_network 12.70.10.0/24, i4 _- m9 E6 Z, S
8 f$ ]: O( \1 m$ {0 E& @$ J- ?
##要启动三台监视器,需要调整监视器数量:
* t0 v4 v }+ ]% i5 G[ceph: root@master0 /]# ceph orch apply mon 3 9 ]( T/ p( ~& H @4 V8 s4 x
Scheduled mon update...
, R0 ^; z# q" l$ m1 ]3 s) o3 {* ?! ^! e5 b9 k
按照官网的说法:Cephadm 仅在已配置子网中配置了 IP 的主机上部署新的监控守护程序
5 P+ M- r- |, q4 t- V' B, _" c3 `在一组特定的主机上部署监视器,请务必在此列表中包含第一个(引导)主机。 C* U# M$ V. H1 o: e
bash 体验AI代码助手 代码解读复制代码
' {( y" ~! J- y( U a X7 b5 Q. k" r% Q6 [# Q$ t( ^
[ceph: root@master0 /]# ceph orch apply mon master0,master1,master2; A" E6 ? l) M: I
Scheduled mon update...
6 I" K# s! y* ~, v: v##加标签' p4 ~. u9 O# `" p
[ceph: root@master0 /]# ceph orch host label add master0 mon
4 y: {5 I8 a2 Q% @1 DAdded label mon to host master0 K% Y2 J" G8 s7 M
[ceph: root@master0 /]# ceph orch host label add master1 mon
$ Q9 A; F# t1 zAdded label mon to host master1
+ f$ W- ~- W& n% ]' {[ceph: root@master0 /]# ceph orch host label add master2 mon
- @7 g9 C0 [* Z. |# ^( F6 d. EAdded label mon to host master2
* c/ F; \' [3 j9 \1 L
7 d! j$ b: @1 W% B# n# Z[ceph: root@master0 /]# ceph orch host ls3 U) {- i+ S" Q9 ~" E
HOST ADDR LABELS STATUS5 G Y9 E+ V8 i
master0 master0 mon0 R9 m9 p1 ~% y1 \9 C6 \1 J
master1 master1 mon+ k6 U5 e& `+ B
master2 master2 mon
7 V O& V Z! q. [" b5 t) U) ], Y
0 C0 y r! r) g& z: T- w[ceph: root@master0 /]# ceph -s
) R6 V9 a$ s- q- n6 z# l. i4 o cluster:5 H0 v% x0 l( A: R
id: e3386564-bb02-11ec-af56-525400299ff76 |' A# E' |# ^! ~/ A9 j
health: HEALTH_WARN
) R0 x- v* o9 r" u# o4 S5 c. B OSD count 0 < osd_pool_default_size 3
! q( q$ {' y- f7 Q+ S0 c" c# P; l* l Q/ x& k
services:
) t1 j8 p# P- h$ V& j- | mon: 3 daemons, quorum master0,master1,master2 (age 88s)
v6 R( ~( ?* i) ?7 _, ?4 N mgr: master0.ojikws(active, since 73m), standbys: master1.uxevld
/ J! @( E4 U; C) b osd: 0 osds: 0 up, 0 in
) F% ~. f+ n; ~3 F# X. z8 t7 T# { U' x; t7 s8 V
data:) S/ V3 G% c, a+ G1 b
pools: 0 pools, 0 pgs
9 ~7 T1 W) x' N" @! |9 E8 g5 x5 H objects: 0 objects, 0 B
2 ^7 J9 r* \, g$ ] usage: 0 B used, 0 B / 0 B avail% C8 D. L9 M" D, c5 p- n7 {
pgs:, L5 V5 E! f6 B: N/ p! e! G+ O6 Z
' O; C/ G* J6 V/ |; U& _4 s加入ODS
, q! {% i, \' O7 O3 {可以看到集群三台机器上面,一共挂载了6块磁盘:* e) E1 N7 Q2 J7 u0 I! b& ]
bash 体验AI代码助手 代码解读复制代码% }1 I& O" q8 ~$ O; D! D
! W3 H7 G6 J/ A% |; m' Y9 {& v[ceph: root@master0 /]# ceph orch device ls6 Q9 ?+ r& W B% U- i0 N" L# T
Hostname Path Type Serial Size Health Ident Fault Available
3 g7 @; F; K, ?9 a3 e, M2 }+ Y7 qmaster0 /dev/vdb hdd 536G Unknown N/A N/A Yes
# ?/ s& {( k. Z& u# Umaster0 /dev/vdc hdd 536G Unknown N/A N/A Yes
p# A9 E# S; t( g' W$ kmaster1 /dev/vdb hdd 536G Unknown N/A N/A Yes
7 o% f0 \0 c; d8 w4 ]master1 /dev/vdc hdd 536G Unknown N/A N/A Yes4 r' b7 m4 ?3 Q
master2 /dev/vdb hdd 536G Unknown N/A N/A Yes
, A1 r% s, H. `; qmaster2 /dev/vdc hdd 536G Unknown N/A N/A Yes0 t7 [) |& p7 U. T# N" I
5 z; v+ O/ Q! I9 i4 y当然在每台机器上面可以列出块设备(/dev/vdb 和 /dev/vdc):: `$ S) ^. @9 y+ ^9 D3 ^
bash 体验AI代码助手 代码解读复制代码
# ^9 e. ]) W D: K8 a+ r& ~: G' ^' R
[ceph: root@master0 /]# lsblk, t. N- Q% B: l% R# g5 H
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT3 q4 }3 N! b6 X/ _/ t" G
sr0 11:0 1 1024M 0 rom' ~$ @' k' ]6 ^3 D; {. n
vda 252:0 0 500G 0 disk
4 i$ ]8 R" d( d8 ? p$ N|-vda1 252:1 0 1G 0 part /rootfs/boot& }( z1 V b$ ~# f" ^1 i; L
`-vda2 252:2 0 499G 0 part
) x: L' d3 g- D# M |-centos-root 253:0 0 50G 0 lvm /rootfs
( }- T$ ] p9 d3 @8 N- Z* s+ V |-centos-swap 253:1 0 7.9G 0 lvm6 y' L/ U0 x E0 \+ z( j: C. @
`-centos-home 253:2 0 441.1G 0 lvm /rootfs/home8 ?$ c+ L; \0 A$ j( p9 Z; E% q9 W0 q
vdb 252:16 0 500G 0 disk$ o" W0 I M9 \) p# e8 g2 t. `
vdc 252:32 0 500G 0 disk
j9 A5 B$ H3 {8 L( C8 M
7 r3 D+ ^- e" P" o- J( d3 M% i按照官网说法:设备满足
1 `+ Q" x4 e( l# s j如果满足以下所有条件,则认为存储设备可用:: F3 n1 k3 [7 U& u/ `5 B: I
0 Z: E/ Q1 d" {6 w, ?7 y5 N2 S
设备不能有分区。
5 c" x3 k6 K* D* J7 Z设备不得具有任何 LVM 状态。# l. M2 Y% l4 W6 z3 J
不得安装该设备。 I# C- X! y- r' \; I
设备不得包含文件系统。# I3 B4 P$ P# g7 }, T) b" `4 [
设备不得包含 Ceph BlueStore OSD。
8 X2 S' |: Y5 Y: G0 D+ t设备必须大于 5 GB。! A) E. _' r1 D. _& X% {
) Y: V; d5 a7 P2 N. oCeph将 拒绝在不可用的设备上配置 OSD
& m# _2 ` o% s可以使用如下命令,将所有可用的磁盘添加到ceph集群中:
" R' X L0 ^6 y, \# W1 \- `% ?) w- K6 j4 Qbash 体验AI代码助手 代码解读复制代码% _" I: S; n# P$ `. W: d: T8 M$ N
4 o; r+ T& f5 b; F6 `1 N[ceph: root@master0 /]# ceph orch apply osd --all-available-devices
* o' }4 Z% ~+ H8 K0 c) nScheduled osd.all-available-devices update...
1 r. G* `, {2 J[ceph: root@master0 /]# ceph -s# j" j8 Y) [6 v: L0 M6 i1 z
cluster:
6 f7 z9 e4 [2 s8 k( t; y% O id: e3386564-bb02-11ec-af56-525400299ff7% ?) Q( ?6 q0 g# E/ _
health: HEALTH_OK
- d: d; w$ X; i2 l* h8 V) I% u, P f4 Y8 K
services:
+ X( _+ K* A( r f7 V) c7 q1 u. H mon: 3 daemons, quorum master0,master1,master2 (age 12m)
$ z' e) l" I0 s4 a, @) ~5 } mgr: master0.ojikws(active, since 84m), standbys: master1.uxevld0 _. J0 L% o! k0 \; n: t
osd: 6 osds: 6 up (since 22s), 6 in (since 22s)
9 W) @* ]( d* l+ E
2 h: F7 n. K3 M& l! Q" B1 K: @! h data:+ _- E1 q! p' i$ x) H" W/ Z
pools: 1 pools, 1 pgs
' v1 S6 n; u4 Q8 H objects: 0 objects, 0 B/ K) ^" l- j- @( p
usage: 6.0 GiB used, 2.9 TiB / 2.9 TiB avail4 I; K& w; B2 a" j2 w
pgs: 1 active+clean
6 S2 u; c" Q8 }- u8 L$ @" l* ^
( a, U* r: s. Y' e; o) q也可以单独加入:
8 ^! R5 d/ p ]- qbash 体验AI代码助手 代码解读复制代码7 d( U' a* z; O* N, o2 W
/ f' h, U/ q$ Y! E! Z& N6 z% `
[ceph: root@master0 /]# ceph orch daemon add osd master0:/dev/vdb
, `( k$ X1 K$ d) T7 X[ceph: root@master0 /]# ceph orch daemon add osd master0:/dev/vdc5 K. y( v. m4 f' j( m7 q! b' s+ X1 F
[ceph: root@master0 /]# ceph orch daemon add osd master1:/dev/vdb
: o6 L2 L, R" v. ?6 b3 }( q7 g" c4 j3 a: O[ceph: root@master0 /]# ceph orch daemon add osd master1:/dev/vdc
* S' W3 x4 B2 n# \& h% B$ b- z[ceph: root@master0 /]# ceph orch daemon add osd master2:/dev/vdb: R7 A) j1 C. W& S; r5 G
[ceph: root@master0 /]# ceph orch daemon add osd master2:/dev/vdc. \! H$ j, l! r3 J
7 ]2 _+ t% n' ~% K
在前端看到:# \& m5 u. S: H9 s4 [
D9 b0 F; E. }( b' G9 `* E1 ?
4 b1 X0 b- V* W7 \/ G挂载块设备
4 c7 D7 b5 K' ?; R4 C' J7 ^2 Nbash 体验AI代码助手 代码解读复制代码) L' T* a: r$ m4 Q( l6 w" {9 Y
! S6 Y) B. c) A) M! X8 K% X
[root@ceph101 tmp]# ceph osd pool create test_rbd 32! V5 Z9 T8 b6 `; ~; c& t
pool 'test_rbd' created
5 m$ F" H+ `- ^; A0 |[root@ceph101 tmp]# ceph df* ?4 g1 Y3 _) f. }7 |/ z& v
--- RAW STORAGE ---: k) l9 }% S s# Q
CLASS SIZE AVAIL USED RAW USED %RAW USED- N) ?' Z( N0 L; z. v/ n0 h
hdd 5.9 TiB 5.8 TiB 1.1 GiB 13 GiB 0.22
: z/ c! Q# W, ^4 N. v# w# S% [) `& ~TOTAL 5.9 TiB 5.8 TiB 1.1 GiB 13 GiB 0.22* `; D" K4 R0 M) q
* G Y' [& t! n3 S4 @
--- POOLS ---2 I: I+ U5 I) `* d
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
" t! c! Q8 u3 q9 D3 r8 a5 ddevice_health_metrics 1 1 0 B 0 0 B 0 1.9 TiB9 d- \( W6 j, W' M$ b, C8 k) S
.rgw.root 24 32 22 KiB 36 6.6 MiB 0 1.9 TiB1 g3 i4 K- k! k3 J
zone_01.rgw.log 31 32 26 KiB 965 55 MiB 0 1.9 TiB
# B8 d2 _: X" U Y6 {zone_01.rgw.control 32 32 0 B 8 0 B 0 1.9 TiB
' [# G7 Q, W7 R: _9 V, \zone_01.rgw.meta 33 8 5.5 KiB 16 2.6 MiB 0 1.9 TiB* u' `- o' t9 [4 H5 h6 I
zone_01.rgw.buckets.index 34 8 672 KiB 55 2.0 MiB 0 1.9 TiB
* U& `- ~7 M% k3 A% Xzone_01.rgw.buckets.data 35 32 2.2 MiB 12 7.9 MiB 0 1.9 TiB& ?; V$ r I3 C: b H& i% j
zone_01.rgw.otp 36 32 0 B 0 0 B 0 1.9 TiB
0 }/ }. B2 y5 G% _! q1 Gcp_pool 42 32 1.4 MiB 2 4.4 MiB 0 1.9 TiB, x0 n7 Z c4 O$ p4 W
test_rbd 43 32 0 B 0 0 B 0 1.9 TiB
' l+ M" F0 Q2 Q) a( h##创建镜像
- x, |9 u4 k! H
6 X$ a6 X5 q8 F[root@ceph101 tmp]# rbd create test_rbd_image_1 --size 10240 -p test_rbd A# d, c. w: q, h
- z( n0 @9 ]' g
[root@ceph101 tmp]# rbd -p test_rbd ls
( i( o, `" F$ R* ^* ?test_rbd_image_18 M/ w1 p2 z/ _
( X9 w, c7 i, J% j
##检看一个RBD镜像的详细信息
; V/ ^& u- u* P& A3 B[root@ceph101 tmp]# rbd --image test_rbd_image_1 info -p test_rbd6 s' U; {$ D* w4 D, c0 I4 S
rbd image 'test_rbd_image_1':1 ]6 k; F6 x; ]( ?& m' S* |: j) n
size 10 GiB in 2560 objects
! N9 e4 j# Y, R/ C# h8 X# y$ k order 22 (4 MiB objects)
# O# x! Q E& d* {' S$ V1 g* n snapshot_count: 0 d) ^. W% |! L" g) d
id: 2836b2f53ea869 B, ^0 W9 U; {$ K7 Z& _( D
block_name_prefix: rbd_data.2836b2f53ea86
0 F% b Z. e/ J3 i: Y format: 2' u: l5 Q# G1 o/ _6 \# e1 ]* e r
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling6 {- L/ x" w6 V3 ? `
op_features:6 A$ b4 \7 u% D* m
flags:
3 E3 }1 r$ N* ]5 P( K1 V3 | create_timestamp: Tue May 10 10:38:16 2022
+ z/ Z+ y4 T: p9 k" z' m access_timestamp: Tue May 10 10:38:16 2022
B# @/ p; F7 H9 M" t$ k modify_timestamp: Tue May 10 10:38:16 2022
' W, N6 T8 _& x h* _% P journal: 2836b2f53ea86
- x% {$ M' h, h# S/ D2 Z mirroring state: disabled2 C& @# I/ T: l; M! a* w
8 w2 U B5 X( h
; T: J2 l2 j) |. U9 N3 R7 Z, Q! [; y[root@ceph101 tmp]# rbd pool stats -p test_rbd
+ @: f: K0 f- X y6 `; S. \/ I1 vTotal Images: 13 f- f+ Y# n [5 X/ C
Total Snapshots: 0. ^& O, T% G; p; u9 U
Provisioned Size: 10 GiB5 ~/ S" |4 `1 j# |. `
[root@ceph101 tmp]# rbd showmapped
! \+ D% \7 x6 c+ g; j8 H" O- ^id pool namespace image snap device. U1 y6 f2 q/ i( W- q+ X
0 cp_pool image2 - /dev/rbd0" H( U$ s1 ~4 W( R( U q
! v$ B8 _8 c' E8 T
[root@ceph101 test_rbd]# umount -f /dev/rbd0, V3 P6 o% T- N. i4 l
#或者& _! G4 F+ c* Z- b+ |( N
' M1 H4 f- P' k X( ~
[root@ceph101 test_rbd]# rbd unmap -f /dev/rbd0; C, ]* E# a3 ~/ c. V
##然后将pool关联应用' _3 m: O# ~- c, }4 W
- [' G( Z6 L# q2 m( l% U* S[root@ceph101 ~]# ceph osd pool application enable test_rbd rbd
0 P% G" n; [" P" H- Kenabled application 'rbd' on pool 'test_rbd'9 S" W7 B9 I( y
[root@ceph101 ~]# rbd map test_rbd/test_rbd_image_11 b" I2 n0 N) ~
rbd: sysfs write failed
# ?& z% [5 z$ d$ Y% p: SRBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable test_rbd/test_rbd_image_1 journaling".! [. p A+ h0 E9 G) |) ]: U% ]
In some cases useful info is found in syslog - try "dmesg | tail".1 u8 Y" V* q# ~: ?0 I6 l% r, N, A. X+ k
rbd: map failed: (6) No such device or address
& g0 C" B7 A7 D, V3 N0 i/ b4 f+ I( [1 a/ z. g6 \) G
[root@ceph101 ~]# uname -r0 Z! _ d0 k. @/ U: q6 f, E# V
5.17.6-1.el7.elrepo.x86_64
1 I% v7 I2 ?% @7 O" J" `[root@ceph101 ~]# dmesg | tail
; k, n% Q! d6 t[ 9.843030] random: crng init done
- {7 Y3 W9 \, S* K- I d6 Y[ 9.843034] random: 7 urandom warning(s) missed due to ratelimiting
% D+ x! e; w6 G* t4 a[ 10.669178] Bridge firewalling registered
7 y0 r, Y B) e; t2 n" V[ 22.239147] process '/bin/ip' started with executable stack" X0 F8 G! o. ?
[ 8185.140070] Key type ceph registered
; L' }0 K1 n8 E9 h) y[ 8185.140395] libceph: loaded (mon/osd proto 15/24)* z8 ~# r/ n! B; P1 n
[ 8185.141923] rbd: loaded (major 251)( n" ?7 ^3 l# }: s
[ 8185.158536] libceph: mon3 (1)12.70.10.184:6789 session established
! g8 ^1 i% h( N7 `% L Y[ 8185.160696] libceph: client175843 fsid 7a367006-c449-11ec-9566-525400ce981f
9 u% M; G8 N* X4 V; B M6 t7 a[ 8185.288221] rbd: image test_rbd_image_1: image uses unsupported features: 0x40
$ s5 m. U, \2 w##根据提示`[ 8185.288221] rbd: image test_rbd_image_1: image uses unsupported features: 0x40`,可以确定内核不支持的features是十六进制0x40,转成十进制是4*16+0*1=64,即2的6次方=64,journaling; J' _# J4 Y2 C* \
#layering: 支持分层**(0次方)**
; N1 U- I6 C# r8 M6 m) b#striping: 支持条带化 v2 **(1次方)**
( @' R( }, o9 b3 |#exclusive-lock: 支持独占锁 **(2次方)**
D- |+ p; F) |( x#object-map: 支持对象映射(依赖 exclusive-lock )**(3次方)**
- p% r* M- v- c* H#fast-diff: 快速计算差异(依赖 object-map )**(4次方)**
& d9 A# ?) W2 k#deep-flatten: 支持快照扁平化操作**(5次方)**) m+ s) M3 @- [' J8 G* r9 t% ^2 w
#journaling: 支持记录 IO 操作(依赖独占锁)**(6次方)**) Z o8 f* R* J1 J9 ^2 y
[root@ceph101 ~]# rbd feature disable test_rbd/test_rbd_image_1 journaling
. p9 n8 M4 Y4 b+ Q$ ?[root@ceph101 ~]# rbd map test_rbd/test_rbd_image_1) v z G' t4 r9 Y6 p7 Y1 K
/dev/rbd0
4 }4 g& A5 N5 l: A7 W' Y6 B H#lsblk 查看磁盘
3 l5 r2 o- K7 e$ }2 F- E3 j
. ]" a$ O. _- q3 M$ t[root@ceph101 test_rbd]# rbd showmapped1 f: W" x- B2 x) v0 d
id pool namespace image snap device& z( L, K! d% I4 I" J
0 test_rbd test_rbd_image_1 - /dev/rbd0
4 l$ P/ u. B2 Z[root@ceph101 ~]# lsblk. |4 P9 |% Z+ z {! G! w
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
5 k* d# p# w% R/ g2 U) v/ e. L4 Urbd0 251:0 0 10G 0 disk& \2 ~. U/ l0 V7 O# ]0 U- }0 L0 S
vdb 252:16 0 500G 0 disk
/ O8 y& E& H# H└─ceph--cbd3517f--a42b--41b9--bdb5--350597fb4873-osd--block--da454e2d--c289--430f--a685--9b437b5a3e00 253:4 0 500G 0 lvm
& D f# A2 Y+ z9 T( C% fsr0 11:0 1 1024M 0 rom9 N( x" N6 e* z
vdc 252:32 0 500G 0 disk
' l- i9 i& S7 I; w└─ceph--797b51d7--f835--43d7--a987--1316a2438933-osd--block--40e4dc65--08e9--4971--9187--2d05208bbb0d 253:3 0 500G 0 lvm
* n5 o4 j$ h7 q) Hvda 252:0 0 500G 0 disk. F' g! m- r1 N0 H9 o$ A
├─vda2 252:2 0 499G 0 part3 _9 I" O3 L" l2 R! s
│ ├─centos-swap 253:1 0 7.9G 0 lvm7 P4 i' ~! V" p- `5 E: k3 s
│ ├─centos-home 253:2 0 441.1G 0 lvm /home" b3 ]- h. ?9 m# m) R3 g0 \3 D
│ └─centos-root 253:0 0 50G 0 lvm /
; b; C& R$ R" ?4 Q( M└─vda1 252:1 0 1G 0 part /boot
" f X! o+ K% `3 i
8 p; V3 Q- a5 u6 B1 J# d
3 n2 ~+ `* G3 C" z* @$ L# 格式化磁盘
" Q* c0 W' T, T# H+ k/ @) ?7 A1 z9 G1 ?) f# \9 t$ L- O
[root@ceph101 ~]# mkfs.ext4 /dev/rbd0
- [3 a% p) U/ X1 j: s" J# kmke2fs 1.42.9 (28-Dec-2013)
4 R. s- ]7 L) t" ^0 d7 b/ N' ZDiscarding device blocks: 完成2 R* z- u7 T) z" z
文件系统标签=
! t# ?& r: O7 [5 LOS type: Linux, r; g2 t! g& z6 w2 M/ ]. m, F+ T
块大小=4096 (log=2)" v4 b! F7 M4 r- f
分块大小=4096 (log=2)
C7 `% O3 y; @: n; q, m- I- ~: xStride=16 blocks, Stripe width=16 blocks
7 h; s# h9 \5 z, i5 U6 J655360 inodes, 2621440 blocks4 G+ v6 n4 ^ [$ H6 a+ X& `8 M9 \
131072 blocks (5.00%) reserved for the super user
0 J' W F8 C. X0 {第一个数据块=0
0 ?6 q# j1 P pMaximum filesystem blocks=2151677952
5 x& p- h. P. k& H' u# }- I5 u80 block groups/ n+ P. M9 W* s. S5 i2 h
32768 blocks per group, 32768 fragments per group4 I8 Z) k! e; u- g+ e
8192 inodes per group
$ b' j$ w& c, iSuperblock backups stored on blocks:
# |3 u/ F4 l6 t( F9 H" ?+ W 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
- {) F0 m$ j5 I. d& {- c" ?+ e9 F$ j$ n
Allocating group tables: 完成
5 j# \4 I0 m o5 y" F* d正在写入inode表: 完成* Z. E# z* F1 c0 ^ ?) f) ~0 _* x
Creating journal (32768 blocks): 完成4 g* b. C2 X+ [8 b
Writing superblocks and filesystem accounting information:
2 V6 H/ X |9 m5 J& g( _ ?. r完成/ d! v( ] M! v' u) Y
. S9 A! G( J, L3 ?6 q" H; X+ S
##创建挂载目录
8 E* R* @ F3 @. |% v4 J# R[root@ceph101 ~]# mkdir test_rbd/0 C' {1 T* u# R( y& v4 x# X
#挂载
" q g; p/ t+ Q4 u9 N& S# F4 m) G[root@ceph101 ~]# mount /dev/rbd0 /root/test_rbd/ i, V. S1 O* H6 }2 c8 g; e- o
[root@ceph101 ~]# lsblk
7 L7 ^/ U! _' f1 \. kNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT( U" m- @# A6 d( q' Z
rbd0 251:0 0 10G 0 disk /root/test_rbd O8 T+ ^" H; q# {! o) s) V
vdb 252:16 0 500G 0 disk8 t$ Y1 P* N. c% u6 J
└─ceph--cbd3517f--a42b--41b9--bdb5--350597fb4873-osd--block--da454e2d--c289--430f--a685--9b437b5a3e00 253:4 0 500G 0 lvm$ j: U. k* ]9 D" w1 v
sr0 11:0 1 1024M 0 rom
2 `0 r8 [( b$ {# z/ W. E6 P" ^vdc 252:32 0 500G 0 disk
* f t) b' k& o4 s! `└─ceph--797b51d7--f835--43d7--a987--1316a2438933-osd--block--40e4dc65--08e9--4971--9187--2d05208bbb0d 253:3 0 500G 0 lvm9 y0 V6 T: ^' `! B0 u: O( m
vda 252:0 0 500G 0 disk( c+ F' i- _# q. V ~: e6 p2 S
├─vda2 252:2 0 499G 0 part% t0 ^( j- f$ f# m* |' F5 z
│ ├─centos-swap 253:1 0 7.9G 0 lvm: h' c. ^/ I2 w) p' S' }
│ ├─centos-home 253:2 0 441.1G 0 lvm /home
3 H" H7 m% N" F+ m6 Z* p9 I1 J# y│ └─centos-root 253:0 0 50G 0 lvm /
# C5 G n$ o0 h7 i4 H! o- Q└─vda1 252:1 0 1G 0 part /boot8 F6 N7 w1 K$ I" F' @2 o% G
8 o. t* I) ^, @( ]
#写入小说文档到挂载的目录
6 l4 I! X- I: ~$ Y+ E6 S; e# S9 \[root@ceph101 ~]# mv bcsj.txt test_rbd/% @# W: i, z& X3 u& U0 v4 x8 j
[root@ceph101 ~]# md5sum test_rbd/bcsj.txt
7 [! c) G6 |- N, `$ D0d615ccd0e1c55f62002134f5cac81cc test_rbd/bcsj.txt0 `5 {! k+ l5 @) ~& E9 ]8 G: R
[root@ceph101 ~]# df -lh, x5 a+ h" f* q9 N( [. F( |
文件系统 容量 已用 可用 已用% 挂载点
3 h! y1 y* }8 C/dev/rbd0 9.7G 15M 9.2G 1% /root/test_rbd2 w* m% J: i/ Q- @
: r. T& a/ ]1 s4 \[root@ceph101 ~]# ceph df1 V( Z3 X7 X: z2 e. \1 @9 H
--- RAW STORAGE ---
) A9 A- r; F8 f7 D M% R# b1 ~CLASS SIZE AVAIL USED RAW USED %RAW USED# S# D9 L7 ]+ d! N8 {9 q
hdd 5.9 TiB 5.8 TiB 1.5 GiB 14 GiB 0.23: r$ B/ O1 y1 T$ S/ M, \
TOTAL 5.9 TiB 5.8 TiB 1.5 GiB 14 GiB 0.232 b9 F" M" k# }
& t* J! ^: p- ]3 O% t( L
--- POOLS ---
0 F' j7 t4 ]9 @9 U- IPOOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
5 j( x/ G# X3 e& E5 m7 wdevice_health_metrics 1 1 0 B 0 0 B 0 1.9 TiB
, D1 R* J% J% Q& Z) l4 i K' \.rgw.root 24 32 22 KiB 36 6.6 MiB 0 1.9 TiB
; r" `; F4 v R! _# I \7 f6 mzone_01.rgw.log 31 32 26 KiB 965 55 MiB 0 1.9 TiB
- X: Z. o$ K% \' Ezone_01.rgw.control 32 32 0 B 8 0 B 0 1.9 TiB" r6 w, i2 M, C
zone_01.rgw.meta 33 8 5.5 KiB 16 2.6 MiB 0 1.9 TiB
+ j; q2 _2 K, i% G: G% @$ Czone_01.rgw.buckets.index 34 8 672 KiB 55 2.0 MiB 0 1.9 TiB
: h/ p$ Q$ {) @" h- [9 ~zone_01.rgw.buckets.data 35 32 2.2 MiB 12 7.9 MiB 0 1.9 TiB
- U# O% U' h& G' z6 ~) r( vzone_01.rgw.otp 36 32 0 B 0 0 B 0 1.9 TiB
; @9 l2 O6 ^+ i. t5 P' y& [cp_pool 42 32 2.2 MiB 2 7.0 MiB 0 1.9 TiB) b. }# M8 ^8 \1 H6 T
test_rbd 43 32 148 MiB 57 446 MiB 0 1.9 TiB) `" [) ~ v& ?# F* K& {' |
; X& {, Y Z+ [4 g, a( f
' o; J3 V t# I# k" S; n对象网关: C. O! ]. y, H6 O
bash 体验AI代码助手 代码解读复制代码: X1 {" h3 |# u& S- K
0 t: E) P% R" x
yum install ceph-radosgw -y \ y! L; v; D* o) s1 F) g+ ?
9 e8 c( G: r8 Z, V. V& s
[ceph: root@ceph1 ceph]# radosgw-admin user create --uid='s3_admin' --display-name='s3_rgw_admin' --access-key='s3_rgw_admin_access_key' --secret-key='s3_rgw_admin_secret_key'0 E) G$ E. U/ h z0 L
{
7 I1 Z J' I2 q: d A( o' v "user_id": "s3_admin",8 X* Q) Y+ A* }5 z2 S! N
"display_name": "s3_rgw_admin",( \8 c' e( R3 I# ]
"email": "",
( y( }/ e" h) `" H9 U5 } "suspended": 0,5 e; w3 s9 N6 L2 I2 g
"max_buckets": 1000,
7 s" o" l% U: x, {% a "subusers": [],9 A5 P; s1 e! q9 \! N: c
"keys": [- W6 |% o( Z& A" z( J: u6 Y
{
% h% f5 N0 _9 F) \ "user": "s3_admin",( F6 Z# B! J6 F5 N8 S
"access_key": "s3_rgw_admin_access_key",6 u3 }0 q \& ^: O/ H; N5 D& R
"secret_key": "s3_rgw_admin_secret_key"
2 k( _6 z x/ f6 y }* ?$ v' O5 W# U1 s I
],
- T$ S+ `0 K) |. V; K; F9 _ "swift_keys": [],9 W% H, p7 N# U) {
"caps": [],
" d, u, @* o J A7 [/ F "op_mask": "read, write, delete",
. T9 r* W" k7 H "default_placement": "",
) F! p' W* r) l }( ` "default_storage_class": "",
H" ?* j" Z; {5 k: L "placement_tags": [],
9 Y" a% Q7 A: F4 G' t "bucket_quota": {
% y! {+ }4 h5 _' A1 o "enabled": false,/ @3 z) q/ A9 Q5 ]" o
"check_on_raw": false,
# `' U" ]! h' b M1 [) D8 j& G "max_size": -1,6 U) a+ T/ ^+ M6 J
"max_size_kb": 0,
% \6 R5 K2 i( v4 {# {- | "max_objects": -18 \) p! A/ u0 m, d
},
E7 E% X% P8 `! F7 e5 f; b "user_quota": {
8 x1 Y3 \: m9 _7 ~/ D! f& |* E "enabled": false,
4 |: G4 g3 T9 ?' f# ^ "check_on_raw": false,( O1 C& J1 [: V; ?$ D
"max_size": -1,/ D* g3 i$ X, `9 E
"max_size_kb": 0,! Y# W+ {' d9 r
"max_objects": -1
! H1 l0 [* _! U$ d5 l' {5 `9 O+ a* N. U },
3 o7 U& r, b; w6 g( ^& ^ "temp_url_keys": [],
) C9 j' g; A" F# c8 | "type": "rgw",9 f: E. D" }" t- k* j- h; p2 i2 S! M
"mfa_ids": []7 [; a' D7 i1 m& A
}
% ]. z! S( `. W6 y- Z1 {# W
9 s z1 R3 L" G[ceph: root@ceph1 ceph]#
5 L O2 Z9 I( J6 C[ceph: root@ceph1 ceph]# radosgw-admin user info --uid='s3_admin'
4 _& J2 }% t0 G' ^9 b{$ K; J! a% L. J5 r" I! G" N
"user_id": "s3_admin",
, j4 M* o3 x( S! i3 B7 ^" J "display_name": "s3_rgw_admin",3 u9 d* ~/ ~, X; ^; e* u; M/ {$ F
"email": "",5 [- m) w8 K Z! g
"suspended": 0,* H# h, R O' a u2 Y- G
"max_buckets": 1000,
( A) X) S, S. Y7 h "subusers": [],: W. k# J8 A2 O/ m
"keys": [8 m4 d1 J& Z& n
{; ?* C- ?3 v+ d) T
"user": "s3_admin"," N! Y) ?7 E P2 `# T8 v8 k
"access_key": "s3_rgw_admin_access_key",
; n. ^3 a) [$ p. H "secret_key": "s3_rgw_admin_secret_key"
; b/ S" e4 {: J+ S }
+ U) H8 q2 O) N( E ],1 t9 j: ~& n; x5 Q9 e
"swift_keys": [],# S! z2 f$ H4 M
"caps": [],
, c7 V' s* S, E, q1 c1 O "op_mask": "read, write, delete",
d8 y9 c. I, P0 R7 k! E2 | "default_placement": "",/ |# S! C" ]+ T! W' t3 Y
"default_storage_class": "",
% D1 v; \8 D2 ? "placement_tags": [],! o8 s. k. O" C5 d; h; n0 x
"bucket_quota": {9 {+ V+ i2 l, X0 s
"enabled": false,
! t" d: O% D$ i, b "check_on_raw": false,
" `3 u( r! W9 Q$ B "max_size": -1,% c% ]3 a) \) R
"max_size_kb": 0,' A1 T; s- L* ]7 h5 m) J: ]
"max_objects": -1# Y/ [0 q( L0 N' D( b e
},
: a8 t: C4 u2 ~1 I8 d( T p7 [ "user_quota": {
* ?4 I0 z4 v7 y* N "enabled": false,
0 q: p/ A+ }% ^% e5 [% V "check_on_raw": false,4 p( r0 b4 R5 j3 Y
"max_size": -1,1 y& P3 }8 U3 V5 W
"max_size_kb": 0,5 ?7 j) z8 ~# Z# `
"max_objects": -1
- ^/ P: f1 A6 I" M },: f) k0 S* `9 ?5 ~/ j" }# A
"temp_url_keys": [],
v8 o# H1 }, @) o+ O- _! K: `# K "type": "rgw",/ t# y% _: y/ w: B0 }: y
"mfa_ids": []
6 {: o k" F; D: e$ y0 L}* m# n& z/ ^. B! K% s1 _
% U3 r* ]: g3 Mceph tell mon.* injectargs --mon_max_pg_per_osd=100 6 p8 H/ B8 ] E; \( X$ U5 J! p
* @: z8 s: ]% F! F+ x8 ?3 ~4 o9 j
#----------------------------------------------------------------------------! X% E/ D* D+ T8 d, ]9 O
radosgw-admin user create --uid=rgw_admin --display-name=rgw_admin --system
' o7 ~8 e2 K! f! E, Vradosgw-admin user info --uid rgw_admin --system; c: z: e; }. [7 S+ R6 z
* t7 C7 I+ h" T
q2 X4 {( X2 u. R: C5 x4 Kradosgw-admin realm create --rgw-realm=realm_1 --default! u- g' m3 C5 A6 i0 P$ u
radosgw-admin zonegroup create \
6 u" K) l3 Y7 N+ } --rgw-realm=realm_1 \8 Z5 t' T6 i! |
--rgw-zonegroup=zone_group_1 \; \" X6 o6 |% O7 Z: t1 @
--endpoints http://ceph101:80 \
# ]/ w8 [/ v' I# V --master --default! d' H. D0 i- D% n4 M3 w# U
: _- h( _" K1 j6 A4 J
2 i3 [/ `" [' P& x( B4 F' I radosgw-admin zone modify \6 W5 o+ y5 P8 T$ ~# m u1 j, ~. @
--rgw-realm=realm_1 \; f$ t4 Y2 d* q$ e
--rgw-zonegroup=zone_group_1 \
$ B0 b) Q {+ G1 z6 U* X" F --rgw-zone=zone_01 \+ Y Y4 o% R9 K9 C, F
--endpoints http://ceph101:80 \; b T+ J7 t7 ] [* s5 H
--access-key=IAWL6PLNFMNM0SLQNWQ0 \
) y" G- Y! R( i; Y9 j, q* A9 i! |2 A --secret=pZTNQ8HThJVXOHBnx5VCP1qJgPGfT9LTMpmwjhAo \
9 `# U) y2 z$ Q( Q! \0 I --master --default % H q% H5 C: t% x3 x. y5 h$ h
$ A) k& z' V" q% F8 {4 A9 ^2 X radosgw-admin period update --commit
8 `% n& I; `9 g# O8 b radosgw-admin period update --rgw-realm=realm_1 B; e6 D2 c1 D8 v$ U; D& M; m3 {# D
ceph orch apply rgw realm_1 zone_01 --placement="1 ceph101"
: \; C h- U/ ]! r& |& |, l" Z4 q
0 x' ]; ?. W* a) e' k2 r& P0 p; Y# G上述命令,一定要在集群健康的情况下运行,否则会出现rgw 的docker进程启动不了的情况,正常情况下,可以看到如下进程:/ Q1 {, C0 H+ L0 O$ Y) V
bash 体验AI代码助手 代码解读复制代码
4 l6 u8 k( S5 ^[root@ceph101 ~]# ceph orch ls# D5 T9 N* E4 Q0 o
NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID
4 T7 M8 p- A! q; K7 ^" \9 g6 z( Valertmanager 1/1 4m ago 2d count:1 quay.io/prometheus/alertmanager:v0.20.0 0881eb8f169f( B' G9 w2 `& T& t
crash 6/6 4m ago 2d * quay.io/ceph/ceph:v15 3edede73a7c4
" b a0 k+ ^/ _; W) lgrafana 1/1 4m ago 2d count:1 quay.io/ceph/ceph-grafana:6.7.4 557c83e11646
5 k+ w3 Y6 o: C+ q2 Emgr 2/2 4m ago 2d count:2 quay.io/ceph/ceph:v15 3edede73a7c42 j- j3 c/ H% y1 F! m8 `! c
mon 5/5 4m ago 2d ceph101;ceph102;ceph103;ceph104;ceph105 quay.io/ceph/ceph:v15 3edede73a7c4
8 }! H Z6 {& F4 ?% inode-exporter 6/6 4m ago 2d * quay.io/prometheus/node-exporter:v0.18.1 e5a616e4b9cf: g9 j m0 |/ f% @7 o/ b
osd.None 12/0 4m ago - <unmanaged> quay.io/ceph/ceph:v15 3edede73a7c4
% E& K/ N9 Z8 c3 j$ K% a7 iprometheus 1/1 4m ago 2d count:1 quay.io/prometheus/prometheus:v2.18.1 de242295e225
# l" I" d5 e& C* @. m5 g3 C8 i7 brgw.realm_1.zone_01 1/1 4m ago 75m ceph101;count:1 quay.io/ceph/ceph:v15 3edede73a7c4+ W4 c0 A' m+ j
2 x" I4 l- u& g- I: W" P# V6 W/ i6 H5 @6 x2 b( k: `9 ^' \
如果rgw进程无法启动的情况,可能是集群不健康,查看log:4 u3 x. S/ I9 ~: d- I4 U
ceph log last cephadm
9 W5 v- J ?8 z+ d) t$ f/var/log/ceph/cephadm.log
+ k) f" _7 c+ cbash 体验AI代码助手 代码解读复制代码[root@ceph101 system]# ceph log last cephadm, F) k2 q, r& ]# _4 t6 N* i
2022-04-27T07:57:57.347323+0000 mgr.ceph101.qhgzmi (mgr.14164) 95889 : cephadm [ERR] Failed to apply rgw.realm_1.zone_group_1acementSpec(hostname='ceph101', network='', name=''), HostPlacementSpec(hostname='ceph102', network='', name=''), HostPlaceme': 'rgw', 'service_id': 'realm_1.zone_group_1', 'unmanaged': False, 'preview_only': False, 'rgw_realm': 'realm_1', 'rgw_zone' 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): Health not ok, will try again when healt% C2 H* p$ j ]! k& } C2 F$ a
Traceback (most recent call last):, M) H" o+ q5 _. X8 H3 C
File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in _apply_all_services
& @5 t/ N$ p# g Q6 r if self._apply_service(spec):
. b, c& ]) J D File "/usr/share/ceph/mgr/cephadm/serve.py", line 511, in _apply_service9 R# @6 X v1 |$ w9 L# ?! @
rgw_config_func(cast(RGWSpec, spec), daemon_id)
- d1 E' S3 A0 n2 ]/ g File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 539, in config
: e' P, ~* q: R self.create_realm_zonegroup_zone(spec, rgw_id)
. _" {8 D! M% O File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 617, in create_realm_zonegroup_zone4 t- F/ R; d' r4 ^! G7 d/ e" C
raise OrchestratorError('Health not ok, will try again when health ok')
3 [0 f) t0 ~6 w2 x! E' C, ?orchestrator._interface.OrchestratorError: Health not ok, will try again when health ok
( `! H; F' |- L; u2022-04-27T07:57:57.353366+0000 mgr.ceph101.qhgzmi (mgr.14164) 95890 : cephadm [ERR] Failed to apply rgw.realm_1.zone_01 specntSpec(hostname='ceph101', network='', name=''), HostPlacementSpec(hostname='ceph102', network='', name=''), HostPlacementSpegw', 'service_id': 'realm_1.zone_01', 'unmanaged': False, 'preview_only': False, 'rgw_realm': 'realm_1', 'rgw_zone': 'zone_01ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): Health not ok, will try again when health ok
, d, C) u8 g, k) X* z0 F+ eTraceback (most recent call last):! e. L2 q# K2 i2 Y. }
File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in _apply_all_services
& ~; j0 ~) U" D if self._apply_service(spec):
2 A8 U& l) V1 v ]4 e8 O File "/usr/share/ceph/mgr/cephadm/serve.py", line 511, in _apply_service
/ {2 ?' L7 }, I- Y; i8 a8 U rgw_config_func(cast(RGWSpec, spec), daemon_id)- ~$ K* r) R/ N; d
File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 539, in config% v, _$ }1 }) S/ u$ q+ F' V
self.create_realm_zonegroup_zone(spec, rgw_id)' [% i. R' I/ v
File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 617, in create_realm_zonegroup_zone$ L- T$ e" k4 T! ] Y
raise OrchestratorError('Health not ok, will try again when health ok')
6 `9 m. v! t5 O$ w. B- I6 Lorchestrator._interface.OrchestratorError: Health not ok, will try again when health ok( R3 c6 j* d9 s- ~6 l+ o+ R
2022-04-27T08:07:22.102133+0000 mgr.ceph101.qhgzmi (mgr.14164) 96175 : cephadm [INF] refreshing ceph104 facts
) p5 ?/ {$ j+ \2022-04-27T08:07:22.103197+0000 mgr.ceph101.qhgzmi (mgr.14164) 96176 : cephadm [INF] refreshing ceph103 facts
, v* M/ P7 O. _5 W. d' ~2022-04-27T08:07:22.105047+0000 mgr.ceph101.qhgzmi (mgr.14164) 96177 : cephadm [INF] refreshing ceph106 facts- Y6 K. t+ K" P" h; f- {
2022-04-27T08:07:22.105643+0000 mgr.ceph101.qhgzmi (mgr.14164) 96178 : cephadm [INF] refreshing ceph105 facts
3 b( _9 Z1 r" l5 v. l2022-04-27T08:07:22.106985+0000 mgr.ceph101.qhgzmi (mgr.14164) 96179 : cephadm [INF] refreshing ceph102 facts- P& @; G& f- _* Q3 b# }( i
2022-04-27T08:07:22.910395+0000 mgr.ceph101.qhgzmi (mgr.14164) 96181 : cephadm [INF] refreshing ceph101 facts, k' ?* N* \5 O7 |! w, `
2022-04-27T08:07:23.599992+0000 mgr.ceph101.qhgzmi (mgr.14164) 96182 : cephadm [ERR] Failed to apply rgw.realm_1.zone_group_1acementSpec(hostname='ceph101', network='', name=''), HostPlacementSpec(hostname='ceph102', network='', name=''), HostPlaceme': 'rgw', 'service_id': 'realm_1.zone_group_1', 'unmanaged': False, 'preview_only': False, 'rgw_realm': 'realm_1', 'rgw_zone' 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): Health not ok, will try again when healt( g) A% `# d/ K2 ]
Traceback (most recent call last):
. e- ]) n6 d8 H; d9 v5 _. ` File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in _apply_all_services+ K% J ~: S6 d. O, x- Q
if self._apply_service(spec):% x# F' p J4 p L% [ y
File "/usr/share/ceph/mgr/cephadm/serve.py", line 511, in _apply_service+ K2 T( E9 Y" |) [0 x; o; X$ z
rgw_config_func(cast(RGWSpec, spec), daemon_id)
5 u; T) L% b: z6 `- D) h File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 539, in config0 d( j8 G8 E P
self.create_realm_zonegroup_zone(spec, rgw_id)6 m( |9 t4 N; l( C; T3 |3 G
File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 617, in create_realm_zonegroup_zone4 J6 m4 U- F* \0 c6 g
raise OrchestratorError('Health not ok, will try again when health ok')
. _) F" c4 x5 Z! u$ m3 n+ yorchestrator._interface.OrchestratorError: Health not ok, will try again when health ok
& h& a4 r# h( C1 U2022-04-27T08:07:23.615964+0000 mgr.ceph101.qhgzmi (mgr.14164) 96183 : cephadm [ERR] Failed to apply rgw.realm_1.zone_01 specntSpec(hostname='ceph101', network='', name=''), HostPlacementSpec(hostname='ceph102', network='', name=''), HostPlacementSpegw', 'service_id': 'realm_1.zone_01', 'unmanaged': False, 'preview_only': False, 'rgw_realm': 'realm_1', 'rgw_zone': 'zone_01ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): Health not ok, will try again when health ok
3 a* j, a$ ~0 d# T2 ?& u5 u! kTraceback (most recent call last):
, {8 `- n6 x+ R4 c# ?: { File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in _apply_all_services
6 Z1 |7 S0 L5 j) { D if self._apply_service(spec):- Q j; M3 o* X5 v& }
File "/usr/share/ceph/mgr/cephadm/serve.py", line 511, in _apply_service6 p6 ` p$ V# n0 S+ ^: }8 J8 B! |$ X
rgw_config_func(cast(RGWSpec, spec), daemon_id)' T* |# h) v; F# ^, M1 Z3 V: T) e
File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 539, in config
) t5 q) `7 ^& s4 I2 ^0 { self.create_realm_zonegroup_zone(spec, rgw_id)# e1 \$ |8 V. }% _, R6 k
File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 617, in create_realm_zonegroup_zone
; W% w# B3 C( z$ G4 ~ raise OrchestratorError('Health not ok, will try again when health ok')
, S8 v: ~, A7 u9 dorchestrator._interface.OrchestratorError: Health not ok, will try again when health ok% o" @1 G* W# C; c' ?+ k" f8 Q
2022-04-27T08:07:23.784884+0000 mgr.ceph101.qhgzmi (mgr.14164) 96184 : cephadm [ERR] Failed to apply rgw.realm_1.zone_group_1acementSpec(hostname='ceph101', network='', name=''), HostPlacementSpec(hostname='ceph102', network='', name=''), HostPlaceme': 'rgw', 'service_id': 'realm_1.zone_group_1', 'unmanaged': False, 'preview_only': False, 'rgw_realm': 'realm_1', 'rgw_zone' 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): Health not ok, will try again when healt
' \; q1 z- K( X# s* mTraceback (most recent call last):) [! ~' F9 q# \: z
File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in _apply_all_services! X$ P. f/ p6 p/ x
if self._apply_service(spec):6 R; U& R) S1 {0 a0 G
File "/usr/share/ceph/mgr/cephadm/serve.py", line 511, in _apply_service
4 p+ x+ z* I' y rgw_config_func(cast(RGWSpec, spec), daemon_id)3 C& Z5 ~+ w/ w3 A) V- n1 T
File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 539, in config7 d) `& ]4 x G$ f
self.create_realm_zonegroup_zone(spec, rgw_id)
8 j. x3 d* V# P" ~0 x* T File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 617, in create_realm_zonegroup_zone5 w8 T) ^% d) O& I* o
raise OrchestratorError('Health not ok, will try again when health ok'), `$ {7 z9 e0 I, u8 `0 V
orchestrator._interface.OrchestratorError: Health not ok, will try again when health ok) S, o- d0 U0 E" F3 W$ m+ P- l4 k
2022-04-27T08:07:23.788497+0000 mgr.ceph101.qhgzmi (mgr.14164) 96185 : cephadm [ERR] Failed to apply rgw.realm_1.zone_01 specntSpec(hostname='ceph101', network='', name=''), HostPlacementSpec(hostname='ceph102', network='', name=''), HostPlacementSpegw', 'service_id': 'realm_1.zone_01', 'unmanaged': False, 'preview_only': False, 'rgw_realm': 'realm_1', 'rgw_zone': 'zone_01ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): Health not ok, will try again when health ok; F1 b+ I9 \5 P6 R
Traceback (most recent call last):
2 Y d- W5 _. Q+ R- x+ M& P5 Q File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in _apply_all_services
! T ]3 o: N6 ]2 t9 K& q+ C. r if self._apply_service(spec):- [9 t m$ q8 S. t" A) F
File "/usr/share/ceph/mgr/cephadm/serve.py", line 511, in _apply_service
. }% F0 Y- {) w rgw_config_func(cast(RGWSpec, spec), daemon_id)
& `$ @ j7 ^/ k* C* ^( P# G3 B x File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 539, in config) m. s3 ^8 C( Z; k h9 c
self.create_realm_zonegroup_zone(spec, rgw_id)
# F1 F) i% T/ i7 u& N$ N File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 617, in create_realm_zonegroup_zone1 f2 D8 d% R q$ b; w! [1 _9 y
raise OrchestratorError('Health not ok, will try again when health ok'). c6 d8 b& t4 q3 t
orchestrator._interface.OrchestratorError: Health not ok, will try again when health ok
9 t- `- n: C1 F5 t; o Z2022-04-27T08:16:33.025623+0000 mgr.ceph101.qhgzmi (mgr.14164) 96463 : cephadm [INF] Saving service rgw.realm_1.zone_01 spec/ Y7 R B% V- ]1 a9 J
2022-04-27T08:16:33.240231+0000 mgr.ceph101.qhgzmi (mgr.14164) 96464 : cephadm [INF] refreshing ceph101 facts4 o2 k3 U' P9 S8 w8 L) m
2022-04-27T08:16:33.248641+0000 mgr.ceph101.qhgzmi (mgr.14164) 96465 : cephadm [INF] refreshing ceph102 facts5 E# R9 e6 ]/ q3 E+ `# U
2022-04-27T08:16:33.250945+0000 mgr.ceph101.qhgzmi (mgr.14164) 96466 : cephadm [INF] refreshing ceph103 facts& ?5 d# O8 b$ n. g2 K! y. x; ^6 A
2022-04-27T08:16:33.252787+0000 mgr.ceph101.qhgzmi (mgr.14164) 96467 : cephadm [INF] refreshing ceph104 facts
+ F& |& D7 ]) [2022-04-27T08:16:33.254250+0000 mgr.ceph101.qhgzmi (mgr.14164) 96468 : cephadm [INF] refreshing ceph105 facts- `0 Z' n3 s$ d( i: Y. m
2022-04-27T08:16:33.256573+0000 mgr.ceph101.qhgzmi (mgr.14164) 96469 : cephadm [INF] refreshing ceph106 facts1 p( `( @! X' `) y
2022-04-27T08:16:34.288319+0000 mgr.ceph101.qhgzmi (mgr.14164) 96470 : cephadm [ERR] Failed to apply rgw.realm_1.zone_group_1acementSpec(hostname='ceph101', network='', name=''), HostPlacementSpec(hostname='ceph102', network='', name=''), HostPlaceme': 'rgw', 'service_id': 'realm_1.zone_group_1', 'unmanaged': False, 'preview_only': False, 'rgw_realm': 'realm_1', 'rgw_zone' 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): Health not ok, will try again when healt! ]' G- \$ }6 Q! Y5 w# n
Traceback (most recent call last):* W* N3 E' ^! S, Q% s& Z' h* r8 z
File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in _apply_all_services# t/ |0 N+ s# ?, l
if self._apply_service(spec):/ y5 B3 ^( Q: T) V
File "/usr/share/ceph/mgr/cephadm/serve.py", line 511, in _apply_service7 w8 z2 o# V* Y2 O& `
rgw_config_func(cast(RGWSpec, spec), daemon_id)
8 g E- `3 @ i, k3 t$ ? File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 539, in config `* z+ w4 w9 U0 Q0 A
self.create_realm_zonegroup_zone(spec, rgw_id)5 x$ A+ M% N* b' C8 {: ?
File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 617, in create_realm_zonegroup_zone$ L% N4 l$ L8 T7 L0 s/ b
raise OrchestratorError('Health not ok, will try again when health ok')
" p6 E6 @5 X# j0 q6 u7 D- p3 @/ gorchestrator._interface.OrchestratorError: Health not ok, will try again when health ok0 X/ m: I! [7 H0 L X- Y. O- K
2022-04-27T08:16:34.292193+0000 mgr.ceph101.qhgzmi (mgr.14164) 96471 : cephadm [ERR] Failed to apply rgw.realm_1.zone_01 specntSpec(hostname='ceph101', network='', name='')]), 'service_type': 'rgw', 'service_id': 'realm_1.zone_01', 'unmanaged': Falsene_01', 'subcluster': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 's
% s/ ~# I* i. ^/ KTraceback (most recent call last):
9 M. t3 O* V$ M" c( ?( V File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in _apply_all_services' W) A# a E4 a4 t* r/ c
if self._apply_service(spec):
" d2 ]6 r$ v2 [: e9 i File "/usr/share/ceph/mgr/cephadm/serve.py", line 511, in _apply_service4 O% {2 y1 L# T7 X7 S, O
rgw_config_func(cast(RGWSpec, spec), daemon_id)! \$ o' Z& t. p, }3 E4 s y1 c$ Y
File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 539, in config
& a; j1 Z' Q7 y3 z) c self.create_realm_zonegroup_zone(spec, rgw_id)
1 \8 w8 O; L" M/ k2 B, K) ^7 v File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 617, in create_realm_zonegroup_zone+ x. O: ~! H- @
raise OrchestratorError('Health not ok, will try again when health ok')0 b# l ]2 ?7 b7 Y
orchestrator._interface.OrchestratorError: Health not ok, will try again when health ok4 B: {! T+ a0 x
5 `" A2 w; y6 Z3 [, d8 n
+ f; d! c" I& f& O# v
: C8 m) k8 f& ]9 m/ |) E' a: o, G进入到docker中
" F; f |' e; _) m) p* M' [, s) wbash 体验AI代码助手 代码解读复制代码docker exec -it 736e1816f245 /bin/sh8 N1 |% T- a$ f
sh-4.4# cd /usr/share/ceph/mgr/cephadm/services/
, x9 i6 v" k+ ^sh-4.4# ls
2 u/ r9 y( a( `) j4 q__init__.py cephadmservice.py container.py iscsi.py monitoring.py nfs.py osd.py& z1 g9 E* ~ q' F: a
sh-4.4# vim cephadmservice.py
. q# D( j; Y$ gsh: vim: command not found
. Z% `9 r& u. J+ J& Bsh-4.4# vi cephadmservice.py9 Z2 `- K G) T, h7 M4 r; I3 L: n
##可以看到是因为python代码查看集群的状态为 Health not ok,所以不能向下进行了。
! i# O" s+ h6 S#进一步,将集群调整为健康状态,再次尝试,可以正常
3 H1 p# K# {( W% t+ |3 i
2 z, r- q9 ^" t" s4 b, s访问s3对象服务,windows下安装S3 Browser
8 q0 j3 K, j$ T; {% \5 c
: ~" |" f. a& c创建用户后,不知道为什么用户被自动删除掉了,重新建立了一下用户,再次绑定zone之后,就可以使用s3 Broswer正常访问了。
) c( L/ h+ c5 ?2 |7 C其他常用命令:
h; r- k, r/ B; p2 B: qbash 体验AI代码助手 代码解读复制代码 radosgw-admin realm delete --rgw-realm=realm_1+ S& s4 L# j, T& P Z
radosgw-admin zonegroup delete --rgw-zonegroup=zone_group_1
# k; e, u/ V# R7 z! E* I radosgw-admin zone delete --rgw-zone=zone_1
; Q* U1 {5 ]$ w- m
F" G( I' [& ]! g; }! }radosgw-admin realm list
0 {" G6 ~$ ^/ F& Rradosgw-admin zonegroup list% f8 \0 Z' s( G: u2 D5 |
radosgw-admin zone list1 s4 o% T. \" P1 q8 }2 s
radosgw-admin zone list --rgw-zonegroup default, }9 m+ d' K! t, }; G
radosgw-admin user info --uid rgw_admin --system
/ d# C3 x1 G3 `# s* a( ^9 l6 s2 P2 X* w
s3cmd mb s3://sec --region=zone_group_1
6 z/ D8 q5 \! M9 L; e4 b z& }
0 H3 P( l2 t, V- I3 i
: y! ?; j5 B' |6 U' M2 S) U1 V/ t配置dashboard:
" Q* n! M5 ~ ~bash 体验AI代码助手 代码解读复制代码 1752 2022-05-09 11:21:28 radosgw-admin user list$ o% B9 P' o; w+ L) [8 T( ?. c9 a
1755 2022-05-09 11:22:06 radosgw-admin user info --uid rgw_admin
% o$ J8 Z. i# L3 S2 i 1757 2022-05-09 11:34:44 vim access.key, C1 A4 ^! y* ^( M
1758 2022-05-09 11:35:00 vim secret.key; W* ]; Y p3 n+ X5 E) W( i
1759 2022-05-09 11:35:21 ceph dashboard set-rgw-api-access-key -i access.key
9 S% s3 Y' `# A& k 1760 2022-05-09 11:35:34 ceph dashboard set-rgw-api-secret-key -i secret.key( ]. x* l, z4 l6 t
1761 2022-05-09 11:36:49 history
" y# A9 p1 W# B" F+ d/ S0 H% B4 x2 m8 z- E4 V' K4 Z/ P9 l* }! t
在前端可以看到8 J. g) h& i, ^$ D! _0 |3 G. q
0 `5 x4 z! S* h6 p- Q! {搭建主-主备份
3 R/ W# S% b2 Q! S# O9 Sbash 体验AI代码助手 代码解读复制代码, ^/ c! ~: M+ r a" f
4 n: x' B/ j$ Nradosgw-admin realm pull --url=http://12.70.10.181:80 --access-key=IAWL6PLNFMNM0SLQNWQ0 --secret=pZTNQ8HThJVXOHBnx5VCP1qJgPGfT9LTMpmwjhAo
Y+ L- {7 G. L( v. m ]
& p: F i( G4 }8 Kradosgw-admin zone modify --rgw-zonegroup=zone_group_1 \
; I7 ]% V" q; F5 v. @0 u# Z --rgw-zone=zone_02 --url=http://ceph101:80 \
* T0 b- b8 |1 V3 K --access-key=IAWL6PLNFMNM0SLQNWQ0 --secret=pZTNQ8HThJVXOHBnx5VCP1qJgPGfT9LTMpmwjhAo \
1 { S" \) {) e) H- O4 W --endpoints=http://ceph1:80
/ [3 ?6 `% p* l! \. ^1 V7 l' Qradosgw-admin period update --commit5 r/ g: P5 G4 e, q6 i& j
ceph orch apply rgw realm_1 zone_02 --placement="1 ceph1"
( Z, _' e; M! P! g" N' a, F: V& h! A* D8 f& t
查看应用的配置
/ C4 s n0 | lbash 体验AI代码助手 代码解读复制代码[root@ceph5 f88b0b1a-c467-11ec-a2b8-525400299ff7]# ceph config dump5 g+ L' A# ^6 j0 c0 N! x( w
4 |3 m" V" j/ q" D$ a
5 l" Z: l) w( j5 Z[root@ceph5 f88b0b1a-c467-11ec-a2b8-525400299ff7]# pwd; }1 ~1 P' |$ P+ K% X6 L
/var/lib/ceph/f88b0b1a-c467-11ec-a2b8-525400299ff79 i9 g; o, V: `- b, B. d
[root@ceph5 f88b0b1a-c467-11ec-a2b8-525400299ff7]# ll
7 N; b" M. K8 e1 G- v; m$ N总用量 0
6 |) O! d( K: t4 l. Y8 vdrwx------ 4 ceph ceph 92 4月 27 18:52 crash
8 q, }# k: D4 Y0 {0 m0 X( ~6 cdrwx------ 2 ceph ceph 133 4月 25 16:07 crash.ceph5: W( T; s/ J/ \& c0 }; t3 M5 y9 g
drwx------ 3 ceph ceph 190 4月 27 17:04 mon.ceph55 v% [9 y( u' }/ }8 q
drwx------ 2 65534 65534 104 4月 25 16:07 node-exporter.ceph5& }" R5 ?% \7 s. J
drwx------ 2 ceph ceph 241 4月 28 19:42 osd.42 O( n( g$ a, ^8 ~' @: T! g5 h
drwx------ 4 root root 88 4月 28 10:59 removed+ T b$ G) C q. k
drwx------ 2 ceph ceph 133 4月 28 19:57 rgw.realm_1.zone_03.ceph5.tapydb
, n! B Q/ H5 k6 U3 ?3 s% p' v" U; U" y8 |1 c: f/ k3 `8 ~, i |# W: }) d
! p0 h2 `7 w q" b9 L4 j2 c9 f5 O! }
[root@ceph5 f88b0b1a-c467-11ec-a2b8-525400299ff7]# ceph config show rgw.realm_1.zone_03.ceph5.tapydb3 y7 g4 |" L s' x) y+ I8 V* b! s9 y" U
NAME VALUE SOURCE OVERRIDES IGNORES
( k) v3 F+ W7 Y; v' n+ U+ Padmin_socket $run_dir/$cluster-$name.$pid.$cctid.asok default
2 J% x7 b5 f/ E8 r: Z5 U( tcontainer_image quay.io/ceph/ceph:v15 mon/ U6 u- ~, o( c, f: j, f2 _& P+ F
daemonize false override$ F2 ] W8 D) h4 K7 n* Y" c
debug_rgw 1/5 default
0 r2 \2 C3 U- x7 Bkeyring $rgw_data/keyring default9 w% D7 |1 e4 O ~1 o! b0 y# {
log_stderr_prefix debug default7 i/ @5 k8 s" B( {: E1 X4 a
log_to_file false default/ h7 Y ?+ s4 p, N3 ?
log_to_stderr true default$ V$ s4 D! k$ F4 j
mon_host [v2:12.70.10.161:3300/0,v1:12.70.10.161:6789/0] [v2:12.70.10.162:3300/0,v1:12.70.10.162:6789/0] [v2:12.70.10.163:3300/0,v1:12.70.10.163:6789/0] [v2:12.70.10.164:3300/0,v1:12.70.10.164:6789/0] [v2:12.70.10.165:3300/0,v1:12.70.10.165:6789/0] file6 u7 r" ~- l2 ^" _! E
no_config_file false override
( g8 N+ V7 U; Xobjecter_inflight_ops 24576 default
6 h' J- t# O, n* m7 e9 Orbd_default_features 61 default9 z8 {; i8 q2 q. [9 O$ y# Y
rgw_frontends beast port=80 mon& \5 V% z; _# ~$ V8 Z
rgw_realm realm_1 mon
8 |# }* B: ~9 t# c, Q7 w+ `rgw_zone zone_03 mon" P3 S; O" K( T( G- d- ^
setgroup ceph cmdline/ x3 d+ {3 Z6 y0 l
setuser ceph cmdline
( Z( P; j* ?" i6 `
0 j* ^5 j8 p4 E5 I对于想修改rgw启动端口的,可以修改配置文件
: s+ V4 R9 d; d# z' d d$ C, f- Tbash 体验AI代码助手 代码解读复制代码##在一个域realm和一个zonegroup下面,可以创建多个zone,每个zone上面可以创建一个rgw,所以再次创建一个rgw网关
8 I8 T* k/ p: z! c- cradosgw-admin realm pull --url=http://12.70.10.181:80 --access-key=IAWL6PLNFMNM0SLQNWQ0 --secret=pZTNQ8HThJVXOHBnx5VCP1qJgPGfT9LTMpmwjhAo; F1 h* e) D0 t. T
6 a- G. l* }* Dradosgw-admin zone modify --rgw-zonegroup=zone_group_1 \
3 _4 ]5 }' j5 z5 i7 [5 y2 o- ^ --rgw-zone=zone_03 --url=http://ceph101:80 \# \/ ]* h R% ~# j7 o( v
--access-key=IAWL6PLNFMNM0SLQNWQ0 --secret=pZTNQ8HThJVXOHBnx5VCP1qJgPGfT9LTMpmwjhAo \
: E% x4 o, R* h% }% I --endpoints=http://ceph5:8088* y; q4 z$ R* l) P+ z
radosgw-admin period update --commit* H$ W' ^% D3 q4 ?1 w) a2 ?
ceph orch apply rgw realm_1 zone_03 --placement="1 ceph1"( P" H; x& h: J! j* u
6 W- v, F2 q. A0 z
/ B# } b u, N0 R8 ~/ H6 V$ i; N+ K7 v( N[root@ceph5 rgw.realm_1.zone_03.ceph5.crocaj]# pwd' r! j+ p' {. o C6 ]) Y
/var/lib/ceph/f88b0b1a-c467-11ec-a2b8-525400299ff7/rgw.realm_1.zone_03.ceph5.crocaj
' a- X4 O" ~8 k- s/ O( E[root@ceph5 rgw.realm_1.zone_03.ceph5.crocaj]# vim! e- I6 s! T& \( L
) }) C' m/ n6 ~5 }`# minimal ceph.conf for f88b0b1a-c467-11ec-a2b8-525400299ff7
1 @; @# D8 q( Q, A; @[global]
) E$ m3 J8 W6 ~ fsid = f88b0b1a-c467-11ec-a2b8-525400299ff7
0 `" L$ }& X8 H: C mon_host = [v2:12.70.10.161:3300/0,v1:12.70.10.161:6789/0] [v2:12.70.10.162:3300/0,v1:12.70.10.162:6789/0] [v2:12.70.10.163:3300/0,v1:12.70.10.163:6789/0] [v2:12.70.10.164:3300/0,v1:12.70.10.164:6789/0] [v2:12.70.10.165:3300/0,v1:12.70.10.165:6789/0]
u7 {+ q; T k: A[client]
, \5 n' b5 ~: Y2 x$ ]' wrgw_frontends = "beast port=8088"
" S& |4 o, u2 n! X6 ^. o7 c( s- y m`; j, S& i2 m9 O3 D- i# q
#然后重启rgw服务) U& u$ f+ Y3 y L
#还可以通过执行ceph config set client.rgw.realm_1.zone_03 rgw_frontends "beast port=18888"9 s# ?) |. \+ z7 {5 ^4 M" U
#去掉config中的配置项,重启服务,也可以生效
; V& l( k4 E" y, c) `#至于精确的配置如:& ?4 m. @4 z1 U
`) T8 l3 i- F7 R' I* Q/ p
[client.rgw.realm_1.zone_03]" X) L& t) V% k( g+ N
rgw_frontends = "beast port=8888"
8 h7 O& V8 I" w* ?6 _`
6 E6 z( ]# Y" y- j#试了很多次,都不好用,官网说的不明白
, ^5 t% |: h# l. R0 Q l
9 P( B( V$ A7 k' k! [; e, Z+ w参见:; I* Q0 m2 y1 @' @2 L* R
$ \: e, G3 R; b# x指定rgw的数据池2 {; Z9 `" ~5 h# q
5 }: \# d9 U5 i+ m
移除osd/ z! v6 _ l8 N6 s3 U; B0 v* W
bash 体验AI代码助手 代码解读复制代码如果是用命令行,操作如下:
. B, f( o8 {1 D, v& O将节点状态标记为out (节点已经不再提供服务)
9 c) D9 ^( I8 s e/ v[root@ceph1 ~]# ceph osd out osd.5. P, N2 { P$ ?' ~& O' ?5 v- l
从crush中移除节点(不删除会影响到当前主机的host crush weight)
; g! Z* y1 J! c+ I4 N[root@ceph1 ~]# ceph osd crush remove osd.5
' s& {; z s4 D删除节点
3 q4 e( S7 t% m) F[root@ceph1 ~]# ceph osd rm osd.5
0 D A* b/ k1 m, I删除节点认证(不删除编号会占住)
! |/ [; M0 q, x$ p5 Y( A8 @[root@ceph1 ~]# ceph auth ls
" R8 [; ?' r! [+ o9 ?" b[root@ceph1 ~]# ceph auth del osd.5. ]/ T3 ?5 t! _& C2 b" \' N. S
[root@ceph1 ~]# ceph orch daemon stop osd.5; f' h r; h0 b- A1 u. H
Scheduled to stop osd.5 on host 'ceph6', i3 }" m$ l: }4 [- J) H% m
[root@ceph1 ~]# ceph health detail
: O* [7 `. D) c! ~HEALTH_WARN 1 failed cephadm daemon(s)
" I9 E1 W% q3 ~# }/ h4 H% R[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)
* ]9 o) S% X9 k S) @ daemon osd.5 on ceph6 is in error state$ x6 E/ ~# U* D2 u5 L
[root@ceph1 ~]# ceph orch daemon rm osd.5 --force
# q3 A- W; I' g# b" mRemoved osd.5 from host 'ceph6'
9 P7 c* B+ R; S; S9 \4 |/ P
3 ~5 ^! ^3 f2 h4 @0 v4 x9 z恢复磁盘:$ O$ C: F, |! ]4 ^, K
找到某个未加载的盘,有两种方式:
# L; O0 R Z' \4 C2 v. lbash 体验AI代码助手 代码解读复制代码[root@ceph6 osd.5_2022-05-11T02:14:55.313464Z]# ceph osd metadata
( ]1 a/ g* l: O3 b( a, j8 \' Z9 |2 w8 `2 y1 M
和
: x) D8 \1 C# Q% v* P. _8 ~bash 体验AI代码助手 代码解读复制代码[root@ceph6 osd.5_2022-05-11T02:14:55.313464Z]# pwd
, q& `. b! V @1 P3 H8 j/var/lib/ceph/f88b0b1a-c467-11ec-a2b8-525400299ff7/removed/osd.5_2022-05-11T02:14:55.313464Z
8 |# e! B- z- F: _+ y7 o[root@ceph6 osd.5_2022-05-11T02:14:55.313464Z]# ll
6 t4 Q2 L. \' m$ B: R总用量 52
; M; S2 y: H" l* s% N# j; R4 B: i! olrwxrwxrwx 1 ceph ceph 93 5月 10 12:47 block -> /dev/ceph-2e1cc736-34d6-4dac-8d7c-f78db028a9eb/osd-block-c831faa6-7cc6-4c04-9709-c33fb29a45f3' z( k" T+ q) d, B
-rw------- 1 ceph ceph 37 5月 10 12:47 ceph_fsid& I- x# n2 b0 G6 P, ?
-rw------- 1 ceph ceph 377 4月 28 19:43 config1 u( q. p' ?+ A9 w+ R, A1 [
-rw------- 1 ceph ceph 37 5月 10 12:47 fsid
/ ^6 Z4 O( N% e$ F-rw------- 1 ceph ceph 55 5月 10 12:47 keyring
5 U% X6 C$ W# |0 c-rw------- 1 ceph ceph 6 5月 10 12:47 ready7 z& [& r$ N9 g6 h# J9 M9 L9 q
-rw------- 1 ceph ceph 3 4月 25 17:04 require_osd_release S+ w9 r* A$ n4 O( ]+ A$ J
-rw------- 1 ceph ceph 10 5月 10 12:47 type7 q/ V3 A* m( u) T$ [# x4 d
-rw------- 1 ceph ceph 38 4月 28 19:43 unit.configured+ u7 i: F' b8 f9 d
-rw------- 1 ceph ceph 48 4月 25 17:04 unit.created
4 W) @& j& Y/ d! T8 c5 r1 v-rw------- 1 ceph ceph 22 4月 28 19:43 unit.image% C) F t4 d' U, t( R: D+ m
-rw------- 1 ceph ceph 931 4月 28 19:43 unit.poststop& B/ G& I) a9 u! l: P! L6 _( b
-rw------- 1 ceph ceph 2035 4月 28 19:43 unit.run5 o9 U2 U1 z5 Z6 Q
-rw------- 1 ceph ceph 2 5月 10 12:47 whoami H1 t; }# C" Q8 x
, n! G v3 l o8 M& j/ b
7 G* B/ |$ h1 ?, a0 M5 B[root@ceph6 osd.5_2022-05-11T02:14:55.313464Z]# lsblk1 O" e3 A2 X, q+ ?
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT5 N7 G" \* n9 b, g" O- e! ~
vdb 252:16 0 500G 0 disk7 i" ]2 `+ }0 V! }7 ^
└─ceph--2e1cc736--34d6--4dac--8d7c--f78db028a9eb-osd--block--c831faa6--7cc6--4c04--9709--c33fb29a45f3 253:4 0 500G 0 lvm
; f& b+ W7 E8 u- |$ asr0 11:0 1 1024M 0 rom% `9 v5 Y* b: [
vdc 252:32 0 500G 0 disk
; e- K/ A( K# `7 O2 r└─ceph--beabac1e--6d55--481b--99e4--db03786e8f78-osd--block--7f9316bd--d011--426f--be86--52b8bfad4c0b 253:3 0 500G 0 lvm! c1 t! k& S1 y7 h1 q0 p- B
vda 252:0 0 500G 0 disk$ @5 f; r" V6 z
├─vda2 252:2 0 499G 0 part
/ g, c1 B3 n/ O3 W, M: v│ ├─centos-swap 253:1 0 7.9G 0 lvm
" s( n: `* x# K7 z4 P2 b2 q. T3 [│ ├─centos-home 253:2 0 441.1G 0 lvm /home
9 N' ?6 X5 q' h6 e2 s│ └─centos-root 253:0 0 50G 0 lvm /$ S* F0 o3 E; m" n8 }
└─vda1 252:1 0 1G 0 part /boot- \) N* r: I6 q0 U
, q9 I2 T( ^2 A7 ?$ m/ t
可用确定是ceph6上面的/dev/vdb块设备9 b6 K8 S: _4 k A7 b& N' O; ?( ?: Y2 ?
bash 体验AI代码助手 代码解读复制代码#zap该磁盘,使其可重新被使用0 k* d/ ~9 s* A; R7 D
3 c0 l8 ] G- T- {! p* ~; S; M2 r
[root@ceph1 ~]# ceph orch device zap ceph6 /dev/vdb --force
4 m5 D$ d: n) w4 q# Q% [/bin/docker: stderr --> Zapping: /dev/vdb
9 { h( c+ G3 Z/ g4 V5 q* i/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/vdb bs=1M count=10 conv=fsync
, |$ _! [0 s% u+ [! q2 ~/bin/docker: stderr stderr: 10+0 records in9 {1 t1 O# T) @, L2 W, I# }
/bin/docker: stderr 10+0 records out
+ z/ k+ O7 _: Q* {9 r9 Q/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 1.1166 s, 9.4 MB/s
9 n; }, z# {- h( a% ~/bin/docker: stderr --> Zapping successful for: <Raw Device: /dev/vdb>
6 d4 z& x% ]' p9 O _! ^, S[root@ceph1 ~]# ceph orch daemon add osd ceph106:/dev/vdc& V! W1 v0 L. r' F0 b& }" |0 M
ceph orch daemon rm osd.5 --force1 O6 J1 w" M8 D4 I7 _
ceph osd out osd.50 w5 t- v& F. }
ceph osd rm osd.56 _: R: D( V9 c: u8 A; R$ A. f
ceph osd crush rm osd.5
! Q7 v& a$ V. ?0 { \0 E6 h
8 a8 I8 ~0 B6 t* F. Kpg故障处理$ f& Q) k3 N) K& M1 T
bash 体验AI代码助手 代码解读复制代码4 E2 U0 h% c" N7 I8 d0 C
; b1 n8 | l" W" m! ^0 g
[root@k8snode001 ~]# ceph health detail
9 P& N* G' l) l9 q" u/ `& PHEALTH_ERR
" C6 T& ^2 `8 W+ m6 P( S! k 1/973013 objects unfound (0.000%); . g6 X) z# t, Y; `' s
17 scrub errors; Possible data damage: 1 pg recovery_unfound, 8 pgs inconsistent, 1 pg repair; Degraded data redundancy: 1/2919039 objects degraded (0.000%), 1 pg degraded) Q# I! s% v& W' q
OBJECT_UNFOUND 1/973013 objects unfound (0.000%)" A4 t4 C: x: Q) k4 ~7 ~! u
pg 2.2b has 1 unfound objects
- t( X, J# z' M9 u# V1 Q7 qOSD_SCRUB_ERRORS 17 scrub errors
# Y. C& Q5 m& Z; O$ n* xPG_DAMAGED Possible data damage: 1 pg recovery_unfound, 8 pgs inconsistent, 1 pg repair+ R- u |/ |) o* Z* D$ f
pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound" i+ E8 F$ P- d0 a/ s/ W0 M
pg 2.44 is active+clean+inconsistent, acting [14,8,21]7 S+ R+ {! s' z: g- |; ]) [( P
pg 2.73 is active+clean+inconsistent, acting [25,14,8]; L" l4 b9 L- _0 j
pg 2.80 is active+clean+scrubbing+deep+inconsistent+repair, acting [4,8,14]9 Z. {+ d( E: |' ^" c4 L
pg 2.83 is active+clean+inconsistent, acting [14,13,6]
( C( T8 P! Z2 e0 x3 ? pg 2.ae is active+clean+inconsistent, acting [14,3,2]8 R1 c/ ~5 w. B- y& m7 x, v( K
pg 2.c4 is active+clean+inconsistent, acting [8,21,14]& B( _5 B* i( | v& p' W/ Q
pg 2.da is active+clean+inconsistent, acting [23,14,15]
" Z d% e1 o. o# Q# x6 k* t0 T pg 2.fa is active+clean+inconsistent, acting [14,23,25]
7 Z* V/ ~$ ~3 r' h( [PG_DEGRADED Degraded data redundancy: 1/2919039 objects degraded (0.000%), 1 pg degraded' {: b! W- ~/ \0 ^: S8 f
pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound; S5 _& N) s* `1 L+ R3 k
# A# h" w4 c8 _- b2.查看pg map5 w* ]9 W/ a0 }; ^) I) U
[root@k8snode001 ~]# ceph pg map 2.2b
" T( X G- z8 Posdmap e10373 pg 2.2b (2.2b) -> up [14,22,4] acting [14,22,4]" h- o5 x1 |1 s* \
从pg map可以看出,pg 2.2b分布到osd [14,22,4]上* U% E4 n' O& ?$ F: X/ w
5 g0 U& }) [' H6 d$ ]# U4 o4 d3.查看存储池状态# m0 v+ r5 B, a
[root@k8snode001 ~]# ceph osd pool stats k8s-1
4 w7 }' \/ G" G; S% m* r1 G0 I3 f2 [pool k8s-1 id 2
" X* _0 V. p. n7 v3 j 1/1955664 objects degraded (0.000%)
5 V0 c# U7 S9 a) E, R- d 1/651888 objects unfound (0.000%)
! D+ s4 }: G/ u% ` client io 271 KiB/s wr, 0 op/s rd, 52 op/s wr
. h( i$ K3 M" J- P' Q1 F
6 J: s- m$ |/ u7 P2 `: P# j[root@k8snode001 ~]# ceph osd pool ls detail|grep k8s-1, r' @1 u9 c" S4 g8 Y4 t' [
pool 2 'k8s-1' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 88 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd: ^7 ]2 A5 H2 \- N5 ?
4.尝试恢复pg 2.2b丢失的块
5 m7 |( |% h( Z2 L, K* L) Q( ?[root@k8snode001 ~]# ceph pg repair 2.2b
% i* M* C1 O; @' D n: _8 Z如果一直修复不成功,可以查看卡住PG的具体信息,主要关注recovery_state,命令如下& ~" t0 ]+ D$ p" @/ y! V( A
[root@k8snode001 ~]# ceph pg 2.2b query$ E' W1 A0 h4 l* F
如果repair修复不了;0 w: ]6 F7 F( z0 T# x9 e; _! L
两种解决方案,回退旧版或者直接删除
$ v/ V0 }5 T* X$ a+ s回退旧版: e* z& V9 J; ~& D1 D
[root@k8snode001 ~]# ceph pg 2.2b mark_unfound_lost revert# l7 _8 N$ Y( j( G& v: Z
直接删除
- d% g+ s6 J) d[root@k8snode001 ~]# ceph pg 2.2b mark_unfound_lost delete
y5 ^2 _5 b$ f' Y
/ v1 S* t6 r; m参考:记一次ceph pg unfound处理过程
{* n7 j# n* D9 O卸载% C1 N2 I1 i+ t' ?' W
卸载过程:
' T- @/ ?" N' h& Q$ abash 体验AI代码助手 代码解读复制代码重命名命令alias ceph='cephadm shell -- ceph'- l m& e% T. S/ x0 x. |$ {' Y* |1 r
#找到fsid
3 K; q2 N. v e# u[root@master1 ~]# ceph -s$ E* w U$ v. _& O
Inferring fsid 008a0d2e-b163-11ec-ba7a-5254004c51c6 O# K7 e7 o; q$ y8 a; z
Inferring config /var/lib/ceph/008a0d2e-b163-11ec-ba7a-5254004c51c6/mon.master1/config
- q# D' e5 Q2 A+ Z9 p) v7 ZUsing recent ceph image quay.io/ceph/ceph@sha256:1b0ceef23cbd6a1af6ba0cbde344ebe6bde4ae183f545c1ded9c7c684239947f9 B7 `3 ] d% Z2 V
2022-04-01T03:20:37.240+0000 7f27a2b6b700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
% @. V( S- H* F2022-04-01T03:20:37.240+0000 7f27a2b6b700 -1 AuthRegistry(0x7f279c05ec00) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
, N7 g; S: Z* u' O9 ~: j2022-04-01T03:20:37.242+0000 7f27a2b6b700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory# K; r, ^# L r. J" v6 J7 z7 w
2022-04-01T03:20:37.242+0000 7f27a2b6b700 -1 AuthRegistry(0x7f27a2b69f90) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx8 w: O& E: O3 |4 s$ Z" p0 A, d
2022-04-01T03:20:37.243+0000 7f27a0907700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
$ h# k2 M, e: l9 e# `; `5 \2022-04-01T03:20:37.243+0000 7f27a2b6b700 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication
1 |0 r% a2 W+ k+ ^$ a0 N& L[errno 13] RADOS permission denied (error connecting to the cluster)
; r5 A H) `+ H9 u- P
! B6 G F, N9 h5 @- y) q" q#使用fsid删除集群8 G; ]' ^5 m7 t+ g* }2 M! T
[root@master1 ~]# cephadm rm-cluster --fsid 008a0d2e-b163-11ec-ba7a-5254004c51c6 --force
* Q" H# z! t7 i+ D" G o/ w) Z
; c& o5 t$ [& ?7 m( M9 M( i: s9 `( D/ y3 Y. j5 B3 @, Q
分发ceph.pub( \: V; T- b4 s" Q- P
playbook -v 2.mon1.yaml -t "find_pub,show_pub,push_pub" --extra-vars "ceph_pub_path=/tmp/ceph/master1/etc/ceph/ceph.pub" + z/ x# {( T$ a
或
9 |5 K6 x5 l3 ` obash 体验AI代码助手 代码解读复制代码playbook -v 2.mon1.yaml -t "find_pub,show_pub,push_pub" -e "ceph_pub_path=/tmp/ceph/master1/etc/ceph/ceph.pub"
3 r6 i1 _( |% p' f: L2 N' {2 u: W/ A- N7 s6 U' L/ O N
使用 ansible master0 -m setup 可以看所有的变量
' z/ [0 M1 H) Q1 S, a" r h) ~参考:ansible.com.cn/docs/playbo…% |/ i' z) x0 u) [
jinja2参考:& Y8 y0 {) u# U
stackoverflow.com/questions/3…- G3 {* Y' K& d% o {7 g
bash 体验AI代码助手 代码解读复制代码 playbook -v 2.mon1.yaml6 V/ A3 C4 o* g1 B% q3 g t
playbook -v 3.push.pub.key.yaml -e "pub_key=/tmp/ceph/master1/etc/ceph/ceph.pub", O/ C" Q2 ~3 ]) j
playbook -v 4.add.host.yaml -t "weave-script" -e "mon1=master1"
; ], s& u# R' p4 l' q) k2 H# I. M, y- h' S# X7 U
本篇介绍Ceph Monitor的子命令,通过子命令的配合实现对MON的管理和配置。# T0 C- w: R4 V3 r1 c
& @8 S6 O$ R8 @5 i9 b1 z9 I- F添加(add) 在某个地址上新增一个名字为的MON服务。
& n. E/ S0 }7 T! u+ C+ E& x! f1 }示例:' k% D# z$ y: |( q
ceph mon add <IPaddr[:port]>/ G" w6 \) f) u) i& I. Z5 M
导出(dump) 显示特定版本的MON map的格式化的信息,该命令可以指定MON map的版本信息,具体示例如下,参数为epoch:
" k6 M( E, e0 d( x6 P/ iceph mon dump {<int[0-]>}
8 h* }. O3 i; M( j' {ceph mon dump 1
: C: L' f0 u2 N- W+ M获取映射(getmap) 获取特定版本的MON map信息,该命令获取的是二进制的信息,可以保持在某个文件中,具体格式如下:4 B+ @$ h. ^' }4 F
ceph mon getmap {<int[0-]>}9 R5 U7 {1 X1 o7 T* w$ h9 D
示例:
/ n5 W/ K* B) z+ H+ \3 Jceph mon getmap 2 –o filename1 z, Z5 W/ P* A9 ~4 l
移除(remove) 移除特定名称的MON服务节点。具体格式如下:
' O& N3 Y; z: P2 z% v$ G+ X, @2 T! zceph mon remove
# _/ d# @) a8 G" C% v' r示例:' \& {- l; _# S; ]- U
ceph mon remove osd3
% v$ O7 T: B% l4 |# [0 w获取状态( stat) 显示MON的摘要状态信息,具体格式如下:* L' I0 ^( Y$ K$ U3 y
ceph mon stat
+ J5 Q. u3 b0 x, `( ?报告状态(mon_status) 报告MON的状态,相对详细,具体格式如下:9 g s3 s- F3 H
ceph mon_status
) h s7 e9 R, {5 f8 U% C- MCeph命令之ceph mon(Monitor管理)* Y* w1 p {& X- T' K( I
; ?# n3 u0 R" I
( Z' {! |: h; S+ P, g
|
|