Ceph 故障排查方法总结

admin · 发表于 2022-7-27 17:39:17

Rbd 无法删除
rbd 无法删除，错误如下：

$ rbd rm nextcloud/mysql
2020-05-13 16:27:46.155 7f024bfff700 -1 librbd::image::RemoveRequest: 0x557a7af027a0 check_image_watchers: image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.

$ rbd info nextcloud/mysql
rbd image 'mysql':
      size 40 GiB in 10240 objects
      order 22 (4 MiB objects)
      id: 17e006b8b4567
      block_name_prefix: rbd_data.17e006b8b4567
      format: 2
      features: layering
      op_features:
      flags:
      create_timestamp: Tue Oct 15 10:47:34 2019
复制
查看当前 rbd 状态：

$ rbd status nextcloud/mysql
Watchers:
      watcher=10.100.21.95:0/115493307 client.67866 cookie=7
复制
发现有节点正在挂载，登入到相应机器进行查看：

$ rbd showmapped
id pool    image                                                    snap device
...
3  nextcloud  mysql                                                    - /dev/rbd3
复制
取消映射：

$ rbd unmap nextcloud/mysql
复制
重新执行删除操作即可：

$ rbd rm nextcloud/mysql
Removing image: 100% complete...done.
复制
暴力解决方案，直接对其添加黑名单，忽略挂载节点：

$ ceph osd blacklist add 10.100.21.95:0/115493307
$ rbd rm nextcloud/mysql
复制
OSD 延迟
查看是否有 osd 延迟：

$ ceph osd perf
osd commit_latency(ms) apply_latency(ms)
  2                0                0
  1                0                0
  0                0                0
复制
碎片整理
查看碎片：

$ xfs_db -c frag -r /dev/mapper/VolGroup-lv_data1
复制
整理碎片：

查看通电时长
查看磁盘通电时长：

$ smartctl -A /dev/mapper/VolGroup-lv_data1
复制
修改副本数量
修改副本数量：

$ ceph osd pool set fs_data2 min_size 1
$ ceph osd pool set fs_data2 size 2
复制
添加 / 删除 pool
添加 / 删除 pool：

$ ceph fs add_data_pool fs fs_data2
$ ceph fs rm_data_pool fs fs_data2
复制
osd 数据均衡分布
osd 数据均衡分布：

$ ceph balancer status
$ ceph balancer on
$ ceph balancer mode crush-compat
复制
mds 无法查询
mds 无法查询:

$ ceph fs status
Error EINVAL: Traceback (most recent call last):
  File "/usr/lib64/ceph/mgr/status/module.py", line 311, in handle_command
return self.handle_fs_status(cmd)
  File "/usr/lib64/ceph/mgr/status/module.py", line 177, in handle_fs_status
mds_versions[metadata.get('ceph_version', "unknown")].append(info['name'])
AttributeError: 'NoneType' object has no attribute 'get'

$ ceph mds metadata
[
{
      "name": "BJ-YZ-CEPH-94-54"
},
{
      "name": "BJ-YZ-CEPH-94-53",
      "addr": "10.100.94.53:6825/4233274463",
      "arch": "x86_64",
      "ceph_release": "mimic",
      "ceph_version": "ceph version 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable)",
      "ceph_version_short": "13.2.10",
      "cpu": "Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz",
      "distro": "centos",
      "distro_description": "CentOS Linux 7 (Core)",
      "distro_version": "7",
      "hostname": "BJ-YZ-CEPH-94-53",
      "kernel_description": "#1 SMP Sat Dec 10 18:16:05 EST 2016",
      "kernel_version": "4.4.38-1.el7.elrepo.x86_64",
      "mem_swap_kb": "67108860",
      "mem_total_kb": "131914936",
      "os": "Linux"
},
{
      "name": "BJ-YZ-CEPH-94-52",
      "addr": "10.100.94.52:6800/3956121270",
      "arch": "x86_64",
      "ceph_release": "mimic",
      "ceph_version": "ceph version 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable)",
      "ceph_version_short": "13.2.10",
      "cpu": "Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz",
      "distro": "centos",
      "distro_description": "CentOS Linux 7 (Core)",
      "distro_version": "7",
      "hostname": "BJ-YZ-CEPH-94-52",
      "kernel_description": "#1 SMP Sat Dec 10 18:16:05 EST 2016",
      "kernel_version": "4.4.38-1.el7.elrepo.x86_64",
      "mem_swap_kb": "67108860",
      "mem_total_kb": "131914936",
      "os": "Linux"
}
]
复制
重启 mds 解决。

cephfs 显示状态正常但无法写入数据
cephfs 显示正常无法使用，一般是有异常 client 导致的，首先查找 mds 是否存在链接，尝试删除链接解决：

$ ceph tell mds.BJ-YZ-CEPH-94-52 session ls
$ ceph tell mds.BJ-YZ-CEPH-94-52 session evict id=834283
复制
每一个 mds 的 id 号不通用，不能跨节点删除。

fs 增加 mds
fs 增加 mds:

$ ceph fs set fs max_mds 2
复制
mon 时区异常
mon 因为时区有部分异常导致报错如下：

$ ceph -s
  cluster:
id:    2f77b028-ed2a-4010-9b79-90fd3052afc6
health: HEALTH_WARN
         9 slow ops, oldest one blocked for 211643 sec, daemons [mon.BJ-YZ-CEPH-94-53,mon.BJ-YZ-CEPH-94-54] have slow ops.

  services:
mon: 3 daemons, quorum BJ-YZ-CEPH-94-52,BJ-YZ-CEPH-94-53,BJ-YZ-CEPH-94-54
mgr: BJ-YZ-CEPH-94-52(active), standbys: BJ-YZ-CEPH-94-54, BJ-YZ-CEPH-94-53
mds: fs-2/2/2 up  {0=BJ-YZ-CEPH-94-52=up:active,1=BJ-YZ-CEPH-94-53=up:active}, 1 up:standby-replay
osd: 36 osds: 36 up, 36 in

  data:
pools: 7 pools, 1152 pgs
objects: 37.66 M objects, 67 TiB
usage: 136 TiB used, 126 TiB / 262 TiB avail
pgs:    1148 active+clean
         4 active+clean+scrubbing+deep

  io:
client: 13 KiB/s rd, 27 MiB/s wr, 2 op/s rd, 19 op/s wr
复制
配置 npt sever：

$ systemctl status ntpd
$ systemctl start ntpd
复制
重启异常的 mon.targe 解决：

$ systemctl status ceph-mon.target
$ systemctl restart ceph-mon.target
复制
1 MDSs report slow requests
报错如下：

$ ceph -s
  cluster:
id:    b313ec26-5aa0-4db2-9fb5-a38b207471ee
health: HEALTH_WARN
         1 MDSs report slow requests
         Reduced data availability: 38 pgs inactive
         Degraded data redundancy: 122006/1192166 objects degraded (10.234%), 102 pgs degraded, 116 pgs undersized
         101 slow ops, oldest one blocked for 81045 sec, daemons [osd.1,osd.2] have slow ops.
复制
重启 mon 即可解决：

$ systemctl restart ceph-mon.target
复制
如果无法解决需要重启 mds 解决：

$ systemctl restart ceph-mds@${HOSTNAME}
复制
Reduced data availability: 38 pgs inactive
报错如下：https://zhuanlan.zhihu.com/p/74323736[1]

$ ceph -s
  cluster:
id:    b313ec26-5aa0-4db2-9fb5-a38b207471ee
health: HEALTH_WARN
         1 MDSs report slow requests
         Reduced data availability: 38 pgs inactive
         145 slow ops, oldest one blocked for 184238 sec, daemons [osd.1,osd.2] have slow ops.

  services:
mon: 3 daemons, quorum master001,master002,master003
mgr: master001(active), standbys: master002, master003
mds: kubernetes-2/2/2 up  {0=master001=up:active,1=master002=up:active}, 1 up:standby
osd: 3 osds: 3 up, 3 in
rgw: 1 daemon active

  data:
pools: 9 pools, 244 pgs
objects: 535.1 k objects, 177 GiB
usage: 470 GiB used, 4.1 TiB / 4.6 TiB avail
pgs:    15.574% pgs unknown
         206 active+clean
         38  unknown

  io:
client: 35 KiB/s wr, 0 op/s rd, 2 op/s wr
复制
此问题属于 pg 丢失数据并且无法自动回复造成的。解决办法是清除 pg 数据让其自动修复，但这样可能会造成数据丢失（如果 size 为 1 则肯定丢失数据）

首先查看异常的 pg：

然后执行 query 查看信息：

$ ceph pg 1.6e query
Error ENOENT: i don't have pgid 1.6e
复制
上述无法查到 pg，通过如下命令查看异常的 pg：

$ ceph pg dump_stuck unclean
ok
PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY
1.74 unknown []       -1    []          -1
1.70 unknown []       -1    []          -1
1.6a unknown []       -1    []          -1
1.2d unknown []       -1    []          -1
1.20 unknown []       -1    []          -1
1.1e unknown []       -1    []          -1
1.1c unknown []       -1    []          -1
1.17 unknown []       -1    []          -1
1.9    unknown []       -1    []          -1
1.29 unknown []       -1    []          -1
1.56 unknown []       -1    []          -1
1.72 unknown []       -1    []          -1
1.45 unknown []       -1    []          -1
1.4e unknown []       -1    []          -1
1.46 unknown []       -1    []          -1
1.22 unknown []       -1    []          -1
1.53 unknown []       -1    []          -1
1.59 unknown []       -1    []          -1
1.24 unknown []       -1    []          -1
1.55 unknown []       -1    []          -1
1.3f unknown []       -1    []          -1
1.38 unknown []       -1    []          -1
1.a    unknown []       -1    []          -1
1.7    unknown []       -1    []          -1
1.34 unknown []       -1    []          -1
1.64 unknown []       -1    []          -1
1.6    unknown []       -1    []          -1
1.32 unknown []       -1    []          -1
1.4    unknown []       -1    []          -1
1.2e unknown []       -1    []          -1
1.31 unknown []       -1    []          -1
1.5e unknown []       -1    []          -1
1.0    unknown []       -1    []          -1
1.42 unknown []       -1    []          -1
1.15 unknown []       -1    []          -1
1.6e unknown []       -1    []          -1
1.41 unknown []       -1    []          -1
1.10 unknown []       -1    []          -1
复制
执行如下命令强制清除 pg 的数据：https://docs.ceph.com/docs/mimic ... troubleshooting-pg/[2]

$ ceph osd force-create-pg 1.74 --yes-i-really-mean-it

# 批量执行
# ceph pg dump_stuck unclean|awk '{print $1}'|xargs -i ceph osd force-create-pg {} --yes-i-really-mean-it
复制
执行完成后即可恢复。

1 clients failing to respond to capability release
报错如下：

$ ceph  health detail
HEALTH_WARN 1 clients failing to respond to capability release
MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release
mdsmaster001(mds.0): Client master003.k8s.shileizcc-ops.com: failing to respond to capability release client_id: 284951
复制
清除次 ID 即可：https://blog.csdn.net/zuoyang1990/article/details/98530070[3]

$ ceph daemon mds.master003 session ls|grep 284951
$ ceph tell mds.master003 session evict id=284951
复制
如果报错如下：

$ ceph tell mds.master003 session evict id=284951
2020-08-13 10:45:03.869 7f271b7fe700  0 client.306366 ms_handle_reset on 10.100.21.95:6800/1646216103
2020-08-13 10:45:03.881 7f2730ff9700  0 client.316415 ms_handle_reset on 10.100.21.95:6800/1646216103
Error EAGAIN: MDS is replaying log
复制
需要到 mds.0 节点执行，否则无法找到次 client。

内核优化
内核优化：https://blog.csdn.net/fuzhongfaya/article/details/80932766[4]

$ echo "8192" > /sys/block/sda/queue/read_ahead_kb
$ echo "vm.swappiness = 0" | tee -a /etc/sysctl.conf
$ sysctl -p
$ echo "deadline" > /sys/block/sd[x]/queue/scheduler

# ssd
# echo "noop" > /sys/block/sd[x]/queue/scheduler
复制
swap 最好是直接关闭，配置内存参数在一定程度上不会生效。

配置文件

40 核心 128 GB 配置文件：

[global]
fsid = 2f77b028-ed2a-4010-9b79-90fd3052afc6
mon_initial_members = BJ-YZ-CEPH-94-52, BJ-YZ-CEPH-94-53, BJ-YZ-CEPH-94-54
mon_host = 10.100.94.52,10.100.94.53,10.100.94.54
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

public network = 10.100.94.0/24
cluster network = 10.100.94.0/24

[mon.a]
host = BJ-YZ-CEPH-94-52
mon addr = 10.100.94.52:6789

[mon.b]
host = BJ-YZ-CEPH-94-53
mon addr = 10.100.94.53:6789

[mon.c]
host = BJ-YZ-CEPH-94-54
mon addr = 10.100.94.54:6789

[mon]
mon data = /var/lib/ceph/mon/ceph-$id

# monitor 间的 clock drift，默认值 0.05
mon clock drift allowed = 1

# 向 monitor 报告 down 的最小 OSD 数，默认值 1
mon osd min down reporters = 1

# 标记一个OSD状态为down和out之前ceph等待的秒数，默认值300
mon osd down out interval = 600

mon_allow_pool_delete = true

[osd]
# osd 数据路径
osd data = /var/lib/ceph/osd/ceph-$id

# 默认 pool pg,pgp 数量
osd pool default pg num =  1200
osd pool default pgp num = 1200

# osd 的 journal 写日志时的大小默认 5120
osd journal size = 20000

# 格式化文件系统类型
osd mkfs type = xfs

# 格式化文件系统时附加参数
osd mkfs options xfs = -f

# 为 XATTRS 使用 object map，EXT4 文件系统时使用，XFS 或者 btrf 也可以使用，默认 false
filestore xattr use omap = true

# 从日志到数据盘最小同步间隔(seconds)，默认值 0.1
filestore min sync interval = 10

# 从日志到数据盘最大同步间隔(seconds)，默认值 5
filestore max sync interval = 15

# 数据盘最大接受的操作数，默认值 500
filestore queue max ops = 25000

# 数据盘能够 commit 的最大字节数(bytes)，默认值 100
filestore queue max bytes = 10485760

# 数据盘能够 commit 的操作数，500
filestore queue committing max ops = 5000

# 数据盘能够 commit 的最大字节数(bytes)，默认值 100
filestore queue committing max bytes = 10485760000

# 前一个子目录分裂成子目录中的文件的最大数量，默认值 2
filestore split multiple = 8

# 前一个子类目录中的文件合并到父类的最小数量，默认值10
filestore merge threshold = 40

# 对象文件句柄缓存大小，默认值 128
filestore fd cache size = 1024

# 并发文件系统操作数，默认值 2
filestore op threads = 32

# journal 一次性写入的最大字节数(bytes)，默认值 1048560
journal max write bytes = 1073714824

# journal一次性写入的最大记录数，默认值 100
journal max write entries = 10000

# journal一次性最大在队列中的操作数，默认值 50
journal queue max ops = 50000

# journal一次性最大在队列中的字节数(bytes)，默认值 33554432
journal queue max bytes = 10485760000

# # OSD一次可写入的最大值(MB), 默认 90
osd max write size = 512

# 客户端允许在内存中的最大数据(bytes), 默认值100
osd client message size cap = 2147483648

# 在 Deep Scrub 时候允许读取的字节数(bytes), 默认值524288
osd deep scrub stride = 1310720

# 并发文件系统操作数, 默认值 2
osd op threads = 32

# OSD 密集型操作例如恢复和 Scrubbing 时的线程, 默认值1
osd disk threads = 10

# 保留 OSD Map 的缓存(MB), 默认 500
osd map cache size = 10240

# OSD 进程在内存中的 OSD Map 缓存(MB), 默认 50
osd map cache bl size = 1280

# 默认值rw,noatime,inode64, Ceph OSD xfs Mount选项
osd mount options xfs = "rw,noexec,nodev,noatime,nodiratime,nobarrier"

# 恢复操作优先级，取值 1-63，值越高占用资源越高, 默认值 10
osd recovery op priority = 20

# 同一时间内活跃的恢复请求数, 默认值 15
osd recovery max active = 15

# 一个 OSD 允许的最大 backfills 数, 默认值 10
osd max backfills = 10

# 开启严格队列降级操作
osd op queue cut off = high

osd_deep_scrub_large_omap_object_key_threshold = 800000
osd_deep_scrub_large_omap_object_value_sum_threshold = 10737418240

[mds]
# mds 缓存大小设置 60GB
mds cache memory limit = 62212254726

# 超时时间默认 60 秒
mds_revoke_cap_timeout = 360

mds log max segments = 51200
mds log max expiring = 51200

mds_beacon_grace = 300

# 对目录碎片大小的硬限制默认 100000
# https://docs.ceph.com/docs/master/cephfs/dirfrags/
mds_bal_fragment_size_max = 500000

## 官方配置 https://ceph.readthedocs.io/en/latest/cephfs/mds-config-ref/

[client]

# RBD缓存, 默认 true
rbd cache = true

# RBD缓存大小(bytes), 默认 335544320（320M）
rbd cache size = 268435456

# 缓存为 write-back 时允许的最大 dirty 字节数(bytes)，如果为0，使用 write-through，默认值为 25165824
rbd cache max dirty = 134217728

# 在被刷新到存储盘前 dirty 数据存在缓存的时间(seconds), 默认值为 1
rbd cache max dirty age = 5

client_try_dentry_invalidate = false

[mgr]
mgr modules = dashboard

# 华为云调优指南 https://support.huaweicloud.com/ ... object_05_0008.html
# https://poph163.com/2020/02/18/c ... %E8%B0%83%E4%BC%98/
复制
full osd
full osd 每个 osd 已经写满上限:https://docs.ceph.com/en/latest/ ... no-free-drive-space[5]

$ ceph osd dump | grep full_ratio
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
复制
集群状态:

$ ceph -s
  cluster:
id:    2f77b028-ed2a-4010-9b79-90fd3052afc6
health: HEALTH_ERR
         2 backfillfull osd(s)
         1 full osd(s)
         2 nearfull osd(s)
         7 pool(s) full
复制
执行 osd 磁盘状态时，如果已经有超过 95% 使用率时则会报错 full osd 则会造成 cluster 无法正常使用：

$ ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZE USE    DATA OMAP    META AVAIL %USE  VAR  PGS
0 hdd 7.27689  1.00000 7.3 TiB 4.7 TiB 4.7 TiB  918 MiB 9.1 GiB 2.5 TiB 65.15 0.84  68
1 hdd 7.27689  1.00000 7.3 TiB 6.1 TiB 6.1 TiB  327 MiB  11 GiB 1.2 TiB 84.07 1.09  67
2 hdd 7.27689  1.00000 7.3 TiB 4.3 TiB 4.3 TiB  924 MiB 8.4 GiB 2.9 TiB 59.70 0.77  67
3 hdd 7.27689  1.00000 7.3 TiB 5.1 TiB 5.1 TiB  807 MiB 9.8 GiB 2.1 TiB 70.57 0.91  66
4 hdd 7.27689  1.00000 7.3 TiB 6.7 TiB 6.7 TiB  770 MiB  13 GiB 583 GiB 92.18 1.19  66
5 hdd 7.27689  1.00000 7.3 TiB 5.5 TiB 5.5 TiB  623 MiB  10 GiB 1.8 TiB 75.87 0.98  66
6 hdd 7.27689  1.00000 7.3 TiB 5.7 TiB 5.7 TiB  602 MiB  11 GiB 1.6 TiB 78.67 1.02  64
7 hdd 7.27689  1.00000 7.3 TiB 5.3 TiB 5.3 TiB  1.1 GiB  10 GiB 1.9 TiB 73.35 0.95  65
8 hdd 7.27689  1.00000 7.3 TiB 5.9 TiB 5.9 TiB  498 MiB  11 GiB 1.4 TiB 81.29 1.05  68
9 hdd 7.27689  1.00000 7.3 TiB 5.1 TiB 5.1 TiB  1.1 GiB 9.8 GiB 2.1 TiB 70.59 0.91  65
10 hdd 7.27689  1.00000 7.3 TiB 6.3 TiB 6.3 TiB  297 MiB  12 GiB 985 GiB 86.78 1.12  61
11 hdd 7.27689  1.00000 7.3 TiB 5.1 TiB 5.1 TiB  923 MiB 9.7 GiB 2.1 TiB 70.56 0.91  67
12 hdd 7.27689  1.00000 7.3 TiB 5.9 TiB 5.9 TiB  203 MiB  11 GiB 1.4 TiB 81.39 1.05  65
13 hdd 7.27689  1.00000 7.3 TiB 5.3 TiB 5.3 TiB  799 MiB  10 GiB 1.9 TiB 73.29 0.95  66
14 hdd 7.27689  1.00000 7.3 TiB 4.9 TiB 4.9 TiB  873 MiB 9.4 GiB 2.3 TiB 67.77 0.88  71
15 hdd 0.29999  1.00000 7.3 TiB 6.9 TiB 6.9 TiB  191 MiB  13 GiB 387 GiB 94.81 1.23  39
16 hdd 7.27689  1.00000 7.3 TiB 5.5 TiB 5.5 TiB  548 MiB  11 GiB 1.8 TiB 75.91 0.98  69
17 hdd 7.27689  1.00000 7.3 TiB 6.7 TiB 6.7 TiB  806 MiB  13 GiB 581 GiB 92.20 1.20  66
18 hdd 7.27689  1.00000 7.3 TiB 4.5 TiB 4.5 TiB  1.4 GiB 8.5 GiB 2.7 TiB 62.43 0.81  66
19 hdd 7.27689  1.00000 7.3 TiB 5.3 TiB 5.3 TiB  1.4 GiB  10 GiB 1.9 TiB 73.28 0.95  65
20 hdd 7.27689  1.00000 7.3 TiB 5.5 TiB 5.5 TiB  705 MiB  11 GiB 1.8 TiB 75.91 0.98  64
21 hdd 7.27689  1.00000 7.3 TiB 6.1 TiB 6.1 TiB  911 MiB  11 GiB 1.2 TiB 84.11 1.09  62
22 hdd 7.27689  1.00000 7.3 TiB 6.1 TiB 6.1 TiB  301 MiB  11 GiB 1.2 TiB 84.03 1.09  66
23 hdd 7.27689  1.00000 7.3 TiB 5.5 TiB 5.5 TiB  401 MiB 9.8 GiB 1.7 TiB 75.96 0.98  67
24 hdd 7.27689  1.00000 7.3 TiB 5.1 TiB 5.1 TiB  1.3 GiB 9.6 GiB 2.1 TiB 70.58 0.91  63
25 hdd 7.27689  1.00000 7.3 TiB 5.1 TiB 5.1 TiB  1.1 GiB 9.7 GiB 2.1 TiB 70.56 0.91  65
26 hdd 7.27689  1.00000 7.3 TiB 5.3 TiB 5.3 TiB  730 MiB  10 GiB 1.9 TiB 73.32 0.95  68
27 hdd 7.27689  1.00000 7.3 TiB 6.1 TiB 6.1 TiB  818 MiB  12 GiB 1.2 TiB 84.08 1.09  62
28 hdd 7.27689  1.00000 7.3 TiB 4.9 TiB 4.9 TiB  587 MiB 9.3 GiB 2.3 TiB 67.84 0.88  68
29 hdd 7.27689  1.00000 7.3 TiB 6.1 TiB 6.1 TiB  215 MiB  11 GiB 1.2 TiB 84.09 1.09  66
30 hdd 7.27689  1.00000 7.3 TiB 6.1 TiB 6.1 TiB  690 MiB  12 GiB 1.2 TiB 84.15 1.09  64
31 hdd 7.27689  1.00000 7.3 TiB 5.5 TiB 5.5 TiB 1020 MiB  10 GiB 1.8 TiB 75.94 0.98  64
32 hdd 7.27689  1.00000 7.3 TiB 6.5 TiB 6.5 TiB  616 MiB  12 GiB 786 GiB 89.45 1.16  66
33 hdd 7.27689  1.00000 7.3 TiB 4.9 TiB 4.9 TiB  622 MiB 8.9 GiB 2.3 TiB 67.84 0.88  66
34 hdd 7.27689  1.00000 7.3 TiB 5.7 TiB 5.7 TiB  102 MiB  11 GiB 1.6 TiB 78.56 1.02  65
35 hdd 7.27689  1.00000 7.3 TiB 5.9 TiB 5.9 TiB  723 MiB  11 GiB 1.4 TiB 81.31 1.05  63
                  TOTAL 262 TiB 202 TiB 202 TiB 25 GiB 381 GiB  60 TiB 77.15
复制
可以手动修改权重解决:

$ ceph osd crush reweight osd.4 0.3
复制
pg 均衡
pg 在默认分配有不合理的地方。https://cloud.tencent.com/developer/article/1664655[6]

$ ceph osd df tree | awk '/osd\./{print $NF" "$(NF-1)" "$(NF-3) }'
osd.0 89 71.20
osd.1 38 94.80
osd.2 92 68.44
osd.3 92 72.36
osd.4 28 76.86
osd.5 64 81.37
osd.6 62 87.90
osd.7 89 78.78
osd.8 52 86.18
osd.9 89 75.44
osd.10 37 96.33
osd.11 102 75.26
osd.12 33 91.41
osd.13 34 95.98
osd.14 59 84.97
osd.15 20 70.92
osd.16 113 89.46
osd.17 30 77.12
osd.18 124 77.11
osd.19 44 95.23
osd.20 65 84.63
osd.21 98 96.71
osd.22 34 95.93
osd.23 62 84.56
osd.24 110 76.63
osd.25 64 82.32
osd.26 59 88.26
osd.27 38 95.83
osd.28 105 79.19
osd.29 36 94.94
osd.30 94 90.79
osd.31 91 81.74
osd.32 12 42.44
osd.33 94 81.32
osd.34 46 86.51
osd.35 37 92.68
复制
reweight-by-pg 按归置组分布情况调整 OSD 的权重:

$ ceph osd reweight-by-pg
moved 0 / 2336 (0%)
avg 64.8889
stddev 58.677 -> 58.677 (expected baseline 7.9427)
min osd.1 with 0 -> 0 pgs (0 -> 0 * mean)
max osd.18 with 168 -> 168 pgs (2.58904 -> 2.58904 * mean)

oload 120
max_change 0.05
max_change_osds 4
average_utilization 18.2677
overload_utilization 21.9212
osd.19 weight 1.0000 -> 0.9500
osd.1 weight 1.0000 -> 0.9500
osd.27 weight 1.0000 -> 0.9500
osd.10 weight 1.0000 -> 0.9500
复制
reweight-by-utilization 按利用率调整 OSD 的权重:

$ ceph osd reweight-by-pg
moved 0 / 2336 (0%)
avg 64.8889
stddev 58.677 -> 58.677 (expected baseline 7.9427)
min osd.1 with 0 -> 0 pgs (0 -> 0 * mean)
max osd.18 with 168 -> 168 pgs (2.58904 -> 2.58904 * mean)

oload 120
max_change 0.05
max_change_osds 4
average_utilization 18.2677
overload_utilization 21.9212
osd.19 weight 1.0000 -> 0.9500
osd.1 weight 1.0000 -> 0.9500
osd.27 weight 1.0000 -> 0.9500
osd.10 weight 1.0000 -> 0.9500
复制
调整写入权重：

$ ceph osd reweight osd.35 0.001
复制
查看当前 osd 信息：

$ ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZE USE    DATA OMAP META AVAIL %USE  VAR  PGS
0 hdd 7.27689  1.00000 7.3 TiB 5.2 TiB 5.2 TiB 1.0 GiB 9.4 GiB 2.0 TiB 71.96 0.86  39
1 hdd 0.00999  0.90002 7.3 TiB 6.9 TiB 6.9 TiB 604 MiB  12 GiB 382 GiB 94.88 1.13  37
2 hdd 7.27689  1.00000 7.3 TiB 5.1 TiB 5.1 TiB 1.2 GiB 8.8 GiB 2.2 TiB 69.55 0.83  34
3 hdd 7.27689  1.00000 7.3 TiB 5.3 TiB 5.3 TiB 812 MiB 9.9 GiB 2.0 TiB 73.15 0.87  34
4 hdd 0.29999  1.00000 7.3 TiB 5.6 TiB 5.6 TiB 185 MiB  12 GiB 1.7 TiB 77.01 0.92  26
5 hdd 3.00000  1.00000 7.3 TiB 6.0 TiB 5.9 TiB 443 MiB  11 GiB 1.3 TiB 81.90 0.98  36
6 hdd 3.00000  1.00000 7.3 TiB 6.5 TiB 6.5 TiB 499 MiB  11 GiB 809 GiB 89.14 1.06  38
7 hdd 7.27689  1.00000 7.3 TiB 5.8 TiB 5.8 TiB 1.2 GiB  11 GiB 1.4 TiB 80.10 0.96  43
8 hdd 3.00000  1.00000 7.3 TiB 6.3 TiB 6.3 TiB 502 MiB  11 GiB 992 GiB 86.69 1.03  36
9 hdd 7.27689  1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.5 GiB 9.8 GiB 1.7 TiB 76.57 0.91  42
10 hdd 0.00999  0.00099 7.3 TiB 7.0 TiB 7.0 TiB 295 MiB  12 GiB 267 GiB 96.41 1.15  37
11 hdd 7.27689  1.00000 7.3 TiB 5.5 TiB 5.5 TiB 1.2 GiB 9.8 GiB 1.7 TiB 76.13 0.91  37
12 hdd 0.00999  1.00000 7.3 TiB 6.7 TiB 6.6 TiB  95 MiB  12 GiB 635 GiB 91.48 1.09  32
13 hdd 0.00999  1.00000 7.3 TiB 7.0 TiB 7.0 TiB 584 MiB  12 GiB 315 GiB 95.78 1.14  34
14 hdd 3.00000  1.00000 7.3 TiB 6.2 TiB 6.2 TiB 974 MiB  11 GiB 1.0 TiB 85.86 1.02  40
15 hdd 0.00999  1.00000 7.3 TiB 5.1 TiB 5.1 TiB 116 KiB  10 GiB 2.2 TiB 70.43 0.84  20
16 hdd 7.27689  1.00000 7.3 TiB 6.6 TiB 6.6 TiB 1.2 GiB  11 GiB 697 GiB 90.64 1.08  43
17 hdd 0.29999  1.00000 7.3 TiB 5.6 TiB 5.6 TiB  40 KiB  12 GiB 1.7 TiB 76.75 0.92  26
18 hdd 7.27689  1.00000 7.3 TiB 5.7 TiB 5.7 TiB 1.9 GiB 9.3 GiB 1.6 TiB 78.01 0.93  53
19 hdd 0.00999  0.00099 7.3 TiB 6.9 TiB 6.9 TiB 1.5 GiB  13 GiB 371 GiB 95.02 1.13  40
20 hdd 3.00000  1.00000 7.3 TiB 6.2 TiB 6.2 TiB 744 MiB  12 GiB 1.0 TiB 85.86 1.02  37
21 hdd 7.27689  0.00099 7.3 TiB 7.0 TiB 7.0 TiB 913 MiB  12 GiB 239 GiB 96.79 1.15  40
22 hdd 0.00999  0.00099 7.3 TiB 7.0 TiB 7.0 TiB 283 MiB  12 GiB 298 GiB 96.00 1.14  34
23 hdd 3.00000  1.00000 7.3 TiB 6.2 TiB 6.2 TiB 515 MiB  11 GiB 1.1 TiB 85.30 1.02  35
24 hdd 7.27689  1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.4 GiB 9.8 GiB 1.6 TiB 77.63 0.93  42
25 hdd 3.00000  1.00000 7.3 TiB 6.0 TiB 6.0 TiB 1.2 GiB  10 GiB 1.3 TiB 82.66 0.99  40
26 hdd 2.00000  1.00000 7.3 TiB 6.5 TiB 6.5 TiB 737 MiB  11 GiB 823 GiB 88.95 1.06  36
27 hdd 0.00999  0.00099 7.3 TiB 7.0 TiB 6.9 TiB 822 MiB  12 GiB 327 GiB 95.61 1.14  37
28 hdd 7.27689  1.00000 7.3 TiB 5.8 TiB 5.8 TiB 859 MiB  10 GiB 1.4 TiB 80.23 0.96  40
29 hdd 0.00999  0.00099 7.3 TiB 6.9 TiB 6.9 TiB 215 MiB  12 GiB 371 GiB 95.02 1.13  36
30 hdd 7.27689  1.00000 7.3 TiB 6.7 TiB 6.7 TiB 1.0 GiB  12 GiB 607 GiB 91.85 1.10  47
31 hdd 7.27689  1.00000 7.3 TiB 6.0 TiB 6.0 TiB 1.2 GiB  10 GiB 1.3 TiB 82.81 0.99  41
32 hdd 0.29999  1.00000 7.3 TiB 3.0 TiB 3.0 TiB  32 KiB 7.1 GiB 4.3 TiB 41.47 0.49  10
33 hdd 7.27689  1.00000 7.3 TiB 6.0 TiB 6.0 TiB 827 MiB 9.7 GiB 1.3 TiB 82.06 0.98  41
34 hdd 2.00000  1.00000 7.3 TiB 6.3 TiB 6.3 TiB 308 MiB  11 GiB 976 GiB 86.90 1.04  33
35 hdd 0.00999  0.00099 7.3 TiB 6.7 TiB 6.7 TiB 613 MiB  12 GiB 540 GiB 92.75 1.11  36
                  TOTAL 262 TiB 220 TiB 219 TiB  27 GiB 391 GiB  42 TiB 83.87
MIN/MAX VAR: 0.49/1.15  STDDEV: 10.62
复制
删除 Cephfs

关闭所有 mds 服务, 需要登入服务器手动关闭:

$ systemctl stop ceph-mds@${HOSTNAME}
复制
删除所需 fs:

$ ceph fs ls
$ ceph fs rm data --yes-i-really-mean-it
复制
SSD 使用

查看当前 OSD 状态: (相关文档:https://blog.csdn.net/kozazyh/article/details/79904219[7])

$ ceph osd crush class ls
[
"ssd"
]
复制
如果使用的 SSD 标识错误，请自定义修改，命令如下, 移除 osd 1 ~ 3 的标识:

$ for i in 0 1 2;do ceph osd crush rm-device-class osd.$i;done
复制
设置 1 ~ 3 标识为 ssd：

$ for i in 0 1 2;do ceph osd crush set-device-class ssd osd.$i;done
复制
创建一个 crush rule:

$ ceph osd crush rule create-replicated rule-ssd default host ssd
$ ceph osd crush rule ls
复制
然后创建 pool 时附带 rule 的名称：

$ ceph osd pool create fs_data 96 rule-ssd
$ ceph osd pool create fs_metadata 16 rule-ssd
$ ceph fs new fs fs_data fs_metadata
复制
crushmap 查看
执行命令如下:

$ ceph osd getcrushmap -o crushmap
$ crushtool -d crushmap -o crushmap
$ cat crushmap
复制
3 monitors have not enabled msgr2
解决如下：

$ ceph mon enable-msgr2
复制
2 daemons have recently crashed
解决如下：https://blog.csdn.net/QTM_Gitee/article/details/106004435[8]

$ ceph crash ls
$ ceph crash archive-all

admin · 发表于 2022-7-29 09:56:46

# systemctl stop ceph-mds@p04.service
# cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary
# cephfs-journal-tool --rank=cephfs:0 journal reset
# cephfs-table-tool all reset session
# systemctl start ceph-mds@p04.service
# ceph mds repaired 0

JOURNAL EXPORT

Before attempting dangerous operations, make a copy of the journal like so:

cephfs-journal-tool journal export backup.bin

Note that this command may not always work if the journal is badly corrupted, in which case a RADOS-level copy should be made (http://tracker.ceph.com/issues/9902).

DENTRY RECOVERY FROM JOURNAL

If a journal is damaged or for any reason an MDS is incapable of replaying it, attempt to recover what file metadata we can like so:

cephfs-journal-tool event recover_dentries summary

This command by default acts on MDS rank 0, pass --rank=<n> to operate on other ranks.

This command will write any inodes/dentries recoverable from the journal into the backing store, if these inodes/dentries are higher-versioned than the previous contents of the backing store. If any regions of the journal are missing/damaged, they will be skipped.

Note that in addition to writing out dentries and inodes, this command will update the InoTables of each ‘in’ MDS rank, to indicate that any written inodes’ numbers are now in use. In simple cases, this will result in an entirely valid backing store state.

Warning

The resulting state of the backing store is not guaranteed to be self-consistent, and an online MDS scrub will be required afterwards. The journal contents will not be modified by this command, you should truncate the journal separately after recovering what you can.

JOURNAL TRUNCATION

If the journal is corrupt or MDSs cannot replay it for any reason, you can truncate it like so:

cephfs-journal-tool [--rank=N] journal reset

Specify the MDS rank using the --rank option when the file system has/had multiple active MDS.

Warning

Resetting the journal will lose metadata unless you have extracted it by other means such as recover_dentries. It is likely to leave some orphaned objects in the data pool. It may result in re-allocation of already-written inodes, such that permissions rules could be violated.

MDS TABLE WIPES

After the journal has been reset, it may no longer be consistent with respect to the contents of the MDS tables (InoTable, SessionMap, SnapServer).

To reset the SessionMap (erase all sessions), use:

cephfs-table-tool all reset session

This command acts on the tables of all ‘in’ MDS ranks. Replace ‘all’ with an MDS rank to operate on that rank only.

The session table is the table most likely to need resetting, but if you know you also need to reset the other tables then replace ‘session’ with ‘snap’ or ‘inode’.

MDS MAP RESET

Once the in-RADOS state of the file system (i.e. contents of the metadata pool) is somewhat recovered, it may be necessary to update the MDS map to reflect the contents of the metadata pool. Use the following command to reset the MDS map to a single MDS:

ceph fs reset <fs name> --yes-i-really-mean-it

Once this is run, any in-RADOS state for MDS ranks other than 0 will be ignored: as a result it is possible for this to result in data loss.

One might wonder what the difference is between ‘fs reset’ and ‘fs remove; fs new’. The key distinction is that doing a remove/new will leave rank 0 in ‘creating’ state, such that it would overwrite any existing root inode on disk and orphan any existing files. In contrast, the ‘reset’ command will leave rank 0 in ‘active’ state such that the next MDS daemon to claim the rank will go ahead and use the existing in-RADOS metadata.

RECOVERY FROM MISSING METADATA OBJECTS

Depending on what objects are missing or corrupt, you may need to run various commands to regenerate default versions of the objects.

# Session tablecephfs-table-tool 0 reset session# SnapServercephfs-table-tool 0 reset snap# InoTablecephfs-table-tool 0 reset inode# Journalcephfs-journal-tool --rank=0 journal reset# Root inodes ("/" and MDS directory)cephfs-data-scan init

Finally, you can regenerate metadata objects for missing files and directories based on the contents of a data pool. This is a three-phase process. First, scanning all objects to calculate size and mtime metadata for inodes. Second, scanning the first object from every file to collect this metadata and inject it into the metadata pool. Third, checking inode linkages and fixing found errors.

cephfs-data-scan scan_extents <data pool>cephfs-data-scan scan_inodes <data pool>cephfs-data-scan scan_links

‘scan_extents’ and ‘scan_inodes’ commands may take a very long time if there are many files or very large files in the data pool.

To accelerate the process, run multiple instances of the tool.

Decide on a number of workers, and pass each worker a number within the range 0-(worker_m - 1).

The example below shows how to run 4 workers simultaneously:

# Worker 0cephfs-data-scan scan_extents --worker_n 0 --worker_m 4 <data pool># Worker 1cephfs-data-scan scan_extents --worker_n 1 --worker_m 4 <data pool># Worker 2cephfs-data-scan scan_extents --worker_n 2 --worker_m 4 <data pool># Worker 3cephfs-data-scan scan_extents --worker_n 3 --worker_m 4 <data pool># Worker 0cephfs-data-scan scan_inodes --worker_n 0 --worker_m 4 <data pool># Worker 1cephfs-data-scan scan_inodes --worker_n 1 --worker_m 4 <data pool># Worker 2cephfs-data-scan scan_inodes --worker_n 2 --worker_m 4 <data pool># Worker 3cephfs-data-scan scan_inodes --worker_n 3 --worker_m 4 <data pool>

It is important to ensure that all workers have completed the scan_extents phase before any workers enter the scan_inodes phase.

After completing the metadata recovery, you may want to run cleanup operation to delete ancillary data generated during recovery.

cephfs-data-scan cleanup <data pool>

USING AN ALTERNATE METADATA POOL FOR RECOVERY

Warning

There has not been extensive testing of this procedure. It should be undertaken with great care.

If an existing file system is damaged and inoperative, it is possible to create a fresh metadata pool and attempt to reconstruct the file system metadata into this new pool, leaving the old metadata in place. This could be used to make a safer attempt at recovery since the existing metadata pool would not be modified.

Caution

During this process, multiple metadata pools will contain data referring to the same data pool. Extreme caution must be exercised to avoid changing the data pool contents while this is the case. Once recovery is complete, the damaged metadata pool should be archived or deleted.

To begin, the existing file system should be taken down, if not done already, to prevent further modification of the data pool. Unmount all clients and then mark the file system failed:

ceph fs fail <fs_name>

Next, create a recovery file system in which we will populate a new metadata pool backed by the original data pool.

ceph fs flag set enable_multiple true --yes-i-really-mean-itceph osd pool create cephfs_recovery_metaceph fs new cephfs_recovery recovery <data_pool> --allow-dangerous-metadata-overlay

The recovery file system starts with an MDS rank that will initialize the new metadata pool with some metadata. This is necessary to bootstrap recovery. However, now we will take the MDS down as we do not want it interacting with the metadata pool further.

ceph fs fail cephfs_recovery

Next, we will reset the initial metadata the MDS created:

cephfs-table-tool cephfs_recovery:all reset sessioncephfs-table-tool cephfs_recovery:all reset snapcephfs-table-tool cephfs_recovery:all reset inode

Now perform the recovery of the metadata pool from the data pool:

cephfs-data-scan init --force-init --filesystem cephfs_recovery --alternate-pool cephfs_recovery_metacephfs-data-scan scan_extents --alternate-pool cephfs_recovery_meta --filesystem <fs_name> <data_pool>cephfs-data-scan scan_inodes --alternate-pool cephfs_recovery_meta --filesystem <fs_name> --force-corrupt <data_pool>cephfs-data-scan scan_links --filesystem cephfs_recovery

Note

Each scan procedure above goes through the entire data pool. This may take a significant amount of time. See the previous section on how to distribute this task among workers.

If the damaged file system contains dirty journal data, it may be recovered next with:

cephfs-journal-tool --rank=<fs_name>:0 event recover_dentries list --alternate-pool cephfs_recovery_metacephfs-journal-tool --rank cephfs_recovery:0 journal reset --force

After recovery, some recovered directories will have incorrect statistics. Ensure the parameters mds_verify_scatter and mds_debug_scatterstat are set to false (the default) to prevent the MDS from checking the statistics:

ceph config rm mds mds_verify_scatterceph config rm mds mds_debug_scatterstat

(Note, the config may also have been set globally or via a ceph.conf file.) Now, allow an MDS to join the recovery file system:

ceph fs set cephfs_recovery joinable true

Finally, run a forward scrub to repair the statistics. Ensure you have an MDS running and issue:

ceph fs status # get active MDSceph tell mds.<id> scrub start / recursive repair

Note

Symbolic links are recovered as empty regular files. Symbolic link recovery is scheduled to be supported in Pacific.

It is recommended to migrate any data from the recovery file system as soon as possible. Do not restore the old file system while the recovery file system is operational.

Note

If the data pool is also corrupt, some files may not be restored because backtrace information is lost. If any data objects are missing (due to issues like lost Placement Groups on the data pool), the recovered files will contain holes in place of the missing data.

[color=rgb(64, 64, 64) !important][backcolor=rgb(243, 246, 246) !important] Previous

[color=rgb(64, 64, 64) !important][backcolor=rgb(243, 246, 246) !important]Next

admin · 发表于 2022-7-29 10:05:00

1、创建一个 Ceph 文件系统
1、首先要创建两个pool，一个是cephfs-data，一个是cephfs-metadate，分别存储文件数据和文件元数据，这个pg也可以设置小一点，这个根据OSD去配置

#ceph osd pool create cephfs-data 256 256
#ceph osd pool create cephfs-metadata 64 64
查看已经创建成功

[root@cephnode01 my-cluster]# ceph osd lspools
1 .rgw.root
2 default.rgw.control
3 default.rgw.meta
4 default.rgw.log
5 rbd
6 cephfs-data
7 cephfs-metadata
关于ceph的日志，可以在/var/log/ceph下可以查看到相关信息

[root@cephnode01 my-cluster]# tail -f /var/log/ceph/ceph
ceph.audit.log                ceph.log                      ceph-mgr.cephnode01.log       ceph-osd.0.log
ceph-client.rgw.cephnode01.log  ceph-mds.cephnode01.log       ceph-mon.cephnode01.log       ceph-volume.log
注：一般 metadata pool 可以从相对较少的 PGs 启动, 之后可以根据需要增加 PGs. 因为 metadata pool 存储着 CephFS文件的元数据, 为了保证安全, 最好有较多的副本数. 为了能有较低的延迟, 可以考虑将 metadata 存储在 SSDs 上.
2、创建一个 CephFS, 名字为 cephfs:需要指定两个创建的pool的名字

#ceph fs new cephfs cephfs-metadata cephfs-data
new fs with metadata pool 7 and data pool 6
3、验证至少有一个 MDS 已经进入 Active 状态，也就是活跃
另外可以看到两个备用的是cephnode01,和cephnode03

#ceph fs status cephfs
cephfs - 0 clients

+------+--------+------------+---------------+-------+-------+
| Rank | State  | MDS    | Activity |  dns  |  inos |
+------+--------+------------+---------------+-------+-------+
|  0 | active | cephnode02 | Reqs: 0 /s | 10  | 13  |
+------+--------+------------+---------------+-------+-------+
+-----------------+----------+-------+-------+
|    Pool    | type |  used | avail |
+-----------------+----------+-------+-------+
| cephfs-metadata | metadata | 1536k | 17.0G |
| cephfs-data | data | 0  | 17.0G |
+-----------------+----------+-------+-------+
+-------------+
| Standby MDS |
+-------------+
|  cephnode01 |
|  cephnode03 |
+-------------+
MDS version: ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable)
4、在 Monitor 上, 创建一个叫client.cephfs的用户，用于访问CephFs

#ceph auth get-or-create client.cephfs mon 'allow r' mds 'allow rw' osd 'allow rw pool=cephfs-data, allow rw pool=cephfs-metadata'
这里会生成一个key,用户需要拿这个key去访问
[client.cephfs]
key = AQA5IV5eNCwMGRAAy4dIZ8+ISfBcwZegFTYD6Q==
查看权限列表，有哪些用户创建了权限

[root@cephnode01 my-cluster]# ceph auth list
client.cephfs
key: AQA5IV5eNCwMGRAAy4dIZ8+ISfBcwZegFTYD6Q==
caps: [mds] allow rw
caps: [mon] allow r
caps: [osd] allow rw pool=cephfs-data, allow rw pool=cephfs-metadata
client.rgw.cephnode01
key: AQBOAl5eGVL/HBAAYH93c4wPiBlD7YhuPY0u7Q==
caps: [mon] allow rw
caps: [osd] allow r
5、验证key是否生效

#ceph auth get client.cephfs
可以看到这个用户是拥有访问cephfs的读写权限的
exported keyring for client.cephfs
[client.cephfs]
key = AQA5IV5eNCwMGRAAy4dIZ8+ISfBcwZegFTYD6Q==
caps mds = "allow rw"
caps mon = "allow r"
caps osd = "allow rw pool=cephfs-data, allow rw pool=cephfs-metadata"
6、检查CephFs和mds状态

#ceph -s 查看集群已经增加mds配置
  cluster:
id:    75aade75-8a3a-47d5-ae44-ec3a84394033
health: HEALTH_OK

  services:
mon: 3 daemons, quorum cephnode01,cephnode02,cephnode03 (age 2h)
mgr: cephnode01(active, since 2h), standbys: cephnode02, cephnode03
mds: cephfs:1 {0=cephnode02=up:active} 2 up:standby
osd: 3 osds: 3 up (since 2h), 3 in (since 2h)
rgw: 1 daemon active (cephnode01)

  data:
pools: 7 pools, 96 pgs
objects: 263 objects, 29 MiB
usage: 3.1 GiB used, 54 GiB / 57 GiB avail
pgs:    96 active+clean

#ceph mds stat
这里显示1个是active状态，2个备用状态
cephfs:1 {0=cephnode02=up:active} 2 up:standby

#ceph fs ls
这里有两个pool
name: cephfs, metadata pool: cephfs-metadata, data pools: [cephfs-data ]
#ceph fs status
1.1 以 kernel client 形式挂载 CephFS
这里使用其他的机器进行挂载，这里是是以prometheus主机挂载，不过这个在哪挂载都可以，kernel主要联系系统内核，和系统内核进行做相互，用这种方式进行挂载文件系统
1、创建挂载目录 cephfs
#mkdir /cephfs

2、挂载目录，这里写集群ceph节点的地址，后面跟创建用户访问集群的key

#mount -t ceph 192.168.1.10:6789,192.168.1.11:6789,192.168.1.12:6789:/ /cephfs/ -o name=cephfs,secret=AQDHjeddHlktJhAAxDClZh9mvBxRea5EI2xD9w==
3、自动挂载
#echo "mon1:6789,mon2:6789,mon3:6789:/ /cephfs ceph name=cephfs,secretfile=/etc/ceph/cephfs.key,_netdev,noatime 0 0" | sudo tee -a /etc/fstab

4、验证是否挂载成功

#stat -f /cephfs
  文件："/cephfs"
ID：4f32eedbe607030e 文件名长度：255    类型：ceph
块大小：4194304 基本块大小：4194304
块：总计：4357    空闲：4357    可用：4357
Inodes: 总计：0       空闲：-1
1.2 以 FUSE client 形式挂载 CephFS
1、安装ceph-common，安装好可以使用rbd,ceph相关命令
这里还是使用我们的内网yum源来安装这些依赖包

yum -y install epel-release
yum install -y ceph-common
2、安装ceph-fuse，ceph的客户端工具，也就是用ceph的方式把这个文件系统挂上
yum install -y ceph-fuse

3、将集群的ceph.conf拷贝到客户端

scp root@192.168.1.10:/etc/ceph/ceph.conf /etc/ceph/
chmod 644 /etc/ceph/ceph.conf
4、使用 ceph-fuse 挂载 CephFS
如果是在其他主机挂载的话，需要这个使用cephfs的key,这个是刚才我们创建好的
直接拿这台服务器上用就可以

[root@prometheus ~]# more /etc/ceph/ceph.client.cephfs.keyring
exported keyring for client.cephfs
[client.cephfs]
key = AQA5IV5eNCwMGRAAy4dIZ8+ISfBcwZegFTYD6Q==
caps mds = "allow rw"
caps mon = "allow r"
caps osd = "allow rw pool=cephfs-data, allow rw pool=cephfs-metadata"

#ceph-fuse --keyring /etc/ceph/ceph.client.cephfs.keyring --name client.cephfs -m 192.168.1.10:6789,192.168.1.11:6789,192.168.1.12:6789 /cephfs/
5、验证 CephFS 已经成功挂载

#df -h
ceph-fuse                         18G    0 18G 0% /cephfs

#stat -f /cephfs
  文件："/cephfs/"
ID：0       文件名长度：255    类型：fuseblk
块大小：4194304 基本块大小：4194304
块：总计：4357    空闲：4357    可用：4357
Inodes: 总计：1       空闲：0
6、自动挂载

#echo "none /cephfs fuse.ceph ceph.id=cephfs[,ceph.conf=/etc/ceph/ceph.conf],_netdev,defaults 0 0"| sudo tee -a /etc/fstab
或
#echo "id=cephfs,conf=/etc/ceph/ceph.conf /mnt/ceph3 fuse.ceph _netdev,defaults 0 0"| sudo tee -a /etc/fstab
7、卸载
#fusermount -u /cephfs

		自动登录	找回密码
密码			注册