ceph 分布式存储手动方式添加mds文件系统

admin · 发表于 2022-7-26 15:23:23

购买主题本主题需向作者支付 5 金钱 才能浏览

admin · 发表于 2022-7-26 15:24:06

#在这里快速回复#bbs168x.cn/forum.php?mod=viewthread&tid=2319&extra=

admin · 发表于 2022-7-26 17:34:20

admin · 发表于 2022-7-26 17:38:50

存储池

创建存储池
ceph osd pool create {pool-name} {pg-num} [{pgp-num}] [replicated] \
      [crush-ruleset-name] [expected-num-objects]
ceph osd pool create {pool-name} {pg-num}  {pgp-num} erasure \
      [erasure-code-profile] [crush-ruleset-name] [expected_num_objects]

例如：

[root@ceph-3 ~]# ceph osd pool create cephfsdata 16 16
pool 'cephfsdata' created
[root@ceph-3 ~]# ceph osd pool create cephfsmetadata 16 16
pool 'cephfsmetadata' created

注意可能会报如下错误：
Error EINVAL: crushtool check failed with -22: crushtool: exec failed: (2) No such file or directory

解决方法：将crushtool模块移动到/usr/bin/目录下

创建ceph文件系统
ceph fs new <fs_name> <metadata> <data>

例如：
ceph fs new cephfs fsmetadata fsdata
[root@ceph-3 ~]# ceph fs new cephfs cephfsmetadata cephfsdata
new fs with metadata pool 3 and data pool 2

ok，到这里你就可以挂载文件系统了。。。

admin · 发表于 2022-7-26 17:43:14

[root@ceph-3 ~]# ceph osd pool ls detail
pool 2 'cephfsdata' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 last_change 115 flags hashpspool stripe_width 0 application cephfs
pool 3 'cephfsmetadata' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 last_change 115 flags hashpspool stripe_width 0 application cephfs

[root@ceph-3 ~]# ceph osd pool set cephfsdata size 3
set pool 2 size to 3
[root@ceph-3 ~]# ceph osd pool set cephfsmetadata size 3
set pool 3 size to 3
[root@ceph-3 ~]# ceph -s
  cluster:
id:    57bf4711-2218-46af-99d6-9c68ae230ce1
health: HEALTH_WARN
         too few PGs per OSD (13 < min 30)

  services:
mon: 3 daemons, quorum ceph-2,ceph-1,ceph-3
mgr: ceph-1(active), standbys: ceph-2, ceph-3
mds: cephfs-1/1/1 up  {0=ceph-2=up:active}, 2 up:standby
osd: 6 osds: 6 up, 6 in

  data:
pools: 2 pools, 32 pgs
objects: 21 objects, 2.19KiB
usage: 652MiB used, 584GiB / 585GiB avail
pgs:    32 active+clean

[root@ceph-3 ~]#

[root@ceph-3 ~]# ceph osd pool set cephfsdata pg_num 32
set pool 2 pg_num to 32
[root@ceph-3 ~]# ceph osd pool set cephfsdata pgp_num 32
set pool 2 pgp_num to 32
[root@ceph-3 ~]# ceph osd pool set cephfsmetadata pg_num 32
set pool 3 pg_num to 32
[root@ceph-3 ~]# ceph osd pool set cephfsmetadata pgp_num 32
set pool 3 pgp_num to 32
[root@ceph-3 ~]# ceph -s
  cluster:
id:    57bf4711-2218-46af-99d6-9c68ae230ce1
health: HEALTH_WARN
         Reduced data availability: 15 pgs inactive, 8 pgs peering
         too few PGs per OSD (29 < min 30)

  services:
mon: 3 daemons, quorum ceph-2,ceph-1,ceph-3
mgr: ceph-1(active), standbys: ceph-2, ceph-3
mds: cephfs-1/1/1 up  {0=ceph-2=up:active}, 2 up:standby
osd: 6 osds: 6 up, 6 in

  data:
pools: 2 pools, 64 pgs
objects: 12 objects, 1.12KiB
usage: 654MiB used, 584GiB / 585GiB avail
pgs:    7.812% pgs unknown
         60.938% pgs not active
         27 activating
         20 active+clean
         12 peering
         5  unknown

  io:
client: 127B/s wr, 0op/s rd, 1op/s wr
recovery: 2B/s, 0objects/s

加载/卸载cephfs过程。

在这里我做的比较简单，就是在ceph集群的机器上做的cephfs加载操作。若在其它客户端进行加载的话，需要将ceph集群配置文件ceph.conf以及访问cephfs的用户keyring复制到对应的客户端。对于通过内核方式加载来说，需要内核模块支持cephfs，对于通过fuse方式加载来说，需要手动安装ceph-fuse这个包。

1、通过内核模块的方式加载cephfs。

#mkdir -p /mnt/mycephfs

#mount -t ceph 127.0.0.1:6789:/ /mnt/mycephfs

之后通过mount命令就可以看到cephfs已经成功的挂载到/mnt/mycephfs目录下。后面就可以和操作本地文件系统一样操作/mnt/mycephfs目录。

2、通过内核模块的方式卸载cephfs。

#umount /mnt/mycephfs

3、通过fuse方式加载cephfs。

#ceph-fuse -m 127.0.0.1:6789 /mnt/mycephfs
[root@ceph-3 ~]# ceph-fuse -m 192.168.120.31:6789 /mnt/mycephfs/
2022-07-26 17:47:13.582325 7fadc9ddc0c0 -1 init, newargv = 0x558a74c0e720 newargc=9
ceph-fuse[30570]: starting ceph client
ceph-fuse[30570]: starting fuse
[root@ceph-3 ~]# df -Th
Filesystem          Type          Size  Used Avail Use% Mounted on
devtmpfs             devtmpfs       1.9G    0  1.9G 0% /dev
tmpfs                tmpfs          1.9G    0  1.9G 0% /dev/shm
tmpfs                tmpfs          1.9G  159M  1.7G 9% /run
tmpfs                tmpfs          1.9G    0  1.9G 0% /sys/fs/cgroup
/dev/mapper/rhel-root xfs             50G  7.6G 42G  16% /
/dev/vda1          xfs          497M  143M  355M  29% /boot
tmpfs                tmpfs          379M    0  379M 0% /run/user/0
/dev/vdb1          xfs             98G  109M 98G 1% /var/lib/ceph/osd/ceph-4
/dev/vdc1          xfs             98G  109M 98G 1% /var/lib/ceph/osd/ceph-5
ceph-fuse          fuse.ceph-fuse  185G    0  185G 0% /mnt/mycephfs

之后通过mount命令就可以看到cephfs已经成功的挂载到/mnt/mycephfs目录下。后面就可以和操作本地文件系统一样操作/mnt/mycephfs目录。

4、通过fuse方式卸载cephfs。

#fusermount -u /mnt/mycephfs

admin · 发表于 2022-7-26 17:43:48

[root@ceph-3 ~]# ceph -s
  cluster:
id:    57bf4711-2218-46af-99d6-9c68ae230ce1
health: HEALTH_OK

  services:
mon: 3 daemons, quorum ceph-2,ceph-1,ceph-3
mgr: ceph-1(active), standbys: ceph-2, ceph-3
mds: cephfs-1/1/1 up  {0=ceph-2=up:active}, 2 up:standby
osd: 6 osds: 6 up, 6 in

  data:
pools: 2 pools, 64 pgs
objects: 21 objects, 2.19KiB
usage: 653MiB used, 584GiB / 585GiB avail
pgs:    64 active+clean

  io:
client: 0B/s wr, 0op/s rd, 0op/s wr
recovery: 0B/s, 0objects/s

		自动登录	找回密码
密码			注册