ceph health detail HEALTH_WARN 1 pools have many more objects per pg than averag

admin · 发表于 2021-9-8 17:27:55

# ceph health detail
HEALTH_WARN 1 pools have many more objects per pg than average
MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average
pool pool-hdd-2 objects per pg (5503) is more than 12.9482 times cluster average (425)

定位问题
[root@lab8106 ~]# ceph -s
cluster fa7ec1a1-662a-4ba3-b478-7cb570482b62
   health HEALTH_WARN
         pool rbd has many more objects per pg than average (too few pgs?)
   monmap e1: 1 mons at {lab8106=192.168.8.106:6789/0}
         election epoch 30, quorum 0 lab8106
   osdmap e157: 2 osds: 2 up, 2 in
         flags sortbitwise
   pgmap v1023: 417 pgs, 13 pools, 18519 MB data, 15920 objects
         18668 MB used, 538 GB / 556 GB avail
               417 active+clean
集群出现了这个警告，pool rbd has many more objects per pg than average (too few pgs?) 这个警告在hammer版本里面的提示是 pool rbd has too few pgs
这个地方查看集群详细信息：
[root@lab8106 ~]# ceph health detail
HEALTH_WARN pool rbd has many more objects per pg than average (too few pgs?); mon.lab8106 low disk space
pool rbd objects per pg (1912) is more than 50.3158 times cluster average (38)
看下集群的pool的对象状态
[root@lab8106 ~]# ceph df
GLOBAL:
SIZE    AVAIL    RAW USED    %RAW USED
556G    538G    18668M       3.28
POOLS:
NAME    ID    USED    %USED    MAX AVAIL    OBJECTS
rbd       6    16071M    2.82       536G    15296
pool1    7       204M    0.04       536G       52
pool2    8       184M    0.03       536G       47
pool3    9       188M    0.03       536G       48
pool4    10    192M    0.03       536G       49
pool5    11    204M    0.04       536G       52
pool6    12    148M    0.03       536G       38
pool7    13    184M    0.03       536G       47
pool8    14    200M    0.04       536G       51
pool9    15    200M    0.04       536G       51
pool10    16    248M    0.04       536G       63
pool11    17    232M    0.04       536G       59
pool12    18    264M    0.05       536G       67
查看存储池的pg个数
[root@lab8106 ~]# ceph osd dump|grep pool
pool 6 'rbd' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 132 flags hashpspool stripe_width 0
pool 7 'pool1' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 134 flags hashpspool stripe_width 0
pool 8 'pool2' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 136 flags hashpspool stripe_width 0
pool 9 'pool3' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 138 flags hashpspool stripe_width 0
pool 10 'pool4' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 140 flags hashpspool stripe_width 0
pool 11 'pool5' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 142 flags hashpspool stripe_width 0
pool 12 'pool6' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 144 flags hashpspool stripe_width 0
pool 13 'pool7' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 146 flags hashpspool stripe_width 0
pool 14 'pool8' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 148 flags hashpspool stripe_width 0
pool 15 'pool9' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 150 flags hashpspool stripe_width 0
pool 16 'pool10' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 152 flags hashpspool stripe_width 0
pool 17 'pool11' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 154 flags hashpspool stripe_width 0
pool 18 'pool12' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 200 pgp_num 200 last_change 156 flags hashpspool stripe_width 0
我们看下这个是怎么得到的
pool rbd objects per pg (1912) is more than 50.3158 times cluster average (38)
rbd objects_per_pg = 15296 / 8 = 1912
objects_per_pg = 15920 /417 ≈ 38
50.3158 = rbd objects_per_pg / objects_per_pg = 1912 / 38
也就是出现其他pool的对象太少，而这个pg少，对象多，就会提示这个了，我们看下代码里面的判断
https://github.com/ceph/ceph/blob/master/src/mon/PGMonitor.cc
int average_objects_per_pg = pg_map.pg_sum.stats.sum.num_objects / pg_map.pg_stat.size();
   if (average_objects_per_pg > 0 &&
      pg_map.pg_sum.stats.sum.num_objects >= g_conf->mon_pg_warn_min_objects &&
      p->second.stats.sum.num_objects >= g_conf->mon_pg_warn_min_pool_objects) {
int objects_per_pg = p->second.stats.sum.num_objects / pi->get_pg_num();
float ratio = (float)objects_per_pg / (float)average_objects_per_pg;
if (g_conf->mon_pg_warn_max_object_skew > 0 &&
   ratio > g_conf->mon_pg_warn_max_object_skew) {
ostringstream ss;
ss << "pool " << name << " has many more objects per pg than average (too few pgs?)";
summary.push_back(make_pair(HEALTH_WARN, ss.str()));
if (detail) {
   ostringstream ss;
   ss << "pool " << name << " objects per pg ("
      << objects_per_pg << ") is more than " << ratio << " times cluster average ("
      << average_objects_per_pg << ")";
   detail->push_back(make_pair(HEALTH_WARN, ss.str()));
}
主要下面的几个限制条件
mon_pg_warn_min_objects = 10000 //总的对象超过10000
mon_pg_warn_min_pool_objects = 1000 //存储池对象超过1000
mon_pg_warn_max_object_skew = 10 //就是上面的存储池的平均对象与所有pg的平均值的倍数关系
解决问题
有三个方法解决这个警告的提示：
删除无用的存储池
如果集群中有一些不用的存储池，并且相对的pg数目还比较高，那么可以删除一些这样的存储池，从而降低mon_pg_warn_max_object_skew这个值，警告就会没有了
增加提示的pool的pg数目
有可能的情况就是，这个存储池的pg数目从一开始就不够，增加pg和pgp数目，同样降低了mon_pg_warn_max_object_skew这个值了
增加mon_pg_warn_max_object_skew的参数值
如果集群里面已经有足够多的pg了，再增加pg会不稳定，如果想去掉这个警告，就可以增加这个参数值，默认为10
总结
这个警告是比较的是存储池中的对象数目与整个集群的pg的平均对象数目的偏差，如果偏差太大就会发出警告
检查的步骤：
ceph health detail
ceph df
ceph osd dump | grep pool
mon_pg_warn_max_object_skew = 10.0
((objects/pg_num) in the affected pool)/(objects/pg_num in the entire system) >= 10.0 警告就会出现
变更记录

		自动登录	找回密码
密码			注册