ceph 分布式存储bucket对象回收lifecycle机制

admin

5 k2 c" P4 x3 w' F0 Z在radosgw中，可以针对bucket设置其对象的生命周期，当对象存活时间超出这个时间后，rgw会对其进行删除回收，这里设置的粒度目前是在bucket，也就是不能针对object设置其失效时间
! j5 O- z1 x1 c6 N6 Q/ i8 [0 qbucket的lc机制弊端是很明显的，在lc打开的情况下，进行lc的做法是遍历整个bucket的对象，找出超过存活时间的对象进行回收，这就意味着，如果单个bucket对象数量十分庞大，这个遍历就会极其消耗性能
  p2 C" N) Z: v5 N1 \& e开始
' E9 H* [* `! s7 p' {7 I首先，与lc相关的参数有
3 o) @6 \5 ^$ I9 C# [6 {  `2 T    "rgw_enable_lc_threads": "true",
- v) F& \/ v0 L- y2 f2 {) l    "rgw_lc_debug_interval": "-1",
    "rgw_lc_lock_max_time": "60",
    "rgw_lc_max_objs": "32",
    "rgw_lc_max_rules": "1000"
    "rgw_lifecycle_work_time": "00:00-06:00"
! M5 m7 M$ ?+ D9 u) t/ E- f默认情况下，bucket的属性标识中对象是永不过期的，对于某个bucket，手动设置其过期时间的接口，使用boto3的做法是
#pip install boto3 -i https://pypi.tuna.tsinghua.edu.cn/simple boto
from boto3.session import Session
' c8 S7 A  N. B6 Y# simport boto3
' B. [4 k4 Y/ }/ V& A1 Pimport uuid
! Z* K" r0 j: |0 }access_key = 'xxx'
, R4 u3 y+ ^  ~# x9 _; u  E0 Tsecret_key = 'xxxxx'
, [5 I3 ~& |1 |0 f, Yhost = 'http://ip:7480'
7 ?8 ?7 M* Z0 I3 {, X# [session = Session(access_key, secret_key)
buckets = ['bucket1']
s3_client = session.client('s3', endpoint_url=host)
- V+ f" `- O6 J( p1 N8 p# Vfor x in buckets:
3 K- y/ \4 `3 O! j% b    s3_client.put_bucket_lifecycle(
* U8 `4 K$ o7 m) u           Bucket=x,
1 m$ _& x8 U# w- u) m7 J3 Z3 ~1 c: P! L           LifecycleConfiguration={
               'Rules': [
                   {
, ~4 h  k7 [$ D& S. q+ B$ h" k/ S. n                       'Expiration':
  F  d! P; `/ R                           {
, H' n! j/ I* a. N: N( ~, a; l                               'Days': 1
                           },
                       'ID': str(uuid.uuid1()),
                       'Status': 'Enabled',
8 Q( C8 A2 \$ n! u& F                       'Prefix': ''
/ a5 r/ ^- u9 k# ?                   }
               ],
& }8 d6 X: A6 b9 V6 i           })
    print s3_client.get_bucket_lifecycle(Bucket=x)
7 B9 K8 h% W3 B# _" q8 U: W设置完成后，可以通过命令radosgw-admin lc list
查看到当前lc的队列
[store@server227 build]$ ./bin/radosgw-admin -k keyring -c ./ceph.conf  lc list
[
    {
        "bucket": ":cpp-bucket1:eace8f62-b901-47ed-b18f-fe2441f33830.4155.1",
' p0 S8 H1 q8 {2 F# e        "status": "COMPLETE"
* N! g. J; u1 U9 @6 `) d    }
  S0 j3 o0 b0 {9 Y2 E* V. p]
4 N9 Z+ a2 O6 T! ^$ k; K代码跟读
lc处理流程的入口是rgw/rgw_lc.cc
8 S8 a) _- Q! s- d8 Y9 F3 \* ~的*RGWLC::LCWorker::entry
函数
2 g& a! H; G1 E# Z6 s( ^void *RGWLC::LCWorker::entry() {
" N/ J% G5 ?3 E0 S$ ?* L4 o& G/ h& l  do {
/ \4 @! I/ }: c$ @" W3 |) ~0 e    utime_t start = ceph_clock_now();
    rgw_lc_debug_interval > 0时或者达到设定的回收时间则返回true
    if (should_work(start)) {
      ldout(cct, 2) << "life cycle: start" << dendl;
      int r = lc->process();
) s$ u8 ?8 r9 P      if (r < 0) {
. c" ^% k* K$ u- f        ldout(cct, 0) << "ERROR: do life cycle process() returned error r=" << r << dendl;
      }
      ldout(cct, 2) << "life cycle: stop" << dendl;
    }
3 b9 _  c- \4 K) v1 s/ J    if (lc->going_down())
      break;
    utime_t end = ceph_clock_now();
    int secs = schedule_next_start_time(start, end);
# n8 V" Q2 G. v7 ?. j, j    utime_t next;
    next.set_from_double(end + secs);
    ldout(cct, 5) << "schedule life cycle next start time: " << rgw_to_asctime(next) << dendl;
+ b. f+ ^8 a5 o1 C- F    lock.Lock();
+ O. n6 z3 |$ `    cond.WaitInterval(lock, utime_t(secs, 0));
  [6 k4 X/ i: {( m4 A) Z5 l# J    lock.Unlock();
  } while (!lc->going_down());
  return NULL;
. j1 _  {/ y; I8 j6 z6 Y2 [% w}
如果使能了lc的功能，lc的线程会在进行一次lc作业后，等待一段时间，再唤醒起来进行下一次的作业，这里函数的第三行，使用should_work
函数判断本次的lc是否进行，默认rgw_lc_debug_interval
, Q3 D6 W* ~$ E- L. [+ }4 R为-1，表示不进行lc的作业，此时会进一步判断是否在设定的回收时间内，如果不在，也不进行lc作业
" K* M6 Y9 |/ N/ t2 j  if (cct->_conf->rgw_lc_debug_interval > 0) {
      * We're debugging, so say we can run */
) T# ]: q$ G2 H6 F      return true;
/ o# j) ~: m  N  } else if ((bdt.tm_hour*60 + bdt.tm_min >= start_hour*60 + start_minute) &&
$ X0 Q- g* F. x             (bdt.tm_hour*60 + bdt.tm_min <= end_hour*60 + end_minute)) {
      return true;
  } else {
, R5 l) D. [7 z5 `. L# H4 p      return false;
: X7 G) @9 v; N- b& _+ q# A  }
跟着lc->process()
; k* e, d$ U4 ]' A9 v1 P看下
int RGWLC::process()
/ P6 L3 g5 |# _/ R8 b{
- f; n6 c+ R0 U! z+ I  int max_secs = cct->_conf->rgw_lc_lock_max_time;
* |2 Z( d- x7 K; I/ G/ a  const int start = ceph::util::generate_random_number(0, max_objs - 1);
  for (int i = 0; i < max_objs; i++) {
    int index = (i + start) % max_objs;
7 S" Z$ j# @6 l7 o& O    int ret = process(index, max_secs);
. ~1 O* k! K( D- V  y' R    if (ret < 0)
      return ret;
, f" ?! K1 X: s3 m& I; ^3 c  }
, x7 D1 e8 F0 H) l5 o% O  return 0;
}
: l7 u" J# Q: W" J- h; A函数比较简短，限制每次作业的时间为rgw_lc_lock_max_time
，默认是60s，结合实际测试看，这个时间并不准确，继续看
int RGWLC::process(int index, int max_lock_secs)
8 R& Z3 r! }4 V{
  根据max_lock_secs来判断一个执行周期的时间长度
6 j- R% I! Y$ D0 [& s# W2 t( K5 }  rados::cls::lock::Lock l(lc_index_lock_name);
3 o& O( S2 y" V& G  D  do {
! D" y. S8 e4 t1 U    utime_t now = ceph_clock_now();
    ......
    从这里开始，正式开始进行对象的lc操作，注意到这是在一个死循环里
8 p0 z1 O! N, k; t& H, Z) Q    ret = bucket_lc_process(entry.first);
    bucket_lc_post(index, max_lock_secs, entry, ret);
  }while(1);
8 b! T2 q8 k1 d, y6 K9 m8 I$ F. hlock的对象是lc_process
，这个函数中会多次请求osd进行lc的前期操作，包括查询head、set一些配置等，最重要的是检查bucket是否配置了lc，开始lifecycle到调用bucket_lc_process
前，花费时间为68ms
，接下来
9 E2 W; z: x5 @5 L  O/ \& Oint RGWLC::bucket_lc_process(string& shard_id)
{
9 J, v. C, f" D" i3 m    对bucket的对象进行回收的操作主要在这个函数中进行
0 n* j- h2 C. j# b+ m  i; k1 Z    ....
' x0 c/ D& J! C$ w0 v  W8 K    设定运行周期
    l.set_duration(time);
    ....
    do {
        objs.clear();
        list_op.params.marker = list_op.get_next_marker();
6 X3 @$ ^9 Q0 |" G# |' q& A        从bucket中选择1000个对象，使用的是bucket list方法
        ret = list_op.list_objects(1000, &objs, NULL, &is_truncated);
        if (ret < 0) {
          if (ret == (-ENOENT))
0 c% a4 v; ?2 o- e5 H- r            return 0;
- K; N. W/ g9 k0 e1 M2 g          ldout(cct, 0) << "ERROR: store->list_objects():" <<dendl;
          return ret;
        }
        
4 C( ]8 l/ l' E        bool is_expired;
        遍历这1000个对象，检查它们是否达到了过期时间
; ?$ G3 j9 U6 K, X        for (auto obj_iter = objs.begin(); obj_iter != objs.end(); ++obj_iter) {
: d8 b: {8 H0 e* ~; W* K          rgw_obj_key key(obj_iter->key);
. D" s- h: S. c; ]  k0 X          ....
* {/ g4 v, G+ |& C+ J: ?* t          if (prefix_iter->second.expiration_date != boost::none) {
            we have checked it before
, Y! |/ o5 {) k. n; O            is_expired = true;
5 c' \) Q& I5 _* q          } else {
            检查对象是否达到过期时间
9 X5 ?  p& f/ e( W9 Q$ f6 R3 {; |            is_expired = obj_has_expired(obj_iter->meta.mtime, prefix_iter->second.expiration);
4 V2 b0 f3 Y/ Z  v          }
5 i( g, d/ r) D! b( I6 ~, {          if (is_expired) {
! T0 l6 F1 x+ L            int ret = store->get_obj_state(&rctx, bucket_info, obj, &state, false);
            if (ret < 0) {
              return ret;
* a6 `% {- O4 j" Q            }
7 M! n! V; }" z- c# V- `8 H+ H            if (state->mtime != obj_iter->meta.mtime) {
              Check mtime again to avoid delete a recently update object as much as possible
: ?9 g- e! D# L) {1 N! U3 O              ldout(cct, 20) << __func__ << "() skipping removal: state->mtime " << state->mtime << " obj->mtime " << obj_iter->meta.mtime << dendl;
              continue;
* b& Y8 _3 K8 i5 G; _- A            }
            ret = remove_expired_obj(bucket_info, obj_iter->key, obj_iter->meta.owner, obj_iter->meta.owner_display_name, true);
            if (ret < 0) {
              ldout(cct, 0) << "ERROR: remove_expired_obj " << dendl;
            } else {
              ldout(cct, 2) << "DELETED:" << bucket_name << ":" << key << dendl;
2 _1 V) z' x' k/ s& D% Q7 X            }
            if (going_down())
              return 0;
          }
; ^. ^8 O$ B5 f        }
      } while (is_truncated);
}
在这个函数中，首先设定运行时间，线程通过bucket list的方式拿到1000个obj，获取使用按顺序的方式，遍历这1000个对象，如果该对象超过了过期时间，则删掉，关于循环的停止条件，is_truncated: if number of objects in the bucket is bigger than max, then truncated
3 @  p( U% B/ L8 q，即要求遍历完整个bucket
这部分耗时主要依赖于bucket内对象的数量，对象越多，则耗时肯定会越多
最后，针对对象中分片情况，进行了处理，在RGWLC::bucket_lc_process
3 n; A& N& G# v的最后
: U+ Y6 V( X3 G1 r& i  ret = handle_multipart_expiration(&target, prefix_map);
  return ret;
其中prefix_map
- s7 ]+ ~* t/ n  c$ C! G0 o表明了对象分片的相关信息
3 q* a2 b4 m. r( k. ^5 r( b7 K, E看下对于分片的处理逻辑
& q  k5 U5 w+ o3 {- yfor (auto prefix_iter = prefix_map.begin(); prefix_iter != prefix_map.end(); ++prefix_iter) {
1 ~: ]  ~! D- n      ......
2 H9 e2 m; q+ [, \  D    do {
( H3 v5 n% m% ^8 |      objs.clear();
      list_op.params.marker = list_op.get_next_marker();
      ret = list_op.list_objects(1000, &objs, NULL, &is_truncated);
6 [  W1 B  F. n8 F; s! d- ?# Y      ......
7 a. X3 k; ]7 t* \    } while(is_truncated);
8 S' ~( d1 E" A6 T% o) b  T}
可以看到，与对象删除一样，分片删除也是一次性取出1000个分片对象进行删除
$ ^; X$ H6 ?* N$ D逻辑总结
首先，在默认情况下，lc线程启动后，根据rgw_lc_debug_interval
及设定的rgw_lifecycle_work_time
) w' T" l! J0 c. W来判断要不要做本次的process，默认rgw_lc_debug_interval
4 _/ \4 p/ Y" f) c6 A% U为-1，表示不做本次lc，另外，在rgw_lc_debug_interval <=0
时，这个时间间隔单位将放大为天，原注释说的是We're in debug mode; Treat each rgw_lc_debug_interval seconds as a day
在决定要做本次lc后，根据rgw_lc_lock_max_time
/ h; E' ]4 O) I来判断每次lc的周期时间长度，然后每次从rgw_lc_max_objs
9 Y% B3 J1 {5 H9 m8 p8 C7 [中选取一个进行运行，增大这个值不会加快回收速度
1 t' N7 _5 T8 `9 F& a关于rgw_lc_max_objs
，它表示的是lc的处理线程的最大数量，它并不是并发的线程，因而增大这个值并不能加快回收速度，它类似于一个分组，bucket被设置lc规则后将被分配到某个lc中，例如，这个值设置为3，它启动回收的时候会随机启动lc.0、lc.1、lc.2中的其中一个，如果某个lc中有bucket则进行回收，测试发现，多个bucket都设置过期的情况下，单个rgw同一时间仅运行一个lc线程，因而设置自动回收速度其实是很慢的
/ t: I$ B* ~( N- M# N接下来，请求并设定一些基本的元数据信息，最重要的是判断bucket是否需要做lc，它检测bucket是否进行了lc的设置，如果设置才做，没有设置，默认是不做
然后，从指定的bucket中每次取出最多1000个有序对象，遍历这些对象并检查他们是否到达过期时间，过期则删除，直到bucket中所有的对象都遍历完成.
针对分片做了特殊处理，删除掉第一个片后，其余分片有专门的逻辑进行进一步的删除
环境中起2个rgw进行测试，发现2个rgw可以同时进行不同bucket的回收，因而，要加快回收速度，只能通过增加rgw的数量实现，单个rgw明显只能串行单线程进行单bucket的回收，不过，因为回收bucket涉及到bucket list，过快的回收速度也可能引发性能问题，需要权衡
' d' \0 s  I: d

admin

16 Operating Ceph Services REPORT DOCUMENTATION BUG#EDIT SOURCE

  a8 ~5 [9 T6 p1 p, h/ q6 R
( `* B" Q( p' O16.1 Operating Ceph Cluster Related Services Using systemd16.2 Restarting Ceph Services Using DeepSea16.3 Shutdown and Start of the Whole Ceph Cluster
: ]2 U, T+ u9 S  d2 z0 |+ b( @# X! ]
- I9 W7 X/ c) `2 e* {" D

You can operate Ceph services either using systemd or using DeepSea.

16.1 Operating Ceph Cluster Related Services Using systemd REPORT DOCUMENTATION BUG#EDIT SOURCE

, N) q6 K; U) K2 _

Use the systemctl command to operate all Ceph related services. The operation takes place on the node you are currently logged in to. You need to have root privileges to be able to operate on Ceph services.

16.1.1 Starting, Stopping, and Restarting Services Using Targets REPORT DOCUMENTATION BUG#EDIT SOURCE
/ B+ W: J3 F! O- O8 z$ D- ]8 r5 _. T
' e% G- r1 W& _$ G( B/ F, \1 [
4 [5 |% g, {7 j4 }  O

To simplify starting, stopping, and restarting all the services of a particular type (for example all Ceph services, or all MONs, or all OSDs) on a node, Ceph provides the following systemd unit files:

cephadm@adm > ls /usr/lib/systemd/system/ceph*.targetceph.targetceph-osd.targetceph-mon.targetceph-mgr.targetceph-mds.targetceph-radosgw.targetceph-rbd-mirror.targetCOPY

. y* @# p5 T& t6 D) c& D8 |; Q6 l) c; k

To start/stop/restart all Ceph services on the node, run:

root # systemctl start ceph.targetroot # systemctl stop ceph.targetroot # systemctl restart ceph.targetCOPY

% m( g, o" }/ `/ W# y$ R

To start/stop/restart all OSDs on the node, run:

root # systemctl start ceph-osd.targetroot # systemctl stop ceph-osd.targetroot # systemctl restart ceph-osd.targetCOPY

: G. m3 n. l: K3 Z( E# @3 X9 R

Commands for the other targets are analogous.

16.1.2 Starting, Stopping, and Restarting Individual Services REPORT DOCUMENTATION BUG#EDIT SOURCE
4 {( G. e# r1 n

You can operate individual services using the following parameterized systemd unit files:

ceph-osd@.serviceceph-mon@.serviceceph-mds@.serviceceph-mgr@.serviceceph-radosgw@.serviceceph-rbd-mirror@.serviceCOPY

' F& a/ u% Q4 H; l

To use these commands, you first need to identify the name of the service you want to operate. See Section 16.1.3, “Identifying Individual Services” to learn more about services identification.

To start, stop or restart the osd.1 service, run:

root # systemctl start ceph-osd@1.serviceroot # systemctl stop ceph-osd@1.serviceroot # systemctl restart ceph-osd@1.serviceCOPY

Commands for the other service types are analogous.

16.1.3 Identifying Individual Services REPORT DOCUMENTATION BUG#EDIT SOURCE
- V6 g% K! w) V/ `# S/ d: B6 u
( {8 K+ g9 W& H, F/ Y. T

You can find out the names/numbers of a particular type of service in several ways. The following commands provide results for ceph* services. You can run them on any node of the Ceph cluster.

To list all (even inactive) services of type ceph*, run:

root # systemctl list-units --all --type=service ceph*COPY

To list only the inactive services, run:

root # systemctl list-units --all --state=inactive --type=service ceph*COPY

9 U  R8 Y5 L* u/ j, o# ~& j

You can also use salt to query services across multiple nodes:

root@master # salt TARGET cmd.shell \ "systemctl list-units --all --type=service ceph* | sed -e '/^$/,$ d'"COPY

Query storage nodes only:

root@master # salt -I 'roles:storage' cmd.shell \ 'systemctl list-units --all --type=service ceph*'COPY

, W) O9 [5 i6 j! J/ q
+ d6 ~! h& R5 q( R16.1.4 Service Status REPORT DOCUMENTATION BUG#EDIT SOURCE


+ o* j9 G' a2 M( S) J

You can query systemd for the status of services. For example:

root # systemctl status ceph-osd@1.serviceroot # systemctl status ceph-mon@HOSTNAME.serviceCOPY

, ]) ]. q- h, G) R" g

Replace HOSTNAME with the host name the daemon is running on.

If you do not know the exact name/number of the service, see Section 16.1.3, “Identifying Individual Services”.

' @) h% l! i) k' N$ P
3 m4 F$ z6 y; a2 `/ _# @3 [16.2 Restarting Ceph Services Using DeepSea REPORT DOCUMENTATION BUG#EDIT SOURCE


+ O9 F  D* j! y& }2 ^

After applying updates to the cluster nodes, the affected Ceph related services need to be restarted. Normally, restarts are performed automatically by DeepSea. This section describes how to restart the services manually.

Tip: Watching the Restart

The process of restarting the cluster may take some time. You can watch the events by using the Salt event bus by running:

root@master # salt-run state.event pretty=TrueCOPY

8 s0 B1 L+ z7 R. H  d  j

Another command to monitor active jobs is

root@master # salt-run jobs.activeCOPY

9 U! P$ a; G6 R/ }* L4 C" b
8 |2 E3 U: l6 h6 v; ^7 v& S16.2.1 Restarting All Services REPORT DOCUMENTATION BUG#EDIT SOURCE
. |" Q* y* e- W4 w9 ~/ T" ]4 A

Warning: Interruption of Services

If Ceph related services—specifically iSCSI or NFS Ganesha—are configured as single points of access with no High Availability setup, restarting them will result in their temporary outage as viewed from the client side.

; Y3 c+ t& O4 ?3 wTip: Samba Not Managed by DeepSea

Because DeepSea and the Ceph Dashboard do not currently support Samba deployments, you need to manage Samba related services manually. For more details, see Chapter 29, Exporting Ceph Data via Samba.

To restart all services on the cluster, run the following command:

root@master # salt-run state.orch ceph.restartCOPY

• For DeepSea prior to version 0.8.4, the Metadata Server, iSCSI Gateway, Object Gateway, and NFS Ganesha services restart in parallel.
• For DeepSea 0.8.4 and newer, all roles you have configured restart in the following order: Ceph Monitor, Ceph Manager, Ceph OSD, Metadata Server, Object Gateway, iSCSI Gateway, NFS Ganesha. To keep the downtime low and to find potential issues as early as possible, nodes are restarted sequentially. For example, only one monitoring node is restarted at a time.
  
  ; T" y' P: w/ `3 J2 J

/ m1 i5 W: J% v

The command waits for the cluster to recover if the cluster is in a degraded, unhealthy state.

4 H7 I: a+ r( ~; o+ c16.2.2 Restarting Specific Services REPORT DOCUMENTATION BUG#EDIT SOURCE
* `8 k6 }" _9 p4 A9 c2 @( p' v

/ U; b' Q( P8 |7 ]

To restart a specific service on the cluster, run:

root@master # salt-run state.orch ceph.restart.service_nameCOPY

For example, to restart all Object Gateways, run:

root@master # salt-run state.orch ceph.restart.rgwCOPY

You can use the following targets:

root@master # salt-run state.orch ceph.restart.monCOPY

root@master # salt-run state.orch ceph.restart.mgrCOPY

- k3 N$ m' l% m" w7 z: C

root@master # salt-run state.orch ceph.restart.osdCOPY

: y" |9 S7 J% V( W$ c

root@master # salt-run state.orch ceph.restart.mdsCOPY

( Y1 Z3 ]( w; n6 @2 o- ?$ `1 m) {1 J. P

root@master # salt-run state.orch ceph.restart.rgwCOPY

: Q, O& [/ D& V5 c* R

root@master # salt-run state.orch ceph.restart.igwCOPY

4 y# `. x7 v$ h4 k2 R

root@master # salt-run state.orch ceph.restart.ganeshaCOPY

  j; I4 a, W4 r' x# _

The restart orchestration checks if the installated binary is newer than the current one, or if configuration changes exist for this daemon and only restarts in those cases. If you run the above command and nothing happens, this is due to these conditions. See Section 16.1.2, “Starting, Stopping, and Restarting Individual Services” for more information.

16.3 Shutdown and Start of the Whole Ceph Cluster REPORT DOCUMENTATION BUG#EDIT SOURCE
+ ?$ e; f+ j( j! e5 D& {  p7 _7 v

There are occasions when you need to stop all Ceph related services in the cluster in the recommended order, and then be able to simply start them again. For example, in case of a planned power outage.

PROCEDURE 16.1: SHUTTING DOWN THE WHOLE CEPH CLUSTER REPORT DOCUMENTATION BUG#

• Shut down or disconnect any clients accessing the cluster.
• To prevent CRUSH from automatically rebalancing the cluster, set the cluster to noout:
  root@master # ceph osd set nooutCOPY
  
  , O1 h) A5 y7 q- [8 U0 T* ?! h1 B
• Disable safety measures and run the ceph.shutdown runner:
  root@master # salt-run disengage.safetyroot@master # salt-run state.orch ceph.shutdownCOPY
  
  
• Power off all cluster nodes:
  root@master # salt -C 'G@deepsea:*' cmd.run "shutdown -h"COPY
  
  
  

: @  k. J: p; [' ?4 x+ DPROCEDURE 16.2: STARTING THE WHOLE CEPH CLUSTER REPORT DOCUMENTATION BUG#
4 A& Q0 u" ^( s

• Power on the Admin Node.
• Power on the Ceph Monitor nodes.
• Power on the Ceph OSD nodes.
• Unset the previously set noout flag:
  root@master # ceph osd unset nooutCOPY
  
  - o' |& M$ _6 e" [+ T! p# [9 M( \
• Power on all configured gateways.
• Power on or connect cluster clients.
  
  

  m( J: M7 L; a. t9 W8 l+ j( n# U
. M3 P1 ^1 a' O# x

admin

16 Operating Ceph Services REPORT DOCUMENTATION BUG#EDIT SOURCE
1 `8 U! W+ H0 s5 s. l16.1 Operating Ceph Cluster Related Services Using systemd
16.2 Restarting Ceph Services Using DeepSea
16.3 Shutdown and Start of the Whole Ceph Cluster
! q: I. N& L: O  g* C! FYou can operate Ceph services either using systemd or using DeepSea.

) i1 L; C! d; i( b- m16.1 Operating Ceph Cluster Related Services Using systemd REPORT DOCUMENTATION BUG#EDIT SOURCE
Use the systemctl command to operate all Ceph related services. The operation takes place on the node you are currently logged in to. You need to have root privileges to be able to operate on Ceph services.

16.1.1 Starting, Stopping, and Restarting Services Using Targets REPORT DOCUMENTATION BUG#EDIT SOURCE
To simplify starting, stopping, and restarting all the services of a particular type (for example all Ceph services, or all MONs, or all OSDs) on a node, Ceph provides the following systemd unit files:
( T+ X- P. @. n. ?9 f- b  M
ls /usr/lib/systemd/system/ceph*.target
ceph.target
; H" k3 b- |9 B: rceph-osd.target
/ t! H  m' y3 X1 _% F2 ?ceph-mon.target
; d/ @6 e1 f- C( k" Iceph-mgr.target
ceph-mds.target
( G8 A/ L6 G6 U5 ~% D, Qceph-radosgw.target
6 ~/ _: T  M, m1 j7 ^' N/ jceph-rbd-mirror.target
COPY
To start/stop/restart all Ceph services on the node, run:

systemctl start ceph.target
! l& d" O$ L) g7 r+ jsystemctl stop ceph.target
  k. _' Q8 \3 Jsystemctl restart ceph.target
1 O) i, z8 N  aCOPY
To start/stop/restart all OSDs on the node, run:
* l% L! p3 R5 q' _3 j
systemctl start ceph-osd.target
systemctl stop ceph-osd.target
2 P: ]) Z$ U6 nsystemctl restart ceph-osd.target
COPY
Commands for the other targets are analogous.

& r3 ?2 y7 Q* ~- i16.1.2 Starting, Stopping, and Restarting Individual Services REPORT DOCUMENTATION BUG#EDIT SOURCE
You can operate individual services using the following parameterized systemd unit files:

ceph-osd@.service
+ Z. @  S. O" g$ vceph-mon@.service
ceph-mds@.service
ceph-mgr@.service
5 G' \' Z9 i. s; P% }3 d2 p- [, Cceph-radosgw@.service
) n+ k% ~# P2 {6 u: ?, x% Oceph-rbd-mirror@.service
4 K& B% W  e# G; T& P6 rCOPY
To use these commands, you first need to identify the name of the service you want to operate. See Section 16.1.3, “Identifying Individual Services” to learn more about services identification.
, m5 p& q5 [0 K9 g# x; d: _
0 _6 d; e: T- N+ h5 O. Z/ s& \To start, stop or restart the osd.1 service, run:
6 E) A0 v: {+ K* K# q
5 m- [3 N" l$ y! x- ^systemctl start ceph-osd@1.service
0 y. k$ G8 J& fsystemctl stop ceph-osd@1.service
systemctl restart ceph-osd@1.service
COPY
Commands for the other service types are analogous.
/ i# O1 ?3 ^6 \  K+ Z6 O7 K1 n0 ^9 v# R
' D5 m% p  K+ g6 S; l6 W16.1.3 Identifying Individual Services REPORT DOCUMENTATION BUG#EDIT SOURCE
You can find out the names/numbers of a particular type of service in several ways. The following commands provide results for ceph* services. You can run them on any node of the Ceph cluster.
0 V2 H# f3 }3 E# s' G/ `# y+ V2 A
1 h, |( M3 p3 W0 P' TTo list all (even inactive) services of type ceph*, run:

0 F0 ^: w! h4 nsystemctl list-units --all --type=service ceph*
: j, N2 @5 M" V! ?9 @9 [+ GCOPY
4 \. `! m$ s) k* j' ], V9 VTo list only the inactive services, run:

systemctl list-units --all --state=inactive --type=service ceph*
0 l3 g1 |  T2 {, _3 [- m; vCOPY
You can also use salt to query services across multiple nodes:

salt TARGET cmd.shell \
# z& _: ~( [3 } "systemctl list-units --all --type=service ceph* | sed -e '/^$/,$ d'"
COPY
$ |0 k) M8 j6 C- f, g* w' Z% |Query storage nodes only:
- C7 l7 W: [. h9 h
$ z1 S% G0 Y, h7 D7 H* zsalt -I 'roles:storage' cmd.shell \
'systemctl list-units --all --type=service ceph*'
COPY
: K2 ]- r/ S# C$ U! o  Q3 a; g! r8 w16.1.4 Service Status REPORT DOCUMENTATION BUG#EDIT SOURCE
/ q1 Y8 ]# Z8 Z" `0 f! KYou can query systemd for the status of services. For example:

systemctl status ceph-osd@1.service
systemctl status ceph-mon@HOSTNAME.service
COPY
1 i' u  {0 H+ [9 XReplace HOSTNAME with the host name the daemon is running on.

' [: a( o+ J9 J: c! k* FIf you do not know the exact name/number of the service, see Section 16.1.3, “Identifying Individual Services”.

" J, ^" s. D# `1 t* Z0 D* Z16.2 Restarting Ceph Services Using DeepSea REPORT DOCUMENTATION BUG#EDIT SOURCE
$ C# D" j7 ]9 t; F7 ^- w* y0 HAfter applying updates to the cluster nodes, the affected Ceph related services need to be restarted. Normally, restarts are performed automatically by DeepSea. This section describes how to restart the services manually.

+ ~) y; E$ j1 l" s, N+ L7 [TipTip: Watching the Restart
The process of restarting the cluster may take some time. You can watch the events by using the Salt event bus by running:
; q. n' c7 j  x4 @' v
+ S) Q/ {) c8 s* a9 O, I' qsalt-run state.event pretty=True
& s& H+ m) F, B- sCOPY
' F/ i( y$ y& g: @# [4 M) ZAnother command to monitor active jobs is

; F( Y9 }+ ?; vsalt-run jobs.active
9 X! E& ~/ E8 w) j  v" }COPY
16.2.1 Restarting All Services REPORT DOCUMENTATION BUG#EDIT SOURCE
, \. P' X3 |2 [% G/ qWarningWarning: Interruption of Services
1 _, Z. G2 X% c% O( {6 wIf Ceph related services—specifically iSCSI or NFS Ganesha—are configured as single points of access with no High Availability setup, restarting them will result in their temporary outage as viewed from the client side.

: t) g# _- t" v+ m! s% e) PTipTip: Samba Not Managed by DeepSea
Because DeepSea and the Ceph Dashboard do not currently support Samba deployments, you need to manage Samba related services manually. For more details, see Chapter 29, Exporting Ceph Data via Samba.
% U0 [3 a: U: ^" `8 u% f7 k+ F4 o
To restart all services on the cluster, run the following command:
$ X$ g* x/ b1 s- v
salt-run state.orch ceph.restart
COPY
For DeepSea prior to version 0.8.4, the Metadata Server, iSCSI Gateway, Object Gateway, and NFS Ganesha services restart in parallel.

For DeepSea 0.8.4 and newer, all roles you have configured restart in the following order: Ceph Monitor, Ceph Manager, Ceph OSD, Metadata Server, Object Gateway, iSCSI Gateway, NFS Ganesha. To keep the downtime low and to find potential issues as early as possible, nodes are restarted sequentially. For example, only one monitoring node is restarted at a time.

The command waits for the cluster to recover if the cluster is in a degraded, unhealthy state.

, ~) B3 t2 j! a1 X16.2.2 Restarting Specific Services REPORT DOCUMENTATION BUG#EDIT SOURCE
% m# [' y9 @- R5 ]* PTo restart a specific service on the cluster, run:

salt-run state.orch ceph.restart.service_name
COPY
+ R0 O( x' o) G4 UFor example, to restart all Object Gateways, run:

& y$ j2 g: g4 csalt-run state.orch ceph.restart.rgw
$ U3 Y6 |4 ^  u3 {; l. F1 ECOPY
9 a7 O8 B- t- JYou can use the following targets:
7 {6 ^9 J$ ^9 ?/ B9 E
$ ^0 ~4 H, E6 v- l0 P, ]salt-run state.orch ceph.restart.mon
COPY
! n0 {* c8 r8 N9 Isalt-run state.orch ceph.restart.mgr
& a, u2 y5 {# \0 t7 M( z" iCOPY
salt-run state.orch ceph.restart.osd
$ e+ t$ R, ^4 ^COPY
9 c6 j+ D' a( m+ H  c  Bsalt-run state.orch ceph.restart.mds
COPY
salt-run state.orch ceph.restart.rgw
# n0 h3 K: @* ^! K% T1 y3 O. l0 {COPY
salt-run state.orch ceph.restart.igw
COPY
( q( n' J8 p1 zsalt-run state.orch ceph.restart.ganesha
" u. l9 g" O& a) t* j1 p: f  LCOPY
: U- _; g! b, ?# K6 [The restart orchestration checks if the installated binary is newer than the current one, or if configuration changes exist for this daemon and only restarts in those cases. If you run the above command and nothing happens, this is due to these conditions. See Section 16.1.2, “Starting, Stopping, and Restarting Individual Services” for more information.

16.3 Shutdown and Start of the Whole Ceph Cluster REPORT DOCUMENTATION BUG#EDIT SOURCE
' ?6 z3 _" O& h/ QThere are occasions when you need to stop all Ceph related services in the cluster in the recommended order, and then be able to simply start them again. For example, in case of a planned power outage.

PROCEDURE 16.1: SHUTTING DOWN THE WHOLE CEPH CLUSTER REPORT DOCUMENTATION BUG#
( z2 b3 Y% F2 NShut down or disconnect any clients accessing the cluster.
! n: M9 r; x0 y0 X
+ E+ X7 m0 E* ETo prevent CRUSH from automatically rebalancing the cluster, set the cluster to noout:
* X! f* w; j. {) Z1 W3 o! M% S
ceph osd set noout
3 _5 w) d; M( A1 e# }COPY
4 H5 X, K, X9 l4 g7 t- I3 s& EDisable safety measures and run the ceph.shutdown runner:

salt-run disengage.safety
salt-run state.orch ceph.shutdown
COPY
( W8 t1 b# x2 N  o. yPower off all cluster nodes:
* a. O9 v% F/ x; m6 g
salt -C 'G@deepsea:*' cmd.run "shutdown -h"
COPY
3 X% w2 `) e" x! k  vPROCEDURE 16.2: STARTING THE WHOLE CEPH CLUSTER REPORT DOCUMENTATION BUG#
# r8 q4 x2 `. u) {  t2 Z0 u4 PPower on the Admin Node.
8 h! X. {, Y8 S, E3 s
Power on the Ceph Monitor nodes.
+ f/ F6 m* w9 U2 F+ \) q9 o* F  [
Power on the Ceph OSD nodes.
8 T; v6 c( W" B
Unset the previously set noout flag:

ceph osd unset noout
COPY
Power on all configured gateways.

. |( m) D8 U' [( zPower on or connect cluster clients.