Ceph 基础篇 -对象存储使用

admin · 发表于 2022-2-9 15:15:20

Ceph 对象网关主要是由 radosgw 接受用户请求，然后与后端librados交互;它主要起到承上启下的功能，对外它能兼容两种接口，一种是S3，另一种是Openstack Swift 接口，这两种接口，都有各自的用户认证机制，所以 Ceph 也提供了一套独立的用户管理机制，这套用户管理机制能同时兼容S3和Swift接口，最终数据会落在 OSD 上面，无论使用 S3 还是 Swift ，它们落到 OSD 上面，都是相同的名称空间;所以你使用 S3 存储的，也可以使用 Swift 进行访问。这是对象存储的一个基本架构，所以如果要使用它，我们需要部署radosgw，才能访问到集群，默认是没有安装的。

bucket 是什么?可以理解为装载对象的容器，它的后端是无限可扩展的存储空间，并且具备安全可靠性。Ceph 对象存储它后端是借助 ceph rados 实现数据容灾的机制，那么他能提供哪些功能呢?

基本功能
RESTful Interface # RESTful风格的接口，实现上传下载及管理功能;
S3- and Swift-compliant APIs # 提供两种风格的 API 接口，兼容 S3 和 Swift;
S3-style subdomains
Unified S3/Swift namespace # 扁平化、统一的的S3/Swift的名称空间;
User management # 为了安全性，也提供用户管理，可以限制对象是可以公共访问，还是授权访问;
Usage tracking # 追踪用户使用情况 rados df
Striped objects # 支持分片上传
Cloud solution integration # 支持云解决方案集成
Multi-site deployment # 支持多站点部署
Multi-site replication # 支持多站点复制
安装 RGW
1. 软件安装
复制
[root@ceph-node01 ~]# rpm -qa |grep ceph
ceph-base-14.2.11-0.el7.x86_64
ceph-mon-14.2.11-0.el7.x86_64
ceph-deploy-2.0.1-0.noarch
python-ceph-argparse-14.2.11-0.el7.x86_64
libcephfs2-14.2.11-0.el7.x86_64
ceph-common-14.2.11-0.el7.x86_64
ceph-selinux-14.2.11-0.el7.x86_64
ceph-mds-14.2.11-0.el7.x86_64
ceph-14.2.11-0.el7.x86_64
python-cephfs-14.2.11-0.el7.x86_64
ceph-osd-14.2.11-0.el7.x86_64
ceph-mgr-14.2.11-0.el7.x86_64
ceph-radosgw-14.2.11-0.el7.x86_64 # 直接使用 yum 安装即可
[root@ceph-node01 ~]#

2. 启动服务，默认启动在7480端口
复制
[root@ compute01 ceph-deploy]# ceph-deploy rgw create compute01

3. 服务检测
复制
[root@ compute01 ceph-deploy]# systemctl status ceph-radosgw@rgw.compute01
● ceph-radosgw@rgw. compute01.service - Ceph rados gateway
Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled; vendor preset: disabled)
Active: active (running) since 一 2020-10-05 20:34:36 EDT; 14s ago
Main PID: 33574 (radosgw)
CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw@rgw.compute01.service
         └─33574 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-node01 --setuser ceph --setgroup ceph

10月 05 20:34:36  compute01 systemd[1]: Started Ceph rados gateway.
10月 05 20:34:36  compute01 systemd[1]: [/usr/lib/systemd/system/ceph-radosgw@.service:13] Unknown lvalue 'LockPersonality' in section 'Service'
10月 05 20:34:36  compute01 systemd[1]: [/usr/lib/systemd/system/ceph-radosgw@.service:14] Unknown lvalue 'MemoryDenyWriteExecute' in ...Service'
10月 05 20:34:36  compute01 systemd[1]: [/usr/lib/systemd/system/ceph-radosgw@.service:17] Unknown lvalue 'ProtectControlGroups' in se...Service'
10月 05 20:34:36  compute01 systemd[1]: [/usr/lib/systemd/system/ceph-radosgw@.service:19] Unknown lvalue 'ProtectKernelModules' in se...Service'
10月 05 20:34:36  compute01 systemd[1]: [/usr/lib/systemd/system/ceph-radosgw@.service:20] Unknown lvalue 'ProtectKernelTunables' in s...Service'
Hint: Some lines were ellipsized, use -l to show in full.
[root@ceph-node01 ceph-deploy]# netstat -antp |grep 7480
tcp 0 0 0.0.0.0:7480 0.0.0.0:* LISTEN 33574/radosgw
[root@ceph-node01 ceph-deploy]# ceph -s
  cluster:
id:    a6d0359e-8295-4e62-8620-54273462e79f
health: HEALTH_OK

  services:
mon: 3 daemons, quorum compute01,compute02,compute03 (age 12d)
mgr: compute03(active, since 4w), standbys: compute01, compute02
osd: 6 osds: 6 up (since 12d), 6 in (since 3w)
rgw: 3 daemons active (compute01, compute02, compute03)

  task status:

  data:
pools: 10 pools, 1568 pgs
objects: 65.20k objects, 258 GiB
usage: 754 GiB used, 4.5 TiB / 5.2 TiB avail
pgs:    1568 active+clean

  io:
client: 0 B/s rd, 15 MiB/s wr, 10 op/s rd, 51 op/s wr

4. 首次访问服务
复制
[root@compute01 ~]# curl http://compute01:9000/
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
[root@compute01 ~]#

提示错误，使用了 anonymous 匿名用户请求所有列表，到这里说明安装完毕了;

5. 修改 RGW 的默认端口 7480 到 9000
复制
[root@ceph-node01 ceph-deploy]# vim ceph.conf
。。。

[client.rgw.compute01]
host = compute01
keyring = /var/lib/ceph/radosgw/ceph-rgw.compute01/keyring
rgw frontends = "civetweb port=9000"
rgw print continue = false

[client.rgw.compute02]
host = compute02
rgw frontends = "civetweb port=9000"
keyring = /var/lib/ceph/radosgw/ceph-rgw.compute02/keyring
rgw print continue = false

[client.rgw.compute03]
host = compute03
rgw frontends = "civetweb port=9000"
keyring = /var/lib/ceph/radosgw/ceph-rgw.compute03/keyring
rgw print continue = false

修改配置文件ceph.conf，为什么修改这个文件呢?因为后面添加节点时，默认是copy的这个配置文件，修改这个配置文件，可以确保集群的唯一性，下面推送配置文件到所有节点;

复制
[root@ceph-node01 ceph-deploy]# ceph-deploy --overwrite-conf config push compute01 compute02 compute03

--overwrite-conf 注意需要使用这个选项，否则提示无法覆盖;

6. 重启服务
复制
[root@compute01 ~]# systemctl restart ceph-radosgw@rgw.compute01.service
[root@compute01 ~]# netstat -antp |grep 9000|grep radosgw
tcp       0    0 0.0.0.0:9000          0.0.0.0:*             LISTEN    2758818/radosgw
[root@compute01 ~]#
7. 验证 80 端口
复制
[root@ceph-node01 ceph-deploy]# curl http://ceph-node01/
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>[root@ceph-node01 ceph-deploy]#

直此，RGW 服务部署完成;

使用 S3 访问 RGW
1. 创建 s3 的兼容用户
复制
[root@ceph-node01 ceph-deploy]# radosgw-admin user create --uid ceph-s3-user --display-name "Ceph S3 User Demo"
{
"user_id": "ceph-s3-user",
"display_name": "Ceph S3 User Demo",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
      {
         "user": "ceph-s3-user",
         "access_key": "V3J9L4M1WKV5O5ECAKPU",
         "secret_key": "f5LqLVYOVNu38cuQwi0jXC2ZTboCSJDmdvB8oeYw"
      }
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
      "enabled": false,
      "check_on_raw": false,
      "max_size": -1,
      "max_size_kb": 0,
      "max_objects": -1
},
"user_quota": {
      "enabled": false,
      "check_on_raw": false,
      "max_size": -1,
      "max_size_kb": 0,
      "max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}

[root@ceph-node01 ceph-deploy]#

注意上面的 access_key 与 secret_key 很重要，要记下来，以备后用; 不记录也没有关系，我们可以使用以下命令查看;

复制
[root@ceph-node01 ceph-deploy]# radosgw-admin user info --uid ceph-s3-user
1.
2. 使用 Ceph SDK 访问 Ceph 集群
官方SDK 使用说明：https://docs.ceph.com/en/latest/ ... g-s3-api-extensions

复制
[root@compute01 ~]# cat s3client.py
import boto
import boto.s3.connection
access_key = '8Y28X33O2YJKI3H6B1WJ'
secret_key = 'ghRR17Zxs2mZrhG8DGwJqTxeTQh64wofOsGB6C5Y'

conn = boto.connect_s3(
      aws_access_key_id = access_key,
      aws_secret_access_key = secret_key,
      host = 'compute01', port = 9000,
      is_secure=False, # uncomment if you are not using ssl
      calling_format = boto.s3.connection.OrdinaryCallingFormat(),
      )
bucket = conn.create_bucket("ceph-s3-bucket")
for bucket in conn.get_all_buckets():
      print "{name}\t{created}".format(
            name = bucket.name,
            created = bucket.creation_date,
      )

[root@compute01 ~]# python s3client.py
ceph-s3-bucket       2022-02-09T06:35:13.546Z
安装完rgw后，会自动创建3个pool，一个是rgw.control、rgw.meta、rgw.log，当我们创建bucket后，还会创建一个rgw.buckets.index pool池;

复制
[root@compute01 ~]# ceph osd lspools
5 images
6 volumes
7 backups
8 vms
9 ceph-test
10 .rgw.root
11 default.rgw.control
12 default.rgw.meta
13 default.rgw.log
14 default.rgw.buckets.index

3. 使用命令行方式操作 rgw
安装命令行工具

复制
[root@compute01 ~]# yum  install s3cmd -y
1.
配置命令行工具

[root@compute01 ~]# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: 8Y28X33O2YJKI3H6B1WJ
Secret Key: ghRR17Zxs2mZrhG8DGwJqTxeTQh64wofOsGB6C5Y
Default Region [US]:

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: 192.168.0.75:9000

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: 192.168.0.75:9000/%(bucket)s

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: no

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

New settings:
  Access Key: 8Y28X33O2YJKI3H6B1WJ
  Secret Key: ghRR17Zxs2mZrhG8DGwJqTxeTQh64wofOsGB6C5Y
  Default Region: US
  S3 Endpoint: 192.168.0.75:9000
  DNS-style bucket+hostname:port template for accessing a bucket: 192.168.0.75:9000/%(bucket)s
  Encryption password:
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

简单使用命令行工具

[root@compute01 ~]# s3cmd ls
2022-02-09 06:35  s3://ceph-s3-bucket

[root@ceph-node01 ~]# s3cmd mb s3://s3cmd-demo
ERROR: S3 error: 403 (SignatureDoesNotMatch)
[root@compute01 ~]# s3cmd mb s3://s3cmd-demo
ERROR: S3 error: 403 (SignatureDoesNotMatch)

.这是需要修改版本，启用v2版本即可

复制
[root@compute01 ~]# sed -i '/signature_v2/s/False/True/g' /root/.s3cfg

再次创建

[root@compute01 ~]# s3cmd mb s3://s3cmd-demo
Bucket 's3://s3cmd-demo/' created

上传单个文件

[root@compute01 ~]# s3cmd put /etc/fstab s3://s3cmd-demo/fatab-demo
upload: '/etc/fstab' -> 's3://s3cmd-demo/fatab-demo'  [1 of 1]
465 of 465 100% in 0s 67.71 KB/s
465 of 465 100% in 0s 67.71 KB/s  failed
WARNING: Upload failed: /fatab-demo (timed out)
WARNING: Waiting 3 sec...
upload: '/etc/fstab' -> 's3://s3cmd-demo/fatab-demo'  [1 of 1]
465 of 465 100% in 0s 361.83 KB/s

出现错误 ERROR: S3 error: 416 (InvalidRange)，原因是上传object对象的时候，需要创建 pool 存储数据，创建 pool 是需要 pg，当 pg 数量不够的情况下，可以将现有的 pg 数量调小，或者修改配置文件调整参数，然后重启 mon 进程，调整方法有三种：

调整 pg_num 和 pgp_num ，默认参数值均为8;
调整 mon_max_pg_per_osd 参数，默认是 300，适当增大;当每个 OSD 中的 PG数量超过这个参数值时，就会报错;https://www.suse.com/support/kb/doc/?id=000019402
增加更多的 OSD 进来;
我们采用第2种方法：

[root@compute01 ~]# cat /etc/ceph/ceph.conf
[global]
fsid = a6d0359e-8295-4e62-8620-54273462e79f
mon_initial_members = compute01,compute02,compute03
mon_host = 192.168.0.75,192.168.0.77,192.168.0.79
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = 192.168.0.0/24
cluster_network = 10.10.10.0/24
rbd_default_features = 3
mon_clock_drift_allowed = 2
mon_clock_drift_warn_backoff = 30
osd_pool_default_size = 3
osd_pool_default_min_size = 2
mon_allow_pool_delete = true
mon_max_pg_per_osd = 1000
osd_crush_update_on_start = false
mon_pg_warn_max_object_skew = 50

[osd]
osd_heartbeat_grace = 20
osd_heartbeat_interval = 5

[client.rgw.compute01]
host = compute01
keyring = /var/lib/ceph/radosgw/ceph-rgw.compute01/keyring
rgw frontends = "civetweb port=9000"
rgw print continue = false

[client.rgw.compute02]
host = compute02
rgw frontends = "civetweb port=9000"
keyring = /var/lib/ceph/radosgw/ceph-rgw.compute02/keyring
rgw print continue = false

[client.rgw.compute03]
host = compute03
rgw frontends = "civetweb port=9000"
keyring = /var/lib/ceph/radosgw/ceph-rgw.compute03/keyring
rgw print continue = false

重启下 monitor daemon 进程

复制
[root@compute01 ~]#  systemctl restart ceph-mon@compute01.service
[root@compute02 ~]# systemctl restart ceph-mon@compute02.service
[root@compute03 radosgw]# systemctl restart ceph-mon@compute03.service

再次上传文件测试

[root@compute01 ~]# s3cmd put /etc/fstab s3://s3cmd-demo/
upload: '/etc/fstab' -> 's3://s3cmd-demo/fstab'  [1 of 1]
465 of 465 100% in 0s 66.26 KB/s

一直卡住，应该是哪里有问题：
检查过程：
[root@compute02 ~]# ceph -s
  cluster:
id:    a6d0359e-8295-4e62-8620-54273462e79f
health: HEALTH_WARN
         Reduced data availability: 32 pgs inactive

  services:
mon: 3 daemons, quorum compute01,compute02,compute03 (age 3m)
mgr: compute03(active, since 4w), standbys: compute01, compute02
osd: 6 osds: 6 up (since 12d), 6 in (since 3w)
rgw: 3 daemons active (compute01, compute02, compute03)

  task status:

  data:
pools: 11 pools, 1600 pgs
objects: 65.22k objects, 258 GiB
usage: 754 GiB used, 4.5 TiB / 5.2 TiB avail
pgs:    2.000% pgs unknown
         1568 active+clean
         32 unknown

  io:
client: 114 KiB/s rd, 9.3 MiB/s wr, 144 op/s rd, 53 op/s wr

[root@compute02 ~]# ceph osd pool ls
images
volumes
backups
vms
ceph-test
.rgw.root
default.rgw.control
default.rgw.meta
default.rgw.log
default.rgw.buckets.index
default.rgw.buckets.data
[root@compute02 ~]# ceph osd pool set default.rgw.buckets.data crush_rule ssd_rule
set pool 15 crush_rule to ssd_rule

完成之后，直接显示成功了：如下done

[root@compute01 ~]# s3cmd put /etc/fstab s3://s3cmd-demo/fatab-demo
upload: '/etc/fstab' -> 's3://s3cmd-demo/fatab-demo'  [1 of 1]
465 of 465 100% in 0s 71.34 KB/s

465 of 465 100% in 81s    5.72 B/s  done
4. 常用操作
复制
# 1. 列出所有 bucket
[root@compute01 ~]# s3cmd ls
2022-02-09 06:35  s3://ceph-s3-bucket
2022-02-09 07:45  s3://s3cmd-demo

# 2. 创建 bucket
[root@compute01 ~]# s3cmd mb s3://gwj-demo/
Bucket 's3://gwj-demo/' created

[root@compute01 ~]# s3cmd ls
2022-02-09 06:35  s3://ceph-s3-bucket
2022-02-09 08:01  s3://gwj-demo
2022-02-09 07:45  s3://s3cmd-demo

# 3. 删除空的bucket
[root@compute01 ~]# s3cmd rb s3://gwj-demo
Bucket 's3://gwj-demo/' removed

[root@compute01 ~]# s3cmd ls
2022-02-09 06:35  s3://ceph-s3-bucket
2022-02-09 07:45  s3://s3cmd-demo

# 4. 上传文件到bucket
[root@compute01 ~]# s3cmd put ip s3://s3cmd-demo
upload: 'ip' -> 's3://s3cmd-demo/ip'  [1 of 1]
5 of 5 100% in 0s 296.75 B/s  done

# 5. 上传目录到 bucket
[root@compute01 ~]# s3cmd put ./ s3://s3cmd-demo/
ERROR: Parameter problem: Use --recursive to upload a directory: ./

[root@compute01 ~]# s3cmd put ./ s3://s3cmd-demo/ --recursive
upload: './.ansible/tmp/ansible-tmp-1638530173.61-8631-41487900105782/AnsiballZ_command.py' -> 's3://s3cmd-demo/.ansible/tmp/ansible-tmp-1638530173.61-8631-41487900105782/AnsiballZ_command.py'  [1 of 177]
106754 of 106754 100% in 0s    2.29 MB/s  done
upload: './.ansible/tmp/ansible-tmp-1638539832.68-4173-41041061132542/AnsiballZ_copy.py' -> 's3://s3cmd-demo/.ansible/tmp/ansible-tmp-1638539832.68-4173-41041061132542/AnsiballZ_copy.py'  [2 of 177]
111789 of 111789 100% in 0s    2.13 MB/s  done
upload: './.ansible/tmp/ansible-tmp-1638539833.48-4173-36283133816628/AnsiballZ_copy.py' -> 's3://s3cmd-demo/.ansible/tmp/ansible-tmp-1638539833.48-4173-36283133816628/AnsiballZ_copy.py'  [3 of 177]
111789 of 111789 100% in 0s  1485.51 KB/s  done

又卡住了：
检查：
[root@compute01 ~]# ceph -s
  cluster:
id:    a6d0359e-8295-4e62-8620-54273462e79f
health: HEALTH_WARN
         Reduced data availability: 32 pgs inactive

  services:
mon: 3 daemons, quorum compute01,compute02,compute03 (age 15m)
mgr: compute03(active, since 4w), standbys: compute01, compute02
osd: 6 osds: 6 up (since 12d), 6 in (since 3w)
rgw: 3 daemons active (compute01, compute02, compute03)

  task status:

  data:
pools: 12 pools, 1632 pgs
objects: 65.40k objects, 258 GiB
usage: 754 GiB used, 4.5 TiB / 5.2 TiB avail
pgs:    1.961% pgs unknown
         1600 active+clean
         32 unknown

  io:
client: 117 KiB/s rd, 11 MiB/s wr, 145 op/s rd, 50 op/s wr

[root@compute01 ~]# ceph osd pool ls
images
volumes
backups
vms
ceph-test
.rgw.root
default.rgw.control
default.rgw.meta
default.rgw.log
default.rgw.buckets.index
default.rgw.buckets.data
default.rgw.buckets.non-ec
[root@compute01 ~]# ceph osd pool set default.rgw.buckets.non-ec crush_rule ssd_rule
set pool 16 crush_rule to ssd_rule

设置完成，上传完成：
upload: './monitor/telegraf' -> 's3://s3cmd-demo/monitor/telegraf'  [part 1 of 6, 15MB] [174 of 177]
15728640 of 15728640 100% in 2s    5.38 MB/s  done
upload: './monitor/telegraf' -> 's3://s3cmd-demo/monitor/telegraf'  [part 2 of 6, 15MB] [174 of 177]
15728640 of 15728640 100% in 0s 38.25 MB/s  done
upload: './monitor/telegraf' -> 's3://s3cmd-demo/monitor/telegraf'  [part 3 of 6, 15MB] [174 of 177]
15728640 of 15728640 100% in 0s 22.08 MB/s  done
upload: './monitor/telegraf' -> 's3://s3cmd-demo/monitor/telegraf'  [part 4 of 6, 15MB] [174 of 177]
15728640 of 15728640 100% in 0s 15.15 MB/s  done
upload: './monitor/telegraf' -> 's3://s3cmd-demo/monitor/telegraf'  [part 5 of 6, 15MB] [174 of 177]
15728640 of 15728640 100% in 1s 11.45 MB/s  done
upload: './monitor/telegraf' -> 's3://s3cmd-demo/monitor/telegraf'  [part 6 of 6, 11MB] [174 of 177]
11599872 of 11599872 100% in 0s 21.31 MB/s  done
upload: './monitor/telegraf_system.conf' -> 's3://s3cmd-demo/monitor/telegraf_system.conf'  [175 of 177]
4322 of 4322 100% in 0s 87.82 KB/s  done
upload: './osds.map' -> 's3://s3cmd-demo/osds.map'  [176 of 177]
5895 of 5895 100% in 0s 120.84 KB/s  done
upload: './s3client.py' -> 's3://s3cmd-demo/s3client.py'  [177 of 177]
659 of 659 100% in 0s 13.74 KB/s  done

# 6. 列举 bucket 中的内容
[root@compute01 ~]# s3cmd ls s3://s3cmd-demo/
                        DIR  s3://s3cmd-demo/.ansible/
                        DIR  s3://s3cmd-demo/.cache/
                        DIR  s3://s3cmd-demo/.gnupg/
                        DIR  s3://s3cmd-demo/.ssh/
                        DIR  s3://s3cmd-demo/monitor/
2022-02-09 08:08       25196  s3://s3cmd-demo/.bash_history
2022-02-09 08:08          18  s3://s3cmd-demo/.bash_logout
2022-02-09 08:08       176  s3://s3cmd-demo/.bash_profile
2022-02-09 08:08       176  s3://s3cmd-demo/.bashrc
2022-02-09 08:08       1077  s3://s3cmd-demo/.cephdeploy.conf
2022-02-09 08:08       100  s3://s3cmd-demo/.cshrc
2022-02-09 08:08          42  s3://s3cmd-demo/.lesshst
2022-02-09 08:08       2248  s3://s3cmd-demo/.s3cfg
2022-02-09 08:08       129  s3://s3cmd-demo/.tcshrc
2022-02-09 08:08       6490  s3://s3cmd-demo/.viminfo
2022-02-09 08:08       34064  s3://s3cmd-demo/1111
2022-02-09 08:08       1797  s3://s3cmd-demo/anaconda-ks.cfg
2022-02-09 08:08       86604  s3://s3cmd-demo/ceph-grafana-dashboards-14.2.8-111.el7cp.noarch.rpm
2022-02-09 08:08    9625568  s3://s3cmd-demo/ceph-mgr-dashboard-14.2.8-111.el7cp.noarch.rpm
2022-02-09 07:52       465  s3://s3cmd-demo/fatab-demo
2022-02-09 07:58       465  s3://s3cmd-demo/fstab
2022-02-09 08:08          5  s3://s3cmd-demo/ip
2022-02-09 08:11       5895  s3://s3cmd-demo/osds.map
2022-02-09 08:11       659  s3://s3cmd-demo/s3client.py

# 7. 下载单个文件
[root@compute01 tmp]# s3cmd get s3://s3cmd-demo/ip
download: 's3://s3cmd-demo/ip' -> './ip'  [1 of 1]
5 of 5 100% in 0s 396.17 B/s  done

[root@compute01 tmp]# ls
ip

# 8. 删除 bucket 中的内容
[root@compute01 tmp]# s3cmd del s3://s3cmd-demo/ip
delete: 's3://s3cmd-demo/ip'

[root@compute01 tmp]# s3cmd get s3://s3cmd-demo/ip
ERROR: Parameter problem: File ./ip already exists. Use either of --force / --continue / --skip-existing or give it a new name.
[root@compute01 tmp]#

# 9. 获取对应的bucket所占用的空间大小
[root@compute01 tmp]# s3cmd du -H s3://s3cmd-demo/
104M    178 objects s3://s3cmd-demo/
[root@compute01 tmp]#  s3cmd du -H s3://s3cmd-demo/.ssh
4K    4 objects s3://s3cmd-demo/.ssh

# 10. 查看bucket文件信息
[root@compute01 tmp]#  s3cmd info s3://s3cmd-demo/fstab
s3://s3cmd-demo/fstab (object):
File size: 465
Last mod:  Wed, 09 Feb 2022 07:58:45 GMT
MIME type: text/plain
Storage: STANDARD
MD5 sum: b3b65116db83a743d6e80ae6c31896ef
SSE:    none
Policy: none
CORS:    none
ACL:    Ceph S3 User Demo: FULL_CONTROL
x-amz-meta-s3cmd-attrs: atime:1644387423/ctime:1638501772/gid:0/gname:root/md5:b3b65116db83a743d6e80ae6c31896ef/mode:33188/mtime:1638501362/uid:0/uname:root

# 11. 两个bucket之间相互cp
[root@compute01 gwj]# s3cmd cp s3://s3cmd-demo/ip s3://test-demo/
remote copy: 's3://s3cmd-demo/ip' -> 's3://test-demo/ip'
[root@compute01 gwj]# s3cmd cp --recursive s3://s3cmd-demo/.ssh s3://test-demo/
remote copy: 's3://s3cmd-demo/.ssh/authorized_keys' -> 's3://test-demo/.ssh/authorized_keys'
remote copy: 's3://s3cmd-demo/.ssh/id_rsa' -> 's3://test-demo/.ssh/id_rsa'
remote copy: 's3://s3cmd-demo/.ssh/id_rsa.pub' -> 's3://test-demo/.ssh/id_rsa.pub'
remote copy: 's3://s3cmd-demo/.ssh/known_hosts' -> 's3://test-demo/.ssh/known_hosts'
[root@compute01 gwj]#

# 12. 两个bucket之间进行mv操作
[root@compute01 gwj]# s3cmd ls s3://s3cmd-demo/.swp
2020-10-06 10:24 12288 s3://s3cmd-demo/.swp
[root@compute01 gwj]# s3cmd mv s3://s3cmd-demo/.swp s3://test-demo/
move: 's3://s3cmd-demo/.swp' -> 's3://test-demo/.swp'
[root@compute01 gwj]# s3cmd ls s3://test-demo/.swp
2020-10-06 10:36 12288 s3://test-demo/.swp
[root@compute01 gwj]# s3cmd ls s3://s3cmd-demo/.swp
[root@compute01 gwj]#

# 13. 列出需要同步的文件和目录，但不进行同步
[root@compute01 ~]# s3cmd sync --dry-run ./ s3://s3cmd-demo
upload: './.swp' -> 's3://s3cmd-demo/.swp'
upload: './ip' -> 's3://s3cmd-demo/ip'
upload: './.cache/abrt/lastnotification' -> 's3://s3cmd-demo/.cache/abrt/lastnotification'
remote copy: 'size.log' -> 'gwj/size.log'
WARNING: Exiting now because of --dry-run
[root@compute01 ~]#

# 14. 在bucket中删除本地不存在的文件
[root@compute01 a]# ls
10.txt 1.txt 2.txt 3.txt 4.txt 5.txt 6.txt 7.txt 8.txt 9.txt
[root@ceph-node01 a]#
[root@ceph-node01 a]# s3cmd ls s3://test2-demo/

[root@compute01 a]# rm -rf 10.txt
[root@compute01 a]# s3cmd sync --delete-removed ./ s3://test2-demo/
delete: 's3://test2-demo/10.txt'
[root@ceph-node01 a]# s3cmd ls s3://test2-demo/

[root@compute01 a]#

使用 swift 访问 RGW
1. 创建 swift 用户
复制
[root@compute01 ceph-deploy]# radosgw-admin subuser create --uid ceph-s3-user --subuser=ceph-s3-user:swift --access=full
{
"user_id": "ceph-s3-user",
"display_name": "Ceph S3 User Demo",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [
      {
         "id": "ceph-s3-user:swift",
         "permissions": "full-control"
      }
],
"keys": [
      {
         "user": "ceph-s3-user",
         "access_key": "V3J9L4M1WKV5O5ECAKPU",
         "secret_key": "f5LqLVYOVNu38cuQwi0jXC2ZTboCSJDmdvB8oeYw"
      }
],
"swift_keys": [
      {
         "user": "ceph-s3-user:swift",
         "secret_key": "ZIOOU8Xcfe3m6ZZapK5P2rU0GGPaiS31chy9yvMW"
      }
],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
      "enabled": false,
      "check_on_raw": false,
      "max_size": -1,
      "max_size_kb": 0,
      "max_objects": -1
},
"user_quota": {
      "enabled": false,
      "check_on_raw": false,
      "max_size": -1,
      "max_size_kb": 0,
      "max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}

[root@compute01 ceph-deploy]#

2. 创建 swift 用户的secret
复制
[root@compute01 ceph-deploy]# radosgw-admin key create --subuser=ceph-s3-user:swift --key-type=swift --gen-secret
{
"user_id": "ceph-s3-user",
"display_name": "Ceph S3 User Demo",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [
      {
         "id": "ceph-s3-user:swift",
         "permissions": "full-control"
      }
],
"keys": [
      {
         "user": "ceph-s3-user",
         "access_key": "V3J9L4M1WKV5O5ECAKPU",
         "secret_key": "f5LqLVYOVNu38cuQwi0jXC2ZTboCSJDmdvB8oeYw"
      }
],
"swift_keys": [
      {
         "user": "ceph-s3-user:swift",
         "secret_key": "0M1GdRTvMSU3fToOxEVXrBjItKLBKtu8xhn3DcEE"
      }
],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
      "enabled": false,
      "check_on_raw": false,
      "max_size": -1,
      "max_size_kb": 0,
      "max_objects": -1
},
"user_quota": {
      "enabled": false,
      "check_on_raw": false,
      "max_size": -1,
      "max_size_kb": 0,
      "max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}

[root@ceph-node01 ceph-deploy]#

3. 需要使用 pip 安装 swift 客户端工具
复制
# 注意如果有pip的话，就不需要再安装了
[root@compute01 ceph-deploy]# curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
。。。
[root@compute01 ceph-deploy]# python get-pip.py
。。。
[root@compute01 ceph-deploy]# pip install python-swiftclient
。。。

4. 使用 swift 命令行工具
复制
[root@compute01 ceph-deploy]# swift -A http://100.73.18.152/auth -U ceph-s3-user:swift -K 0M1GdRTvMSU3fToOxEVXrBjItKLBKtu8xhn3DcEE list
ceph-s3-bucket
s3cmd-demo
[root@compute01 ceph-deploy]#

5. 配置成环境变量的形式
复制
[root@compute01 ceph-deploy]# cat /etc/profile
。。。
export ST_AUTH=http://100.73.18.152/auth
export ST_USER=ceph-s3-user:swift
export ST_KEY=0M1GdRTvMSU3fToOxEVXrBjItKLBKtu8xhn3DcEE
[root@ceph-node01 ceph-deploy]# source /etc/profile
[root@ceph-node01 ceph-deploy]# swift list
ceph-s3-bucket
s3cmd-demo
[root@compute01 ceph-deploy]#

6. 创建bucket
复制
[root@compute01 ceph-deploy]# swift post swift-demo
[root@compute01 ceph-deploy]# swift list
ceph-s3-bucket
s3cmd-demo
swift-demo
[root@compute01 ceph-deploy]#

7. 上传单个文件测试
复制
[root@compute01 ceph-deploy]# swift upload swift-demo /etc/fstab
Object HEAD failed: http://100.73.18.152/swift/v1/swift-demo/etc/fstab 416 Requested Range Not Satisfiable
[root@compute01 ceph-deploy]#

8. 上传目录测试
复制
[root@compute01 a]# swift upload swift-demo /etc/fstab
etc/fstab
[root@compute01 a]#

9. 常用操作
复制
# 1. 列举所有 bucket
[root@compute01 a]# swift list
ceph-s3-bucket
s3cmd-demo
swift-demo
test-demo
test2-demo
[root@compute01 a]#

# 2. 列举所有 bucket
[root@compute01 a]# swift list --lh
0 0 2020-10-06 04:13:10 ceph-s3-bucket
37 3.6M 2020-10-06 04:34:49 s3cmd-demo
2360 33M 2020-10-06 08:07:55 swift-demo
7 16K 2020-10-06 10:32:02 test-demo
9 387 2020-10-06 11:17:00 test2-demo
2.4K 36M
[root@compute01 a]#

# 3. 列举单个 bucket
[root@compute01 a]# swift list swift-demo

# 4. 上传单个文件到bucket
[root@compute01 a]# swift upload swift-demo /etc/fstab
etc/fstab
[root@compute01 a]#

# 5. 上传目录到指定的bucket
[root@compute01 a]# swift upload swift-demo /etc/

# 6. swift 状态信息
[root@compute01 a]# swift stat
                                 Account: v1
                              Containers: 5
                                 Objects: 2413
                                    Bytes: 38701415
Objects in policy "default-placement-bytes": 0
  Bytes in policy "default-placement-bytes": 0
Containers in policy "default-placement": 5
   Objects in policy "default-placement": 2413
      Bytes in policy "default-placement": 38701415
                  X-Openstack-Request-Id: tx000000000000000001302-005f7c5afd-a638-default
            X-Account-Bytes-Used-Actual: 45948928
                              X-Trans-Id: tx000000000000000001302-005f7c5afd-a638-default
                              X-Timestamp: 1601985277.38095
                           Content-Type: text/plain; charset=utf-8
                           Accept-Ranges: bytes
[root@compute01 a]#

# 7. 创建 bucket
[root@compute01 a]# swift post swift-test
[root@compute01 a]# swift list
ceph-s3-bucket
s3cmd-demo
swift-demo
swift-test
test-demo
test2-demo
[root@compute01 a]#

# 8. 删除 bucket
[root@compute01 a]# swift delete swift-demo

# 9. 删除指定 object
[root@compute01 a]# swift delete swift-test root/a/1.txt
root/a/1.txt
[root@compute01 a]#

# 10. 上传大文件时可以使用-S指定分片大小
[root@compute01 a]# swift upload swift-test /home/log.txt
home/log.txt
[root@compute01 a]# swift upload swift-test -S 102400000 /home/log2.txt
home/log2.txt segment 5
home/log2.txt segment 3
home/log2.txt segment 1
home/log2.txt segment 0
home/log2.txt segment 2
home/log2.txt segment 4
home/log2.txt
[root@ceph-node01 a]#

总结
从概念、安装、到实践使用，简单介绍了对象存储命令行工具。

admin · 发表于 2022-2-10 15:06:53

http://www.bbs168x.cn/forum.php? ... tid=2088&extra=
相关的对象存储用户操作可以从此处借鉴

		自动登录	找回密码
密码			注册

Ceph 基础篇 -对象存储使用

浏览过的版块