ceph-deploy之配置安装使用对象存储

admin · 发表于 2022-2-9 09:58:36

eph存储显著的简化了安装和配置Ceph Object Gateway，

Gateway进程嵌入到Civetweb，所以你需要安装一个web服务，或者配置FastCGI，
另外，ceph-deploy可以安装gateway包，生成key，配置数据目录，创建gateway实例。
Tip：Civetweb默认使用7480端口，您必须打开端口7480，或将端口设置为Ceph配置文件中的首选端口（例如端口80）。
安装Ceph Object Gateway1、保证开启端口
2、在admin节点安装Ceph Object Gateway包到client-node节点
ceph-deploy install --rgw <client-node> [<client-node>...]
例子：
ceph-deploy install --rgw cephhost1 cephhost2
创建Ceph Object Gateway实例 ceph-deploy rgw create <client-node> 例子：
ceph-deploy rgw create cephhost1

一旦gateway运行，你可以访问它的7480端口（例如：http://cephhost1:7480）
配置Ceph Object Gateway实例1、通过修改ceph.conf配置文件修改默认端口，添加标题为 [client.rgw.<client-node>]的部分，
用ceph client节点hostname（hostname -s）替换<client-node>.

如果节点name是cephhost1，那么在[global]部分之后，添加如下：
[
rgw_frontends = "civetweb port=80"
注意：
确保在rgw_frontends键/值对中的端口=端口号之间不留空白，
如果您打算使用端口80，请确保Apache服务器未运行，否则将与Civetweb发生冲突。在这种情况下，我们建议删除Apache。

2、重启使新配置端口生效
systemctl restart ceph-radosgw.service
3、如果开启了防火墙，检查端口在防火墙中打开。如果未打开，添加端口重载firewall生效
firewall-cmd --list-all
firewall-cmd --zone=public --add-port 80/tcp --permanent
firewall-cmd --reload
4、现在可以发起一个未认证请求，并且返回一个结果
#request
curl http://<client-node>:80
#result
<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Owner>
<ID>anonymous</ID>
<DisplayName></DisplayName>
</Owner>
<Buckets>
</Buckets>
</ListAllMyBucketsResult>

admin · 发表于 2022-2-9 10:04:03

ceph对象存储
顾名思义，对象存储以对象方式管理数据。每一个对象存储数据、元数据以及一个唯一的标识符。对象存储不能直接被操作系统当成本地或者远程文件系统访问。它只能在应用程序级别通过API访问。ceph提供的对象存储接口是RADOS网关，它建立在ceph RADOS层之上。RADOS网关为应用程序提供兼容S3或者Swift的RESTful API接口，以便将数据以对象方式存储到ceph集群中。
在生产环境中，如果你在ceph对象存储上有大量的工作负载，则你应该使用专用的物理服务器来配置RADOS网关，另外你可以考虑将所有的monitor节点配置成RADOS网关。

安装radosgw相关包
yum -y install ceph-radosgw ceph
1
创建用户
为ceph创建rados网关用户及密钥环，登录任意一台ceph monitor节点上，并执行下列命令
创建密钥环

ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
输出
creating /etc/ceph/ceph.client.radosgw.keyring
1
2
3
chmod +r /etc/ceph/ceph.client.radosgw.keyring
1
这时候/etc/ceph/ceph.client.radosgw.keyring 文件还是空的
为RADOS网关实例生成网关用户以及密钥，这里的RADOS网关实例名是gateway

ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway --gen-key
1
cat /etc/ceph/ceph.client.radosgw.keyring
[client.radosgw.gateway]
key = AQBWuqBf5apFDxAAAbqsG0NTx8lehGoNpcPVJQ==
1
2
3
为密钥增加功能

ceph-authtool -n client.radosgw.gateway --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
1
cat /etc/ceph/ceph.client.radosgw.keyring
[client.radosgw.gateway]
key = AQBWuqBf5apFDxAAAbqsG0NTx8lehGoNpcPVJQ==
caps mon = "allow rw"
caps osd = "allow rwx"
1
2
3
4
5
将密钥添加到ceph集群中

ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring
输出
added key for client.radosgw.gateway
1
2
3
分发密钥到ceph rados网关节点上

scp /etc/ceph/ceph.client.radosgw.keyring radosgw节点的hostname:/etc/ceph/ceph.client.radosgw.keyring
1
由于这里的rados网关节点和monitor节点在一台机器上，所以不必分发。

为rados网关创建池
ceph osd pool create .rgw 128 128
1
创建radosgw网关数据目录
mkdir -p /var/lib/ceph/radosgw/ceph-ceph01.gateway
1
Civetweb方式配置rgw
增加配置
给ceph添加一个网关配置，添加如下配置到ceph monitor节点的ceph.conf文件中，并将该文件移动到RADOS网关节点。确保主机名是RADOS网关的主机名。

[client.radosgw.gateway]
host=ceph01
keyring=/etc/ceph/ceph.client.radosgw.keyring
log file=/var/log/ceph/client.radosgw.gateway.log
rgw_frontends = civetweb port=80
1
2
3
4
5
civetweb默认监听在7480端口，上述的配置中显示指定监听端口为80(port=80)
复制配置文件到rgw节点，这里rgw节点在ceph节点上，不必复制。

scp  /etc/ceph/ceph.conf radosgw节点的hostname:/etc/ceph/ceph.conf
1
启动rgw实例
systemctl start ceph-radosgw@radosgw.gateway.service
1
说明：ceph-radosgw@radosgw.gateway.service中gateway为具体的实例名，这个实例名要和ceph.conf中配置的一致。

验证
rgw节点curl

curl localhost:80
正确输出
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>[root@ceph01 ceph]#
1
2
3
或者浏览器输入rgw节点ip
正确输出如下

apache fastcgi方式配置rgw（方法1简单）
安装httpd
yum -y install httpd
1
配置httpd
cat > /etc/httpd/conf.d/rgw.conf << EOF
<VirtualHost *:80>
ServerName localhost
DocumentRoot /var/www/html

ErrorLog /var/log/httpd/rgw_error.log
CustomLog /var/log/httpd/rgw_access.log combined

# LogLevel debug

RewriteEngine On

RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]

SetEnv proxy-nokeepalive 1

ProxyPass / fcgi://localhost:9000/

</VirtualHost>
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
配置ceph
[client.radosgw.gateway]
host=ceph01
keyring=/etc/ceph/ceph.client.radosgw.keyring
rgw_socket_path=/var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file=/var/log/ceph/client.radosgw.gateway.log
rgw frontends = fastcgi socket_port=9000 socket_host=0.0.0.0
#rgw dns name = ceph-rgw.objectstore.com
rgw print continue=false
1
2
3
4
5
6
7
8
启动httpd
systemctl start httpd
1
启动ceph-radosgw.target服务
systemctl start ceph-radosgw@radosgw.gateway.service
1
验证
curl http://ceph01
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>[root@ceph01 yum.repos.d]#
1
2
或者浏览器输入rgw节点ip
正确输出如下

apache fastcgi方式配置rgw（方法2复杂）
安装yum-plugin-priorities
yum -y install yum-plugin-priorities
1
配置mod_fastcgi的yum源
cat > /etc/yum.repos.d/ceph-fastcgi.repo <<EOF
[fastcgi-ceph-basearch]
name=FastCGI basearch packages for Ceph
baseurl=http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/
enabled=1
priority=2
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

[fastcgi-ceph-noarch]
name=FastCGI noarch packages for Ceph
baseurl=http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/
enabled=1
priority=2
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

[fastcgi-ceph-basearch]
name=FastCGI source packages for Ceph
baseurl=http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/
enabled=1
priority=2
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
安装相关包
yum -y install httpd mod_fastcgi ceph-radosgw ceph
1
配置rados网关
rados网关的配置包括apache和fastcgi的配置以及ceph密钥的生成。
编辑/etc/httpd/conf/httpd.conf文件以配置apache

cp /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.bak
1
设置ServerName = 主机名
1
确保下面这行存在并且是非注释状态

cat /etc/httpd/conf/httpd.conf|egrep "rgw|rewrite"
LoadModule rewrite_module modules/mod_rewrite.so
1
2
编辑/etc/httpd/conf.d/fastcgi.conf文件以配置FastCGI
确保FastCGI模块开启

cat  /etc/httpd/conf.d/fastcgi.conf |grep "fastcgi_module"
LoadModule fastcgi_module modules/mod_fastcgi.so
1
2
关闭FastCgiWrapper

cat  /etc/httpd/conf.d/fastcgi.conf |grep -i "fastcgiwrapper"
FastCgiWrapper Off
1
2
设置对象网关脚本

cat > /var/www/html/s3gw.fcgi <<
#!/bin/sh
exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway
EOF
1
2
3
4
授权脚本
chmod +x /var/www/html/s3gw.fcgi
chown apache.apache /var/www/html/s3gw.fcgi
1
2
在/etc/httpd/conf.d/目录下创建网关配置文件rgw.conf

cat >/etc/httpd/conf.d/rgw.conf<<EOF
FastCgiExternalServer /var/www/html/s3gw.fcgi -socket /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
<VirtualHost *:80>
ServerName ceph01
ServerAdmin {zhanghao@ceicloud.com}
DocumentRoot /var/www/html
RewriteEngine On
RewriteRule ^/(.*) /s3gw.fcgi?%{QUERY_STRING}[E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
<IfModule mod_fastcgi.c>
<Directory /var/www/html>
Options +ExecCGI
AllowOverride ALL
SetHandler fastcgi-script
Order allow,deny
Allow from all
AuthBasicAuthoritative Off
</Directory>
</IfModule>
AllowEncodedSlashes On
Errorlog /var/log/httpd/error.log
CustomLog /var/log/httpd/access.log combined
ServerSignature Off
</VirtualHost>
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
添加ceph配置
给ceph添加一个网关配置，添加如下配置到ceph monitor节点的ceph.conf文件中，并将该文件移动到RADOS网关节点。确保主机名是RADOS网关的主机名。

[client.radosgw.gateway]
host=ceph01
keyring=/etc/ceph/ceph.client.radosgw.keyring
rgw socket path=/var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file=/var/log/ceph/client.radosgw.gateway.log
#rgw frontends = fastcgi socket_port=80 socket_host=0.0.0.0
rgw dns name = ceph-rgw.objectstore.com
rgw print continue=false
1
2
3
4
5
6
7
8
设置文件权限
调整RADOS网关节点上的/var/log/httpd、/var/run/ceph和/var/log/ceph这三个文件的所有权和权限，并设置SELinux为Permissive。

chown apache:apache /var/log/httpd/
chown apache:apache /var/run/ceph/
chown apache:apache /var/log/ceph/
1
2
3
启动Apache和ceph RADOS网关服务，忽略遇到的任何警告。

启动httpd
systemctl start httpd
1
启动ceph-radosgw.target服务
systemctl start ceph-radosgw@radosgw.gateway.service
1
验证
curl http://ceph01
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>[root@ceph01 yum.repos.d]#
1
2
或者浏览器输入rgw节点ip
正确输出如下

Nginx fastcgi方式配置rgw
添加ceph配置
给ceph添加一个网关配置，添加如下配置到ceph monitor节点的ceph.conf文件中，并将该文件移动到RADOS网关节点。确保主机名是RADOS网关的主机名。

[client.radosgw.gateway]
host=ceph01
keyring=/etc/ceph/ceph.client.radosgw.keyring
rgw_socket_path=/var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file=/var/log/ceph/client.radosgw.gateway.log
rgw frontends = fastcgi
rgw print continue=false
rgw_content_length_compat = true
1
2
3
4
5
6
7
8
安装nginx
yum -y install epel-release
yum -y install nginx
1
2
配置nginx
配置nginx服务，在/etc/nginx/nginx.conf文件的http段下添加如下内容：

http {
server {
      listen 80 default;
      server_name {hostname};
location / {
         fastcgi_pass_header Authorization;
         fastcgi_pass_request_headers on;
         fastcgi_param QUERY_STRING  $query_string;
         fastcgi_param REQUEST_METHOD $request_method;
         fastcgi_param CONTENT_LENGTH $content_length;
         fastcgi_param CONTENT_LENGTH $content_length;

         if ($request_method = PUT) {
                  rewrite ^ /PUT$request_uri;
         }

         include fastcgi_params;
         fastcgi_pass unix:/var/run/ceph/ceph.radosgw.gateway.sock;
      }

      location /PUT/ {
         internal;
         fastcgi_pass_header Authorization;
         fastcgi_pass_request_headers on;

         include fastcgi_params;
         fastcgi_param QUERY_STRING  $query_string;
         fastcgi_param REQUEST_METHOD $request_method;
         fastcgi_param CONTENT_LENGTH $content_length;
         fastcgi_param  CONTENT_TYPE $content_type;
         fastcgi_pass unix:/var/run/ceph/ceph.radosgw.gateway.sock;
      }
}

注意： fastcgi_pass 指向的路径需要与ceph.conf中配置的路径一致。
修改nginx启动用户
编辑/etc/nginx/nginx.conf文件，修改user为root

user root;

启动nginx
systemctl start nginx

cat /var/log/nginx/error.log
2020/11/03 15:19:09 [crit] 26789#0: *23 connect() to unix:/var/run/ceph/ceph.radosgw.gateway.fastcgi.sock failed (13: Permission denied) while connecting to upstream, client: 192.168.229.114, server: ceph01, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/ceph/ceph.radosgw.gateway.fastcgi.sock:", host: "ceph01"

验证
curl http://ceph01
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>[root@ceph01 yum.repos.d]#

或者浏览器输入rgw节点ip
正确输出如下

报错
配置nginx fastcgi rgw时候权限错误
curl http://ceph01
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.16.1</center>
</body>
</html>

nginx错误日志

2020/11/03 15:19:09 [crit] 26789#0: *23 connect() to unix:/var/run/ceph/ceph.radosgw.gateway.fastcgi.sock failed (13: Permission denied) while connecting to upstream, client: 192.168.229.114, server: ceph01, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/ceph/ceph.radosgw.gateway.fastcgi.sock:", host: "ceph01"

解决
修改nginx启动用户
编辑/etc/nginx/nginx.conf文件，修改user为root

user root;

重载nginx配置

nginx -s reload

apache fastcgi配置rgw（方法2复杂）报错
curl http://ceph01
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
zhanghao@ceicloud.com to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>

[Tue Nov 03 15:31:06.955924 2020] [:error] [pid 28243] (13)Permission denied: [client 192.168.229.114:36712] FastCGI: failed to connect to server "/var/www/html/s3gw.fcgi": connect() failed
[Tue Nov 03 15:31:06.956045 2020] [:error] [pid 28243] [client 192.168.229.114:36712] FastCGI: incomplete headers (0 bytes) received from server "/var/www/html/s3gw.fcgi"

授权

chmod 777 -R /var/run/ceph/

换了报错，但还是权限的问题

[Tue Nov 03 15:39:19.598498 2020] [:error] [pid 29128] (111)Connection refused: [client 192.168.229.114:36768] FastCGI: failed to connect to server "/var/www/html/s3gw.fcgi": connect() failed
[Tue Nov 03 15:39:19.598595 2020] [:error] [pid 29128] [client 192.168.229.114:36768] FastCGI: incomplete headers (0 bytes) received from server "/var/www/html/s3gw.fcgi"

检查httpd的启动用户，发现启动用户不一致

[root@ceph01 yum.repos.d]# ps -ef|grep httpd
root    29125    1  0 15:38 ?       00:00:00 /usr/sbin/httpd -DFOREGROUND
apache 29127 29125  0 15:38 ?       00:00:00 /usr/sbin/httpd -DFOREGROUND
apache 29128 29125  0 15:38 ?       00:00:00 /usr/sbin/httpd -DFOREGROUND
apache 29129 29125  0 15:38 ?       00:00:00 /usr/sbin/httpd -DFOREGROUND
apache 29130 29125  0 15:38 ?       00:00:00 /usr/sbin/httpd -DFOREGROUND
apache 29131 29125  0 15:38 ?       00:00:00 /usr/sbin/httpd -DFOREGROUND
root    29414 12349  0 15:43 pts/3 00:00:00 grep --color=auto httpd

解决：
用root用户启动nginx
具体方法百度，但不是只改配置文件User Group那么简单。

创建rados网关用户
要访问对象存储，需要为RADOS网关创建用户。这些用户的账户将由访问权限以及密钥标识，客户端可以使用这些账户来执行ceph对象存储操作。

复制ceph密钥
从monitor节点复制admin的密钥到rados的网关节点

scp /etc/ceph/ceph.client.admin.keyring RADOS网关节点ip:/etc/ceph/

从rados网关节点执行命令确定集群可达
ceph -s

创建rados网关用户
这是会生成该用户的access_key和secret_key，这两个密钥用于访问ceph对象存储。

radosgw-admin user create --uid=mona --display-name='Monika Singh' --email=mona@example.com
输出
{
"user_id": "mona",
"display_name": "Monika Singh",
"email": "mona@example.com",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
      {
         "user": "mona",
         "access_key": "JDRTJS0766NOL89YXR8X",
         "secret_key": "Sg6QTkXMs79epxSUEvwFmjVNWgqvWI2Jkll4KiNQ"
      }
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
      "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1
},
"user_quota": {
      "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1
},
"temp_url_keys": []
}

说明：执行这条命令之后会自动创建存储池，创建的存储池如下

ceph osd pool ls
.rgw.root
default.rgw.control
default.rgw.data.root
default.rgw.gc
default.rgw.log
default.rgw.users.uid
default.rgw.users.email
default.rgw.users.keys

创建一个mano的子用户用于swift访问
radosgw-admin subuser create --uid=mona --subuser=mona:swift --access=full --secret=secretkey --key-type=swift
{
"user_id": "mona",
"display_name": "Monika Singh",
"email": "mona@example.com",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [
      {
         "id": "mona:swift",
         "permissions": "full-control"
      }
],
"keys": [
      {
         "user": "mona",
         "access_key": "JDRTJS0766NOL89YXR8X",
         "secret_key": "Sg6QTkXMs79epxSUEvwFmjVNWgqvWI2Jkll4KiNQ"
      }
],
"swift_keys": [
      {
         "user": "mona:swift",
         "secret_key": "secretkey"
      }
],
"caps": [
      {
         "type": "buckets",
         "perm": "*"
      },
      {
         "type": "metadata",
         "perm": "*"
      },
      {
         "type": "users",
         "perm": "*"
      },
      {
         "type": "zone",
         "perm": "*"
      }
],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
      "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1
},
"user_quota": {
      "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1
},
"temp_url_keys": []
}

为访问用户增加必要的能力
radosgw-admin caps add --uid=mona --caps='zone=*'
输出
{
"user_id": "mona",
"display_name": "Monika Singh",
"email": "mona@example.com",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
      {
         "user": "mona",
         "access_key": "JDRTJS0766NOL89YXR8X",
         "secret_key": "Sg6QTkXMs79epxSUEvwFmjVNWgqvWI2Jkll4KiNQ"
      }
],
"swift_keys": [],
"caps": [
      {
         "type": "zone",
         "perm": "*"
      }
],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
      "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1
},
"user_quota": {
      "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1
},
"temp_url_keys": []
}

访问对象存储（IP方式访问）
S3 API访问
客户端安装s3cmd
yum -y install s3cmd

生成s3.cfg配置文件
s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: 1F0D2GRLPRU9ENSB689J  # 粘贴服务端生成的Access Key
Secret Key: M5AmCuh8XcWnKXvBUJ8orE90z6508YGDtbvIA0h4  # 粘贴服务端生成的Secret Key
Default Region [US]:  # 直接回车即可

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: 192.168.229.114  # 输入对象存储的IP地址

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: %(bucket).192.168.229.114  # 输入对象存储的bucket地址

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: # 空密码回车
Path to GPG program [/usr/bin/gpg]: # 回车

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: No  # 是否使用https，选no

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: # 留空回车

New settings:
  Access Key: 1F0D2GRLPRU9ENSB689J
  Secret Key: M5AmCuh8XcWnKXvBUJ8orE90z6508YGDtbvIA0h4
  Default Region: US
  S3 Endpoint: 192.168.229.114
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket).192.168.229.114
  Encryption password:
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] n  #输入n

Save settings? [y/N] y  # y 要保存配置文件
Configuration saved to '/root/.s3cfg'  # 最后配置文件保存的位置/root.s3cfg

生成的s3.cfg配置文件内容如下
cat /root/.s3cfg
[default]
access_key = 1F0D2GRLPRU9ENSB689J
access_token =
add_encoding_exts =
add_headers =
bucket_location = US
ca_certs_file =
cache_file =
check_ssl_certificate = True
check_ssl_hostname = True
cloudfront_host = cloudfront.amazonaws.com
connection_pooling = True
content_disposition =
content_type =
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encrypt = False
expiry_date =
expiry_days =
expiry_prefix =
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = 192.168.229.114
host_bucket = %(bucket).192.168.229.114
human_readable_sizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key =
limit = -1
limitrate = 0
list_md5 = False
log_target_prefix =
long_listing = False
max_delete = -1
mime_type =
multipart_chunk_size_mb = 15
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
public_url_use_https = False
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
restore_priority = Standard
secret_key = M5AmCuh8XcWnKXvBUJ8orE90z6508YGDtbvIA0h4
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
signurl_use_https = False
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
stats = False
stop_on_error = False
storage_class =
throttle_max = 100
upload_id =
urlencoding_mode = normal
use_http_expect = False
use_https = False
use_mime_magic = True
verbosity = WARNING
website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
website_error =
website_index = index.html

创建桶
s3cmd mb s3://buck1
输出
Bucket 's3://buck1/' created

s3cmd mb s3://my-bucket
输出
Bucket 's3://my-bucket/' created

查看桶
s3cmd ls
输出
2020-11-04 02:43  s3://buck1
2020-11-04 02:30  s3://my-bucket

往桶中传数据
s3cmd put /etc/hosts s3://buck1
输出
upload: '/etc/hosts' -> 's3://buck1/hosts'  [1 of 1]
304 of 304 100% in 3s 92.11 B/s  done

swift访问
客户端安装swift客户端
yum -y install python-setuptools
yum -y install python-pip
pip install --upgrade pip -i https://mirrors.aliyun.com/pypi/simple
pip install --upgrade setuptools -i https://mirrors.aliyun.com/pypi/simple
pip install python-swiftclient -i https://mirrors.aliyun.com/pypi/simple

swift创建并查看桶
swift -V 1.0 -A http://192.168.229.114/auth -U mona:swift -K secretkey post swift-buck
swift -V 1.0 -A http://192.168.229.114/auth -U mona:swift -K secretkey list
输出
buck1
my-bucket
swift-buck

访问对象存储（DNS方式访问）
ceph对象存储支持S3和swift兼容的API。为了利用ceph对象存储的能力，我们需要配置S3或者swift接口。接下来我们依次为这两种接口做一个基本配置。高级配置请查阅它们各自的文档。

S3 API访问
Amazon的简单存储服务（S3）通过Web接口（例如REST）为用户提供存储服务。ceph通过RESTful API兼容S3。S3客户端应用程序能够通过access和密钥来访问ceph对象存储。下面我们来配置它，除非特别指明，否则以下命令都在ceph-rgw节点上执行。
radosgw用户应该有足够的能力来处理S3请求。为radosgw用户（ID为mona）增加必要的能力。

安装dns服务
yum -y install bind* -y

配置dns服务
cat /etc/named.conf
options {
listen-on port 53 { 127.0.0.1;192.168.229.114; };
listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
recursing-file  "/var/named/data/named.recursing";
secroots-file "/var/named/data/named.secroots";
allow-query    { localhost;192.168.0.0/16; };

zone "objectstore.com" IN {
type master;
file "db.objectstore.com";
      allow-update {none;};
};

说明：ip地址根据实际情况更改。

cat >/var/named/db.objectstore.com <<EOF
@ 86400 IN SOA objectstore.com. root.objectstore.com. (
      20091028 ; serial yyyy-mm-dd
         10800 ; serial every 15 min
         3600 ; serial every hour
      3600000 ; expire after 1 month +
         86400) ; min ttl of 1 day
@ 86400 IN NS objectstore.com.
@ 86400 IN A  192.168.229.114
* 86400 IN CNAME @
EOF

编辑/etc/resolv.conf文件
cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 114.114.114.114
search objectstore.com
nameserver 192.168.229.114

检查配置
named-checkconf /etc/named.conf
1
named-checkzone objectstore.com /var/named/db.objectstore.com
正确输出
zone objectstore.com/IN: loaded serial 20091028
OK

启动dns服务
systemctl start named

测试dns配置
dig ceph01.objectstore.com
nslookup ceph01.objectstore.com

在客户端的/etc/resolv.conf文件增加配置
cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 114.114.114.114
search objectstore.com
nameserver 192.168.229.114

测试客户端的dns配置
yum install bind-utils
dig ceph01.objectstore.com
nslookup ceph01.objectstore.com

客户端安装s3cmd
yum -y install s3cmd

生成s3.cfg配置文件
s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: 1F0D2GRLPRU9ENSB689J  # 粘贴服务端生成的Access Key
Secret Key: M5AmCuh8XcWnKXvBUJ8orE90z6508YGDtbvIA0h4 # 粘贴服务端生成的Secret Key
Default Region [US]: # 直接回车即可

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: ceph01.objectstore.com  # 输入对象存储的域名

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: %(bucket).ceph01.objectstore.com  # 输入对象存储的bucket地址

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: # 空密码回车
Path to GPG program [/usr/bin/gpg]:  #回车

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: No #输入No

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:  #回车

New settings:
  Access Key: 1F0D2GRLPRU9ENSB689J
  Secret Key: M5AmCuh8XcWnKXvBUJ8orE90z6508YGDtbvIA0h4
  Default Region: US
  S3 Endpoint: ceph01.objectstore.com
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket).ceph01.objectstore.com
  Encryption password:
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] n #输入n

Save settings? [y/N] y #输入y
Configuration saved to '/root/.s3cfg' # 最后配置文件保存的位置/root.s3cfg

生成的s3.cfg配置文件内容如下
cat /root/.s3cfg
[default]
access_key = 1F0D2GRLPRU9ENSB689J
access_token =
add_encoding_exts =
add_headers =
bucket_location = US
ca_certs_file =
cache_file =
check_ssl_certificate = True
check_ssl_hostname = True
cloudfront_host = cloudfront.amazonaws.com
connection_pooling = True
content_disposition =
content_type =
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encrypt = False
expiry_date =
expiry_days =
expiry_prefix =
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = ceph01.objectstore.com
host_bucket = %(bucket).ceph01.objectstore.com
human_readable_sizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key =
limit = -1
limitrate = 0
list_md5 = False
log_target_prefix =
long_listing = False
max_delete = -1
mime_type =
multipart_chunk_size_mb = 15
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
public_url_use_https = False
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
restore_priority = Standard
secret_key = M5AmCuh8XcWnKXvBUJ8orE90z6508YGDtbvIA0h4
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
signurl_use_https = False
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
stats = False
stop_on_error = False
storage_class =
throttle_max = 100
upload_id =
urlencoding_mode = normal
use_http_expect = False
use_https = False
use_mime_magic = True
verbosity = WARNING
website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
website_error =
website_index = index.html

vim /root/.s3cfg
host_base = ceph01.objectstore.com
host_bucket = %(bucket)s.ceph01.objectstore.com

创建桶
s3cmd mb s3://buck1
输出
Bucket 's3://buck1/' created

s3cmd mb s3://my-bucket
输出
Bucket 's3://my-bucket/' created

查看桶
s3cmd ls
输出
2020-11-04 02:43  s3://buck1
2020-11-04 02:30  s3://my-bucket

往桶中传数据
s3cmd put /etc/hosts s3://buck1
输出
upload: '/etc/hosts' -> 's3://buck1/hosts'  [1 of 1]
304 of 304 100% in 3s 92.11 B/s  done

swift访问
客户端安装swift客户端
yum -y install python-setuptools
yum -y install python-pip
pip install --upgrade pip -i https://mirrors.aliyun.com/pypi/simple
pip install --upgrade setuptools -i https://mirrors.aliyun.com/pypi/simple
pip install python-swiftclient -i https://mirrors.aliyun.com/pypi/simple

swift创建并查看桶
swift -V 1.0 -A http://ceph01.objectstore.com/auth -U mona:swift -K secretkey post swift-buck
swift -V 1.0 -A http://ceph01.objectstore.com/auth -U mona:swift -K secretkey list
输出
buck1
my-bucket
swift-buck

		自动登录	找回密码
密码			注册