ceph luminous 提供iscsi服务

admin · 发表于 2021-7-10 17:41:02

开始安装之前必须要满足以下条件：
正在运行的Ceph Luminous版本集群或更高版本的存储群集；（本文我们采用的是M版本）
RHEL/CentOS 7.5或者Linux kernel v4.16或更新；
同时拥有以下几个包：
targetcli-2.1.fb47 or    newer package
python-rtslib-2.1.fb64 or    newer package
tcmu-runner-1.3.0 or    newer package
ceph-iscsi-config-2.4 or    newer package
ceph-iscsi-cli-2.5 or    newer package
注意：我们本次安装ISCSI GW是在OSD 节点上安装
由于很多童鞋可能获取不到rpm包，所以我这里整理了一下所需要的rpm并且分享了出来，以下是链接
链接: https://pan.baidu.com/s/1piNymp6oWLNHvXKaxBszLg 密码: 4vag
安装部分就不说了只要把rpm包下载到本地，安装很简单，下面来说下配置部分：
首先创建一个iscsi gw的配置文件
# touch /etc/ceph/iscsi-gateway.cfg
里面内容如下：
[config]
# Name of the Ceph storage cluster. A suitable Ceph configuration file allowing
# access to the Ceph storage cluster from the gateway node is required, if not
# colocated on an OSD node.
cluster_name = ceph

# Place a copy of the ceph cluster's admin keyring in the gateway's /etc/ceph
# drectory and reference the filename here
gateway_keyring = ceph.client.admin.keyring

# API settings.
# The API supports a number of options that allow you to tailor it to your
# local environment. If you want to run the API under https, you will need to
# create cert/key files that are compatible for each iSCSI gateway node, that is
# not locked to a specific node. SSL cert and key files *must* be called
# 'iscsi-gateway.crt' and 'iscsi-gateway.key' and placed in the '/etc/ceph/' directory
# on *each* gateway node. With the SSL files in place, you can use 'api_secure = true'
# to switch to https mode.

# To support the API, the bear minimum settings are:
api_secure = false

# Additional API configuration options are as follows, defaults shown.
# api_user = admin
# api_password = admin
# api_port = 5001
# trusted_ip_list = 192.168.0.10,192.168.0.11
注意：需要修改的地方就行把trusted_ip_list替换成自己实际场景的节点ip
完事之后重启iscsi服务
# systemctl daemon-reload
# systemctl enable rbd-target-api
# systemctl start rbd-target-api
OK到此为止，iscsi gw已经安装配置完成，下面开始创建iscsi target

1.执行gwcli命令进入iscsi

2.创建iscsi target
> /> cd /iscsi-target
> /iscsi-target>  create iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw

3.创建iSCSI网关。以下使用的IP是用于iSCSI数据传输的IP,它们可以与trusted_ip_list中列出的用于管理操作的IP相同，也可以不同，看有没有做多网卡分离
> /iscsi-target> cd iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/gateways
> /iscsi-target...-igw/gateways>  create ceph-gw-1 10.172.19.21 skipchecks=true
> /iscsi-target...-igw/gateways>  create ceph-gw-2 10.172.19.22 skipchecks=true

4.创建一个rbd设备disk_1
> /iscsi-target...-igw/gateways> cd /disks
> /disks> create pool=rbd image=disk_1 size=90G

5.创建iqn
> /disks> cd /iscsi-target/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts
> /iscsi-target...eph-igw/hosts>  create iqn.1994-05.com.redhat:rh7-client

6.创建chap的用户名密码，并且chap必须设置，否则服务端是禁止连接的
> /iscsi-target...at:rh7-client>  auth chap=myiscsiusername/myiscsipassword

7.增加磁盘到客户端
> /iscsi-target...at:rh7-client> disk add rbd.disk_1

OK，到这里就配置完成了，我们看下最终是什么样的
iscsi.png

安装配置完之后就可以挂载使用了，一般也就是windows/Linux，我们先说下Linux下挂载方式

1.安装iscsi工具
# yum install iscsi-initiator-utils
# yum install device-mapper-multipath

2.配置多路径
# mpathconf --enable --with_multipathd y
devices {
      device {
            vendor                "LIO-ORG"
            hardware_handler    "1 alua"
            path_grouping_policy "failover"
            path_selector       "queue-length 0"
            failback             60
            path_checker          tur
            prio                alua
            prio_args             exclusive_pref_bit
            fast_io_fail_tmo    25
            no_path_retry       queue
      }
}

3.重启服务
# systemctl reload multipathd

4.配置chap认证
修改配置客户端的名称为之前设置的名称
# vi /etc/iscsi/initiatorname.iscsi
InitiatorName= iqn.1994-05.com.redhat:rh7-client
修改chap认证配置文件
# vi /etc/iscsi/iscsid.conf
node.session.auth.authmethod = CHAP
node.session.auth.username = myiscsiusername
node.session.auth.password = myiscsipassword

5.查询iscsi target
# iscsiadm -m discovery -t -st 10.172.19.21

6.连接target
# iscsiadm -m node -T iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw -l
# multipath –ll

admin · 发表于 2021-7-10 21:21:50

[root@ceph01 ~]# vi /etc/yum.repos.d/CentOS-Base.repo

[Storage]
name=CentOS-$releasever - Storage
) L. a3 c' W Y6 W$ Pbaseurl=https://mirrors.huaweicloud.com/centos/7.7.1908/storage/x86_64/ceph-nautilus/
$ `( p+ \+ q8 I' f) l( xgpgcheck=0
enabled=1

[Gluster]
name=CentOS-$releasever - Storage
0 {6 u- Y$ O- z5 m, c+ Pbaseurl=https://mirrors.huaweicloud.com/centos/7.7.1908/storage/x86_64/gluster-5// t0 h+ y7 v% M3 z
gpgcheck=0
enabled=1

[ceph-iscsi]
name=CentOS-$releasever - Storage. O# G6 h9 k o' G4 c+ g% C
baseurl=https://mirrors.huaweicloud.com/ceph/ceph-iscsi/3/rpm/el7/noarch/
5 g0 @/ o6 Z" _, ?gpgcheck=0
enabled=1

[epel]
name=CentOS-$releasever - Storage/ M* Y. R4 w4 l
baseurl=https://mirrors.huaweicloud.com/epel/7Server/x86_64/1 C; \' `6 ^5 v; M
gpgcheck=0
enabled=1

admin · 发表于 2021-7-10 21:37:05

直接yum install试试可以发现，只有targetcli和python-rtslib能装上，而且版本都比官网说的要低，好吧，麻烦来了。

经过一段几个小时*的搜索……从下面链接中找到了rpm包：

Build new RPM for 3.0
Missing ceph-iscsi-cli package
新建repo文件：

sudo vim /etc/yum.repo.d/iscsi.repo
cat /etc/yum.repo.d/iscsi.repo

[ceph-iscsi]
name=Ceph-iscsi
baseurl=https://4.chacra.ceph.com/r/ceph-iscsi/master/88f3f67981c7da15448f140f711a1a8413d450b0/centos/7/flavors/default/noarch/
priority=1
gpgcheck=0

[tcmu-runner]
name=tcmu-runner
baseurl=https://3.chacra.ceph.com/r/tcmu-runner/master/eef511565078fb4e2ed52caaff16e6c7e75ed6c3/centos/7/flavors/default/x86_64/
priority=1
gpgcheck=0

[python-rtslib]
name=python-rtslib
baseurl=https://2.chacra.ceph.com/r/python-rtslib/master/67eb1605c697b6307d8083b2962f5170db13d306/centos/7/flavors/default/noarch/
priority=1
gpgcheck=0
这里我使用的是本地源，将上面的包下载到本地源：

sudo yum install --downloadonly --downloaddir=yum/ceph-iscsi/ targetcli python-rtslib tcmu-runner ceph-iscsi
createrepo -p -d -o yum/ yum/
注意到这里没有包含targetcli的repo，因为没有找到，使用yum基础的Base源或者是Ceph源可以安装targetcli-2.1.fb46-7.el7.noarch.rpm，虽然官网需要的是targetcli-2.1.fb47 or newer package，但在后续使用中发现没有影响，所以这里就不用管targetcli了。

那么这里下载下来的就是：

targetcli-2.1.fb46-7.el7.noarch.rpm
python-rtslib-2.1.fb68-1.noarch.rpm
tcmu-runner-1.4.0-0.1.51.geef5115.el7.x86_64.rpm
ceph-iscsi-3.2-8.g88f3f67.el7.noarch.rpm
ceph-iscsi网关初始配置
如果使用的不是集群内的节点作为ceph-iscsi网关，那就需要进行一些初始的配置。

安装ceph。
从集群中的一台机器上拷贝/etc/ceph/ceph.conf到本机的/etc/ceph/ceph.conf。
从集群中的一台机器上拷贝/etc/ceph/ceph.client.admin.keyring到本机的/etc/ceph/ceph.client.admin.keyring。
当然第2和3步可以直接在deploy节点使用ceph-deploy admin {node-gateway}，{node-gateway}就表示网关节点的名字。

可见这里就是为了将ceph-iscsi网关节点变成一个admin节点。

这时在网关节点上应该可以执行相关命令操作ceph集群，例如sudo ceph -s查询当前集群的状态。

安装配置iscsi
这里官网建议先切换到root用户，方便一点：

su root
在两个网关节点上都安装iscsi（注意到上面已经将相关包下载到了本地源，所以可以直接yum安装）：

yum install -y ceph-iscsi
服务启动：

先创建rbd pool，如果没有的话。

ceph osd lspools

1 cephfs_data
2 cephfs_metadata

ceph osd pool create rbd 128

pool 'rbd' created
创建并修改/etc/ceph/iscsi-gateway.cfg文件：

vim /etc/ceph/iscsi-gateway.cfg
cat /etc/ceph/iscsi-gateway.cfg

[config]
Name of the Ceph storage cluster. A suitable Ceph configuration file allowing
access to the Ceph storage cluster from the gateway node is required, if not
colocated on an OSD node.
cluster_name = ceph

Place a copy of the ceph cluster's admin keyring in the gateway's /etc/ceph
drectory and reference the filename here
gateway_keyring = ceph.client.admin.keyring

API settings.
The API supports a number of options that allow you to tailor it to your
local environment. If you want to run the API under https, you will need to
create cert/key files that are compatible for each iSCSI gateway node, that is
not locked to a specific node. SSL cert and key files *must* be called
'iscsi-gateway.crt' and 'iscsi-gateway.key' and placed in the '/etc/ceph/' directory
on *each* gateway node. With the SSL files in place, you can use 'api_secure = true'
to switch to https mode.

To support the API, the bear minimum settings are:
api_secure = false

Additional API configuration options are as follows, defaults shown.
api_user = admin
api_password = admin
api_port = 5001
trusted_ip_list = 192.168.90.234,192.168.90.235
上面trusted_ip_list填写的就是两台网关的ip（这里不讨论多网卡的情况）。

在另外一台网关上复制这个文件：

sudo scp cluster@node2:/etc/ceph/iscsi-gateway.cfg /etc/ceph/iscsi-gateway.cfg
在两台网关上都开启rbd-target-api服务：

systemctl daemon-reload
systemctl enable rbd-target-api
systemctl start rbd-target-api
配置：（在其中一台网关配置就行）

gwcli
cd /iscsi-targets

create iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
ok

cd iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/gateways

/iscsi-target...-igw/gateways> create node2 192.168.90.234
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
ok

/iscsi-target...-igw/gateways> create node3 192.168.90.235
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
ok

/iscsi-target...-igw/gateways> cd /disks

create pool=rbd image=disk_1 size=200G
ok

cd /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts

/iscsi-target...csi-igw/hosts> create iqn.1994-05.com.redhat:rh7-client
ok

/iscsi-target...at:rh7-client> auth username=myiscsiusername password=myiscsipassword
ok

/iscsi-target...at:rh7-client> disk add rbd/disk_1
ok
配置完成，可以看到我当前的目录结构：

ls
o- / ......................................................................... [...]
  o- cluster ......................................................... [Clusters: 1]
  | o- ceph .......................................................... [HEALTH_WARN]
  | o- pools .......................................................... [Pools: 3]
  | | o- cephfs_data ... [(x3), Commit: 0.00Y/2028052096K (0%), Used: 2029431878b]
  | | o- cephfs_metadata .... [(x3), Commit: 0.00Y/2028052096K (0%), Used: 77834b]
  | | o- rbd ................ [(x3), Commit: 200G/2028052096K (10%), Used: 15352b]
  | o- topology ................................................ [OSDs: 6,MONs: 3]
  o- disks ........................................................ [200G, Disks: 1]
  | o- rbd ............................................................ [rbd (200G)]
  | o- disk_1 ................................................ [rbd/disk_1 (200G)]
  o- iscsi-targets ............................... [DiscoveryAuth: None, Targets: 1]
o- iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ..................... [Gateways: 2]
   o- disks .......................................................... [Disks: 1]
   | o- rbd/disk_1 ............................................... [Owner: node3]
   o- gateways ............................................ [Up: 2/2, Portals: 2]
   | o- node2 ............................................. [192.168.90.234 (UP)]
   | o- node3 ............................................. [192.168.90.235 (UP)]
   o- host-groups .................................................. [Groups : 0]
   o- hosts .............................................. [Hosts: 1: Auth: CHAP]
      o- iqn.1994-05.com.redhat:rh7-client .......... [Auth: CHAP, Disks: 1(200G)]
      o- lun 0 ................................ [rbd/disk_1(200G), Owner: node3]
linux客户端配置
在作为客户端的linux主机上。

安装相关组件：

sudo yum install -y iscsi-initiator-utils
sudo yum install -y device-mapper-multipath
开启multipathd服务并进行配置：

sudo mpathconf --enable --with_multipathd y

sudo vim /etc/multipath.conf
sudo cat /etc/multipath.conf

devices {
      device {
            vendor                "LIO-ORG"
            hardware_handler    "1 alua"
            path_grouping_policy "failover"
            path_selector       "queue-length 0"
            failback             60
            path_checker          tur
            prio                alua
            prio_args             exclusive_pref_bit
            fast_io_fail_tmo    25
            no_path_retry       queue
      }
}
修改客户端名称：

sudo vim /etc/iscsi/initiatorname.iscsi
sudo cat /etc/iscsi/initiatorname.iscsi

InitiatorName=iqn.1994-05.com.redhat:rh7-client
修改chap认证配置文件：

sudo vim /etc/iscsi/iscsid.conf
sudo cat /etc/iscsi/iscsid.conf

...
*************
CHAP Settings
*************

To enable CHAP authentication set node.session.auth.authmethod
to CHAP. The default is None.
node.session.auth.authmethod = CHAP

To set a CHAP username and password for initiator
authentication by the target(s), uncomment the following lines:
node.session.auth.username = myiscsiusername
node.session.auth.password = myiscsipassword
...

发现target：

sudo iscsiadm -m discovery -t st -p 192.168.90.234

192.168.90.234:3260,1 iqn.2003-01.org.linux-iscsi.rheln1
192.168.90.235:3260,2 iqn.2003-01.org.linux-iscsi.rheln1
登入target：

sudo iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.rheln1 -l
查看是否成功：

sudo multipath -ll
mpatha (360014050fedd563975249adb2e84e978) dm-2 LIO-ORG ,TCMU device
size=200G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='queue-length 0' prio=50 status=active
| `- 3:0:0:0 sdc 8:32 active ready running
`-+- policy='queue-length 0' prio=10 status=enabled
  `- 2:0:0:0 sdb 8:16 active ready running
在fdisk中就可以直接看到这个“硬盘”：

sudo fdisk -l

...
Disk /dev/mapper/mpatha: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 524288 bytes

		自动登录	找回密码
密码			注册

ceph luminous 提供iscsi服务

配置yum文件ceph-iscsi

浏览过的版块