In this article, we learn to deploy ceph cluster on ubuntu 18.04. Three nodes are used for this study.
We target to deploy the most recent ceph release which is called Pacific. With this release, we can use cephadm to create a ceph cluster by bootstrapping on a single host and expanding the cluster to additional hosts.
Intro to ceph
Whether you want to provide Ceph Object Storage and/or Ceph Block Device services to Cloud Platforms, deploy a Ceph Filesystem or use Ceph for another purpose, all Ceph Storage Cluster deployments begin with setting up each Ceph Node, your network, and the Ceph Storage Cluster. A Ceph Storage Cluster requires at least one Ceph Monitor, Ceph Manager, and Ceph OSD (Object Storage Daemon). The Ceph Metadata Server is also required when running Ceph Filesystem clients.
Monitors: A Ceph Monitor (ceph-mon) maintains maps of the cluster state, including the monitor map, manager map, the OSD map, and the CRUSH map. These maps are critical cluster state required for Ceph daemons to coordinate with each other. Monitors are also responsible for managing authentication between daemons and clients. At least three monitors are normally required for redundancy and high availability.
Managers: A Ceph Manager daemon (ceph-mgr) is responsible for keeping track of runtime metrics and the current state of the Ceph cluster, including storage utilization, current performance metrics, and system load. The Ceph Manager daemons also host python-based plugins to manage and expose Ceph cluster information, including a web-based dashboard and REST API. At least two managers are normally required for high availability.
Ceph OSDs: A Ceph OSD (object storage daemon, ceph-osd) stores data, handles data replication, recovery, rebalancing, and provides some monitoring information to Ceph Monitors and Managers by checking other Ceph OSD Daemons for a heartbeat. At least 3 Ceph OSDs are normally required for redundancy and high availability.
MDSs: A Ceph Metadata Server (MDS, ceph-mds) stores metadata on behalf of the Ceph Filesystem (i.e., Ceph Block Devices and Ceph Object Storage do not use MDS). Ceph Metadata Servers allow POSIX file system users to execute basic commands (like ls, find, etc.) without placing an enormous burden on the Ceph Storage Cluster.
Ceph stores data as objects within logical storage pools. Using the CRUSH algorithm, Ceph calculates which placement group should contain the object, and further calculates which Ceph OSD Daemon should store the placement group. The CRUSH algorithm enables the Ceph Storage Cluster to scale, rebalance, and recover dynamically.
Deploy a ceph storage cluster
Prepare Ubuntu Linux and packages
From ceph installation guide, the following system requirements must be met before deployment.
Python 3
Systemd
Podman or Docker for running containers
Time synchronization (such as chrony or NTP)
LVM2 for provisioning storage devices
root@host1:~# cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
Upgrade to Ubuntu 18.0.4
root@host1:~# apt install update-manager-core
root@host1:~# do-release-upgrade -c
Checking for a new Ubuntu release
New release '18.04.5 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
root@host1:~# do-release-upgrade
root@host1:~# cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
Install python3
root@host1:~# apt-get install python3
Install docker
Refer to here
Install ntp
root@host1:~# apt-get install ntp
root@host1:~# service ntp start
root@host1:~# timedatectl set-timezone UTC
Install lvm2
root@host1:~# apt-get install lvm2
Check and disable firewall status
root@host1:~# ufw status
Add cluster nodes to /etc/hosts
Configure passwordless ssh from primary host to the others
Install cephadm
The cephadm command can
bootstrap a new cluster
launch a containerized shell with a working Ceph CLI
aid in debugging containerized Ceph daemons
root@host1:# curl –silent –remote-name –location https://github.com/ceph/ceph/raw/pacific/src/cephadm/cephadm
root@host1:# ls
cephadm
root@host1:~# chmod +x cephadm
root@host1:# ./cephadm add-repo –release pacific
root@host1:# ./cephadm install
root@host1:~# which cephadm
/usr/sbin/cephadm
Bootstrap a new cluster
The first step in creating a new Ceph cluster is running the cephadm bootstrap command on the Ceph cluster’s first host. The act of running the cephadm bootstrap command on the Ceph cluster’s first host creates the Ceph cluster’s first “monitor daemon”, and that monitor daemon needs an IP address. You must pass the IP address of the Ceph cluster’s first host to the ceph bootstrap command, so you’ll need to know the IP address of that host.
root@host1:~# cephadm bootstrap --mon-ip <host1-ip> --allow-fqdn-hostname
Ceph Dashboard is now available at:
URL: https://host1:8443/
User: admin
Password: btauef87vj
Enabling client.admin keyring and conf on hosts with "admin" label
You can access the Ceph CLI with:
sudo /usr/sbin/cephadm shell --fsid ad30a6fc-068f-11ec-8323-000c29bf98ea -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
Please consider enabling telemetry to help improve Ceph:
ceph telemetry on
For more information see:
https://docs.ceph.com/docs/pacific/mgr/telemetry/
Bootstrap complete.
root@host1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a946ae868dbc prom/alertmanager:v0.20.0 "/bin/alertmanager -…" 6 minutes ago Up 6 minutes ceph-ad30a6fc-068f-11ec-8323-000c29bf98ea-alertmanager.host1
504d9271b24c ceph/ceph-grafana:6.7.4 "/bin/sh -c 'grafana…" 6 minutes ago Up 6 minutes ceph-ad30a6fc-068f-11ec-8323-000c29bf98ea-grafana.host1
622a5e234406 prom/prometheus:v2.18.1 "/bin/prometheus --c…" 6 minutes ago Up 6 minutes ceph-ad30a6fc-068f-11ec-8323-000c29bf98ea-prometheus.host1
6c2b0440d4c1 prom/node-exporter:v0.18.1 "/bin/node_exporter …" 6 minutes ago Up 6 minutes ceph-ad30a6fc-068f-11ec-8323-000c29bf98ea-node-exporter.host1
8bc618e9ffa3 ceph/ceph "/usr/bin/ceph-crash…" 6 minutes ago Up 6 minutes ceph-ad30a6fc-068f-11ec-8323-000c29bf98ea-crash.host1
b57a021238ba ceph/ceph:v16 "/usr/bin/ceph-mgr -…" 7 minutes ago Up 7 minutes ceph-ad30a6fc-068f-11ec-8323-000c29bf98ea-mgr.host1.ltfphc
e812853ef17d ceph/ceph:v16 "/usr/bin/ceph-mon -…" 7 minutes ago Up 7 minutes ceph-ad30a6fc-068f-11ec-8323-000c29bf98ea-mon.host1
Enable ceph CLI
To execute ceph commands, you can also run commands like this:
root@host1:~# cephadm shell -- ceph -s
Inferring fsid ad30a6fc-068f-11ec-8323-000c29bf98ea
Inferring config /var/lib/ceph/ad30a6fc-068f-11ec-8323-000c29bf98ea/mon.host1/config
Using recent ceph image ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb
cluster:
id: ad30a6fc-068f-11ec-8323-000c29bf98ea
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum host1 (age 9m)
mgr: host1.ltfphc(active, since 10m)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
Cephadm does not require any Ceph packages to be installed on the host. However, it recommends enabling easy access to the ceph command.
You can install the ceph-common package, which contains all of the ceph commands, including ceph, rbd, mount.ceph (for mounting CephFS file systems), etc.:
root@host1:~# cephadm add-repo --release pacific
Installing repo GPG key from https://download.ceph.com/keys/release.gpg...
Installing repo file at /etc/apt/sources.list.d/ceph.list...
Updating package list...
Completed adding repo.
root@host1:~# cephadm install ceph-common
Installing packages ['ceph-common']...
root@host1:~# ceph -v
ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@host1:~# ceph status
cluster:
id: ad30a6fc-068f-11ec-8323-000c29bf98ea
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum host1 (age 11m)
mgr: host1.ltfphc(active, since 12m)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
Adding additional hosts to the cluster
To add each new host to the cluster, perform two steps:
Install the cluster’s public SSH key in the new host’s root user’s authorized_keys file:
root@host1:# ssh-copy-id -f -i /etc/ceph/ceph.pub root@host2
root@host1:# ssh-copy-id -f -i /etc/ceph/ceph.pub root@host3
Tell Ceph that the new node is part of the cluster:
root@host1:# ceph orch host add host2 –labels _admin
root@host1:# ceph orch host add host3 –labels _admin
Wait for a while until the monitor detects the new hosts. Verify the new added hosts as below.
root@host1:~# cat /etc/ceph/ceph.conf
# minimal ceph.conf for ad30a6fc-068f-11ec-8323-000c29bf98ea
[global]
fsid = ad30a6fc-068f-11ec-8323-000c29bf98ea
mon_host = [v2:<host2-ip>:3300/0,v1:<host2-ip>:6789/0] [v2:<host3-ip>:3300/0,v1:<host3-ip>:6789/0] [v2:<host1-ip>:3300/0,v1:<host1-ip>:6789/0]
root@host1:~# ceph status
cluster:
id: ad30a6fc-068f-11ec-8323-000c29bf98ea
health: HEALTH_WARN
clock skew detected on mon.host2, mon.host3
OSD count 0 < osd_pool_default_size 3
services:
mon: 3 daemons, quorum host1,host2,host3 (age 115s)
mgr: host1.ltfphc(active, since 30m), standbys: host2.dqlsnk
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
Adding storage
To add storage to the cluster, either tell Ceph to consume any available and unused device:
ceph orch apply osd --all-available-devices
Or Deploy OSDs with specified storage devices.
Listing storage devices
In order to deploy an OSD, there must be a storage device that is available on which the OSD will be deployed.
Run this command to display an inventory of storage devices on all cluster hosts:
root@host1:~# ceph orch device ls
Hostname Path Type Serial Size Health Ident Fault Available
host2 /dev/sdb hdd 85.8G Unknown N/A N/A Yes
host3 /dev/sdb hdd 85.8G Unknown N/A N/A Yes
host1 /dev/sdb hdd 85.8G Unknown N/A N/A Yes
A storage device is considered available if all of the following conditions are met:
- The device must have no partitions.
- The device must not have any LVM state.
- The device must not be mounted.
- The device must not contain a file system.
- The device must not contain a Ceph BlueStore OSD.
- The device must be larger than 5 GB.
Ceph will not provision an OSD on a device that is not available.
Creating new OSDs
There are a few ways to create new OSDs:
After running the above command:
- If you add new disks to the cluster, they will automatically be used to create new OSDs.
- If you remove an OSD and clean the LVM physical volume, a new OSD will be created automatically.
If you want to avoid this behavior (disable automatic creation of OSD on available devices), use the unmanaged parameter:
ceph orch apply osd --all-available-devices --unmanaged=true
For example:
ceph orch daemon add osd host1:/dev/sdb
In our case, we use the following commands to create OSDs for the three nodes. We only need run the commands from host1.
root@host1:~# ceph orch daemon add osd host1:/dev/sdb
Created osd(s) 0 on host 'host1'
root@host1:~# ceph orch daemon add osd host2:/dev/sdb
Created osd(s) 1 on host 'host2'
root@host1:~# ceph orch daemon add osd host3:/dev/sdb
Created osd(s) 2 on host 'host3'
root@host1:~# ceph status
cluster:
id: ad30a6fc-068f-11ec-8323-000c29bf98ea
health: HEALTH_WARN
clock skew detected on mon.host2, mon.host3
59 slow ops, oldest one blocked for 130 sec, mon.host2 has slow ops
services:
mon: 3 daemons, quorum host1,host2,host3 (age 2m)
mgr: host1.ltfphc(active, since 102s), standbys: host2.dqlsnk
osd: 3 osds: 3 up (since 7m), 3 in (since 7m)
data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 15 MiB used, 240 GiB / 240 GiB avail
pgs: 1 active+clean
Rry Run
The –dry-run flag causes the orchestrator to present a preview of what will happen without actually creating the OSDs.
For example:
ceph orch apply osd --all-available-devices --dry-run
Create a pool
Pools are logical partitions for storing objects. When you first deploy a cluster without creating a pool, Ceph uses the default pools for storing data.
By default, Ceph makes 3 replicas of RADOS objects. Ensure you have a realistic number of placement groups. Ceph recommends approximately 100 per OSD and always use the nearest power of 2.
root@host1:~# ceph osd lspools
1 device_health_metrics
root@host1:~# ceph osd pool create datapool 128 128
pool 'datapool' created
root@host1:~# ceph osd lspools
1 device_health_metrics
2 datapool
root@host1:~# ceph osd pool ls detail
pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 22 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth
pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 39 flags hashpspool stripe_width 0
root@host1:~# ceph osd pool get datapool all
size: 3
min_size: 2
pg_num: 128
pgp_num: 128
crush_rule: replicated_rule
hashpspool: true
nodelete: false
nopgchange: false
nosizechange: false
write_fadvise_dontneed: false
noscrub: false
nodeep-scrub: false
use_gmt_hitset: 1
fast_read: 0
pg_autoscale_mode: on
On the admin node, use the rbd tool to initialize the pool for use by RBD:
[ceph: root@host1 /]# rbd pool init datapool
Create rbd volume and map to a block device on the host
The rbd command enables you to create, list, introspect and remove block device images. You can also use it to clone images, create snapshots, rollback an image to a snapshot, view a snapshot, etc.
root@host1:~# rbd create --size 512000 datapool/rbdvol1
root@host1:~# rbd map datapool/rbdvol1
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable datapool/rbdvol1 object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
root@host1:~# dmesg | tail
[50268.015821] cgroup: cgroup: disabling cgroup2 socket matching due to net_prio or net_cls activation
[59168.019848] Key type ceph registered
[59168.020080] libceph: loaded (mon/osd proto 15/24)
[59168.023667] rbd: loaded (major 252)
[59168.028478] libceph: mon2 <host1-ip>:6789 session established
[59168.028571] libceph: mon2 <host1-ip>:6789 socket closed (con state OPEN)
[59168.028594] libceph: mon2 <host1-ip>:6789 session lost, hunting for new mon
[59175.101037] libceph: mon0 <host1-ip>:6789 session established
[59175.101413] libceph: client14535 fsid ad30a6fc-068f-11ec-8323-000c29bf98ea
[59175.105601] rbd: image rbdvol1: image uses unsupported features: 0x38
root@host1:~# rbd feature disable datapool/rbdvol1 object-map fast-diff deep-flatten
root@host1:~# rbd map datapool/rbdvol1
/dev/rbd0
root@host1:~# rbd showmapped
id pool namespace image snap device
0 datapool rbdvol1 - /dev/rbd0
root@host1:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 128G 0 disk
├─sda1 8:1 0 127G 0 part /
├─sda2 8:2 0 1K 0 part
└─sda5 8:5 0 975M 0 part
sdb 8:16 0 80G 0 disk
└─ceph--bc7eff08--2ac6--44a5--b941--5444c4a8600a-osd--block--b4dfb938--05af--413d--a327--18d26fc75b8d 253:0 0 80G 0 lvm
rbd0 252:0 0 100G 0 disk
root@host1:~# ls -la /dev/rbd/datapool/rbdvol1
lrwxrwxrwx 1 root root 10 Aug 26 19:35 /dev/rbd/datapool/rbdvol1 -> ../../rbd0
root@host1:~# ls -la /dev/rbd0
brw-rw---- 1 root disk 252, 0 Aug 26 19:35 /dev/rbd0
root@host1:~# rbd status datapool/rbdvol1
Watchers:
watcher=<host1-ip>:0/2778790200 client.14556 cookie=18446462598732840967
root@host1:~# rbd info datapool/rbdvol1
rbd image 'rbdvol1':
size 100 GiB in 25600 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 38bebe718b2f
block_name_prefix: rbd_data.38bebe718b2f
format: 2
features: layering, exclusive-lock
op_features:
flags:
create_timestamp: Thu Aug 26 19:31:29 2021
access_timestamp: Thu Aug 26 19:31:29 2021
modify_timestamp: Thu Aug 26 19:31:29 2021
Create filesystem and mount rbd volume
You can use Linux standard commands to create filesystem on the volume and mount it for different purpose.
Troubleshooting
- Ceph does not support pacific or later on centos7.8
If you are installing Ceph with version of pacific on CentOS 7.8, you may see the following issue.
$ cat /etc/*release
CentOS Linux release 7.8.2003 (Core)
NAME="CentOS Linux"
$ uname -r
5.7.12-1.el7.elrepo.x86_64
# ./cephadm add-repo --release pacific
ERROR: Ceph does not support pacific or later for this version of this linux distro and therefore cannot add a repo for it
You can install Ceph with version “octopus” instead.
$ ./cephadm add-repo --release octopus
Writing repo to /etc/yum.repos.d/ceph.repo...
Enabling EPEL...
Completed adding repo
Note: cephadm is new in Ceph release v15.2.0 (Octopus) and does not support older versions of Ceph.
Invalid GPG Key
$ ./cephadm install
Installing packages [‘cephadm’]…
Non-zero exit code 1 from yum install -y cephadm
yum: stdout Loaded plugins: fastestmirror, langpacks, priorities
yum: stdout Loading mirror speeds from cached hostfile
yum: stdout * base: pxe.dev.purestorage.com
yum: stdout * centosplus: pxe.dev.purestorage.com
yum: stdout * epel: mirror.lax.genesisadaptive.com
yum: stdout * extras: pxe.dev.purestorage.com
yum: stdout * updates: pxe.dev.purestorage.com
yum: stdout 279 packages excluded due to repository priority protections
yum: stdout Resolving Dependencies
yum: stdout –> Running transaction check
yum: stdout —> Package cephadm.noarch 2:15.2.14-0.el7 will be installed
yum: stdout –> Finished Dependency Resolution
yum: stdout
yum: stdout Dependencies Resolved
yum: stdout
yum: stdout ================================================================================
yum: stdout Package Arch Version Repository Size
yum: stdout ================================================================================
yum: stdout Installing:
yum: stdout cephadm noarch 2:15.2.14-0.el7 Ceph-noarch 55 k
yum: stdout
yum: stdout Transaction Summary
yum: stdout ================================================================================
yum: stdout Install 1 Package
yum: stdout
yum: stdout Total download size: 55 k
yum: stdout Installed size: 223 k
yum: stdout Downloading packages:
yum: stdout Public key for cephadm-15.2.14-0.el7.noarch.rpm is not installed
yum: stdout Retrieving key from https://download.ceph.com/keys/release.gpg
yum: stderr warning: /var/cache/yum/x86_64/7/Ceph-noarch/packages/cephadm-15.2.14-0.el7.noarch.rpm: Header V4 RSA/SHA256 Signature, key ID 460f3994: NOKEY
yum: stderr
yum: stderr
yum: stderr Invalid GPG Key from https://download.ceph.com/keys/release.gpg: No key found in given key data
Traceback (most recent call last):
File “./cephadm”, line 8432, in
main()
File “./cephadm”, line 8420, in main
r = ctx.func(ctx)
File “./cephadm”, line 6384, in command_install
pkg.install(ctx.packages)
File “./cephadm”, line 6231, in install
call_throws(self.ctx, [self.tool, ‘install’, ‘-y’] + ls)
File “./cephadm”, line 1461, in call_throws
raise RuntimeError(‘Failed command: %s’ % ‘ ‘.join(command))
RuntimeError: Failed command: yum install -y cephadm
Based on Ceph Documentation , execute the following to install the release.asc key.
$ rpm --import 'https://download.ceph.com/keys/release.asc'
Install cephadm package again and it succeeds.
$ ./cephadm install
Installing packages ['cephadm']...
$ which cephadm
/usr/sbin/cephadm
Failed to add host during bootstrap
$ cephadm bootstrap –mon-ip 192.168.1.183
Adding host host1…
Non-zero exit code 22 from /usr/bin/docker run –rm –ipc=host –net=host –entrypoint /usr/bin/ceph -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=host1 -v /var/log/ceph/ccc938de-0c30-11ec-8c3f-ac1f6bc8d268:/var/log/ceph:z -v /tmp/ceph-tmpdqxjp0ly:/etc/ceph/ceph.client.admin. keyring:z -v /tmp/ceph-tmpmt5hrjo9:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v15 orch host add host1
/usr/bin/ceph: stderr Error EINVAL: Failed to connect to host1 (host1).
/usr/bin/ceph: stderr Please make sure that the host is reachable and accepts connections using the cephadm SSH key
/usr/bin/ceph: stderr
/usr/bin/ceph: stderr To add the cephadm SSH key to the host:
/usr/bin/ceph: stderr > ceph cephadm get-pub-key > ~/ceph.pub
/usr/bin/ceph: stderr > ssh-copy-id -f -i ~/ceph.pub root@host1
/usr/bin/ceph: stderr
/usr/bin/ceph: stderr To check that the host is reachable:
/usr/bin/ceph: stderr > ceph cephadm get-ssh-config > ssh_config
/usr/bin/ceph: stderr > ceph config-key get mgr/cephadm/ssh_identity_key > ~/cephadm_private_key
/usr/bin/ceph: stderr > chmod 0600 ~/cephadm_private_key
/usr/bin/ceph: stderr > ssh -F ssh_config -i ~/cephadm_private_key root@host1
ERROR: Failed to add host : Failed command: /usr/bin/docker run –rm –ipc=host –net=host –entrypoint /usr/bin/ceph -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=host1 -v /var/log/ceph/ccc938de-0c30-11ec-8c3f-ac1f6bc8d268:/var/log/ceph:z -v /tmp/ ceph-tmpdqxjp0ly:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpmt5hrjo9:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v15 orch host add host1
Note: If there are multiple networks and interfaces, be sure to choose one that will be accessible by any host accessing the Ceph cluster.
Make sure passwordless ssh is configured on each host.
Remove ceph cluster
$ cephadm rm-cluster –fsid ccc938de-0c30-11ec-8c3f-ac1f6bc8d268 –force
ceph-common installation failure
$ cephadm install ceph-common
Installing packages [‘ceph-common’]…
Non-zero exit code 1 from yum install -y ceph-common
yum: stdout Loaded plugins: fastestmirror, langpacks, priorities
yum: stdout Loading mirror speeds from cached hostfile
yum: stdout * base: pxe.dev.purestorage.com
yum: stdout * centosplus: pxe.dev.purestorage.com
yum: stdout * epel: mirror.lax.genesisadaptive.com
yum: stdout * extras: pxe.dev.purestorage.com
yum: stdout * updates: pxe.dev.purestorage.com
yum: stdout 279 packages excluded due to repository priority protections
yum: stdout Resolving Dependencies
yum: stdout –> Running transaction check
yum: stdout —> Package ceph-common.x86_64 1:10.2.5-4.el7 will be installed
yum: stdout –> Processing Dependency: python-rbd = 1:10.2.5-4.el7 for package: 1:ceph-common-10.2.5-4.el7.x86_64
yum: stdout Package python-rbd is obsoleted by python3-rbd, but obsoleting package does not provide for requirements
yum: stdout –> Processing Dependency: python-rados = 1:10.2.5-4.el7 for package: 1:ceph-common-10.2.5-4.el7.x86_64
yum: stdout Package python-rados is obsoleted by python3-rados, but obsoleting package does not provide for requirements
yum: stdout –> Processing Dependency: hdparm for package: 1:ceph-common-10.2.5-4.el7.x86_64
yum: stdout –> Processing Dependency: gdisk for package: 1:ceph-common-10.2.5-4.el7.x86_64
yum: stdout –> Processing Dependency: libboost_regex-mt.so.1.53.0()(64bit) for package: 1:ceph-common-10.2.5-4.el7.x86_64
yum: stdout –> Processing Dependency: libboost_program_options-mt.so.1.53.0()(64bit) for package: 1:ceph-common-10.2.5-4.el7.x86_64
yum: stdout –> Running transaction check
yum: stdout —> Package boost-program-options.x86_64 0:1.53.0-28.el7 will be installed
yum: stdout —> Package boost-regex.x86_64 0:1.53.0-28.el7 will be installed
yum: stdout –> Processing Dependency: libicuuc.so.50()(64bit) for package: boost-regex-1.53.0-28.el7.x86_64
yum: stdout –> Processing Dependency: libicui18n.so.50()(64bit) for package: boost-regex-1.53.0-28.el7.x86_64
yum: stdout –> Processing Dependency: libicudata.so.50()(64bit) for package: boost-regex-1.53.0-28.el7.x86_64
yum: stdout —> Package ceph-common.x86_64 1:10.2.5-4.el7 will be installed
yum: stdout –> Processing Dependency: python-rbd = 1:10.2.5-4.el7 for package: 1:ceph-common-10.2.5-4.el7.x86_64
yum: stdout Package python-rbd is obsoleted by python3-rbd, but obsoleting package does not provide for requirements
yum: stdout –> Processing Dependency: python-rados = 1:10.2.5-4.el7 for package: 1:ceph-common-10.2.5-4.el7.x86_64
yum: stdout Package python-rados is obsoleted by python3-rados, but obsoleting package does not provide for requirements
yum: stdout —> Package gdisk.x86_64 0:0.8.10-3.el7 will be installed
yum: stdout —> Package hdparm.x86_64 0:9.43-5.el7 will be installed
yum: stdout –> Running transaction check
yum: stdout —> Package ceph-common.x86_64 1:10.2.5-4.el7 will be installed
yum: stdout –> Processing Dependency: python-rbd = 1:10.2.5-4.el7 for package: 1:ceph-common-10.2.5-4.el7.x86_64
yum: stdout Package python-rbd is obsoleted by python3-rbd, but obsoleting package does not provide for requirements
yum: stdout –> Processing Dependency: python-rados = 1:10.2.5-4.el7 for package: 1:ceph-common-10.2.5-4.el7.x86_64
yum: stdout Package python-rados is obsoleted by python3-rados, but obsoleting package does not provide for requirements
yum: stdout —> Package libicu.x86_64 0:50.2-4.el7_7 will be installed
yum: stdout –> Finished Dependency Resolution
yum: stdout You could try using –skip-broken to work around the problem
yum: stdout You could try running: rpm -Va –nofiles –nodigest
yum: stderr Error: Package: 1:ceph-common-10.2.5-4.el7.x86_64 (base)
yum: stderr Requires: python-rbd = 1:10.2.5-4.el7
yum: stderr Available: 1:python-rbd-10.2.5-4.el7.x86_64 (base)
yum: stderr python-rbd = 1:10.2.5-4.el7
yum: stderr Available: 2:python3-rbd-15.2.14-0.el7.x86_64 (Ceph)
yum: stderr python-rbd = 2:15.2.14-0.el7
yum: stderr Error: Package: 1:ceph-common-10.2.5-4.el7.x86_64 (base)
yum: stderr Requires: python-rados = 1:10.2.5-4.el7
yum: stderr Available: 1:python-rados-10.2.5-4.el7.x86_64 (base)
yum: stderr python-rados = 1:10.2.5-4.el7
yum: stderr Available: 2:python3-rados-15.2.14-0.el7.x86_64 (Ceph)
yum: stderr python-rados = 2:15.2.14-0.el7
Traceback (most recent call last):
File “/usr/sbin/cephadm”, line 6242, in
r = args.func()
File “/usr/sbin/cephadm”, line 5073, in command_install
pkg.install(args.packages)
File “/usr/sbin/cephadm”, line 4931, in install
call_throws([self.tool, ‘install’, ‘-y’] + ls)
File “/usr/sbin/cephadm”, line 1112, in call_throws
raise RuntimeError(‘Failed command: %s’ % ‘ ‘.join(command))
RuntimeError: Failed command: yum install -y ceph-common
cephadm log
/var/log/ceph/cephadm.log
rbd image map failed
[ceph: root@host1 /]# rbd map datapool/rbdvol1
modinfo: ERROR: Module alias rbd not found.
modprobe: FATAL: Module rbd not found in directory /lib/modules/5.7.12-1.el7.elrepo.x86_64
rbd: failed to load rbd kernel module (1)
rbd: sysfs write failed
In some cases useful info is found in syslog - try “dmesg | tail”.
rbd: map failed: (2) No such file or directory
[root@host1 ~]# modprobe rbd
[root@host1 ~]# lsmod | grep rbd
rbd 106496 0
libceph 331776 1 rbd
rbd image map failed on other cluster nodes
[ceph: root@host2 /]# rbd map datapool/rbdvol5 –id admin
2021-09-21T19:49:49.384+0000 7f91ea781500 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: sysfs write failed
2021-09-21T19:49:49.387+0000 7f91ea781500 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2021-09-21T19:49:49.387+0000 7f91ea781500 -1 AuthRegistry(0x5633b09431e0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2021-09-21T19:49:49.388+0000 7f91ea781500 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2021-09-21T19:49:49.388+0000 7f91ea781500 -1 AuthRegistry(0x7fffd357c350) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2021-09-21T19:49:49.389+0000 7f91d9b68700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2021-09-21T19:49:49.389+0000 7f91da369700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2021-09-21T19:49:49.389+0000 7f91dab6a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2021-09-21T19:49:49.389+0000 7f91ea781500 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication
rbd: couldnot connect to the cluster!
In some cases useful info is found in syslog - try “dmesg | tail”.
rbd: map failed: (22) Invalid argument
[ceph: root@host2 /]# ls /etc/ceph
ceph.conf rbdmap
Copy the /etc/ceph/ceph.keyring from admin node host1 to host2
[ceph: root@host2 /]# ls /etc/ceph
ceph.conf ceph.keyring rbdmap
[ceph: root@host2 /]# rbd map datapool/rbdvol5
[ceph: root@host2 /]# rbd device list
id pool namespace image snap device
0 datapool rbdvol5 - /dev/rbd0
Reference
