How to uninstall ceph storage cluster
Check the pools, images and OSDs
[ceph: root@host1 /]$ ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 83.83411 root default
-3 27.94470 host host1
0 ssd 3.49309 osd.0 up 1.00000 1.00000
1 ssd 3.49309 osd.1 up 1.00000 1.00000
2 ssd 3.49309 osd.2 up 1.00000 1.00000
3 ssd 3.49309 osd.3 up 1.00000 1.00000
4 ssd 3.49309 osd.4 up 1.00000 1.00000
5 ssd 3.49309 osd.5 up 1.00000 1.00000
6 ssd 3.49309 osd.6 up 1.00000 1.00000
7 ssd 3.49309 osd.7 up 1.00000 1.00000
-5 27.94470 host host2
8 ssd 3.49309 osd.8 up 1.00000 1.00000
9 ssd 3.49309 osd.9 up 1.00000 1.00000
10 ssd 3.49309 osd.10 up 1.00000 1.00000
11 ssd 3.49309 osd.11 up 1.00000 1.00000
12 ssd 3.49309 osd.12 up 1.00000 1.00000
13 ssd 3.49309 osd.13 up 1.00000 1.00000
14 ssd 3.49309 osd.14 up 1.00000 1.00000
15 ssd 3.49309 osd.15 up 1.00000 1.00000
-7 27.94470 host host3
16 ssd 3.49309 osd.16 up 1.00000 1.00000
17 ssd 3.49309 osd.17 up 1.00000 1.00000
18 ssd 3.49309 osd.18 up 1.00000 1.00000
19 ssd 3.49309 osd.19 up 1.00000 1.00000
20 ssd 3.49309 osd.20 up 1.00000 1.00000
21 ssd 3.49309 osd.21 up 1.00000 1.00000
22 ssd 3.49309 osd.22 up 1.00000 1.00000
23 ssd 3.49309 osd.23 up 1.00000 1.00000
[ceph: root@host1 /]$ ceph osd lspools
1 device_health_metrics
2 datapool
[ceph: root@host1 /]$ rbd showmapped
id pool namespace image snap device
0 datapool rbdvol1 - /dev/rbd0
1 datapool rbdvol2 - /dev/rbd1
2 datapool rbdvol3 - /dev/rbd2
3 datapool rbdvol4 - /dev/rbd3
Remove the images and pools
[ceph: root@host1 /]$ rbd unmap /dev/rbd0
[ceph: root@host1 /]$ rbd unmap /dev/rbd1
[ceph: root@host1 /]$ rbd unmap /dev/rbd2
[ceph: root@host1 /]$ rbd unmap /dev/rbd3
[ceph: root@host1 /]$ rbd showmapped
[ceph: root@host1 /]$ rbd rm datapool/rbdvol1
Removing image: 100% complete...done.
[ceph: root@host1 /]$ rbd rm datapool/rbdvol2
Removing image: 100% complete...done.
[ceph: root@host1 /]$ rbd rm datapool/rbdvol3
Removing image: 100% complete...done.
[ceph: root@host1 /]$ rbd rm datapool/rbdvol4
Removing image: 100% complete...done.
[ceph: root@host1 /]$ ceph osd pool rm datapool datapool --yes-i-really-really-mean-it
Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool
[ceph: root@host1 /]$ ceph tell mon.\* injectargs '--mon-allow-pool-delete=true'
mon.host1: mon_allow_pool_delete = 'true'
mon.host1: {}
mon.host3: mon_allow_pool_delete = 'true'
mon.host3: {}
mon.host2: mon_allow_pool_delete = 'true'
mon.host2: {}
[ceph: root@host1 /]$ ceph osd pool rm datapool datapool --yes-i-really-really-mean-it
pool 'datapool' removed
Remove the OSDs
[ceph: root@host1 /]$ for i in `seq 0 23`
> do
> ceph osd down $i && ceph osd destroy $i --force
> done
marked down osd.0.
destroyed osd.0
[omitted...]
[ceph: root@host1 /]$ ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 83.83411 root default
-3 27.94470 host host1
0 ssd 3.49309 osd.0 destroyed 1.00000 1.00000
1 ssd 3.49309 osd.1 destroyed 1.00000 1.00000
2 ssd 3.49309 osd.2 destroyed 1.00000 1.00000
3 ssd 3.49309 osd.3 destroyed 1.00000 1.00000
4 ssd 3.49309 osd.4 destroyed 1.00000 1.00000
5 ssd 3.49309 osd.5 destroyed 1.00000 1.00000
6 ssd 3.49309 osd.6 destroyed 1.00000 1.00000
7 ssd 3.49309 osd.7 destroyed 1.00000 1.00000
-5 27.94470 host host2
8 ssd 3.49309 osd.8 destroyed 1.00000 1.00000
9 ssd 3.49309 osd.9 destroyed 1.00000 1.00000
10 ssd 3.49309 osd.10 destroyed 1.00000 1.00000
11 ssd 3.49309 osd.11 destroyed 1.00000 1.00000
12 ssd 3.49309 osd.12 destroyed 1.00000 1.00000
13 ssd 3.49309 osd.13 destroyed 1.00000 1.00000
14 ssd 3.49309 osd.14 destroyed 1.00000 1.00000
15 ssd 3.49309 osd.15 destroyed 1.00000 1.00000
-7 27.94470 host host3
16 ssd 3.49309 osd.16 destroyed 1.00000 1.00000
17 ssd 3.49309 osd.17 destroyed 1.00000 1.00000
18 ssd 3.49309 osd.18 destroyed 1.00000 1.00000
19 ssd 3.49309 osd.19 destroyed 1.00000 1.00000
20 ssd 3.49309 osd.20 destroyed 1.00000 1.00000
21 ssd 3.49309 osd.21 destroyed 1.00000 1.00000
22 ssd 3.49309 osd.22 destroyed 1.00000 1.00000
23 ssd 3.49309 osd.23 up 1.00000 1.00000
Remove the cluster hosts
[ceph: root@host1 /]$ ceph orch host rm host3
Removed host 'host3'
[ceph: root@host1 /]$ ceph orch host rm host2
Removed host 'host2'
[ceph: root@host1 /]$ ceph orch host rm host1
Removed host 'host1'
Check if there is ceph daemon running
[ceph: root@host1 /]$ ceph orch ps host3
No daemons reported
[ceph: root@host1 /]$ ceph orch ps host2
No daemons reported
[ceph: root@host1 /]$ ceph orch ps host1
No daemons reported
Remove the ceph storage cluster
[root@host1 ~]$ cephadm rm-cluster --fsid fec2332e-1b0b-11ec-abbe-ac1f6bc8d268 --force
[root@host1 ~]$ cephadm ls
[]
Cleanup the ceph configuration files
[root@host1 ~]$ rm -rf /etc/ceph
[root@host1 ~]$ rm -rf /var/lib/ce
ceph/ cephadm/ certmonger/
[root@host1 ~]$ rm -rf /var/lib/ceph*
Cleanup the ceph block devices
Do the following on each cluster node.
[root@host1 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 3.5T 0 disk
├─nvme0n1p3 259:4 0 3.5T 0 part
│ ├─vgroot-lvswap01 253:1 0 4G 0 lvm
│ └─vgroot-lvroot 253:0 0 3.5T 0 lvm /
├─nvme0n1p1 259:2 0 1G 0 part /boot/efi
└─nvme0n1p2 259:3 0 500M 0 part /boot
nvme3n1 259:6 0 3.5T 0 disk
└─ceph--ab144c40--73d6--49bc--921b--65025c383bb1-osd--block--2b965e29--b194--4363--8c96--20ab5b97db33 253:3 0 3.5T 0 lvm
nvme2n1 259:5 0 3.5T 0 disk
└─ceph--b1ffe76d--1043--43a2--848b--6ba117e71a75-osd--block--0d6ff85d--9c49--43a0--98a3--c519fbb20b9c 253:4 0 3.5T 0 lvm
nvme1n1 259:1 0 3.5T 0 disk
[root@host1 ~]$ for i in `seq 2 9`; do dd if=/dev/zero of=/dev/nvme${i}n1 bs=1M count=1000; done
[root@host1 ~]$ reboot