FlamingBytes

The Priority Queue is implemented with binary heap. To understand how it’s implemented, refer to this post. Python provides a built-in implementation of the Priority Queue data structure.

For insertion, it uses the put function. The get function dequeues the highest priority item from the queue.

To use the Priority Queue class object:

1
2
3

from queue import PriorityQueue
    
q = PriorityQueue()

To insert a tuple pair which has a priority associated with it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

q.put((2, "Joe"))
q.put((1, "Tim"))
q.put((4, "Watson"))

print(q.get())

print(q.empty())
print(q.full())

q.put((3, "Michael"))
print(q.get())

while not q.empty():
    print(q.get())

print(q.empty())

Output:

(1,'Tim')
False
False
(2,'Joe')
(3,'Michael')
(4,"watson)
True

To simply insert values as priority:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

q.put(12)
q.put(20)
print(q.qsize())
q.put(15)
print(q.qsize())
while not q.empty():
    print(q.get())
print(q.empty())

Output:

2
3
12
15
20
True

Priority Queue is an extension of Queue with following properties

• each item has a priority associated with it
• the item with high priority is dequeued before other item with low priority

Binary heap is generally preferred implementation for Priority Queue. It takes time O(log n) to re-heapify the remaining items if the min/max item is removed.The same amount of time O(log n) is taken to insert an new item in the heap.

The key functions to implement in Priority Queue include:

• insert(val)
• extractMin()/extractMax()
• remove(i)
• getMin()/getMax()
• changePriority(i, val)

Implementation in Python:

def left_child(k):
    return 2 * k + 1


def right_child(k):
    return 2 * k + 2


def is_leaf(k):
    return 2 * k >= len(mylist)


def parent(k):
    return (k - 1) // 2


def getMin():
    return mylist[0]


def heapify(k):
    if is_leaf(k):
        return

    n = len(mylist)

    # get left and right child index
    left = left_child(k)
    right = right_child(k)

    # find the smallest elements among current node, its left child and right child if exists
    smallest = k
    if left < n and mylist[k] > mylist[left]:
        smallest = left
    if right < n and mylist[smallest] > mylist[right]:
        smallest = right

    # swap current node and the smallest child
    if smallest != k:
        mylist[k], mylist[smallest] = mylist[smallest], mylist[k]
        heapify(smallest)


def shift_up(i):
    while i > 0 and mylist[i] < mylist[parent(i)]:
        mylist[i], mylist[parent(i)] = mylist[parent(i)], mylist[i]
        i = parent(i)


def insert(val):
    mylist.append(val)
    shift_up(len(mylist) - 1)


def extractMin():
    min = mylist[0]
    mylist[0] = mylist[len(mylist) - 1]
    mylist.pop(len(mylist) - 1)
    heapify(0)
    return min


def remove(i):
    mylist[i] = getMin() - 1
    shift_up(i)
    extractMin()


def print_heap():
    for i in range(len(mylist) // 2):
        left = 2 * i + 1
        right = 2 * i + 2
        if left < len(mylist) and right < len(mylist):
            print(
                f"parent: {mylist[i]} left-child: {mylist[left]} right-child: {mylist[right]}"
            )
        elif left < len(mylist):
            print(f"parent: {mylist[i]} left-child: {mylist[left]} right-child: None")
            
mylist = []            

Build the binary heap:

insert(6)
insert(4)
insert(8)
insert(2)
insert(9)
insert(7)
print(mylist)

Output:
[2,4,7,6,9,8]


         4        4         2         2          2
4  -->  /   -->  / \  -->  / \  -->  / \  -->  /    \ 
       6        6   8     4   8     4   8     4     7
                         /         / \       / \   /
                        6         6   9     6   9 8

Insert a new item:

insert(1)
print(mylist)

Output:
[1,4,2,6,9,8,7]

      2             2             1
    /   \         /   \         /   \
   4     7  -->  4     1  -->  4     2
  / \   / \     / \   / \     / \   / \
 6   9 8   1   6   9 8   7   6   9 8   7

Extract the min item:

extractMin()
print(mylist)

Output:
[2,4,7,6,9,8]

      1             7             2
    /   \         /   \         /   \
   4     2  -->  4     2  -->  4     7
  / \   / \     / \   /       / \   / 
 6   9 8   7   6   9  8      6   9 8

Remove the item at specified position:

remove(1)
print(mylist)

Output:
[2,6,7,8,9]

       2                1(2-1)         8              2              2
     /   \            /   \          /   \          /   \          /   \
    4     7  -->     2     7  -->   2     7  -->   8     7  -->   6     7
   / \   /          / \   /        / \            / \            / \
  6   9 8          6   9 8        6   9          6   9          8   9

What is min-heap and max-heap

A heap is a data structure based on binary tree. It’s designed to efficiently access the smallest or largest element in a collection of items. It follows properties of a complete binary tree. So, it’s also known as binary heap.

A min-heap(max-heap) is a binary tree such that

• The data in each node is less(greater) than or equal to the data in node’s children.
• The binary tree is complete. A complete binary tree means all levels are completely filled except the last level and the last level has all nodes as left as possible.

How is binary heap represented

A binary heap is typically represented as an array.

• The root node is arr[0]
• The parent node for the ith node is arr[(i-1)/2]
• the left child node for the ith node is arr[2*i+1]
• the right child node for the ith node is arr[2*i+2]

Level order traversal is used to achieve the array representation.

Example:

binary tree:
              6(0)
            /      \
          4(1)     8(2)
         /  \     /
       2(3) 9(4) 7(5)
       
array: [6,4,8,2,9,7]

Application of heap

The heap is broadly used to solve the following problems. We will learn more in future posts. Here we will be focus on the heap implementation.

• Heap sort in O(nlogn) time
• Priority queue
• Graph algorithms
• Many other problems

What is heapify

Heapify is the process to build the heap data structure using binary tree.

We start the process from the first index of non-leaf node.

     6             6             6             2             2
   /   \         /   \         /   \         /   \         /   \
  4     8  -->  4     7  -->  2     7  -->  6     7  -->  4     7
 / \   /       / \   /       / \   /       / \   /       / \   /
2   9 7       2   9 8       4   9 8       4   9 8       6   9 8

Implementaion

def left_child(mylist, k):
    return 2 * k + 1


def right_child(mylist, k):
    return 2 * k + 2


def is_leaf(mylist, k):
    return 2 * k >= len(mylist)


def heapify(mylist, k):
    if is_leaf(mylist, k):
        return

    n = len(mylist)

    # get left and right child index
    left = left_child(mylist, k)
    right = right_child(mylist, k)

    # find the smallest elements among current node, left and right child
    smallest = k
    if left < n and mylist[k] > mylist[left]:
        smallest = left
    if right < n and mylist[smallest] > mylist[right]:
        smallest = right

    # swap current node and the smallest child
    if smallest != k:
        mylist[k], mylist[smallest] = mylist[smallest], mylist[k]
        heapify(mylist, smallest)


def build_min_heap(mylist):
    n = len(mylist) // 2 - 1
    for i in range(n, -1, -1):
        heapify(mylist, i)


def parent(k):
    return (k - 1) // 2


def insert(mylist, val):
    mylist.append(val)

    k = len(mylist) - 1
    while k > 0 and mylist[k] < mylist[parent(k)]:
        mylist[k], mylist[parent(k)] = mylist[parent(k)], mylist[k]
        k = parent(k)


def remove(mylist):
    min = mylist[0]
    mylist[0] = mylist[len(mylist) - 1]
    mylist.pop(len(mylist) - 1)
    heapify(mylist, 0)
    return min


def print_heap(mylist):
    for i in range(len(mylist) // 2):
        left = 2 * i + 1
        right = 2 * i + 2
        if left < len(mylist) and right < len(mylist):
            print(
                f"parent: {mylist[i]} left-child: {mylist[left]} right-child: {mylist[right]}"
            )
        elif left < len(mylist):
            print(f"parent: {mylist[i]} left-child: {mylist[left]} right-child: None")

test_list = []
insert(test_list, 6)
insert(test_list, 4)
insert(test_list, 8)
insert(test_list, 2)
insert(test_list, 9)
insert(test_list, 7)
print(test_list)
print_heap(test_list)

test_list_1 = [6, 4, 8, 2, 9, 7]
build_min_heap(test_list_1)
print(test_list_1)
print_heap(test_list_1)

remove(test_list_1)
print(test_list_1)
print_heap(test_list_1)

What is queue

Like stack, queue is also a linear data structure in which the items are stored in a First-In-First-Out(FIFO)manner.

The queue may include the following key functions.

• enqueue() - add an item to the end of the queue
• dequeue() - remove the front item
• size() - return the number of items in queue
• peek() - return the front item
• is_empty() - check if the queue is empty

How to implement

There are various ways to implement a queue in Python. In this post, we will learn how to implement it with the following ways.

• list
• collections.deque
• queue.Queue
• linked list

using list

In Python, list is like dynamic array. The items in list are stored contiguously in memory. Thus, the random access to list is as fast as array. However, inserting/deleting item in front of the queue is slow because it requires shifting other items with time complexity O(n).

st = []
st.append(1)
st.append(2)
st.append(3)
print("stack size: {}".format(len(st)))
print(st.pop(0))
print(st.pop(0))
print(st.pop(0))
print("stack size: {}".format(len(st)))

using collections.deque

deque is pronounced “deck” and stands for “double-ended queue”. It is built upon a doubly linked list which allows insert/remove nodes from both ends with constant time O(1). Deque is the prefered method to implement a queue in Python. Note that deque is not thread-safe.

from collections import deque

st = deque()
st.append(1)
st.append(2)
st.append(3)
print("stack size: {}".format(len(st)))
print(st.popleft())
print(st.popleft())
print(st.popleft())
print("stack size: {}".format(len(st)))

using queue.Queue

If you need multi-threading, you can use queue.Queue module. However, it comes at a performance cost to achieve the thread-safety.

from queue import Queue

st = Queue()
st.put(1)
st.put(2)
st.put(3)
print("stack size: {}".format(st.qsize()))
print(st.get())
print(st.get())
print(st.get())
print("stack size: {}".format(st.qsize()))

using linked list

You also can implement your own queue with the linked list data structure.

class Node:
    def __init__(self, val):
        self.val = val
        self.next = None


class MyQueue:
    def __init__(self):
        self.front = self.rear = None
        self.size = 0

    def get_size(self):
        return self.size

    def is_empty(self):
        return self.size == 0

    def enqueue(self, val):
        n = Node(val)
        if self.rear == None:
            self.front = n
            self.rear = n
        else:
            self.rear.next = n
            self.rear = n

        self.size += 1

    def dequeue(self):
        if self.is_empty():
            raise Exception("empty stack")

        val = self.front.val
        self.front = self.front.next
        self.size -= 1
        return val

    def __str__(self):
        curr = self.front
        st_str = ""
        while curr:
            st_str += str(curr.val) + "->"
            curr = curr.next
        return st_str[:-2]


qu = MyQueue()
qu.enqueue(1)
qu.enqueue(2)
qu.enqueue(3)
print("queue: {}".format(qu))
print("queue size: {}".format(qu.size))
print(qu.dequeue())
print(qu.dequeue())
print(qu.dequeue())
print("queue size: {}".format(qu.size))

What is stack

Stack is a linear data structure in which the items are in a Last-In-First-Out(LIFO)manner.

The stack may include the following key functions.

• push() - add an item to the top of the stack
• pop() - remove the item at the top
• size() - return the number of items in stack
• peek() - return the item from top of the stack
• is_empty() - check if the stack is empty

How to implement

There are various ways to implement a stack in Python. In this post, we will learn how to implement it with the following ways.

• list
• collections.deque
• queue.LifoQueue
• linked list

using list

In Python, list is like dynamic array. The items in list are stored contiguously in memory. Thus, the random access to list is as fast as array. However, as it grows bigger than the allocated memory it currently holds, it needs to have more memory allocation to increase the capacity dynamically. In such case, the stack append call could take longer time.

st = []
st.append(1)
st.append(2)
st.append(3)
print("stack size: {}".format(len(st)))
print(st.pop())
print(st.pop())
print(st.pop())
print("stack size: {}".format(len(st)))

Using deque

deque is pronounced “deck” and stands for “double-ended queue”. It is built upon a doubly linked list which allows insert/remove nodes from both ends with constant time(O(1)). Deque is the prefered method to implement a stack in Python. Note that deque is not thread-safe.

from collections import deque

st = deque()
st.append(1)
st.append(2)
st.append(3)
print("stack size: {}".format(len(st)))
print(st.pop())
print(st.pop())
print(st.pop())
print("stack size: {}".format(len(st)))

Using LifoQueue

If you need multi-threading, you can use queue.LifoQueue module. However, it comes at a performance cost to achieve the thread-safety.

from queue import LifoQueue

st = LifoQueue()
st.put(1)
st.put(2)
st.put(3)
print("stack size: {}".format(st.qsize()))
print(st.get())
print(st.get())
print(st.get())
print("stack size: {}".format(st.qsize()))

Using linked list

You also can implement your own stack with the linked list data structure.

class Node:
    def __init__(self, val):
        self.val = val
        self.next = None


class Stack:
    def __init__(self):
        self.top = None
        self.size = 0

    def get_size(self):
        return self.size

    def is_empty(self):
        return self.size == 0

    def push(self, val):
        n = Node(val)
        if self.top == None:
            self.top = n
        else:
            n.next = self.top
            self.top = n

        self.size += 1

    def pop(self):
        if self.is_empty():
            raise Exception("empty stack")

        val = self.top.val
        self.top = self.top.next
        self.size -= 1
        return val

    def peek(self):
        if self.is_empty():
            raise Exception("empty stack")

        return self.top.val

    def __str__(self):
        curr = self.top
        st_str = ""
        while curr:
            st_str += str(curr.val) + "->"
            curr = curr.next
        return st_str[:-2]


st = Stack()
st.push(1)
st.push(2)
st.push(3)
print("Stack: {}".format(st))
print("stack size: {}".format(st.size))
print("stack top: {}".format(st.peek()))
print(st.pop())
print(st.pop())
print(st.pop())
print("stack size: {}".format(st.size))

aa

1. ls - The most frequently used command in Linux to list directories
2. pwd - Print working directory command in Linux
3. cd - Linux command to navigate through directories

mkdir - Command used to create directories in Linux

mv - Move or rename files in Linux

cp - Similar usage as mv but for copying files in Linux

rm - Delete files or directories

touch - Create blank/empty files

ln - Create symbolic links (shortcuts) to other files

cat - Display file contents on the terminal

clear - Clear the terminal display

echo - Print any text that follows the command

less - Linux command to display paged outputs in the terminal

man - Access manual pages for all Linux commands

uname - Linux command to get basic information about the OS

whoami - Get the active username

tar - Command to extract and compress files in Linux

grep - Search for a string within an output

head - Return the specified number of lines from the top

tail - Return the specified number of lines from the bottom

diff - Find the difference between two files

cmp - Allows you to check if two files are identical

comm - Combines the functionality of diff and cmp

sort - Linux command to sort the content of a file while outputting

export - Export environment variables in Linux

zip - Zip files in Linux

unzip - Unzip files in Linux

ssh - Secure Shell command in Linux

service - Linux command to start and stop services

ps - Display active processes

kill and killall - Kill active processes by process ID or name

df - Display disk filesystem information

mount - Mount file systems in Linux

chmod - Command to change file permissions

chown - Command for granting ownership of files or folders

ifconfig - Display network interfaces and IP addresses

traceroute - Trace all the network hops to reach the destination

wget - Direct download files from the internet

ufw - Firewall command

iptables - Base firewall for all other firewall utilities to interface with

apt, pacman, yum, rpm - Package managers depending on the distro

sudo - Command to escalate privileges in Linux

cal - View a command-line calendar

alias - Create custom shortcuts for your regularly used commands

dd - Majorly used for creating bootable USB sticks

whereis - Locate the binary, source, and manual pages for a command

whatis - Find what a command is used for

top - View active processes live with their system usage

useradd and usermod - Add new user or change existing users data

passwd - Create or update passwords for existing users

https://www.digitalocean.com/community/tutorials/linux-commands

Before upgrade

$ cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

$ uname -r
3.10.0-693.el7.x86_64

$ cat /boot/grub2/grubenv
saved_entry=CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)

Install the newer kernel package

$ rpm -ivh kernel-ml-5.7.12-1.el7.elrepo.x86_64.rpm
warning: kernel-ml-5.7.12-1.el7.elrepo.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID baadae52: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:kernel-ml-5.7.12-1.el7.elrepo    ################################# [100%]

$ rpm -ivh kernel-ml-devel-5.7.12-1.el7.elrepo.x86_64.rpm
warning: kernel-ml-devel-5.7.12-1.el7.elrepo.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID baadae52: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:kernel-ml-devel-5.7.12-1.el7.elre################################# [100%]

$  rpm -qa | grep kernel
kernel-ml-devel-5.7.12-1.el7.elrepo.x86_64
kernel-3.10.0-693.el7.x86_64
kernel-tools-3.10.0-693.el7.x86_64
kernel-ml-5.7.12-1.el7.elrepo.x86_64
kernel-tools-libs-3.10.0-693.el7.x86_64

Set the default boot kernel entry

Notes: if /etc/grub2.cfg does not exist, do next step to rebuild the GRUB2 and come back later after that.

$ awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg
CentOS Linux (5.7.12-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-8ef36acf9f544b90bf0621450fe05f75) 7 (Core)

$ grub2-set-default 0 
$ grep saved /boot/grub2/grubenv
saved_entry=0

Rebuild the GRUB2 configuration

$ grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.7.12-1.el7.elrepo.x86_64
Found initrd image: /boot/initramfs-5.7.12-1.el7.elrepo.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-693.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-693.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-8ef36acf9f544b90bf0621450fe05f75
Found initrd image: /boot/initramfs-0-rescue-8ef36acf9f544b90bf0621450fe05f75.img
done

Reboot the system

$ reboot

Verify the new kernel version

$ uname -r
5.7.12-1.el7.elrepo.x86_64
$ cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

Reference

• https://www.golinuxcloud.com/set-default-boot-kernel-version-old-previous-rhel-linux/
• https://www.tecmint.com/install-upgrade-kernel-version-in-centos-7/

Check the pools, images and OSDs

[ceph: root@host1 /]$ ceph osd tree
ID  CLASS  WEIGHT    TYPE NAME             STATUS  REWEIGHT  PRI-AFF
-1         83.83411  root default
-3         27.94470      host host1
 0    ssd   3.49309          osd.0             up   1.00000  1.00000
 1    ssd   3.49309          osd.1             up   1.00000  1.00000
 2    ssd   3.49309          osd.2             up   1.00000  1.00000
 3    ssd   3.49309          osd.3             up   1.00000  1.00000
 4    ssd   3.49309          osd.4             up   1.00000  1.00000
 5    ssd   3.49309          osd.5             up   1.00000  1.00000
 6    ssd   3.49309          osd.6             up   1.00000  1.00000
 7    ssd   3.49309          osd.7             up   1.00000  1.00000
-5         27.94470      host host2
 8    ssd   3.49309          osd.8             up   1.00000  1.00000
 9    ssd   3.49309          osd.9             up   1.00000  1.00000
10    ssd   3.49309          osd.10            up   1.00000  1.00000
11    ssd   3.49309          osd.11            up   1.00000  1.00000
12    ssd   3.49309          osd.12            up   1.00000  1.00000
13    ssd   3.49309          osd.13            up   1.00000  1.00000
14    ssd   3.49309          osd.14            up   1.00000  1.00000
15    ssd   3.49309          osd.15            up   1.00000  1.00000
-7         27.94470      host host3
16    ssd   3.49309          osd.16            up   1.00000  1.00000
17    ssd   3.49309          osd.17            up   1.00000  1.00000
18    ssd   3.49309          osd.18            up   1.00000  1.00000
19    ssd   3.49309          osd.19            up   1.00000  1.00000
20    ssd   3.49309          osd.20            up   1.00000  1.00000
21    ssd   3.49309          osd.21            up   1.00000  1.00000
22    ssd   3.49309          osd.22            up   1.00000  1.00000
23    ssd   3.49309          osd.23            up   1.00000  1.00000

[ceph: root@host1 /]$ ceph osd lspools
1 device_health_metrics
2 datapool

[ceph: root@host1 /]$ rbd showmapped
id  pool      namespace  image    snap  device
0   datapool             rbdvol1  -     /dev/rbd0
1   datapool             rbdvol2  -     /dev/rbd1
2   datapool             rbdvol3  -     /dev/rbd2
3   datapool             rbdvol4  -     /dev/rbd3

Remove the images and pools

[ceph: root@host1 /]$ rbd unmap /dev/rbd0
[ceph: root@host1 /]$ rbd unmap /dev/rbd1
[ceph: root@host1 /]$ rbd unmap /dev/rbd2
[ceph: root@host1 /]$ rbd unmap /dev/rbd3

[ceph: root@host1 /]$ rbd showmapped

[ceph: root@host1 /]$ rbd rm datapool/rbdvol1
Removing image: 100% complete...done.
[ceph: root@host1 /]$ rbd rm datapool/rbdvol2
Removing image: 100% complete...done.
[ceph: root@host1 /]$ rbd rm datapool/rbdvol3
Removing image: 100% complete...done.
[ceph: root@host1 /]$ rbd rm datapool/rbdvol4
Removing image: 100% complete...done.

[ceph: root@host1 /]$ ceph osd pool rm datapool datapool --yes-i-really-really-mean-it
Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool

[ceph: root@host1 /]$ ceph tell mon.\* injectargs '--mon-allow-pool-delete=true'
mon.host1: mon_allow_pool_delete = 'true'
mon.host1: {}
mon.host3: mon_allow_pool_delete = 'true'
mon.host3: {}
mon.host2: mon_allow_pool_delete = 'true'
mon.host2: {}
[ceph: root@host1 /]$ ceph osd pool rm datapool datapool --yes-i-really-really-mean-it
pool 'datapool' removed

Remove the OSDs

[ceph: root@host1 /]$ for i in `seq 0 23`
> do
> ceph osd down $i && ceph osd destroy $i --force
> done
marked down osd.0.
destroyed osd.0
[omitted...]

[ceph: root@host1 /]$  ceph osd tree
ID  CLASS  WEIGHT    TYPE NAME             STATUS     REWEIGHT  PRI-AFF
-1         83.83411  root default
-3         27.94470      host host1
 0    ssd   3.49309          osd.0         destroyed   1.00000  1.00000
 1    ssd   3.49309          osd.1         destroyed   1.00000  1.00000
 2    ssd   3.49309          osd.2         destroyed   1.00000  1.00000
 3    ssd   3.49309          osd.3         destroyed   1.00000  1.00000
 4    ssd   3.49309          osd.4         destroyed   1.00000  1.00000
 5    ssd   3.49309          osd.5         destroyed   1.00000  1.00000
 6    ssd   3.49309          osd.6         destroyed   1.00000  1.00000
 7    ssd   3.49309          osd.7         destroyed   1.00000  1.00000
-5         27.94470      host host2
 8    ssd   3.49309          osd.8         destroyed   1.00000  1.00000
 9    ssd   3.49309          osd.9         destroyed   1.00000  1.00000
10    ssd   3.49309          osd.10        destroyed   1.00000  1.00000
11    ssd   3.49309          osd.11        destroyed   1.00000  1.00000
12    ssd   3.49309          osd.12        destroyed   1.00000  1.00000
13    ssd   3.49309          osd.13        destroyed   1.00000  1.00000
14    ssd   3.49309          osd.14        destroyed   1.00000  1.00000
15    ssd   3.49309          osd.15        destroyed   1.00000  1.00000
-7         27.94470      host host3
16    ssd   3.49309          osd.16        destroyed   1.00000  1.00000
17    ssd   3.49309          osd.17        destroyed   1.00000  1.00000
18    ssd   3.49309          osd.18        destroyed   1.00000  1.00000
19    ssd   3.49309          osd.19        destroyed   1.00000  1.00000
20    ssd   3.49309          osd.20        destroyed   1.00000  1.00000
21    ssd   3.49309          osd.21        destroyed   1.00000  1.00000
22    ssd   3.49309          osd.22        destroyed   1.00000  1.00000
23    ssd   3.49309          osd.23               up   1.00000  1.00000

Remove the cluster hosts

[ceph: root@host1 /]$ ceph orch host rm host3
Removed host 'host3'
[ceph: root@host1 /]$ ceph orch host rm host2
Removed host 'host2'
[ceph: root@host1 /]$ ceph orch host rm host1
Removed host 'host1'

Check if there is ceph daemon running

[ceph: root@host1 /]$ ceph orch ps host3
No daemons reported
[ceph: root@host1 /]$ ceph orch ps host2
No daemons reported
[ceph: root@host1 /]$ ceph orch ps host1
No daemons reported

Remove the ceph storage cluster

[root@host1 ~]$ cephadm rm-cluster --fsid fec2332e-1b0b-11ec-abbe-ac1f6bc8d268 --force
[root@host1 ~]$ cephadm ls
[]

Cleanup the ceph configuration files

[root@host1 ~]$ rm -rf /etc/ceph
[root@host1 ~]$ rm -rf /var/lib/ce
ceph/       cephadm/    certmonger/
[root@host1 ~]$ rm -rf /var/lib/ceph*

Cleanup the ceph block devices

Do the following on each cluster node.

[root@host1 ~]$ lsblk
NAME                                                                                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1                                                                                               259:0    0  3.5T  0 disk
├─nvme0n1p3                                                                                           259:4    0  3.5T  0 part
│ ├─vgroot-lvswap01                                                                                   253:1    0    4G  0 lvm
│ └─vgroot-lvroot                                                                                     253:0    0  3.5T  0 lvm  /
├─nvme0n1p1                                                                                           259:2    0    1G  0 part /boot/efi
└─nvme0n1p2                                                                                           259:3    0  500M  0 part /boot
nvme3n1                                                                                               259:6    0  3.5T  0 disk
└─ceph--ab144c40--73d6--49bc--921b--65025c383bb1-osd--block--2b965e29--b194--4363--8c96--20ab5b97db33 253:3    0  3.5T  0 lvm
nvme2n1                                                                                               259:5    0  3.5T  0 disk
└─ceph--b1ffe76d--1043--43a2--848b--6ba117e71a75-osd--block--0d6ff85d--9c49--43a0--98a3--c519fbb20b9c 253:4    0  3.5T  0 lvm
nvme1n1                                                                                               259:1    0  3.5T  0 disk

[root@host1 ~]$ for i in `seq 2 9`; do dd if=/dev/zero of=/dev/nvme${i}n1 bs=1M count=1000; done
[root@host1 ~]$ reboot

This post tries to provide general steps for system performance analysis. We are going to cover the following objectives.

• The performance goals
• Workload characterization
• Drill-down analysis
• Traditional performance tools
• BCC/BPF performance tools

The Goals of performance analysis

In general, the goals of performance analysis are to improve end-user performance and reduce the operating cost. To achieve this, it’s important to make the performance measurable. We often use the following metrics to measure performance.

• Rates - operation or request rates per second
• Throughput - data transferred per second
• Latency - the time to accomplish a operation or request in milliseconds
• Resource utilization - the resource usage in percentage
• Cost - Price/performance ratio

The rates, throughput and latency are usually the most important metrics to check if the certain performance goal is met. For example, the throughput measured in MiB/s for a daily database backup is too slow to complete in a given backup window. We need to investigate the issue from the backup application to system layer in order to find a solution to improve the performance. For a second example, the latency at cloud native storage volume layer is as high as 20ms while the underneath SSD disk latency shows less than 1ms. This requires further analysis at volume layer in order to find out the cause for the 19ms latency.

Performance optimization is endless effort. It depends on the goal you are targeting. So, setting the goals is the first step before you involve further performance analysis activities.

Workload characterization

Performance analysis is a process to analyze systematically. Understanding the system/application configuration and applied workload are often needed before you do further performance analysis. This is the workload characterization.

The workload characterization tries to answer the following questions:

• What’s the running application? What’s the major components/features used in the application?
• What is the schedule to run the workload? What is the job concurrency?
• What are the read/write patterns? Mixed read and write, or read/write only workload?
• What are the rates, throughput and latency at application level?
• What is the performance concern?

Sometimes, you may get a description of the workload from end-users. However, the workload and its configuraiton are usualaly not described clearly enough by the users. It’s worth to characterize the workload with custom profiler. An application level workload profiler can be developed for this purpose. But this often requires application expertise. At system level, you may leverage the BCC/BPF performance tools to profile the workload.

Drill-down analysis

The drill-down analysis is to find a clue and drill deeper until you find the root cause for the performance issue.

The general process for drill-down analysis would be like these steps.

1. Examine the high level performance metrics and identify the degraded performance point
2. For the target workload point with degradation, lazer focus on the four major system resources(CPU, memory, disk I/O and network) to see what is the potential bottleneck
3. If it’s hardware bottleneck, it might be resolved by scaling up and scaling out the system resources. Otherwise, it could be a software bottleneck either from kernel space or user space.
4. Find a clue based on the collected metrics to drill down to the next level. Software bottleneck analysis often requires profiling and tracing effort to pinpoint the culprit.

To identify a hardware bottleneck, you would check if any of the four major resources are saturated. For example, the system must be CPU bound if the CPU utilization is above 90%. The system must be disk I/O bound if the disk is 100% busy and wait queue is unexpected large.

It’s likely that you could not find the root cause with one round of analysis if you go with the wrong direction. You have to repeat the above steps to identify the right direction for RCA. Keep in mind, finding a needle in haystack is not an easy work. You must be patient.

Traditional performance tools

During the drill-down performance analysis, you can use the following Linux built-in tools. They are simple but very powerful to help determine the next direction on the way of performance analysis.

• uptime - system loads in past 1 minute, 5 minutes and 15 minutes
• dmesg - system error messages
• vmstat - overview of system resource usage(CPU, Memory and disk/network I/O)
• mpstat - per-CPU usage in different states
• pidstat - CPU usage per process
• iostat - disk I/O statistics(throughput, IOPS, latency, etc)
• netstat/sar - network throughput, TCP/IP connection stats
• top - CPU/Memory usage per process and more

Please refer to this post for more detail on how to use Linux traditional tools to analyze performance.

BCC/BPF performance tools

While the traditional tools always gives us a first look at the system performance especially on the resource usage, we can use(or create) BCC/BPF tools for further performance analysis. Please refer to this post for more detail.

Intro to AWS CLI

The AWS Command Line Interface (AWS CLI) is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.

AWS CLI install

To install AWS CLI on MacOS:

$ curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
$ sudo installer -pkg AWSCLIV2.pkg -target /
$ which aws
/usr/local/bin/aws
$ aws --version
aws-cli/2.10.3 Python/3.9.11 Darwin/20.4.0 exe/x86_64 prompt/off

Configure AWS credentials

The AWS CLI stores sensitive credential information that you specify with aws configure in a local file ~/.aws/credentials.

$ aws configure

AWS Access Key ID [****************6N7Q]: [Your AWS access Key ID]

AWS Secret Access Key [****************+Ic+]: [Your AWS Secret Access Key]

Default region name [None]: [Your Region Name]

Default output format [None]:

$ cat ~/.aws/credentials

[default]

aws_access_key_id = [Your AWS access Key ID]

aws_secret_access_key = [Your AWS Secret Access Key]

aws_region = [Your Region Name]

AWS CLI Commands

To get commands help:

$ aws eks help
$ aws ec2 help

To list and describe clusters:

$ aws eks list-clusters
$ aws eks describe-cluster --name [your-cluster-name]

To list and describe nodegroups:

$ aws eks list-nodegroups --cluster-name [your-cluster-name]
$ aws eks describe-nodegroup --cluster-name [your-cluster-name] --nodegroup-name [you-nodegroup-name]

To delete multiple EBS volumes(example):

$ aws ec2 describe-volumes --filters "Name=tag:Name,Values=gp3-volume-*"  | egrep "VolumeId" | awk '{print $NF}' | sed 's/\"//g;s/\,//' > vol_ids

$ for id in `cat vol_ids`
do
    aws ec2 delete-volume --volume-id $id
done

$ aws ec2 describe-volumes --filters "Name=tag:Name,Values=gp3-volume-*"  | egrep "VolumeId" | awk '{print $NF}'  | wc -l
0

To create multiple EBS volumes(example):

$ for i in `seq 1 8`
do
aws ec2 create-volume --volume-type gp3 --size 256 --iops 16000 --throughput 1000 --availability-zone us-east-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=gp3-volume-$i}]"
done

To attach EBS volume to EC2 instance:

$ aws ec2 describe-volumes --filters "Name=tag:Name,Values=gp3-volume-1" --query "Volumes[*].{ID:VolumeId}"
[
    {
        "ID": "vol-0101002b66d4fc211"
    }
]

$ aws ec2 attach-volume --volume-id vol-0101002b66d4fc211 --instance-id i-0c2b7553a99a7277b --device /dev/sdf
{
    "AttachTime": "2023-03-04T01:58:37.155000+00:00",
    "Device": "/dev/sdf",
    "InstanceId": "i-0c2b7553a99a7277b",
    "State": "attaching",
    "VolumeId": "vol-0101002b66d4fc211"
}

To describe instance:

$ aws ec2 describe-instances --instance-ids i-0c2b7553a99a7277b | egrep "DeviceName|Status|VolumeId"
"DeviceName": "/dev/sdf",
"Status": "attached",
"VolumeId": "vol-0101002b66d4fc211"
"Status": "attached"

To verify the attached volumes in EC2 instance:

[ec2-user@ip-192-168-25-183 ~]$ ls -la /dev/sdf
lrwxrwxrwx 1 root root 7 Mar  4 01:58 /dev/sdf -> nvme1n1

[ec2-user@ip-192-168-25-183 ~]$ lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1       259:0    0  256G  0 disk
├─nvme0n1p1   259:1    0  256G  0 part /
└─nvme0n1p128 259:2    0    1M  0 part
nvme1n1       259:3    0  256G  0 disk

Reference

• https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html