Shell, sed and awk scripting

Posted on Edited on In , Views:

Shell

Special variables in Shell:

  • $0 - The filename of the current script.
  • $# - The number of arguments supplied to a script.
  • $* - All the arguments are double quoted. If a script receives two arguments, $* is equivalent to $1 $2.
  • $@ - All the arguments are individually double quoted. If a script receives two arguments, $@ is equivalent to $1 $2.
  • $? - The exit status of the last command executed.
  • $$ - The process number of the current shell. For shell scripts, this is the process ID under which they are executing.
  • $! - The process number of the last background command.

Enable ssh passwordless login for multiple servers:

$ rpm -ivh sshpass-1.06-1.el7.x86_64.rpm

$ for i in `seq 0 239`
do
    sshpass -p "password" ssh-copy-id -i /root/.ssh/id_rsa.pub -o StrictHostKeyChecking=no server$i
done

Sed

$ sed -n 5p <file>

Replace specific line with new string:

$ sed -i '2s/.*/aa/' /file/path

Add line after string match:

$ sed -i '/SERVER = str1/a SERVER = "str2"' /file/path

Remove strings from beginning of lines:

# Remove 40 characters from beginning of lines
$ cat file | sed -r 's/.{40}//'

Remove leading and tailing spaces:

$ echo $diskstats
252       2 dev/dev1029144966969016657 150 0 1200 8031 2297170 2606 165332568 46106742 0 684017 44948515 5 0 72 4
$ echo $diskstats | awk '{$1="";$2="";$3="";print}' | sed 's/^[ \t]*//;s/[ \t]*$//;'
150 0 1200 8031 2297170 2606 165332568 46106742 0 684017 44948515 5 0 72 4

Replace multiple digits in a string

$ echo fio.read_16k_8jobs_20210927_103626.out | sed 's/fio\.//g;s/\.out//g;' | awk '{gsub("_[0-9]{5,}","",$1); print}'
read_16k_8jobs

awk

Use awk regex to match variable passed from shell

str=abc
awk  '/'$str'/ {print $0 }' filename

Or

str=abc
awk  -v pattern=$str '$0 ~ pattern{print $0 }' filename

Misc

download a file

$ sudo curl -vL -o testfile.zip 'https://example.com/remote_testfile.zip'

wget directory:

$ wget -r -np -R "index.html*" http://example.com/configs/.vim/

Sync Mac Address during boot

$ cat /etc/rc.d/rc.local
new_mac=`cat /sys/class/net/eno16780032/address`
sed -i "s/HWADDR=.*/HWADDR=$new_mac/" /etc/sysconfig/network-scripts/ifcfg-eno16780032
service network restart

Check command exit status

for pool in $pools 
do 
    for vol in $vols
    do 
        pxctl v i $vol | grep $pool > /dev/null 2>&1
        if [ $? -eq 0 ]; then echo $vol; break; fi
    done
done