BASH Script Performace

Today we will look at some bash code snippests and the performance issues. Lets first look at the problem and the implemented solution:

Problem: We needed to log the output of the ps command for all the process’s. This was required to be done on per minute basis and the output was required in comma separated files. So, here is what was implemented:

pslog=`ps -e -opid,ppid,user,nlwp,pmem,vsz,rss,s,time,stime,pri,nice,pcp:u,args|grep -v PID|sort -r -k 13,13`
        OLD_IFS=$IFS
        IFS=$'\n'
        logarr=( $pslog )
        for LOGLINE in ${logarr[@]}
        do
                LOGLINE=`echo $LOGLINE|awk '{OFS=",";print $1,$2,$3,$4,$5,$6,$7,:$8,$9,$10,$11,$12,$13,$14}'`
                echo $LOGLINE >> output
        done
        IFS=$OLD_IFS

This was working well and there were no issues. But suddenly we started seeing issues with the reported CPU usages. We would see that whenever this script was running the CPU usage was high, specially if there were too many process’s/thread’s on the system during that time.  This code was definitely part of a very large code base, and at this point of time we did not know what was causing the issues.

Backup of files in the directory.

I was working on some scripts and the changes that I was making in the scripts was very dynamic, which I did want to keep backing up in the version control system. But for the peace of my mind, I wanted to keep a copy of the scripts, whenever it was in working state.

Since I had multiple files, so it would make more sense to have a script that could copy all the files in the current directory to “old” directory without over-writing the existing files. So, I wrote a script that would postfix the files with a number. With this approach, finally what I had was the following:

Linux hardware details.

Here is one of the scripts that I found on the net while searching for something … Note the URL for the script in the Description.

#!/bin/bash -
#===============================================================================
#
#          FILE:  linux_hw.sh
#
#         USAGE:  ./linux_hw.sh
#
#   DESCRIPTION:  http://www.howtogeek.com/howto/solaris/get-the-processor-type-on-solaris/
#
#       OPTIONS:  ---
#  REQUIREMENTS:  ---
#          BUGS:  ---
#         NOTES:  ---
#        AUTHOR: Amit Agarwal (aka), amit.agarwal@roamware.com
#       COMPANY: Roamware India Pvt Ltd
#       CREATED: 09/13/2011 03:57:34 PM IST
# Last modified: Sun Oct 30, 2011  04:59PM
#      REVISION:  ---
#===============================================================================

function linux_hw_CPU {
	typeset num=0
	typeset name=""
	typeset cores=""
	name="$( cat /proc/cpuinfo | awk -F: '
/vendor_id/ { vendor=$2 }
/model name/ { model=$2 }
/cpu MHz/ {
if( model ~ "Hz" ) {speed=""} else { speed=$2? MHz" };
print vendor, model, speed; }
		' | tail -1
	)"

        num=$(if [ -r /proc/vmware/cpuinfo ]; then awk '/pcpu/ { print NF-1 }' /proc/vmware/cpuinfo; else cat /proc/cpuinfo | grep processor| wc -l; fi)

	# ESX: mas info sobre logical/cores/packages
	if [ -r /proc/vmware/sched/ncpus ]
	then
		cores=$( echo $( cat /proc/vmware/sched/ncpus ) )
	fi

	echo $num $( echo "$name ($cores)" | enclose )
}

function enclose {
	tr -s " " | sed -e "s/^/\"/; s/$/\"/; s/\"\ /\"/; s/\ \"/\"/"
}

function linux_hw_CPU {

	typeset num=0
	typeset name=""
	typeset cores=""

	name="$(
		cat /proc/cpuinfo | awk -F: '
/vendor_id/ { vendor=$2 }
/model name/ { model=$2 }
/cpu MHz/ {
if( model ~ "Hz" ) {speed=""} else { speed=$2" MHz" };
print vendor, model, speed; }
		' | tail -1
	)"

	num=$(
		if [ -r /proc/vmware/cpuinfo ]
		then
			awk '/pcpu/ { print NF-1 }' /proc/vmware/cpuinfo
		else
			cat /proc/cpuinfo | grep processor| wc -l
		fi

	)

	if grep -q "physical id" /proc/cpuinfo || grep "siblings" /proc/cpuinfo
	then
		chip_count=$( grep "physical id" /proc/cpuinfo | sort -u | wc -l )
		chip_core=$( grep "siblings" /proc/cpuinfo | tail -1 | cut -d: -f2 )
		cores="($chip_count chips x $chip_core cores)"
	fi

	# Blades HP con
	if [ -x /sbin/hpasmcli ]
	then
		chip_name=$( /sbin/hpasmcli -s "SHOW SERVER" | grep "Name" | head -1 | cut -d: -f2 )
		chip_speed=$( /sbin/hpasmcli -s "SHOW SERVER" | grep "Speed" | head -1 | cut -d: -f2 )
		chip_core=$( /sbin/hpasmcli -s "SHOW SERVER" | grep "Core" | head -1 | cut -d: -f2 )
	fi

	# ESX: mas info sobre logical/cores/packages
	if [ -r /proc/vmware/sched/ncpus ]
	then
		cores="($( echo $( cat /proc/vmware/sched/ncpus ) ))"
	fi

	# Linux Itanium IA64
	if grep -q -i itanium /proc/cpuinfo
	then
		name="$(
		grep "vendor" /proc/cpuinfo | cut -d: -f2- | tail -1 ) $(
		grep "arch " /proc/cpuinfo | cut -d: -f2- | tail -1 ) $(
		grep "family" /proc/cpuinfo | cut -d: -f2- | tail -1 ) $(
		grep "cpu MHz" /proc/cpuinfo | cut -d: -f2- | cut -d. -f1 | tail -1 ) Mhz"

		chip_count=$( grep "physical id" /proc/cpuinfo | sort -u | wc -l )
		chip_core=$( grep "siblings" /proc/cpuinfo | tail -1 | cut -d: -f2 )
		cores="($chip_count chips x $chip_core cores)"
	fi

	echo $num $( echo "$name $cores" | enclose )
}

linux_hw_CPU

Related articles

• linux?????? (weijianhui16.wordpress.com)
• How to check the uptime of a process (or an application) running in Linux machine using shell script? (singztechmusings.wordpress.com)
• Some examples of grep command in unix (javarevisited.wordpress.com)

Add ssh key to remote host

If you are working on recent versions of the *nix OS like Fedora or Ubuntu then you would know about the commad ssh-copy-id. But if you land up using one of the older versions like Solaris or something where the command is not present, then probably you need a simpler solution to this. One of the simplest solution is with a lot of assumtions, simply copy the id_rsa file to remote server and hope it works. And here is  a script to do just that:

using `!#$’ to referance backward-word

Here is something that I found on the commandlinefu

cp /work/host/phone/ui/main.cpp !#$:s/host/target

Ah well…its very interesting and very useful.

Some explanation:

!# – means the current command (similar to !! being the last command)

$ – represents the last parameter

:s – is used for substitution.

Related articles

• CommandLineFu: Amazing Collection Of Command Line Gems [Linux & Mac] (makeuseof.com)
• Oh, the Iron-y (ftalphaville.ft.com)

Compare files excluding certain lines.

Quick tip, you can use any expression for the sed commands in the (). With this trick you can redirect the stdout of 2 commands to the diff command. This might become very useful, if you want to compare 2 files, excluding the first  line.

diff <(sed '1d' file) <(sed '1d' file2)

More interesting example is where the string ABC is converted to abc before comparing in the second file with the following command:

Cont: Get yourself some more conkyrc files.

Last time we got ourselves some conkyrc files from the ubuntu forums. But that scripts gets the files only from the First page of the thread. Lets extend this further and get the script to get all the conkyrc files.

There are some 1048 pages in the thread, I am showing pages 1 to 3 but you can change 3 to whatever number you want 🙂

count=0
for i in {1..3}
do
    >.test
    >conkyrc
	echo "Getting page $i"
	curl http://ubuntuforums.org/showthread.php?t=281865\&page=$i |\\
         sed -n '// d'|\
         sed 's##\n-----------------------------------\n\n\n#' \
         >conkyrc
	dos2unix conkyrc
	cp conkyrc .test
	while [ $(wc -l .test|sed 's/[^0-9]//g') != 0 ]
	do
		sed -n '1,/------------------------/ p' .test|sed '$d' >conkyrc.$count
		diff .test conkyrc.$count |sed 's/^<.//'|sed '1, /---------------------/ d;2d'>.test
		((count++))
	done
	echo "Files so far are : $count"
done

Related articles

• Get yourself some conkyrc files. (amit-agarwal.co.in)
• HTC-Like Clock / Weather Conky Configuration (putokaz.wordpress.com)
• Split and join file on Mac OSX Terminal (lezeng.wordpress.com)