Awk on Amit Agarwal Linux Blog

Apache server stats with small and useful bash script.

Fri, 29 Nov 2013 01:35:36 +0000

Just copy this script to your web-server cgi-bin directory and enjoy.

The script with show the common errors like 404 Error, Internal Server Error and others. It will show the User agent distribution using simple commands like grep, uniq, awk and so on.

You would need to change the tfile – which is temporary file and also the access.log path in the next line.

Just re-direct the output to some file with html extenstion. You could even put this in the cron which re-directs the output to some html in server document root.

cvs add files recursively – not already in repository

Tue, 12 Nov 2013 01:25:03 +0000

When you have a lot of files in some repository and you have added a couple of new, in CVS there is no command to add just the new ones to the repository, so here is a workaround for that.

cvs status 2>/dev/null | awk '{if ($1=="?")print "cvs add -kb " $2}'

Well, if you are adding text files then you might want to remove the “-kB” in the cvs command above.

BASH Script Performace

Fri, 06 Jan 2012 06:38:04 +0000

Today we will look at some bash code snippests and the performance issues. Lets first look at the problem and the implemented solution:

Problem: We needed to log the output of the ps command for all the process’s. This was required to be done on per minute basis and the output was required in comma separated files. So, here is what was implemented:

pslog=`ps -e -opid,ppid,user,nlwp,pmem,vsz,rss,s,time,stime,pri,nice,pcp:u,args|grep -v PID|sort -r -k 13,13`
        OLD_IFS=$IFS
        IFS=$'\n'
        logarr=( $pslog )
        for LOGLINE in ${logarr[@]}
        do
                LOGLINE=`echo $LOGLINE|awk '{OFS=",";print $1,$2,$3,$4,$5,$6,$7,:$8,$9,$10,$11,$12,$13,$14}'`
                echo $LOGLINE >> output
        done
        IFS=$OLD_IFS

This was working well and there were no issues. But suddenly we started seeing issues with the reported CPU usages. We would see that whenever this script was running the CPU usage was high, specially if there were too many process’s/thread’s on the system during that time. This code was definitely part of a very large code base, and at this point of time we did not know what was causing the issues.

Rekursive Grep on Solaris or AIX Systems without GNU egrep -r funcionality

Fri, 16 Dec 2011 03:00:27 +0000

If you work regularly on a Solaris or systems which do not have the “-r” (recursive grep) for grep, then you know what a lifesaver this command can be.

Here is one from command line fu:

find . -type f -exec awk '/linux/ { printf "%s %s: %s\n", FILENAME, NR, $0; }' {} \;

The benefit of using awk here is that you can print the line number also 🙂

Linux hardware details.

Mon, 28 Nov 2011 11:20:25 +0000

Here is one of the scripts that I found on the net while searching for something … Note the URL for the script in the Description.

#!/bin/bash -
#===============================================================================
#
#          FILE:  linux_hw.sh
#
#         USAGE:  ./linux_hw.sh
#
#   DESCRIPTION:  http://www.howtogeek.com/howto/solaris/get-the-processor-type-on-solaris/
#
#       OPTIONS:  ---
#  REQUIREMENTS:  ---
#          BUGS:  ---
#         NOTES:  ---
#        AUTHOR: Amit Agarwal (aka), amit.agarwal@roamware.com
#       COMPANY: Roamware India Pvt Ltd
#       CREATED: 09/13/2011 03:57:34 PM IST
# Last modified: Sun Oct 30, 2011  04:59PM
#      REVISION:  ---
#===============================================================================

function linux_hw_CPU {
	typeset num=0
	typeset name=""
	typeset cores=""
	name="$( cat /proc/cpuinfo | awk -F: '
/vendor_id/ { vendor=$2 }
/model name/ { model=$2 }
/cpu MHz/ {
if( model ~ "Hz" ) {speed=""} else { speed=$2? MHz" };
print vendor, model, speed; }
		' | tail -1
	)"

        num=$(if [ -r /proc/vmware/cpuinfo ]; then awk '/pcpu/ { print NF-1 }' /proc/vmware/cpuinfo; else cat /proc/cpuinfo | grep processor| wc -l; fi)

	# ESX: mas info sobre logical/cores/packages
	if [ -r /proc/vmware/sched/ncpus ]
	then
		cores=$( echo $( cat /proc/vmware/sched/ncpus ) )
	fi

	echo $num $( echo "$name ($cores)" | enclose )
}

function enclose {
	tr -s " " | sed -e "s/^/\"/; s/$/\"/; s/\"\ /\"/; s/\ \"/\"/"
}

function linux_hw_CPU {

	typeset num=0
	typeset name=""
	typeset cores=""

	name="$(
		cat /proc/cpuinfo | awk -F: '
/vendor_id/ { vendor=$2 }
/model name/ { model=$2 }
/cpu MHz/ {
if( model ~ "Hz" ) {speed=""} else { speed=$2" MHz" };
print vendor, model, speed; }
		' | tail -1
	)"

	num=$(
		if [ -r /proc/vmware/cpuinfo ]
		then
			awk '/pcpu/ { print NF-1 }' /proc/vmware/cpuinfo
		else
			cat /proc/cpuinfo | grep processor| wc -l
		fi

	)

	if grep -q "physical id" /proc/cpuinfo || grep "siblings" /proc/cpuinfo
	then
		chip_count=$( grep "physical id" /proc/cpuinfo | sort -u | wc -l )
		chip_core=$( grep "siblings" /proc/cpuinfo | tail -1 | cut -d: -f2 )
		cores="($chip_count chips x $chip_core cores)"
	fi

	# Blades HP con
	if [ -x /sbin/hpasmcli ]
	then
		chip_name=$( /sbin/hpasmcli -s "SHOW SERVER" | grep "Name" | head -1 | cut -d: -f2 )
		chip_speed=$( /sbin/hpasmcli -s "SHOW SERVER" | grep "Speed" | head -1 | cut -d: -f2 )
		chip_core=$( /sbin/hpasmcli -s "SHOW SERVER" | grep "Core" | head -1 | cut -d: -f2 )
	fi

	# ESX: mas info sobre logical/cores/packages
	if [ -r /proc/vmware/sched/ncpus ]
	then
		cores="($( echo $( cat /proc/vmware/sched/ncpus ) ))"
	fi

	# Linux Itanium IA64
	if grep -q -i itanium /proc/cpuinfo
	then
		name="$(
		grep "vendor" /proc/cpuinfo | cut -d: -f2- | tail -1 ) $(
		grep "arch " /proc/cpuinfo | cut -d: -f2- | tail -1 ) $(
		grep "family" /proc/cpuinfo | cut -d: -f2- | tail -1 ) $(
		grep "cpu MHz" /proc/cpuinfo | cut -d: -f2- | cut -d. -f1 | tail -1 ) Mhz"

		chip_count=$( grep "physical id" /proc/cpuinfo | sort -u | wc -l )
		chip_core=$( grep "siblings" /proc/cpuinfo | tail -1 | cut -d: -f2 )
		cores="($chip_count chips x $chip_core cores)"
	fi

	echo $num $( echo "$name $cores" | enclose )
}

linux_hw_CPU

linux?????? (weijianhui16.wordpress.com)
How to check the uptime of a process (or an application) running in Linux machine using shell script? (singztechmusings.wordpress.com)
Some examples of grep command in unix (javarevisited.wordpress.com)

Total upload and download on any interface.

Sun, 05 Jun 2011 01:29:21 +0000

Something that I have been searching for a long time, finally some look inside the <a class=“zem_slink” title=“Procfs” rel=“wikipedia” href=“http://en.wikipedia.org/wiki/Procfs" _mce_href=“http://en.wikipedia.org/wiki/Procfs">/proc got me what I wanted. Hope this will save someone’s day. So far, the only option I could think of was to run conky which becomes a little heave on some of the very old laptops I have to use sometimes 🙁

  <td>
    <div class="text codecolorer">
      &nbsp;cat /proc/net/dev|grep eth2|awk '{print $2/1024/1024"&nbsp; "$10/1024/1024}'
    </div>
  </td>
</tr>

CPUSPEED Slowing you down? (jarneil.wordpress.com)
Interface Curmudgeons versus Gestures (lingpipe-blog.com)
Alias in .bashrc (michaeldaranto.wordpress.com)

Ranking of the most frequently used commands

Mon, 31 Jan 2011 22:54:43 +0000

Lets take a quick look at how to get the most frequently used commands on you shell. So what we need to do is this:

  <td>
    <div class="text codecolorer">
      history | awk '{print $2}' | awk 'BEGIN {FS="|"}{print $1}' | sort | uniq -c | sort -n | tail | sort -nr
    </div>
  </td>
</tr>

So, how did we arrive at this and will this always work? No it might not always work. A typical example is where HISTTIMEFORMAT variable is set. In that case, if you check history, you will see that after the number column we have time and date in the specified format, in which case, you will get wrong information from the above command. Anyways, forgetting these special cases, lets go to how we got this command:

List of commands you use most often

Sun, 25 Jul 2010 03:53:08 +0000

$ history | <a class="zem_slink" title="AWK" rel="homepage" href="http://cm.bell-labs.com/cm/cs/awkbook/index.html">awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head > /tmp/cmds | <a class="zem_slink" title="Gnuplot" rel="homepage" href="http://www.gnuplot.info/">gnuplot -persist <(echo 'plot "/tmp/cmds" using 1:xticlabels(2) with boxes') Plot your most used commands with gnuplot.

<a href="http://www.commandlinefu.com/commands/view/5845/list-of-commands-you-use-most-often">View this command to comment, vote or add to favourites
<a href="http://feeds2.feedburner.com/commands/by/sthrs">View all commands by <a href="http://feeds2.feedburner.com/commands/by/sthrs">sthrs

by David Winterbottom (<a href="http://codeinthehole.com">codeinthehole.com)

concatenate compressed and uncompressed logs

Thu, 15 Jul 2010 02:10:28 +0000

<span style="font-size: x-large;"><a href="http://feedproxy.google.com/~r/Command-line-fu/~3/iwFUyltYgjM/concatenate-compressed-and-uncompressed-logs">concatenate compressed and uncompressed logs

$ find /var/log/apache2 -name 'access.loggz' -exec <a class="zem_slink freebase/guid/9202a8c04000641f800000000001abd9" title="Gzip" rel="homepage" href="http://www.gzip.org/">zcat {} ; -or -name 'access.log' -exec cat {} ; This command allows you to stream your log files, including gziped files, into one stream which can be piped to <a class="zem_slink freebase/guid/9202a8c04000641f800000000000584b" title="AWK" rel="homepage" href="http://cm.bell-labs.com/cm/cs/awkbook/index.html">awk or some other command for analysis.

Note: if your version of ’find’ supports it, use:

List of commands you use most often

Wed, 16 Jun 2010 02:17:01 +0000

<span style="font-size: x-large;"><a href="http://feedproxy.google.com/~r/Command-line-fu/~3/6MxHiSiINdo/list-of-commands-you-use-most-often">List of commands you use most often

$ history | <a class="zem_slink freebase/en/awk" title="AWK" rel="homepage" href="http://cm.bell-labs.com/cm/cs/awkbook/index.html">awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head > /tmp/cmds | gnuplot -persist <(echo 'plot "/tmp/cmds" using 1:xticlabels(2) with boxes') Plot your most used commands with gnuplot.

<a href="http://www.commandlinefu.com/commands/view/5845/list-of-commands-you-use-most-often">View this command to comment, vote or add to favourites
<a href="http://feeds2.feedburner.com/commands/by/sthrs">View all commands by <a href="http://feeds2.feedburner.com/commands/by/sthrs">sthrs

Determine what process is listening on a port on Solaris, without lsof

Tue, 08 Jun 2010 09:13:37 +0000

: Image by Metsuke iLife via Flickr

$ for x in `ptree | <a class="zem_slink freebase/en/awk" title="AWK" rel="homepage" href="http://cm.bell-labs.com/cm/cs/awkbook/index.html">awk '{print $1}’`; do pfiles $x | <a class="zem_slink freebase/en/grep" title="Grep" rel="wikipedia" href="http://en.wikipedia.org/wiki/Grep">grep ${PORT} > /dev/null 2>&1; if [ x”$?” == ”x0” ]; then ps -ef | grep $x | grep -v grep; fi; done 2> /dev/null

redet – build regular expression (regex) for any program.

Fri, 04 Jun 2010 02:06:46 +0000

Even if someone is a regex guru there are times that people get confused with some regular expression and want to either test it out or want to some help. Behold, redet is here. Lets start with installing redet:

sudo yum install redet

Once you are done with installation, just run it with redet. First you need to load the sample data, select your program and then enter the regex to test it out. Here are some screenshots to get you started.

Awk Introduction Tutorial – 7 Awk Print Examples

Wed, 19 May 2010 16:38:43 +0000

: Image by Getty Images via Daylife

awk examples

Awk Introduction Tutorial – 7 Awk Print Examples This is the first article on the new awk tutorial series. We’ll be posting several articles on awk in the upcoming weeks that will cover all features of awk with practical examples. In this article, let us review the fundamental awk working methodology along with 7 practical awk print examples. Note: Make sure you review our earlier […]

Splitting strings with IFS

Thu, 04 Feb 2010 15:58:43 +0000

Today I want to discuss splitting strings into tokens or “words”. I previously discussed how to do this with the <a href="http://bashcurescancer.com/reading-a-file-line-by-line.html">IFS variable and promised a more in depth discussion. Today, I will make the case on WHY to use IFS to split strings as opposed to using a subshell combined with <a class="zem_slink freebase/en/awk" title="AWK" rel="homepage" href="http://cm.bell-labs.com/cm/cs/awkbook/index.html">awk or cut.

I wrote this script which reads the /etc/password file line-by-line and prints the <a class="zem_slink freebase/en/user" title="User (computing)" rel="wikipedia" href="http://en.wikipedia.org/wiki/User_%28computing%29">username of any user which has a UID greater than 10 and has the shell of /sbin/nologin. Each test function performs this task 10 times to increase the length of the test:

Understand Awk Variables with 3 Practical Examples

Thu, 28 Jan 2010 08:02:21 +0000

<a id="aptureLink_CLV4LbC1Dr" style="padding: 0px 6px; float: right;" href="http://gnuwin32.sourceforge.net/packages/mawk.htm"><img style="border: 0px none;" title="Mawk for Windows" src="http://blog.amit-agarwal.com/wp-content/uploads/img.zemanta.com/ph/400x270_WebClip" alt="" width="400px" height="270px" /> This article is part of the on-going Awk Tutorial and Examples series. Like any other <a class="zem_slink freebase/en/programming_language" title="Programming language" rel="wikipedia" href="http://en.wikipedia.org/wiki/Programming_language">programming languages, Awk also has user defined variables and built-in variables. In this article let us review how to define and use <a class="zem_slink freebase/en/awk" title="AWK" rel="homepage" href="http://cm.bell-labs.com/cm/cs/awkbook/index.html">awk variables. Awk variables should begin with the letter, followed by it can consist of alpha numeric characters or underscore. Keywords […]

Display the output of a command from the first line until the first instance of a regular expression.

Mon, 04 Jan 2010 22:46:55 +0000

: Image via Wikipedia

| perl -n -e 'print "$_" if 1 … /«a class="zem_slink freebase/en/regular_expression" title="Regular expression" rel="wikipedia" href="http://en.wikipedia.org/wiki/Regular_expression">regex>/;# This <a class="zem_slink freebase/en/command_line_interface" title="Command-line interface" rel="wikipedia" href="http://en.wikipedia.org/wiki/Command-line_interface">command line will display the output of , from the first line of output, until the first time it sees a <a class="zem_slink freebase/en/pattern_matching" title="Pattern matching" rel="wikipedia" href="http://en.wikipedia.org/wiki/Pattern_matching">pattern matching .

Display a block of text with delineated by a start pattern and an end pattern

Fri, 18 Dec 2009 07:43:41 +0000

Taken idea from commandlinefu.com

The command will display a segment from the file from the start pattern to the end pattern.

function viewsegment() { tail -n +`fgrep -n -m 1 “$1” $3 | head -`fgrep -n -m 1 “$2” $3 }
Display a block of text with delineated by a start pattern and an end pattern

Awk on Amit Agarwal Linux Blog

Apache server stats with small and useful bash script.

cvs add files recursively – not already in repository

BASH Script Performace

Rekursive Grep on Solaris or AIX Systems without GNU egrep -r funcionality

Linux hardware details.

Related articles

Total upload and download on any interface.

Related articles

Ranking of the most frequently used commands

List of commands you use most often

concatenate compressed and uncompressed logs

<span style="font-size: x-large;"><a href="http://feedproxy.google.com/~r/Command-line-fu/~3/iwFUyltYgjM/concatenate-compressed-and-uncompressed-logs">concatenate compressed and uncompressed logs

List of commands you use most often

<span style="font-size: x-large;"><a href="http://feedproxy.google.com/~r/Command-line-fu/~3/6MxHiSiINdo/list-of-commands-you-use-most-often">List of commands you use most often

Determine what process is listening on a port on Solaris, without lsof

redet – build regular expression (regex) for any program.

Awk Introduction Tutorial – 7 Awk Print Examples

Splitting strings with IFS

Understand Awk Variables with 3 Practical Examples

Display the output of a command from the first line until the first instance of a regular expression.

Display a block of text with delineated by a start pattern and an end pattern