faster bash operations on files with File Descriptors.

2011-06-11 2 min read Bash Learning Linux

I was writing a bash script that would do some operations and read and write to file. Seems that that was pretty simple with

  <td>
    <div class="text codecolorer">
      while read line<br /> <br /> do<br /> <br /> done<file
    </div>
  </td>
</tr>

1
2
3
4
5

and then use redirection operations like “>” and “»” to write to file. Done with the script pretty fast. So far so good, when I went for real life tests, no one was interested in using it, why? Simple, it was simply taking too long. The file was reading about 10K lines and writing about 50 lines and was taking about more than 10 minutes.

Unix shell script for removing duplicate files

2011-05-16 1 min read Bash Linux

The following shell script finds duplicate (2 or more identical) files and outputs a new shell script containing commented-out rm statements for deleting them (copy-paste from here):

::: updated on 02 May 20121, seems like wordpress did not like it so well so reformatting the code :::::::

#!/bin/bash -
#===============================================================================
#
#          FILE:  a.sh
#
#         USAGE:  ./a.sh
#
#   DESCRIPTION:
#
#       OPTIONS:  ---
#  REQUIREMENTS:  ---
#          BUGS:  ---
#         NOTES:  ---
#        AUTHOR: Amit Agarwal (aka), amit.agarwal@roamware.com
#       COMPANY: blog.amit-agarwal.co.in
#       CREATED: 02/05/12 06:52:08 IST
# Last modified: Wed May 02, 2012  07:03AM
#      REVISION:  ---
#===============================================================================

OUTF=rem-duplicates.sh;
echo "#!/bin/sh" >$OUTF;
find "$@" -type f -exec md5sum {} \; 2>/dev/null | sort --key=1,32 | uniq -w 32 -d |cut -b 1-32 --complement |sed 's/^/rm -f/' >>$OUTF

Pretty good one line, I must say 🙂

cd across parallel directories

2011-03-14 1 min read Bash Linux

Here is a simple and fast way to cd across parallel directory.

  <td>
    <div class="text codecolorer">
      cd ${PWD/test/actual}
    </div>
  </td>
</tr>

Linux command line tips: wget (ghacks.net)
Article Directory Submissions (tjantunen.com)
Article Directory Submissions (cash-bandit.com)
Create a repository from your fedora DVD (annupandey.wordpress.com)

some bash exercise for you

2011-02-15 1 min read Bash

Here is some teaser for you, can you figure out how this works:

  <td>
    <div class="text codecolorer">
      echo ls<br /> <br /> !!:1-$
    </div>
  </td>
</tr>

1
2
3

ovigia: #tips: #Bash Shell Scripts and Tutorials | BashShell.net (bashshell.net)
perl is faster than bash in some cases. (amit-agarwal.co.in)
bash math (amit-agarwal.co.in)

Ranking of the most frequently used commands

2011-01-31 2 min read Linux Uncategorized

Lets take a quick look at how to get the most frequently used commands on you shell. So what we need to do is this:

  <td>
    <div class="text codecolorer">
      history | awk '{print $2}' | awk 'BEGIN {FS="|"}{print $1}' | sort | uniq -c | sort -n | tail | sort -nr
    </div>
  </td>
</tr>

So, how did we arrive at this and will this always work? No it might not always work. A typical example is where HISTTIMEFORMAT variable is set. In that case, if you check history, you will see that after the number column we have time and date in the specified format, in which case, you will get wrong information from the above command. Anyways, forgetting these special cases, lets go to how we got this command:

shopt causes bash completion to stop working.

2011-01-25 1 min read Bash

Couple of days, I added some scripts to my bash startup and suddenly realized that bash completions stopped working. This is tricky situation in bash to figure out what went wrong. So, I set my foot down to make sure that I find out the culprit. Thus I started with method of elimination. So, I started with nothing in .bashrc and kept adding my stuff in batches, until the said feature was broken. Once that happened I started removing the added ones, one by one, until my completion started working again.