perl is faster than bash in some cases.

Some days back, I had to generate some data to be uploaded to a database. As usual I assumed that bash should be faster and hence wrote the script to create the files in bash. But I found that even after 5 hours I was only 10% done with the data generation. Now that would mean that it would take around 50 hours to complete the data generation. Something did not look correct to me and I asked one of my colleague. He suggested I do a strace.

A quick strace command on the PID was shocking but very clear on what was happening.

1
strace  -p <PID>

Here’s a explanation of what was happening:

We saw that for every write there was

write(1, “a\n”, 2)                      = 2
dup2(10, 1)                             = 1
fcntl64(10, F_GETFD)                    = 0x1 (flags FD_CLOEXEC)
close(10)                               = 0

We knew that these are very costly calls for CPU and immediately understood what we should do. What was actually happening was that for each of the echo command the FD was being opened, file appended and then FD closed. This made it very clear why the script was running so slow. So, I quickly did some test to very that this will fix the issue I was facing.

I wrote one bash and one perl script to test this and did the time on these. Here are the programs and the output of time on them.

1
2
3
4
5
echo a > test
echo a > test
echo a > test
echo a > test
echo a > test

time output:

real    0m0.020s
user    0m0.004s
sys    0m0.005s

1
2
3
4
5
6
7
open FILE, ">test";
print FILE "test";
print FILE "test";
print FILE "test";
print FILE "test";
print FILE "test";
close FILE;

time output:

real    0m0.035s
user    0m0.001s
sys    0m0.008s

one more test to confirm the result

1
2
3
4
5
echo a >> test
echo a >> test
echo a >> test
echo a >> test
echo a >> test

time output:

real    0m0.018s
user    0m0.006s
sys    0m0.003s

As you can see the perl script took a lot lesser user time on the CPU and that is because the file was opened only once and then once all the output was written to the file, the file was closed so file operations in perl are much less than that in the similar bash script. The time taken in the bash script can be decreased drastically if we use open in the bash script also. So, the lesson that I learned was if there are some operations that you can remove from your script, even if they do not seem to be serious issue in the begining, you can improve the performance greatly.

Enhanced by Zemanta

Are you swapped? Increase the performance of Linux machine.

With the ever increasing cost of the Hardware, the amount of physical RAM available on the system is increasing day by day. For example, couple of years back, I had a system which was very high end Desktop with 256MB RAM and today I have a 2GB RAM Desktop. So, whats the point.

The Linux systems (right word should be kernel) are desiged to use both RAM and swap partition. Swap partition is a partition on Hard disk and is used mostly like RAM. Problem is that HDD access is always slower than RAM access and hence inherently, the system will work little slower even if you have enough RAM not to use swap. The term \”swappiness\” is used to determine how the kernel should try to seam-balance between the use of RAM and swap. By default, most of the distro\’s have a swappiness of 60. A higher value of swappiness means that the RAM will be swapped out faster.

There are two ways to look at the swappiness:

1) If the user has a higher swappiness then the used memory will be swapped faster to the swap and thus free\’ing the RAM for other useful purposes.

2) A lower value of swappiness would mean that bloaty applications will not be swapped and thus when the user returns to the application, the application would load faster or rather look faster as there will be no swapping from the swap to the RAM.

I personally keep the swappiness to a value of 100 in the Desktop. But then, anyway I kill firefox as soon as I am done and restart when required.

If you see that the RAM is underutilized or feel that the system performance is not that good then you can tickle with this setting and set it to 10-15. How to do it :

Login as roo (\”su -\”)

echo 15 > /proc/sys/vm/swappiness

This will take effect immediately, but if you want to change this value permanently then you can do this(again as root):

echo \”sys.vm.swappiness = 10\” > /etc/sysctl.conf

So play with your swappiness 🙂

\"Enhanced
Enhanced by Zemanta

Visualizing system performance data

Bookmark this category

Visualizing system performance data

$ vmstat 2 10 | awk \'NR > 2 {print NR, $13}\' | gnuplot -e "set terminal png;set output \'v.png\';plot \'-\' u 1:2 t \'cpu\' w linespoints;" draw `vmstat ` data using gnuplot

\"commandlinefu.com\"/

by David Winterbottom (codeinthehole.com)


URL: http://feedproxy.google.com/~r/Command-line-fu/~3/jjk4Htw9rvo/visualizing-system-performance-data