perl is faster than bash in some cases.

Some days back, I had to generate some data to be uploaded to a database. As usual I assumed that bash should be faster and hence wrote the script to create the files in bash. But I found that even after 5 hours I was only 10% done with the data generation. Now that would mean that it would take around 50 hours to complete the data generation. Something did not look correct to me and I asked one of my colleague. He suggested I do a strace.

A quick strace command on the PID was shocking but very clear on what was happening.

1
strace  -p <PID>

Here’s a explanation of what was happening:

We saw that for every write there was

write(1, “a\n”, 2)                      = 2
dup2(10, 1)                             = 1
fcntl64(10, F_GETFD)                    = 0x1 (flags FD_CLOEXEC)
close(10)                               = 0

We knew that these are very costly calls for CPU and immediately understood what we should do. What was actually happening was that for each of the echo command the FD was being opened, file appended and then FD closed. This made it very clear why the script was running so slow. So, I quickly did some test to very that this will fix the issue I was facing.

I wrote one bash and one perl script to test this and did the time on these. Here are the programs and the output of time on them.

1
2
3
4
5
echo a > test
echo a > test
echo a > test
echo a > test
echo a > test

time output:

real    0m0.020s
user    0m0.004s
sys    0m0.005s

1
2
3
4
5
6
7
open FILE, ">test";
print FILE "test";
print FILE "test";
print FILE "test";
print FILE "test";
print FILE "test";
close FILE;

time output:

real    0m0.035s
user    0m0.001s
sys    0m0.008s

one more test to confirm the result

1
2
3
4
5
echo a >> test
echo a >> test
echo a >> test
echo a >> test
echo a >> test

time output:

real    0m0.018s
user    0m0.006s
sys    0m0.003s

As you can see the perl script took a lot lesser user time on the CPU and that is because the file was opened only once and then once all the output was written to the file, the file was closed so file operations in perl are much less than that in the similar bash script. The time taken in the bash script can be decreased drastically if we use open in the bash script also. So, the lesson that I learned was if there are some operations that you can remove from your script, even if they do not seem to be serious issue in the begining, you can improve the performance greatly.

Enhanced by Zemanta

http://dmartin.org/content/things-i-can-do-linux-i-cant-do-windows

Here\’s a nice article on Windows vs Linux. I have been reading on these for quite some time now, but this one really sums it up all very well. I hope you enjoy it as much as I did.

Since I often hear from friends and people on the Internet about things they can\’t do in Linux that they could on Windows, I thought I\’d write up a list of things I can do in Linux that I can\’t do in Windows.

  1. Update every single piece of software on my system with a single action. This is one of the main reasons I run Linux.  Sure, Windows has Windows Update, but that only updates the operating system, Office, and a few other things.  For every Linux distribution I\’ve used (Gentoo, Red Hat, Suse, Ubuntu), updating is simple.  When you update, you have every application, every library, every script – every single piece of software upgraded automatically for you.  And on most of them, they will check for updates automatically and notify you.  This is great for security, fixing bugs quickly, and getting the latest in features.
  2. Update nearly everything on my computer without a reboot. On Linux, there is only one thing that requires a reboot after updates.  The kernel.  And even then you can continue to run on the previous kernel.  You just need to reboot to get the benefit of using the new kernel (say, if it has a bug fix or a new feature).  In Windows, many of the updates to even non-critical software require reboots.
  3. Keep my system secure without software that consumes my system resources, requires my time, and frequently nags me.  Basic requirement for a secure Windows box include:
    1. Running antivirus protection.  AV software consumes resources and requires routine scans.
    2. A software firewall like ZoneAlarm or the one built into Vista that constantly asks you if you want to allow software to contact the Internet.  More time on your part.
    3. Running Adblock Adaware and/or Spybot Search & Destroy on a routine basis, consuming your time, and requiring your manual intervention.  People often forget or don\’t \”get around to it\”.
    4. Never trusting software.  You have to go through life assuming every bit of software and every website on the Internet is going to screw you over.  What a sorry state of affairs that is.

    All of this requires your attention, slows your computer, and ruins the open experience of the Internet.  None of this is necessary in Linux.  You get your software through your distribution.  As long as you can trust your distribution, you can trust the software available.  Having a firewall is a good thing even in Linux, but most of us have a firewall built into our Cable and DSL modems, or our wi-fi router.  A software firewall in windows is as much used to keep malware from calling out as it is to keep outside intrusions from coming in, and you don\’t have the same concerns in Linux (since, as I said, you can trust your software).

  4. Run an entire operating system for free without pirating software, and without breaking the law. Most Window\’s users seem to accept that breaking the law is okay, because it is pretty much required.  Either you break the law, or spend countless thousands of dollars on the software you need.  You may not think it is a big deal, but if you own a home like I do, you are putting it at risk.  While unlikely, the potential is there for software companies to come after you just like the RIAA has come after countless people.  With Linux, this isn\’t necessary.  You can run the software you need without paying for it, and without breaking the law.  I know I sleep better at night.
  5. Take my settings with me where ever I go. In Linux, all your personal settings are stored in your Home folder, most in folders that begin with a period (like .gaim).  So, I can copy all these settings from one computer to another.  I can put these settings on a USB drive.  When I switched from Gentoo to Ubuntu, I kept all my settings.  On Windows, some settings are under your home folder and some are in the registry.  So your settings are not portable.
  6. Run Internet Explorer 5.0, 5.5, 6.0, and 7.0 on the same desktop. I have all installed thanks to the wonderful IEs4Linux project.  I can even run them side-by-side if I want.  For a web developer, that\’s huge.  Testing browser compatibility to that level on Windows requires multiple machines or something like VMWare.  Further, when I run IE under Linux, I don\’t have to worry about any malware or virus getting onto my system.
  7. Understand everything that is going on in my computer. Using Windows is like working with a black box.  You can see the outside, but you have no idea what is going on inside.  If you hit snags, your only option is to hope Microsoft fixes it.  Or, perhaps you can submit a bug report to Microsoft, spending your time improving software that  a company makes billions from.  Under Linux, you can look at the system logs, where you can see most issues.  You can search for the log messages on Google, and can usually track the cause and often find a fix.  If not, I can even go look at the source code to find the offending problem.  Granted, most people aren\’t capable or don\’t have the time to look at the source code.  But the fact that tens-of-thousands of geeks do is often very, very helpful.  And if you do spend the time filling out a bug report, you are helping other people just like yourself, not contributing your time to a rich software company.
  8. Customize every aspect of my desktop. In Windows, you are more or less stuck with what you are given.  Sure, you can install buggy skinning engines, or you can pay Microsoft extra for the ability to put skins on your desktop.  But even these aren\’t very adaptive.  It\’s just a new coat of paint on the same desktop.  Under Linux, I can choose the window manager, the desktop environment, the theme, the GTK engine, the icon theme, the special effects (see Beryl or Compiz), the filesystem browser, and so on.  Nearly every aspect of the system has competitive options.  If you look around the internet at screenshots of various Linux desktops, you rarely see two that look the same.
  9. Benefit from competition between projects for each system on my computer. As I mention in point 8, there are options for every aspect of the Linux desktop.  Not only is it fun to try the various options, but it leads to better software as multiple projects compete against each other to be the best.  Can you imagine competing printing backends, competing desktop environments, or competing USB mounting systems on Windows?  I\’ve been a Linux user for 3 years now, and I\’ve seen remarkable changes in systems used on the Linux desktop, from critical systems (XFree86 switched to X.Org, auto-mounting systems) to non-critical  (my CD-Rom eject button works!).
  10. Run thousands of great pieces of software that only run on Linux. Just like Windows, Linux has software that doesn\’t run on Windows.  Great pieces of software like Amarok, Bluefish, Neverball, Gnumeric, K3B, Beryl, gdesklets, and MythTV.  I know this is a chicken-and-egg point, where Windows has the exact same situation. Too often I hear \”I can\’t switch to Linux because it doesn\’t run [insert Windows software]\”.  My reason for pointing it out is just to make it clear that this is a two-way street.
  11. Learn about, support, and appreciate the value of free software. I believe free software is important to us all.  Even if you use non-free software, the free software movement ensures checks and balances on non-free software by offering an alternative.  By running a free operating system and becoming involved in the community, I\’ve contributed to free software, even if only in a small way.

The main point I frequently try to make is that you can\’t expect features to be 1-to-1 when switching operating systems.  We like Adobe Photoshop or Microsoft Office because we have used it for years, and we are used to it.  We hate change.  It\’s natural that people have a desire for everything to be identical.  Did we expect Windows to be exactly like something else when we started using it?  Probably not.

When you learn closed-source proprietary software like Photoshop or Office, you have spent your time indenturing yourself to a lifetime of spending $700 every so many years.  And the same goes for every company you work for that you insist you need Office or Photoshop.  And if you don\’t think that your company\’s expenses affect your salary, think again.

Conversely, if you take the time to learn open and free systems like Linux, Gimp, or OpenOffice, you now have given yourself a lifetime of perpetually free software.  The value of that is quite profound.  No more worrying about installing Office on more than one computer and running into activation issues.  I have OpenOffice installed on all 5 computers I own, and my flash drive where I can run it on any computer I wish.

Bottom line is, yes, you will have to spend time learning Linux and the software running on Linux if you choose to switch.  But by doing so, you\’ve set yourself up for a lifetime of free computing.  For many of you, that\’s going to be 40, 50, 60, 70 years.  A period of learning isn\’t so hard to swallow, when you can see the value of doing so.

My advice, should you choose to try Linux, is to forget about making it exactly like Windows.  You will spend countless hours, and you will fail.  Once you spend signicant time on Linux, every time you use a Windows computer you will say \”Bah, I could do xyz if I was on Linux\”.

If you are a Linux user, and you have other things you think should be on this list, please contribute comments below.

Contd:: Evaluation of Distro\’s

Foresight

Foresight is a desktop operating system featuring an intuitive user interface and a showcase of the latest desktop software, giving users convenient and enjoyable access to their music, photos, videos, documents, and Internet resources.

\"Foresight\"

I liked their logo. Quite simple and intutive.

\"Foresight

Logo goes quite well with the Foreground image. Both are same in color and hence it gives you a feeling of theme and not just ramdomly selected foreground.

\"Foresight

And definately their Desktop is quite impressive. Very different and elegnant. Worth a look. It definately meets the promise as on their website —

As a Linux distribution, Foresight sets itself apart by eliminating the need for the user to be familiar with Linux, combining a user-focused desktop environment on top of Conary. As the most technically innovative software management system available today, Conary ensures that users can efficiently search, install, and manage all the software on the Foresight system, including bringing in the latest features and fixes without waiting for a major release. Thanks to Conary, Foresight developers can provide those features and fixes through rolling releases, allowing updates to be pushed out as soon as they are available. The latest version of Foresight is 2.0.5 which contains an up to date snapshot of the Foresight respository.

Byzantine

ByzanineOS is a software internet appliance with a home entertainment bias. It is based on a networked Linux distribution/bootable system with Mozilla providing access to a range of services and applications.

ByzanineOS fits in 32MB (or 48MB) of media and should work on any x86 based PC.

Simple to use OS with most of the applications supported on Mozilla. Quite different again.

\"Byzantine \"Byzantine \"Byzantine

FreeNAS

Liked the concept that you really don\’t need a X window if you really don\’t want to do anything on the system. It runs a webserver and mostly all configuration needs to be done from the web interface. You don\’t get a shell to work on. You just have a menu with 4-5 options to set a few things and all other things and montitoring needs to be done from the Web Interface.

So what is it? It is network attached Storage OS.

\"FreeNAS \"FreeNAS \"FreeNAS \"FreeNAS \"FreeNAS \"FreeNAS

DSL

It did not detect my mouse and I could not do much without the mouse. But I was running it on the QEMU so might be that is the issue. I just got one screenshot. Will try again later with different settings to see if I can get it working.

\"DSL\"

Linux Mint

It did not come up completely. But I was running it on the QEMU so might be that is the issue. I just got one screenshot. Will try again later with different settings to see if I can get it working.

\"Linux

Do let me know what you think of these various distro\’s.