Latest Burps

In this article I'll be talking about a trick to make Arch Linux's package manager much faster in cases where its databases (/var/lib/pacman) have not yet found their way into the disk cache. It was inspired by a comment Vladimir made on Arch's bugtracker. Please note that this doesn't apply to SSDs or very speedy HDDs.

What's so special about pacman's databases

A characteristic of them is that they are composed of many small files. For each package in the 'sync' databases (packages that exist in Arch's repos) 2 files are stored, and in the case of the 'local' database (installed packages) 3 files are stored per package. To give you an idea, there are 12010 files inside /var/lib/pacman on the computer I'm currently typing this. :)

Alright, so you've got a few thousand files, but they are small!

The problem isn't the size of these files (around 33MiB on my desktop, without filesystem overhead). The slowdown is caused by the fact that to parse all these files, the HDD must do lots of seek operations which take some milliseconds each. Times the number of files; it adds up. Also, this delay is especially noticeable on lower RPM hard drives.

Enough preliminary talk, what's our plan of attack?!

We want to minimize the number of seeks that occur during the first time pacman's databases are accessed after boot. It would be great if the needed data could be read sequentially off of the HDD. Enter the concept of loop devices. So, what we are going to do is stick the whole /var/lib/pacman directory inside its own filesystem which will live inside a single fixed-size file.

I'm sold, show me the codes!

OK, here's what we'll do:

1. Get a root shell.
2. Go into /var/lib.
3. Create a 100MiB file called pacman.ext3. This will hold the loop filesystem.
4. Format it as ext3. (I suppose reiserfs or some other filesystem that handles small files well could be used instead during this step, but ext3 is fast enough for our purposes and very reliable.)
5. Create a tarball of the pacman directory which contains the databases.
6. Add a new entry for the loop filesystem to /etc/fstab.
7. Mount the loop filesystem. After this, the /var/lib/pacman directory will appear empty.
8. Extract the tarball from step 5, which will populate the newly created filesystem.
9. Done!

Following are the commands for the above steps:

I don't like, kill it with fire!

If you decide you want to revert what has be done, the steps are:

1. Get a root shell.
2. Go into /var/lib.
3. Create a tarball of the pacman directory which contains the databases.
4. Unmount the loop filesystem. After this, the /var/lib/pacman directory will be outdated!
5. Remove the entry for the loop filesystem from /etc/fstab.
6. Get rid of the now outdated pacman directory.
7. Extract the tarball from step 3, which will recreate the pacman directory.
8. Done!

Following are the commands for the above steps:

Conclusion

That's about it. Please don't attempt this if you don't understand what it does exactly. Consider this a disclaimer. If you bork your pacman databases, you'll most likely end up reinstalling Arch. Keep this in mind!

On the other hand, the speed-up is phenomenal. On my laptop, which has a 5400 RPM hard drive, the time required to do a 'pacman -Syu' right after boot has been shrunk from 1 minute and 38 seconds to just 6 seconds. Simply amazing. :D

Other notes

There also exists a script that does the same thing more or less. I found it after I had converted my desktop machine, and it's called pacman-cage. I do like to know exactly what goes on on my system, so I don't see me using it, but you could if you want. :)

I finished compiling the initial version of my résumé, based on the excellent moderncv LaTeX class. I'm not planning on starting to look for a job very soon, but I think it's practical to have a solid base résumé to which you can add things as you go along until you're ready to apply for a job opening.

Many thanks to a good friend of mine for pointing out some serious mistakes I had made in the first few drafts of my résumé.

Edit: Hum, forgot to add a link to it. So, here it is: Evangelos Foutras' résumé (or, PDF). :)

I participated in this year's Toughest Developer Puzzle Ever challenge, and actually managed to finish in the 95th place, which I'm pretty excited about! Solving the puzzles (and there were 100 [*] of them) turned out to be quite challenging, and my head begun to hurt at some point. However, it was well worth the effort; I found it to be a very rewarding experience and I learned a few new things along the way. :)

Many thanks go to the awesome people at woot.com who helped me get unstuck at points where I couldn't move on, even after having tried several different approaches or stared at a problem for extended periods of time. You can find my questions in the dedicated TDPE thread.

I'm really looking forward to next year's puzzles! :D

[*] Well, around 10 levels near the end were freebies, so make that 90. :P

This might be an odd post, but here it goes.

I'm a 22 years old male, baptized "Christian", who decided to silently drop the whole religion thing a couple of years ago (by not going to church and not caring about religion in general.) The problem is my mother, who is very religious and sees my infidelity as something that's her fault.

Up until now, I had played along in Christian customs (but without going to church at all,) not wanting to upset other people; choosing the easy way to live life as an atheist I guess. Lately I've become more intolerant of this —excuse the word— crap.

Things got a bit intense yesterday when my mother repeatedly called me outside as the epitaph of Christ was passing by. I refused to go and kept telling her that I don't believe in religion. It was quite obvious that she didn't like my response. For the next couple of hours, she'd come into my room and start preaching, telling me how wrong I am. That was last night.

Tonight is the resurrection of Christ, and the end of Lent. At ~23:15 I went to the kitchen to cut some tomatoes and onions to make me a couple of souvlakia after the Lent is over. My mother comes in and tells me in an ironic voice that since I don't believe in Christianity I should go ahead and eat whatever I want with no need to wait another 45 minutes. We argued calmly for a few minutes and even though I could see that she didn't mean anything she said, I proceeded to set up the electric grill. That's when she went nuts.

Shortly after plugging in the grill, I had become the enemy. She became furious at me and said that she'd much rather spend next Easter with other people (relatives) that believe in religion. Understanding that nothing good would come out if I had become angry too and had responded with harsh words, I unplugged the grill and returned, somewhat hungry, to my room.

So here I am, hungry and sad, writing this post. I doubt my mother is the only person with such strong religious belief, and this really disappoints me. Seriously, fuck this shit, fuck it! However, because life goes on, and I can't pick another family to be part of, I would appreciate any advice on how to handle the situation that has arisen, now and in the future, by other atheists that have witnessed or experienced similar circumstances.

And to all theists out there: If I show you respect and don't make fun of your beliefs or lifestyle, I rightfully demand the same treatment from you.

Update

First of all, many thanks to all the wonderful people over at reddit.com/r/atheism; your responses here and in the thread on reddit are truly enlightening. As the second comment below says, I now realize that I might have indeed stepped all over my mother's traditions with the way I behaved yesterday. There isn't much that can be done about that now, and it probably doesn't matter either; it can been seen as a learning experience.

After about an hour of the incident, the atmosphere had been restored back to friendly, which is comforting and shows that religion can become at most a small obstacle, not a 5 meter tall wall, at least in this situation. :)

Tomorrow I'll be presenting a website at my university as part of an assignment. This website will be running off of my netbook at home. So, what can go wrong that would make it unavailable before or during the presentation? Let's see (not an exhausting list):

• Power outage, since the netbook won't restart itself when power is restored
• Network issues, either at the ISP-level or the wireless connection to the router
• Software failure; Linux is very stable but shit happens
• Hardware failure; according to smartctl, the machine has accumulated 5553 power-on hours

Of course, the combined chance of any of the above events occurring is pretty slim. For argument's sake, let's assume that the probability of the website becoming unavailable at some point during the 2-hour window of the presentation is 1%. That means there is a 99% probability that nothing unforeseen will happen and everything will go smoothly as planned.

Unfortunately, that's not good enough. Murphy's law states:

Anything that can go wrong will go wrong.

The question then is, what can be done to increase the chances of success and reduce the risk significantly? Meet probabilities.

At this point, let's introduce a remote server to the game which holds a clone of the website and can act as a fallback, shall the netbook disappear from the Internet. This server also has a 1% probability of failing during the specified time window, independently of the first machine.

So, are we better off with two copies of the website online? My guess is yes, but let's go ahead and calculate the probabilities:

• Event A: First server goes down, P(A) = 0.01
• Event B: Second server goes down, P(B) = 0.01
• Events A and B at the same time: P(A and B) = P(A)P(B) = 0.0001 or 0.01%

We went from 99% availability to 99.99%! That 0.99%, I believe, is worth the extra few minutes required to duplicate the website on a second, remote, server. Tomorrow, I'll learn if it's enough too. :)

16:55  * foutrelis is wondering whether he should start a personal blog on /me.com
16:56 < foutrelis> Seems like a good idea; not sure what I'd write about though. :3
16:57 < foutrelis> AlexC_: do you have a personal blog/site besides 3dteapot.co.uk?
16:57 < AlexC_> nope, I was going to. But never got around to it
16:57 < AlexC_> I may sometime, though
16:58 < foutrelis> I'm not imagining anything huge or serious, just a dumping ground for ideas and stuff .-.
16:58 < AlexC_> yeah same here actually
16:58 < foutrelis> We should both start it today! :D
16:59 < foutrelis> Could be fun.
16:59 < AlexC_> I already have the domains for it :P
17:00 < AlexC_> a quick install of 2.5 and done hehe
17:00 < foutrelis> Same. :3
17:01 < foutrelis> Writing in English would be challenging for me, but it'd be interesting at the same time. :D
17:01 < AlexC_> challenging? Your english is great
17:02 < foutrelis> It's not that my English is bad, but I'm not used to writing articles in it. :3
17:02 < foutrelis> Alright, here we go. /me does a Git clone of TCM

(It took some fiddling around to get the rewrite rules in Cherokee just right, but don't tell anyone! :3)