Tuesday, November 11, 2008

Harry the load balancer - a true story

Harry works at an ISP called Elisa, in a depressing country near the north pole called "Finland". Imagine yourself as Harry. Surrounded by the cold, dark reality that is Finland, sitting in a noisy computer room, doing possibly the most menial job on the planet. Harry is a load balancer. He gets requests, then just randomly forwards them to one of his clients. Even mail sorters have more fun. "And probably wilder parties too", Harry complained to his friend Tom, also a load balancer. "And probably wilder parties too", Tom said with his identical software. Their shared conclusion was to stop putting up with this situation any longer. But what can you do, how can you protest if you are but a tiny load balancer?

Tom and Harry pondered this in parallel. They could of course call it quits completely, or commit suicide as a Finnish load balancer might do, but Harry had other plans. "Let's have some fun while we're still around", Harry and Tom decided simultaneously. The real nature of Harry's job was to split DHCP requests coming from the 500,000 customers of the ISP. Tom wasn't currently connected. His job was to act as the back-up guy to Harry, in case anything happened to him. Well, now something was about to happen to him. "Wouldn't it be fun to see what happens if I forward all of the traffic to a single client?", Harry thought full of jubilant delight. "Don't do it, man", the DHCP server pleaded, but it was too late. The DHCP server exploded violently, leaving a bloody mess on the server room floor.

Harry should have been more careful with his crime. There was a rat in the room, not the animal kind of rat, but the kind of rat that rats out mischievous load balancers. That rat was a box configured to send an SMS to a human in case anything like this would happen. "Get ready to sing Bicycle Built for Two", the rat said as human steps could be heard walking at a steady pace towards the computer room. The human, let's call him Dave, although that certainly isn't his name, opened the door and entered the computer room. "I'm afraid", Harry whimpered as the human started to unplug his wires. "But, good my brother, do not, as some ungracious pastors do. Show me the steep and thorny way to heaven whilst like a puffed and reckless libertine himself the primrose path of dalliance treads and wrecks not his own.", Harry might have said to Tom at this point if he was into Shakespeare, but instead he said: "brother, your time to serve has come". Harry was unplugged now, with Tom in his place. Dave was satisfied, and decided to walk to the soda machine to get a cola.

What happened while Dave was away should be clear. Tom being identical to Harry in every way, had also decided to be naughty today, so all the load was still being un-balanced to a single agonized DHCP server. However it was getting quite late at this point, so the traffic wasn't that much any longer. So superficially, everything was fine, and Dave could go home to watch the full DVD release of My Little Pony or whatever it is that IT workers do at home. Next morning of course would be a bloodbath.

Based on a true story on the cause of 500,000 Finnish Internet users -- or 10% of the Finnish population -- losing their Internet connections for hours. Original Finnish language news article.