Keeping a highly available web application online is no joke. Everything above 99% is extremely impressive; that means that you battled the forces of erosion and probably even deployed some pretty neat features without even a hiccup from your users’ perspective. I always feel great when I get our weekly New Relic status report email – it’s a good indication of how well I did my job in the previous week. And for a couple weeks now I’m happy to report I’ve been very proud indeed, with 100% uptime on the Hipstamatic web application.

How do you achieve numbers like these? Unfortunately getting to 100% isn’t an easy road, and I want to state up front that I also don’t think it’s a realistic goal. Issues you can’t control can ruin your uptime number, and you shouldn’t feel broken up about that. It happens to everybody. But it’s always good setting goals that are difficult to achieve, and this one is no different.

So what’s the secret to 100% uptime?

The Proper Refactoring proceeds apace, but I think in my last post I gave the impression that the Hipstamatic Rails project is inefficient or, even worse, slow. Nothing could be further from the truth; over the course of two years I’ve been continuously improving the project to be more responsive and much, much faster. How much faster? Well, unfortunately, I don’t have metrics from the first months I worked at Synthetic. But we were using XML and then plist to generate our responses to the iPhone app, and that process was achingly slow: I would estimate 200ms on average.

Now, take a look at our average response time over the last month.

Considering the web of external services Hipstamatic depends on for much of its operation, I’m proud of our 115ms average response time. Proud but not satisfied – hence the need for the Proper Refactoring, and I am optimistic that it will lead to a net performance gain for us and our users. There’s no reason we can’t achieve 50-70ms response times with better caching and slimmer applications.

Over the same time period that our response time has dropped, our user base has grown exponentially, and so too our traffic. At the beginning of my tenure at Synthetic our site was receiving close to 100,000 hits a day, and nearly all of that web traffic: now community.hipstamatic.com sees about a million requests a day, most of that API traffic generated from our iPhone applications. That’s an enormous amount of growth, and much of that over the course of just one or two explosive months.

Synthetic is a team of extremely talented individuals. But as our main Rails programmer and only server administrator, I wanted to discuss the lessons I personally learned in making Hipstamatic’s web site and web services fast. (Or, at the very least, a lot faster.)

The Rails application backing Hipstamatic is very, very large. It started over two years ago as a Rails 2.1 project, and has been continuously improved since then – moving to Rails 3.2, adding in redis and resque, and then adding in elasticsearch. During that time the database has bounced around continuously in size and importance as we move data from MySQL to data stores that are better suited for it. And while at the start it handled only contests and submissions, since then we’ve added in orders, the family album, D-Series support, and even more exciting behind-the-scenes stuff.

And as you can imagine from a project that’s undergone continuous improvement for a long time, it’s kind of a mess. A lot of stuff was done without an eye towards our future needs, and, even more embarrassingly, a lot of stuff was done with a future need in mind – and, of course, that need never materialized, so the code is named or structured improperly.

The temptation with any project as large and old as this is to do the Big Rewrite. I’ve been involved in a few of those, and my advice regarding them is quite simple:

I volunteer in the cyber center at the Center on Halsted. They’re always looking for people to teach new classes, so I figured I would teach an introduction to programming class – an introduction through Ruby. Besides being my favorite computer language, I think Ruby makes a great beginner’s language: it has none of the strange, computer-oriented concepts that make most programming languages difficult to learn, and it even reads like pure, simple English. Instead of pointers and variable typing, you have sensible enumerators and object orientation. And also it’s super fun.

Unfortunately, getting stuff installed on the computers at the cyber center is kind of a headache. Like most chronically underfunded but well-meaning institutions, there are about three levels of bureaucracy between you and getting anything done. And teaching Ruby without the aid of irb would be next to impossible. So what’s a guy to do?

Simple: create irb online.

Rails provides a really awesome ability to avoid Law of Demeter violations – the Module#delegate method. The Law of Demeter is an informal programming guideline, intended to make your code more obvious and more reusable: objects should only call methods on other objects, not objects of those objects. To provide a more concrete example:

1
2

  @user.address.street_address # Law of Demeter violation: @user should not reach into address!
  @user.street_address # So much better...

Rails, because it’s cool, provides a quick and easy pattern for making this work:

1
2
3
4
5
6
7

class User < ActiveRecord::Base
  has_one :address
  delegate :street_address, :to => :address

end

User.new.street_address # Calls address.street_address

Check out the delegate documentation for more information on how this functionality works. But a question I came across on Stack Overflow today asked: what do you when you want to delegate all methods? I do something similar in Dynamoid and wanted to talk about how to make this pattern sensible and performant.

Hipstamatic uses the pretty awesome Family Album feature for people to like and react to each others’ photos. You can create either a magic album – an album that matches to a combination of criteria including accounts, geolocation, tags and descriptions – or a curated album, selecting photos directly that you want to include. The latter is a pretty straight-forward association and isn’t very interesting to talk about, but I wanted to discuss briefly the methods we used to implement magic albums and what we finally settled on. It involved a lot of setting up elasticsearch and percolation, and ultimately I think it’s a very durable, excellent solution for anyone wanting to index a lot of data and retrieve it extremely quickly.

I initially thought of titling this post something more inflammatory, like “Why Haml Sucks” or “Only Losers Use Haml.” But the truth is Haml does anything but suck. It’s actually quite elegant; the syntax is clean, not needing closing tags is just really cool, and it’s very fast to read. It seems like it would be an ideal language to replace HTML, just like SASS and CoffeeScript are abstractions of and (to a certain extent) replacements for CSS and JavaScript, respectively.

So why do I bang my head against my desk every time I see someone using it in a view?

Phil Karlton, someone I can only presume is a pretty smart programmer, said “there are only two hard things in Computer Science: cache invalidation and naming things.” He’s totally right; cache invalidation is one of the biggest headaches when designing highly usable, highly available websites and is something that I’m sure every Rails programmer worth their salt has struggled with. (Naming things is also a pain but not the focus of this post.)

Outside of the guiding framework of Rails, it can be difficult to manage your own Ruby code, as I’ve discovered in the process of creating Dynamoid. MVC provides a clear and concise framework that encapsulates the functionality of most web applications really well; but outside of web applications, exactly where and how to put your code is a lot less clear. I thought I’d share some of my experiences and thoughts on doing this in my own Gem in the hopes that the lessons I’ve learned will aid others.

Over the weekend I spent some time getting Fnordmetric set up in an application. On the surface it looks really cool and quite nifty, but I ran into some trouble getting it configured how I wanted it and figured I’d make a post about it. I think I might have been struggling against the conventions in it a little too much, but it was still an instructive battle.