Things You Should Do Now

Phorge Flavor Text (Sundries)

Describes things you should do now when building software, because the cost to do them increases over time and eventually becomes prohibitive or impossible.

Overview

If you're building a hot new web startup, there are a lot of decisions to make about what to focus on. Most things you'll build will take about the same amount of time to build regardless of what order you build them in, but there are a few technical things which become vastly more expensive to fix later.

If you don't do these things early in development, they'll become very hard or impossible to do later. This is basically a list of things that would have saved Facebook huge amounts of time and effort down the road if someone had spent a tiny amount of time on them earlier in the development process.

See also Things You Should Do Soon for things that scale less drastically over time.

Start IDs At a Gigantic Number

If you're using integer IDs to identify data or objects, don't start your IDs at 1. Start them at a huge number (e.g., 2^33) so that no object ID will ever appear in any other role in your application (like a count, a natural index, a byte size, a timestamp, etc). This takes about 5 seconds if you do it before you launch and rules out a huge class of nasty bugs for all time. It becomes incredibly difficult as soon as you have production data.

The kind of bug that this causes is accidental use of some other value as an ID:

// Load the user's friends, returns a map of friend_id => true
$friend_ids = user_get_friends($user_id);

// Get the first 8 friends.
$first_few_friends = array_slice($friend_ids, 0, 8);

// Render those friends.
render_user_friends($user_id, array_keys($first_few_friends));

Because array_slice() in PHP discards array indices and renumbers them, this doesn't render the user's first 8 friends but the users with IDs 0 through 7, e.g. Mark Zuckerberg (ID 4) and Dustin Moskovitz (ID 6). If you have IDs in this range, sooner or later something that isn't an ID will get treated like an ID and the operation will be valid and cause unexpected behavior. This is completely avoidable if you start your IDs at a gigantic number.

Only Store Valid UTF-8

For the most part, you can ignore UTF-8 and unicode until later. However, there is one aspect of unicode you should address now: store only valid UTF-8 strings.

Assuming you're storing data internally as UTF-8 (this is almost certainly the right choice and definitely the right choice if you have no idea how unicode works), you just need to sanitize all the data coming into your application and make sure it's valid UTF-8.

If your application emits invalid UTF-8, other systems (like browsers) will break in unexpected and interesting ways. You will eventually be forced to ensure you emit only valid UTF-8 to avoid these problems. If you haven't sanitized your data, you'll basically have two options:

• do a huge migration on literally all of your data to sanitize it; or
• forever sanitize all data on its way out on the read pathways.

As of 2011 Facebook is in the second group, and spends several milliseconds of CPU time sanitizing every display string on its way to the browser, which multiplies out to hundreds of servers worth of CPUs sitting in a datacenter paying the price for the invalid UTF-8 in the databases.

You can likely learn enough about unicode to be confident in an implementation which addresses this problem within a few hours. You don't need to learn everything, just the basics. Your language probably already has a function which does the sanitizing for you.

Never Design a Blacklist-Based Security System

When you have an alternative, don't design security systems which are default permit, blacklist-based, or otherwise attempt to enumerate badness. When Facebook launched Platform, it launched with a blacklist-based CSS filter, which basically tried to enumerate all the "bad" parts of CSS and filter them out. This was a poor design choice and lead to basically infinite security holes for all time.

It is very difficult to enumerate badness in a complex system and badness is often a moving target. Instead of trying to do this, design whitelist-based security systems where you list allowed things and reject anything you don't understand. Assume things are bad until you verify that they're OK.

It's tempting to design blacklist-based systems because they're easier to write and accept more inputs. In the case of the CSS filter, the product goal was for users to just be able to use CSS normally and feel like this system was no different from systems they were familiar with. A whitelist-based system would reject some valid, safe inputs and create product friction.

But this is a much better world than the alternative, where the blacklist-based system fails to reject some dangerous inputs and creates security holes. It also creates product friction because when you fix those holes you break existing uses, and that backward-compatibility friction makes it very difficult to move the system from a blacklist to a whitelist. So you're basically in trouble no matter what you do, and have a bunch of security holes you need to unbreak immediately, so you won't even have time to feel sorry for yourself.

Designing blacklist-based security is one of the worst now-vs-future tradeoffs you can make. See also "The Six Dumbest Ideas in Computer Security":

http://www.ranum.com/security/computer_security/

Fail Very Loudly when SQL Syntax Errors Occur in Production

This doesn't apply if you aren't using SQL, but if you are: detect when a query fails because of a syntax error (in MySQL, it is error 1064). If the failure happened in production, fail in the loudest way possible. (I implemented this in 2008 at Facebook and had it just email me and a few other people directly. The system was eventually refined.)

This basically creates a high-signal stream that tells you where you have SQL injection holes in your application. It will have some false positives and could theoretically have false negatives, but at Facebook it was pretty high signal considering how important the signal is.

Of course, the real solution here is to not have SQL injection holes in your application, ever. As far as I'm aware, this system correctly detected the one SQL injection hole we had from mid-2008 until I left in 2011, which was in a hackathon project on an underisolated semi-production tier and didn't use the query escaping system the rest of the application does.

Hopefully, whatever language you're writing in has good query libraries that can handle escaping for you. If so, use them. If you're using PHP and don't have a solution in place yet, the Phorge implementation of qsprintf() is similar to Facebook's system and was successful there.