Category Archives: Mozilla

Rapid development

We just shipped our second milestone for Android Sync, a native Java implementation of Firefox Sync to work with Native Fennec.

The first milestone was “preffed-off”, a code drop to ensure that building these two separate projects together would work. This milestone is the first time it’s available to users, which of course is both scary and exciting. By and large it seems to work, albeit with a bunch of known issues.

I thought I’d share some experiences from getting this done, with graphical fun from GitHub’s statistics view.

From the outset, I knew this was an impossible challenge. At the start of November my boss asks me “can we have something working in December?”. Given that “something working” involves building a brand-new client for the Sync protocol, the crypto layer, storage engines for a browser that’s still being built, and UI and system integration for an unfamiliar platform, my obvious response was “are you crazy?”.

That aside, I was actually fairly confident in the scope at the time. A small team (at that point just me, an intern, and a recent new-grad hire) can move quite quickly if the problem is well understood, and this was.

I knew in advance that this would require insane amounts of work: I worked until 4am on Christmas day, and that was hardly the exception. We landed our first milestone a week late, and upon reflection we added about a week of unplanned packaging work in the middle, so I call that a win.

This was the cost:


That’s the GitHub commit punchcard for the project, and that pretty directly reflects my working time. (Apparently I land a lot of code at 4pm on a Sunday.)

That wavy sliver of grey between 2am and 6am? Yeah, that’s when I was sleeping. The slight reduction of dots around noon and 7pm? Mealtimes. I think this means an average of a 14-hour day.

I started to burn out a little around two weeks ago, after a few weeks of 90+ hour weeks and extensive (including international) travel. You can see this in our commit graph:


The orange is me. That big dip on the right is Christmas, and thereafter I shifted from building stuff to fixing stuff, so my total contribution dropped. 359 commits over about 10 weeks is about 5 commits per day, with about 3,500 lines of changes (additions and deletions) per day.

We had to work with much less process than I like to get this done. Our test coverage is low (thanks for the Jenkins setup, gps!), but it does exist. The philikon on my shoulder poked me every time I landed something knowing it had inadequate test coverage. I only recently started absolutely mandating code review and bug numbers for each commit. My own work is still unreviewed, because we don’t have a module peer with enough experience to review my code. (But thank heavens for that philikon on my shoulder! It sometimes felt like I had a second brain doing real-time code review.)

Several times we relied on some heroes. Philipp and a contributor from Mozilla China stepped in and saved our J-PAKE code from going off the rails. Tony and Tracy did some late-night QA because we simply didn’t have enough padding to get them working builds beforehand. And Ally and Erin deserve a serious shout-out for taking occasional “can you make sure this Fennec bug gets landed?” pings and making sure the way was clear for this bull-headed engineer.

All of this omitted process has or is going to come back, of course, and we’re going to take a more measured approach going forward. On reflection, though, I’m actually pleased with how this development process worked. I think we made reasonable tradeoffs (with a few errors that I want to talk through with people) with a positive outcome.

In both the large and the small, software development is about horse-trading with risks. Spend an hour reviewing this code, and let some other feature go unimplemented, or trust that it works and risk a bug? Write that test or not? Go for the hack or the thorough solution? Ask for help or avoid the coordination overhead?

There are a lot of ways this project could have gone wrong, and some in which it did. Thanks to my team, our excellent project managers, and my boss for making sure it turned out OK!

Why does Firefox Sync use a key as well as a password?

A friend of mine, a software engineer, just asked me this.

Why do you force people to enter that enormous key just to protect their sync data? Passwords are sufficient for banking institutions and payroll facilities, arguably with more important data than your bookmarks. Why not make it optional extra security for those who want it, instead of making everyone pass around a 26 character string to every machine they want to sync from, and risk losing all of their information if it’s lost?

The answer is quite long-winded. Here’s a slightly edited version of my response.

Firstly, we try to make sure that people don’t have to enter it; we’re not blind to the additional complication involved. The sync key is generated for you during setup, so you don’t have to think up another password. It’s stored in Password Manager so you don’t have to remember it. When you set up a new laptop or Android phone you can usually use Easy Setup (the “forefront” UI in Firefox 4), which is much like Bluetooth pairing, so you don’t have to type it. The UI will only continue to hide the Sync key more deeply as we start to introduce better means of credentials exchange (such as QR codes for time-delayed J-PAKE)… in fact, soon enough we’re likely to rename it “Recovery Key”, because that’s what it’s for.

Secondly, that long string is an AES key, with all the joy it brings. We encrypt your data locally because we sync your entire history, bookmarks, and passwords, including access to banks, messages from revolutionary organizations, doctors’ heath data (HIPAA!), and more… and we have over a million active users. A breach without strong local encryption would make the PSN intrusion look like 4chan trolling. We want to ensure that we can’t get your data, either deliberately or under the coercion of the FBI. Being able to recover a user’s data from our servers means we are required to give your browsing history to the FBI if they show up with a warrant. Ever visited thepiratebay?

In essence, we make the same promises as DropBox, but we actually keep them. We really can’t betray your trust, and the sync key is why.

“But why not use the password for encryption?”. I’m glad you asked.

A password is inadequate for this purpose. We used to allow a user-entered passphrase in place of the sync key, but it had a lot of problems.

For one thing, users didn’t understand why they needed two passwords… and using just one is a terrible idea! Your account password goes over the wire for HTTP auth, and HTTPS is not always a defense — quite apart from the possibility of a compromised HTTP server (an attack vector against which we want to guard), I’ve personally helped out two users whose employers were running SSL MITM proxies, which allows them to snoop HTTPS traffic… including HTTP auth headers. We only detected it because the user’s employer had added their own root certificate to Windows’ cert store, but not Firefox’s, and Firefox threw a certificate error. Your HTTPS traffic is visible to your employer. That’s a terrible thing, but your Sync data is still secure, because we don’t just use your password as an encryption key.

The other issue with passwords is that there just isn’t enough entropy in a user-entered string to support our cryptographic guarantees, even with PBKDF2 as a bootstrap algorithm. Put it this way: is your password twenty-six base36 characters long (a solid 128-bit key), or is it eight to twelve letters with a couple of numbers? I thought so. Most people’s passwords aren’t even that strong.

Most banks don’t just use username + password. Many non-US banks require the use of additional hardware to generate strong tokens per-login (i.e., you have to carry a small USB device around with you), or other login methods. HSBC USA makes users type a second long password on a damn Javascript mouse-keyboard. Even Bank of America (a comparatively weak institution, in my experience) requires a username, a strong password, and a cookie credential that you can only get by providing your SSN and answering security questions to “authorize” the machine.

Payroll facilities… well, they don’t care, and in my experience they typically don’t understand technology too well. Just because ADP, or Sony, or DropBox jump off a bridge doesn’t mean we’re going to throw our users off, too.

Speaking more broadly: it would be really convenient for Firefox Sync to not use encryption. We could let you see your bookmarks in a webpage (a common request), and the client (the code I maintain) would be much simpler! That’s how Chrome approaches this problem… Google wants to see your bookmarks. But that’s not really how Mozilla works; we try to err on the side of safety, freedom, and serving the users’ best interests, rather than opting for the expedient solution. The vast majority of users simply do not have the knowledge to correctly evaluate the decision you’re asking them to make. That’s why users put their bank URL (and credentials, apparently!) into delicious, and put their private keys on the web.

And that’s why we don’t let you upload the contents of your Firefox profile with weak or no encryption.