Category Archives: Engineering

Different kinds of storage

I’ve been spending most of my time so far on Project Tofino thinking about how a user agent stores data.

A user agent is software that mediates your interaction with the world. A web browser is one particular kind of user agent: one that fetches parts of the web and shows them to you.

(As a sidenote: browsers are incredibly complicated, not just for the obvious reasons of document rendering and navigation, but also because parts of the web need to run code on your machine and parts of it are actively trying to attack and track you. One of a browser’s responsibilities is to keep you safe from the web.)

Chewing on Redux, separation of concerns, and Electron’s process model led to us drawing a distinction between a kind of ‘profile service’ and the front-end browser itself, with ‘profile’ defined as the data stored and used by a traditional browser window. You can see the guts of this distinction in some of our development docs.

The profile service stores full persistent history and data like it. The front-end, by contrast, has a pure Redux data model that’s much closer to what it needs to show UI — e.g., rather than all of the user’s starred pages, just a list of the user’s five most recent.

The front-end is responsible for fetching pages and showing the UI around them. The back-end service is responsible for storing data and answering questions about it from the front-end.

To build that persistent storage we opted for a mostly event-based model: simple, declarative statements about the user’s activity, stored in SQLite. SQLite gives us durability and known performance characteristics in an embedded database.

On top of this we can layer various views (materialized or not). The profile service takes commands as input and pushes out diffs, and the storage itself handles writes by logging events and answering queries through views. This is the CQRS concept applied to an embedded store: we use different representations for readers and writers, so we can think more clearly about the transformations between them.

Where next?

One of the reasons we have a separate service is to acknowledge that it might stick around when there are no browser windows open, and that it might be doing work other than serving the immediate needs of a browser window. Perhaps the service is pre-fetching pages, or synchronizing your data in the background, or trying to figure out what you want to read next. Perhaps you can interact with the service from something other than a browser window!

Some of those things need different kinds of storage. Ad hoc integrations might be best served by a document store; recommendations might warrant some kind of graph database.

When we look through that lens we no longer have just a profile service wrapping profile storage. We have a more general user agent service, and one of the data sources it manages is your profile data.

Humane code review

In the year I’ve been at Mozilla I’ve submitted almost two hundred patches for review. My email archives suggest that I’ve had 26 reviews not granted, and 164 granted.

I’ve also been reviewing patches for Sync since January of 2011. And in the past few months our team — and thus the number of people whose code I review, or see being reviewed — has grown significantly.

This has given me a lot of food for thought on the topic of code review and motivation, so here’s a Saturday afternoon blog post!

From one valid (if extreme) perspective, we don’t own our code, and we have no investment in it. All code is bad, and a positive review brings only grim satisfaction. A negative review is a joyous opportunity to learn, not a personal insult. We toil beneath our hooded robes.

Very few people can be this detached from their output. (I try!)

A “review granted” or “review not granted” email brings most of us a burst of emotion; even having reached a very low-ego place with respect to my code (it’s hard to make it through two hundred code reviews and still derive my self-worth from a patch!), I still feel good when I get r+, and a mild shot of frustration and disappointment when I get r-. Despite the fantastic opportunity to learn inherent in each review, we need to fight through our emotions to get to the point of learning. Marko Kloos writes about a similar process in his discussion of writers’ rejection letters.

If I still sometimes feel these things, I thought, then what about people who aren’t like me? Perhaps they’re volunteer contributors with other demands on their time. Perhaps they’re just less motivated, or have more self-worth riding on each patch. Perhaps they are junior engineers who feel that they have to prove themselves. A blunt, factual, “this is totally wrong” review, as so many of us tend to give, can be a shattering blow to a person’s motivation. Even people with practice dissociating their ego from their work can take a motivation hit; it’s not just the personal involvement, but also the frustration at a setback.

I can already hear the comments: “but they shouldn’t feel like that! It’s just a code review!”. My rebuttal is “but people do, because most people are not like you, and not accounting for that can be damaging in all kinds of ways”.

For a lot of people, how you phrase your code review will dictate whether the patch ever lands at all, or at least how soon it lands.

I’ve experienced a few approaches to code review that I think might help.

The absolute first thing is to provide a useful review. Just like an email, if your review doesn’t tell the author what they need to do to get the patch approved, then they need to double their effort: they not only have to fix the patch, but they have to put in the time to find out how first! The more clearly you can state what you want, and ideally how to get there (“try looking at this test to see how to do it”), the more likely it is that the author will be able to keep on rolling.

The second is an elaboration on the first: engage. Don’t just throw an r-; walk over (or call, or IRC) and talk through it, perhaps pairing to get to a patch that’ll pass review. Rather than just “no”, spend a little time to make sure that the contributor is actually moving forward and motivated. That might mean you spend some time trying to solve someone else’s problem. That’s fine; it’s an investment in the code, just like code review, but it’s also an investment in the author!

The third is to be careful with your wording. Follow the same kinds of tips that you read in magazines about having a constructive argument: avoiding finger-pointing, focusing on solutions or improvements rather than problems, etc. This is a little bit of a hack, but people don’t stop being people when they open Bugzilla. There is a difference between “you forgot to clear session state here, so the following test exercises the completely wrong thing (you idiot)” and “we should make sure that the following test still works when you clear session state at the end of this one, like this…”. It’s small, but a few of those per review, and a few reviews a month, will eventually leave a sour taste in a contributor’s mouth. Do you want people to be apprehensive about asking you for review?

On a related note, be careful with your flagsMarco Bonardo once reviewed one of my patches. I was new to that component, and he gave me very detailed feedback on everything from style to direction. Something that stuck in my mind was that he cleared the review flag and set feedback+, rather than review-. The patch wasn’t anywhere near good enough to land, but rather than coming across as a rejection it was presented as “great start, keep it up!”. This put me in a positive mood and encouraged me… and I’m pretty unfazed about negative reviews! That technique went straight into my reviewing toolbox. (This is the same kind of diplomatic response that normal human beings use all the time in social situations: “do you like my new jacket?” “it really complements your jeans!”.)

Similarly, as a patch author I sometimes find a flaw in my own code and clear my review? flag; as a reviewer one can do the same, which has a smaller ego hit than that big minus sign.

Does anyone else have any good tips?

Squeaky doors

I work (remotely) with a senior development manager. His office door squeaks.

I know that he’s been in that office for several years, yet every time we have a video conference — multiple times each week — I’m greeted by a series of loud creaks as each physical participant enters the room.

I’m pretty sure he no longer notices the squeaky office door; perhaps it’s even a charming, reassuring quirk of his environment. Maybe he jokes “oh yes, you get used to it”, or “haha, one day I’ll bring in some oil”.

I notice the squeaky door. In fact, I’m a little surprised that anyone could ignore it for so long.

The squeaky door is, of course, a parable about bad tools or environments. (It’s also absolutely real: if I wasn’t a thousand miles away, I would have oiled that door a long time ago!)

When I moved into my shared office as a new PhD student, many years ago, the door squeaked. When I went to get coffee it would squeak. Running down the hall for a printout? Squeak, squeak. The very next day I brought in a small bottle of oil, and two minutes later the door was swinging silently.

It pays dividends to spend a little time working on removing the frictions we encounter every day: from oiling hinges, to moving furniture, all the way to switching build tools or deciding to work remotely. (I’d call a frustrating commute a big friction!)


I’ve always had a little bit of an obsession with tools. Building, buying, using, admiring.

All kinds of tools. I have an attachment to the shiny ones — pocket knives, clicking open and fitting the hand; watches; firearms, steel reciprocating and rotating under huge stresses with minuscule tolerances. The dirty, heavy ones, too; these were part of my youth. Large metalworking lathes, striking off big curls of swarf; old drill presses with the chuck key attached by a chain, carefully positioned so that the safety guard would interfere if the chuck key were accidentally left in place.


These all have some things in common: they’re built with singularity of purpose; they have been refined over years (sometimes millennia); and, whilst they demand respect from the user (these are not toys), their workings and affordances are obvious to their intended audience. There is no mollycoddling, no wizards, no DRM.

Having an ample collection of tools is a joy. With the right tool, any job is easy. Without it, one struggles, messes up, gets dirty, gets frustrated.

Yesterday I changed a couple of wheels on my truck. Each wheel weighs perhaps 70lb with the tire mounted.

My jack stands were too large for the restricted space around each jacking point, so I had to do one wheel at a time, using just my shop jack. The lever on the jack was too short, so I had to stretch under the rear of the (elevated!) truck to work it. Lifting the wheels into position on my own was a challenge — they are 33″ across and at least a foot deep. My torque wrench was at the limits of its range, and still needed help from my breaker bar extension. Even my heavy-duty impact wrench had difficulty breaking the factory lug nuts loose.

A job that would have been easy on a small car turned out to be a sweaty, dirty mess. Inadequate tools. Lesson learned.

Making tools as you need them is an important skill, and an important attitude. Machinists, mechanics, woodworkers… these people are all used to building jigs and tools to make their jobs possible, or to make difficult tasks repeatable. I am often surprised at the number of software developers who only use tools, never making their own. They will perform tiresome tasks by hand because Visual Studio doesn’t have a plugin for it, or because their shell isn’t up to the task. They will allow their existing tools to dictate their technology selection, because it never occurs to them to make their own. Perhaps they aren’t lazy enough (laziness being a chief virtue of a programmer), or maybe they have a high tolerance for frustration. Me, I get annoyed every time I have to use Remote Desktop to manually install a test environment. “You mean I have to use the mouse?!”

Programming tools are one of the few pieces of software for which you, the developer, are also the end user: scratching an itch. It should be second nature to build them; even to spend half or more of your time building tools to do your work for you. (From one perspective, this is what macros are all about.)

I don’t just mean making new tools, either: we should constantly look for existing processes, tools, systems, and components which are inadequate, no longer fit, or could be expanded to make our lives better, and replace them. These are the levers by which we move the world: shouldn’t we constantly look for longer ones?

One of the most frustrating work situations I’ve encountered is when my two previous points collided: having bad tools, and entrenched organizational forces that prevented the introduction of new, better ways of doing things. Few within the organization were aware of the alternatives, and the cost of changing was high, so hundreds or thousands of people plodded on each day, doing the software equivalent of building pyramids out of high-tech bricks, but moving them with rolling logs and frayed rope.

There’s no good way out of this situation: nobody can justify the disruption of a major shift in tools to upper management — the old way works, so it’s hard to even suggest it. If you’re lucky, some brave soul will do their best to incrementally improve the old technology: low-profile rolling logs, sharper stone axes. “Crappy Tool v2.0, Now With Slightly Less Suck”. There is no leader to drive the change past the entrenched resistance, because nobody who can justify the expense actually suffers from the bad tools.

The only solution, I think, is to avoid stagnation at all costs, because stagnation eventually becomes impossible to escape. Just as with engineering debt, if you build up too much it can be very difficult to shift. Be mindful of over-attachment to old tools, and constantly strive to use the best. Re-evaluate, re-implement, and replace.

And don’t ever hire anyone who doesn’t build their own tools.

Time for a new technical blog

The barrier to entry of my extremely clunky old blog was preventing me from writing… so here’s a new one. If only all things in life were so easy. Over time I might scavenge and re-import good stuff from the old into the new, but don’t count on it.

What’s in a name? As those who’ve worked with me are aware, I’m fond of the old saw that once you’re done with the first 80% of making something (particularly software), you’ve still got the last 80% to go. I aim to cover the whole 160% here.

Expect neat little code snippets, thoughts on user experience, and grand commentary on the distortion of Alexander’s architectural pattern language into the horror that is the modern software patterns movement. Or just stuff that’s too long for Twitter.