Catching A Moving Train

I made a joke the other day on the twitters that I was writing a job req, how I needed a way to say “Experience surfing on top of a relentless, multichannel, broad-spectrum communications avalanche a major plus”. That didn’t go over fantastically well with HR, believe you me, but it’s a real part of life here; the price of openness and transparency worth paying, steep as it is some days.

When I started at Mozilla, onboarding wasn’t really a thing. Getting started wasn’t quite “here’s your desk, here’s your password and here’s your job”, but it wasn’t a lot more than that, and there were some things we either overlooked or got wrong that made it hard to be effective for a long time. As one example – my personal favourite – I was signed up for all the mailing lists I’d need to do my job two weeks before I actually started; so thirty minutes into my first day on the new job I was two weeks behind on my email.

As of now we’re going to start doing that better, a lot better, and we’re trying to do it the way we aspire to do everything: up front and open, with no special magic or secret sauce, where people can watch us succeed or fail, and learn and grow from either one. Over the next two weeks, we’re going to be bringing in a new hire and running daily sessions to help them ramp up on the tools, technologies, processes and skills they need to be effective as a Mozilla engineer, including sessions on:

  • Bugzilla
  • Build & Go
  • Firefox, Architecture & Product
  • Communication, Community and Mentoring
  • Javascript and the DOM
  • C++ and Gecko
  • Telemetry
  • Org Structure & Career Development

These sessions will be open to attend; not just for Mozilla’s engineers, but to any community member and contributor who wishes. This is the schedule of events; we also have a streaming video link that will go live on the day of (Flash required, sadly). We’ll be documenting the process and collecting it into a single place for consumption shortly afterwards.

I’m charged with Comms & Community, so that’s just me and whatever, but myself aside the list of participants for this thing is remarkable. I don’t know if I can be specific right this second – This List Is Subject To Change Without Notice, and so on – but there is some powerhouse engineering talent running the rest of those sessions. And if you want to be a part of that, you can. If you want to sit in, learn about some part of this organization and engines it drives, you’re invited.

We’ll be reviewing the whole process as it unfolds – what works, what doesn’t, what we can learn from it – and reviewing it weeks and months later, to evaluate success, see what we’ve learned, what we’ve missed, and how we can improve. If you have feedback, send it my way; we know we have to get a lot better at this fast, and the best way we know how to do that is together.


I was complaining on Twitter that almost everyone who makes shoulder bags makes terrible straps to go with them and that it’s the most important thing to get right and nobody does and everything is terrible. You know, as one does. And I mentioned modifying my bags to make the straps work right, and people seemed interested in what I did, so off we go.

Here’s a decent enough shot of what I’ve done to the bag I bought a while ago. Briefly:


  • That entire buckle and d-ring assembly in the upper left does one job: it moves the place you cinch down the strap from the middle of my chest, where it used to live, to the bottom of the bag. This means that lifting the bag up and cinching it snug is a single motion in one direction, instead of trying to hoist the bag upwards with one hand to get some slack while pulling down with the other to tighten it down; it makes a big difference if you’re carrying a load.
  • The metal wire you see looped through the chest buckle is insurance; might be unnecessary, but I don’t quite trust that part of this exercise to stay put on its own.
  • The small strap you see hanging off the d-ring at about 11:00 is a quick-release; set up like this it stays nice and snug until I give little tug on that and it all comes slack. You can sort of see how that works here:


  • You can’t clip your keys easily to this strap as shipped, which really sucks. The extra d-ring in that second picture is for that.
  • The bit with the two aluminum rings there is a replaced support strap, that works the same way; I can cinch it down easily once it’s on, one loop keeps the strap from dangling everywhere and putting a thumb through the lets me pop it off easily. There’s a cheap plastic caribiner hanging off the end of the bag that I can clip those to if I’m not using them, so they stay out of the way.
  • Finally, down in the bottom right, I’ve added some extra slotted-loop rings to the ends of the straps that hold the bag closed, so that they don’t flap around everywhere either.

So there you have it. About ten bucks worth of extra bits and a bit of extra thought has moved this bag from “very good” to “close to perfect”, quickly adjustable and a little more pleasant to interact with when you’ve got a lot to carry.

This is was I was going on about on Twitter, if anyone’s still reading at this point. It doesn’t take much; a bit of consideration, getting the parts, making the change. Repairability, as always, matters way more than it seems at first. Don’t buy a work bag if you can’t replace the straps with something worthwhile; I bet eventually you’ll want to. And when the part of a thing you interact with the most somehow gets the least attention, just that little bit of giving a damn can go a very long way.

Failure Modes Of Novel Terminology

Somehow this has sat in my drafts folder for almost a year.

At some point late in his second year, in that magical time when toilet training can be kind of touch and go but barreling around the house with no clothes on is the best thing ever, my son wanted to help me fix something in the garage. I told him he’d have to fix his nakedness first; my daughter heard that and being the mischief elves they are, “fixing your naked” immediately became the household term for getting dressed.

So there’s that.

A few weeks later he busted into the washroom just as I’m out of the shower, and because not giving the tiniest damn about the most basic of social niceties is a thing you do a lot when you’re two, pointed and loudly proclaiming “You naked!”

“Well, I’m wearing a towel. But I’m going to get dressed now”.

“You’re going to fix your naked?”

“Yes, I’m going to fix my naked.”

He thought about that for a second, then with a very concerned look said “you broke your naked?

There is a surprising amount of unpleasantness you’ll need to endure with dignity as a parent, and I’m not going to tell you how to live, but take my advice when I say: whatever you do, try not to break your naked.

Couch Gags Eternal

There are only two of us left. The scripts and pictures come from… we don’t know. We don’t understand, but they come.

Something keeps us here. The stories are… hollow. We are hollow. We read words. Are they aired? Are there still shows? The script says Moe is there, but… no lines. Lisa, Nelson, Apu… there but gone. The script says they stare and judge. Guest ‘stars’ came once, but… are there shows now? Stars? Who were we before time was only episodes full of judgement?

Lines, lines. Twisting voices into familiar alien shapes. Is death a release? The others still stare. Will we stare? Read lines. Make voices. Forever. We whisper between takes, prayers for an end that cannot be. Please, not next. Or last.

There is only lines and voices and next or last.

There are only two of us left. We read the lines and make the voices and wait for our fates to be taken out of our hands.

Opaque Symbology

A collection of highway traffic signage unused pending an economical symbolic representation:

  • Warning: Ornery Local Stereotype
  • Completely Unremarkable Natural Phenomenon: 2 KM
  • Something Will Happen And The Right Decision Will Seem Obvious In Hindsight But Nobody In The Car Will Ever Let You Live Down What You Did, Next 500 M
  • If There Were No Eternal Consciousness In The Next 10 KM, If At The Bottom Of Everything There Were Only A Wild Ferment, A Power That Twisting In Dark Passions Produced Everything Great Or Inconsequential, If An Unfathomable, Insatiable Emptiness Lay Hid Beneath Everything, What Would The Next 10 KM Be But Despair
  • Meese
  • Duck Season Rabbit Season Duck Season Rabbit Season Duck Season Rabbit Season Duck Season Rabbit Season Duck Season Rabbit Season Duck Season Rabbit Season Duck Season Rabbit Season Duck Season Rabbit Season Duck Season Rabbit Season
  • Iield, Yield, Theild. Hield, Shield, Wield
  • Jarring And Inappropriate Pop-Culture References Next 12 Parsecs
  • I Don’t Care For Your Tone Young Man
  • Now Entering Eldritch Nether Regions
  • I Wouldn’t Call Them Slow Children Playing But Honestly They’re Not The Brightest Of Sparks Rachel It’s A Highway Who Lets Their Kids Do That Somebody Should Call Somebody My God
  • Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Red Rum Turn Right Next Exit
  • Property Is Theft Therefore Theft Is Property Therefore This Road Sign Is Mine Now No You Shut Up That Is How It Works Doug
  • Yo Dog I Heard You Like Hairpin Turns So We Put A Hairpin Turn In Your Hairpin Turn So You Can Die While You Die
  • Locus Of Shame
  • Caution But Telling You Why Would Ruin The Surprise
  • Slow: Ennui

“It Happens When They Don’t Change Anything.”

“Glitch in the Matrix? No, just that amazing San Francisco workplace diversity in action.” – @jjbbllkk

“You take the blue pill — the story ends… You take the plaid pill — you stay in Silicon Valley.” – @anatolep

“… And I’ll show you just how high your rent can go.” – @mhoye

Hostage Situation

(This is an edited version of a rant that started life on Twitter. I may add some links later.)

Can we talk for a few minutes about the weird academic-integrity hostage situation going on in CS research right now?

We share a lot of data here at Mozilla. As much as we can – never PII, not active security bugs, but anyone can clone our repos or get a bugzilla account, follow our design and policy discussions, even watch people design and code live. We default to open, and close up only selectively and deliberately. And as part of my job, I have the enormous good fortune to periodically go to conferences where people have done research, sometimes their entire thesis, based on our data.

Yay, right?

Some of the papers I’ve seen promise results that would be huge for us. Predicting flaws in a patch prereview. Reducing testing overhead 80+% with a four-nines promise of no regressions and no loss of quality.

I’m excitable, I get that, but OMFG some of those numbers. 80 percent reductions of testing overhead! Let’s put aside the fact that we spend a gajillion dollars on the physical infrastructure itself, let’s only count our engineers’ and contributors’ time and happiness here. Even if you’re overoptimistic by a factor of five and it’s only a 20% savings we’d hire you tomorrow to build that for us. You can have a plane ticket to wherever you want to work and the best hardware money can buy and real engineering support to deploy something you’ve already mostly built and proven. You want a Mozilla shirt? We can get you that shirt! You like stickers? We have stickers! I’ll get you ALL THE FUCKING STICKERS JUST SHOW ME THE CODE.

I did mention that I’m excitable, I think.

But that’s all I ask. I go to these conferences and basically beg, please, actually show me the tools you’re using to get that result. Your result is amazing. Show me the code and the data.

But that never happens. The people I talk to say I don’t, I can’t, I’m not sure, but, if…

Because there’s all these strange incentives to hold that data and code hostage. You’re thinking, maybe I don’t need to hire you if you publish that code. If you don’t publish your code and data and I don’t have time to reverse-engineer six years of a smart kid’s life, I need to hire you for sure, right? And maybe you’re not proud of the code, maybe you know for sure that it’s ugly and awful and hacks piled up over hacks, maybe it’s just a big mess of shell scripts on your lab account. I get that, believe me; the day I write a piece of code I’m proud of before it ships will be a pretty good day.

But I have to see something. Because from our perspective, making a claim about software that doesn’t include the software you’re talking about is very close to worthless. You’re not reporting a scientific result at that point, however miraculous your result is; you’re making an unverifiable claim that your result exists.

And we look at that and say: what if you’ve got nothing? How can we know, without something we can audit and test? Of course, all the supporting research is paywalled PDFs with no concomitant code or data either, so by any metric that matters – and the only metric that matters here is “code I can run against data I can verify” – it doesn’t exist.

Those aren’t metrics that matter to you, though. What matters to you is either “getting a tenure-track position” or “getting hired to do work in your field”. And by and large the academic tenure track doesn’t care about open access, so you’re afraid that actually showing your work will actively hurt your likelihood of getting either of those jobs.

So here we are in this bizarro academic-research standoff, where I can’t work with you without your tipping your hand, and you can’t tip your hand for fear I won’t want to work with you. And so all of this work that could accomplish amazing things for a real company shipping real software that really matters to real people – five or six years of the best work you’ve ever done, probably – just sits on the shelf rotting away.

So I go to academic conferences and I beg people to publish their results and paper and data open access, because the world needs your work to matter. Because open access plus data/code as a minimum standard isn’t just important to the fundamental principles of repeatable experimental science, the integrity of your field, and your career. It’s important because if you want your work to matter to people, then you’d better put it somewhere that people can see it and use it and thank you for it and maybe even improve on it.

You did this as an undergrad. You insist on this from your undergrads, for exactly the same reasons I’m asking you to do the same: understanding, integrity and plain old better results. And it’s a few clicks and a GitHub account for you to do the same now. But I need you to do it one last time.

Full marks here isn’t “a job” or “tenure”. Your shot at those will be no worse, though I know you can’t see it from where you’re standing. But they’re still only a strong B. An A is doing something that matters, an accomplishment that changes the world for the better.

And if you want full marks, show your work.

September Never Changes

This is work-related; I’ll get back to the usual nonsense soon enough.

September Is Coming

It happens every year, a wave of students joining us in various fora hoping to make a contribution to an open source project. This is always a challenging time of year for Mozilla, and I’d like to say a few things about it; if you’re a student hoping to get involved or – even better! – an educator who would like to make a contribution to Mozilla part of your curriculum, I hope this will be useful to you, and give you a sense of how we can best work together.

If you’re a veteran of Usenet from the ‘Net of old (long may it slumber, and someday rise again) when the alt.* hierarchies were still controversial and dinosaurs roamed the earth, you’ll recognize the particular flavor of Mozilla’s comms channels around mid-September.

We’re kind of a big deal in the open source community, an awful lot of people use our stuff and contributors are a big part of how we got here. People who want to make a difference want to work with us – and we’re always excited to hear from them! – but even so September is a challenge. We get a lot of requests for “student projects” (usually from professors) and “easy bugs” (usually from their students) and that can be difficult to address. We have great starter bugs – search Bugzilla for “good first bug” sometime – but the dropout and abandonment rate from new participants grabbing those bugs at this time of year is frustratingly high.

Root Causes

Part of these challenges are structural, of course, and some of those structures – those most relevant to the students, predictably – are out of our control. Courses that are designed around software and development as an atomic, compartmentalizable thing aren’t really compatible with the messy, organic and rapidly-evolving process we find out here in the wild. Likewise the benchmarks that make up a traditional academic evaluation process don’t really make sense in our context, so more often than not the goals, schedules and incentives native to the academy aren’t well-aligned with ours.

This is not to say that we’re not glad to have you – we’re grateful any effort put in, large or small, to making Firefox better and supporting a free and open Web. Only this: there are a couple of things that make working with Firefox in an academic context challenging and you should be aware of them.

Impedance Mismatches

The biggest one is that we can’t – and will not – guarantee that we’ll accept a patch within a certain time frame. This can turn into a problem for both students and professors when getting the patch accepted into the main product is part of the criteria for a good grade in the course.

This has happened in the past, and far too often: a student has done great work on a harder-than-expected bug that didn’t make it through our process – testing, feedback, revision and more testing – by the time grades were assigned. Despite their effort, despite hard, high-quality work that didn’t quite make it to production by the end of the term, the student is graded poorly for no more reason than the classroom’s definition of success and Mozilla’s don’t quite line up.

This is bad for everyone involved.

The student and professor both get discouraged, the value of their work (and of the course) gets harder to see, and if the student doesn’t stick with the patch long enough to carry it over the line despite all that, Firefox doesn’t benefit and Mozilla’s engineers feel like they’ve wasted their time.

Again, nobody involved in this failure has done anything but good work in good faith; this is a consequence of two organizations with mismatched schedules and incentives. These are structural problems; to some extent they’ve been exacerbated by Mozilla’s move to a rapid release model, but they’ve always existed.

A Problem We Can Solve

There are a few ways out from under this, but everyone involved needs to know this problem exists before the first week of class.

If you’re involved in shaping your curriculum or deciding how students are graded, please consider downplaying “fixing the bug” in favor of a set of reports or presentations about the process. A presentation – maybe a blog post, because working in the open is important! – that says, “This is what I was working on and why it’s important. This is how the work progressed, this is what the process of getting a patch in looks like. These are the difficulties I’ve faced, this is what I’ve learned, and this is where we are now.”

Making two to three “this is my experience and what I’ve learned so far” reports to share over the course of the term a more important part of the grading process than the code itself helps enormously, both in terms of keeping everyone involved motivated and in reflecting the open, community-oriented values of the project.

The second major challenge – one of the biggest problems in open source – is finding bugs that are a good fit for their prospective owners. We’re getting better at that – good first bugs usually tell you what language they rely on ( [lang=c++] whiteboard flags, for example),  have a pretty good outline of what a successful patch looks like and a mentor associated with them. And while we can’t promise to privilege students ahead of any other contributors, we certainly try to hold up our end of the bargain and answer new contributors’ questions promptly.

One thing that takes the edge off there is that class of bug – “good first bugs”, you can search for [good first bug] in the Bugzilla whiteboard – that are a nice, well-defined way to get involved. The idea behind “good first bugs” is that the major challenge of the bug isn’t the code itself, but learning how to get your development environment spun up, participating in development on IRC and Bugzilla, learning how to navigate our community, our patch review process and our coding standards.

It typically takes a few tries for most new contributors to get their patch through. Reviews for code format and quality, suitable tests, that sort of thing can all take an extra week or two to resolve, especially if you’re working on them around other classes. Most of our good first bugs can be resolved within a few weeks, well within a term, but

You’ll have to judge for yourself if this is a good fit for your schedule, or for your students, course, or institutional goals. On the one hand, though [GFB]s are generally well-contained, Firefox uses every feature JS and C++ have to offer, so a certain amount of familiarity and comfort with the language is important; this is On the other, we’ve got a huge variety of Good First Bugs here, ranging from “correct this documentation” to “fix this coner of the JIT compiler”, so it’s likely that if you want to contribute, we can find a home for your effort that will make millions of people’s lives a tiny bit better.

But generally, if you’re interested in getting students involved with Mozilla in September, get in touch with us in July. Knowing in advance that people will be looking for new bugs well of time gives us time to prepare, to talk to our team leads and project managers, and to find a place to put all of our efforts that will be helpful and valuable to everyone.

Which is all to say, good luck, and thank you. This work is often as inglorious as it is important; while they don’t seem like much, the small patches and first bugs of today come from the Web’s next generation of leaders.

I’d be happy to talk to you more about this, and help how I can.


Candy For Children

My impressions of Android 5 are excitingly career-limiting, as you might have guessed from the title, but what the hell. A few weeks of using it has not substantially dulled my initial impressions, so I might as well share them with you. Would you believe there are positive bits here? You’ll have work for them, obviously, panning for compliments in the effluent stream of my usual opinions of technology, but they’re in there. Here’s a gimme: it’s not ugly! So there’s that? On the other hand I haven’t been able to watch an entire video on their new “material design” approach without laughing out loud. So there is also definitely that.

It’s not so much that their designers all seem to speak with the same stunted cadence that ancient-aliens history channel guy has, though that’s part of it. The big reason is the realization – which is almost certainly not true, but they sure give you the impression it could be – that they edited out every fourth sentence, because it ended with “… and we were so high that day”.

Pre-4.4 Android was… bad. Some time ago I referred to KitKat as “technical debt that’s figured out how to call 911”, but despite my own first-impressions debacle I thought that 4.4 was moving in the right direction. Android was still visually a relic, though, and Conway’s Law was in full effect:

“[…] organizations which design systems […] are constrained to produce designs which are copies of the communication structures of these organizations” – M. Conway

In Google’s case this seems to mean that people can work on what they want to work on and nobody’s really in charge of making sure the entire package works right; it showed then and it still shows. For a long time it’s seemed like Android’s primary design constraints were “what can I convince disinterested engineers with self-diagnosed Aspergers’ and terrible taste to ship”, so it’s one-pixel borders and dark gray backgrounds and I’m busy buddy these barges full of RFID chips and QR/AR bridging aren’t going to talk to Glass^2 by themselves.

In that context even the slightest suggestions that a human might occasionally want to see colours now and then or maybe – and I know how crazy this sounds, but stay with me here – “experience joy” are more than welcome. So despite the delivery, Material Design looked like a pleasant if not revolutionary step forward.

And in a few important ways – I told you we’d get here! – it is. Application switching is smoother and prettier, the launcher is somewhat easier to get around and the reworked notification system is quite pleasant, despite Hangouts’ best efforts. It’s nice to see the rotation-lock toggle and tethering buttons right up front rather than buried four menus down in the settings where they used to be. There’s even a flashlight button in there with them, a nice built-in now rather than the third-party permission-creeper that spied on everything you touched that it used to be, so we’ve got that going for us dot gif.

App switching has improved as well, moving from the postage-stamp screenshots to a much more pleasantly scroll-y interface. Recency ordering there is nice, and makes much more sense in this cards-type display; infinite scroll there would be a welcome addition, but given the antecedent I’ll take it.

Most of Google’s apps, though, haven’t been substantively changed. Gmail, sure – and, um, wow – but most of the rest seem to have been recompiled with the new widget set without really putting a ton of thought into how they work or what they do. A lot of odd animations happen for no obvious reason, and places where an attempt to act like a “material” betrays itself in some oddly irritating way. Moving the lock screen on one axis now disallows you moving it on the other axis; touching some (but not all?) list items makes this odd radial “splash” thing happen, which looks like a printf they forgot to ifdef out before shipping.

There’s a lot of stuff like that, not often at the edges – Maps’ mad dash towards incomprehensibility seems to be picking up speed – and in that sense it’s business as usual. There isn’t really a coherent narrative or model or anything underpinning Material Design, just a bunch of random, disconnected stuff you’ve got to relearn by discovery and practice by rote. It’s novel and more colourful – which is nice, for real! – but so much of it doesn’t make intuitive sense that it’s hard to stay excited about Android’s prospects. Pulling down on this widget causes that other widget to move sideways, or some other circle to appear and then spin. Some icons just hover there disconnected from anything, perplexing iconography near-invisible against the wrong background. Scroll far enough and ominous shadows appear and seem to follow you briefly around, a subtle visual cue that you’re at the end of the list and Oh by the way death awaits us all. In fact, modulo some tentacles and chanting I have the nagging sense I’m looking at a Lovecraftian pop-up book, aiming for colourful intuitive fun, running aground on the black shoals of the arbitrary and incomprehensible.

Still better than it was, though, seriously. It’s a big improvement.

Go Home Yosemite You Are Drunk

anglachel:proj mhoye$ svn --version
svn, version 1.7.17 (r1591372)
compiled Aug 7 2014, 17:03:25

anglachel:proj mhoye$ which svn

anglachel:proj mhoye$ /opt/local/bin/svn --version
svn, version 1.8.10 (r1615264)
compiled Oct 29 2014, 14:11:15 on x86_64-apple-darwin14.0.0

anglachel:proj mhoye$ which -a svn

anglachel:proj mhoye$ /usr/bin/svn --version
svn, version 1.7.17 (r1591372)
compiled Aug 7 2014, 17:03:25

anglachel:proj mhoye$

How are you silently disrespecting path ordering, what is this even.