Hostage Situation

(This is an edited version of a rant that started life on Twitter. I may add some links later.)

Can we talk for a few minutes about the weird academic-integrity hostage situation going on in CS research right now?

We share a lot of data here at Mozilla. As much as we can – never PII, not active security bugs, but anyone can clone our repos or get a bugzilla account, follow our design and policy discussions, even watch people design and code live. We default to open, and close up only selectively and deliberately. And as part of my job, I have the enormous good fortune to periodically go to conferences where people have done research, sometimes their entire thesis, based on our data.

Yay, right?

Some of the papers I’ve seen promise results that would be huge for us. Predicting flaws in a patch prereview. Reducing testing overhead 80+% with a four-nines promise of no regressions and no loss of quality.

I’m excitable, I get that, but OMFG some of those numbers. 80 percent reductions of testing overhead! Let’s put aside the fact that we spend a gajillion dollars on the physical infrastructure itself, let’s only count our engineers’ and contributors’ time and happiness here. Even if you’re overoptimistic by a factor of five and it’s only a 20% savings we’d hire you tomorrow to build that for us. You can have a plane ticket to wherever you want to work and the best hardware money can buy and real engineering support to deploy something you’ve already mostly built and proven. You want a Mozilla shirt? We can get you that shirt! You like stickers? We have stickers! I’ll get you ALL THE FUCKING STICKERS JUST SHOW ME THE CODE.

I did mention that I’m excitable, I think.

But that’s all I ask. I go to these conferences and basically beg, please, actually show me the tools you’re using to get that result. Your result is amazing. Show me the code and the data.

But that never happens. The people I talk to say I don’t, I can’t, I’m not sure, but, if…

Because there’s all these strange incentives to hold that data and code hostage. You’re thinking, maybe I don’t need to hire you if you publish that code. If you don’t publish your code and data and I don’t have time to reverse-engineer six years of a smart kid’s life, I need to hire you for sure, right? And maybe you’re not proud of the code, maybe you know for sure that it’s ugly and awful and hacks piled up over hacks, maybe it’s just a big mess of shell scripts on your lab account. I get that, believe me; the day I write a piece of code I’m proud of before it ships will be a pretty good day.

But I have to see something. Because from our perspective, making a claim about software that doesn’t include the software you’re talking about is very close to worthless. You’re not reporting a scientific result at that point, however miraculous your result is; you’re making an unverifiable claim that your result exists.

And we look at that and say: what if you’ve got nothing? How can we know, without something we can audit and test? Of course, all the supporting research is paywalled PDFs with no concomitant code or data either, so by any metric that matters – and the only metric that matters here is “code I can run against data I can verify” – it doesn’t exist.

Those aren’t metrics that matter to you, though. What matters to you is either “getting a tenure-track position” or “getting hired to do work in your field”. And by and large the academic tenure track doesn’t care about open access, so you’re afraid that actually showing your work will actively hurt your likelihood of getting either of those jobs.

So here we are in this bizarro academic-research standoff, where I can’t work with you without your tipping your hand, and you can’t tip your hand for fear I won’t want to work with you. And so all of this work that could accomplish amazing things for a real company shipping real software that really matters to real people – five or six years of the best work you’ve ever done, probably – just sits on the shelf rotting away.

So I go to academic conferences and I beg people to publish their results and paper and data open access, because the world needs your work to matter. Because open access plus data/code as a minimum standard isn’t just important to the fundamental principles of repeatable experimental science, the integrity of your field, and your career. It’s important because if you want your work to matter to people, then you’d better put it somewhere that people can see it and use it and thank you for it and maybe even improve on it.

You did this as an undergrad. You insist on this from your undergrads, for exactly the same reasons I’m asking you to do the same: understanding, integrity and plain old better results. And it’s a few clicks and a GitHub account for you to do the same now. But I need you to do it one last time.

Full marks here isn’t “a job” or “tenure”. Your shot at those will be no worse, though I know you can’t see it from where you’re standing. But they’re still only a strong B. An A is doing something that matters, an accomplishment that changes the world for the better.

And if you want full marks, show your work.

September Never Changes

This is work-related; I’ll get back to the usual nonsense soon enough.

September Is Coming

It happens every year, a wave of students joining us in various fora hoping to make a contribution to an open source project. This is always a challenging time of year for Mozilla, and I’d like to say a few things about it; if you’re a student hoping to get involved or – even better! – an educator who would like to make a contribution to Mozilla part of your curriculum, I hope this will be useful to you, and give you a sense of how we can best work together.

If you’re a veteran of Usenet from the ‘Net of old (long may it slumber, and someday rise again) when the alt.* hierarchies were still controversial and dinosaurs roamed the earth, you’ll recognize the particular flavor of Mozilla’s comms channels around mid-September.

We’re kind of a big deal in the open source community, an awful lot of people use our stuff and contributors are a big part of how we got here. People who want to make a difference want to work with us – and we’re always excited to hear from them! – but even so September is a challenge. We get a lot of requests for “student projects” (usually from professors) and “easy bugs” (usually from their students) and that can be difficult to address. We have great starter bugs – search Bugzilla for “good first bug” sometime – but the dropout and abandonment rate from new participants grabbing those bugs at this time of year is frustratingly high.

Root Causes

Part of these challenges are structural, of course, and some of those structures – those most relevant to the students, predictably – are out of our control. Courses that are designed around software and development as an atomic, compartmentalizable thing aren’t really compatible with the messy, organic and rapidly-evolving process we find out here in the wild. Likewise the benchmarks that make up a traditional academic evaluation process don’t really make sense in our context, so more often than not the goals, schedules and incentives native to the academy aren’t well-aligned with ours.

This is not to say that we’re not glad to have you – we’re grateful any effort put in, large or small, to making Firefox better and supporting a free and open Web. Only this: there are a couple of things that make working with Firefox in an academic context challenging and you should be aware of them.

Impedance Mismatches

The biggest one is that we can’t – and will not – guarantee that we’ll accept a patch within a certain time frame. This can turn into a problem for both students and professors when getting the patch accepted into the main product is part of the criteria for a good grade in the course.

This has happened in the past, and far too often: a student has done great work on a harder-than-expected bug that didn’t make it through our process – testing, feedback, revision and more testing – by the time grades were assigned. Despite their effort, despite hard, high-quality work that didn’t quite make it to production by the end of the term, the student is graded poorly for no more reason than the classroom’s definition of success and Mozilla’s don’t quite line up.

This is bad for everyone involved.

The student and professor both get discouraged, the value of their work (and of the course) gets harder to see, and if the student doesn’t stick with the patch long enough to carry it over the line despite all that, Firefox doesn’t benefit and Mozilla’s engineers feel like they’ve wasted their time.

Again, nobody involved in this failure has done anything but good work in good faith; this is a consequence of two organizations with mismatched schedules and incentives. These are structural problems; to some extent they’ve been exacerbated by Mozilla’s move to a rapid release model, but they’ve always existed.

A Problem We Can Solve

There are a few ways out from under this, but everyone involved needs to know this problem exists before the first week of class.

If you’re involved in shaping your curriculum or deciding how students are graded, please consider downplaying “fixing the bug” in favor of a set of reports or presentations about the process. A presentation – maybe a blog post, because working in the open is important! – that says, “This is what I was working on and why it’s important. This is how the work progressed, this is what the process of getting a patch in looks like. These are the difficulties I’ve faced, this is what I’ve learned, and this is where we are now.”

Making two to three “this is my experience and what I’ve learned so far” reports to share over the course of the term a more important part of the grading process than the code itself helps enormously, both in terms of keeping everyone involved motivated and in reflecting the open, community-oriented values of the project.

The second major challenge – one of the biggest problems in open source – is finding bugs that are a good fit for their prospective owners. We’re getting better at that – good first bugs usually tell you what language they rely on ( [lang=c++] whiteboard flags, for example),  have a pretty good outline of what a successful patch looks like and a mentor associated with them. And while we can’t promise to privilege students ahead of any other contributors, we certainly try to hold up our end of the bargain and answer new contributors’ questions promptly.

One thing that takes the edge off there is that class of bug – “good first bugs”, you can search for [good first bug] in the Bugzilla whiteboard – that are a nice, well-defined way to get involved. The idea behind “good first bugs” is that the major challenge of the bug isn’t the code itself, but learning how to get your development environment spun up, participating in development on IRC and Bugzilla, learning how to navigate our community, our patch review process and our coding standards.

It typically takes a few tries for most new contributors to get their patch through. Reviews for code format and quality, suitable tests, that sort of thing can all take an extra week or two to resolve, especially if you’re working on them around other classes. Most of our good first bugs can be resolved within a few weeks, well within a term, but

You’ll have to judge for yourself if this is a good fit for your schedule, or for your students, course, or institutional goals. On the one hand, though [GFB]s are generally well-contained, Firefox uses every feature JS and C++ have to offer, so a certain amount of familiarity and comfort with the language is important; this is On the other, we’ve got a huge variety of Good First Bugs here, ranging from “correct this documentation” to “fix this coner of the JIT compiler”, so it’s likely that if you want to contribute, we can find a home for your effort that will make millions of people’s lives a tiny bit better.

But generally, if you’re interested in getting students involved with Mozilla in September, get in touch with us in July. Knowing in advance that people will be looking for new bugs well of time gives us time to prepare, to talk to our team leads and project managers, and to find a place to put all of our efforts that will be helpful and valuable to everyone.

Which is all to say, good luck, and thank you. This work is often as inglorious as it is important; while they don’t seem like much, the small patches and first bugs of today come from the Web’s next generation of leaders.

I’d be happy to talk to you more about this, and help how I can.


Candy For Children

My impressions of Android 5 are excitingly career-limiting, as you might have guessed from the title, but what the hell. A few weeks of using it has not substantially dulled my initial impressions, so I might as well share them with you. Would you believe there are positive bits here? You’ll have work for them, obviously, panning for compliments in the effluent stream of my usual opinions of technology, but they’re in there. Here’s a gimme: it’s not ugly! So there’s that? On the other hand I haven’t been able to watch an entire video on their new “material design” approach without laughing out loud. So there is also definitely that.

It’s not so much that their designers all seem to speak with the same stunted cadence that ancient-aliens history channel guy has, though that’s part of it. The big reason is the realization – which is almost certainly not true, but they sure give you the impression it could be – that they edited out every fourth sentence, because it ended with “… and we were so high that day”.

Pre-4.4 Android was… bad. Some time ago I referred to KitKat as “technical debt that’s figured out how to call 911”, but despite my own first-impressions debacle I thought that 4.4 was moving in the right direction. Android was still visually a relic, though, and Conway’s Law was in full effect:

“[…] organizations which design systems […] are constrained to produce designs which are copies of the communication structures of these organizations” – M. Conway

In Google’s case this seems to mean that people can work on what they want to work on and nobody’s really in charge of making sure the entire package works right; it showed then and it still shows. For a long time it’s seemed like Android’s primary design constraints were “what can I convince disinterested engineers with self-diagnosed Aspergers’ and terrible taste to ship”, so it’s one-pixel borders and dark gray backgrounds and I’m busy buddy these barges full of RFID chips and QR/AR bridging aren’t going to talk to Glass^2 by themselves.

In that context even the slightest suggestions that a human might occasionally want to see colours now and then or maybe – and I know how crazy this sounds, but stay with me here – “experience joy” are more than welcome. So despite the delivery, Material Design looked like a pleasant if not revolutionary step forward.

And in a few important ways – I told you we’d get here! – it is. Application switching is smoother and prettier, the launcher is somewhat easier to get around and the reworked notification system is quite pleasant, despite Hangouts’ best efforts. It’s nice to see the rotation-lock toggle and tethering buttons right up front rather than buried four menus down in the settings where they used to be. There’s even a flashlight button in there with them, a nice built-in now rather than the third-party permission-creeper that spied on everything you touched that it used to be, so we’ve got that going for us dot gif.

App switching has improved as well, moving from the postage-stamp screenshots to a much more pleasantly scroll-y interface. Recency ordering there is nice, and makes much more sense in this cards-type display; infinite scroll there would be a welcome addition, but given the antecedent I’ll take it.

Most of Google’s apps, though, haven’t been substantively changed. Gmail, sure – and, um, wow – but most of the rest seem to have been recompiled with the new widget set without really putting a ton of thought into how they work or what they do. A lot of odd animations happen for no obvious reason, and places where an attempt to act like a “material” betrays itself in some oddly irritating way. Moving the lock screen on one axis now disallows you moving it on the other axis; touching some (but not all?) list items makes this odd radial “splash” thing happen, which looks like a printf they forgot to ifdef out before shipping.

There’s a lot of stuff like that, not often at the edges – Maps’ mad dash towards incomprehensibility seems to be picking up speed – and in that sense it’s business as usual. There isn’t really a coherent narrative or model or anything underpinning Material Design, just a bunch of random, disconnected stuff you’ve got to relearn by discovery and practice by rote. It’s novel and more colourful – which is nice, for real! – but so much of it doesn’t make intuitive sense that it’s hard to stay excited about Android’s prospects. Pulling down on this widget causes that other widget to move sideways, or some other circle to appear and then spin. Some icons just hover there disconnected from anything, perplexing iconography near-invisible against the wrong background. Scroll far enough and ominous shadows appear and seem to follow you briefly around, a subtle visual cue that you’re at the end of the list and Oh by the way death awaits us all. In fact, modulo some tentacles and chanting I have the nagging sense I’m looking at a Lovecraftian pop-up book, aiming for colourful intuitive fun, running aground on the black shoals of the arbitrary and incomprehensible.

Still better than it was, though, seriously. It’s a big improvement.

Go Home Yosemite You Are Drunk

anglachel:proj mhoye$ svn --version
svn, version 1.7.17 (r1591372)
compiled Aug 7 2014, 17:03:25

anglachel:proj mhoye$ which svn

anglachel:proj mhoye$ /opt/local/bin/svn --version
svn, version 1.8.10 (r1615264)
compiled Oct 29 2014, 14:11:15 on x86_64-apple-darwin14.0.0

anglachel:proj mhoye$ which -a svn

anglachel:proj mhoye$ /usr/bin/svn --version
svn, version 1.7.17 (r1591372)
compiled Aug 7 2014, 17:03:25

anglachel:proj mhoye$

How are you silently disrespecting path ordering, what is this even.

Social Engineering

I gave this talk at FSOSS last week, in which I try to reclaim the term “Social Engineering”, so that it stops meaning “get the receptionist to give you their password” and starts meaning “Measuring community growth and turning that into processes and practices that work.”

I thought it went well, though listening to it I can see I’ve got a couple of verbal tics to work on. Gotta stop using ‘um’ and ‘right’ as punctuation.

Rogue Cryptocurrency Bootstrapping Robots

Cuban Shoreline

I tried to explain to my daughter why I’d had a strange day.

“Why was it strange?”

“Well… There’s a thing called a cryptocurrency. ‘Currency’ is another word for money; a cryptocurrency is a special kind of money that’s made out of math instead of paper or metal.”

That got me a look. Money that’s made out of made out of math, right.

“… and one of the things we found today was somebody trying to make a new cryptocurrency. Now, do you know why money is worth anything? It’s a coin or a paper with some ink on it – what makes it ‘money’?”

“… I don’t know.”

“The only answer we have is that it’s money if enough people think it is. If enough people think it’s real, it becomes real. But making people believe in a new kind of money isn’t easy, so what this guy did was kind of clever. He decided to give people little pieces of his cryptocurrency for making contributions to different software projects. So if you added a patch to one of the projects he follows, he’d give you a few of these math coins he’d made up.”


“Right. Kind of weird. And then whoever he is, he wrote a program to do that automatically. It’s like a little robot – every time you change one of these programs, you get a couple of math coins. But the problem is that we update a lot of those programs with our robots, too. Our scripts run, our robots, and then his robots try to give our robots some of his pretend money.”


“So that’s why my day was weird. Because we found somebody else’s programs trying to give our programs made-up money, in the hope that this made-up money would someday become real.”


“What did you to today?”

“I painted different animals and gave them names.”

“What kind of names?”

“French names like zaval.”

“Cheval. Was it a good day?”

“Yeah, I like painting.”

“Good, good.”

(Charlie Stross warned us about this. It’s William Gibson’s future, but we still need to clean up after it.)

A Beautiful, Momentary Friendship

For about ten minutes this morning I was in a beautiful relationship.

I bike to work in the morning, and I’m pretty aggressive about it. I’m one of the scofflaw cyclists people like to complain about while they’re spending a few hours every day slowly dying in gridlock. I move so much faster than traffic, though, that their opinions hardly matter. Off peak hours (whenever those are) you can make a case for driving, I suppose? But in rush hour, in this city, nothing is faster than me. TTC, Porsche, Ducati, doesn’t matter.

Today, though.

This morning I’m cranking down the road, not full out but sure not dawdling, when a woman about my age riding with fenders and a pannier – a pannier! Wicker! – blows past me like it’s not even a thing. Whoosh.

This cannot stand, of course; the machismo bullshit is strong with me at moments like this. It’s a rare day and a rare treat for me that I get a rabbit to chase on my ride in, so I can’t miss this; I gear down take off after her.

After a while I catch up, start drafting – the two of us are flying down the road – and then pass her, but I’m not shaking her, oh no. She was not having that. I beat her to a light by about two lengths but she timed it better, got out in front of me again, took a better line around the traffic and started stretching her lead. She was raising her game here, and I did not have an easy time catching up. By the time I do I’m feeling it, and looking over it doesn’t look like I’m pushing her anywhere near as hard as she’s pushing me.

We went back and forth like that for about ten minutes, past everyone, trading leads and drafting around traffic and go go go until finally her commute took her north near where I turned south. I was grinning like a lunatic at the end of it, and she seemed happy as well; we shared a nod and went our separate ways, and that was that.

Whoever you are, that was one of the best rides in I can remember. I hope it was as much of a blast for you as it was for me.

Well done, and thank you.

Insecurity Theatre

Your new password must contain a mix of:

  • uppercase letters
  • lowercase letters
  • numbers
  • symbols
  • symbols that are also numbers
  • illuminati symbols
  • hobo signs
  • occult symbols (not illuminati)
  • old girlfriend’s phone numbers
  • hieroglyphs
  • fragrances
  • H.P. Lovecraft references
  • exotic spices
  • descriptions of that favorite sweater you lost in a breakup that one time
  • secret regrets
  • controversial onomatopoeia
  • limericks about a thermostat
  • vaguely sexual innuendos
  • anagrams of a word you can’t spell
  • favorite emoji
  • least favorite emoji
  • turnips
  • shrugs
  • ennui
  • cursory pats on the back
  • long stares into the middle distance
  • moments of quiet yearning for lost love (unrelated to sweater or secret regret)
  • cups of OK coffee
  • sense of resigned inevitability (minimum three)
  • irish setters
  • tweed hats

No repeat characters.

Practical Orthodonty

My dentist expressed some concern today that when he pokes my gums with a piece of sharp metal, they’re prone to bleeding.

My observation that almost every part of me has that quality was not well-received. He proposed some treatment for it, but when I told him I’d pay quite a bit extra to have him take whatever that is and dip my entire body in it that part of the conversation didn’t go all that swimmingly either. I said he could hold me by the heel, I know how this works, but no.

Anyway, long story short, apparently dentists have no sense of humour and flossing your entire body won’t make you invulnerable.

Now you know.

The Knife Shop


In Tsukiji there’s a small, open-fronted shop called “Tsukiji Masamoto”, and it’s packed. The walls are lined floor to ceiling with knives in various wooden cases, row after row of every tool you’d need to separate one part of some animal from another.

Their shapes were unfamiliar to my western eyes, specialized tools for jobs I know nothing about. Even the local equivalent of the west’s workaday one-size-fits-all chef’s knife, the santoku, seems to come in more shapes and sizes than makes sense. The cleaver-like usuba bocho doesn’t have an obvious western counterpart, the hard angles of the usagisaki hocho, the “eel knife”, likewise. And the savage economy of the soba kiri or udon kiri – literally “noodle knife”, because that’s all they’re for – looks more like the business end of something stylized and cruel than a common household utensil in its own right.

Most striking is the maguro bocho, made for filleting four hundred pounds of tuna in a single motion; some seven feet long, five of it blade, they seem more like a Daimyo’s tool than a fishmonger’s. It’s hard to believe they’re useful until you see how big a full-sized tuna can be; the nature of the tool becomes clear once you understand the nature of the job, as usual.

When I visited there was a man squared up over a whetstone out front, a man who looks like he’s made out of old leather and dock rope. He was holding a hon deba to the wheel in hands you could mistake for a bag of walnuts. He seemed to have been there forever; as far as I’m concerned he’s probably still there, a small man who stands like an old mountain. Tsukiji seemed to have been built around him; I had the impression some shogun’s son had found him standing in front of that wheel when it was still called Edo and returned home to say, father, we do need a fish market, and there is a man already there sharpening knives. Respectfully, father, I don’t think he’s going anywhere for anyone.

As I was watching him work he lifted the hon deba off the wheel and peered closely at its edge for a long time before he lowered the knife and stared at the sky for just as long. For a moment I could almost see a hint of dissatisfaction and then nothing; he put the edge back to the wheel, I moved on.

I think about this a lot; I wish I’d been able to ask him what he’d seen.