Hostage Situation

(This is an edited version of a rant that started life on Twitter. I may add some links later.)

Can we talk for a few minutes about the weird academic-integrity hostage situation going on in CS research right now?

We share a lot of data here at Mozilla. As much as we can – never PII, not active security bugs, but anyone can clone our repos or get a bugzilla account, follow our design and policy discussions, even watch people design and code live. We default to open, and close up only selectively and deliberately. And as part of my job, I have the enormous good fortune to periodically go to conferences where people have done research, sometimes their entire thesis, based on our data.

Yay, right?

Some of the papers I’ve seen promise results that would be huge for us. Predicting flaws in a patch prereview. Reducing testing overhead 80+% with a four-nines promise of no regressions and no loss of quality.

I’m excitable, I get that, but OMFG some of those numbers. 80 percent reductions of testing overhead! Let’s put aside the fact that we spend a gajillion dollars on the physical infrastructure itself, let’s only count our engineers’ and contributors’ time and happiness here. Even if you’re overoptimistic by a factor of five and it’s only a 20% savings we’d hire you tomorrow to build that for us. You can have a plane ticket to wherever you want to work and the best hardware money can buy and real engineering support to deploy something you’ve already mostly built and proven. You want a Mozilla shirt? We can get you that shirt! You like stickers? We have stickers! I’ll get you ALL THE FUCKING STICKERS JUST SHOW ME THE CODE.

I did mention that I’m excitable, I think.

But that’s all I ask. I go to these conferences and basically beg, please, actually show me the tools you’re using to get that result. Your result is amazing. Show me the code and the data.

But that never happens. The people I talk to say I don’t, I can’t, I’m not sure, but, if…

Because there’s all these strange incentives to hold that data and code hostage. You’re thinking, maybe I don’t need to hire you if you publish that code. If you don’t publish your code and data and I don’t have time to reverse-engineer six years of a smart kid’s life, I need to hire you for sure, right? And maybe you’re not proud of the code, maybe you know for sure that it’s ugly and awful and hacks piled up over hacks, maybe it’s just a big mess of shell scripts on your lab account. I get that, believe me; the day I write a piece of code I’m proud of before it ships will be a pretty good day.

But I have to see something. Because from our perspective, making a claim about software that doesn’t include the software you’re talking about is very close to worthless. You’re not reporting a scientific result at that point, however miraculous your result is; you’re making an unverifiable claim that your result exists.

And we look at that and say: what if you’ve got nothing? How can we know, without something we can audit and test? Of course, all the supporting research is paywalled PDFs with no concomitant code or data either, so by any metric that matters – and the only metric that matters here is “code I can run against data I can verify” – it doesn’t exist.

Those aren’t metrics that matter to you, though. What matters to you is either “getting a tenure-track position” or “getting hired to do work in your field”. And by and large the academic tenure track doesn’t care about open access, so you’re afraid that actually showing your work will actively hurt your likelihood of getting either of those jobs.

So here we are in this bizarro academic-research standoff, where I can’t work with you without your tipping your hand, and you can’t tip your hand for fear I won’t want to work with you. And so all of this work that could accomplish amazing things for a real company shipping real software that really matters to real people – five or six years of the best work you’ve ever done, probably – just sits on the shelf rotting away.

So I go to academic conferences and I beg people to publish their results and paper and data open access, because the world needs your work to matter. Because open access plus data/code as a minimum standard isn’t just important to the fundamental principles of repeatable experimental science, the integrity of your field, and your career. It’s important because if you want your work to matter to people, then you’d better put it somewhere that people can see it and use it and thank you for it and maybe even improve on it.

You did this as an undergrad. You insist on this from your undergrads, for exactly the same reasons I’m asking you to do the same: understanding, integrity and plain old better results. And it’s a few clicks and a GitHub account for you to do the same now. But I need you to do it one last time.

Full marks here isn’t “a job” or “tenure”. Your shot at those will be no worse, though I know you can’t see it from where you’re standing. But they’re still only a strong B. An A is doing something that matters, an accomplishment that changes the world for the better.

And if you want full marks, show your work.


  1. Tyga
    Posted July 24, 2015 at 11:57 pm | Permalink

    I’m a practical greybeard who returned to academia to do research on stuff that I felt needed to be done.

    I have been privy to some pretty amazing research claims. Whenever I dig deeper, it turns out that the claims (I have yet to meet an exception to my claims – but I’m sure there must be some out there) are based on very carefully controlled environments and using code that is not only ugly, but riddled with flaws. It’s like doing cellular biology with microscopes with distorting optics.

    Even within the research community there are calls for “reproducible research” – but for the above reasons (I suspect) very little has come to pass so far.

    Lest I sound too depressing, I offer a solution …

    Virtually all public funded universities are expected to enter into industry collaborated research. In many countries (for example Australia) the government provides tax incentives for such collaborative research. So identify work at universities where incentives exist for collaborative research, engage on specific research projects – typically continuation of already completed promising work. That way Mozilla will become privy to the information, techniques and knowledge you desire at relatively modest costs.

    BTW in Germany, Sweden, France, Norway, a great deal of software research happens in such an environment of collaboration between industry and academia.

    Academics who engage in such projects actually end up enhancing their reputations. Partly because such work brings in more research funding / grants. Hate to put like this, but from what I’ve seen, the amount of money brought in by a researcher counts for even more than their publications output.

  2. Mike
    Posted July 25, 2015 at 3:00 am | Permalink

    So you sign an NDA and send some guys to inspect the work in a developer-controlled cleanroom. They can take datasets in and report on the results, but nothing else.

  3. Posted July 25, 2015 at 6:09 am | Permalink

    “Even if you’re overoptimistic by a factor of five and it’s only a 20% savings we’d hire you tomorrow to build that for us.”

    Really? Maybe Mozilla will. Most companies won’t. I’ve done some pretty serious open source work in multiple projects over the years, used by at least dozens of for-profit enterprises. (Some of the stuff I’m a major contributor but not originator of easily reaches into the thousands of customers.)

    I frequently receive emails from salaried full time employees of these companies begging for help. OTOH

    Number of job offers: 0
    Number of paid consulting contracts: 0

    Someone with even more successful projects, and perhaps a more entrepreneurial mindset, can indeed get occasional gigs from their work. However the default approach (> 99% of the time) is that the developers who are already employees try and learn the things on their own, even if it’s going to take them weeks to do something the original developer could knock out in a couple of hours. (Insert standard anecdote about knowing which knob to turn.)

    Developer time is not fungible. Companies are not set up to enable hiring, or even contracting, to move quickly and efficiently. In fact, procedures are in place to prevent this.

    Suppose I came to you with a product that delivered 80% reduction in testing overhead at Mozilla. Furthermore, imagine that I could show you the code and it was good. I could fully satisfy your reasonable questions about the product. Now imagine I have an exploding offer to start somewhere else at standard Silicon Valley salaries, and it starts tomorrow. I’d rather work at Mozilla but I have to start tomorrow or it doesn’t happen.

    Can you, personally, make that happen? Can anybody at Mozilla?

    More realistically, suppose I come to you with a fix that will save Mozilla one developer-month. Can you or any other engineer authorize a payment of one developer’s salary for a week to purchase it? Or would the overhead of getting the payment approved be so much hassle that the actual engineers, the ones who know how much time and effort will be saved, would just go ahead and take four times as long to do the same thing?

  4. Robert Zeh
    Posted July 26, 2015 at 11:51 am | Permalink

    I work on trading systems, and I track research on verification, software development, and programming languages.

    I’ve implemented enough algorithms out of graduate level text books and papers that I no longer trust papers without downloadable code and tests. I’ve been burned too many times by results I can’t duplicate.

    Authors who don’t respond — in any way at all — to bug reports are another annoyance.

    It makes me suspect that Tyag is right, and that much of the code that CS papers are based on only works under the proper planetary alignment.

  5. Posted July 27, 2015 at 4:21 pm | Permalink

    As part of a book I am writing a book on empirical software engineering I regularly email researchers asking for a copy of their data. I get varying responses, but most of the time if the author still has the data they are happy to send me a copy.

    In general most research claims are intellectually dishonest in that they are making improvements in areas that nobody is interested in or suggesting approaches that won’t have the desired effect, e.g., predicting which files are most likely to contain faults and propose that whatever the code in these files does be completely reimplemented.

    There are only two papers based on Firefox data that I think are worthwhile, one on faults and the other on benchmarking (no link to hand, but I remember it was an Astralian MSc that Mozilla was in contact with).

    In my experience Elliotte Rusty Harold is correct, developers would always prefer to do it themselves than pay somebody else to do it. The secret is to target managers not developers.

  6. Posted July 30, 2015 at 6:37 pm | Permalink

    You’re right — the majority of the current system is setup up to at best not care about releasing code/data along with research and at worst actively discourages it. At least on the surface.

    The growing swell (tiny bubble?) from within the community for “reproducible research” has a polarizing effect too. On one hand, many of us think it’s long since overdue, and on the other hand there are some pretty spectacular cases of bad code or data analysis being put into the spotlight for the early adopters that are retroactively releasing their stuff. Even if a researcher feels their claims are solid, the latter point can cause a fear.

    As the first commenter pointed out, incentivizing the practice is likely the best way from here to there. Bringing in grant money is /definitely/ one very shiny looking bean on an academic’s CV — far shinier than opening your code/data — for most of the bean counters out there (er…hiring committees). Even non-monetary industry collaborations would look far better than research openly released. Engineering the open release of code/data into the existing framework is likely to be far more successful than fundamentally shifting how things are done cold turkey.

    Disclaimer: I fully agree that a work loses much of its integrity if it can’t be reproduced, and so my final slide in every academic talk includes a URL to where they can get access to it all. But then again I’m not doing anything that Mozilla (or most companies) would care about ;).