August 31, 2020

Consequences Of Code

Filed under: a/b,digital,documentation,doom,fail,hate,interfaces,losers,vendetta — mhoye @ 9:24 pm

[Content warning: There’s descriptions of psychological and animal abuse in here, because I’m talking about Facebook. This gets ugly fast.]

The idea behind DevOps – a consolidation of the developers who created software and the operational teams who deployed and maintained it, a meaningful distinction once upon a time – was simple. You, developer, will spend couple of days a month carrying the pager that wakes somebody up when your software fails. It’s telling that “you should own the consequences of your professional actions” was considered some sort of revolutionary insight in this industry, but let’s put that aside for the moment; that model of service development and deployment where engineers own and experience the failure modes of code they wrote, this sudden, shocking alignment of incentives – what do you mean, I’m going to be sitting in the economy seats during the maiden flight of this airplane I designed? – has driven dramatic improvements in too many organizations to count.

There are some exceptions, though.

Many Facebook employees reportedly weren’t satisfied with Zuckerberg’s explanation for the lack of action on the Kenosha Guard page, BuzzFeed reported. “We need to get better at avoiding mistakes and being more proactive,” one employee commented on the livestream of the Facebook meeting. ”Feels like we’re caught in a cycle of responding to damage after it’s already been done rather than constructing mechanisms to nip these issues before they result in real harm.” Employees have also blamed Zuckerberg personally for the company’s repeated failure to adequately address hate on its platform, with one telling BuzzFeed that Zuckerberg “seems truly incapable of taking personal responsibility for decisions and actions at Facebook.””

Many Facebook employees “weren’t satisfied”, golly. Sure, they all showed up for work the next day but think of all that dissatisfaction. Then compare it to what their content moderators – the people they pay to deal with the consequences of the software they’ve written – go through for fifteen bucks an hour every fucking day.

For the six months after he was hired, Speagle would moderate 100 to 200 posts a day. He watched people throw puppies into a raging river, and put lit fireworks in dogs’ mouths. He watched people mutilate the genitals of a live mouse, and chop off a cat’s face with a hatchet. He watched videos of people playing with human fetuses, and says he learned that they are allowed on Facebook “as long as the skin is translucent.” He found that he could no longer sleep for more than two or three hours a night. He would frequently wake up in a cold sweat, crying.

Who could have predicted that “operational mistakes” might happen in an environment like that?

In a companywide meeting on Thursday, Facebook CEO Mark Zuckerberg said that a militia page advocating for followers to bring weapons to an upcoming protest in Kenosha, Wisconsin, remained on the platform because of “an operational mistake.”

… which is just the most obvious and egregious lie. 3 overlooked reports might, conceivably, be an operational mistake. Four hundred is a policy decision.

The reason Facebook’s engineers, managers and leadership don’t and will never take operational responsibility for their code – the reason they won’t ever put the people who write their software and the people subject to the worst consequences of it in the same State, much less the same building – is simple: if Facebook’s engineers and managers had to spend one week every quarter doing the moderation work they fob off on underpaid contractors, Facebook wouldn’t exist in a year. And everyone working there knows that.

If you work at Facebook, quit. You might have good intentions – the best intentions, just really great intentions, fantastic intentions – but you know who you are and what you’re complicit in. Your intentions are just the bedtime stories you’re telling your conscience so you can sleep at night. You have a choice. Do better.

July 30, 2020



“A lunatic is easily recognized. He is a moron who doesn’t know the ropes. The moron proves his thesis; he has logic, however twisted it may be. The lunatic on the other hand, doesn’t concern himself at all with logic; he works by short circuits. For him, everything proves everything else. The lunatic is all idée fixe, and whatever he comes across confirms his lunacy. You can tell him by the liberties he takes with common sense, by his flashes of inspiration, and by the fact that sooner or later he brings up the Templars.”
– Umberto Eco, Foucault’s Pendulum.

… but it’s basically impossible to talk about this stuff without sounding like a lunatic, so let’s press on. I suppose that’s never stopped me before.

Hey, does anyone remember the tagline from the Majestic conspiracy game back in the day: “The Game Plays You”? Hold that thought.

You might have seen the argument from Adrian Hon recently, that the QAnon conspiracy theory is actually an ARG:

… and I knew I’d seen an argument that general shape before, but I couldn’t remember where; the “bottomless ARG” idea, I mean. It hit me earlier this week, shortly before the phrase “Alien DNA and Demon Sperm” became a part of this year’s pantheon of nonsensical headline nightmares: that was C.S. Lewis’ description of occultism, and the occult in general.

Lewis saw occultism as a sort of psychological snare, a set of endlessly self-referential symbols of symbols of symbols with no ultimate referent, a bottomless semiotic rathole for the overcurious inquirer designed to perpetually confuse and distract the mind. Beaudrillard, incidentally – creator of the term “hyperreal” – saw modern finance, and particularly advertising, in the same light – a set of self-referential symbols ultimately disconnected from reality, meaningful only in their own context, self-sustaining only to people trapped in that interlocking mesh.

Seeing through this lens makes the underpinnings of Facebook’s deep-seated resistance to admit the existence of, much less take responsibility for, much less do anything about, the running river of fake news, conspiracy theories and racist agitprop on that platform understandable: Facebook isn’t a social network: Facebook is an ARG Platform. It’s indiscriminate, unpoliced Alternate-Reality-As-A-Service.

“Whatever the rhythm was, luck rewarded us, because, wanting connections, we found connections — always, everywhere, and between everything. The world exploded in a whirling network of kinships, where everything pointed to everything else, everything explained everything else… “
– Umberto Eco, Foucault’s Pendulum

And with an audience already wound up in this unfiltered, overpopulated hyperreality-as-service, you barely need to do any work at all to kickstart the sort of amazing, self-sustaining paranoia-fulfillment engine that would have put the last few centuries’ foil-hatted quasi-mystic conspiriographists’ jaws right on the floor. All you need is enough people in rough proximity who feel frightened and powerless, a compelling seed crystal – the antivax fraud, the QAnon clownshow, a thousand others, it barely matters as long as it’s got a sharp hook – and this cancerous hyperculture machine pretty much bootstraps itself, making in-group celebrities out wannabe James Burkes pulling obscure facts together and drip-feeding the occasional five-like dopamine hit to the noobs explaining that you can’t spell “Rosicrucianist Aliens” without “Clintons”.

(For a while I was using that as a first-pass test for newsfeeds: if I replace “the Clintons” with “the Templars” and say this out loud… do I sound like a crank? I’ve never mentioned it, because I don’t need people who already sound like cranks emailing me to say “of course, it was right in front of us the whole time”, but we’re playing way past that now. But if I’ve accidentally added something to the collective lunatic lexicon – the lexographia lunacii, as it was first described in fifteen-forty-never because I just made that up – then I will seriously owe Chelsea an apology.)

Facebook’s ongoing negligence aside, what makes the Q-loons fascinating is that despite all its modern trappings, once that meme set its hooks into a vulnerable population (and psychological vulnerability is the name of the game, out there in the fever swamps) this wide-open extremely-2020 conspiracy-ARG is structurally nothing more a massively-multiplayer version of every vintage occult ceremony in history. I mean, the baseline aesthetic is trash, but still; it’s just a bunch of lunatic imagery, strange incantations and oddball ceremonials whose only reason to exist is to justify the time people have spent bringing it into existence.

“I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth.”
– Umberto Eco, Foucault’s Pendulum

So now thousands and thousands of people are participating, without realizing it, in a massively-distributed, collaborative occult ceremony, tying every scrap of fact and coincidence of the world together into this giant fractal-sefirot red-yarn serial-killer wall, drawing lines of imaginary digital salt from symbol to symbol to meaningless symbol, each utterly disconnected from anything more real than their own paranoid helplessness and fear.

Another way to say that is: QAnon is an occult conspiracy whose nefarious secret purpose is convincing themselves that an occult conspiracy actually exists.

It’s strangely beautiful in a way, until you understand what you’re seeing; Foucault’s Pendulum rewritten as a cryptofascist fever-swamp MMORPG. I love that book, and seeing this is like being offered the Maltese Falcon and handed a seagull drowned in crude.

“You see, Casaubon, even the Pendulum is a false prophet. You look at it, you think it’s the only fixed point in the cosmos. but if you detach it from the ceiling of the Conservatoire and hang it in a brothel, it works just the same. And there are other pendulums: there’s one in New York, in the UN building, there’s one in the science museum in San Francisco, and God knows how many others. Wherever you put it, Foucault’s Pendulum swings from a motionless point while the earth rotates beneath it. Every point of the universe is a fixed point: all you have to do is hang the Pendulum from it.”
– Umberto Eco, Foucault’s Pendulum

I’m not sure what to do with that information, sadly. If we’ve learned anything from the ongoing death throes of the 20th century, it’s that in an information dense and hyperconnected society, that robust commitments to social infrastructure – public health, public education, public journalism, social justice, social welfare and full employment – aren’t some sort of lefty, feel-good hippie political niceties; these are national security issues, and their failures expose an attack surface on participatory representative democracy. And while I don’t think Facebook profits from disinformation, they’re definitely complicit and definitely profiting from ignorance, powerlessness and helplessness. The disinformation, the conspiracy theories and racist agitprop, the antivax gongshow, Q and a thousand others festering alternate realities are just byproducts of that fear and desperation, parasites that have latched on to a vulnerable population from an accommodating platform happy to look the other way, wash their hands of the whole thing and let the machine grind away.

“I believe that you can reach the point where there is no longer any difference between developing the habit of pretending to believe and developing the habit of believing.
– Umberto Eco, Foucault’s Pendulum

I sometimes wonder what makes Facebook’s staff think the algorithms they’ve built, to give their users whatever keeps them clicking away uncritically, aren’t pointed back at them? Do they not wonder, or just not ask?

May 5, 2020

The Shape Of The Machine

Filed under: digital,documentation,doom,interfaces,vendetta — mhoye @ 11:18 am


The AMP people are rebranding their “AMP Stories” as “web stories“, raising that bland flag to the strained smiles of their publisher client-states, so it’s about time for me to click “post” on this. I was ranting about AMP a few weeks ago, partly because it’s a minor fraud being carried out at the expense of open standards and the common good, but mostly because of what an incredible social engineering exercise it’s been inside the big G.

The most remarkable thing about this machine – specifically, this corner of the massive Google monitor-and-monetize engine – how well they’ve broken up every part of the process internally at Google. The team working on AMP is making web pages lighter-weight and faster to load, and that sure seems like a net good in the world. Elsewhere, the search team has built out a preference for surfacing faster-loading and lighter-weight web pages to mobile users, which I’m sure was a decision with some sort of empirical basis and on its face seems like a good idea too. That a site would show up better in search results because it’s snappier or more responsive seems reasonable. And finally Google sells ads and shares ad revenue and that keeps a lot of sites alive these days, and I guess that’s good as well.

But when you put all the parts of that machine together under one P&L-statement roof suddenly you’re looking at a thinly veiled extortion scheme; as a publisher you either use AMP or your ad revenue goes away. Google could have approached the “be better on mobile” problem, search optimization and revenue sharing any number of ways, obviously, but the one they’ve chosen and built out is the one that guarantees that either you let them middleman all of your traffic or they cut off your oxygen.

That’s a real nice revenue stream you’ve got there; it’d be a shame if anything happened to it.

It’s got a certain sinister elegance to it, that mercenary Vintage Microsoft flavor, but Vintage Microsoft never pretended they were anything other than mercenary and this is all about the pretending. It’s not on the same level as Facebook, obviously, where they’ve managed to compartment their org structure so carefully that complicity in war crimes and other atrocities leaves facebookers feeling like they’re the ones being portrayed unfairly, but it’s still an accomplishment. That’s the most fascinating part of this exercise, to my mind: that Google has managed to structure this surveillance-and-value-extraction machine entirely out of people who are convinced that they, personally, are doing good for the world. The stuff they’re working on isn’t that bad – we’ve got such beautiful intentions! – and it’s interesting work and the perks here are really good… You can kind of watch their eyes de-focus for a second when you bring it up; it takes them a minute to remember who pays for their laundry and snacks.

April 27, 2020

Side Scroller

Filed under: arcade,documentation,interfaces,linux,toys — mhoye @ 8:35 am

I’ve never met Ian Albert, but years ago he painstakingly scraped and pasted together a set of maps and backgrounds from a various oldschool games, an effort that’s helped me in a bunch of odd little ways over the years and for which I’m grateful. Of particular interest today are the original Super Mario Brothers maps; for the sake of this exercise, let’s start with world 1, level 1.

ImageMagick and FFMpeg are a pair of “classically-Linux” command-line tools, in terms of how insanely complex and opaque they appear until you’ve worked with them for a bit and can sort of see the logic of their approaches. Even then the documentation takes some getting used to – the man page should just say “don’t bother, go to the website” – and even then you’ve gotta kind of fumble your way towards competence if you want to use them day to day.

Well, maybe you don’t, but I sure do. In any case once you know they exist you muddle your way to doing a lot with them. In particular, “convert” from the ImageMagick tool suite lets you upscale some of those Mario-level gifs to PNGs, like so:

$> convert mario-1-1.gif -scale 300% mario-1-1.png

We’re doing this conversion because FFMpeg (apparently?) doesn’t like to pan over gifs as an input stream but is happy to do that with PNGs, and scaling it up gets you an image size better suited for modern screens. We’re admittedly scaling up and then compressing something that eventually gets upscaled again, which looks like it should bea waste of effort. I’ve tested it, though, and on this machine at least it looks like movie upscaling comes out a lot mushier than static image upscaling and this approach is quite a bit crisper.

In any case, then you run “file” on that resulting image to see how big it is:

$> file ./mario-1-1.png
./mario-1-1.png: PNG image data, 10152 x 672, 4-bit colormap, non-interlaced

Do a bit of loose math to figure out your frame width and subtract 16/9 * 672 – that is, the aspect ration of your monitor times the height of the image – from the length – to get the number you need to work with next – in my case rounding to 1200, it’s 8952.

That’s the number of frames you’re going to tell FFMpeg to pan across, like so:

$> ffmpeg -loop 1 -framerate 5 -i mario-1-1.png -vf crop=1200:672:n:0 -frames:v 8952 -pix_fmt yuv420p mario-1-1.mp4

Now, order of operations and operation context both matters in FFMpeg usage, which adds a degree of complexity to figuring out wtf you’re doing with it, but walking through that command:

the “-loop” option is specific to the image processing part of ffpmeg, and in turn specific to some image-processing formats, so “loop 1” might or might not error out saying “unrecognized option”, depending on where you put it in the command line and which image types you’re choosing to process, which is not super helpful. In this case, it works for .png input files, and it means “go through this set of input images once”. We’ll get back to “-framerate” in a moment.

“-i” is input the png of the mario level we made earlier. The rest of this command is where the proverbial action is.

“-vf” means “create a filtergraph”, which is FFMpeg-ese for “transform the set of input images you’ve decoded in the following way”. “The following” can get pretty crazy, as you might imagine, but fortunately for us this will be reasonably simple in intent, despite the somewhat daunting syntax.

In this case, it means “crop out a sub-image from the given input image, of with 1200 and height 672, starting at horizontal offset “n” and vertical offset 0. “n” in this case is implicitly provided by the “frames” part, as we iterate over the frames from zero to the value of “-frames:v”

The “-pix_fmt yuv420p” part – “pixel format”, is what that means – I don’t really understand, beyond the fact that FFMpeg can encode videos in way more formats than browsers can easily decode, and its’ default idea of “best” doesn’t work everywhere. This incantation seems to fix that, which isn’t particularly satisfying but is definitely part of the whole fumbling-towards-competence part I mentioned.

In any case, the “-framerate 5” part is the interesting bit. That’s there because about nine thousand frames – 8952 specifically – divided by the number of seconds in a 30 minute meeting is very close to five. Five frames per second is really slow, so the resulting output video is, as predicted by our basic arithmetic, a lazy 29 minutes and 50 seconds long:

… and that’s the story of how you make a videoconference background that scrolls slowly through a Mario level over the course of half an hour.

A few notes:

  • If you leave out the framerate option and just want to see it scroll by at a default 25 frames per second, the movie is five minutes and change, which is amusingly a few seconds longer than the best speedruns of the entire game.
  • That crop=1200:672:n:0 option elides a lot of possible complexity; there’s an entire mathematical-expression interpreter under the hood of of crop and all the other FFMPeg filters, so if you want a 1080p movie panning diagonally across some of the many classic and modern works of art that are available now from any number of places, you can roll your own with relative ease.
  • The temptation to edit these to say something like “Thank you, Mario! But Peach went to another meeting.” is strong; if I get around to that, the fonts are here or maybe here.
  • I really need to get out of the house more. I guess we all do?

Update: A friend points me at FFMprovisr:

“FFmpeg is a powerful tool for manipulating audiovisual files. Unfortunately, it also has a steep learning curve, especially for users unfamiliar with a command line interface. This app helps users through the command generation process so that more people can reap the benefits of FFmpeg.”

Thank you, Sumana!

April 13, 2020


Filed under: documentation,fail,life,losers,vendetta — mhoye @ 3:18 pm

March 28, 2020


Filed under: a/b,documentation,fail,future,losers,lunacy,vendetta — mhoye @ 2:04 pm

March 18, 2020


Filed under: analog,documentation,interfaces,life,mozilla,work — mhoye @ 9:56 pm


As far as I can tell, 100% of the google results for “burnout” or “recognizing burnout” boil down to victim-blaming; they’re all about you, and your symptoms, and how to recognize when you’re burning out. Are you frustrated, overwhelmed, irritable, tired? Don’t ask for help, here’s how to self-diagnose! And then presumably do something.

What follows is always the most uselessly vague advice, like “listen to yourself” or “build resiliency” or whatever, which all sounds great and reinforces that the burden of recovery is entirely on the person burning out. And if you ask about the empirical evidence supporting it, this advice is mostly on par with leaving your healing crystals in the sun, getting your chakras greased or having your horoscope fixed by changing your birthday.

Resiliency and self-awareness definitely sound nice enough, and if your crystals are getting enough sun good for them, but just about all of this avoiding-burnout advice amounts to lighting scented candles downwind of a tire fire. If this was advice about a broken leg or anaphylaxis we’d see it for the trash it is, but because it’s about mental health somehow we don’t call it out. Is that a shattered femur? Start by believing in yourself, and believing that change is possible. Bee stings are just part of life; maybe you should take the time to rethink your breathing strategy. This might be a sign that breathing just isn’t right for you.

Even setting that aside: if we could all reliably self-assess and act on the objective facts we discerned thereby, burnout (and any number of other personal miseries) wouldn’t exist. But somehow here we are in not-that-world-at all. And as far as I can tell approximately none percent of these articles are ever about, say, “how to foster an company culture that doesn’t burn people out”, or “managing people so they don’t burn out”, or “recognizing impending burnout in others, so you can intervene.”

I’ll leave why that might be as an exercise for the reader.

Fortunately, as in so many cases like this, evidence comes to the rescue; you just need to find it. And the best of the few evidence-based burnout-prevention guidelines I can find come from the field of medicine where there’s a very straight, very measurable line between physician burnout and patient care outcomes. Nothing there will surprise you, I suspect; “EHR stress” (Electronic Health Records) has a parallel in our lives with tooling support, and the rest of it – sane scheduling, wellness surveys, agency over meaningful work-life balance and so on – seems universal. And it’s very clear from the research that recognizing the problem in yourself and in your colleagues is only one, late step. Getting support to make changes to the culture and systems in which you find yourself embedded is, for the individual, the next part of the process.

The American Medical Association has a “Five steps to creating a wellness culture” document, likewise rooted in gathered evidence, and it’s worth noting that the key takeaways are that burnout is a structural problem and mitigating it requires structural solutions. “Assess and intervene” is the last part of the process, not the first. “Self-assess and then do whatever” is not on the list at all, because that advice is terrible and the default setting of people burning out is self-isolation and never, ever asking people for the help they need.

We get a lot of things right where I work, and we’re better at taking care of people now than just about any other org I’ve ever heard of, but we still need to foster an “if you see something, say something” approach to each others’ well being. I bet wherever you are, you do too. Particularly now that the whole world has hard-cutover to remote-only and we’re only seeing each other through screens.

Yesterday, I told some colleagues that “if you think somebody we work with is obviously failing at self-care, talk to them”, and I should have been a lot more specific. This isn’t a perfect list by any means, but if you ask someone how they’re doing and they can’t so much as look you in the eye when they answer, see that. If you’re talking about work and they start thumbing their palms or rubbing their wrists or some other reflexive self-soothing twitch, notice. If you ask them about what they’re working on and they take a long breath and longer choosing their words, pay attention. If somebody who isn’t normally irritable or prone to cynical or sardonic humor starts trending that way, if they’re hunched over in meetings looking bedraggled when they normally take care of posture and basic grooming, notice that and say so.

If “mental health” is just “health” – and I guarantee it is – then burnout is an avoidable workplace injury, and I don’t believe in unavoidable mental-health injuries any more than I believe in unavoidable forklift accidents. Keep an eye out for your colleagues. If you think somebody you work with is failing at self-care, talk to them. Maybe talk to a friend, maybe talk to their manager or yours.

But say something. Don’t let it slide.

March 6, 2020

Brace For Impact

I don’t spend a lot of time in here patting myself on the back, but today you can indulge me.

In the last few weeks it was a ghost town, and that felt like a victory. From a few days after we’d switched it on to Monday, I could count the number of human users on any of our major channels on one hand. By the end, apart from one last hurrah the hour before shutdown, there was nobody there but bots talking to other bots. Everyone – the company, the community, everyone – had already voted with their feet.

About three weeks ago, after spending most of a month shaking out some bugs and getting comfortable in our new space we turned on federation, connecting Mozilla to the rest of the Matrix ecosystem. Last Monday we decommissioned for good, closing the book on a 22-year-long chapter of Mozilla’s history as we started a new one in our new home on Matrix.

I was given this job early last year but the post that earned it, I’m guessing, was from late 2018:

I’ve mentioned before that I think it’s a mistake to think of federation as a feature of distributed systems, rather than as consequence of computational scarcity. But more importantly, I believe that federated infrastructure – that is, a focus on distributed and resilient services – is a poor substitute for an accountable infrastructure that prioritizes a distributed and healthy community. […] That’s the other part of federated systems we don’t talk about much – how much the burden of safety shifts to the individual.

Some inside baseball here, but if you’re wondering: that’s why I pushed back on the idea of federation from the beginning, for all invective that earned me. That’s why I refused to include it as a requirement and held the line on that for the entire process. The fact that on classically-federated systems distributed access and non-accountable administration means that the burden of personal safety falls entirely on the individual. That’s not a unique artifact of federated systems, of course – Slack doesn’t think you should be permitted to protect yourself either, and they’re happy to wave vaguely in the direction of some hypothetical HR department and pretend that keeps their hands clean, as just one example of many – but it’s structurally true of old-school federated systems of all stripes. And bluntly, I refuse to let us end up in a place where asking somebody to participate in the Mozilla project is no different from asking them to walk home at night alone.

And yet here we are, opting into the Fediverse. It’s not because I’ve changed my mind.

One of the strongest selling points of Matrix is the combination of powerful moderation and safety tooling that hosting organizations can operate with robust tools for personal self-defense available in parallel. Critically, these aren’t half-assed tools that have been grafted on as an afterthought; they’re first-class features, robust enough that we can not only deploy them with confidence, but can reasonably be held accountable by our colleagues and community for their use. In short, we can now have safe, accountable infrastructure that complements, rather than comes at the cost, of individual user agency.

That’s not the best thing, though, and I’m here to tell you about my favorite Matrix feature that nobody knows about: Federated auto-updating blocklist sharing.

If you decide you trust somebody else’s decisions, at some other organization – their judgment calls about who is and is not welcome there – those decisions can be immediately and automatically reflected in your own. When a site you trust drops the hammer on some bad actor that ban can be adopted almost immediately by your site and your community as well. You don’t have to have ever seen that person or have whatever got them banned hit you in the eyes. You don’t even need to know they exist. All you need to do is decide you trust that other site judgment and magically someone persona non grata on their site is precisely that grata on yours.

Another way to say that is: among people or communities who trust each other in these decisions, an act of self-defense becomes, seamlessly and invisibly, an act of collective defense. No more everyone needing to fight their own fights alone forever, no more getting isolated and picked off one at a time, weakest first; shields-up means shields-up for everyone. Effective, practical defensive solidarity; it’s the most important new idea I’ve seen in social software in years. Every federated system out should build out their own version, and it’s very clear to me, at least, that is going to be the table stakes of a federated future very soon.

So I feel pretty good about where we’ve ended up, and where we’re going.

In the long term, I see that as the future of Mozilla’s responsibility to the Web; not here merely to protect the Web, not merely to defend your freedom to participate in the Web, but to mount a positive defense of people’s opportunities to participate. And on the other side of that coin, to build accountable tools, systems and communities that promise not only freedom from arbitrary harassment, but even freedom from the possibility of that harassment.

I’ve got a graph here that’s pointing up and to the right, and it’s got nothing to do with scraping fractions of pennies out of rageclicks and misery; just people making a choice to go somewhere better, safer and happier. Maybe, just maybe, we can salvage this whole internet thing. Maybe all is not yet lost, and the future is not yet written.

February 20, 2020

Synchronous Messaging: We’re Live.

Filed under: digital,documentation,future,irc,mozilla,vendetta,work — mhoye @ 5:15 pm


After a nine month leadup,, our Matrix-based replacement for IRC, has been up running for about a month now.

While we’ve made a number of internal and community-facing announcements about progress, access and so forth, we’ve deliberately run this as a quiet, cautious, low-key rollout, letting our communities find their way to chat.m.o and Matrix organically while we sort out the bugs and rough edges of this new experience.

Last week we turned on federation, the last major step towards opening Mozilla to the wider Matrix ecosystem, and it’s gone really well. Which means that as of last week, Mozilla’s transition from IRC to Matrix is within arm’s reach of done.

The Matrix team have been fantastic partners throughout this process, open to feedback and responsive to concerns throughout. It’s been a great working relationship, and as investments of effort go one that’s already paying off exactly the way want our efforts to pay off, with functional, polish and accessibility improvements that benefit the entire Matrix ecosystem coming from the feedback from the Mozilla community.

We still have work to do, but this far into the transition it sure feels like winning. The number of participants in our primary development channels has already exceeded their counterparts on IRC at their most active, and there’s no sign that’s slowing down. Many of our engineering and ops teams are idling or archiving their Slack channels and have moved entirely to Matrix, and that trend isn’t slowing down either.

As previously announced, we’re on schedule to turn off IRC.m.o at the end of the month, and don’t see a reason to reconsider that decision. So far, it looks like we’re pretty happy on the new system. It’s working well for us.

So: Welcome. If you’re new to Mozilla or would like to get involved, come see us in the #Introduction channel on our shiny new Matrix system. I hope to see you there.

February 18, 2020

Dexterity In Depth

Filed under: a/b,academic,documentation,interfaces,mozilla,science,vendetta,work — mhoye @ 10:50 am


I’m exactly one microphone and one ridiculous haircut away from turning into Management Shingy when I get rolling on stuff like this, because it’s just so clear to me how much this stuff matters and how little sense I might be making at the same time. Is your issue tracker automatically flagging your structural blind spots? Do your QA and UX team run your next reorg? Why not?

This all started life as a rant on Mastodon, so bear with me here. There are two empirically-established facts that organizations making software need to internalize.

The first is that by wide margin the most significant predictive indicator that there will be a future bug in a piece of software is the relative orgchart distance of the people working on it. People who are working on a shared codebase in the same room but report to different VPs are wildly more likely to introduce errors into a codebase than two people who are on opposite sides of the planet and speak different first languages but report to the same manager.

The second is that the number one predictor that a bug will be resolved is if it is triaged correctly – filed in the right issue tracker, against the right component, assigned to the right people – on the first try.

It’s fascinating that neither of the strongest predictive indicators of the most important parts of a bug’s lifecycle – birth and death – actually take place on the developers’ desk, but it’s true. In terms of predictive power, nothing else in the software lifecycle comes close.

Taken together, these facts give you a tools to roughly predict the effectiveness of collaborating teams, and by analyzing trends among bugs that are frequently re-assigned or re-triaged, can give you a lot of foresight into how, where and why a company need to retrain or reorganize those teams. You might have read Agile As Trauma recently, in which Dorian Taylor describes agile development as an allergic reaction to previously bad management:

The Agile Manifesto is an immune response on the part of programmers to bad management. The document is an expression of trauma, and its intellectual descendants continue to carry this baggage. While the Agile era has brought about remarkable advancements in project management techniques and development tools, it remains a tactical, technical, and ultimately reactionary movement.

This description is strikingly similar to – and in obvious tension with – Clay Shirky’s description of bureaucracy as the extractive mechanism of complexity and an allergic reaction to previous institutional screwups.

Bureaucracies temporarily suspend the Second Law of Thermodynamics. In a bureaucracy, it’s easier to make a process more complex than to make it simpler, and easier to create a new burden than kill an old one.

… which sounds an awful lot like the orgchart version of “It’s harder to read code than to write it”, doesn’t it?

I believe both positions are correct. But that tension scribes the way forward, I think, for an institutional philosophy that is responsive, flexible and empirically grounded, in which being deliberate about the scale, time, and importance of different feedback cycles gives an organization the freedom to treat scaling like a tool, that the signals of different contexts can inform change as a continuum between the macro and micro levels of organizational structure and practice. Wow, that’s a lot of words in a strange order, but hear me out.

It’s not about agile, or even agility. Agility is just the innermost loops, the smallest manifestation of a wide possible set of tightly-coupled feedback mechanisms. And outside the agile team, adjacent to the team, those feedback loops may or may not exist however much they need to, up and down the orgchart (though there’s not often much “down” left in the orgchart, I’ve noticed, where most agile teams live…) but more importantly with the adjacent and complementary functions that agile teams rely on.

It is self-evident that how teams are managed profoundly affects how they deliver software. But agile development (and every other modern developer-cult I’m aware of) doesn’t close that loop, and in failing to do so agile teams are reduced to justifying their continued existence through work output rather than informing positive institutional change. And I don’t use “cult” lightly, there; the current state of empirical evaluation of agile as a practice amounts to “We agiled and it felt good and seemed to work!” And feeling good and kinda working is not nothing! But it’s a long way from being anything more than that.

If organizations make software, then starting from a holistic view of what “development” and “agility” means and could be, looking carefully at where feedback loops in an organization exist, where they don’t and what information they circulate, all that suggests that there are reliable set of empirical, analytic tools for looking at not just developer practice, but the organizational processes around them. And assessing, in some measurable, empirical way, the real and sustainable value of different software development schools and methodologies.

But honestly, if your UX and QA teams aren’t informing your next reorg, why not?

Older Posts »

Powered by WordPress