I don’t spend a lot of time in here patting myself on the back, but today you can indulge me.
In the last few weeks it was a ghost town, and that felt like a victory. From a few days after we’d switched it on to Monday, I could count the number of human users on any of our major channels on one hand. By the end, apart from one last hurrah the hour before shutdown, there was nobody there but bots talking to other bots. Everyone – the company, the community, everyone – had already voted with their feet.
About three weeks ago, after spending most of a month shaking out some bugs and getting comfortable in our new space we turned on federation, connecting Mozilla to the rest of the Matrix ecosystem. Last Monday we decommissioned IRC.Mozilla.org for good, closing the book on a 22-year-long chapter of Mozilla’s history as we started a new one in our new home on Matrix.
I was given this job early last year but the post that earned it, I’m guessing, was from late 2018:
I’ve mentioned before that I think it’s a mistake to think of federation as a feature of distributed systems, rather than as consequence of computational scarcity. But more importantly, I believe that federated infrastructure – that is, a focus on distributed and resilient services – is a poor substitute for an accountable infrastructure that prioritizes a distributed and healthy community. […] That’s the other part of federated systems we don’t talk about much – how much the burden of safety shifts to the individual.
Some inside baseball here, but if you’re wondering: that’s why I pushed back on the idea of federation from the beginning, for all invective that earned me. That’s why I refused to include it as a requirement and held the line on that for the entire process. The fact that on classically-federated systems distributed access and non-accountable administration means that the burden of personal safety falls entirely on the individual. That’s not a unique artifact of federated systems, of course – Slack doesn’t think you should be permitted to protect yourself either, and they’re happy to wave vaguely in the direction of some hypothetical HR department and pretend that keeps their hands clean, as just one example of many – but it’s structurally true of old-school federated systems of all stripes. And bluntly, I refuse to let us end up in a place where asking somebody to participate in the Mozilla project is no different from asking them to walk home at night alone.
And yet here we are, opting into the Fediverse. It’s not because I’ve changed my mind.
One of the strongest selling points of Matrix is the combination of powerful moderation and safety tooling that hosting organizations can operate with robust tools for personal self-defense available in parallel. Critically, these aren’t half-assed tools that have been grafted on as an afterthought; they’re first-class features, robust enough that we can not only deploy them with confidence, but can reasonably be held accountable by our colleagues and community for their use. In short, we can now have safe, accountable infrastructure that complements, rather than comes at the cost, of individual user agency.
That’s not the best thing, though, and I’m here to tell you about my favorite Matrix feature that nobody knows about: Federated auto-updating blocklist sharing.
If you decide you trust somebody else’s decisions, at some other organization – their judgment calls about who is and is not welcome there – those decisions can be immediately and automatically reflected in your own. When a site you trust drops the hammer on some bad actor that ban can be adopted almost immediately by your site and your community as well. You don’t have to have ever seen that person or have whatever got them banned hit you in the eyes. You don’t even need to know they exist. All you need to do is decide you trust that other site judgment and magically someone persona non grata on their site is precisely that grata on yours.
Another way to say that is: among people or communities who trust each other in these decisions, an act of self-defense becomes, seamlessly and invisibly, an act of collective defense. No more everyone needing to fight their own fights alone forever, no more getting isolated and picked off one at a time, weakest first; shields-up means shields-up for everyone. Effective, practical defensive solidarity; it’s the most important new idea I’ve seen in social software in years. Every federated system out should build out their own version, and it’s very clear to me, at least, that is going to be the table stakes of a federated future very soon.
So I feel pretty good about where we’ve ended up, and where we’re going.
In the long term, I see that as the future of Mozilla’s responsibility to the Web; not here merely to protect the Web, not merely to defend your freedom to participate in the Web, but to mount a positive defense of people’s opportunities to participate. And on the other side of that coin, to build accountable tools, systems and communities that promise not only freedom from arbitrary harassment, but even freedom from the possibility of that harassment.
I’ve got a graph here that’s pointing up and to the right, and it’s got nothing to do with scraping fractions of pennies out of rageclicks and misery; just people making a choice to go somewhere better, safer and happier. Maybe, just maybe, we can salvage this whole internet thing. Maybe all is not yet lost, and the future is not yet written.
While we’ve made a number of internal and community-facing announcements about progress, access and so forth, we’ve deliberately run this as a quiet, cautious, low-key rollout, letting our communities find their way to chat.m.o and Matrix organically while we sort out the bugs and rough edges of this new experience.
Last week we turned on federation, the last major step towards opening Mozilla to the wider Matrix ecosystem, and it’s gone really well. Which means that as of last week, Mozilla’s transition from IRC to Matrix is within arm’s reach of done.
The Matrix team have been fantastic partners throughout this process, open to feedback and responsive to concerns throughout. It’s been a great working relationship, and as investments of effort go one that’s already paying off exactly the way want our efforts to pay off, with functional, polish and accessibility improvements that benefit the entire Matrix ecosystem coming from the feedback from the Mozilla community.
We still have work to do, but this far into the transition it sure feels like winning. The number of participants in our primary development channels has already exceeded their counterparts on IRC at their most active, and there’s no sign that’s slowing down. Many of our engineering and ops teams are idling or archiving their Slack channels and have moved entirely to Matrix, and that trend isn’t slowing down either.
As previously announced, we’re on schedule to turn off IRC.m.o at the end of the month, and don’t see a reason to reconsider that decision. So far, it looks like we’re pretty happy on the new system. It’s working well for us.
So: Welcome. If you’re new to Mozilla or would like to get involved, come see us in the #Introduction channel on our shiny new Matrix system. I hope to see you there.
I’m exactly one microphone and one ridiculous haircut away from turning into Management Shingy when I get rolling on stuff like this, because it’s just so clear to me how much this stuff matters and how little sense I might be making at the same time. Is your issue tracker automatically flagging your structural blind spots? Do your QA and UX team run your next reorg? Why not?
This all started life as a rant on Mastodon, so bear with me here. There are two empirically-established facts that organizations making software need to internalize.
The first is that by wide margin the most significant predictive indicator that there will be a future bug in a piece of software is the relative orgchart distance of the people working on it. People who are working on a shared codebase in the same room but report to different VPs are wildly more likely to introduce errors into a codebase than two people who are on opposite sides of the planet and speak different first languages but report to the same manager.
The second is that the number one predictor that a bug will be resolved is if it is triaged correctly – filed in the right issue tracker, against the right component, assigned to the right people – on the first try.
It’s fascinating that neither of the strongest predictive indicators of the most important parts of a bug’s lifecycle – birth and death – actually take place on the developers’ desk, but it’s true. In terms of predictive power, nothing else in the software lifecycle comes close.
Taken together, these facts give you a tools to roughly predict the effectiveness of collaborating teams, and by analyzing trends among bugs that are frequently re-assigned or re-triaged, can give you a lot of foresight into how, where and why a company need to retrain or reorganize those teams. You might have read Agile As Trauma recently, in which Dorian Taylor describes agile development as an allergic reaction to previously bad management:
The Agile Manifesto is an immune response on the part of programmers to bad management. The document is an expression of trauma, and its intellectual descendants continue to carry this baggage. While the Agile era has brought about remarkable advancements in project management techniques and development tools, it remains a tactical, technical, and ultimately reactionary movement.
This description is strikingly similar to – and in obvious tension with – Clay Shirky’s description of bureaucracy as the extractive mechanism of complexity and an allergic reaction to previous institutional screwups.
Bureaucracies temporarily suspend the Second Law of Thermodynamics. In a bureaucracy, it’s easier to make a process more complex than to make it simpler, and easier to create a new burden than kill an old one.
… which sounds an awful lot like the orgchart version of “It’s harder to read code than to write it”, doesn’t it?
I believe both positions are correct. But that tension scribes the way forward, I think, for an institutional philosophy that is responsive, flexible and empirically grounded, in which being deliberate about the scale, time, and importance of different feedback cycles gives an organization the freedom to treat scaling like a tool, that the signals of different contexts can inform change as a continuum between the macro and micro levels of organizational structure and practice. Wow, that’s a lot of words in a strange order, but hear me out.
It’s not about agile, or even agility. Agility is just the innermost loops, the smallest manifestation of a wide possible set of tightly-coupled feedback mechanisms. And outside the agile team, adjacent to the team, those feedback loops may or may not exist however much they need to, up and down the orgchart (though there’s not often much “down” left in the orgchart, I’ve noticed, where most agile teams live…) but more importantly with the adjacent and complementary functions that agile teams rely on.
It is self-evident that how teams are managed profoundly affects how they deliver software. But agile development (and every other modern developer-cult I’m aware of) doesn’t close that loop, and in failing to do so agile teams are reduced to justifying their continued existence through work output rather than informing positive institutional change. And I don’t use “cult” lightly, there; the current state of empirical evaluation of agile as a practice amounts to “We agiled and it felt good and seemed to work!” And feeling good and kinda working is not nothing! But it’s a long way from being anything more than that.
If organizations make software, then starting from a holistic view of what “development” and “agility” means and could be, looking carefully at where feedback loops in an organization exist, where they don’t and what information they circulate, all that suggests that there are reliable set of empirical, analytic tools for looking at not just developer practice, but the organizational processes around them. And assessing, in some measurable, empirical way, the real and sustainable value of different software development schools and methodologies.
But honestly, if your UX and QA teams aren’t informing your next reorg, why not?
Karl Germain once said that “Magic is the only honest profession. A magician promises to deceive you and he does.”
This is sort of a review of a game, I guess. It’s called Superliminal.
“Every magic trick consists of three parts, or acts. The first part is called the pledge. The magician shows you something ordinary: a deck of cards, a bird, or a man. He shows you this object, and pledges to you its utter normality. Perhaps he asks you to inspect it to see that it is indeed real, unaltered, normal. But, of course, it probably isn’t. The second act is called the turn. The magician takes the ordinary something and makes it do something extraordinary. Now you’re looking for the secret. But you won’t find it, because of course you’re not really looking. You don’t really want to know. You want to be fooled. But you wouldn’t clap yet. Because making something disappear isn’t enough. You have to bring it back. That’s why every magic trick has a third act. The hardest part. The part we call the Prestige.”
— Cutter (Michael Caine), “The Prestige”
You’ve probably heard David Foster Wallace’s speech to the graduates of Kenyon College in 2005, the “This Is Water” speech.
It is about the real value of a real education, which has almost nothing to do with knowledge, and everything to do with simple awareness; awareness of what is so real and essential, so hidden in plain sight all around us, all the time, that we have to keep reminding ourselves over and over: “This is water, this is water.”
It’s possibly the greatest College Graduation Speech of all time, both in its mastery of the form and the surgical precision of its own self-serving subversion of that same form. “This is a standard requirement of US commencement speeches, the deployment of didactic little parable-ish stories”, right after the parable-ish story. “I am not the wise old fish”, followed by an explanation of the whole point of the fish story. Over and over again, throughout, we’re shown the same sleight of hand:.
Tell your audiences that they’re too smart to want a certain thing and give it to them anyway. Remind everyone that they’re too hip for corny dad sermonizing and then double down on the corny dad sermonizing. This is a great way to write a commencement speech—not by avoiding platitudes, but by drawing an enchanted circle around yourself where the things we thought were platitudes can be revealed as dazzling truths. Where all of us can be consoled, if only for an instant, by the notion that the insight we lack has been here all along! Just hiding inside of our clichés.
I don’t think Harnette’s cynicism in that LitHub article, pointed at the pernicious consequences of Wallace’s “cult of sincerity” is the whole story. She’s not wrong, but there’s more: if you’ve got the right eyes to see it the outline of the Prestige is there, the empty space where the third act didn’t happen. The part where this long, drawn out paean, begging for sincerity and authenticity and “simple awareness” reveals itself for what it is, a cry for help from somebody whose inner monologue does not shut up or take its foot off the gas so much as a millimeter for anyone or any reason ever. A plea for a simplicity from somebody whose mind simply won’t, that nobody saw.
Because of course you’re not really looking. You don’t really want to know; you want to be fooled. I know that this stuff probably doesn’t sound fun and breezy or grandly inspirational the way a video game review is supposed to sound. But it’s just about capital-T Time to stop using this gag.
@pamela :
I will never, ever tire of seamlessly transitioning from the end of Mean Girls to the beginning of Hackers with the same song, this was a damn *gift* given to us by the movie industry
@mhoye :
@pamela Has somebody actually crossfaded the video for this?
@pamela :
@mhoye not that I’ve found, but I live in hope…
@kiethzg :
@pamela @mhoye Sounds like a fun little project to start off my weekend with!
@pamela @mhoye I actually got distracted with even sillier things, but! Finally did this. Then watched it on a loop for a bit. Then remembered I should actually upload it somewhere! So here it is:
I really love the idea of jumping from movie to completely unrelated movie through a common song and a smooth soundtrack crossfade. The only rule, really, is that the song you jump into a movie with has to be earlier in the movie than the one you jump out with. Anyone out there got a dataset of movie soundtracks I could use to cobble together an Oracle Of Bacon-like tool for figuring out the forward soundtrack distance between movies?
I often wonder where we’d be if Google had spent their don’t-be-evil honeymoon actually interviewing people for some sort moral or ethical framework instead of teaching a generation of new hires that the important questions are all about how many piano tuners play ping pong on the moon.
You might have seen the NYTimes article on hypertargeted product placement, one of those new magical ideas that look totally reasonable in an industry where CPU cycles are cheap and principles are expensive.
I just wanted to make sure we all understood that one extremely intentional byproduct of that will breathe new life into the old documnent-canary trick of tailoring sensitive text with unique punctuation or phrasing in particularly quotable passages to identify leakers, and has been purpose-built as a way to precision-target torrent seeders or anyone else who shares media. “We only showed this combination of in-product signal to this specific person, therefore they’re the guilty party” is where this is going, and that’s not an accident.
The remedy, of course, is going to be cooperation. Robust visual diffs, scene hashes and smart muting (be sure to refer to They Live for placeholder inspiration) will be more than enough to fuzz out discoverability for even a moderately-sized community. As it frequently is, the secret ingredient is smart people working together.
In any case, I’m sure that all right thinking people can agree that ads are the right place to put graffiti. So I’m looking forward to all the shows that are turned into hijacked art-project torrents the moment they’re released, and seeing