Quality Speakings

Unfortunately my suite of annoying verbal tics – um right um right um, which I continue to treat like Victor Borge’s phonetic punctuation – are on full display here, but I guess we’ll have to live with that. Here’s a talk I gave at the GTA Linux User Group on “The State Of Mozilla”, split into the main talk and the Q&A sections. I could probably have cut a quarter of that talk out by just managing those twitches better, but I guess that’s a project for 2019. In the meantime:


The talk:


The Q&A afterwards:

The preview on that second one is certainly unflattering. It ends on a note I’m pretty proud of, though, around the 35 minute mark.

I should go back make a note of all the “ums” and “rights” in this video and graph them out. I bet it’s some sort of morse-coded left-brain cry for help.

Time Dilation


[ https://www.youtube.com/embed/JEpsKnWZrJ8 ]

I riffed on this a bit over at twitter some time ago; this has been sitting in the drafts folder for too long, and it’s incomplete, but I might as well get it out the door. Feel free to suggest additions or corrections if you’re so inclined.

You may have seen this list of latency numbers every programmer should know, and I trust we’ve all seen Grace Hopper’s classic description of a nanosecond at the top of this page, but I thought it might be a bit more accessible to talk about CPU-scale events in human-scale transactional terms. So: if a single CPU cycle on a modern computer was stretched out as long as one of our absurdly tedious human seconds, how long do other computing transactions take?

If a CPU cycle is 1 second long, then:

  • Getting data out of L1 cache is about the same as getting your data out of your wallet; about 3 seconds.
  • At 9 to 10 seconds, getting data from L2 cache is roughly like asking your friend across the table for it.
  • Fetching data from the L3 cache takes a bit longer – it’s roughly as fast as having an Olympic sprinter bring you your data from 400 meters away.
  • If your data is in RAM you can get it in about the time it takes to brew a pot of coffee; this is how long it would take a world-class athlete to run a mile to bring you your data, if they were running backwards.
  • If your data is on an SSD, though, you can have it six to eight days, equivalent to having it delivered from the far side of the continental U.S. by bicycle, about as fast as that has ever been done.
  • In comparison, platter disks are delivering your data by horse-drawn wagon, over the full length of the Oregon Trail. Something like six to twelve months, give or take.
  • Network transactions are interesting – platter disk performance is so poor that fetching data from your ISP’s local cache is often faster than getting it from your platter disks; at two to three months, your data is being delivered to New York from Beijing, via container ship and then truck.
  • In contrast, a packet requested from a server on the far side of an ocean might as well have been requested from the surface of the moon, at the dawn of the space program – about eight years, from the beginning of the Apollo program to Armstrong, Aldrin and Collin’s successful return to earth.
  • If your data is in a VM, things start to get difficult – a virtualized OS reboot takes about the same amount of time as has passed between the Renaissance and now, so you would need to ask Leonardo Da Vinci to secretly encode your information in one of his notebooks, and have Dan Brown somehow decode it for you in the present? I don’t know how reliable that guy is, so I hope you’re using ECC.
  • That’s all if things go well, of course: a network timeout is roughly comparable to the elapsed time between the dawn of the Sumerian Empire and the present day.
  • In the worst case, if a CPU cycle is 1 second, cold booting a racked server takes approximately all of recorded human history, from the earliest Indonesian cave paintings to now.

Licensing Edgecases

While I’m not a lawyer – and I’m definitely not your lawyer – licensing questions are on my plate these days. As I’ve been digging into one, I’ve come across what looks like a strange edge case in GPL licensing compliance that I’ve been trying to understand. Unfortunately it looks like it’s one of those Affero-style, unforeseen edge cases that (as far as I can find…) nobody’s tested legally yet.

I spent some time trying to understand how the definition of “linking” applies in projects where, say, different parts of the codebase use disparate, potentially conflicting open source licenses, but all the code is interpreted. I’m relatively new to this area, but generally speaking outside of copying and pasting, “linking” appears to be the critical threshold for whether or not the obligations imposed by the GPL kick in and I don’t understand what that means for, say, Javascript or Python.

I suppose I shouldn’t be surprised by this, but it’s strange to me how completely the GPL seems to be anchored in early Unix architectural conventions. Per the GPL FAQ, unless we’re talking about libraries “designed for the interpreter”, interpreted code is basically data. Using libraries counts as linking, but in the eyes of the GPL any amount of interpreted code is just a big, complicated config file that tells the interpreter how to run.

At a glance this seems reasonable but it seems like a pretty strange position for the FSF to take, particularly given how much code in the world is interpreted, at some level, by something. And honestly: what’s an interpreter?

The text of the license and the interpretation proposed in the FAQ both suggest that as long as all the information that a program relies on to run is contained in the input stream of an interpreter, the GPL – and if their argument sticks, other open source licenses – simply… doesn’t apply. And I can’t find any other major free or open-source licenses that address this question at all.

It just seems like such a weird place for an oversight. And given the often-adversarial nature of these discussions, given the stakes, there’s no way I’m the only person who’s ever noticed this. You have to suspect that somewhere in the world some jackass with a very expensive briefcase has an untested legal brief warmed up and ready to go arguing that a CPU’s microcode is an “interpreter” and therefore the GPL is functionally meaningless.

Whatever your preferred license of choice, that really doesn’t seem like a place we want to end up; while this interpretation may be technically correct it’s also very-obviously a bad-faith interpretation of both the intent of the GPL and that of the authors in choosing it.

The position I’ve taken at work is that “are we technically allowed to do this” is a much, much less important question than “are we acting, and seen to be acting, as good citizens of the larger Open Source community”. So while the strict legalities might be blurry, seeing the right thing to do is simple: we treat the integration of interpreted code and codebases the same way we’d treat C/C++ linking, respecting the author’s intent and the spirit of the license.

Still, it seems like something the next generation of free and open-source software licenses should explicitly address.

The Last Days Of 20A0


Science International – What Will They Think Of Next

At first blush this is a statement on the crude reproductive character of mass culture.

But it also serves as a warning about the psychohistorical destruction to come, the stagnation after revolution, the failure to remix.

I need to write this down, because I forget things sometimes, and I think what I heard today was important. Not to me, the time for me or almost anyone else alive on Earth today to make a difference has passed, but someone, somewhere might be able to make something of this, or at least find it helpful, or something. Once I’m done, I’m going to seal it up in a pipe, coat it in wax, and chuck it into the ravine. Maybe someday someone will read this, and try to put things together. If they’re allowed to.

It’s happening again.

The Phantom Time Hypothesis, developed by Heribert Illig, proposes that error and falsification have radically distorted the historical record. In his analysis, we have dilated the course of true events, so that they appear to cover far greater lengths of time than in fact passed. The so-called dark ages, for example, only appear that way because those centuries were mere decades.

You can feel it, can’t you? The relentless immediacy of crisis over crisis, the yawning void the endless emergency is stretched taut to obscure. The soul-bending psychological trauma; even moments of optimism seem unfairly compressed, hyperdense self-referential memetic shards landing like cartoon anvils and sublimated into vapor by the meteoric heat of the Next Thing. The spiritual torniquet of the perpetually immediate present twisting tighter, fractions of degrees at a time.

The space: do we not all feel it? The space. It may be said that the consumer cultures of the 1980s and 1990s, successively exhorting us to embrace artifice and then soul-crushing blandness, were manufactured to “cure” the residual confusion and cultural inconsistency that resulted from the methods used to effect mankind’s collective psychic displacement. The hidden “space,” however, manifests itself in curious ways – the obsession with youth and physical condition in those born in the 1960s and 1970s; oddities in climate change data; the apparently freakish pace of economic change in what we believe now to be the 1980s; and so forth.

You can hear fragments of the past that remain, the warning signs engineered to survive their own absence singing the speed, the mass of this oncoming train to anyone foolish or optimistic enough (and is there a difference, at this remove?) to put an ear to the tracks. It’s happening again; here we are in the moments before the moment, and it can’t be an accident that those who seem most adept in this psychosocial twilight, deftly navigating unmoored in cold storms of this howling psychic gyre are people who’ve lost their anchors or thrown them overboard by choice in the name of some dark mirrored vision of liberty or mere expediency, in the long calm of the before. They’re just one more set of symptoms now, signs of symbols nested in symbols whose ultimate referents are burned to ash beneath them.

It is happening again.

But the problem is a real one, not a mere intellectual game. Because today we live in a society in which spurious realities are manufactured by the media, by governments, by big corporations, by religious groups, political groups — and the electronic hardware exists by which to deliver these pseudo-worlds right into the heads of the reader, the viewer, the listener. Sometimes when I watch my eleven-year-old daughter watch TV, I wonder what she is being taught. The problem of miscuing; consider that. A TV program produced for adults is viewed by a small child. Half of what is said and done in the TV drama is probably misunderstood by the child. Maybe it’s all misunderstood. And the thing is, Just how authentic is the information anyhow, even if the child correctly understood it? What is the relationship between the average TV situation comedy to reality?

What’s left but what’s next, the twisting, the tightening, the inevitable snap; the collective spasm, the absence that will pass for absolution. The last fracturing as the cabals of consensus and permitted history are ground into the microcults gnawing at the fraying edges of tomorrow’s interstitials, memetic remixes remixed as memetic merchandise and malformed memories. Veracity hitting the kalidoscopic crystal of the weaponized postmodern like a bird hitting a window.

It. Is. Happening. Again.

We can’t say we weren’t warned.

I don’t know if that man was crazy or not, but I think he was sane. As he was leaving, he said something about putting my house underwater. Please, don’t let them brush me away. Don’t let them hide us. Try and find more, I know there’s got to be more people who tried to leave something behind. Don’t let the world die in vain. Remember us.

We were here, and there was something here worth saving. There was such a thing as now, and we fought for it. We’ll leave the artifacts, hidden and codified as we have before, as best we’re able. Watch for them. Listen. You’ll be able to hear the Next Time, the shape and speed and mass of it approaching, and it may not be too late to throw it off the tracks. Reassemble this moment, rebuild who we were out of the hidden shards we’ve left. Hone yourselves to the gleaming edges you’ll need with the tools we’ve left you. Put your ear to the rails and listen.

No piece of information is superior to any other. Power lies in having them all on file and then finding the connections. There are always connections; you have only to want to find them.

We were here. This was real. Remember us.

Planetary Alignment

I’m going to be closing down a number of disused or underused sub-Planets off of Planet Mozilla later this week. If you have any objections to this, you should let me know directly.

I’m probably going to do it anyway, but I promise I’ll at least hear you out.

Thirty-Five Minutes Ago

Well, that’s done.

mhoye@ANGLACHEL:~/src/planet-content/branches/planet$ git diff | grep "^-name" | wc -l
401
mhoye@ANGLACHEL:~/src/planet-content/branches/planet$ git commit -am "The Great Purge of 2017"

Purging the Planet blogroll felt a lot like being sent to exorcise the ghosts of my own family estate. There were a lot of old names, old memories and more than a few recent ones on the business end of the delete key today.

I’ve pulled out all the feeds that errored out, everyone who isn’t currently involved in some reasonably direct capacity in the Mozilla project and a bunch of maybes that hadn’t put anything up since 2014, and I’d like the record to show that I didn’t enjoy doing any of that.

If you believe your feed was pulled in error, please file a bug so I can reinstate it directly.

Durable Design

Flip

It seems like small thing, but it’s an engineering detail I’ve always had a lot of respect for.

That picture is of a Flip video camera with the lid off, a product from about nine years ago. It was a decent little video camera at a time that phones weren’t up to it, storing a bit over an hour of 720p video with decent sound. The company that made them, Pure Digital Technologies, was bought by Cisco in 2009 for about $590M and shut down less than two years later. Their last product – that ultimately never shipped – could stream video live to the Web, something we wouldn’t really see from a pocket-sized device until Periscope and (now-dead) Meerkat took a run at it five years later.

The thing I wanted to call attention to, though, is the shape of that case. The Flip shipped with a custom rectangular battery that had the usual extra charging smarts in it and you could charge off USB, like all civilized hardware that size. But it also gave you the option of putting in three absolutely standard, available-everywhere AAA batteries instead, after that exotic square thing finally died.

You only get to run the camera about two-thirds as long, sure. But long after they’ve stopped making those custom batteries or supporting the device itself, the fact of the matter is: you can still run it at all. It may not be the best thing around, but it’s also not in a landfill. It still does everything it said it would; my kids can make movies with it and they’re good fun. It didn’t suddenly become junk just because the people who made it aren’t around anymore.

I’ve often wondered what those product meetings looked like at Pure Digital. Who pushed for that one extra feature that might give their product a few extra years of life, when so many market forces were and are pushing against it. What did they see, that convinced them to hold the line on a feature that few people would ever use, or even notice? You see it less and less every day, in software and hardware alike – the idea that longevity matters, that maybe repair is better than replace.

If you’re still out there, whoever made this what it was: I noticed. I think it matters, and I’m grateful. I hope that’s worth something.

Cleaning House

Current status:


Current Status

When I was desk-camping in CDOT a few years ago, one thing I took no small joy in was the combination of collegial sysadminning and servers all named after cities or countries that made a typical afternoon’s cubicle chatter sound like a rapidly-developing multinational diplomatic crisis.

Change management when you’re module owner of Planet Mozilla and de-facto administrator of a dozen or so lesser planets is kind of like that. But way, way better.

Over the next two weeks or I’m going to going to be cleaning up Planet Mozilla, removing dead feeds and culling the participants list down to people still actively participating in the Mozilla project in some broadly-defined capacity. As well, I’ll be consuming decommissioning a number of uninhabited lesser under- or unused planets and rolling any stray debris back into Planet Mozilla proper.

With that in mind, if anything goes missing that you expected to survive a transition like that, feel free to email me or file a bug. Otherwise, if any of your feeds break I am likely to be the cause of that, and if you find a planet you were following has vanished you can take some solace in the fact that it was probably delicious.

The Minimum Viable Context

2017-05-09_08-06-58

This is not a subtweet; if I thought this should be about you, I’d have said so to your face months ago. If you get all the way through it and still kind of suspect it’s about you, though, you should spend some time looking inward and gear yourself up to deal with whatever you find in there, rattling the chains.

I’ve started and stopped writing this a couple of times now. Some drafts have been didactic, other self-congratulatory. “Blogging isn’t real if it’s not the first draft”, I’ve read somewhere, but I’ve never been able to do that; writing has always been a slog from what I’ve got written to what I can just barely sense I could. If I wanted to flatter myself I’d wheel out the old Mozart/Beethoven analogy, but that feels too much like fishing for compliments and besides, that garbage was in an early draft too.

So let’s lead with the punchline. Here’s the checklist: does everyone on your team…

  1. have a shared understanding of success?
  2. know what everyone else’s role is, and what they need to do their job well?
  3. know how their work contributes to the team’s success?
  4. know how their team’s success contributes to their own?

If you’re surveying the field from the executive suite and need big-picture, master-class management advice, well. This is not that. Talk to my friends Shappy and Johnath at Raw Signal. If you understand what they’re offering you know better than to look for it here. What I’ve got here is penny-ante table stakes, the difference between a team and a handful of people sharing the same corner of an orgchart. It is not complicated; it should, in theory, be trite. But to borrow a line, the fact is that in the day-to-day trenches of adult existence banal platitudes can have life-or-death importance.

In theory, you’d think hitting 4 out of 4 would be not just easy, but expected. In practice, in my experience, you’ll be lucky to make it to 2.

A few months ago I was asked to help a team out of the weeds. Getting into the details would be a disservice, so I won’t; in the broad strokes, I’m talking about a cross-discipline team of smart, invested people doing an important job. But for whatever reason, something – several somethings, it turned out – had gone really, really wrong. Execution, morale and retention were all going south. Everyone knew it, but nobody was really sure what had happened or what to do about it.

So I talked to a lot of people, I read a lot of mailing lists and bugs, and offered some advice.

If you’ve been around the team-dysfunction block before, you know there are plenty of probable causes. Shakeout from a reorg, a company pivoting hard, a team managing some sudden turnover, maybe the organization has grown from everyone being in the same room to nobody even being in the same city. Maybe you’ve hit that critical mass where communicating has suddenly gone from something nobody needed to worry about to something nobody remembers how to do. Maybe the one person who made it work left, maybe it’s just been that way so long nobody remembers the possibility that it could be different.

The advice I had for them was straightforward, a word I love for the veneer of upright nobility it adds to a phrase I could just as easily close out with “simple” or “obvious”. Get everyone into the same room for a few days, preferably away from everyone’s home base. Start the first day by having everyone give a talk about their jobs, not some name-and-title intro but a deep dive into what their job involves and the information, context and resources they need to do it well. Have some conversations – some public, some privately – between team leads and members about personal or professional goals and growth paths.

And then take the roadmap and the entire work backlog for the team and – ideally in the last meeting of that first day – print it out, stand up in front of everyone and drop it on the floor. Then tell everyone to come back the next day ready to start fresh.

The goal of this exercise was to make all the hidden costs – all the hidden work, all the hidden friction, everything people couldn’t see through the lens of their own disciplines – visible. And then, with that information, to take a hard reset. To narrow the team scope down to one or two tightly focused, high-impact features that could ship soon, and – critically – explicitly stop working on everything else. That sounded a bit dramatic, maybe impossible – I’ve been called worse – but nothing else seemed like it would work at all.

Because when I was asking my questions, the answers I got were mostly about the questions those teammates were asking each other. And it wasn’t hard to spot a common theme.

“If only it weren’t for the people, the goddamned people,” said Finnerty, “always getting tangled up in the machinery. If it weren’t for them, earth would be an engineer’s paradise.” – Kurt Vonnegut, “Player Piano”, 1952

Does everyone on the team understand that when you ask a designer to make a new button, that you’re asking them for a few dozen hours of product and market research, and a few more of design and testing, and not half an hour in Illustrator drawing pretty pictures? Does everyone really get that accommodating that schema change means refactoring a pile of business logic or backup processes? Did you all notice that you were asking for a contractual change with a major partner when you said “just change this string”?

I made those questions up for this post; the real ones were different in the specifics but definitely not in substance. You realize that you’re asking for the entire process, not just the output at the end, right? Why don’t you just?

You’ve seen this. You’ve probably even asked questions like them; I sure have. And unchallenged, even the mildest case of engineer’s disease left untreated will fester; eventually cultural rot sets in. We don’t really have a word for the long decline that happens next, the eventual checking out that happens the moment you clock in. The septic shock, the team’s paralysis and organ failures of core people ragequitting near the end. But you’ve seen that, too.

“You should focus on a small number of things” and “it helps to understand how your colleagues do their best work” is not exactly going to spur a revolution in technical leadership. I get that. But: don’t mistake the roadmap for the terrain. If you’ve made that plan without a clear, shared idea of where you’re going, how everyone can help you get there, and why you’re going at all? Then it’s hard to see how that will succeed, much less give rise to the kind of work or working environment you can be proud of. So toss it. Do the work of understanding where and who you are, and draw the map from there to somewhere that matters.

I told you this was table stakes, and I was not kidding about that at all. I wanted to help them get to a point where everyone on the team could confidently go 4 for 4 on the list, to get them to necessary so they could launch themselves at sufficient. And now, a couple of months later, I think it worked. They’re not all the way there yet – culture’s got a lot of inertia, and if I ever find a way to hard-pivot a whole org I’ll let you know – but they’re on the way, with a lot of clarity about what they’re doing, how they’re going to get it done together, and why it matters.

So: what about your team? Does everyone on your team have a shared understanding of success? Do you know what everyone else’s role is, and what they need to do their job well? Do you know how your work, and theirs, contributes to the team’s success and to your own?

Or does your team – maybe, possibly, kind of, just – suck at being a team?

You should do something about that. What are you going to do about that?

Trimming The Roster

This is a minor administrative note about Planet Mozilla.

In the next few weeks I’ll be doing some long-overdue maintenance and cleaning out dead feeds from Planet and the various sub-Planet blogrolls to help keep them focused and helpful.

I’m going to start by scanning existing feeds and culling any that error out every day for the next two weeks. After that I’ll go down the list of remaining feeds individually, and confirm their author’s ongoing involvement in Mozilla and ask for tagged feeds wherever possible. “Involved in Mozilla” can mean a lot of things – the mission, the many projects, the many communities – so I’ll be happy to take a yes or no and leave it at that.

The process should be pretty painless – with a bit of luck you won’t even notice – but I thought I’d give you a heads up regardless. As usual, leave a comment or email me if you’ve got questions.