I would like to thank the City of Toronto for warning me away from this clearly haunted staircase that’s somewhere in the Don Valley.
Your Mulders, Coopers and Constantines get all the limelight, but as far as I’m concerned, in the fight against your garden variety supernatural malefactors, it’s the workaday maintenance and cleanup crews that are the real heroes.
If you’ve updated a mainstream Linux recently and it feels like everything is inexplicably crashy and unreliable when you’re trying to get anything of substance done, particularly on smaller machines, it’s not just you.
Fedora and Ubuntu have both started shipping with systemd-oomd – an unproven SystemD OOM-killer – turned by default, on the desktop versions of their products.
To break that incantation down – “single-user desktop machines run the system daemon out of memory daemon by default” – into what it means and explain how we got here would be an exercise in ritual humiliation that I have no inclination to pursue. A 17th-century grammarian two centuries too early for the light bulb might glance at that phrase and ask how the infernal ‘machine’ in question had become so thoroughly curse-ed that the word “daemon” is in there twice, and that would be an excellent question. Here in the 21st century it’s still an excellent question.
But in short, an “oom killer” is a program whose only job is to shut down – “kill” in the parlance of Unix – processes that are hoarding enough memory that the stability of the system is at risk. That is, when the whole system is out of memory, and one process is threatening to starve all the other processes out. The idea was born at a time when computers were insanely expensive, and it makes sense as a tool in exactly one setting: as a last line of defence on multi-user systems being shared by people doing work the system operators don’t care about at all.
A consumer desktop machine’s operator is also their sole user, and who almost certainly cares about no part of this computer’s operation at all but the work they’re doing. It is makes precise opposite of sense, and shipping an oom-killer at all – much less an unproven, buggy oom-killer – in that context was incredibly irresponsible. Prioritizing “system responsiveness” over literally whatever the human at the keyboard of the computer they own was working on at the time is an insult. It guarantees that the user experience of Linux will always be “it crashes at the worst possible time, every time”.
You can turn off systemd-oomd – I say again, “the system daemon out of memory daemon”, because nonsense unix-terminology salad is apparently just how we roll now – with:
sudo systemctl disable --now systemd-oomd
(you get down) followed by
sudo systemctl mask systemd-oomd
(and you stay down.)
Max Planck said, once upon a time, that science advances one funeral at a time. Maybe the Year Of Linux On The Desktop isn’t coming until people who grew up with phones take the reins from the people who grew up on mainframes. Maybe that’s already happened, and the Unix on the Desktop we’re stuck with is MacOS.
Getting this thing spun back up was a bit of an adventure – as you are no doubt unfortunately aware: software – and I wanted a few things out of my new setup, among those being “self-hosted WordPress backed by SQLite”, “https”, “cleanly importing the old data” and a few other modest housekeeping improvements. I’m starting with a network-facing Linux instance here, and I’m going to gloss over a few pretty hairy steps like “set up Apache2”, for which I apologize. I’ve been doing this for a long time; if you’ve found your way here in the weeks or months after this has been published and need more detailed instructions, email me.
Once you’ve got your Linux instance and Apache stood up straight, the rest of the preparation steps are straightforward:
“Have Certbot do its thing” was actually remarkable: I am just a simple unfrozen caveman and your “hosted services” confuse and frighten me, and it’s been probably a decade since I’ve looked into this space so being able to just… cause a cert to occur? Without needing to put a credit card into anything or fax things to companies whose software you wouldn’t run with a gun to your head? Amazing. The Certbot+LetsEncrypt process was so fast and painless that I was actually startled, and “pleasantly startled by software” is a rare feeling I deeply mistrust. But it worked right on the first try, pretty wild.
Getting WordPress working right with SQLite was largely the opposite of that. Who doesn’t love debugging other people’s PHP for db-constraint correctness problems, or hand-editing XML? Well, everyone. That’s who doesn’t love it. You, me, everyone.
Before turning on Apache you’re going to want to take WP-SQLite-DB up on their offer to put the DB somewhere sensibly non-default, you might need to toss it and start fresh later, but fortunately if your database is SQLite that’s “mv” and not “oh god now I need to re-learn how MySQL works again”, and that on its own will make the whole effort worthwhile.
Anyway, let’s get into the two bits of unpleasantness.
The first thing you’re going to need to do is manually clean up your old WordPress backup. I don’t know when this bug was introduced, but WordPress-current will not import its own backups correctly. Specifically, when there is a line break and tabs between <content:encoded> tags and the <![CDATA[[ tags that follow it, the importer will assume that tab-tab-tab-tab-<![CDATA[[ is the part of the content you’re importing rather than an indication that it’s encoded. This manifests itself in – if you’re lucky! – all your old blogposts losing most of their formatting on import. But that’s not the best part.
No, the best part is that once you import that backup, if you open thus-afflicted posts in the WYSIWYG editor they’re all blank. Fun race condition: if you touch the post with the classic text-editor option first, it assumes it’s all text, and escapes all those brackets for you when you switch to visual. But if you touch it with the visual editor first, it assumes everything in those CDATA brackets is invisible, and helpfully scrubs it clean for you. Super helpful either way, but opening the editor for a post you know exits, if imperfectly, on your sites and discovering that the editor believes that page to be blank? You need to have a certain mental model of this class of malfeasance to figure that out, I think, and I’m not sure if I should be proud I can connect those dots easily or annoyed that this is the person I’ve spent my life becoming.
Like I said, I’ve been doing this a long time. Get you a man who can do both.
Anyway: I’m sorry I don’t have a regression range for you, but if you’re importing old WordPress backups into WordPress 6 you will definitely need to inspect that backup XML file, get your regex on and strip all the whitespace between the <content-or-whatever:encoded>
tags and the <![CDATA[[
bits or you’re gonna have a bad time dot gif.
At this point you’ve got a WordPress instance that you can stand up, data that you can import, and superficially this will appear to work. The problem you’re going to hit when you’re trying to configure WordPress through its usual menus – for example, disabling comments – is that saving those options gives you a dialog saying your changes have been saved, but they won’t have changed.
WordPress won’t show any errors in the UI, but will instead inform you of the problem by dumping a web page into your Apache logs. It looks something like this:
[Tue Jun 21 17:18:07.151358 2022] [php7:notice] [pid 12535] [client IPADDR:57288] WordPress database error <div style="clear:both"> </div><div class="queries" style="clear:both; margin_bottom:2px; border: red dotted thin;">Queries made or created this session were<br/>\r\n\t<ol>\r\n\t\t<li>Raw query:\nUPDATE `wp_options` SET `option_value` = NULL WHERE `option_name` = 'default_ping_status'</li>\r\n\t\t<li>Rewritten:\nUPDATE wp_options SET option_value = NULL WHERE option_name = 'default_ping_status'</li>\r\n\t\t<li>With Placeholders:\nUPDATE wp_options SET option_value = NULL WHERE option_name = :param_0 </li>\r\n\t\t<li>Prepare:\nUPDATE wp_options SET option_value = NULL WHERE option_name = :param_0 </li>\r\n\t\t<li>Executing:\narray (\n 0 => 'default_ping_status',\n)</li>\r\n\t</ol>\r\n</div><div style="clear:both; margin_bottom:2px; border: red dotted thin;" class="error_message" style="border-bottom:dotted blue thin;">Error occurred at line 1713 in Function execute_query. <br/> Error message was: Error while executing query! Error message was: SQLSTATE[23000]: Integrity constraint violation: 19 NOT NULL constraint failed: wp_options.option_value </div><pre>#0 WP_SQLite_DB\\PDOEngine->get_error_message() called at [/opt/wordpress/wp-content/db.php:2736]\n#1 WP_SQLite_DB\\wpsqlitedb->query(UPDATE `wp_options` SET `option_value` = NULL WHERE `option_name` = 'default_ping_status') called at [/opt/wordpress/wp-includes/wp-db.php:2476]\n#2 wpdb->update(wp_options, Array ([option_value] => Array ([value] => ,[format] => %s,[charset] => ,[length] => )), Array ([option_name] => Array ([value] => default_ping_status,[format] => %s,[charset] => ,[length] => ))) called at [/opt/wordpress/wp-includes/option.php:496]\n#3 update_option(default_ping_status, ) called at [/opt/wordpress/wp-admin/options.php:322]\n</pre> for query UPDATE `wp_options` SET `option_value` = NULL WHERE `option_name` = 'default_ping_status' made by update_option, WP_SQLite_DB\\wpsqlitedb->query, WP_SQLite_DB\\wpsqlitedb->print_error, referer: SITE/wp-admin/options-discussion.php
… which I think we can agree is less than helpful, but with some distillation we discover:
Error message was: SQLSTATE[23000]: Integrity constraint violation: 19 NOT NULL constraint failed: wp_options.option_value
This is the second part of the unpleasantness, and this took quite a while to figure out; that SQLite has some database constraints that WordPress doesn’t meet and isn’t interested in meeting. I’m not an expert – I think the general thrust of the problem is that where SQLite defaults to strict, MySQL defaults to sticking a crayon in its nose – but some precision chicanery in options.php seems to sort us out. In wp-admin/options.php at around the 320th line or so:
if ( isset( $_POST[ $option ] ) ) {
$value = $_POST[ $option ];
if ( ! is_array( $value ) ) {
$value = trim( $value );
}
$value = wp_unslash( $value );
}
// You probably shouldn't do this, but it makes WordPress work on SQLite.
$value = $value ?? '' ;
// If this breaks anything I am very sorry. It works on my machine. -mhoye
update_option( $option, $value );
A single line of code. The precise, surgical elegance of a scalpel taped to a brick. But it works. For me, on my machine.
Shut down Apache, rename the DB file and restart it, and there you are. A clean install of WordPress running (far as I can tell…) correctly backed by SQLite3.
Nothing to it.
Addendum: “Nothing to it”, he said, full of hubris. Well, haha. If you upgrade your Linux box – Ubuntu, in my case, via do-release-upgrade – you’ll need to reinstall libapache2-mod-php8.0
and php8.0-sqlite3
afterwards, and afterwards maybe do some aggressive career counselling for whoever decided that dumping an entire web page into my logs was OK.
Apologies for the interruption, and the ugly defaults. Content has returned and full service will resume shortly. Please enjoy our new LetsEncrypt cert, a tolerable theme will be along shortly.
I have some recovery-editing to do, because the the WordPress import/export process is remarkably bad at importing things that have been exported.
On his deathbed, in the final moments, an angel appears radiant and glorious by Machiavelli’s bedside. Niccolo, it says in a gentle, sonorous whisper, your time here comes to an end. Come, the hour has struck that you must move on; know in your rest that your works will endure.
Among kings, queens, he asks in relief? Princes and ministers, their consorts and counsels?
No, the angel smiled, leaning close. No, Niccolo, no. It’s mostly assholes in airports.
“what”
A lot of them are moving their lips while they read.
“what is… how many?”
About half. I’d say half.
Me, unixy old person: “… and before end to end encryption was standardized, that meant that something called replay attacks were possible. To mitigate this, kerberos authentication was invented, which…”
A young unixy person: “Can I pet Kerberos?”
Me: “what”
“Kerberos was named for the three-headed dog that guards the gates of the underworld, right?”
“… yes?”
“Can I pet the dog, is my question.”
“…”
“…”
“Ok, no, but file that bug immediately.”
I had to ride out for an urgent cross-town delivery yesterday, and after blowing through 15km of fogged-out greyscale city like a fast ghost I found myself in a dimly lit elevator thinking to myself, self, maybe you did get to be that William-Gibson movie extra you wanted to be. Good work, self.
Already, the children of Earth were the most terrifying creatures in the galaxy. They became the stuff of horror stories, nightly warnings told to children; huge, hulking, brutish things, that hacked and slashed and stabbed and shot and burned and survived, that built monstrous metal things that rumbled across the landscape and blasted buildings to ruin.
All that preserved us was their lack of space flight. In their obsession with murdering one another, the humans had locked themselves into a rigid framework of physics that thankfully omitted the equations necessary to achieve interstellar travel.
They became our bogeymen. Locked away in their prison planet, surrounded by a cordon of non-interference, prevented from ravaging the galaxy only by their own insatiable need to kill one another. Gruesome and terrible, yes – but at least we were safe.
Or so we thought.
The cities were called Hiroshima and Nagasaki. In the moment of their destruction, the humans unlocked a destructive force greater than any of us could ever have believed possible. It was at that moment that those of us who studied their technology knew their escape to be inevitable, and that no force in the universe could have hoped to stand against them.
The first human spacecraft were… exactly what we should have expected them to be. There were no elegant solar wings, no sleek, silvered hulls plying the ocean of stars. They did not soar on the stellar currents. They did not even register their existence. Humanity flew in the only way it could: on all-consuming pillars of fire, pounding space itself into submission with explosion after explosion. Their ships were crude, ugly, bulky things, huge slabs of metal welded together, built to withstand the inconceivable forces necessary to propel themselves into space through violence alone.
It was almost comical. The huge, dumb brutes simply strapped an explosive to their backs and let it throw them off of the planet.
I keep thinking about the intersection of the “Humans are terrifying, humans are in fact Space Orcs, (or Space Fae?) stories about how humans clearly the most dangerous and/or blessed aliens out there by far” stories that periodically bubble up out of Tumblr, and the somewhat painful realization that the Star Trek series is secretly a hidden prequel to Iain Banks culture novels, by dint of the whole Hidden Strong AI Everywhere issue.
There’s obviously a lot about Star Trek series that doesn’t really stand up to scrutiny, but here’s the thing: despite all their hemming and hawing about how Lt. Data was a unique and special synthetic intelligence, about how purely inorganic sentience is this rare thing, the fact of it is that moment LaForge asks the computer for “an adversary worthy of Data”, the Holodeck delivers a whole-ass, self-aware parasentient synthetic intelligence running on their local computation strata in the form of Professor Moriarty, at the cost of… a few moments of heavy processor flex. And apart from that momentary CPU spike, exactly nobody so much as remarked on it.
The implication is that human starship-class systems – and in all likelihood any and probably every human system bigger than a tricorder – are at the very least self aware and almost certainly suprasentient. And from the perspective of any other guest or adjacent culture watching the humans bumble around the universe while this is happening right in their laps, this must be astonishingly terrifying. Put aside the fact of humans are these unstoppable biological engines that brew up toxic chemical propellants so they can drink them for fun, eat chemical weapons purely for the sensation, fart methane by accident and can walk off a gunshot; they travel the stars in the company of gods they’ve built for their own amusement and… they don’t even talk about it?
Imagine the discussion, at the Galactic Council Meeting:
“The humans, we know the humans, but somehow we didn’t know this. As monstrous their history, as terrifying their physiologies, as – dare I say it – fundamentally alien their notions, in our most desperate hours they have appeared as allies, engineers, champions. Among our largest galactic cohabitants, their presence is considered a mysterious if ominous blessing, for our smallest they are – bafflingly, for no motive we can discern – a sword, shield and loyal ally. But… this? How could we have known, or understood, this?”
“Set aside their terrifying predilections, their monstrous history, the absurd resilience of their physiology. They travel the stars cradled in the arms of idle gods of their own devising and this is so unremarkable, so commonplace to them that they not only rarely remark on it, but they rarely so much as notice? How could we have even suspected this? What can we do? About this, about them? What could we do, but play along?”
I’ve often thought “this video game should not have been made into a bad movie”, but Tenet is first time I’ve ever thought “this bad movie should have been a video game.”
An hour into it and I can’t see myself finishing it. Action sequence, cutscene with tediously explanatory dialogue, action sequence, cutscene with tediously explanatory dialogue, bleh.
On the bright side, “Tenet serves up all the cerebral spectacle audiences expect from a Christopher Nolan production” is one of the most Federer-grade backhanded compliments I’ve ever seen, so we have that.