blarg?

Profanity, Followed By More Profanity, Updated With More Profanity.

I’d like to talk about Ultimate, and how Coop’s team beat us the other night and how he’s a crafty player, but it’s been a bad few days, so I’m going to talk about that instead.

In light of Shaver’s Surprise Guest Administrator incident, I took a good hard look at the Nexus project, and, lo and behold, we’re hosed. Nexus, and four other machines on campus, have been compromised. So my days have recently looked a lot like I suspect his did, go buy new drives, do the reinstalls and then spend time I don’t really have doing forensic work on the old drive to figure out what else has gone wrong.

Boy, had things ever gone wrong. It’s not completely clear which way the plague spread, but a naive comparison of dates between bitchcake and nexus seems to show that nexus was compromised first. Either way, my account was the infection vector.

From what we can tell, the compromised machines had been set up to run IRC bots, presumably for a denial-of-service attack at some point in the future, or just as part of some script-kiddie-related penis-size contest. We haven’t got that far yet; I’ve been busy fighting on-campus fires and helping Shawn reinstall Debian on the afflicted machines. We’ve tracked the person responsible to, apparently, Romania, via some IRC-related sleuthing, and developed what appears to be a reasonably good protocol for preventing this from happening in the future.

The “Romania” thing really disappoints me, because I had really hoped that I’d be able to hold this person’s throat in my hands. Ever seen that bit in Pulp Fiction where Vincent Vega’s talking about how is car got keyed? It would have been worth it, if I could have just caught him doing it.

We also found an incredible list of other compromised machines in the logs.

I’ve contacted all the people on campus and on our subnet who were on the list. I’m really not sure what else I can do, aside from battening down the hatches and trying to keep it from happening again, but I have this list of maybe four hundred compromised machines here from all over the world and I’m not sure what to do with it. About half of them end in “undernet.org”, though, and I know exactly what I’m going to do about those.

God Dammit.

Update: As it stands, the evidence is that the spread of the infection went from Engsoc here at Carleton to Nexus, and from there to Bitchcake. And it gets better: apparently the people at Engsoc knew about a break-in back in the middle of January, and didn’t tell anyone.

Shaver was actually apologizing to me about this. I’m going to have to clear that up with him, and maybe beg forgiveness while I’m at it.

After I find somebody appropriate at Engsoc, bite their goddamned arm off and pound them down with it.

9 Comments | Skip to comment form

  1. Nick

    Undernet.org? Who’re they? What exactly are you going to do about those? I am intrigued. I expect gore, but maybe that’s because I’ve been playing a lot of Halo lately. Dave got an X-Box.

  2. Mike Hoye

    Undernet is the Vanier of the internet.

  3. Mike Kozlowski

    Man, that’s a pain in the ass when that happens. I’ve had a couple of Linux machines rooted in my day, too. I’ve decided that Linux is too insecure for production work these days, and that Windows is the way to go — far less likely to have anything bad happen to you if you’re not checking your email on it.

  4. sean

    Mike… do i even have to say it.

    ps. i’ve given up nethack for lent :) Thoth will be angry.

    s.

  5. Melanie

    I remember way back when there were these prank emails sent around saying “watch out for viruses being passed via email”, and everyone laughed, because those in the know knew that this was impossible! So Microsoft went and invented a way to do it. Or rather invented a way to let others do it.

    Not that this has anything to do with your problem, Mike. Sorry – sucks rocks.

  6. uncle kev

    Ugh. Being on the receiving end of rooted machines in production and customer environments, I totally feel your pain. I’m curious to hear what exploit was used to compromise the boxes (I’m guessing folks had ssh set up to allow unauthenticated logins, and from there they just used local exploits), and the curiosity goes beyond the merely morbid, as I have a few boxes here or there.

    I would go beyond beating the shit out of the folks at Engsoc, I’d bring it up with Carleton’s IS department. When their machine got rooted, they had an obligation to disclose it to their user community. It’s disappointing an organisation that’s had a network presence as long as it has is that irresponsible. I would have expected they’d behave better.

    Really sorry to hear about that, and if there’s any stories you want to swap or practices to share, let me know.

    P.S. – Because of you, I have had Mark Morrison rebounding between my ears for three days. GD catchy-bad Return of the Mac.

  7. Mike Hoye

    So far, we’ve got:

    • Trojaned SSH on Engsoc leads to
    • access on Nexus, leads to
    • local root-exploit on Nexus (Suckit) allowed by a pre-2.4.24 kernel
    • Sniffers installed, /bin/false replaced with a shell, etc.

    So, best practices are now:

    • Keep up to date on security updates, especially in sshd and kernel
    • Force regular password changes
    • Daily chkrootkit
    • install Tripwire

    We’ve also had to do reinstalls of all our other machines, and mail people all over the place who’ve also been compromised. Unlike, say, some other people.

  8. Kosta Jilkine

    We had it a bit worse, I think. They got in through Oren’s account. Bastard.

    Afterwards, the bot farm was used to ddos us.

    They brought down our colo for 30 min until they pulled our server from the net.

  9. dusty

    Do me a big favor: beat the living sh*t out of them. We got hit by the same thing.. this time it was Oren’s account that was the way in. Apparently we made them angry.. they turned their botfarm against us and put us and our colo-co out for 30 minutes. As it is right now, our server is still down, of course. It has effectively been down since Thursday.

    Dusty