It’s gone. It’s all gone.

Okay, it’s not all gone, but enough is gone that—

Perhaps I should start over. I am, after all, starting over.

About three months ago, I got the first notification that the eleven-year-old single hard drive in my colocated server, Prisma, was failing: a single Unrecoverable Sector Error message, quietly emailed to root. I didn’t see it when it came in, of course. I’d long ago given up reading root‘s email; I’d set DenyHosts, RootKit Hunter, and cron to email me everything while I was debugging some problem, solved it, figured I’d clean up after myself later, and then got sidetracked. I had dozens, perhaps hundreds, of these kinds of projects: half-finished, oft-remembered, never-important-enough to preëmpt anything fun or interesting. So when that message arrived—well, when I noticed it—I did what I usually did with such things: I said to myself, “I should do something about that,” and then I went back to whatever it was I was doing.

A week or so ago, the Great Work opted to force the issue. The error that had started with one bad sector had become fifteen, and then one morning I woke up and I started getting really weird mail errors; I could read my mail just fine, but I wasn’t getting any new mail, and I wasn’t able to delete the old mail any more. I thought to myself, “okay, my laptop is four years old, I’ve seen a few errors on Prisma, I’m thinking really hard that I should probably take a backup soon, please don’t tell me I have to repair two computers at once.” A bit of websearching later, and I logged onto the server to check my mail logs. I didn’t see any errors in them, so then I got really confused and started digging into the syslog. And that’s where I found it.

Not in the logs; in the timestamps. The server was still up and running, but the mail log, the syslog—in fact, all the logs—had stopped updating at the same time. But the server was still up. That meant something was wrong with—

root@prisma:/tmp# touch foo
touch: unable to open foo: read-only file system

root@prisma:/tmp# 

ohno

I started my first attempted system recovery that night by rebooting the server to restore the file system. This had the unfortunate side effect of kicking off a forced fsck of the only hard drive in the system, and fsck was, to say the least, unhappy with the state of affairs. The drive was in such sorry state that the automated fsck actually failed, forcing me to run it manually. Every time it hit a bad sector, I had to hit Y for it to continue, forcing me to sit through its Liturgy of Neglect. It did eventually end, though, and after a second reboot the server came back online and prompted me to log back in.

“Qapla’“, I exclaimed, and immediately started to bzip2 the contents of the hard drive so I could download the compressed files. If I hadn’t started this whole process after midnight, I might have realized that trying to write to the drive would exacerbate the drive failure. I didn’t, however, and about two-thirds of the way into the job, the file system crapped out again. Still awkwardly locked onto this whole “create an archive first” idea, I rebooted the server a second time.

This time, fsck didn’t bother to hide its contempt. It just straight-up started throwing sectors into /lost+found and dared me to tell it no. I got about thirty minutes (!!) into this bad idea when sanity struck me and I started just trying to rip the hard drive as-is using WinSCP and a remote root login to scrape whatever bits I could get off of the disk before it completely died.

---

The disk copy was at about twenty percent two days later when my hosting provider very helpfully contacted me to let me know they’d been seeing some alarms with my server, and did I want any help. I thanked them for their notification, told them I was going to need a new hard drive, and asked them to please pend the ticket until my backups were done. They thanked me and I went back to my copy, but by day three with only a quarter of the drive downloaded, I started to sweat, and the support emails started coming in more frequently. “Just checking in to see if you need help,” or “Hey these alarms are still going off; do you need anything,” or “the smoke has stopped; is it safe?” That sort of thing.

My spouse very helpfully suggested running rsync directly from the NAS. I snapped something incoherent and fiddled with my WinSCP jobs. They very wisely left me alone. On the fourth day of madness, I realized that I no longer trusted the hard drive to last until my job was done, and I started thrashing about looking for anything that could speed up the job. A few hours later, I finally listened to my betters and tried running rsync directly from the NAS. When I saw files coming down in a rush, I nearly cried with relief.

My spouse forgave me, too, which was nice.

As soon as the last download job was done—01h00 of the fifth day with no mail—I contacted the techs and asked them to please install the new hard drive, attaching the old one as a secondary, and I would commence with the OS reinstall. Half-an hour later I realized I’d forgotten to download one very important folder: /root. This was where I’d been storing all my custom-built admin tools, but more importantly this is where I’d been storing my database backups. I may have saved all my physical files, but I didn’t save anything that had only been stored in the database.

The server is back, of course. I still have a bunch of work to do, more of which I think will get done this time than last. I say that now, of course, but I’ve learned a few more things since then, and I’ve been figuring out what needs to come back and what doesn’t. If I have files of yours, and you want them, let me know; I’ll do what I can to get them to you, even if I have to upload them here so you can archive them yourself. If I didn’t manage to save it, whatever it was, I’m sorry. The old drive now hangs like a digital appendix in the newly reborn Prisma.

But so, so much has been lost. Twenty-five years of blogging history, gone. All the rough copies of my novels, several short stories, all gone. The accumulation of so much history has just been wiped clean. The flames of transformation have caught up with me again, and this time they’ve taken more than my present; they’ve taken my past.

And yet, I find it hard to feel sad at the loss. I’m not the person I was when I started writing the blog. I’ve been through multiple relationships, living arrangements, jobs, genders, and selves since then. I’ve moved across the country twice and dated across multiple national borders. I’ve revised all of those texts and most of those short stories. Despite the initial rush of self-loathing and frustration, what’s settled in my crop is a sense of liberation, of having been able to scrub clean a bunch of the past and simply start new.

I wonder if this is how the phoenix feels, ascending from its funeral pyre.