Klaus on Tilde Town

Saving your offline server with... irssi!

After announcing my Bring Back Blogs! challenge, my self-hosted server went offline. This couldn't have been any more ironical, because my announcement did include the personal tech goals that I was intending to lay down, one of them being an increase in self-hosting and reducing my reliance in third-party services. Oops! What a way to start the year, huh? '^_^

To aggravate the situation, I was a few thousand kilometers away from my house - and, thus, my server - at the time, and did not have plans to go back until a much later time to try to fix physical issues had they happened. Uh-oh. Thus, I was left with a disappointing trade-off: either wait out and suffer the downtime of weeks without posting, or use my tilde.town backup mirror to keep up the blogging.

Both were quite expensive, turned out, because my server also carried other services that needed, including part of my backups. I had to think about something else to try.

Surprisingly, though, I was back up and running the day after the incident, and perhaps even more surprisingly, managed to do so with quite an obscure solution: the irssi IRC client!

This is more or less how my procedure went.

Tor didn't work

So the main line is down and I can't ping or do anything with it. Don't I have backup routes or something?

Oh yes, I do: way before I started hosting services available over the internet, way before I could configure my ISP's modem to allow a NAT passthrough, I had first self-hosted via a hidden service! This was my first trick to access my files and git repos away from home and, even though it took some time, it worked well.

I figured that what probably had happened was that my ISP's router rebooted or suffered a hiccup of sorts, which led the IP address to change, and thus the domain to point to something nonexistent. Nope, I never read about BIND to properly get it configured. But that would be easy to fix, all I would have to do was:

Easy-peasy, but for some odd reason, I couldn't SSH into the hidden service. It simply appeared to be down altogether. Adding the -vv flags wouldn't reveal any more information about the demise of my server, and just like that my backup line was dead. No route in sight, apparently. At this moment I began to wonder what could've possible happened to bring the server down: power failure? Hard drive crash? Some software crash that locked up the system? DOS attack?

Irssi to the rescue!

I was about to give up and live with the downtime when I noticed something almost by chance, in what was probably the last place I'd look: the tmux pane of my terminal that was running irssi, the IRC client. Before we proceed, though, some background:

See, I have this little trick where I make a channel in a large IRC network and have my server join there to "hold the room" in sort of way, as well as all other devices that I am using. Having all my devices accessing this channel allows me have a shared clipboard through which I can pass interesting links and other text snippets across them in a quick way (if something is sensitive that requires encryption, I can do something like this strategy with CyberChef).

Most importantly, however, by having my server holding the channel all the time, if a specific device gets disconnected without receiving a message, I can simply access the server's user and replay the message, since the server is the one "guaranteed" to see all the messages.

So, having glanced at the irssi window, my thoughts suddenly went like: "wait a second, could be that my server is still connected to IRC?" I switched channels to look and, surprisingly, I saw the nick from my server right there, standing tall. So there was no internet connection problem, after all! It was a much lesser problem in the end.

If the server is up, why can't I access it via Tor? Very good question that I still can't answer. If only I could find that nick's IP address from its connection to the IRC server! And thankfully, there's a way to do that. Run the command /whois $nick into the input box and you'll find a lot of information that is exposed to the IRC server and, if you're lucky, the host IP address is included if not masked or obfuscated somehow. Mine wasn't, and sure enough, I found my new public IP address from there.

Final test: do a raw ssh -i ~/.ssh/mykey user@ipaddress to see if it works and sure enough it did. Victory. Next, all I had to do was run the script that updates DNS records and my server was rendered online again.

Lessons learned

Whew! It's always easier to see via hindsight, but I missed out on an opportunity to use automation to let the problem solve itself. My DNS updating could've been done in an automatic way by the means of a cron job running every 3 hours or so.

You could try making the job more frequent to reduce the window of downtime, but some free DNS servers state that updating too quickly is against their terms of use and could boot you out. For that, you can make a script that checks first if the address has changed before actually requesting the update. This curl snippet, for example, will return your public IP address:

curl --silent https://lite.duckduckgo.com/lite?q=ip | awk '/Your IP address is/ { print $5 }'

Yay for scraping!

In a more serious way, you might actually want to study and set up something like BIND to properly synchronize your IP and domain in a less hacky way. I didn't do it, and frankly feel quite lazy to do so, but it's the way to go to make self-hosting much more robust, especially if your router supports it!

Finally, who would think that IRC would save the day in this sort of way, right? This will definitely be a trick I'll keep up the sleeve for the rest of my self-hosting adventures, but also shows a big warning about privacy: your IP and ISP information is exposed to the IRC users unless you cloak it or obfuscate it in some other manner. Be careful out there if you need to be anonymous in a channel!

Have you ever used an unorthodox method to "fix" your server like this one? How did it go? Let me know on Mastodon!

This post is number #41 of my #100DaysToOffload project. Follow my progress through Mastodon!

Last updated on 01/21/23