Blinkenshell Blog

I said I should be doing some more technical posts so here we go! Blinkenshell runs the SSH server on a non-standard port (mainly port 2222, but also port 443 for people trying to avoid some firewalls), and the reason for doing this is to avoid some automated bots that go around the internet scanning for open SSH servers and trying different bruce-force attacks to log in. I still think this non-standard port helps a bit, but there’s always some more persistent attackers out there that find SSH servers running on other ports and start hammering away there as well, and for this reason we have fail2ban. Fail2ban will scan through any logfiles you specify and apply certain filters/patterns to find failed login attempts, and if there are repeated failed attempts it will try to perform different actions like blocking the source IP using iptables. For Blinkenshell I try to avoid running too many programs on the main SSH server triton because of security reasons, so our setup is a little bit different.

The first part of the puzzle is rsyslog running on triton sending syslog messages over UDP to a separate log collection server. This is useful for several reasons like trying to figure out why a server crashed or to get forensic data after a host was compromised. In this case we’re going to use it to be able to run fail2ban on a separate host from the one we’re trying to protect. So we install fail2ban on the log collection server and set up a new “jail” that will listen for logs coming from triton. Something like this:

[sshd-triton]
port = ssh,2222
logpath = /var/log/remote/triton.log
enabled = true
filter = sshd[mode=aggressive]
banaction = route

The second part of the puzzle is hinted at the last line “banaction = route” here. Since fail2ban will not run on the same server as the server we’re trying to protect any iptables rules that we install locally on the fail2ban server will not stop the attackers. The idea here is to add any banned IPs to the local routing table, and then send these routes to the firewall and drop the traffic there. This will require a routing protocol daemon running on the fail2ban server that talks to a routing daemon on the firewall.

In my case the fail2ban server runs on Linux so here I’ll choose FRRouting (FRR) for the routing daemon. My preferred choice of routing protocol for cases where you want to specify some routing policy is BGP, so I’ll enable the bgpd in /etc/frr/daemons and systemctl restart frr. You can then enter a cisco-style cli using the command “vtysh”, and go into configure mode using “configure”. Sample config:

ip prefix-list BLACKHOLE seq 5 deny 10.0.0.0/8 le 32
ip prefix-list BLACKHOLE seq 10 deny 194.14.45.0/24 le 32
ip prefix-list BLACKHOLE seq 15 permit 0.0.0.0/0 ge 32
!
route-map BLACKHOLE permit 10
 match ip address prefix-list BLACKHOLE
!
router bgp <myasn>
 neighbor <firewallip> remote-as <firewallas>
 !
  address-family ipv4 unicast
   redistribute kernel route-map BLACKHOLE

To try and avoid any accidental dropping of legitimate traffic I’ll add a little route-map to deny my local prefixes first, and then allow any /32 routes. The “redistribute kernel” line is what will actually take the routes that fail2ban added to the kernel routing table using “banaction = route” and add them to the BGP table.

On the firewall end I’m using OpenBSD so there we will be using openbgd instead of FRR, so it’s a completely different syntax! I’m actually already running bgpd for other things on the firewall (maybe more updates on this later!), but the relevant parts to this fail2ban blackhole thing are:

prefix-set accept-blackhole-in {
    0.0.0.0/0 prefixlen = 32
}
neighbor <fail2ban ip> {
    remote-as <fail2ban as>
    descr "fail2ban"
}

allow from <fail2ban ip> prefix-set accept-blackhole-in set pftable "blackhole"

Again, another filter to only accept /32 routes so we don’t ruin other routing by some misconfiguration. The other key part here is: set pftable “blackhole”. This will add any routes received from this neighbor to a table in pf. We can then refer to this table when writing firewall rules in pf like so:

table <blackhole> persist
block in quick log on $if from <blackhole> to any

Or if you want you can re-route the traffic to some honeypot etc with “route-to” options in pf. This will block all traffic from the banned IP in the firewall, so that IP address will not be able to reach any other services hosted behind the firewall either. This could be seen as extra protection, but it could also cause even more confusion for users that accidentaly type the wrong password too many times and then can suddenly not reach any other services either 🙂 We’ll see how it goes!

Let me know if you’re interested in more technical stuff like this

I’m happy to report that the big maintenance window yesterday was successful and we are now running live on the new server hardware! I’ve spent a lot of time earlier this week doing final preparations and planning the exact steps to take during the migration because I knew there was going to be a lot of work to do during the migration and many things that could go wrong. I felt a bit nervous going to bed on Friday evening before the big day, but I also knew I had done a lot of preparation so I still slept very well 🙂

Saturday morning started with some final preparations relating to the network setup. I had to disconnect both internet and my management connection to the new server to be able to then change it over to the final network configuration. I didn’t want to have to spend lots of time just trying to get back in to the server if anything went wrong so I set up an of out-of-bands connection to the server via a separate laptop for emergencies, then I rebooted the firewall on the new server to apply all the final changes and crossed my fingers I would still be able to log in after moving some network cables over to their new ports, and it all worked out on the first try!

The next step was to copy over a lot of data to the new server. I had tested this out with some less important virtual machines earlier so I knew roughly what to expect. I shut down the web server at 09:43 (according to twitter) and started the copy which took around 20 minutes. Here I also have to convert from vmdk format of disks used by ESX into the qcow2 format used by the KVM hypervisor. Things progressed pretty much as expected here and I continued with the mail services, directory services, ircd and lastly shut down the SSH/triton server.

Once all the services were down I could start copying all of the user home directories. I had prepared this by doing a ZFS snapshot and transferring it to the new server the day before. I did a final snapshot and then started an incremental “zfs send” operation to sync over any files that had changed in the last day. This seemed to work well, but then there was an error message for just one directory. I went to investigate and found that some of the directories did not have the snapshot that I thought I transferred the day before. This is exactly the kind of problem I did not want to run in to at this point 🙂 I knew doing a full copy of all the directories would take somewhere around two hours and I did not feel like sitting around waiting for that long so I devised a little script that would transfer just the missing directories over which was much faster, crisis averted! 🙂 At this point it’s around 12:31.

The next step is to actually disconnect the old server from the internet entirely and move over to the new server and firewall setup. This involved some more network patch cabling, lots of firewall rules and some routing. This went pretty well but there’s always some firewall rules that you miss, which sometimes requires doing some tcpdump work to figure out what’s actually going on. Anyway, around 14:01 things were starting to look pretty good in this area as well.

Next was lots of messing around with NFS exports, since I moved from a Solaris based OS called Nexenta to Linux the options for sharenfs had changed somewhat. Also more firewall rules.

Mail was the first service I wanted to get up and running so that’s what I started working on next. I know mail servers should try and resend mail for something like two days before giving up, but if I ran in to any problems I wanted to have as much time as possible to figure it out before any emails would get lost. This went well and I only had minor configurations to update to get it up and running. I also started up directory services and ircd which went pretty well, just some regular OS updates etc. Now it’s around 16:22.

Web services were next, here I had to do some more troubleshooting and I had actually forgotten to copy some data for the wiki so I had go back and move that over. At 18:23 web was back up and I was starting to feel pretty confident 🙂

Lastly I started up the new SSH shell server that I have been preparing for about a month. It has the same hostname and ssh-keys etc as the old triton server and I tried to replicate the environment as well so hopefully it’s not totally strange. It’s running Ubuntu linux instead of Gentoo as I mentioned in the previous post. Here I had actually misconfigured the primary IP-address and when I went to change it I messed up the NFS mounts which make things very weird I had a hard time ever shutting the server off because of hanging processes. Eventually I got back using the correct IP and I sent a message on twitter at 20:48 letting people know it was possible to log back in again. I took a well deserved break and had a a fancy hipster beer and watched some Netflix to relax 🙂

As far as I know things are mostly working fine on triton now, but there has been reports of some weird color garbage on the terminal after detaching from gnu screen (time to change to tmux?). I haven’t able to figure this one out so if you know what’s causing it please message me. Also two users had problems this morning that their /home got unmounted so I had to manually remount it. I’m not sure what was causing this but I’ll keep an eye on it. Other than that it’s mostly been some missing packages etc that I have been installing as we go.

There’s still a lot more to do before the move is complete but so far I’m very happy with things and I’m a bit less worried about the old server breaking down. Next on the agenda for me is getting blinkenbot and signup back up and running. There’s also work to get IPv6 back, and also lots of work on the back end infrastructure. Please let me know if you want to read more about the new setup, I’m thinking I should write more about how the final setup looks like now (or when it’s more finished).

User websites has been a part of Blinkenshell since the very early days, and the format has stayed pretty much the same. However, the web has evolved a lot and nowadays many visitors expect HTTPS by default. Because of this I’ve setup a Let’s Encrypt wildcard certificate for *.u.blinkenshell.org and I hope to migrate over user websites there.

For now, any new users signing up will automatically get the new domain name <username>.u.blinkenshell.org and HTTPS. Existing accounts still keep their <username>.blinkenshell.org and no HTTPS for the time being, but contact me if you want to migrate over to the new domain name and HTTPS. I can also set up a redirect from the old domain name so external links will not break when making the move.

On another note, Blinkenshell is still running PHP 5.x but will have to migrate over to PHP 7.x very shortly so if you have some old code running make sure it’s compatible as soon as possible!