Maintanance downtime 11 August

Tomorrow, Saturday the 11th of August, I’m going to take down all Blinkenshell services to move the physical server to a new location. I’m going to shut the server down at around 14.00 CET, and it’s probably going to be offline until late evening. I’m also going to install some new harddrives and perform the regular software updates etc. I’ll try to keep you updated on twitter as the maintanance is going along.

Posted in maintenance | Leave a comment

Harddrive replacement downtime

I got an email this morning saying one of the drives in the raid-z array had failed, so I’m going to have to replace that drive. Unfortunately Blinkenshell does not have some cool enterprise server with hot-swap drives so this means I’ll have to take the server down.

I’m planning on taking the server down this Wednesday the 17th, the downtime is planned from 18.00 CET to 20.00 CET.

There has also been some problems with processes getting killed by the OOM-killer for no apparent reason (blinkenbot for example) the last couple of days, so I’m going to try upgrading to a newer kernel.

Update1: The server is back up, everything looks good so far!

Posted in maintenance | 3 Comments

5 Year Anniversary!

Blinkenshell turns five years old this week, the domain name was registered on the 10th of August 2006. It’s a pretty long time for a shell provider, and I think Blinkenshell is better than ever. The community is slowly and steadily growing bigger and stronger, and we’ve just had some nice server upgrades too.

I’m celebrating by offering the 12 months Supporter subscription for 200 SEK (previously 300 SEK) if you get it before the 14th of August. I’ve also (permanently) raised the RSS memory limit from 48M to 64M on all supporter accounts (it’s 32M on regular accounts). So if you’ve been thinking of showing your support by getting a supporter account, now is the time to get one! 😉

I’m very happy to see that the community is also active when I’m not there. I’ve tried to build things in such a way that the important parts can be run by the community if I’m away. The signup process is fully automated, and it’s regulating itself and encouraging new users to participate in the community. You can vote to kick users from the IRC-channel, and there are resource limits in place so one program should not be able to bring the server down. And it actually works very well, I can be away on vacation for a week and everything seems to be working just fine. There is very little abuse, and people seems to be getting along in the IRC channel too. Thanks to all of you in this wonderful community for making this possible! I guess I would have quit a long time ago like many of the other free shell providers if it wasn’t for the great community. You can rest assured I will keep Blinkenshell running, atleast another 5 years if it’s up to me 🙂

Posted in Uncategorized | 3 Comments

Server maintanance done

The server maintanance is now done, and it took even longer than I guessed. 10+ hours of software and hardware upgrades, and battling with software bugs. And there will probably be some issues left to resolve in the comming days. 🙂

Please report bugs or strange behavious to me and I’ll take a look at it as soon as I can!

Here is the story of my day with upgrading the server, I ran into quite a few problems:

The hardware actually worked as expected, everything started up on the first try and all the memory was ok. Not even any dead harddrives! The software was a bit worse though. First up was ESXi upgrades, I had downloaded all the required patches the day before, and was ready to start. Or so I thought. It turned out I had downloaded patches for ESX instead of ESXi, doh!

Not a huge problem though, downloading the the first set of patches went in a couple of minutes. Today however, VMWares support site was “temporarily down for maintanance” almost the entire day. I managed to get two of the three required patches (4.1 U1 + 2 patches) by googling for the direct link instead of going through the patch search tool, but I could not find the last patch. I had spent quite a few hours on this already, so I felt I had to start making some progress. Installing the patches went much more smoothly when they were for the correct OS! (and after I figured out you can’t use from behind NAT apparently)

I started booting up the virtual machines, and everything looked good. The main SSH server Triton would not quite boot properly though, but I could get a prompt halfway through the init process. I started looking around for anything odd, maybe some package which was not of the correct version. When doing an update of the package list the machine froze, and actually the entire ESXi host died with a “Purple screen of death”. After another PSOD or two I figured out it had something to do with network traffic, and google turned turned up something about a bug affecting ESXi 4.1 U1 on VMs with multiple CPUs and the vmxnet3 driver. It turned out this bug had been fixed in the patch which I could not find from before, frustrating! Many hours had passed by this time, and there was maybe an hour left until I was supposed to be finished with everything. Thankfully I managed to google the direct link for the last patch also (VMwares site still not working), and after the patch eveything seemed to be working fine.

I also hade some issues with upgrading NexentaStor, downloading the patches from the nexentastor shell did not work properly. I managed to manually the updates though, and then run the upgrade-thing from within the nexentastor shell. The second part of the upgrade was supposed to be from version 3.0.5 -> 3.1.0, but this did not work very well either. It turned out after some more googling that the had actually pulled version 3.1.0 from the servers, but not put up any message on the main site or anything that would give a hint when you tried upgrading from the CLI. Somewhere in the forums there was a post saying they were working on some distribution bug, so I could not get this upgrade at all.

Next up was a bug in the new Gentoo init system baselayout2/openrc, it only affected me because I did not use a pretty recent kernel option that apparently impoves boot performance or something like that (CONFIG_DEVTMPFS). I didn’t get much of a clue from the software this time either, it just stopped booting about halfway through the init process and froze. Much googling and a kernel compilation later it actually seemed like I was pretty close to finished, only a couple of hours after the deadline I had set 😛

Then I had to fight a bit with grsecurity and the autostart-scripts (root does not have read-permission in users home directories, but that’s where the autostart scripts are stored, messy…), but about 20.30 I could open up for logins to the main SSH server again!

I hope there will be atleast a couple of years until I have to take down the server and do software maintanance like that again, puh. Sorry for the longest blog post I’ve ever written, but maybe someone will find it interesting 😛

Posted in downtime, maintenance, Uncategorized | 2 Comments

Server maintanance 7th of August

As I said in my previous post, I’m going to take down Blinkenshell to upgrade the RAM and do the regular round of software upgrades. I’m going to take the server down at around 10.00 AM CET this Sunday (August 7). I plan to have everything up and running again by 18.00 CET. All services are going to be affected; mail, web and shell.

Please save and exit before I shut down the server. If you haven’t configured your irssi/whatever to autoconnect to all your favourite networks and channels, now might be a good time to fix that. Also check out the auto-start feature to automatically start a screen:ed irssi/whatever when the server boots up.

Posted in maintenance | Leave a comment

RAM Upgrade

So yeah… I ordered 32GB of RAM the other day. The main server is using a bit more than the current 8GB of RAM, so it’s using memory compression and swap and other workarounds at the moment. It’s actually working pretty well considering, but there’s no room for expansion so I finally decided to get some more RAM. Special thanks to everyone who has signed up for supporter accounts and donated! (Nerdy details: I ordered four 8GB DDR3 1333MHz ECC REG quad-rank x8-modules)

I’m going to announce a maintanance stop pretty soon. It’s probably going to be pretty long like 8 hours or so, since it’s also time to upgrade the virtual hypervisor, storage server, shell server etc. The virtual hypervisor and storage server and some parts has been running for almost 300 days without upgrading/reboot so it’s about time now. 🙂

Posted in hardware, maintenance | 1 Comment

IPv6 Day

Today is the World IPv6 Day, when many parts of the Internet community will try out IPv6 on a larger scale. A lot of websites will be publishing AAAA (IPv6) DNS records for their main addresses, so users with IPv6 will be accessing the websites over IPv6 by default.

Blinkenshell has been available over IPv6 for a while on a special URL, but from today the regular URL is accessible via IPv6 as well. SSH and IRC are also available via IPv6.

If your ISP does not provide native IPv6 yet, you can get a free IPv6-tunnel from Hurricane Electtric for example.

Posted in internet | Leave a comment

Downtime on Triton

I’m taking Triton down because of some issues with NFS, many processes is hanging. I’m going to be upgrading some software on Triton and other services also, so the downtime will probably be 1-2 hours.

Update 14:24: Making good progress, but still maybe 1 hour left before everything is up and running.

Update 17.51: Everything should be up and running now. Sorry for the delay, I ran into some problems while upgrading the LDAP server.

Posted in downtime | Leave a comment

Pallas and SSL Update

I’m going to remove the Pallas SSH server since it’s not used very much, only a couple of users are logging in. I guess this means people are not bothered very much by any delays to Sweden. It was nice as a backup service when Triton was down, but it’s a bit to expensive for just that.

Instead, I’m going to spend some money on a “real” SSL certificate. The new certificate for HTTPS etc should be accepted by default in all major desktop browsers. You can browse this blog, the wiki, the webmail and phpmyadmin with the new SSL certificate.

Update 2011-02-11: Pallas is now offline!

Posted in services, web | Leave a comment

Snapshots and Wiki updates

I’ve enabled a feature of ZFS called snapshots for all supporter accounts. This allows for you to recover deleted or changed files from the last 7 days. More details on the Wiki.

I’ve also updated many other pages on the wiki to reflect some changes and to clearify. There is also a new howto on Process Management in a Linux shell, with some details about memory management on Blinkenshell.

Posted in services, shell, web | Leave a comment