Happy 2013

Posted on January 5, 2013 by independence

It’s been a while since I wrote anything here, but I thought I might summarize the past year and tell you a bit about what’s been happening behind the scenes at Blinkenshell. The service is much the same, but there has been some changes that you might not be aware of.

In August I moved from Linköping to Stockholm, and the main server moved with me. There was quite a long downtime here, but everything went well overall. I’ve also started a new job in Stockholm, so I’ve not put very much time into Blinkenshell this autumn.

In November when I was just about to start my one week vacation to go to Dreamhack and help build the network infrastructure, something happened with the server I use to tunnel IPv4 traffic. This was ofcourse on a friday, so the service provided did not respond until monday. It tunrned out that the server had rebooted, and the harddrive was not ok. They helped me install a new server the next week I tried to recreate the configuration from old backups and some from memory. Everything did come back up online, but it was a long downtime of several days.

Everything seemed to work fine for some weeks, but on the 21st of December the new server I had been moved to stopped responding. Since this was also on a Friday, and there was Christmas holidays after that, the service provider has still not responded to me about why the server went down. A friend of mine helped me set up a new server for tunnelling the IPv4 traffic now, so everything is back online. This was however an even longer downtime than the one in November.

The new server means we do have a new IPv4 address, and some IRC networks has not updated their I-lines yet so we’ve had issues connecting to Freenode and IRCnet for a couple of days for example. I do however think this new server should help improve response times since it has better connectivity in general.

I believe the new server will be more stable then the old was was this last autumn, but I can’t guarantee anything as usual. I hope you will stick around and see what happens in 2013, and a big welcome to all the new members that continue to join this great community! 🙂

Posted in Uncategorized | 3 Comments

Mosh – Mobile Shell

Posted on August 19, 2012 by independence

If you’re a mobile SSH user, you might want to take a look at Mosh

You can run mosh on Blinkenshell only if you have a supporter account (since it requires an open UDP port).

Posted in software | 1 Comment

Maintanance downtime 11 August

Posted on August 10, 2012 by independence

Tomorrow, Saturday the 11th of August, I’m going to take down all Blinkenshell services to move the physical server to a new location. I’m going to shut the server down at around 14.00 CET, and it’s probably going to be offline until late evening. I’m also going to install some new harddrives and perform the regular software updates etc. I’ll try to keep you updated on twitter as the maintanance is going along.

Posted in maintenance | Leave a comment

Harddrive replacement downtime

Posted on August 14, 2011 by independence

I got an email this morning saying one of the drives in the raid-z array had failed, so I’m going to have to replace that drive. Unfortunately Blinkenshell does not have some cool enterprise server with hot-swap drives so this means I’ll have to take the server down.

I’m planning on taking the server down this Wednesday the 17th, the downtime is planned from 18.00 CET to 20.00 CET.

There has also been some problems with processes getting killed by the OOM-killer for no apparent reason (blinkenbot for example) the last couple of days, so I’m going to try upgrading to a newer kernel.

Update1: The server is back up, everything looks good so far!

Posted in maintenance | 3 Comments

5 Year Anniversary!

Posted on August 8, 2011 by independence

Blinkenshell turns five years old this week, the domain name was registered on the 10th of August 2006. It’s a pretty long time for a shell provider, and I think Blinkenshell is better than ever. The community is slowly and steadily growing bigger and stronger, and we’ve just had some nice server upgrades too.

I’m celebrating by offering the 12 months Supporter subscription for 200 SEK (previously 300 SEK) if you get it before the 14th of August. I’ve also (permanently) raised the RSS memory limit from 48M to 64M on all supporter accounts (it’s 32M on regular accounts). So if you’ve been thinking of showing your support by getting a supporter account, now is the time to get one! 😉

I’m very happy to see that the community is also active when I’m not there. I’ve tried to build things in such a way that the important parts can be run by the community if I’m away. The signup process is fully automated, and it’s regulating itself and encouraging new users to participate in the community. You can vote to kick users from the IRC-channel, and there are resource limits in place so one program should not be able to bring the server down. And it actually works very well, I can be away on vacation for a week and everything seems to be working just fine. There is very little abuse, and people seems to be getting along in the IRC channel too. Thanks to all of you in this wonderful community for making this possible! I guess I would have quit a long time ago like many of the other free shell providers if it wasn’t for the great community. You can rest assured I will keep Blinkenshell running, atleast another 5 years if it’s up to me 🙂

Posted in Uncategorized | 3 Comments

Server maintanance done

Posted on August 7, 2011 by independence

The server maintanance is now done, and it took even longer than I guessed. 10+ hours of software and hardware upgrades, and battling with software bugs. And there will probably be some issues left to resolve in the comming days. 🙂

Please report bugs or strange behavious to me and I’ll take a look at it as soon as I can!

Here is the story of my day with upgrading the server, I ran into quite a few problems:

The hardware actually worked as expected, everything started up on the first try and all the memory was ok. Not even any dead harddrives! The software was a bit worse though. First up was ESXi upgrades, I had downloaded all the required patches the day before, and was ready to start. Or so I thought. It turned out I had downloaded patches for ESX instead of ESXi, doh!

Not a huge problem though, downloading the the first set of patches went in a couple of minutes. Today however, VMWares support site was “temporarily down for maintanance” almost the entire day. I managed to get two of the three required patches (4.1 U1 + 2 patches) by googling for the direct link instead of going through the patch search tool, but I could not find the last patch. I had spent quite a few hours on this already, so I felt I had to start making some progress. Installing the patches went much more smoothly when they were for the correct OS! (and after I figured out you can’t use vihostupdate.pl from behind NAT apparently)

I started booting up the virtual machines, and everything looked good. The main SSH server Triton would not quite boot properly though, but I could get a prompt halfway through the init process. I started looking around for anything odd, maybe some package which was not of the correct version. When doing an update of the package list the machine froze, and actually the entire ESXi host died with a “Purple screen of death”. After another PSOD or two I figured out it had something to do with network traffic, and google turned turned up something about a bug affecting ESXi 4.1 U1 on VMs with multiple CPUs and the vmxnet3 driver. It turned out this bug had been fixed in the patch which I could not find from before, frustrating! Many hours had passed by this time, and there was maybe an hour left until I was supposed to be finished with everything. Thankfully I managed to google the direct link for the last patch also (VMwares site still not working), and after the patch eveything seemed to be working fine.

I also hade some issues with upgrading NexentaStor, downloading the patches from the nexentastor shell did not work properly. I managed to manually the updates though, and then run the upgrade-thing from within the nexentastor shell. The second part of the upgrade was supposed to be from version 3.0.5 -> 3.1.0, but this did not work very well either. It turned out after some more googling that the had actually pulled version 3.1.0 from the servers, but not put up any message on the main site or anything that would give a hint when you tried upgrading from the CLI. Somewhere in the forums there was a post saying they were working on some distribution bug, so I could not get this upgrade at all.

Next up was a bug in the new Gentoo init system baselayout2/openrc, it only affected me because I did not use a pretty recent kernel option that apparently impoves boot performance or something like that (CONFIG_DEVTMPFS). I didn’t get much of a clue from the software this time either, it just stopped booting about halfway through the init process and froze. Much googling and a kernel compilation later it actually seemed like I was pretty close to finished, only a couple of hours after the deadline I had set 😛

Then I had to fight a bit with grsecurity and the autostart-scripts (root does not have read-permission in users home directories, but that’s where the autostart scripts are stored, messy…), but about 20.30 I could open up for logins to the main SSH server again!

I hope there will be atleast a couple of years until I have to take down the server and do software maintanance like that again, puh. Sorry for the longest blog post I’ve ever written, but maybe someone will find it interesting 😛

Posted in downtime, maintenance, Uncategorized | 2 Comments

Server maintanance 7th of August

Posted on August 2, 2011 by independence

As I said in my previous post, I’m going to take down Blinkenshell to upgrade the RAM and do the regular round of software upgrades. I’m going to take the server down at around 10.00 AM CET this Sunday (August 7). I plan to have everything up and running again by 18.00 CET. All services are going to be affected; mail, web and shell.

Please save and exit before I shut down the server. If you haven’t configured your irssi/whatever to autoconnect to all your favourite networks and channels, now might be a good time to fix that. Also check out the auto-start feature to automatically start a screen:ed irssi/whatever when the server boots up.

Posted in maintenance | Leave a comment

RAM Upgrade

Posted on July 30, 2011 by independence

So yeah… I ordered 32GB of RAM the other day. The main server is using a bit more than the current 8GB of RAM, so it’s using memory compression and swap and other workarounds at the moment. It’s actually working pretty well considering, but there’s no room for expansion so I finally decided to get some more RAM. Special thanks to everyone who has signed up for supporter accounts and donated! (Nerdy details: I ordered four 8GB DDR3 1333MHz ECC REG quad-rank x8-modules)

I’m going to announce a maintanance stop pretty soon. It’s probably going to be pretty long like 8 hours or so, since it’s also time to upgrade the virtual hypervisor, storage server, shell server etc. The virtual hypervisor and storage server and some parts has been running for almost 300 days without upgrading/reboot so it’s about time now. 🙂

Posted in hardware, maintenance | 1 Comment

IPv6 Day

Posted on June 8, 2011 by independence

Today is the World IPv6 Day, when many parts of the Internet community will try out IPv6 on a larger scale. A lot of websites will be publishing AAAA (IPv6) DNS records for their main addresses, so users with IPv6 will be accessing the websites over IPv6 by default.

Blinkenshell has been available over IPv6 for a while on a special URL, but from today the regular URL http://blinkenshell.org/ is accessible via IPv6 as well. SSH and IRC are also available via IPv6.

If your ISP does not provide native IPv6 yet, you can get a free IPv6-tunnel from Hurricane Electtric for example.

Posted in internet | Leave a comment

Downtime on Triton

Posted on February 20, 2011 by independence

I’m taking Triton down because of some issues with NFS, many processes is hanging. I’m going to be upgrading some software on Triton and other services also, so the downtime will probably be 1-2 hours.

Update 14:24: Making good progress, but still maybe 1 hour left before everything is up and running.

Update 17.51: Everything should be up and running now. Sorry for the delay, I ran into some problems while upgrading the LDAP server.

Posted in downtime | Leave a comment