If you’re interested in some statistics for Blinkenshell check out this page: http://blinkenshell.org/stats/
These graphs are generated with the excellent Munin monitoring/graphing tool. If you think there are any other graphs that might be interesting for users let me know.
As you might have noticed there was some issues with the server yesterday. The website was down most of the day, and some people had problems logging in via SSH. This was related to an issue with the fileserver. There has actually been some issues with performance also (freeze for a couple of minutes at 01.30 CET). I planning to make a scheduled stop to fix this performance issue at some point, but because of the problems yesterday I had to fix it right away. All services was down between 19.00 and 00.30 CET yesterday because of this maintanance. The maintenance went well, and everything should be up and running as usual now.
I did expect there might be some problems with the new server after the migration, but this problem showed up a bit later than I would have thought. The server had been running well for about three weeks before this happened, but I guess that’s how it is sometimes 🙂
The performance issues was related to a new feature in ZFS called deduplication. It’s been around in the enterprise space for quite a while, but it’s very new in ZFS. I had deduplication enabled for the VM datastore for a while when testing the server this summer, but disabled it after a couple of days. I did however not realize that the data that had already been deduplicated would be kept in that state after I disabled deduplication on the filesystem. This caused the kernel to keep a huge deduplication table in memory which in turn caused other problems when the load on the server increased. I had to move all data away from the filesystem, destroy the entire ZFS pool, create a new one and copy the data back. That’s why the stop was quite long 🙂
The main SSH server Triton rebooted today at about 14.15 CET. The server got a kernel panic after I reloaded the grsec policy. I’ve probably reloaded the grsec policy hundreds of times before, so I’m guessing this bug is pretty rare. There is currently no updates to the stable kernel, so if it’s a bug in the kernel we’ll have to wait until the fix gets there. In the meantime I’ll be more careful with reloading the grsec policy 😉
Other servers like mail and web were not affected. Triton was back online at about 14.18 CET. Sorry for the inconvenience!
While migrating to the new server I took the opportunity to make some changes to the signup system here on Blinkenshell. The biggest change is in terms of time and code rewritten is probably the move from cPickle datastore backend to a MySQL one. You might not notice very much of this change at first, but there are some subtle improvements. You don’t have to wait until someone logs out from the signup program to vouch for them for example.
The next big thing is that users are automatically added to the system when they reach step 6. No more waiting for Independence to come online (unless you send Paypal donations, they still have to be registered manually). Because of this it’s even more important that the vouch step is done right, and therefore I’ve added a feature that has been suggested a couple of times before: “negative vouches”. This is only available from the signup program at the moment (not via Blinkenbot). After you choose someone to vouch for, you can make the vouch positive (+1 p) or negative (-2 p). A user has to reach +2 p to get to the next step.
I’ve also updated most of the questions on the quiz to make them more relevant and up to date. If you want you can retake the quiz in the signup program to see if you still got it 😉
When the server migration was done a couple of days back I wrote about a new type of “Supporter” account that would be introduced, and today I’m announcing that offer.
It will be a new type of account with more features and larger resources, but hosted on the same server and with the same basic rules (with one exception: bots and bouncers are allowed). The main thing with this account is a larger quota (1000MB) and the ability to host dynamic web pages (web pages with CGI/PHP, MySQL db).
The price for this account will start from 25 SEK per month for a 12 month subscription. There is no setup fee, and you can get a full refund within 14 days if you are not satisfied with the service. Payment is done via PayPal (or creditcard via Paypal).
I hope this offer will complement the free account in a good way for people who feel they have some money to spare, or need the extra features and higher resource limits. I know the dynamic webpage-feature was “stolen” away from the free account, but I hope you understand and don’t think I’m too much of a sell out 😉
More details about the new account: Supporter account
Today it’s 4 years since the blinkenshell.org domain name was registered and this project really started to take form. I tried to get the server migration done before today, and it is almost done. I’ve been working on migrating signup and blinkenbot tonight, and that’s mostly done too. There are of course lots of bugs and quirks still left to work out, but the big parts are in place now. I’m very happy with the new server, and I hope you will be too 🙂 I will write more details about what has changed with the signup and webhosting soon. But until then, happy 4 year anniversary everyone!
Ok, so all user accounts was migrated yesterday. There was some problems with the temporary passwords for some users. If your temp password does not work, contact me on IRC or via email and I’ll generate a new one.
There was also a problem where users could not chmod public_html to enable their websites. This should be fixed now.
Another problem that I’ve heard a lot about is “ERROR Closing Link: user[free.blinkenshell.org] (Too many connections from your IP)” when connecting to BlinkenIRC. This is because mimas does not have a trust for connections from Triton. Instead, you should use the new local IRCd; connect to sao or sao.blinkenshell.local instead of mimas and everything will work as usual.
Random bits and pieces: IPv6 is not enabled yet. Some IRC networks has not updated their trusts yet, we’re still waiting for freenode. If you want to connect to IRCnet, please try irc.swipnet.se. Bitlbee should be working now. The sv_SE locale does not exist anymore, please use en_US.UTF8 (you can change that in your ~/.profile file).
Webmail is available at https://webmail.blinkenshell.org/
Signups are still disabled, and Blinkenbot is still offline. I’ll start working on that tomorrow.
If you have any other problems ask in the IRC channel or email me at independence at blinkenlights.se.
The server migration is taking a bit longer than expected, but I hope it’s going to get done tonight. While the scripts are doing their work I’m going to write a bit about the changes that will be introduced with this new server.
I’ve been working with the new server since about December of 2009, so about eight months. The project was a bit larger than I imagined at first, but now I think I’ve got all the important pieces the way I want them. Everything is not 100% finished, and I’m sure you are going to find lots of bugs and unexpected behavior. But most of this will be sorted out in the next couple of weeks. I felt I had to do the migration at some point even if everything was not 100% complete since it would have taken me maybe another year to get everything exactly right 😉 But here we are, I hope it all goes well. Now something about the new configuration.
I’ve posted about the new server hardware before, and a bit about the general software pieces. It’s running VMWare ESXi with a bunch of virtual machines, the one you will probably use the most is Triton. This server will replace titan as the main SSH server where you run all irssi processes etc. There will also be a file server, a firewall, a mail server, a web server, a domain controller and so on. They’re all running a bunch of different OSes, but I will be sticking to Gentoo on the SSH server Triton.
Every user will have a ZFS filesystem on the file server with 100MB quota. This is shared for the normal home directory, public_html/website and email account. It’s also compressed on the fly, so if you save lots of textfiles or logs you can fit a whole lot more than 100MB in there. I think this is a great improvement.
The SSH server Triton is using more features from Grsecurity, most noticeably the RBAC system. This is an added security layer that will help me sleep better at night 😉 The RBAC policy might need some work, if some commands fail with permission denied etc it might be something I forgot to put in the policy. This will get better in the next couple of weeks as users start using the server and reporting bugs to me.
Another thing I’ve decided to change is the website hosting. I was not completely satisfied with the old solution, it was hard to use with dynamic scripts because of the reverse proxy, and the performance was quite bad. I’ve decided to skip the reverse proxy and I’m using another method to execute CGI (and PHP) scripts. However, this new solution does require much more resources on the server. Because of this I’ve decided to make the web hosting with dynamic scripts an optional part that is only available on “supporter” accounts. These accounts will require some payment, more on that later. Websites with static content will not be affected. I will allow users who has previously had a website with dynamic content to continue hosting them at no cost, email me and I’ll fix it. To not expose scripts with passwords etc I’ve set the permissions of all public_html directories to 700 so world can not read them. Change back to 755 if you want to enable your website.
There are lots of other changes too, I’ll keep posting new entries and updating the wiki. But this is it for tonight I think 🙂
Today at 18.00 CET I’m going to attempt to migrate Blinkenshell to the new server that I’ve been working on. At 18.00, all processes will be killed and all logins will be disabled. Please exit all your running processes (screens etc) before this.
After the migration, you will be able to log in to the old server titan.blinkenshell.org with your old password. Here you will receive a new temporary password that you can use to log in to the new server triton.blinkenshell.org (or ssh.blinkenshell.org). Also on port 2222. Use the temporary password to log in and choose a new password.
The downtime for the migration should hopefully not be more than an hour or two, but it’s a quite complicated operation so it might take longer. If it does not work out at all we might have to go back to the old server and try the migration again at a later date.
More information about the new server and changes will be published later on. Stay tuned.
As you probably have noticed NickServ disappeared from our little IRC network a while back. I’m not sure of the reason behind this, the guy who hosted it hasn’t been online for a long time and doesn’t answer to email. Maybe he’s away or he got bored. Whatever reason, we can’t sit around with no NickServ for too long. No new registrations can be accepted etc, so we had to get something back online, and that’s what I’ve been doing today.
I’ve set up an IRCd on the new server that I bought a couple of months back, and configured new IRC services such as NickServ and ChanServ on there. Around at 16.30 I linked up the new IRCd with services. I had however forgot to configure a session limit exception for Blinkenshell so everyone got G-lined (banned from the network) 🙂 That was quickly fixed, but since it was a G-line and not a disconnect all clients has to reconnect manually. Sorry about that. Other than that, things seems to be working fine now.
I also used this opportunity to change some other things. The network name was previously Allshells, and the network was intended for all sorts of shell providers. That never really took of however and there was only #blinkenshell and a couple of small channels. Since the other IRC operators are gone now, I decided to make this a network dedicated for Blinkenshell only. Thus the new name of our network is BlinkenIRC!
I’m also going to make this a SSL-only network, no remote unencrypted connections will be accepted. Connections from titan are basically going to localhost so there is no need for SSL there. Users who are connecting remotely to the IRC network will however have to make some changes. SSL connections are accepted on port 6697, port 6667 will be closed entierly soon. The SSL certificates are self-signed or signed by CAcert, but if you accept the certificate once and save the fingerprint you should be reasonably safe.
I’m also happy announce that djweezy has contributed with a third IRC server for the network hosted in the US, thanks!
To sum up, we now have two new IRC servers and a new NickServ. The new NickServ database is completely empty so you have to register your nickname again (and verify them in the signup program again). The easiest way to connect is via irc.blinkenshell.org on port 6697 with SSL enabled. More details on the servers and DNS-names is available here: BlinkenIRC
The new shell server is on the way btw, everything will probably be finished within a month or so.
Wiki Changes