Those of you who follow our state site probably know we have been migrating all our servers to a new rack cabinet on Sunday. The operation went very smooth and things were back online within an hour.
Sunday around midnight (CET) one of our servers went down shortly. Because of the disk encryption, all services dependent on this machine have been down until one of the admins woke up to see the situation. We have quickly restored the server and everything went back online, but only for short time. After we've put things up online, we sent email to the datacenter asking if it was possible that someone mistakenly pulled the plug on our server (things happen). While we were happy with services back online and updating our social feed and our state site, we noticed that the server has rebooted again. Apparently one of the on-site datacenter engineers went over to the rack, saw that one cable was not plugged all the way in, and he attempted to push the plug. While doing it he noticed the server going down. In order to not cause any more unexpected downtimes, muppeth rushed to the datacenter to see what the issue is. Ready with duct-tape, tie-rips and a hammer (always have a hammer with you, you never know) he proceeded to inspect the situation. Indeed one of the cables was slightly out. Pushing the cable in position seems to be solving the issue. The cable sits now very nicely in the socket and does not seem to be wanting to go anywhere.
We deeply apologize for the downtime and hope your week has started better for you then for us.
We wish ourself and all of you smooth, productive, chill, and happy rest of the week!