The downtime is over and we should never be down that long again. We now have excellent backup and restore scripts that are compatible with the new infrastructure and should be compatible with any infrastructure in the future.
There will be a nontrivial amount of downtime next weekend as we migrate to a new server.
The server has to be shut down to avoid any database updates while the database is being backed up, and the entire backup process takes about 3 hours (the new server is faster).
Then a redeploy, restore data, and we should be good to go. Old server will stay around in case things go horribly wrong.
Under 5 minutes of downtime. Not bad. That concludes the maintenance for today. There will be more in the foreseeable as we make some serious upgrades to our infrastructure. The new setup is 🔥 🤓
Our server is taking some time off today. If it turns out to take too long, the original will be restored. There is always a contingency plan. #sysadminlife
Took a little longer than expected, but we're back.
Upgrade complete. It went reasonably smooth and improvements were made so it will go even better next time.
Your lovely sysadmin may or may not have forgotten to restore the back last Tuesday. They may have also forgotten that the [old] server was off and that they should hop to it ASAP. What we can be sure of is that the sysadmin did migrate everything to the new server! This means the backup scripts work reasonable well and things should be more stable now. Happy Sunday!
We apologize for our extended downtime (a bit over a week, I think). We had an unexpected 15-hour power outage at our data center, and our recovery processes still need some work. We have updated our processes and there is even drastic talk of trying them BEFORE the next unplanned incident. Stay tuned.
Mostly hackers, mostly in Urbana, IL, talking to each other & our friends on like-minded servers without giving our personal data to the marketing machine.