The downtime is over and we should never be down that long again. We now have excellent backup and restore scripts that are compatible with the new infrastructure and should be compatible with any infrastructure in the future.
There will be a nontrivial amount of downtime next weekend as we migrate to a new server.
The server has to be shut down to avoid any database updates while the database is being backed up, and the entire backup process takes about 3 hours (the new server is faster).
Then a redeploy, restore data, and we should be good to go. Old server will stay around in case things go horribly wrong.
Under 5 minutes of downtime. Not bad. That concludes the maintenance for today. There will be more in the foreseeable as we make some serious upgrades to our infrastructure. The new setup is 🔥 🤓
Our server is taking some time off today. If it turns out to take too long, the original will be restored. There is always a contingency plan. #sysadminlife
Upgrade complete. It went reasonably smooth and improvements were made so it will go even better next time.
Your lovely sysadmin may or may not have forgotten to restore the back last Tuesday. They may have also forgotten that the [old] server was off and that they should hop to it ASAP. What we can be sure of is that the sysadmin did migrate everything to the new server! This means the backup scripts work reasonable well and things should be more stable now. Happy Sunday!
OpenSSH taking minutes to become available, booting takes half an hour (2018)
https://news.ycombinator.com/item?id=20463251
#hackernews #tech
We apologize for our extended downtime (a bit over a week, I think). We had an unexpected 15-hour power outage at our data center, and our recovery processes still need some work. We have updated our processes and there is even drastic talk of trying them BEFORE the next unplanned incident. Stay tuned.
@admin Downtime turned out to be about 3 minutes to actually do the migration. Everything should be solid now. If you experience any issues, let me know.
@admin backed everything up and then decided to improve (and test) the deployment scripts. Up for now, will go down briefly later today.
@admin Mystery solved! It was an image size limit issue. Now posting up to 20MB files should work. If you want to post something larger, let us know.