admin @admin@hax0rbana.social

Apologies for the recent instability. We should be stable now for a bit, but I am going to have to schedule some maintenance in the near future.

It's been almost a year since we had this level of trouble though, so although downtime is certainly undesirable, we're doing reasonably well.

The downtime is over and we should never be down that long again. We now have excellent backup and restore scripts that are compatible with the new infrastructure and should be compatible with any infrastructure in the future.

There will be a nontrivial amount of downtime next weekend as we migrate to a new server.

The server has to be shut down to avoid any database updates while the database is being backed up, and the entire backup process takes about 3 hours (the new server is faster).

Then a redeploy, restore data, and we should be good to go. Old server will stay around in case things go horribly wrong.

Under 5 minutes of downtime. Not bad. That concludes the maintenance for today. There will be more in the foreseeable as we make some serious upgrades to our infrastructure. The new setup is 🔥 🤓

Our server is taking some time off today. If it turns out to take too long, the original will be restored. There is always a contingency plan. #sysadminlife

Just checked the stats and we have three 9s of uptime. Not bad, all things considered.

Took a little longer than expected, but we're back.

Time for some brief maintenance, brb

Upgrade complete. It went reasonably smooth and improvements were made so it will go even better next time.

Server will undergo some planned maintenance tomorrow morning. Things have been steadily improving and I expect things to go smoothly. Our automation has removed a good deal of human error!

Maintenance is scheduled for tomorrow morning. It should be just a few minutes of downtime, if everything goes according to plan.

Your lovely sysadmin may or may not have forgotten to restore the back last Tuesday. They may have also forgotten that the [old] server was off and that they should hop to it ASAP. What we can be sure of is that the sysadmin did migrate everything to the new server! This means the backup scripts work reasonable well and things should be more stable now. Happy Sunday!

Planned outage on Tuesday while we migrate to a new server. The backup process currently takes 5 hours (which is part of the reason to move to a better server). If successful, this should be the last extended downtime for a while.

Ahh, the unplanned outages. This time it was a couple days of no Internet. We are adopting a more ...aggressive... escalation process.

We apologize for our extended downtime (a bit over a week, I think). We had an unexpected 15-hour power outage at our data center, and our recovery processes still need some work. We have updated our processes and there is even drastic talk of trying them BEFORE the next unplanned incident. Stay tuned.

@admin Downtime turned out to be about 3 minutes to actually do the migration. Everything should be solid now. If you experience any issues, let me know.

@admin backed everything up and then decided to improve (and test) the deployment scripts. Up for now, will go down briefly later today.

Migrating a VM to a new server and expect our instance to be down for about half an hour. Sorry for the late notice.

Apparently not everything came back up after an unexpected power outage, and it took us several days to figure out what was down. *embarrassed face* We think we are visible to the Internet again. Sorry for the outage; we are making plans for monitoring to help us catch down services instantly.