Hello everyone! Wow what a stressful past few days it has been. PaperDemon was down for the past 3 days straight. But we are finally back online!
For security reasons I can't go into detail on what happened but here's the gist of it.
On April 1st, we deployed a new version of the paperdemon web application but it wasn't able to build properly. This created an outage of the website that lasted approximately 3 hours. I was able to put a bandaid solution on it to get it to build and run in the short term. It was determined that in order to fix this instability in the long term, we needed to do some maintenance on the server.
I contacted my web host and asked them to begin the maintenance as soon as possible to avoid further outages upon deployment. They proceeded with the maintenance. Unfortunately a series of various things went wrong with this maintenance work causing misconfigurations of the server and causing the website to be down for approximately 64 hours. It was not something I could fix by myself because sys admin work is not my strength. It's the web hosts responsibility to maintain the infrastructure and basic functions of the server. I tried my best to simplify things with the web application to make it easier for them to see the problems and troubleshoot the issues. After a lot of back and forth they were able to fix the problems on their end. Then, I was able to get the web application back online.
These past few days have been incredibly stressful. Having the website down just created this awful feeling of despair and pain in the pit of my stomach. It hurts our community to be down that long. And it's difficult when you can't really do anything but send emails and wait.
As for what I can do to avoid such a situation in the future, probably the best solution is to change to more modern containerized cloud based infrastructure. But the work involved to transition to that is not trivial and would put more responsibilities on my shoulders to maintain and manage configurations of the servers. I will spend some time to review and consider if we're going to spend time doing that soon or not. It would mean we'd have to put a lot of other things on hold.
Thank you all for your understanding and for staying with us through all of this!
There's still a little bit more maintenance we need to do tomorrow which might result in the site being down for another hour. But for now, I need to go to bed. I have not slept well these past few days due to the stress.
If you notice any bugs or things not working properly, please report them in the help forum, help section of the discord, or email [email protected]