As you may have noticed, the ES Designs Manila server became extremely flaky last week. The server was hanging in a way that defied all my server monitoring and restarting processes. Towards the end of the week, the server was hanging without crashing at seemingly random intervals (that averaged around 30 minutes). I was getting alerts at this frequency 24/7, which made sleep difficult.

After working on the server for two days, I think that I have it back to normal. The problems that I was having were the result of an influx of comment spam on a system that was already bloated with it. I have managed to remove a large amount of the spam from the system and implemented some changes to the server that should make it more difficult for it to accumulate and easier to remove completely when it does get through.

During the process, I turned off Manila's ability to send out email. Handling the comment spam and sending out email confirmation with every new post was too much a burden on the servers resources. I will turn the mail back on tomorrow if the server continues to run smoothly (and allows me to get some sleep).

I apologize for the problems and hope that the steps I am taking will limit the impact of comment spam in the future.