On January 1st this site was taken offline for 12 hours while server upgrades were performed. Upgrades including increasing the amount of installed memory (Web Server 1Gb -> 4Gb & Database Server 1Gb -> 2Gb). I also replaced Hard Drives in both servers as each was showing potential signs of failure. These new drives were Seagate Barracuda 500 Gb Serial ATA drives. I also added new SATA controller card in the Database server. In comparing benchmarks from before and after the upgrades there seems to be a significant improvement in performance which will hopefully be seen here on this site.
Network Outages:
Over the course of the past few weeks, My ISP has been experiencing periodic connectivity issues. They wish to express our apologies for these outages and emphasize that they are doing everything feasible to keep them under control and provide a final resolution. In short, they have determined that the outages are caused by extremely aggressive denial of service attacks on the ISP's network. While they are monitoring 24/7 for the attacks to mitigate the impact, they are also installing additional equipment which will resolve this permanently.
The ISP initially experienced an unexpected network failure where interfaces were resetting causing errors and BGP sessions to fail. With the first occurrence, The ISP contacted it's circuit vendors for testing of the circuits, which resulted in no issues located. We prepared equipment for replacement, but with the sudden onset and almost instant disappearance of the cause, maintenance was not determined as effective at that time.
The issue did not reoccur for almost a week, at which point the same symptoms appeared. With this, maintenance was scheduled and performed replacing the likely hardware. The frequency increased with the next occurrence of network issues being a couple days later, and at this point we became very highly concerned. A sequence of maintenances was performed replacing all equipment in any way involved, and as logging was failing to provide any insights, we began monitoring the network 24/7 manually.
Through this manual monitoring it was determined that extremely targeted denial of service attacks were being performed. They would attack, obtain very rapid results, and then stop the attack just as quickly. However, having now located the cause, we were able to mitigate the attacks before they could cause network damage. Once it was realized that the original attack method had been neutralized, the attacks shifted styles.
The new attack method is one of saturating the network with anywhere from 100 to 200 thousand packets per second. These packets are also designed to cause maximum impact to the routers involved. While the networking equipment in place is able to handle up to ten times the normal load, it was not designed to handle an attack with the sole purpose of taking the network down.
The ISP is monitoring 24/7 to mitigate any attack which comes in, contact our upstream providers to block it from even reaching us, and to maintain network stability. Additionally, they are installing additional equipment which is designed to protect the network from just such an attack. Again, we apologize for the problems that this has caused you.