The #DDOS of SEOAndy.net and What We Are Doing
Update 17/Aug/14 - 10% Rate Attack - http://www.seoandy.net/maintenance/ ===
Even now a day after it's peak, as I write this I am keenly aware that there is an on going DDOS (distributed denial of service) Attack against SEOAndy.net - so whilst I will attempt to fill you in regarding everything I know, please note that things are changing fairly quickly as more data becomes available and analysed.
Background of this attack
Taken from an update earlier (4/8/14): Over the past few weeks SEOAndy has been the subject of a number of direct Denial of Service Attacks. These attacks are nothing new for the site which has been secured from such attacks for a number of years, and this was upgraded just a few months ago. However over the past few weeks we’ve had a larger spate than usual, last weekend and today though we’ve seen the DOS attacks become Distrubted (DDOS).
The one aim of such attacks is to prevent users, such as yourself, from reaching the website – and where possible to force access to the administration system for various means.
At this point we want to assure you that the data we do collect about you, our friends, your email has not been comprimised due to security systems within the website. We also don’t store more than 100 emails within the SEOAndy website also for this reason – ultimately, We care about you and your data.
Further to this: We did also hold email addresses along side comments and names of those who make a comment - this information for a number of weeks has been within the Disqus commenting system and not held here on the website.
What We Are Doing:
Expanding on previous updates: In total we’ve had a few hours down time over the past 2 weeks, this for us is a huge amount. We are used to an hour maybe once every 6 months for various reasons, but the 3 or 4 hours we’ve had recently is big for us and a concern. (We are currently working to check exactly the length of our downtime due to this specific attack).
Over the past few months we’ve implemented the following steps:
- CloudLinux The CL system reinforces an already strong and secure platform of Cpanel/Apache. CloudLinux means we can limit resources based on an account and direct resources as required. This ultimately means that each account cannot "overrun" it's resources and thus it should not bring other websites down on the same server. This prevents issues across the server as well as ensuring that any large attacks are mitigated on contact. It does however, often, mean that when high levels of genuine traffic are met, the site breaks - thus we constantly "twiddle" with settings for the best performance all round.
- Hardened the WordPress Install We’ve continued out process of reducing the surface attack area of wordpress (which can be fairly large due to the size of the site). This will be on going work to prevent future possible attacks.
- Changed the Firewall / Proxy We have now implemented a new firewall system. The new firewall is special in that it is built for the purpose of protecting wordpress from DOS (and other) attacks. This firewall came online an around 6pm (officially) and has so far blocked 200 unique IP’s from requesting files which are commonly part of a DOS and should only be used by administrators. These are IP's which continue to request files and don't "disappear" after 1 or 100 attempts. I am currently using CloudProxy from Sucuri - an awesome tool set these guys have including a website malware scanner for free!
Over the coming period I will post more about hardening wordpress for security, using a proxy for dns and using a cdn for protection and speed. There is too much to say on each to include them in this single post.
The Current State of Play
Right now we are still looking through our logs to find the exact origins of this attack. However our preliminary data and that from our new proxy means that we do have some data - it all correlates with each other, so we're pretty sure its right (but we will keep digging to find the root of the issue).
To the right is the 6 hours tracked yesterday and how request which were blocked came through.
It's important to note that initally a IP not whitelisted will not be blocked until it attempts something malicious at which point it will.
Interesting this data varies from our other data a little from the 10% showing as XMLRPC attempts, which wer know was a huge part of previous attacks.
As of writing this (5 aug 14 - 7am) 10% of requests have this morning been blocked to the site. Ultimately this indicates the attack is on going, though at a somewhat vastly reduced rate. However, the website has remained solid and live since the roll-out of the new proxy.
Due to the on going attack and the number of times SEOAndy has been hit by DOS attacks over the last year, it is evident that we are a target of attacks - and not a victim. This basically means that we are being hit on purpose and not simply as a mistake by a passing bot trying to hit another target. This makes our work a little more tricky to ensure with one loophole closing we do not open another.
What Next for SEOAndy?
Today the future of SEOAndy remains clouded. Ultimately SEOAndy for a long time has not broke-even, and it's something I never minded before, but costs are now spiralling and as such I will for a while be considering my options. If you've any ideas on how we can ensure the future of this website and resource please do get in touch, comments, tweets, emails etc - you know where I am.
The instant future is that I believe SEOAndy for now is safe from being taken offline by another DOS, security is being tightened across the board to ensure this.
Finally, although we've been dealing with DOS and DDOS attacks for a while, if you've any tips to offer on mitigating and preventing attacks please use the comments below.
A special thanks must go to a number of people and companies; my rockstar hosting company SW Broadband for their stellar customer service and solid hosting performance, CloudProxy for helping get the service rolling asap (also Hover domains for helping push through changes to domain DNS) and of course my friends who have supported me ensuring I don't yet have a breakdown over this, this includes those on our email newsletter list who have offered many kind words.
In case of outage repeated at http://pastebin.com/nf2hpfd2