TIMELINE: 14:36 UTC 05/25/2016 to 07:00 UTC 05/26/2016
TIMELINE OF ACTIONS:
- 2016-05-25 14:36 UTC - 5 of our DNS servers started to receive high amounts of traffic which started to saturate the network interfaces
- 2016-05-25 14:41 UTC - We started investigating the issue along with traffic source assessment
- 2016-05-25 14:50 UTC - We started rerouting traffic of the attacked DNS's to other DNS's which highlighted the target points
- 2016-05-25 15:38 UTC - We identified the pattern of DDoS attack which was using domain registered in China and was patterned as malicious behaviour most likely using a botnet with more than 1k unique IP address from across the globe
- 2016-05-25 16:31 UTC - Due to traffic volumes, we created firewall rules to filter-out attackers however, while this rule could be applied on our DNS's, and effectively drop all responses for attacker's requests, the incoming traffic was still able to saturate our interfaces due to a domain registrars unvalidated DNS point.
- 2016-05-25 17:00 UTC - We have reported this problem to the reference domain registrar where the offending domain was registered, as well as attempting to directly contact the domain owner while we continued trying to manage the inbound traffic with rate-limiting on requests.
- 2016-05-25 20:59 UTC - Working with our upstream partners we blocked the attackers and moved two dns servers (dns6.pointhq.com and dns8.pointhq.com) to completely new IP addresses once we had stabilised the interfaces. We have updated all records in PointDNS which were pointing to to the target IP addresses to match the new ones. At this point the impact on systems had been minimised, with service monitoring
- 2016-05-26 07:00 UTC - Post attack, following monitoring of continued traffic filtering, we have begun the process of reverting all routing/IP changes to their original form
- 2016-05-26 08:39 UTC - dns8.pointhq.com IP reverted to original: 18.104.22.168
- 2016-05-26 11:41 UTC - dns9.pointhq.com IP reverted to original: 22.214.171.124
- 2016-05-26 12:10 UTC - dns10.pointhq.com IP reverted to original: 126.96.36.199
- 2016-05-26 14:24 UTC - dns6.pointhq.com IP reverted to original: 188.8.131.52
The following is a summary of actual downtime on each DNS server over the course of the period of the DDoS attack from 14:36 UTC 05/25/2016 to 07:00 UTC 05/26/2016. The times indicated represent a sum total of availability impact for the duration, and do not represent a continuous period.
While this attack was sizeable (the attack took out one of our geo-partners entire infrastructure in that region) and having multiple pan-global sources, the impact to customers was generally low with little effect on service usage levels for majority of PointDNS customers.
However, for a limited number of users of the service exclusively using the affected DNS servers, the impact was greater. And for some of those affected, the issue was being compounded due to some using the IP addresses of the DNS servers instead of the canonical names.
During this period, our response times to tickets was far from ideal, which was also further exacerbated by our status page updates not reflecting the progression of the incident, which was down to human error and communication oversight in an attempt to resolve the issue quicker. This had a knock-on effect of causing customers to be left without appropriate updates to progress to allow them to take decisions to minimise impact on them. Following this, we have added an internal review for DDoS incident management to guarantee better communication levels.