Single customer triggered bug that caused global internet outage: Fastly
On June 8, a global internet outage took the world by surprise at around 3:00pm Indian Standard Time. The outage affected many prominent news outlets and social media platforms including CNN, The Verge, The New York Times, The Guardian, and Reddit. According to Fastly, the cloud computing service that faced the underlying issue, an unnamed single customer triggered a bug that caused the outage.
Fastly's outage impacted client websites for almost an hour
The origin server and cache servers that constitute the underlying Content Delivery Network (CDN) for the affected websites are operated by the company Fastly. The outage that lasted around an hour had an impact on approximately 80% of Fastly's servers and even impacted British Government websites. When services resumed, performance was impacted due to the high load on the origin servers.
Customer's configuration change contained specific circumstances that triggered the bug
In a blogpost, Fastly's head of engineering and infrastructure, Nick Rockwell, said that a bug had been introduced in Fastly's code on May 12. It lay dormant until an unwary customer applied a "valid configuration change" on June 8 that "included the specific circumstances that triggered the bug," causing 85% of Fastly's CDN to return errors. Fastly has not yet named the customer.
Fastly blog post apologized to customers for the outage
Rockwell said that even though there were specific conditions that triggered the outage, Fastly should have anticipated it. He agreed that the company provides many reputed clients with mission-critical services. Rockwell thanked the community for support and apologized to customers for the outage. The Guardian reported that Fastly's CDN is one of the largest on the internet, rivaling Amazon Web Services and Cloudflare's offerings.
Outage highlights dangers of few CDNs handling almost all traffic
The recent outage certainly highlighted the risks of just a few CDNs handling enormous volumes of internet traffic. Moreover, it highlighted how damaging bugs can be, especially the ones in critical code that could be inadvertently triggered. It sends chills down one's spine thinking of what could happen if bad actors leveraged similar bugs in code belonging to major CDN providers.
