Amazon is currently experiencing “high error rates with S3 in US-EAST-1,” causing a massive outage for sites, apps, and services across the web. The AWS service health dashboard was also temporarily affected by the outage. Amazon says it is working at repairing S3 and that they believe they have identified the root cause.
@awscloud Care to share the architecture of this status page as an anti-pattern?
— Thorben Heins (@thorbenheins) February 28, 2017
The outage is affecting many popular sites, such as Quora, Netflix, Splitwise, Business Insider, Giphy, Trello, IFTTT, many publishers’ image hosting, filesharing in Slack, and the Docker Registry Hub.
WordPress businesses are also currently affected, especially those that host customer downloads. WooCommerce customers are currently unable to access downloads they purchased. Similarly Envato customers are having difficulty accessing downloads and content.
— WPBakery (@wpbakery) February 28, 2017
— WooCommerce (@WooCommerce) February 28, 2017
Amazon S3 has identified high error rate issue that’s impacting the Envato Market specifically our downloads and content. We are monitoring!
— Envato Help (@envato_help) February 28, 2017
Downloading and updating our premium plugins is possible again, after a short downtime due to problems at S3. Sorry for the inconvenience!
— Yoast (@yoast) February 28, 2017
Due to the AWS outage, we are experiencing problems with sales, support, and development. Please accept our apologies…we’ll be back soon!
— WP Ninjas, LLC (@wpninjas) February 28, 2017
Joost de Valk, CEO and founder of Yoast, said the company experienced minor effects from the outage but has already been planning on switching from S3 to a new storage provider.
“The outage doesn’t seem to have affected our revenue much,” de Valk said. “It was slightly annoying and led to some images not working and people not being able to download their plugins for a while, which is always a shame. However, not directly related to this, we’re already looking at ditching S3. That’s because our new hosting setup at SiteGround combined with CDN from MaxCDN actually negates the need for S3 entirely.”
Other companies that have AWS integrated into their support services experienced more disruption due to customers not being able to receive help.
“Obviously our website is hosted using AWS technology through Pagely,” WP Ninjas co-founder James Laws said. “I’m not sure how they’ve been affected directly, but we have noticed intermittent downtime. Perhaps the biggest impact is that our support service is built on AWS and with it down we are completely unable to provide any support to our users.”
Laws said the company has had fairly decent uptime with AWS in the past and that the idea of switching services because of an outage would not be worth the effort.
“The truth is that 100% uptime is more a fantasy than anything,” Laws said. “The idea of having to move a website or change a support system temporarily or even permanently for a short period of downtime would be pretty daunting. You probably could create contingency plans for something like this, but the technical and administrative costs are not generally worth it in my opinion.”
The outage serves as a painful reminder of how dependent the web is on cloud storage providers and how few services have a backup plan for instances like these.
At 12:52 PM PST Amazon released an update, promising improvements for customers within the hour: “We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour.” The ability to retrieve, list, and delete was fully recovered within half an hour and Amazon continues to work on fixing the ability to add new objects to S3.