Businesses and their supply chain management teams can take heart that in this year of continuous non-stop disruptive incidents, even the mighty, most prepared and influential industry participants are not immune by such incidents.

In this Supply Chain Matters and subsequent blog postings, we highlight examples of untimely supply chain and customer fulfillment disruptions involving both Amazon and Apple.

 

Significant Amazon AWS Outage

Yesterday, a major outage involving Amazon Web Services (AWS) Cloud infrastructure hosting division impacted operations and services for quite a number of other businesses including Amazon’s own customer fulfillment centers.

The incident reportedly began yesterday around 10:45 am when system users began encountering access problems. It was not until early evening that services began to recover.

Amazon AWS itself began communicating an hour later that they were aware of impacts to multiple AWS APIs in the division’s U.S. Region located in Virginia. The communication noted the root cause being an impairment of several network access devices. and that the impact was further delaying the division’s ability to actually provide updates.

Reported outages reportedly impacted U.S. based airlines Delta and Southwest, sites Disney Plus Netflix Venmo and a host of others including Amazon’s own operations. The Associated Press acknowledged that its abilities to publish news were inoperable for most of yesterday as a result of the outage.

Business network CNBC reported that Amazon operational workers on social media were sharing photos of customer fulfillment centers at a total standstill because electronic sensors were not operable. There were other reports indicating that certain contracted Amazon last-mile customer delivery drivers were idled because of a lack of access to delivery systems. The New York Times reported that updates from Amazon workers appearing on Facebook and other social media forums indicated that operational supervisors tried to make the best of the outage hoping that system access would return. Employees were reportedly either asked to take early lunch breaks, take on facility maintenance chores, or offered he opportunity to take unpaid time to leave their work shifts earlier than scheduled.

The Verve reported that the Amazon.com website itself as well as services related to Alexa assistant, Kindle books, Amazon Music and other services were also impacted as were cameras associated with Ring and Wyze.

Commenting on yesterday’s AWS outage, Peter Groucutt, Managing Director at Databarracks indicated to Supply Chain Matters, in-part:

An enormous number of SaaS business services are hosted on AWS and when these tools are hit, the knock-on effect impacts thousands of downstream businesses. The technology supply-chain is a big, interconnected web, but often it leads back to AWS, Microsoft Azure or Google.

The other positive we can take from this incident is that it was localised to specific AWS Regions. In many ways, we shouldn’t think of AWS as just one cloud, but rather a collection of connected data centres spread across the world. It is designed so that an issue in US-East-1 doesn’t affect Seoul or London.”

Thus far, Amazon has provided limited information as to the extent or impact of yesterday’s outage on overall customer fulfillment activities.

Supply Chain Matters is of the view that the impact will likely be confined, but to what extent will depend on what on online platform provider elects to disclose. The scope of the impact will likely be provided by Amazon’s customers in the notions of delayed delivery commitments.

True to Murphy’s Law, the timing of this hours-long system outage coming at the surge of holiday fulfillment volume activity is a marker for Amazon’s U.S. customer fulfillment teams in that the system outage caused an operational halt. In these times of same day and two-hour online delivery commitments, full operational halts for any online fulfillment facility are a cause for concern and action. The question is why an outage occurring in one region could not be readily overcome by switching to a redundant Cloud operation from a backup region. We have little doubt that this will garner the immediate attention of senior Amazon operational and AWS management.

Indeed, even the best in preparations can be impacted by points of vulnerability.

The takeaway for businesses and their associated supply chain management and IT strategy teams is in continually assessing a deemed mission critical Cloud software’s provider’s dependencies on specific hosted Cloud providers and their detailed systems outage recovery plans.

Bob Ferrari

 

© Copyright 2021, The Ferrari Consulting and Research Group and the Supply Chain Matters® blog. All rights reserved.