The AWS outage exposes the fragility of our data-hungry digital economy

The Amazon Web Services (AWS) outage, reported by The Verge, took down a massive number of applications and websites yesterday (25th November). But that shouldn’t really be that surprising – AWS is far and away the most widely used public cloud platform. If anything, its impact only further emphasises the extent to which we all depend on Amazon.

However, there’s one element of the story that has largely passed unnoticed: the cause of the outage. Despite the lack of interest, it is, I think, revealing about both the nature of the web and how companies actually use AWS. It also highlights an interesting irony: while huge amounts of data are supposed to ensure business resilience and adaptability, the complexity of the systems that facilitate its collection can actually increase the chance of failure and ‘downtime’.

Kinesis Data Streams: the cause of the AWS outage

The cause of the AWS outage has been identified by AWS as a problem with Kinesis Data Streams. To most people, this might seem like a bit of trivial technical detail – but Kinesis Data Streams is actually an interesting product within the AWS ecosystem. It’s what is called a data streaming platform, which means it allows its users to collect and process huge amounts of data extremely quickly. As the copy on the AWS website says:

“KDS can continuously capture gigabytes of data per second from hundreds of thousands of sources such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events. The data collected is available in milliseconds to enable real-time analytics use cases such as real-time dashboards, real-time anomaly detection, dynamic pricing, and more.”

The ability to capture this data isn’t particularly novel. There are, today, many commercially available tools that help organizations do it; AWS is just another possible option on the market for companies that need want to be able to track such extensive data sets. But just because it’s fairly unremarkable doesn’t mean it’s not significant, or, indeed, worthy of critique.

A tool of surveillance capitalism

It’s not unfair to call KDS a tool of surveillance capitalism: it’s specifically designed to help companies collect and process huge amounts of data. This point is important, because it’s tempting to just think of ‘cloud’ as simple, straightforward hosting. I need to put my website or app somewhere, so I’ll park it on AWS.

But that’s obviously not the case. Public cloud platforms like AWS are multifaceted and complex things that enable and encourage certain practices.

You might even say that because Amazon has built its business on its own form of surveillance capitalism, it’s only natural that it would want to encourage such similar forms of surveillance throughout the digital economy. This benefits Amazon in two ways: first, it normalises surveillance. Then it ensures that the company has a monopoly on the tools that enable it.

Read more news and analysis on privacy issues in the digital era.

Surveillance capitalism and system vulnerability

The AWS outage is interesting not just because it underlines the extent to which we all depend on Amazon, but also because it highlights how surveillance capitalism has the capacity to undermine itself. The level of complexity required for it to operate across the internet creates new weaknesses and vulnerabilities.

This suggests two things. On the one hand data surveillance is built on far flimsier foundations than it often appears; on the other, it demonstrates that its benefits significantly outweigh the risk and effort required to make it successful.