In my experience with AWS to date, I've only witnessed one AWS account get hacked, and it was big.

It was nothing to do with AWS and everything to do with the customer.

A crypto take over 🥶

The admin / security team were lock out of the AWS account...

The account was spending, big time 💸

Even worse, the customer had taken out an AWS managed support contract with a 3rd party who managed their hardware MFA keys and root account credentials, they had never tested the process. What was logged as a p1 incident took what seemed like a lifetime for an actionable response.

Eventually the security team got in with AWS and suspended the account, but that caused another major issue..

They had never tested the re-provisioning of the mission critical hosted workloads (and there were many, all sharing the same account) to another AWS account.

A total disaster that cost north of a few million.

• Customer data leaked.

• Backups lost.

• The total recovery time for a critical suite of workloads was over a month

The stuff that Cloud nightmares are made of...

Just some of the many lessons they learned when doing AWS at scale:

• Segregate AWS account environments

• Implement time bound privileged access management & POLP

• Limit all console access to time bound, essential usage only

• Backup into vaults (logically air gapped allow for sharing to other accounts)

• Test root MFA login processes & have a clear and tested path of escalation for anomalies, change and access requests.

• Ensure workloads can be re-provisioned into new accounts through IaC

• Ban VPC peering

• Implement E/W traffic inspection

• Build an account security baseline with all the baseline AWS security features enabled

• Vend new accounts to a high set of standards, consistently

• Detect, analyse and take action on drift control

• Actively threat hunt, especially using AI

• Scan all code, especially dependencies

• Encrypt & rotate all secrets, keys and credentials

• Understand the AWS Shared Responsibility Model

• Build and agree Cloud standards applicable to all staff and suppliers, ensure that they are followed, escalate at CxO level when violated.

Don't think this could ever happen to you?

Here's AWS's own customer provided playbook for that exact scenario 👇

Make sure that you never need to follow it.

Call to Action

Thank you again for subscribing to AWS at Scale. If you like my content then please visit these posts online and share them across your socials and support me by tagging @leewynne

All the best, Lee