What happened
Replaying the SCARLETEEL intrusion, reported February 2023
- 1
Into the cluster
Not statedAn internet-exposed service in the self-managed Kubernetes cluster is exploited; its image carried a known vulnerability. A miner and a credential script are downloaded and run in its container.
- 2
Node role from metadata
Not statedA script in the container reads the worker node's IAM role credentials from IMDSv1. Calls with them to create IAM users, groups and access keys are all denied.
- 3
The account read
Not statedThe node role, meant to read one S3 bucket but able to read everything in the account, downloads the Lambda functions' code and reads IAM user keys from their environment variables. More than 1 TB is read from the S3 buckets; which credential made those reads is not stated.
- 4
Logs, then the second account
Not statedAn IAM user compromised earlier stops some CloudTrail logging. A Terraform state file in S3 holds a clear-text key of an IAM user of a second AWS account, and that account refuses every call but GetCallerIdentity.
Proprietary software and its keys were taken from Lambda functions, more than 1 TB was read from S3 buckets, some CloudTrail logging was disabled, and an IAM user key of a second account was found; the second account refused the attacker's calls.
One read-everything node role carried the chain
No cloud control reaches the vulnerable image of a service you expose from your own cluster, so the chain starts outside the cloud. What turned a foothold in one container into the theft of Lambda code and more than 1 TB of S3 data was the worker node's IAM role: IMDSv1 handed its credentials to the container, and the role could read the whole account. IMDSv2 with a hop limit of 1 on the worker instances refuses that read, and a permissions boundary that keeps the role to its one bucket refuses the Lambda calls. A Kubernetes NetworkPolicy that excepts the metadata address caps the read where the network plugin enforces the policy. In a member account of an organization, a service control policy caps who can stop the trails, and bucket and key policies on the Terraform state bucket cap who can read the key it holds. In the second account, the well-scoped IAM user is why every call made with that key failed except GetCallerIdentity.
Could this have been prevented?
In hindsight, not as a whole. 2 controls would have refused a step outright, and 12 controls would have limited one. 1 step has no cloud-provider control. The counts below are control assessments, not attack stages: several can address the same step. The same replay with the controls in place before the incident shows where each phase would have been refused or bounded.
- 2
- Prevented
- 12
- Contained
- 1
- Not covered
How it could have been prevented
The same replay with the controls in place. Green marks where a step is refused, blue where it is bounded, and red where no control reaches it. Open a row to read the control, why it acts, and its caveat.
Replaying the SCARLETEEL intrusion, reported February 2023
1 · Into the cluster
2 Contained1 Not covered3 on other clouds2 · Node role from metadata
1 Prevented4 Contained6 on other clouds3 · The account read
1 Prevented2 Contained6 on other clouds4 · Logs, then the second account
4 Contained9 on other cloudsKey facts
- Sysdig published its write-up on February 28, 2023; it names no victim and gives no date or time for any step of the intrusion
- The entry was an internet-exposed service in a self-managed Kubernetes cluster; Sysdig's Director of Threat Research said its image carried a known vulnerability that no policy or process prevented
- The worker node's IAM role was meant to read one S3 bucket but could read everything in the account, Lambda included
- Attempts to create IAM users and groups and to bind new access keys to existing users were all denied for lack of permissions
- More than 1 TB was read from S3, including customer scripts, troubleshooting tools and logs; S3 data events were not logged, so the objects read are unknown
- A Terraform state file in S3 held a clear-text access key of an IAM user of a second AWS account; that user was well scoped, and every call failed except GetCallerIdentity
Sources
Every fact in this breakdown comes from the public reporting below. The replay is a model of that reporting, not a reconstruction of the environment it describes.
- 1.SCARLETEEL: Operation leveraging Terraform, Kubernetes, and AWS for data theftSysdig Threat Research Team · Feb 28, 2023 · ResearchEntry vector, IMDSv1 credential theft, denied persistence calls, role scope, Lambda code and environment variable theft, S3 reads, CloudTrail logging disabled, Terraform state key, second account enumeration, response, recommendations
- 2.How to Detect SCARLETEEL with Sysdig SecureSysdig · Mar 29, 2023 · ResearchRecommendations, restated chain
- 3.Novel cloud attack pivoted from K8s to Lambda, pulled IAM keys from TerraformThe Stack · Mar 1, 2023 · NewsEntry vector detail, restated chain
- 4.SCARLETEEL hackers use advanced cloud skills to steal source code, dataBleepingComputer · Feb 28, 2023 · NewsRestated chain, recommendations
- 5.SCARLETEEL 2.0: Fargate, Kubernetes, and CryptoSysdig Threat Research Team · Jul 11, 2023 · ResearchLink between the February and July write-ups
The estate in the replay is illustrative and drawn at the level of attack stages; the victim's real environment is not public. Verdicts describe what a control does when it is in place before the incident. They are not an assessment of the victim's security program.