Skip to main content
Back
IntrusionKubernetes · AWS
9 min read

SCARLETEEL 2.0 from JupyterLab pods to AWS admin and crypto mining

Victim
An unnamed Sysdig customer's AWS account
When
Not stated; reported Jul 11, 2023Disclosed Jul 11, 2023
Entry
JupyterLab notebook containers in a Kubernetes cluster on EC2 were exploited (how is not stated), and scripts in them read the worker node role's credentials from IMDSv1.

Sysdig's Threat Research Team watched the attack in a customer's AWS account and published it on July 11, 2023; it names no victim and dates no step. The actor exploited JupyterLab notebook pods in a Kubernetes cluster, how is not stated, read the worker node role's keys from IMDSv1, and created access keys for an admin user whose capitalized name fell outside a deny statement on key creation. With admin rights it created new users and keys and launched 42 mining instances, and the noise of the launches got it caught.

What happened

Outside parties
Attacker S3-Compatible Storage
Attacker Collection Hosts
Sysdig Threat Research
Internet
EC2 Instance MetadataIMDSv1 on some nodes
Kubernetes Cluster on EC2
Victim AWS Account
JupyterLab Notebook Pods
Kubernetes API Server
Worker Node Role
Admin IAM User
New Users and Keys
42 Mining Instances
Secrets Manager

Replaying the SCARLETEEL 2.0 intrusion, reported July 2023

  1. 1

    Foothold in the notebook pods

    Not stated

    JupyterLab notebook containers in a Kubernetes cluster, exposed to the public, are exploited. No source names the vulnerability or says how the containers were reached.

  2. 2

    Node credentials and tooling

    Not stated

    Scripts in the pods read the worker node role's keys from IMDSv1; an IMDSv2 attempt fails at the default hop limit of 1. They also search files and other containers, peirates queries the Kubernetes API, and the AWS CLI is pointed at the attacker's own S3-compatible storage. No source says what the searches or the API calls returned.

  3. 3

    Escalation to admin

    Not stated

    With the node role's keys, Pacu runs reconnaissance and some EC2 launches fail. Access keys are created for an admin user whose capitalized name falls outside a deny statement, then new users and new keys for every user hold the account.

  4. 4

    Mining and the last attempts

    Not stated

    42 c5.metal and r5a.4xlarge instances mine Monero until the noise of the launches gets the actor caught. After admin access is limited, attempts at Secrets Manager and SSH keys fail for lack of privileges.

Impact

The actor gained AdministratorAccess through an admin user a deny statement missed, created persistent users and keys, and launched 42 mining instances before it was caught; later attempts at Secrets Manager and SSH keys failed. Whether data was exfiltrated is not stated.

Outcome

A node role and an admin name the deny missed carried the chain

No cloud control fixes the flaw in the notebooks you expose, so the chain starts at the pods; inbound rules on the worker nodes cap who can reach them. IMDSv1 handed the worker node role's keys to code in the pods, while IMDSv2 at the default hop limit of 1 already refused the one script that tried it, and a Kubernetes NetworkPolicy that blocks the metadata address does the same. The deny statement on access-key creation matched user names case-sensitively, so an admin named with a capital A got new keys; a service control policy keyed on the caller has no such gap, although which identity made the call is not stated. A service control policy that allows only the instance types your account uses bounds what even an admin can launch in a member account of an organization; the sources do not say whether this account was one, or which instance types it used.

Could this have been prevented?

In hindsight, not as a whole. 2 controls would have refused a step outright, and 14 controls would have limited one. 2 steps have no cloud-provider control. The counts below are control assessments, not attack stages: several can address the same step. The same replay with the controls in place before the incident shows where each phase would have been refused or bounded.

2
Prevented
14
Contained
2
Not covered

How it could have been prevented

The same replay with the controls in place. Green marks where a step is refused, blue where it is bounded, and red where no control reaches it. Open a row to read the control, why it acts, and its caveat.

Outside parties
Attacker S3-Compatible Storage
Attacker Collection Hosts
Sysdig Threat Research
Internet
EC2 Instance MetadataIMDSv1 on some nodes
Kubernetes Cluster on EC2
Victim AWS Account
JupyterLab Notebook Pods
Kubernetes API Server
Worker Node Role
Admin IAM User
New Users and Keys
42 Mining Instances
Secrets Manager

Replaying the SCARLETEEL 2.0 intrusion, reported July 2023

1 · Foothold in the notebook pods

2 Contained3 on other clouds

2 · Node credentials and tooling

2 Prevented6 Contained2 Not covered6 on other clouds

3 · Escalation to admin

4 Contained9 on other clouds

4 · Mining and the last attempts

2 Contained6 on other clouds

Key facts

  • Sysdig published its write-up on July 11, 2023; it names no victim and gives no date for any step of the attack
  • The entry was JupyterLab notebook containers in a Kubernetes cluster, exposed to the public; no source names the vulnerability
  • One script's IMDSv2 attempt failed because the instance kept the default hop limit of 1; on nodes that allowed IMDSv1, a plain request returned the node role's keys
  • A deny statement on iam:CreateAccessKey matched only lowercase admin user ARNs, so an admin user named with a capital A got new access keys; which identity made the call is not stated
  • The actor launched 42 c5.metal and r5a.4xlarge instances to mine Monero; Sysdig's conservative estimate was over $4,000 per day had the attack not been stopped
  • After admin access was limited, attempts to read Secrets Manager secrets and update SSH keys failed for lack of privileges

Sources

Every fact in this breakdown comes from the public reporting below. The replay is a model of that reporting, not a reconstruction of the environment it describes.

  1. 1.
    SCARLETEEL 2.0: Fargate, Kubernetes, and CryptoSysdig Threat Research Team · Jul 11, 2023 · ResearchEntry through JupyterLab containers, credential sources tried, IMDSv1 success and IMDSv2 hop-limit failure, AWS CLI and Pacu, S3-compatible endpoint outside CloudTrail, peirates calls, Pandora malware, deny-statement bypass, persistence users and keys, 42 mining instances, detection and the failed later attempts
  2. 2.
    Scarleteel 2.0 and the MITRE ATT&CK frameworkSysdig · Sep 6, 2023 · ResearchEntry container exposed to the public, credential-file search, Kubernetes API calls in the jupyter namespace, deny-statement bypass
  3. 3.
    Scarleteel 2.0 and the MITRE ATT&CK FrameworkCloud Security Alliance (republished from Sysdig) · Jan 2, 2024 · ResearchText of the deny statement on iam:CreateAccessKey, Kubernetes API calls made with the pod's service account credentials
  4. 4.
    SCARLETEEL Fine-Tunes AWS and Kubernetes Attack TacticsThe New Stack · Jul 13, 2023 · NewsReport of the write-up, quote of its author

The estate in the replay is illustrative and drawn at the level of attack stages; the victim's real environment is not public. Verdicts describe what a control does when it is in place before the incident. They are not an assessment of the victim's security program.

More cloud attack breakdowns

All breakdowns