What happened
Replaying the Kubeflow April 2020 intrusion
- 1
Dashboard on the internet
Before April 2020 (not stated)Operators publish the Kubeflow dashboard through a public load balancer, and anyone who reaches it can deploy containers.
- 2
Notebook service account
April 2020 (day not stated)Code in a Jupyter notebook uses the pod's mounted service account to deploy the miner container through the Kubernetes API.
- 3
Mining on the cluster
April 2020 (day not stated)The container pulls a miner image from a public repository and mines on the cluster's nodes, often powerful machine-learning nodes.
A cryptocurrency miner ran on the compute of tens of Kubernetes clusters, per Microsoft, often powerful machine-learning nodes; no data theft is reported.
Where Kubernetes and cloud controls fit
If you run Kubeflow, a ValidatingAdmissionPolicy that refuses a public LoadBalancer Service guards the dashboard only while the users who change the Service cannot remove the policy, and the sources do not say which rights they held. A policy that admits only images from your approved registry would have refused the miner. With automountServiceAccountToken set to false on the notebook's ServiceAccount, a notebook pod created after the change holds no token to deploy with, though a pod that already runs keeps its own. Each cloud has its own guardrail against the public load balancer, and the sources do not say which cloud the clusters ran on. ValidatingAdmissionPolicy became generally available in 2024, after the incident.
Could this have been prevented?
In hindsight, not as a whole. 1 control would have refused a step outright, and 2 controls would have limited one. 1 step has no cloud-provider control. The counts below are control assessments, not attack stages: several can address the same step. The same replay with the controls in place before the incident shows where each phase would have been refused or bounded.
- 1
- Prevented
- 2
- Contained
- 1
- Not covered
How it could have been prevented
The same replay with the controls in place. Green marks where a step is refused, blue where it is bounded, and red where no control reaches it. Open a row to read the control, why it acts, and its caveat.
Replaying the Kubeflow April 2020 intrusion
1 · Dashboard on the internet
1 Contained4 on other clouds2 · Notebook service account
1 Contained1 Not covered3 · Mining on the cluster
1 Prevented4 on other cloudsKey facts
- Microsoft's Azure Security Center team saw a suspect image from a public repository deployed on many clusters during April 2020 and published the campaign on June 10, 2020
- Operators had changed the Istio ingress gateway Service to type LoadBalancer, which published the Kubeflow central dashboard to the internet; by default the dashboard is reachable only inside the cluster
- Anyone who reached the exposed dashboard could perform operations in Kubeflow, including deploying containers; how anonymous access was allowed is not stated
- The attacker deployed the miner container with the mounted service account of a Jupyter notebook pod, which by default may deploy containers in its namespace
- The image ran a cryptocurrency miner on tens of Kubernetes clusters, per Microsoft; one news report said thousands, a count the Microsoft post does not give
- The sources do not say which cloud the clusters ran on, who ran the campaign, how long the miners ran or how they were removed
Sources
Every fact in this breakdown comes from the public reporting below. The replay is a model of that reporting, not a reconstruction of the environment it describes.
- 1.Misconfigured Kubeflow workloads are a security riskMicrosoft Security (Azure Security Center) · Jun 10, 2020 · ResearchEntry vector, deployment route, payload, scale, recommendations
- 2.Kubernetes Falls to Cryptomining via Machine-Learning FrameworkThreatpost · Jun 11, 2020 · NewsRe-report of the chain, disputed count
- 3.Kubeflow Targeted in XMRig Monero Cryptomining CampaignBank Info Security (ISMG) · Jun 12, 2020 · NewsRe-report of the chain, deployment speed (unattributed)
The estate in the replay is illustrative and drawn at the level of attack stages; the victim's real environment is not public. Verdicts describe what a control does when it is in place before the incident. They are not an assessment of the victim's security program.