Skip to main content
Back
IntrusionKubernetes
5 min read

Kubeflow cryptomining through exposed dashboards

Victim
Unnamed operators of tens of Kubernetes clusters running Kubeflow
When
April 2020; day not statedDisclosed Jun 10, 2020
Entry
The Kubeflow central dashboard, published to the internet when operators changed the Istio ingress gateway Service to type LoadBalancer, accepted operations from anyone.

In April 2020, per Microsoft, an attacker reached Kubeflow dashboards that operators had published to the internet by changing the Istio ingress gateway Service to a public load balancer. From a Jupyter notebook, the attacker used the pod's mounted service account to deploy a container whose image, from a public repository, ran a cryptocurrency miner on tens of clusters. Microsoft does not say which cloud the clusters ran on, who the operators were, or how long the miners ran.

What happened

External Parties
Public image repository
Internet
Kubernetes API Servernotebook service account
Machine-Learning Kubernetes Cluster
Kubeflow Dashboard
Jupyter Notebook Pod
Miner Container

Replaying the Kubeflow April 2020 intrusion

  1. 1

    Dashboard on the internet

    Before April 2020 (not stated)

    Operators publish the Kubeflow dashboard through a public load balancer, and anyone who reaches it can deploy containers.

  2. 2

    Notebook service account

    April 2020 (day not stated)

    Code in a Jupyter notebook uses the pod's mounted service account to deploy the miner container through the Kubernetes API.

  3. 3

    Mining on the cluster

    April 2020 (day not stated)

    The container pulls a miner image from a public repository and mines on the cluster's nodes, often powerful machine-learning nodes.

Impact

A cryptocurrency miner ran on the compute of tens of Kubernetes clusters, per Microsoft, often powerful machine-learning nodes; no data theft is reported.

Outcome

Where Kubernetes and cloud controls fit

If you run Kubeflow, a ValidatingAdmissionPolicy that refuses a public LoadBalancer Service guards the dashboard only while the users who change the Service cannot remove the policy, and the sources do not say which rights they held. A policy that admits only images from your approved registry would have refused the miner. With automountServiceAccountToken set to false on the notebook's ServiceAccount, a notebook pod created after the change holds no token to deploy with, though a pod that already runs keeps its own. Each cloud has its own guardrail against the public load balancer, and the sources do not say which cloud the clusters ran on. ValidatingAdmissionPolicy became generally available in 2024, after the incident.

Could this have been prevented?

In hindsight, not as a whole. 1 control would have refused a step outright, and 2 controls would have limited one. 1 step has no cloud-provider control. The counts below are control assessments, not attack stages: several can address the same step. The same replay with the controls in place before the incident shows where each phase would have been refused or bounded.

1
Prevented
2
Contained
1
Not covered

How it could have been prevented

The same replay with the controls in place. Green marks where a step is refused, blue where it is bounded, and red where no control reaches it. Open a row to read the control, why it acts, and its caveat.

External Parties
Public image repository
Internet
Kubernetes API Servernotebook service account
Machine-Learning Kubernetes Cluster
Kubeflow Dashboard
Jupyter Notebook Pod
Miner Container

Replaying the Kubeflow April 2020 intrusion

1 · Dashboard on the internet

1 Contained4 on other clouds

2 · Notebook service account

1 Contained1 Not covered

3 · Mining on the cluster

1 Prevented4 on other clouds

Key facts

  • Microsoft's Azure Security Center team saw a suspect image from a public repository deployed on many clusters during April 2020 and published the campaign on June 10, 2020
  • Operators had changed the Istio ingress gateway Service to type LoadBalancer, which published the Kubeflow central dashboard to the internet; by default the dashboard is reachable only inside the cluster
  • Anyone who reached the exposed dashboard could perform operations in Kubeflow, including deploying containers; how anonymous access was allowed is not stated
  • The attacker deployed the miner container with the mounted service account of a Jupyter notebook pod, which by default may deploy containers in its namespace
  • The image ran a cryptocurrency miner on tens of Kubernetes clusters, per Microsoft; one news report said thousands, a count the Microsoft post does not give
  • The sources do not say which cloud the clusters ran on, who ran the campaign, how long the miners ran or how they were removed

Sources

Every fact in this breakdown comes from the public reporting below. The replay is a model of that reporting, not a reconstruction of the environment it describes.

  1. 1.
    Misconfigured Kubeflow workloads are a security riskMicrosoft Security (Azure Security Center) · Jun 10, 2020 · ResearchEntry vector, deployment route, payload, scale, recommendations
  2. 2.
    Kubernetes Falls to Cryptomining via Machine-Learning FrameworkThreatpost · Jun 11, 2020 · NewsRe-report of the chain, disputed count
  3. 3.
    Kubeflow Targeted in XMRig Monero Cryptomining CampaignBank Info Security (ISMG) · Jun 12, 2020 · NewsRe-report of the chain, deployment speed (unattributed)

The estate in the replay is illustrative and drawn at the level of attack stages; the victim's real environment is not public. Verdicts describe what a control does when it is in place before the incident. They are not an assessment of the victim's security program.

More cloud attack breakdowns

All breakdowns