Skip to main content
Back
DemonstrationKubernetes · AWS · Google Cloud
6 min read

Wiz Research's SAPwned demonstration on SAP AI Core

Environment
SAP AI Core, from Wiz Research's own customer account
When
not stated; before the Jan 25, 2024 report – May 15, 2024Disclosed Jul 17, 2024
Entry
An ordinary customer ran code in a Kubernetes pod through an Argo Workflow file, then bypassed the Istio sidecar to reach a trusted internal network of unauthenticated services.

Wiz Research worked as an ordinary customer of SAP AI Core, SAP's managed service for training AI workloads, and followed a chain of five findings from its own training pod to cluster administrator on the service's shared Kubernetes cluster. On the way, a Grafana Loki endpoint returned AWS secrets, six AWS EFS file systems answered with no credential, and an unauthenticated Helm Tiller server installed a cluster-admin pod that read Kubernetes Secrets holding customer cloud keys and an SAP read-write Google Container Registry key. Wiz reported the findings to SAP on January 25, 2024, SAP deployed fixes for all of them on May 15, 2024, and both state that no customer data was compromised.

What the demonstration did

Research team
Wiz Research
Internet
AWS S3 APILoki log bucket
Customer Training Pod
AI Core Internal Network
+ cluster-admin pod
+ Kubernetes Secrets
+ tenant pods
SAP AI Core Cluster
+ Container Registry
+ SAP images
+ read-write registry key
SAP Google Cloud project
Argo Workflow
Training Code
Istio Sidecar
Grafana Loki
Six EFS File Systems
Helm v2 Tiller

Replaying Wiz Research's SAPwned demonstration / published July 2024; the run date is not stated

  1. 1

    Code in a pod, past Istio

    Not dated · before Jan 25, 2024

    A training job runs the researchers' code in a Kubernetes pod, and two pod settings the admission controller missed take its traffic out of the Istio fence.

  2. 2

    A trusted internal network

    Not dated · before Jan 25, 2024

    Inside the network, a Loki config endpoint returns AWS secrets to its S3 log bucket, and six EFS file systems serve customer files with no credential.

  3. 3

    Cluster admin, then cloud keys

    Not dated · before Jan 25, 2024

    An unauthenticated Tiller server installs a cluster-admin pod, which reads Kubernetes Secrets holding customer cloud keys and an SAP read-write registry key.

Impact

From an ordinary customer pod, the chain reached cluster-admin on the shared SAP AI Core cluster and read Kubernetes Secrets in plaintext, including customer cloud keys (AWS, SAP HANA, Docker Hub) and a read-write SAP Google Container Registry key; Wiz and SAP both state no customer data was compromised.

Outcome

What the demonstration shows about your shared clusters

The chain needed no cloud vulnerability. Once the pod left the Istio fence, a config endpoint, six EFS file systems and a Tiller server all answered with no authentication, because the internal network was trusted. On your cloud, a security group on file-share mount targets and a guardrail on long-lived keys cap two of those steps, for keys created after the guardrail is set. The rest sits in Kubernetes: admission policies bound the pod fields and the service accounts a training pod may carry, and NetworkPolicies keep training pods' egress to what they need.

Would the controls have stopped it?

In hindsight, no control would have stopped it. 9 controls would have limited how far it went. 4 steps have no cloud-provider control. The counts below are control assessments, not attack stages: several can address the same step. The same replay with the controls in place before the incident shows where each phase would have been refused or bounded.

0
Prevented
9
Contained
4
Not covered

How it could have been prevented

The same replay with the controls in place. Green marks where a step is refused, blue where it is bounded, and red where no control reaches it. Open a row to read the control, why it acts, and its caveat.

Research team
Wiz Research
Internet
AWS S3 APILoki log bucket
Customer Training Pod
AI Core Internal Network
+ cluster-admin pod
+ Kubernetes Secrets
+ tenant pods
SAP AI Core Cluster
+ Container Registry
+ SAP images
+ read-write registry key
SAP Google Cloud project
Argo Workflow
Training Code
Istio Sidecar
Grafana Loki
Six EFS File Systems
Helm v2 Tiller

Replaying Wiz Research's SAPwned demonstration / published July 2024; the run date is not stated

1 · Code in a pod, past Istio

1 Contained2 Not covered

2 · A trusted internal network

4 Contained4 on other clouds

3 · Cluster admin, then cloud keys

4 Contained2 Not covered2 on other clouds

Key facts

  • Wiz Research began as an SAP AI Core customer with basic permissions; an Argo Workflow file it submitted spawned a Kubernetes pod running its own code, by design
  • The admission controller blocked root but allowed shareProcessNamespace, which exposed the Istio sidecar's Istiod token, and UID 1337, which Istio's iptables rules exclude, so the pod's traffic was no longer restricted
  • A Grafana Loki /config endpoint returned the AWS secrets Loki used for Amazon S3, and six AWS EFS file systems answered on NFS port 2049 with no credential, holding code and training data in folders by customer ID
  • An unauthenticated Helm v2 Tiller server on gRPC port 44134 revealed registry and Artifactory secrets and installed a Helm package that spawned a pod with cluster-admin privileges
  • With cluster admin, Wiz read Kubernetes Secrets in plaintext, including its own account's AWS, SAP HANA and Docker Hub keys and an SAP Google Container Registry key it confirmed as read-write
  • Wiz reported to SAP on January 25, 2024, bypassed the first patch with two new vulnerabilities on February 28, and SAP deployed fixes for all reported vulnerabilities on May 15, 2024; SAP stated that no customer data or systems were impacted

Sources

Every fact in this breakdown comes from the public reporting below. The replay is a model of that reporting, not a reconstruction of the environment it describes.

  1. 1.
    SAPwned: SAP AI vulnerabilities expose customers' cloud environments and private AI artifactsWiz Research · Jul 17, 2024 · ResearchEntry, admission controller gaps, Istio bypass, Loki AWS secrets, EFS shares, Tiller secrets, cluster admin, customer secrets, GCR key, disclosure timeline, no customer data compromised
  2. 2.
    SAP Core AI bugs allowed access to internal network servers, say researchersThe Register · Aug 6, 2024 · NewsSAP statement, patch sequence, Black Hat presentation
  3. 3.
    SAP AI Core Vulnerabilities Allowed Service Takeover, Customer Data AccessSecurityWeek · Jul 18, 2024 · NewsCoverage, permissions needed, patch date

The estate in the replay follows the researchers' description of their own test environment. Verdicts describe what a control does when it is in place before the run. They are not an assessment of any company's security program.

More cloud attack breakdowns

All breakdowns