Skip to main content
Back
DemonstrationAWS · Kubernetes · SaaS
2 min read

Hugging Face research found isolation gaps in inference and Spaces builds

Environment
Hugging Face research demonstration by Wiz
When
Research in 2024; individual test dates not statedDisclosed Apr 4, 2024
Entry
A research model running on shared inference infrastructure obtains its EKS node identity.

Wiz found two separate isolation gaps while testing Hugging Face. A model running on the shared inference service could obtain credentials belonging to its host. In a separate test, a build for Hugging Face Spaces could write to an internal registry that stored container images. Changing another customer’s image was a potential consequence. The reports do not document such a change.

What the demonstration did

Research team
Wiz
Internet
Hugging Face
Inference node
Spaces registry

Research demonstration / Wiz

  1. 1

    Inference workload identity

    2024

    A research model running on shared inference infrastructure obtains its EKS node identity.

  2. 2

    Spaces build registry

    2024

    In a separate test, Wiz finds that a Spaces build can write to the platform’s internal registry.

Impact

Wiz demonstrated exposure of infrastructure credentials and separate registry write access. Potential effects on other users’ models or Spaces were described as risks. The reports do not document altered customer images.

Would the controls have stopped it?

Separate model workloads from the node identity

On the inference nodes, requiring IMDSv2 and limiting the token response to one network hop can keep ordinary container workloads from receiving the node role. The Spaces registry is a separate boundary that needs authenticated, narrowly scoped writes.

Incident control assessments

0
Prevented
1
Contained
1
Not covered

How it could have been prevented

The same replay with the controls in place. Green marks where a step is refused, blue where it is bounded, and red where no control reaches it. Open a row to read the control, why it acts, and its caveat.

Research team
Wiz
Internet
Hugging Face
Inference node
Spaces registry

Research demonstration / Wiz

1 · Inference workload identity

1 Contained

2 · Spaces build registry

1 Not covered

Key facts

  • Wiz examined hosted model inference and the separate Hugging Face Spaces build environment.
  • A research inference workload could access its Amazon EKS node identity.
  • In a separate Spaces test, the build environment had write access to an internal container registry.
  • Wiz and Hugging Face published accounts of their work to improve platform isolation on April 4.

Sources

Every fact in this breakdown comes from the public reporting below. The replay is a model of that reporting, not a reconstruction of the environment it describes.

  1. 1.
    Hugging Face works with Wiz to strengthen AI cloud securityWiz Research · Apr 4, 2024 · ResearchWiz demonstrated exposure of infrastructure credentials and separate registry write access. Potential effects on other users’ models or Spaces were described as risks. The reports do not document altered customer images.
  2. 2.
    Hugging Face partners with Wiz Research to Improve AI SecurityHugging Face · Apr 4, 2024 · Vendor reportHugging Face describes its collaboration with Wiz and changes to its platform security program.
  3. 3.
    Configure the Instance Metadata Service optionsAWS · undated · Vendor reportAWS documents how token requirements and response hop limits affect container access to instance metadata.

The estate in the replay follows the researchers' description of their own test environment. Verdicts describe what a control does when it is in place before the run. They are not an assessment of any company's security program.

More cloud attack breakdowns

All breakdowns