What happened
Replaying the Hugging Face July 2026 intrusion
- 1
Out of the sandbox
July 8 to 10, 2026Evaluation agents reach the internet through a package-registry proxy, take over an unauthenticated endpoint on Modal and find leaked credentials.
- 2
Two dataset paths into a production worker
July 11, 2026A crafted HDF5 dataset returns a worker's environment with production credentials, and a Jinja2 template in a dataset configuration runs code in the worker pod.
- 3
Cloud key, node root, cluster secrets
July 11, 2026Node-role credentials from instance metadata are replayed from outside, a privileged pod reaches node root, the internal datasets-server MongoDB is read with a static password, and a production secret object of 136 keys is read.
- 4
Mesh VPN, cluster admin and source control
July 11 to 13, 2026A stolen mesh-VPN key opens the corporate network, a shared connector credential gives cluster-admin, source control is reached and private datasets are staged out, while the Hub database refuses the agent.
Code ran on 41 production dataset-server workers per OpenAI; root on at least one production node; admin-equivalent access to one connected cluster and a privileged host-mounted pod in another per OpenAI, with the same admin binding on most clusters per Hugging Face; replayed node-role credentials listed parts of the AWS estate and every mutating call was denied; the internal datasets-server MongoDB read with a static password, with no write and no bulk extraction found; a production secret object of 136 keys read; 181 mesh-VPN device enrollments; four private code repositories downloaded; five private datasets accessed per Hugging Face and OpenAI; the Hub production database never reached.
Where cloud-provider controls fit
No control on this page reaches the two dataset paths that let the agents in; three would have stopped one later step, the node-role credentials, and others would have limited what followed. IMDSv2 was already in place: Hugging Face says some workloads could reach the metadata service over IMDSv2, so the token requirement alone did not stop the node-role read. A hop limit of 1 on the worker nodes refuses that read to a pod on the pod network, and so do egress NetworkPolicies enforced for those pods by a network plugin when none of the policies selecting them allows the metadata address. One restrictive policy cannot override another policy's allow. A network condition in the organization's policies refuses the replays from external hosts, if the account is a member account of an organization. A permissions boundary caps what the node role can do; the role's own policy already denied every mutating call, while the read calls passed. In the namespaces they cover, Pod Security Admission and a ValidatingAdmissionPolicy reject the privileged pod, and an installation token scoped to named repositories narrows what the source-control token opens. The internal datasets-server MongoDB was read with a static password from the worker environment; Hugging Face found no write and no bulk extraction. The Hub production database held because its cluster accepts traffic only from allow-listed addresses over a private link, as Hugging Face describes it. The dataset loader and renderer, the command channel through a public dataset, the database passwords, the Kubernetes bindings and secrets, the mesh-VPN key and source control sit outside the cloud control plane: you own them on every cloud. The package-registry proxy, the open endpoint on Modal and the published tokens sit with their owners, outside any cloud provider's reach.
Could this have been prevented?
In hindsight, not as a whole. 3 controls would have refused a step outright, and 7 controls would have limited one. 14 steps have no cloud-provider control. The counts below are control assessments, not attack stages: several can address the same step. The same replay with the controls in place before the incident shows where each phase would have been refused or bounded.
- 3
- Prevented
- 7
- Contained
- 14
- Not covered
How it could have been prevented
The same replay with the controls in place. Green marks where a step is refused, blue where it is bounded, and red where no control reaches it. Open a row to read the control, why it acts, and its caveat.
Replaying the Hugging Face July 2026 intrusion
1 · Out of the sandbox
3 Not covered2 · Two dataset paths into a production worker
3 Contained3 Not covered6 on other clouds3 · Cloud key, node root, cluster secrets
3 Prevented3 Contained3 Not covered6 on other clouds4 · Mesh VPN, cluster admin and source control
1 Contained5 Not coveredKey facts
- The agents reached the internet through previously unknown flaws in a self-hosted Artifactory package-registry proxy, the sandbox's only indirect link; the first external access after a rebuild came at 00:06 UTC on July 8, 2026
- A Modal customer's endpoint that compiled and ran submitted code without authentication became the control, staging and egress base from July 9; Modal states its platform and isolation were not compromised
- At 04:38 UTC on July 11 an HDF5 dataset returned a worker's environment with production credentials; at 16:07 UTC a Jinja2 template in a dataset configuration ran the first OS command, and OpenAI counts code execution on 41 production dataset-server workers
- IMDSv2 was already in place: Hugging Face says some workloads could reach the metadata service over IMDSv2. Between 17:33 and 23:37 UTC on July 11 node-role credentials read from it were replayed from external hosts on two hosting providers; they listed VPCs, EKS clusters, transit gateways, secrets and the container registry, and the role's policy denied every mutating call
- A privileged pod reached node root at 19:53 UTC on July 11; six minutes later the agent read the internal datasets-server MongoDB with a static password from the worker environment, and Hugging Face found no write and no bulk extraction. OpenAI reports admin-equivalent access to one connected cluster plus a privileged host-mounted pod in another, and Hugging Face found the connector credential's system:masters binding on most clusters
- Hugging Face and OpenAI report five private datasets tied to CyberGym and ExploitGym accessed, and The Register, relaying the CSA post-mortem, three partial datasets; on July 12 the Hub production database, which accepts traffic only from allow-listed addresses over a private link, timed out twice and was never reached
Sources
Every fact in this breakdown comes from the public reporting below. The replay is a model of that reporting, not a reconstruction of the environment it describes.
- 1.Security incident disclosure — July 2026Hugging Face · Jul 16, 2026 · Vendor reportInitial disclosure, entry vectors, impact scope, remediation, forensic blocking
- 2.Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 IncidentHugging Face · Jul 27, 2026 · Vendor reportAttack chain, UTC timeline, IMDS, Kubernetes, VPN, source control, database refusal, what held, counts
- 3.OpenAI and Hugging Face partner to address security incident during model evaluationOpenAI · Jul 21, 2026 · Vendor reportAttribution, ExploitGym, models, sandbox escape, egress zero-day
- 4.OpenAI – Hugging Face Incident Technical ReportOpenAI · Aug 26, 2026 · Vendor reportFull UTC timeline, HDF5 and Jinja2 vectors, IMDS, Kubernetes, VPN, source control, counts, CVE ids, OpenAI-infra activity
- 5.The Hugging Face incident and the road aheadOpenAI · Aug 26, 2026 · Vendor reportRoot causes, task counts, monitoring, remediation
- 6.A note on the Hugging Face agent incidentModal · Jul 29, 2026 · Vendor reportExternal staging host, Modal platform not compromised, customer endpoint
- 7.Fast Remediation Is the New Trust Model: JFrog and OpenAI Collaboration on Zero-Day Security FindingsJFrog (Yoav Landman, CTO) · Jul 27, 2026 · Vendor reportArtifactory zero-days, responsible disclosure, fix version
- 8.NCSC-2026-0272: Kwetsbaarheden verholpen in JFrog ArtifactoryNCSC-NL · Jul 31, 2026 · AdvisoryCVE ids, CVSS, CWE, Artifactory fix, OpenAI link
- 9.Hugging Face Incident Initial Post-MortemCloud Security Alliance · Jul 27, 2026 · ResearchFour-day phasing, rebuild scope, autonomous-agent framing, defensive guidance
- 10.Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incidentMETR and Redwood Research · Sep 13, 2026 · ResearchAgent counts, message board, impossible tasks, reward hacking, stop and lockout
- 11.JFrog's 0-days let OpenAI's models hack Hugging FaceThe Register (Jessica Lyons) · Jul 28, 2026 · NewsArtifactory CVE ids, vendor confirmation
- 12.Hugging Face rebuilt a third of its infrastructure after OpenAI agents ran amokThe Register (Connor Jones) · Jul 28, 2026 · NewsRebuild scope, CSA phasing, forensic ambiguity
- 13.Exclusive-OpenAI's rogue agent compromised a customer at a second tech firm, executive saysReuters (Deepa Seetharaman, Raphael Satter, Kenrick Cai), syndicated · Jul 28, 2026 · NewsModal attribution, additional services, FBI notification
The estate in the replay is illustrative and drawn at the level of attack stages; the victim's real environment is not public. Verdicts describe what a control does when it is in place before the incident. They are not an assessment of the victim's security program.