What happened
Replaying the Codecov January 2021 intrusion
- 1
A key in a public image
Not datedAn HMAC key for a Google Cloud Storage service account is extracted from an intermediate layer of the public Codecov Self-Hosted Docker image.
- 2
The uploader changed in Cloud Storage
Jan 31 to Apr 1, 2021, in 108 windowsCloud Storage accepts requests signed with the leaked key from outside Codecov, and the Bash Uploader object is replaced with a version that sends data out.
- 3
CI environments sent out
Jan 31 to Apr 1, 2021Customers' CI jobs fetch the changed script and run it, and it posts each job's git remote and environment variables to a server outside Codecov.
- 4
Customer systems reached
Jan 31 to Apr 18, 2021; cloud use unconfirmedGitHub credentials taken from CI reach customers' repositories, which are cloned or accessed; cloud provider keys taken from CI may have been used, which no source confirms.
- 5
Found and closed
Apr 1 to 15, 2021A customer's checksum check exposes the change on April 1, 2021 and Codecov remediates the script that day; Codecov revokes the key, rotates its production keys and emails affected users on April 15.
The Bash Uploader in Codecov's Google Cloud Storage was changed in 108 windows of time between January 31 and April 1, 2021, and each CI job that ran it sent its git remote and environment variables to a server outside Codecov. Customers report an exposed release-signing key, exposed or possibly exposed GitHub tokens, cloned repositories and records of personal information in source code. No source gives the number of affected customers or confirms that a cloud provider key taken from CI was used.
Where cloud-provider controls fit
The key sat in a public image layer, where no cloud-provider control looks, and the script ran inside your CI jobs with whatever their environment held. The cloud-side decision was Codecov's: a Google Cloud Organization Policy constraint that refuses service account HMAC-signed requests would have refused the key, provided nothing Codecov ran depended on such requests. An IAM deny policy on object writes limits who can replace the script, and a locked retention policy limits when a published object can be replaced; the sources do not name the key's service account or say how the script was published. On your side, jobs that reach your clouds through federated identities, not keys stored in CI variables, leave no cloud key in the environment a script can send out; whether any cloud key taken this way was used is unconfirmed. An egress allow-list caps what leaves a self-hosted runner. The checksum check in your pipeline, your CI service's hosted runners and your GitHub tokens own the rest: no cloud-provider policy reaches them.
Could this have been prevented?
In hindsight, not as a whole. 1 control would have refused a step outright, and 10 controls would have limited one. 6 steps have no cloud-provider control. The counts below are control assessments, not attack stages: several can address the same step. The same replay with the controls in place before the incident shows where each phase would have been refused or bounded.
- 1
- Prevented
- 10
- Contained
- 6
- Not covered
How it could have been prevented
The same replay with the controls in place. Green marks where a step is refused, blue where it is bounded, and red where no control reaches it. Open a row to read the control, why it acts, and its caveat.
Replaying the Codecov January 2021 intrusion
1 · A key in a public image
1 Not covered2 · The uploader changed in Cloud Storage
1 Prevented3 Contained7 on other clouds3 · CI environments sent out
1 Contained2 Not covered3 on other clouds4 · Customer systems reached
6 Contained2 Not covered8 on other clouds5 · Found and closed
1 Not coveredKey facts
- Codecov's root cause analysis said the attacker extracted an HMAC key for a Google Cloud Storage service account from an intermediate layer of the public Codecov Self-Hosted Docker image; when and how was not stated
- Per Codecov, the changed Bash Uploader was in effect during 108 windows of time between January 31 and April 1, 2021; their lengths were not stated
- One added line posted the output of git remote -v and env to a third-party server, and Codecov said it was the only change to the script
- On April 1, 2021 a customer reported that the downloaded script's SHA256 did not match the one published on GitHub; Codecov remediated the script that day and emailed affected users on April 15
- HashiCorp reported its release-signing GPG key exposed with no evidence of use; Twilio reported an exposed GitHub user token and Coalition a possibly exposed one, with repositories cloned at both; Mercari reported 27,889 records exposed in its source code
- No source gave the number of affected customers or confirmed that a cloud provider key taken from CI was used; unnamed investigators told Reuters of access to hundreds of customer networks, which Codecov did not confirm
Sources
Every fact in this breakdown comes from the public reporting below. The replay is a model of that reporting, not a reconstruction of the environment it describes.
- 1.Bash Uploader Security UpdateCodecov · Apr 29, 2021 · Vendor reportstart date, entry vector, Google Cloud Storage key, malicious change, exfiltrated data, affected products, self-hosted scope, discovery, remediation, customer notification, indicators of compromise, law enforcement, forensic firm, attribution not determined
- 2.Post-Mortem / Root Cause Analysis (April 2021)Codecov · Jun 8, 2021 · Vendor reportroot cause, HMAC key, service account, Docker image layer, Google Cloud Storage object change, detection, response roles, recovery, Docker image rebuild, storage monitoring, new uploader, key management, federal investigators, no other intrusions
- 3.Codecov Releases New Detections for Supply Chain CompromiseCISA · Apr 30, 2021 · Advisoryofficial confirmation of dates, customer guidance, certificate reissue
- 4.HCSEC-2021-12 - Codecov Security Event and HashiCorp GPG Key ExposureHashiCorp · May 4, 2021 · Vendor reportdownstream exposure, signing key, CI pipelines, response start date, key rotation, re-signing, no evidence of misuse
- 5.Twilio's Response to the Recent Codecov VulnerabilityTwilio · May 4, 2021 · Vendor reportdownstream exposure, Codecov notification, GitHub notification, user token, cloned repositories, email addresses, secret rotation, no modification
- 6.Rapid7's Response to Codecov IncidentRapid7 · Sep 3, 2024 · Vendor reportdownstream exposure, single CI server, source code repositories, internal credentials, customer alert data, external forensics, no production access
- 7.Mercari's response to the Codecov vulnerability and related notification on personal information exposureMercari · May 22, 2021 · Vendor reportdownstream exposure, credential revocation, GitHub notification, GitHub access dates, source code access, exposed record counts, regulator report, no damage to users
- 8.Incident report in response to the recent Codecov breachCoalition · May 7, 2021 · Vendor reportdownstream exposure, GitHub notification, read-only user token, clone date, secret rotation, no PII, no modification
- 9.Codecov hackers breached hundreds of restricted customer sites: sourcesReuters (syndicated on Yahoo News) · Apr 19, 2021 · Newsreported scope, reported automation, FBI investigation, customer statements, customer base
- 10.Codecov hackers gained access to Monday.com source codeBleepingComputer · May 18, 2021 · Newsdownstream exposure, read-only source code copy, customer URL file, key rotation, no modification
The estate in the replay is illustrative and drawn at the level of attack stages; the victim's real environment is not public. Verdicts describe what a control does when it is in place before the incident. They are not an assessment of the victim's security program.