Skip to main content
Back
IntrusionGoogle Cloud · SaaS
10 min read

Codecov Bash Uploader supply-chain compromise

Victim
Codecov and its CI users; Codecov gives no count of affected customers, HashiCorp, Twilio, Rapid7, Mercari and Coalition describe their own exposure, and monday.com's account is known through a news report of its SEC filing
When
Jan 31, 2021 – Apr 18, 2021Disclosed Apr 15, 2021
Entry
An HMAC key for a Google Cloud Storage service account, left in an intermediate layer of the public Codecov Self-Hosted Docker image; when and how the attacker found it is not stated.

An attacker extracted an HMAC key for a Google Cloud Storage service account from a layer of Codecov's public Self-Hosted Docker image. From January 31, 2021 the key was used to change the Bash Uploader in Cloud Storage, and CI jobs that fetched and ran it sent their git remote and environment variables to a server outside Codecov until a customer's checksum check exposed the change on April 1. GitHub tokens taken this way reached customers' repositories; whether a cloud provider key taken from CI was used is unconfirmed.

What happened

External Parties
Receiving servers
Docker Hub
Shasum check
Internet
Cloud Storage APIHMAC-signed requests
Customers' CI
Codecov's Google Cloud
+ cloned repositories
+ internal credentials
+ personal data records
Customers' Repositories
+ keys kept in CI
+ production environments
+ use unconfirmed
Customers' Clouds
Bash Uploader in CI
CI Environment Variables
Service Account HMAC Key
Bash Uploader Object

Replaying the Codecov January 2021 intrusion

  1. 1

    A key in a public image

    Not dated

    An HMAC key for a Google Cloud Storage service account is extracted from an intermediate layer of the public Codecov Self-Hosted Docker image.

  2. 2

    The uploader changed in Cloud Storage

    Jan 31 to Apr 1, 2021, in 108 windows

    Cloud Storage accepts requests signed with the leaked key from outside Codecov, and the Bash Uploader object is replaced with a version that sends data out.

  3. 3

    CI environments sent out

    Jan 31 to Apr 1, 2021

    Customers' CI jobs fetch the changed script and run it, and it posts each job's git remote and environment variables to a server outside Codecov.

  4. 4

    Customer systems reached

    Jan 31 to Apr 18, 2021; cloud use unconfirmed

    GitHub credentials taken from CI reach customers' repositories, which are cloned or accessed; cloud provider keys taken from CI may have been used, which no source confirms.

  5. 5

    Found and closed

    Apr 1 to 15, 2021

    A customer's checksum check exposes the change on April 1, 2021 and Codecov remediates the script that day; Codecov revokes the key, rotates its production keys and emails affected users on April 15.

Impact

The Bash Uploader in Codecov's Google Cloud Storage was changed in 108 windows of time between January 31 and April 1, 2021, and each CI job that ran it sent its git remote and environment variables to a server outside Codecov. Customers report an exposed release-signing key, exposed or possibly exposed GitHub tokens, cloned repositories and records of personal information in source code. No source gives the number of affected customers or confirms that a cloud provider key taken from CI was used.

Outcome

Where cloud-provider controls fit

The key sat in a public image layer, where no cloud-provider control looks, and the script ran inside your CI jobs with whatever their environment held. The cloud-side decision was Codecov's: a Google Cloud Organization Policy constraint that refuses service account HMAC-signed requests would have refused the key, provided nothing Codecov ran depended on such requests. An IAM deny policy on object writes limits who can replace the script, and a locked retention policy limits when a published object can be replaced; the sources do not name the key's service account or say how the script was published. On your side, jobs that reach your clouds through federated identities, not keys stored in CI variables, leave no cloud key in the environment a script can send out; whether any cloud key taken this way was used is unconfirmed. An egress allow-list caps what leaves a self-hosted runner. The checksum check in your pipeline, your CI service's hosted runners and your GitHub tokens own the rest: no cloud-provider policy reaches them.

Could this have been prevented?

In hindsight, not as a whole. 1 control would have refused a step outright, and 10 controls would have limited one. 6 steps have no cloud-provider control. The counts below are control assessments, not attack stages: several can address the same step. The same replay with the controls in place before the incident shows where each phase would have been refused or bounded.

1
Prevented
10
Contained
6
Not covered

How it could have been prevented

The same replay with the controls in place. Green marks where a step is refused, blue where it is bounded, and red where no control reaches it. Open a row to read the control, why it acts, and its caveat.

External Parties
Receiving servers
Docker Hub
Shasum check
Internet
Cloud Storage APIHMAC-signed requests
Customers' CI
Codecov's Google Cloud
+ cloned repositories
+ internal credentials
+ personal data records
Customers' Repositories
+ keys kept in CI
+ production environments
+ use unconfirmed
Customers' Clouds
Bash Uploader in CI
CI Environment Variables
Service Account HMAC Key
Bash Uploader Object

Replaying the Codecov January 2021 intrusion

1 · A key in a public image

1 Not covered

2 · The uploader changed in Cloud Storage

1 Prevented3 Contained7 on other clouds

3 · CI environments sent out

1 Contained2 Not covered3 on other clouds

4 · Customer systems reached

6 Contained2 Not covered8 on other clouds

5 · Found and closed

1 Not covered

Key facts

  • Codecov's root cause analysis said the attacker extracted an HMAC key for a Google Cloud Storage service account from an intermediate layer of the public Codecov Self-Hosted Docker image; when and how was not stated
  • Per Codecov, the changed Bash Uploader was in effect during 108 windows of time between January 31 and April 1, 2021; their lengths were not stated
  • One added line posted the output of git remote -v and env to a third-party server, and Codecov said it was the only change to the script
  • On April 1, 2021 a customer reported that the downloaded script's SHA256 did not match the one published on GitHub; Codecov remediated the script that day and emailed affected users on April 15
  • HashiCorp reported its release-signing GPG key exposed with no evidence of use; Twilio reported an exposed GitHub user token and Coalition a possibly exposed one, with repositories cloned at both; Mercari reported 27,889 records exposed in its source code
  • No source gave the number of affected customers or confirmed that a cloud provider key taken from CI was used; unnamed investigators told Reuters of access to hundreds of customer networks, which Codecov did not confirm

Sources

Every fact in this breakdown comes from the public reporting below. The replay is a model of that reporting, not a reconstruction of the environment it describes.

  1. 1.
    Bash Uploader Security UpdateCodecov · Apr 29, 2021 · Vendor reportstart date, entry vector, Google Cloud Storage key, malicious change, exfiltrated data, affected products, self-hosted scope, discovery, remediation, customer notification, indicators of compromise, law enforcement, forensic firm, attribution not determined
  2. 2.
    Post-Mortem / Root Cause Analysis (April 2021)Codecov · Jun 8, 2021 · Vendor reportroot cause, HMAC key, service account, Docker image layer, Google Cloud Storage object change, detection, response roles, recovery, Docker image rebuild, storage monitoring, new uploader, key management, federal investigators, no other intrusions
  3. 3.
    Codecov Releases New Detections for Supply Chain CompromiseCISA · Apr 30, 2021 · Advisoryofficial confirmation of dates, customer guidance, certificate reissue
  4. 4.
    HCSEC-2021-12 - Codecov Security Event and HashiCorp GPG Key ExposureHashiCorp · May 4, 2021 · Vendor reportdownstream exposure, signing key, CI pipelines, response start date, key rotation, re-signing, no evidence of misuse
  5. 5.
    Twilio's Response to the Recent Codecov VulnerabilityTwilio · May 4, 2021 · Vendor reportdownstream exposure, Codecov notification, GitHub notification, user token, cloned repositories, email addresses, secret rotation, no modification
  6. 6.
    Rapid7's Response to Codecov IncidentRapid7 · Sep 3, 2024 · Vendor reportdownstream exposure, single CI server, source code repositories, internal credentials, customer alert data, external forensics, no production access
  7. 7.
    Mercari's response to the Codecov vulnerability and related notification on personal information exposureMercari · May 22, 2021 · Vendor reportdownstream exposure, credential revocation, GitHub notification, GitHub access dates, source code access, exposed record counts, regulator report, no damage to users
  8. 8.
    Incident report in response to the recent Codecov breachCoalition · May 7, 2021 · Vendor reportdownstream exposure, GitHub notification, read-only user token, clone date, secret rotation, no PII, no modification
  9. 9.
    Codecov hackers breached hundreds of restricted customer sites: sourcesReuters (syndicated on Yahoo News) · Apr 19, 2021 · Newsreported scope, reported automation, FBI investigation, customer statements, customer base
  10. 10.
    Codecov hackers gained access to Monday.com source codeBleepingComputer · May 18, 2021 · Newsdownstream exposure, read-only source code copy, customer URL file, key rotation, no modification

The estate in the replay is illustrative and drawn at the level of attack stages; the victim's real environment is not public. Verdicts describe what a control does when it is in place before the incident. They are not an assessment of the victim's security program.

More cloud attack breakdowns

All breakdowns