Skip to main content
Back
DemonstrationAWS · SaaS
6 min read

Permiso's Bedrock honeypot: LLMjacking through a leaked AWS key

Environment
Permiso's honeypot AWS account (staged)
When
Jun 25, 2024 – Aug 6, 2024Disclosed Oct 3, 2024
Entry
Permiso placed a long-lived AWS access key and its secret in a file on GitHub, on purpose, to stage what had happened at its clients; actors began using it within minutes.

Permiso P0 Labs placed a long-lived AWS access key and its secret in a file on GitHub on June 25, 2024, on purpose, to stage the Bedrock abuse it had seen at its clients. Within minutes actors used the key; actors from twelve ASNs checked which Bedrock models it could use and called the console's own model access APIs, and the quarantine policy AWS attached on August 1 denied no Bedrock action. From August 5 one actor ran more than 75,000 successful model invocations in two days for a sexual roleplay service, and on August 6 AWS removed the account's ability to launch additional Bedrock resources.

What the demonstration did

External Parties
Permiso P0 Labs
Actors · twelve ASNs
Roleplay service
+ 2 more
Internet
AWS APIleaked long-lived key
GitHub
Public GitHub Repository
Amazon Bedrock
Permiso's Honeypot AWS Account
Key File
Key's IAM Identity
Bedrock Model Access
Invocation Logging

Replaying Permiso's Bedrock honeypot demonstration / staged in Permiso's own AWS account, June to August 2024

  1. 1

    A bait key on GitHub

    Jun 25, 2024

    Permiso places a long-lived AWS access key and its secret in a public GitHub file on purpose, and actors use the key from the internet within minutes.

  2. 2

    Finding and opening the models

    Jun 25 – Aug 6, 2024

    Actors from twelve ASNs check which Bedrock models the key can use and call the console's own model access APIs; whether those requests opened the models is not stated.

  3. 3

    The quarantine gap and the hijack

    Aug 1–6, 2024

    AWS attaches its quarantine policy to the key on August 1, but the policy denies no Bedrock action; from August 5 one actor runs more than 75,000 successful model invocations in two days.

  4. 4

    Blocked and reported

    Aug 6 and 15, 2024

    AWS removes the account's ability to launch additional Bedrock resources on August 6, and Permiso reports the findings to Anthropic and AWS on August 15.

Impact

Actors used the deliberately leaked key from June 25 to August 6, 2024 and ran more than 75,000 successful model invocations in two days for a sexual roleplay service. The quarantine policy AWS applied on August 1 did not cover Bedrock, and AWS added Bedrock actions to it on October 2.

Outcome

What the honeypot shows about your AWS keys

A long-lived access key that reaches a public repository is used within minutes, and AWS's quarantine is reactive: in August 2024 it denied no Bedrock action. What you control is what the key can do before anyone finds it. In the member accounts of an organization, a service control policy can refuse new long-term access keys and IAM users' calls that carry no MFA, leave Amazon Bedrock off the services an account may use, deny model access requests outside approved administrators, though a model already enabled stays usable, and allow invocations only for approved models and from your own networks. GitHub push protection can refuse a push that carries the key, unless the pusher can bypass it. When AWS restricts an abused account, and what its quarantine denies, is AWS's call.

Would the controls have stopped it?

In hindsight, no control would have stopped it. 7 controls would have limited how far it went. 3 steps have no cloud-provider control. The counts below are control assessments, not attack stages: several can address the same step. The same replay with the controls in place before the incident shows where each phase would have been refused or bounded.

0
Prevented
7
Contained
3
Not covered

How it could have been prevented

The same replay with the controls in place. Green marks where a step is refused, blue where it is bounded, and red where no control reaches it. Open a row to read the control, why it acts, and its caveat.

External Parties
Permiso P0 Labs
Actors · twelve ASNs
Roleplay service
+ 2 more
Internet
AWS APIleaked long-lived key
GitHub
Public GitHub Repository
Amazon Bedrock
Permiso's Honeypot AWS Account
Key File
Key's IAM Identity
Bedrock Model Access
Invocation Logging

Replaying Permiso's Bedrock honeypot demonstration / staged in Permiso's own AWS account, June to August 2024

1 · A bait key on GitHub

3 Contained1 Not covered6 on other clouds

2 · Finding and opening the models

2 Contained6 on other clouds

3 · The quarantine gap and the hijack

2 Contained1 Not covered3 on other clouds

4 · Blocked and reported

1 Not covered

Key facts

  • Permiso placed a long-lived AWS access key and its secret in a file on GitHub on June 25, 2024 to simulate what had occurred at earlier clients; within minutes, actors tried the key against Amazon Bedrock and other services
  • Between June 25 and August 6, 2024, actors from twelve ASNs called nine Bedrock APIs, among them PutUseCaseForModelAccess, CreateFoundationModelAgreement and PutFoundationModelEntitlement; whether those requests opened the models or Permiso enabled them before the leak is not stated
  • AWS found the exposed key on June 25 and again on August 1, 2024, when it attached the AWSCompromisedKeyQuarantineV2 policy; the policy then denied no Bedrock action, and AWS added five Bedrock actions to it on October 2, 2024
  • Over two days from August 5, 2024, the honeypot recorded more than 75,000 successful model invocations, almost all from one IP address and mostly against Claude Sonnet
  • On August 6, 2024 AWS removed the account's ability to launch additional Bedrock resources, about 35 hours after the invoke attempts reached the thousands and about 42 days after the key was first used maliciously
  • Krebs on Security reported a $3,500 AWS bill for what Permiso called a two-day experiment; Permiso's own timeline runs from June 25 to August 6, 2024

Sources

Every fact in this breakdown comes from the public reporting below. The replay is a model of that reporting, not a reconstruction of the environment it describes.

  1. 1.
    When AI Gets Hijacked: Exploiting Hosted Models for Dark RoleplayingPermiso (P0 Labs) · Oct 3, 2024 · ResearchHoneypot setup, leak on GitHub, Bedrock API sequence, model access request APIs, invocation count, source of the invocations, AWS notices and quarantine, AWS restriction, AWS statement
  2. 2.
    A Single Cloud Compromise Can Feed an Army of AI Sex BotsKrebs on Security · Oct 3, 2024 · NewsLeak on GitHub, key used within minutes, bill for the experiment, quarantine restrictions and Bedrock
  3. 3.
    AWSCompromisedKeyQuarantineV2 (AWS Managed Policy reference)Amazon Web Services · Oct 2, 2024 · AdvisoryQuarantine policy content, Bedrock actions in the policy, edit date

The estate in the replay follows the researchers' description of their own test environment. Verdicts describe what a control does when it is in place before the run. They are not an assessment of any company's security program.

More cloud attack breakdowns

All breakdowns