What happened
Replaying the JW Player intrusion
- 1
The monitoring dashboard becomes public
Before discoveryAn AWS load balancer admitted internet traffic to the unauthenticated Weave Scope dashboard. The cluster controller had reversed an engineer’s manual restriction on its security group.
- 2
A privileged monitoring agent exposes host capacity
Day 3JW Player traced the miner running on its development and staging nodes to the monitoring agent, which had been deployed with privileged access to each host.
Impact
The miner consumed one CPU core on every development and staging node. JW Player reported uninterrupted service and no access to customer data.
Could this have been prevented?
Keep the dashboard’s network restrictions in its deployment configuration
For a dashboard used only by staff, declare the approved source networks in Kubernetes so the AWS load balancer keeps the same rules after a deployment. On current clusters, teams that can replace a privileged monitoring agent can also enforce the Baseline Pod Security Standards before the replacement runs.
Incident control assessments
- 1
- Prevented
- 1
- Contained
- 0
- Not covered
How it could have been prevented
The same replay with the controls in place. Green marks where a step is refused, blue where it is bounded, and red where no control reaches it. Open a row to read the control, why it acts, and its caveat.
Replaying the JW Player intrusion
1 · The monitoring dashboard becomes public
1 Contained2 · A privileged monitoring agent exposes host capacity
1 PreventedKey facts
- The dashboard had no sign-in requirement.
- AWS logs showed that Kubernetes had reversed a manual change to the load balancer’s security group.
- The miner used one full CPU core on each affected node.
- Responders isolated one node for investigation and replaced the cluster nodes.
- JW Player found no access to the Kubernetes API during the intrusion.
Sources
Every fact in this breakdown comes from the public reporting below. The replay is a model of that reporting, not a reconstruction of the environment it describes.
- 1.How A Cryptocurrency Miner Made Its Way onto Our Internal Kubernetes ClustersBrian Choy, JW Player DevOps · Sep 20, 2018 · Vendor reportResponding engineer’s firsthand investigation, observed resource use and recovery.
- 2.Security group rulesAmazon Web Services · undated · Vendor reportInbound source restrictions and the union of rules in attached security groups.
- 3.Pod Security AdmissionKubernetes · undated · Vendor reportNamespace enforcement, exemptions and the boundary at pod admission.
- 4.Pod Security StandardsKubernetes · undated · Vendor reportThe Baseline profile prohibits privileged containers and hostPath volumes.
- 5.Kubernetes v1.25: Pod Security Admission Controller in StableKubernetes · Aug 25, 2022 · Vendor reportRelease history for the present-day admission control.
The estate in the replay is illustrative and drawn at the level of attack stages; the victim's real environment is not public. Verdicts describe what a control does when it is in place before the incident. They are not an assessment of the victim's security program.