Addressing Cross-Request KV Cache Exposure in Workers AI
Security Incident Report: GLM-5.3 Model Vulnerability
On September 8, 2026, a security researcher from Knostic identified a critical tenant isolation vulnerability affecting the GLM-5.3 Workers AI model. The flaw, which persisted between August 27 and September 9, 2026, resulted in a limited number of instances where users received AI-generated responses containing data from other customers’ previous requests. Following the disclosure, the affected model was taken offline for 40 minutes to facilitate an immediate investigation and remediation. Cloudflare has confirmed that no other models hosted on the Workers AI platform were impacted by this issue.
Technical Root Cause Analysis
The vulnerability stemmed from an interaction between a configuration mismatch and a logic error within the SGLang inference engine, which utilizes a disaggregated prefill/decode architecture. In this setup, prefill GPUs generate the Key-Value (KV) cache, which is then transferred to decode GPUs for response generation. The KV cache utilizes a shared pool of fixed-size memory pages that are not zeroed out between requests for performance reasons, relying instead on the engine to manage ownership and lifecycle tracking.
The incident was driven by two primary factors. First, the GLM-5.3 model was deployed with a context window of 1,048,576 tokens, while the available KV cache pool on the prefill GPUs was limited to 681,280 tokens. Although this mismatch was known, pre-launch testing failed to detect the resulting data leakage because the system returned successful HTTP status codes without alerting operators to output incoherence.
Second, a bug in the SGLang engine’s handling of rejected requests occurred during disaggregated operations. When a prompt exceeded the available KV pool size, the scheduler marked the request as rejected and substituted a one-token placeholder. However, the admission logic in both prefill and decode nodes incorrectly identified these rejected requests as active. Consequently, the decode node proceeded to generate responses while reading from stale memory pages that still contained data from prior user sessions, leading to the cross-request data exposure.
Remediation and Hardening
Upon identifying the flaw, the engineering team implemented a two-fold mitigation strategy. To address the configuration gap, the KV cache pool was expanded to hold approximately 2.04 million tokens, ensuring that the pool size exceeds the maximum supported context window. This change guarantees that valid requests never trigger the problematic rejection path.
Furthermore, the team collaborated with the SGLang maintainers to patch the engine’s underlying logic. The updated configuration ensures that requests carrying a pending rejection are retired immediately at the admission queue level, preventing any KV transfer or bootstrap process. Additionally, the decode node now performs mandatory verification to ensure the number of tokens transferred matches the reserved memory allocation. These fixes, incorporated into SGLang v0.5.20 and subsequent releases, provide a robust defense against similar failure paths. The organization has also updated its internal validation protocols to monitor KV pool metrics directly against context length requirements, ensuring future deployments maintain strict isolation integrity.
