Cloudflare published a postmortem on October 26 for a tenant isolation bug in its Workers AI platform. Between August 27 and September 9, the GLM 5.3 endpoint returned 155 responses that contained text from other customers' prompts. Every one of them went to a prompt longer than 681,280 tokens. Cloudflare knew that number before the model launched.
The write-up by Kevin Flansburg and Michelle Chen runs 16 minutes and names its own mistakes. I read it twice. It is the best inference postmortem I have seen, and the failure it describes will happen to other people who run the same stack.
Two numbers that didn't agree
Workers AI runs GLM 5.3 with SGLang, an open-source inference engine, in a disaggregated setup. One pool of GPUs does prefill, which reads the prompt and produces the KV cache. That cache then goes over RDMA, through Mooncake, to a second pool of GPUs that generate the reply. The KV cache on each GPU is a shared pool of pages, all the same size, and every request gets a set of pages for its lifetime.
SGLang does not let you set the size of that pool. It takes whatever GPU memory is left after loading the weights and carves that into pages. The context length of the model is a separate setting. Nothing in the engine checks that the two agree.
On Cloudflare's prefill nodes the pool came to 681,280 tokens. The model was advertised at 1,048,576. The team saw the mismatch in the startup logs and the metrics before launch. They expected the engine to reject any prompt that didn't fit, and their benchmark with long prompts returned HTTP 200 with a clean stream, so they shipped. The tests checked status codes and throughput. Nobody read the output.
What the engine did with a prompt that didn't fit
When a prompt exceeds the pool, SGLang's scheduler marks the request as rejected, records the error to send back, and swaps the prompt for a placeholder of one token so the request can drain through the normal batch path without allocating memory. On a single node this works. The placeholder runs, the batch finishes, the client gets a 400.
In disaggregated mode it broke. The admission queue between prefill and decode checked whether a request was finished, not whether it had been rejected. So the rejected request kept going. Prefill computed one page of KV cache for the placeholder and transferred that one page. Decode had reserved eight pages for the full prompt, received a "transfer complete" message, and ran attention across all eight. One page held the placeholder. The other seven held whatever earlier requests had left behind: other users' prompts and partial generations.
The client got a 200. The token count in the response matched the long prompt. The reply was coherent, it arrived faster than a prefill of eight pages should take, and it was built from somebody else's data. Every request over the threshold did this, from launch day on.
Someone had filed this exact bug against SGLang as issue #30233 on July 6, seven weeks before GLM 5.3 went live. Cloudflare found it while investigating the incident.
Forty minutes to pull the model, one day to fix it
Gadi Evron of Knostic filed the report through HackerOne on September 8 at 23:11 UTC. Cloudflare started verifying at 08:00 the next morning, declared an incident at 13:22, and reproduced the leak with two independent accounts within 25 minutes. The model went private at 14:00. By 14:35 they had the root cause, confirmed because a canary prefill node with a bigger pool was immune. All prefill nodes had the corrected config by 14:52. A rebuilt deployment with a Cloudflare patch to the engine went out at 19:04, and the model was public again at 19:40.
The config fix was blunt. They matched the GLM 5.3 settings to GLM 5.2, which freed enough memory that the prefill pool now holds about 2.04 million tokens, double the context window. A valid prompt can never reach the rejection path. The engine fix retires a rejected request at the admission queue with its original error before any KV transfer happens. That landed upstream in SGLang v0.5.20, with follow-ups in v0.5.21 on October 2 and v0.5.22 on October 19. Cloudflare met the SGLang maintainers at RadixArk on September 10, and the two teams held the disclosure until October 26 so the fixes could ship first.
Cloudflare says the 155 affected requests look like ordinary use of long prompts once its own testing and the researcher's are excluded, and that it found no evidence anyone exploited the bug on purpose. The post describes no direct customer notification, only the status page update and the model going private. If you sent GLM 5.3 a prompt over 681,280 tokens in that window, ask.
The same bug is waiting in other people's clusters
Cloudflare's own framing is that this is one bug in a class. A KV page is freed, reused, or read while something else still has a claim on it. On a single decode node, token generation, host offload, the transfer receiver, and the abort path can all hold the same page at once. Across nodes, prefill and decode keep separate views of who owns what, and a dropped abort message or a late transfer leaves one side wrong. The post lists eleven upstream SGLang changes for races in this family, most of them from the last two months.
Their longer answer is a tag on every page, checked at read time, so a mismatched page fails the request instead of returning someone else's text. The prototype cost under 1% in throughput and tail latency. They turned it on for GLM 5.2 in mid-August and turned it off shortly after over reliability problems. It was also tied to specific attention backends, so it isn't a general fix yet. They are now applying formal verification to the RDMA transfer protocol, the same approach they used on their DNS and consensus code.
I have run shared infrastructure for a long time, and the lesson here is older than LLMs. Memory shared across tenants with no enforced isolation is an incident with a delay on it. The engine allocates pages from a common pool, hands raw references across a network, and trusts bookkeeping on two machines to agree. That is the design, and it is the design everyone serving large models at scale uses, because the alternative costs GPU memory. Cloudflare got burned by a known mismatch and a known upstream bug, and a test suite that never read a response body let both through.
If you run SGLang in prefill/decode mode, Cloudflare's advice is short. Read the KV pool size out of the startup log on every node, after every change to weights, quantization, parallelism, or memory fraction, and lower the context length if the pool is smaller. Do not rely on a proxy to enforce the limit. Upgrade to v0.5.22. Turn on deferred KV release on both roles. And add a test that puts a marker in the prompt and checks the output for anyone else's marker, because an HTTP 200 proves nothing.
One commenter on Hacker News wondered why more serverless inference providers haven't disclosed something like this. My guess is they haven't looked. Cloudflare had a bug bounty program and a canary node. Most providers running the same engine have neither, and the bug sat in the SGLang tracker from July until a paying customer's data came out the other end. I expect at least one more disclosure of this shape before the year is out, and I expect it to come from a provider that didn't find it themselves.
