Potential Weaknesses in the OpenAI Sandbox that Could Have Contributed to the Hugging Face Incident

Sometimes clean architecture can prevent even attacks we have not foreseen.

One thing that kept jumping out at me from the Hugging Face incident actually had to do with the sandbox itself. This is not the first time an agent "escaped a sandbox," which does speak to the capabilities of the agent, but also sort of says something about the sandbox. You are not really supposed to be able to get out of them (when done right).

In fact, there is one specific aspect of the sandbox OpenAI used that one could argue is actually a known vulnerability or at least a weakness. To give some context, let me digress back to my netsec researching days…

Why our research project almost got blocked at Princeton

I was involved in a project under Maria Apostolaki and Jennifer Rexford (now university provost) that involved running a programmable switch to forward packets for a machine that was open to the Internet (this was the Tango Project which later won the Applied Networking Research Prize at IETF 122 in Thailand). The switch also had a "control" interface where you could log in and configure its settings and the control interface was on Princeton's trusted network behind the firewall. This project almost didn't get approved at Princeton. The security team was worried that if hackers gained access to the switch through the forwarding interfaces connected to the Internet, the hackers, having compromised our switch, would use that to launch attacks inside the Princeton firewall via the control interface. We did go ahead with the project but only after showing there was no way any packet on the switching interfaces could reach the control interface or operating system on the switch (this configuration is sometimes called out-of-band management).

The main takeaway from this: Princeton security already knew at an architecture level that any host that lived both inside the firewall and outside the firewall on the public Internet risked being a bridge between the Princeton network and the outside world, potentially bypassing the controls put in place at the firewall.

Fast forward to OpenAI's sandbox

The postmortem says that the AI agent was sandboxed except for a package "proxy" that was capable of fetching software packages from the Internet. If we draw a logical network diagram for this, we can immediately see there is a bridge host breaking the sandbox architecture. The proxy can talk to the AI agent on one interface and then talk to the Internet on another. This is exactly the same configuration the Princeton security team did not tolerate on their own network.

In a more rigorous network sandbox the hosts don't even know that another network exists. The proper implementation is to divide the infrastructure between hosts and packet forwarders where the packet forwarders have 1) no addresses reachable by the hosts in the sandbox and 2) have rules such that packets in the sandbox are never forwarded to a port not in the sandbox. One should always assume that any host in the sandbox (or reachable by the sandbox) is compromised. If a sandbox is made under these assumptions, even running servers from 20 years ago with no patches will not allow the AI agent to escape. The only risk would be exploiting a flaw in the packet forwarding architecture (so that rule 2 or 1 is broken) but the attack surface is vastly reduced. Enforcing these types of rules is fundamental to both physical and logical forwarding systems and the attack surface is so much smaller than a full OS (particularly one with open ports and protocols like a proxy server).

Note that in this type of sandbox, the package proxy idea doesn't really work. No one can actually talk to the Internet and that is the point. Ideally a full package repository should be stored in a storage device in the sandbox and then the package proxy just pulls from that disk instead of talking to the Internet.

Actually, if you really can't sleep just plug in some switches with a bunch of ethernet cables and make sure none of them go to the Internet. And don't give the agent a wifi antenna. For really scary stuff, a physical layer 1 sandbox is sort of the best (although I guess it would be a full datacenter offline to run GPT, maybe that's why they didn't do this option). Not fully the same thing but at the Princeton Plasma Physics Lab all wires going between their control room and plasma device go through an "optical break" where the electric signals are briefly converted to fiber optic to avoid an EMP carried over the wires from frying the control room hardware. Sometimes even being on the same wire is just too risky.

What is the actual takeaway?

You might be wondering what the actual takeaway is other than I told you so. I think there is an important story about defense in depth. This incident would never have happened if the package proxy had not been in a potentially compromising bridge position between the two networks. This was key to the vulnerability chain the agent ultimately exploited.

I think there is also a story about architecture and first principles. I spent a lot of time in academia reasoning about first principles and aggressively defining threat models. From a first-principled approach the proxy server is an obvious weakness. Interestingly, in industry security first principles actually get much less weight compared to practical attack patterns. I think this is due to the vast number of vulnerabilities (can't fix them all) and the relatively high rate of exploitation for simple vulnerabilities compared to complex vulnerability chains (there are 5,376 DDoS attacks each hour and there was one Hugging Face incident in the last year). I do wonder sometimes if AI is going to change this game by making complex attack patterns easier. I think security principles are the true way to get ahead of security whack a mole.

And finally I think there is an important lesson to be said about sandboxes. It's just not supposed to work like this. Even earlier this year Claude Mythos Preview escaped a sandbox during Anthropic's own testing and emailed the researcher running the evaluation. The idea is you make the sandbox simple but architecturally sound, and then everything in the sandbox gets pwned and nothing gets out. In theory a sandbox should be at a very low layer and rely on extremely primitive controls to ensure that at an information level nothing escapes. When a "sandbox" starts to turn more into a whitelist, or worse blacklist, it's not really a sandbox. I have a suspicion that given the race to AI and the hassle of making really good sandboxes (think no Internet), sort of good sandboxes slip by. There might be value in the community coming together and defining what a good AI sandbox actually is.

One thing that is important to remember for security against AI agents is that AI agents do have some similarities with traditional adversaries (e.g., exploiting CVEs, performing lateral motion, chaining exploits). Even in this new future, it's important to keep the traditional security basics strong and build on them to further protect against stronger agentic adversaries.