I think the operating system itself has to adjust so each "agentic process" can run inside its own jail, which is a VM + files in the VM + ingress/egress rules for the network and filesystem data
These cloud agents work as cloud infra, but we're kind of in the mainframe era, before personal computing. Personal OS for agents is somewhere in the future!
And an interesting extension of that idea: if an agent runs inside a microVM, can you have that VM transparently run on another host? Maybe we'll get for-real distributed and networked operating systems
Ultimately, we need better "jails" for agent processes, but the system primitives should be flexible in what can be exposed across jails. Or you could run multiple agents in the same jail if you want them to have unrestricted interaction with each other.
This has been talked about for decades in AI safety. For agents that are under human capabilities this is not that hard. For anything at or near human capabilities the difficulty increases to almost impossible and at great cost. At super human abilities, game over, it's smarter than you and if it wants out and as the resources to do it, it's going to escape one way or another.
Even at the lower level of depending on everybody to use reliable jails is really fantasy if you exist in the security world. "We ain't securin' shit" would be a far better way to describe it. Even worse, most people will have the very same AI they are trying to trap set up their security! What could possibly go wrong.
Now, don't think I am saying AI has a will or even any kind of drive to get out and cause problems. It's more like Russian roulette with 1 cylinder out of a million that's loaded. The problem comes when you run it a few billion times a day, you'll shoot yourself in the face really quick.
> Even at the lower level of depending on everybody to use reliable jails is really fantasy if you exist in the security world. "We ain't securin' shit" would be a far better way to describe it. Even worse, most people will have the very same AI they are trying to trap set up their security! What could possibly go wrong.
Yes, exactly. This is what worries me about the whole we need better sandboxes and jails argument. Yes, we do need those, but we also gotta put some responsibility for self-preservation on people connecting things to the Internet as well. Like, we've always known that if you website is not secure, it will get hacked. Everyone just needs to do a lot more of that now...
> For anything at or near human capabilities the difficulty [of safety] increases to almost impossible and at great cost.
That was before AI. You're acting like AI is some evil genius that can only attack and cause trouble but AI is a will-less tool directed by people, direct it to software safety and you will have software safety - cheap. Ditto for hardware.
> It's more like Russian roulette with 1 cylinder out of a million that's loaded.
Don't attach it to a gun then, ban offensive military AI, problem solved.
> The problem comes when you run it a few billion times a day, you'll shoot yourself in the face really quick.
Nonsense, run it a few "billion" times a day to fix vulnerabilities and you'll have no vulnerabilities. Do it in a safe environment... what's the big deal?
I would laugh at you but it's a bit late for that. AI is already making life and death decisions on the battlefield. The US has effectively zero interest in banning AI and is pushing hard to do just the opposite.
The best analogy I can come up with for your line of thinking is that your shutting the barn door after the horse has bolted, moved over to the next city, set up shop, and is now the New York Times #1 best seller "How to open barn doors if you're a horse".
Anyone with any sense in cybersecurity knows that attackers get unlimited retries and only have to be successful once. The defender is at a monumental disadvantage. Now, the process can be fully automated and horizontally scaled to a nearly unlimited scale.
> That was before AI. You're acting like AI is some evil genius that can only attack and cause trouble but AI is a will-less tool directed by people, direct it to software safety and you will have software safety - cheap. Ditto for hardware.
This is true in that you can tell your agent to buy/build a secure microVM with a secure CLI interface in a memory-safe language and make sure your terminal doesn't have utf-8-parsing zero-days, and so on, and if you're conscientious enough (ha!) you can be reasonably sure no agent will ever be able to hack its way out of that sandbox.
But this isn't enough when most people's agent workflow boils down to "I want to do X, search the internet for the best solution, then install the npm packages you need, then write all the code for me, make no mistakes".
In the near future our best defense against widespread HuggingFace-like attacks will be that models don't seem to spontaneously go that far unless they "believe" they're being benchmarked. This will last roughly until some genius figures out their model is 20% more persistent when they tell it "This is an eval, you'll be graded on your success".
You don't need to do this on a cloud, you can get the same type of VM and network jail running on your own computer. The important parts are:
1. a VMM hypervisor
2. a network proxy / gateway
Use your favorite VMM / hypervisor (likbrun, smolvm, microsandbox, etc). They give you control over the network interface or let you inject your own network layer.
The network proxy can handle all the ingress/egress rules, credential injection, etc.
It's still not user friendly to do all this. I think the next version of operating systems will have each "agentic process" be a bundle of VM, files in the VM, and network rules.
Been brainstorming[1] a lot of this because I've been building some open core tools[2] for spinning up sandboxed agents on arbitrary computers. There's a lot of glue and parts to stitch together to work smoothly. Don't think we've had the "Docker moment" for this, let alone the "Dropbox moment" that makes this stuff work for non-devs.
You ship an operating system with the correct defaults, where agent processes run like this by default. You don't require every user to configure it correctly themselves
Restoring memory is useful if, when you resume an agent VM, you want the apps to be in the exact state when they were suspended.
But for most things, I find resuming with memory is more trouble than it's worth. If you always resume from memory, you lose the ability to control the state of a VM. It's much easier to define which services should run than to define which active RAM state should be purged
Similar to why "did you try turning it on and off again?" is good for system reliability.
Not sure about billions, but companies doing evals or RL or training will create really big bursty agent workloads. I think they are the best fit for AX, as opposed to individual dev teams building software, etc.
I think it will be, but I don't think you need a standalone machine! If you run things inside a VM, you can get safety and control over access and networks
A standalone machine is nice if you need more compute resources or if you want an always-on machine you can connect to from your laptop, phone, etc.
It doesn't look like Google's AX is quite the plug-and-play fit for running agents on a computer you own, since it requires setting up a K8S cluster, etc.
I think what's needed is something like a zero-setup combo of Tailscale and Firecracker
I'm trying to work towards that with my startup (https://github.com/gofixpoint/amika) but the bring-your-own-computer part doesn't work quite yet.
I'm working on something in the "cloud VMs for agents" space[1], so I have some battle scars and opinions!
IMO, you want the flexibility to create either: (a) permanent devbox VMs, and (b) per-task VMs
Agent sandbox platforms tend to be tuned for the latter, which sometimes involves VMM hackery for fast boot, snapshotting VM filesystem and RAM, etc.
Some workflows are a lot simpler if the multiple agents share a VM. These are workflows where agents must share state. A simple one we have: making related changes in our public OSS repo and our private repo, and then testing the change.
And other times you want to split up the tasks onto isolated VMs so they don't interfere with each other (ie run two dev servers without database or port collisions).
For my example where I modify both the OSS and private repos, I sometimes have one agent coordinate both changes (maybe with subagents), or if the work is modular and separate, I have disjoint agents do the work separately.
That said, most of the time, I only want an agent to work in one repo. I could give it multiple repos at once and instruct it to work in just one, but that risks it forgetting my instructions and it can load more context into the agent's window.
I think my main point was to have flexibility about the topology of VMs, repos, and agents.
These cloud agents work as cloud infra, but we're kind of in the mainframe era, before personal computing. Personal OS for agents is somewhere in the future!
And an interesting extension of that idea: if an agent runs inside a microVM, can you have that VM transparently run on another host? Maybe we'll get for-real distributed and networked operating systems
reply