Skip to main content

Sandboxes

A sandbox is a small Linux machine that belongs to one conversation. The agent runs shell commands and edits files in it. The agent loop stays on the Distri server. Only the tools run inside the sandbox.

Distri runs sandboxes on OpenSandbox. The runtime sits behind one Rust trait, so you can plug in another one.

What a sandbox gives an agent​

  • A machine per conversation. Each thread gets its own sandbox. Two conversations never share files or processes.
  • A persistent /workspace. The sandbox mounts a volume at /workspace and sets HOME=/workspace. Files there survive pauses, restarts and new sandboxes for the same thread.
  • Four tools. Bash, Read, Write and Edit run inside the sandbox. Bash keeps its working directory between calls.
  • Every entry point. The tools come from the workspace, not the agent definition. An agent called over the API or A2A, a bot on Telegram, WhatsApp, Slack or Discord, an embedded agent in your product, and sub-agents started with invoke_agent all get the same sandbox for the same thread.

The agent declares nothing. If an agent already has its own tool named Bash, Read, Write or Edit, its own tool wins.

Turn it on​

The sandbox is a workspace default, like the default model.

  1. The server needs a sandbox provider. Set SANDBOX_URL and SANDBOX_API_KEY, and set DISTRI_SANDBOX_DEFAULT=on. See Configuration.
  2. Turn it on for the workspace. Add "sandbox": true to the workspace settings:
{
"sandbox": true
}

From then on, every run in that workspace gets the four tools. Workspaces without the setting are unchanged.

How it works​

Distri keeps one binding per thread in Postgres (sandbox_bindings). It records the provider, the sandbox id, the volume name, the state and the current directory.

Before each tool call, Distri makes sure the thread's sandbox is ready:

The sandbox isDistri does
runninguses it as is
pausedresumes it, then waits until it answers
gone, or never createdcreates a new one on the thread's volume, mounted at /workspace

Only one of these checks runs per thread at a time. Parallel tool calls in one turn share one sandbox.

Pause and resume. On Kubernetes, a pause deletes the pod and keeps the volume. A resume starts a new pod on the same volume. Anything on /workspace comes back. Memory, running processes and packages installed outside /workspace (for example with apt install) do not.

Expiry. A sandbox expires after OPENSANDBOX_TIMEOUT_SECS (one hour by default). The next tool call creates a new sandbox on the same volume, so the files are still there.

Timings. On Distri's Kubernetes node with gVisor, a new sandbox answers in about 6.4 s with the image already on the node. A resume answers in about 3.4 s. On a laptop k3d cluster, a create takes about 6 s and a resume about 2 s. With OpenSandbox in Docker mode, a create takes under a second and a resume under 100 ms.

Providers​

Distri ships one provider: OpenSandbox. It gives Distri a lifecycle API (create, pause, resume, delete) and execd, an agent inside each sandbox that runs commands and moves files. It runs on Docker for local work and on Kubernetes in production.

Other runtimes plug in by implementing the SandboxProvider trait from distri-core (distri_core::sandbox). E2B, Daytona, Modal, or your own containers or VMs all fit. Trimmed:

#[async_trait::async_trait]
pub trait SandboxProvider: Send + Sync + std::fmt::Debug {
/// Short stable name stored with bindings, e.g. "opensandbox".
fn name(&self) -> &'static str;

async fn create(&self, spec: &SandboxSpec) -> SandboxResult<SandboxInfo>;
async fn status(&self, sandbox_id: &str) -> SandboxResult<SandboxInfo>;
async fn pause(&self, sandbox_id: &str) -> SandboxResult<()>;
async fn resume(&self, sandbox_id: &str) -> SandboxResult<()>;
/// Delete the sandbox. Its volume is kept.
async fn delete(&self, sandbox_id: &str) -> SandboxResult<()>;

/// Find a sandbox by metadata (e.g. one a timed-out create still made).
async fn find_by_metadata(&self, key: &str, value: &str) -> SandboxResult<Vec<SandboxInfo>> { Ok(vec![]) }
/// Wait until commands and files can be served.
async fn wait_ready(&self, sandbox_id: &str, timeout: Duration) -> SandboxResult<()> { Ok(()) }

async fn exec(
&self,
sandbox_id: &str,
req: &ExecRequest,
on_chunk: Option<UnboundedSender<ExecChunk>>,
) -> SandboxResult<ExecOutput>;
async fn read_file(&self, sandbox_id: &str, path: &str) -> SandboxResult<Vec<u8>>;
/// Write a file, creating parent directories.
async fn write_file(&self, sandbox_id: &str, path: &str, content: &[u8]) -> SandboxResult<()>;
}

A SandboxSpec carries the image, the volume to mount at /workspace, environment variables, metadata and a timeout. The provider maps its own states onto Pending, Running, Pausing, Paused, Resuming, Stopping and Gone.

A second trait, SandboxBinder, maps a run's thread to a sandbox. It has two methods: ensure (running, paused or gone, as in the table above) and set_cwd. Distri Cloud implements it on Postgres. A self-hosted server can implement it on its own store.

Security​

Distri's production sandboxes run on a dedicated Kubernetes node (k3s) with these locks:

  • gVisor. Every sandbox pod runs under the gVisor runtime (runsc). An admission policy refuses any pod in the sandbox namespace that does not.
  • No open internet. A NetworkPolicy lets a sandbox reach only cluster DNS and an egress proxy. The proxy (Squid) allows HTTPS to an allowlist: package registries (PyPI, npm, crates.io, Debian) and GitHub. Everything else gets a 403. Private addresses, the node, other sandboxes and the cloud metadata endpoint are unreachable. pip, npm, cargo, git, curl and apt use the proxy by themselves.
  • Three locks on the API. Cloudflare Access sits in front of the sandbox API. OpenSandbox's API key guards /v1. Each sandbox is created with secureAccess, so traffic into it needs a per-sandbox token.
  • One volume per thread. A thread's files live on its own volume. Secrets such as LLM keys and connection tokens stay on the Distri server and never enter the sandbox.

Run it locally with Kubernetes​

This sets up OpenSandbox on a laptop cluster the same way production runs it: the Kubernetes runtime with the agent-sandbox controller as the workload provider. It uses k3d (k3s in Docker). kind works the same way.

Local differences from production:

  • No gVisor. Sandboxes use the default runtime (runc). gVisor needs a runtime on the node, which k3d does not have.
  • No ingress gateway. Distri reaches execd through the OpenSandbox server's proxy instead (SANDBOX_USE_SERVER_PROXY=true). OpenSandbox keeps that path for compatibility; production uses the gateway.
  • No egress proxy. Sandboxes can reach the internet.

You need Docker (Docker Desktop, Colima or OrbStack), k3d, kubectl, helm, git and jq. On macOS:

brew install k3d kubectl helm jq

1. Create a cluster​

k3d cluster create distri-sandbox --wait
kubectl get nodes

2. Install agent-sandbox​

kubectl apply --server-side -f \
https://github.com/kubernetes-sigs/agent-sandbox/releases/download/v1.0.11/sandbox.yaml
kubectl -n agent-sandbox-system rollout status deploy/agent-sandbox-controller --timeout=180s

3. Namespaces, permissions and an API key​

OpenSandbox 1.2.1's charts do not give the server access to agent-sandbox's Sandbox objects. Add that, plus the namespace the sandboxes live in:

kubectl create namespace opensandbox-system
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Namespace
metadata: { name: opensandbox }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: opensandbox-agent-sandbox }
rules:
- apiGroups: ["agents.x-k8s.io"]
resources: ["sandboxes", "sandboxes/status"]
verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: opensandbox-server-agent-sandbox }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: opensandbox-agent-sandbox }
subjects:
- { kind: ServiceAccount, name: opensandbox-server, namespace: opensandbox-system }
EOF

export SANDBOX_API_KEY=$(openssl rand -hex 24)
kubectl -n opensandbox-system create secret generic opensandbox-api-key \
--from-literal=api-key="$SANDBOX_API_KEY"

4. Install OpenSandbox​

The charts are not published to a Helm repository. Install them from the release tag:

git clone --depth 1 --branch release-1.2.1 https://github.com/opensandbox-group/OpenSandbox.git
helm dependency build OpenSandbox/manifests/charts/opensandbox

Save this as values-local.yaml. The charts pull from a registry in mainland China by default, so it points every image at Docker Hub:

base:
enabled: true
fastSandbox:
namespaces:
createSystem: false # we made opensandbox-system above

opensandbox-server:
server:
image:
repository: docker.io/opensandbox/server
tag: release-1.2.1
env:
- name: OPENSANDBOX_SERVER_API_KEY
valueFrom:
secretKeyRef: { name: opensandbox-api-key, key: api-key }
configToml: |
[server]
host = "0.0.0.0"
port = 80

[runtime]
type = "kubernetes"
execd_image = "docker.io/opensandbox/execd:release-1.2.1"

[kubernetes]
namespace = "opensandbox"
workload_provider = "agent-sandbox"
sandbox_create_timeout_seconds = 300

[agent_sandbox]
shutdown_policy = "Delete" # pause deletes the pod, keeps the volume

[storage]
volume_default_size = "5Gi"

# Production adds gVisor here:
# [secure_runtime]
# type = "gvisor"
# k8s_runtime_class = "gvisor"

opensandbox-controller:
controller:
image:
repository: docker.io/opensandbox/controller
tag: release-1.2.1
namespaces: [opensandbox]
snapshot:
imageCommitterImage: docker.io/opensandbox/image-committer:release-1.2.1

ingress-gateway:
enabled: false
opensandbox-node-agent:
enabled: false
fast-sandbox:
enabled: false

Then install:

helm upgrade --install opensandbox OpenSandbox/manifests/charts/opensandbox \
-n opensandbox-system -f values-local.yaml --wait --timeout 10m
kubectl -n opensandbox-system get pods

The first install pulls several images and can take a few minutes.

If the pods sit in ContainerCreating or ImagePullBackOff, the cluster cannot pull images by itself. This happens with some Docker setups on macOS. Pull on the host and import them, then delete the stuck pods:

for i in opensandbox/server:release-1.2.1 opensandbox/controller:release-1.2.1 \
opensandbox/execd:release-1.2.1 python:3.12-slim; do docker pull "$i"; done
k3d image import -c distri-sandbox opensandbox/server:release-1.2.1 \
opensandbox/controller:release-1.2.1 opensandbox/execd:release-1.2.1 python:3.12-slim
kubectl -n opensandbox-system delete pod --all

Import python:3.12-slim too, or whatever image you set in OPENSANDBOX_IMAGE. The sandboxes themselves use it.

5. Reach the API​

kubectl -n opensandbox-system port-forward svc/opensandbox-server 8090:80

In another terminal:

curl -s http://localhost:8090/health
curl -s -H "OPEN-SANDBOX-API-KEY: $SANDBOX_API_KEY" http://localhost:8090/v1/sandboxes

The first call answers {"status":"healthy"}. The second answers an empty list: {"items":[],...}. A call without the key gets a 401.

To try a sandbox by hand, create one with a volume, wait for Running, then run a command through its execd:

H="OPEN-SANDBOX-API-KEY: $SANDBOX_API_KEY"
ID=$(curl -s -H "$H" -H 'content-type: application/json' http://localhost:8090/v1/sandboxes -d '{
"image": {"uri": "python:3.12-slim"},
"entrypoint": ["sleep", "infinity"],
"resourceLimits": {"cpu": "500m", "memory": "512Mi"},
"timeout": 900,
"volumes": [{"name": "workspace", "mountPath": "/workspace",
"pvc": {"claimName": "try-workspace", "storage": "1Gi"}}]
}' | jq -r .id)

curl -s -H "$H" http://localhost:8090/v1/sandboxes/$ID | jq -r .status.state # repeat until Running

curl -s -N -H "$H" -H 'content-type: application/json' \
http://localhost:8090/v1/sandboxes/$ID/proxy/44772/command \
-d '{"command": "python3 -c \"print(1+1)\"", "cwd": "/workspace"}'

curl -s -X DELETE -H "$H" http://localhost:8090/v1/sandboxes/$ID

The command answers a stream of JSON events, ending with {"type":"execution_complete","exit_code":0,...}. Deleting the sandbox keeps the try-workspace volume.

6. Point Distri at it​

Add to the server's .env:

DISTRI_SANDBOX_DEFAULT=on
SANDBOX_URL=http://localhost:8090
SANDBOX_API_KEY=<the key from step 3>
SANDBOX_USE_SERVER_PROXY=true # no ingress gateway locally

Restart the server and set "sandbox": true on a workspace.

Then run the provider contract test from the distri-cloud checkout, with the same variables set. It creates a sandbox, runs commands, writes and reads files, pauses, resumes, deletes and recreates on the same volume, and prints the timings:

cargo test -p distri-cloud --test test_sandbox_provider -- --nocapture

A passing run prints a line like this:

TIMINGS provider=http://localhost:8090 create_ms=6206 exec_ms=226 pause_ms=1562 resume_ms=1944 recreate_on_volume_ms=1315
test provider_contract ... ok

The second test in the file, ensure_reuses_resumes_and_recreates_on_the_volume, also needs DATABASE_URL for Postgres. Without SANDBOX_URL, both tests skip.

Clean up​

k3d cluster delete distri-sandbox

Quicker: OpenSandbox in Docker mode​

For quick local work you can skip Kubernetes. OpenSandbox can run sandboxes as plain Docker containers:

uvx opensandbox-server init-config ~/.sandbox.toml --example docker
uvx opensandbox-server

Set SANDBOX_URL to the server's address (by default http://localhost:8080) and SANDBOX_API_KEY to the key in ~/.sandbox.toml, if you set one. Volumes become Docker named volumes. Pause and resume work, and the files on /workspace stay.

Configuration​

The server reads these environment variables.

VariableDefaultWhat it does
DISTRI_SANDBOX_DEFAULToffon lets workspaces with "sandbox": true use sandboxes.
SANDBOX_URLnoneThe OpenSandbox server, e.g. http://localhost:8090. Required. OPENSANDBOX_URL also works.
SANDBOX_API_KEYnoneOpenSandbox's API key, sent as OPEN-SANDBOX-API-KEY. OPENSANDBOX_API_KEY also works.
SANDBOX_ACCESS_CLIENT_ID, SANDBOX_ACCESS_CLIENT_SECRETnoneA Cloudflare Access service token, sent on every call when both are set.
SANDBOX_USE_SERVER_PROXYfalseReach execd through the OpenSandbox server instead of the ingress gateway.
OPENSANDBOX_IMAGEpython:3.12-slimThe sandbox image.
OPENSANDBOX_CPU1CPU limit per sandbox.
OPENSANDBOX_MEMORY1GiMemory limit per sandbox.
OPENSANDBOX_CPU_REQUEST250mCPU reserved per sandbox (Kubernetes only).
OPENSANDBOX_MEMORY_REQUEST512MiMemory reserved per sandbox (Kubernetes only).
OPENSANDBOX_TIMEOUT_SECS3600Seconds until a sandbox expires. The next tool call recreates it on the same volume.
OPENSANDBOX_VOLUME_SIZE5GiSize of a new thread volume. Docker volumes ignore it.
OPENSANDBOX_SECURE_ACCESSon (off for a localhost URL)Create sandboxes with secureAccess, so traffic into a sandbox needs its own token.