> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orcapods.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Local Development
> Set up a full Orca development environment with hot reload on all components.
This is a self-hosting guide. The hosted product at [https://www.orcapods.ai](https://www.orcapods.ai) needs none of this — sign in and go.
Orca's source repository is currently private. This guide applies to teams with source access (design partners and licensed self-hosters). If that's you and you don't have access yet, email [support@okik.io](mailto:support@okik.io).
## Repository Layout
```mermaid theme={null}
flowchart TB
repo["orca/"]
runtime["agent-runtime/
Go workspace module
conductor + runner"]
worker["agent-worker/
Node.js agent sidecar"]
dashboard["dashboard/
React + Vite frontend"]
landing["landing/
React + Vite landing site"]
docker["docker/
Dockerfiles"]
docs["docs/
Documentation"]
compose["docker-compose.yml"]
makefile["Makefile"]
gowork["go.work
Go workspace definition"]
repo --> runtime
repo --> worker
repo --> dashboard
repo --> landing
repo --> docker
repo --> docs
repo --> compose
repo --> makefile
repo --> gowork
```
***
## Prerequisites
| Tool | Version | Purpose |
| ---------- | --------------------- | --------------------------------- |
| Go | 1.25+ | Conductor & runner |
| Node.js | 20+ | agent-worker, dashboard & landing |
| npm or Bun | npm 10+ / current Bun | JS package management |
| Docker | 24+ | Optional compose stack |
| Make | any | Dev workflow shortcuts |
***
## Initial Setup
```bash theme={null}
git clone https://github.com/okikorg/orca.git
cd orca
# Install JS dependencies
cd agent-worker && npm install && cd ..
cd dashboard && npm install && cd ..
cd landing && npm install && cd ..
# Verify Go workspace
cd agent-runtime && go mod download && cd ..
```
***
## Development Modes
### Mode 1: Sidecars Only (Fastest Iteration)
Use this when you're working on the conductor or runner only and don't need to change the sidecar.
```bash theme={null}
# Start sidecars as Docker containers (detached)
make up
# Terminal 1: Runner
make runner
# Terminal 2: Conductor
make conductor
# Terminal 3: Dashboard
make frontend
```
`make up` starts the Node.js sidecars using Docker Compose so you don't need to manage them manually.
### Mode 2: All Processes (Full Hot Reload)
Use this when you're changing the sidecar code.
```bash theme={null}
# Terminal 1: Sidecar (Node.js watch mode)
cd agent-worker
MODE=general PORT=7072 ANTHROPIC_API_KEY=sk-ant-... \
node --watch src/server.js
# Terminal 2: Runner
RUNNER_PORT=7070 RUNNER_BASE_URL=http://localhost:7070 \
RUNNER_CAPABILITIES=general \
GENERAL_SIDECAR_URL=http://localhost:7073 \
CONDUCTOR_BASE_URL=http://localhost:8080 \
go run ./agent-runtime/cmd/runner
# Terminal 3: Conductor
CONDUCTOR_PORT=8080 RUNNER_URLS=http://localhost:7070 \
go run ./agent-runtime/cmd/conductor
# Terminal 4: Dashboard
cd dashboard
npm run dev
```
### Mode 3: Full Docker Stack
Use this for production-parity testing.
```bash theme={null}
# Build all images and start the compose stack
make dev
# Backend proxy: http://localhost:8080
# Dashboard: http://localhost:5173 (run make frontend)
# Conductor API: http://localhost:8080/api
# Metrics: http://localhost:8080/metrics
```
### Landing Site
Use this when you're changing the standalone marketing site. It is a separate Vite app that reuses the dashboard design tokens. The landing site ships dark by default, exposes a nav theme toggle, and stores an explicit light/dark preference in `localStorage` as `orca-theme`; keep `landing/index.html` and `landing/src/lib/theme.ts` in sync when changing theme defaults.
```bash theme={null}
cd landing
npm run dev
```
***
## Environment Variables
Copy this into a `.env` file at the repo root for Docker Compose, or export them in your shell for process mode.
```bash theme={null}
# Required
ANTHROPIC_API_KEY=sk-ant-...
# Optional providers
OPENAI_API_KEY=sk-...
GOOGLE_GENERATIVE_AI_API_KEY=AI...
GROQ_API_KEY=gsk_...
# Optional tools
TAVILY_API_KEY=tvly-...
# Conductor (optional overrides)
CONDUCTOR_PORT=8080
AGENT_ORC_RUNS_DIR=/tmp/orca-runs
AGENT_ORC_PLANS_DIR=/tmp/orca-plans.jsonl
# Runner (optional overrides)
RUNNER_PORT=7070
RUNNER_BASE_URL=http://localhost:7070
MCP_BASE_URL=http://localhost:7070
RUNNER_CAPABILITIES=claude,codex,general
CLAUDE_SIDECAR_URL=http://localhost:7071
CODEX_SIDECAR_URL=http://localhost:7072
GENERAL_SIDECAR_URL=http://localhost:7073
# Shared storage (optional, SeaweedFS in local compose)
AWS_ENDPOINT_URL_S3=http://localhost:8333
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=agent-orc
AWS_SECRET_ACCESS_KEY=agent-orc-secret
S3_BUCKET=agent-artifacts
S3_FORCE_PATH_STYLE=true
# Pay-first billing (the conductor integration is off by default in dev)
BILLING_EMIT_ENABLED=false
BILLING_INTERNAL_URL=http://localhost:8091
BILLING_INTERNAL_SIGNING_KEY_CURRENT=base64-encoded-32-byte-key
# Optional for real Polar calls; unset uses the billing service's local mock
POLAR_ACCESS_TOKEN=polar_...
POLAR_SERVER=sandbox
POLAR_CREDIT_PACK_PRODUCT_IDS=2000:prod_20,10000:prod_100,20000:prod_200
POLAR_WEBHOOK_SECRET=whsec_...
# Connected Apps via the MCP bridge (optional)
COMPOSIO_API_KEY=...
MCP_BRIDGE_INTERNAL_SIGNING_KEY_CURRENT=base64-encoded-32-byte-key
COMPOSIO_SUBJECT_PEPPER=base64-encoded-stable-pepper
# Orchestration tools are registered by default.
# Set to off to remove delegation/plan tools from the runner registry.
AGENT_ORC_ORCHESTRATION_TOOLS=off
```
***
## Runner Authentication
The runner trusts the `X-Tenant-ID` header outright, because its only legitimate caller is the conductor. `RUNNER_AUTH_SECRET` is what proves a caller *is* the conductor: the conductor stamps it on every runner request, and the runner rejects anything else with `401` before the tenant header is even read.
| Variable | Requirement | Description |
| -------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `RUNNER_AUTH_SECRET` | Required outside dev | Shared secret of at least 32 bytes, set identically on the runner and the conductor. Unset leaves the runner unauthenticated, which is the local default; the runner refuses to boot without it when `AGENT_ORC_ENV` is any non-`dev` value. |
`/healthz` and `/readyz` stay open, or a gated liveness probe would restart-loop the runner. Everything else is gated, `/metrics` included (it keeps its own bearer gate on top).
Leave it unset locally. `make demo` runs the conductor and runners on loopback, and a sandbox worker reaching the runner through a tunnel holds no shared secret, so the gate would block exactly the path you are testing.
***
## Dynamic Sandbox Workers
Profiles with `workerMode: "sandbox"` need a session-scoped signing key and either a runner-managed substrate or an out-of-band worker. The runner fails closed when `WORKER_TOKEN_HMAC_KEY` is unset or decodes to fewer than 32 bytes: managed workers cannot launch, `/worker/*` calls are rejected, and the sandbox session's MCP endpoint requires a valid worker token.
| Variable | Requirement | Description |
| -------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `WORKER_TOKEN_HMAC_KEY` | Required | Raw, hex, base64, or base64url secret of at least 32 bytes. Keep it distinct from `RUN_TOKEN_HMAC_KEY`. |
| `WORKER_SUBSTRATE` | Optional | `process`, `docker`, `daytona`, or `e2b`. Unset leaves worker launch to an external operator. This is also the **default** for profiles that name none. |
| `WORKER_SUBSTRATES` | Optional | Comma-separated extra substrates this runner can also launch on, e.g. `daytona,docker`. Together with `WORKER_SUBSTRATE` this is the set a profile's `workerSubstrate` may choose from. |
| `WORKER_IMAGE_` | Per substrate | Worker image for one substrate, e.g. `WORKER_IMAGE_E2B`, `WORKER_IMAGE_DAYTONA`, `WORKER_IMAGE_DOCKER`. Falls back to `WORKER_IMAGE`. Required once a runner serves more than one substrate, because the same worker is a different artifact on each. |
| `WORKER_RUNNER_URL` | Usually required | Runner URL reachable from the worker. Falls back to `MCP_BASE_URL`; a Docker Desktop worker commonly needs `http://host.docker.internal:7070`. |
| `WORKER_ENTRYPOINT` | Process only | Path to `agent-worker/src/server.js`. |
| `NODE_BIN` | Process only | Optional Node executable override. |
| `WORKER_IMAGE` | Docker, Daytona, or E2B | Agent-worker container image for Docker; pre-built snapshot name for Daytona; template id for E2B. |
| `DOCKER_BIN` | Docker only | Optional Docker CLI override. |
| `WORKER_DOCKER_NETWORK` | Docker only | Optional Docker network attached to launched workers. |
| `WORKER_CPU_MILLIS` | Docker only | CPU request passed to Docker. Daytona worker sizing belongs to the snapshot. |
| `WORKER_MEMORY_MB` | Docker only | Memory request passed to Docker. Daytona worker sizing belongs to the snapshot. |
| `DAYTONA_API_KEY` | Daytona only | Required Daytona credential. |
| `DAYTONA_API_URL` | Daytona only | Optional Daytona API base URL override. |
| `DAYTONA_TOOLBOX_URL` | Daytona only | Optional toolbox proxy host root. Do not append `/toolbox`. |
| `DAYTONA_TARGET` | Daytona only | Optional Daytona target. |
| `WORKER_DOMAIN_ALLOWLIST` | Daytona only | Optional comma-separated extra public DNS domains. The runner also derives public callback, credentialed provider API, configured base-URL override, and OTLP endpoint hosts; loopback names, IP addresses, and other non-public hosts are dropped. Daytona allows at most 20 entries. |
| `E2B_API_KEY` | E2B only | Required E2B credential. `E2B_KEY` is accepted as an alias. |
| `E2B_BRIDGE_URL` | E2B only | **Required.** URL of a deployed `e2b-bridge/`. E2B's per-sandbox exec is gRPC-Web to the in-sandbox `envd` daemon, not REST on the control plane, so the bridge is the only way a lease can tail the worker log. The runner refuses to boot without it. |
| `E2B_BASE_URL` | E2B only | Optional control-plane base URL override (BYOC / self-hosted). |
| `WORKER_IDLE_TIMEOUT` | Daytona or E2B | Provider-side backstop, for a runner that dies before it can pause anything; defaults to `15m`. Set `0` to disable it. Keep it longer than `WORKER_IDLE_PAUSE` or the provider fires first and the runner's own policy never applies (the runner warns at boot when it does). On E2B this is a **wall-clock lifetime**, not an idle timer: it counts from launch and from each resume regardless of activity. |
| `WORKER_IDLE_PAUSE` | All substrates | How long a worker may sit between runs before the runner pauses it; defaults to `5m`. Set `0` to opt out and leave reclaim to the substrate. This is the platform's idle policy and it applies on every substrate, including docker and process, which have no provider-side auto-stop at all. A paused worker resumes on the next run with its conversation intact. |
| `WORKER_AUTO_ARCHIVE` | Daytona only | How long a **stopped** worker sandbox keeps its disk before Daytona archives it to cold storage; defaults to `2h`. Set `0` to leave Daytona's own default, which is 7 days. This is the only reclaim that reaches a worker whose runner died, because stopping frees compute but not disk. |
| `WORKER_RESUME_TIMEOUT` | Daytona only | Bound on resuming a worker sandbox; defaults to `5m`, against the provider's own 90s. Resuming an archived worker restores its filesystem from cold storage, and a resume that outruns this fails the run instead of merely being slow. |
Stopping a sandbox frees compute but **not disk**. A stopped Daytona sandbox holds its full allocation against the organisation's total disk limit (30 GiB on tier 1), so three abandoned 8 GiB workers are enough to make every subsequent launch fail with `400 ... Total disk limit exceeded` while nothing at all is running. Only archiving returns that disk.
Two traps in Daytona's semantics make the safe-looking setting the wrong one: the default archive interval is **7 days**, and `0` means the **maximum** interval rather than "disabled". `WORKER_AUTO_ARCHIVE` exists because no runner-side cleanup can reach an orphan, by definition: the runner that owned it is gone.
Setting `WORKER_IDLE_TIMEOUT=0` disables the provider auto-stop, and auto-archive only ever applies to a sandbox that is already stopped, so the two together leave an orphan's disk allocated indefinitely. The runner warns at boot (`worker_substrate.orphan_disk_unreclaimable`) when it sees that combination.
### E2B differs from Daytona in three ways that matter
Both run through the same provider-neutral `workersubstrate.NewCloud`, so the substrate code is identical. What differs is what the provider does underneath it.
**The timeout kills by default.** E2B's `timeout` is a wall-clock time-to-live, and reaching it destroys the sandbox rather than stopping it. Daytona's auto-stop preserves the disk, which is the assumption `WORKER_IDLE_TIMEOUT` is written against. The runner therefore always sets `autoPause` for worker sandboxes, which converts the timer into a pause and makes the two providers mean the same thing. Nothing in a deployment should turn that off; without it a worker is destroyed mid-session and takes the conversation with it.
The upside of the same mechanism: E2B's auto-pause captures a full **memory** snapshot, so a resumed worker comes back with its processes and in-memory state intact. Daytona's pause degrades to stop plus start, which is why the worker persists its state to sandbox disk and rehydrates on resume. That fallback is still correct on E2B, just no longer load-bearing.
**There is no auto-archive.** `WORKER_AUTO_ARCHIVE` has no E2B equivalent, so a paused worker holds its disk until something deletes it. On Daytona, archiving is the one reclaim that reaches a worker whose runner died; on E2B there is none, and orphans accumulate against the team quota until they are cleaned up out of band.
**There is no domain allow-list.** `WORKER_DOMAIN_ALLOWLIST` builds an exact DNS allow-list, which Daytona accepts and E2B cannot express: its network controls are CIDR-shaped (`allow_internet_access`, `network.denyOut`). An E2B worker sandbox therefore has unrestricted egress, and the containment the Daytona path gets from a derived allow-list is simply absent.
The runner names both absences at boot under `worker_substrate.e2b_gaps`, because neither ever surfaces as an error downstream.
**Concurrency is not the constraint it is on Daytona.** The Daytona tier caps concurrent worker sandboxes at roughly two (memory) and accumulated stopped sandboxes at 30 GiB (disk), and both bite during ordinary testing. Four concurrent E2B sandboxes allocate without complaint, and 17 paused ones coexist holding \~390 GB, so neither cap applies in the same way. That makes the missing auto-archive a slow leak rather than an outage: nothing fails, the paused sandboxes simply accumulate until someone deletes them.
`WORKER_IDLE_TIMEOUT=0` means "disable the provider backstop" on Daytona. On E2B it cannot mean that, and the failure is silent: an omitted timeout does not disable anything, it falls back to E2B's own default of **15 seconds**. Every worker would auto-pause at roughly the time it takes to boot, and the only symptom would be a cold-start timeout naming the substrate. The runner refuses to start with `WORKER_SUBSTRATE=e2b` and a zero `WORKER_IDLE_TIMEOUT` rather than serve that.
### Letting an agent choose its substrate
A runner configured for more than one substrate lets each profile pick, via `workerSubstrate` on the agent (the **Runs in** control under Compute in the agent form):
```bash theme={null}
WORKER_SUBSTRATE=e2b \
WORKER_SUBSTRATES=daytona,docker \
WORKER_IMAGE_E2B= \
WORKER_IMAGE_DAYTONA= \
WORKER_IMAGE_DOCKER=orca-agent-worker:dev \
go run ./agent-runtime/cmd/runner
```
Resolution order for one profile is: `workerSubstrate` if set, else `sandbox.provider` (the shell's) **if this runner serves it**, else the runner's default. Inheriting the shell is the common case and needs no configuration: an agent given an E2B shell almost always wants its worker on E2B, and having to say so twice is how the two drift apart.
The two sources fail differently, on purpose. An **explicit** `workerSubstrate` the runner does not serve is an error, because someone chose it and quietly running elsewhere would hide the mistake. An **inherited** one is only a hint: a docker-only runner serving an agent whose shell is on E2B falls back to docker rather than failing a profile nobody edited.
Per-substrate images are not optional once a runner serves several. "The worker image" is not one artifact: the same worker is an E2B template id, a Daytona snapshot name, and a local Docker reference, built by three different pipelines and never interchangeable. A runner serving two substrates from a single `WORKER_IMAGE` hands each of them the other's identifier, and the failure surfaces as a provider-side "not found" naming neither the substrate nor the setting.
### Routing across a mixed fleet
Runners advertise their substrates on `/runner/info` (`substrates`, plus `substratesKnown`), and the conductor routes a new session to a runner that serves the profile's `workerSubstrate`. A fleet where one runner has E2B credentials and another has Daytona therefore works without the caller knowing which is which.
Four rules make that safe:
* **The mode routes even when the substrate does not.** `workerMode: "sandbox"` means a worker has to be launched, which is a different fact from *where*, and most sandbox profiles name no substrate at all. Reading "no substrate named" as "no requirement" put a per-session-worker agent on a runner that launches nothing: the session was accepted and then narrated `starting agent worker` up to its three-minute timeout, on a fleet whose own placement report already said *nothing will start*. A profile that needs a worker is refused at create when no eligible runner launches one, rather than falling back to the shared sidecars, because the author opted into a per-session worker and running elsewhere silently is the failure this reporting exists to prevent.
* **Only an explicit choice routes to a *particular* substrate.** An inherited one (the shell's provider) is a hint the broker may ignore in favour of the runner default, so routing on it would refuse runners that would have served the session perfectly well.
* **A named substrate nothing serves is a hard failure**, not a fallback: `no runner serves worker substrate "daytona" (fleet serves: docker, e2b)`. The runtime path widens to the whole fleet when nothing matches, because a runner advertising no capabilities serves them all. Substrates have no such contract, so widening would swap a clear error for a cold-start timeout on a runner that never could have run it.
* **Unknown is not none.** A runner too old to report substrates stays eligible, so a fleet mid-upgrade does not read as an outage. A runner that reports an empty list is excluded, because it would accept the session and fail its first run. `substratesKnown` is what tells those apart.
The choice is **not** validated when the agent is saved. Which substrates exist is a property of the fleet, not of the profile. Routing catches the common case at session create; the runner still re-checks at launch and fails with an error listing what it serves (`this runner does not serve substrate "daytona" (configured: docker, e2b)`), which is what catches a session pinned to a runner whose config changed under it. The runner logs its own set once at boot under `worker_substrate.configured`.
Placement is read when a worker is **launched**, not on every run. Editing an agent's substrate mid-session leaves the live worker where it is, because re-placing it would destroy the conversation it holds; the change takes effect on that session's next worker.
### How a worker reaches its runner
A worker never listens on a port; it dials out. So something has to be reachable from inside the sandbox, and the runner itself must not be that thing, because `RUNNER_AUTH_SECRET` gates every `/runner/*` route and a worker running inside a tenant's sandbox cannot hold a shared secret.
The conductor forwards instead. It mounts exactly six routes, unauthenticated by Clerk but each one verifying a worker token whose session claim must match the session in its own path, then forwards to the runner that owns that session with the runner-auth secret attached:
```
GET /worker/sessions/{id}/next-run
POST /worker/sessions/{id}/events
POST /worker/sessions/{id}/state
POST|GET|DELETE /runner/sessions/{id}/mcp
```
The last one is easy to miss and breaks differently: it is the worker's MCP reverse channel, minted into each run command from `WORKER_RUNNER_URL`. Point that at the runner and every run starts fine, then fails the moment the agent calls a tool.
It is minted from `WORKER_RUNNER_URL` and **not** from `MCP_BASE_URL`, and the two must not be collapsed. `MCP_BASE_URL` is the address a STATIC sidecar dials, and a static sidecar is on the private network and sends no auth on this callback by contract. A sandbox worker is on the public internet and reaches the runner only through the conductor's proxy, which demands a worker token on every request. Setting `MCP_BASE_URL` to the conductor to serve sandbox workers therefore re-points every static sidecar at that proxy too, and the DEFAULT path -- almost every agent -- starts answering `invalid or missing worker token` on its first tool call while the sandbox path it was changed for works fine. That shipped to prod once.
The worker process token is valid for 24 hours because it must survive across runs. The capability token embedded in `sessionMcpUrl` is minted separately for each run: it defaults to 2 hours when the run has no deadline, otherwise it uses the remaining run deadline plus 5 minutes of grace, with a 5-minute minimum and a 24-hour signing ceiling. Both are bearer credentials scoped to one tenant and session.
| Variable | Where | Description |
| ----------------------- | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `WORKER_TOKEN_HMAC_KEY` | Conductor **and** runner | Same value on both. The runner mints; the conductor verifies to decide what to forward. Unset on the conductor leaves the proxy unmounted (it logs `worker_proxy.disabled`) and every worker times out in cold start. |
| `RUNNER_AUTH_SECRET` | Conductor **and** runner | Stamped on each forwarded request. |
| `WORKER_RUNNER_URL` | Runner | The **conductor's** public URL, not the runner's. Drives both halves of a sandbox worker's connectivity: its transport and its MCP callback. |
| `MCP_BASE_URL` | Runner | The **runner's own** address, reachable from the static sidecars on the private network. Must NOT be the conductor: that points the shared-sidecar path at the worker proxy, which 401s every static session's tool calls. |
The conductor meters this machine-to-machine surface separately from browser traffic. RPM tiers require rate limiting to be enabled; the in-flight cap is always enforced because long polls hold connections and goroutines for their full duration.
| Variable | Default | Scope |
| ---------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `AGENT_ORC_RATELIMIT_SANDBOX_EVENTS_RPM` | `600` | Public `POST /api/sandbox-events/{provider}` requests per client IP. |
| `AGENT_ORC_RATELIMIT_WORKER_RPM` | `240` | Authenticated worker transport requests per verified session. |
| `AGENT_ORC_RATELIMIT_WORKER_UNAUTH_RPM` | `120` | Worker transport requests that fail token verification, per client IP. |
| `AGENT_ORC_RATELIMIT_WORKER_INFLIGHT` | `24` | Concurrent proxied worker requests per verified session. A non-positive value restores the default rather than disabling the cap. |
An exhausted tier or in-flight cap returns `429 Too Many Requests`. `AGENT_ORC_RATELIMIT_ENABLED=false` disables the RPM tiers but not `AGENT_ORC_RATELIMIT_WORKER_INFLIGHT`.
Setting one of these RPM values to `0` means something stronger than it does for the `RUNS`/`SESSIONS`/`TOPOLOGY` tiers. Those routes still fall back to the general per-tenant and per-IP ceilings; these do not, because they are excluded from the per-IP tier by design so each is charged once to its own bucket. A `0` here therefore leaves that route with no request bound at all, and only the resource bounds (the fan-out limit and the in-flight cap) remain.
Locally, `scripts/worker-tunnel-guard.mjs` (below) plays the same role for a tunnel pointed at a single runner.
For a local process worker:
```bash theme={null}
WORKER_TOKEN_HMAC_KEY=replace-with-at-least-32-bytes \
WORKER_SUBSTRATE=process \
WORKER_RUNNER_URL=http://localhost:7070 \
WORKER_ENTRYPOINT="$PWD/agent-worker/src/server.js" \
go run ./agent-runtime/cmd/runner
```
For Daytona, build the worker image as a snapshot before starting the runner. The snapshot entrypoint must be `/app/sandbox-entrypoint.sh`; it starts the outbound worker, writes diagnostic output to `/tmp/orca-worker.log`, and keeps the sandbox available so the runner can read that log after an early worker exit. Size CPU, memory, and disk when pushing the snapshot, because Daytona rejects resource overrides when creating a sandbox from a snapshot.
```bash theme={null}
daytona snapshot push --name \
--entrypoint /app/sandbox-entrypoint.sh
WORKER_TOKEN_HMAC_KEY=replace-with-at-least-32-bytes \
WORKER_SUBSTRATE=daytona \
WORKER_IMAGE= \
WORKER_RUNNER_URL=https:// \
DAYTONA_API_KEY=... \
go run ./agent-runtime/cmd/runner
```
For E2B, build the worker image as a template first. The template plays the same role as the Daytona snapshot, from a different pipeline: E2B reads a Dockerfile rather than pushing a local image. Use the repository target, which builds `agent-worker/e2b.Dockerfile`:
```bash theme={null}
make worker-e2b-template # -> template name orca-agent-worker
```
That Dockerfile is separate from `agent-worker/Dockerfile` for two reasons. It is Debian-based rather than Alpine, because E2B injects a glibc-linked `envd` daemon into the guest and a musl base leaves it unable to start, with no error worth reading. And it sets no `CMD`, `ENTRYPOINT`, or start command at all.
Do **not** give the template a start command (`-c` / `--cmd`). E2B runs a template's start command **once at build time** and captures it in the snapshot, and E2B does not expose environment supplied at sandbox creation to it. A worker started that way would hold a build-time environment forever: no `ORCA_SESSION_ID`, no run token, no runner URL. It is the one substrate where creating a sandbox starts nothing, so the runner execs `/app/sandbox-entrypoint.sh` itself after create, in the background, with the session environment (`workersubstrate`'s `entrypointStarter`, implemented by the E2B provider).
The entrypoint still has to be `/app/sandbox-entrypoint.sh` and still tees to `/tmp/orca-worker.log`, because that log is how you diagnose a worker that exits before it calls home. The path is `workersubstrate.WorkerEntrypointPath`; change one and you must change the other, or the start becomes a silent `127` inside a backgrounded process.
`e2b-bridge/` must be deployed and reachable before the runner starts. It is what gives the Go provider a flat HTTP surface over E2B's per-sandbox gRPC-Web `envd` API. The runner refuses to boot without `E2B_BRIDGE_URL`, and the post-create start goes through it too, so an unreachable bridge fails the launch rather than producing an empty sandbox.
```bash theme={null}
WORKER_TOKEN_HMAC_KEY=replace-with-at-least-32-bytes \
WORKER_SUBSTRATE=e2b \
WORKER_IMAGE= \
WORKER_RUNNER_URL=https:// \
E2B_API_KEY=... \
E2B_BRIDGE_URL=https:// \
go run ./agent-runtime/cmd/runner
```
When a Daytona worker is testing against a local runner, do not expose the runner directly through a tunnel. Run the repository's allowlist proxy and point the tunnel at the proxy instead; it forwards only the authenticated worker callback routes and the session MCP route:
```bash theme={null}
node scripts/worker-tunnel-guard.mjs \
--listen 17098 \
--upstream http://localhost:17099
```
Set `WORKER_RUNNER_URL` and `MCP_BASE_URL` to the public tunnel URL. Daytona's domain list is exact rather than additive, so the runner derives public callback hosts, each injected provider credential's API host (honoring supported base-URL overrides), and public configured OTLP hosts. Loopback names such as `localhost`, IP addresses, and other non-public hosts are omitted because a cloud sandbox cannot reach them and Daytona rejects the entire allocation if any domain is invalid. Use `WORKER_DOMAIN_ALLOWLIST` only for additional operator-approved public egress destinations. At runner boot, `worker_substrate.domain_allowlist` logs the final list.
To receive Daytona sandbox state notifications while testing locally, expose only the conductor's event ingress through a separate allowlist proxy, then point the public tunnel at that proxy:
```bash theme={null}
node scripts/worker-tunnel-guard.mjs \
--listen 17099 \
--upstream http://localhost:8080 \
--allow '^/api/sandbox-events/[^/]+$'
```
Configure the provider webhook as `https:///api/sandbox-events/daytona`. Do not tunnel the conductor directly: `--allow` replaces the proxy's default worker callback routes and keeps the rest of `/api/*` unreachable from that hostname.
The first run for a sandbox-worker session starts the worker. Later runs reuse it, and deleting the session closes it. Daytona sandboxes carry an `orca.session` label with the session ID so operators and diagnostic scripts can locate the matching sandbox without inspecting redacted environment variables. A worker that never reaches its first long poll causes the run to fail after the bounded cold-start wait instead of hanging indefinitely.
### Compare Static and Sandbox Workers
With the conductor and both worker paths already configured, run the parity harness from the repository root:
```bash theme={null}
node scripts/worker-parity.mjs \
--api http://localhost:8080 \
--runtime vercel \
--model openai:gpt-4.1-mini
```
The harness creates timestamped `static` and `sandbox` profiles, runs the same prompt through each sequentially, and exits non-zero unless both paths succeed and agree on terminal event type, collapsed event-type shape, `session_init` presence, and token reporting. It deliberately does not compare model response text. The output also reports the sandbox cold-start time relative to the static sidecar.
### Check That a Conversation Survives Pause and Resume
A cloud sandbox loses everything held in the worker's memory when it is paused, because the provider stops the container and restarts it from disk. The per-session worker lifetime depends on the worker having written its state to disk first, so that path needs its own check:
```bash theme={null}
DAYTONA_API_KEY=... node scripts/worker-pause-resume.mjs \
--api http://localhost:18080
```
The script plants a number in turn one, stops the sandbox through the provider API rather than through the runner (which is how an idle sweeper or a provider-initiated stop would arrive), then asks for the number back on the same session. It exits non-zero unless the second turn recalls it.
This check is meaningful only against a cloud substrate. `docker pause` retains the container's memory, so under the Docker substrate the conversation survives whether or not anything was ever persisted.
***
## Makefile Targets
| Target | Description |
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `make up` | Start sidecars via Docker Compose (detached) |
| `make down` | Stop Docker Compose containers |
| `make runner` | Start the runner binary (foreground) |
| `make conductor` | Start the conductor binary (foreground) |
| `make billing` | Start the pay-first billing service on `:8091` after sidecars are up |
| `make frontend` | Install dashboard dependencies when package metadata changes, then start the Vite dev server |
| `make frontend-prod` / `make dash-prod` | Start the local dashboard dev server against the deployed Railway conductor and VirtualFS services |
| `make dev` | Build images + start full compose stack |
| `make demo` | Start the local demo stack with containers, two runners, conductor, billing on `:8091`, MCP bridge on `:8092`, VirtualFS on `:8081`, dashboard, and e2b bridge |
| `make vfs-shell` | Start a local VirtualFS browser playground with the mixed profile (RAM `/agents`, disk `/pools` and `/data`, and optional S3 `/seaweed`) |
| `make vfs-shell-strict` | Start the VirtualFS browser playground with strict production-style mount visibility (every mount unavailable, prod profile) |
| `make build` | Build conductor, runner, VirtualFS, billing, and MCP bridge binaries to `./bin/` |
| `make test` | Run all tests (Go + Node.js) |
`make demo` runs `demo-free-ports` first, including ports `8091`, `8092`, and `8081`, then starts `./bin/billing` on `:8091`, `./bin/mcp-bridge` on `:8092`, and `./bin/vfs serve --addr :8081 --s3-bucket "$S3_BUCKET" --s3-endpoint "$AWS_ENDPOINT_URL_S3"` with `OTEL_SERVICE_NAME=vfs` before launching the dashboard. It generates `BILLING_INTERNAL_SIGNING_KEY_CURRENT`, `MCP_BRIDGE_INTERNAL_SIGNING_KEY_CURRENT`, and `COMPOSIO_SUBJECT_PEPPER` in `.env` when they are absent; keep the generated subject pepper stable because changing it re-keys derived Composio subjects. Connected Apps is available from [http://localhost:5173/settings](http://localhost:5173/settings); the bridge still starts when `COMPOSIO_API_KEY` is unset, but the app gallery remains empty.
When `WORKER_TUNNEL_URL` is set, `make demo` runs `runner-b` sandbox workers on Daytona instead of the substrate from `.env`, using `WORKER_DAYTONA_SNAPSHOT` or the default `orca-agent-worker-dyn`. It uses the tunnel URL for both the worker callback and session MCP endpoint; `pi` and `vercel` profiles route to this runner. `DAYTONA_API_KEY` must be configured. These sandboxes idle-pause rather than exit, so list them with `daytona sandbox list` after testing and delete any you no longer need.
Set `WORKER_E2B_TEMPLATE` as well and `runner-b` serves **both** substrates, so an agent can pick `Runs in: e2b` or inherit it from an E2B shell; daytona stays the default. Build the template with `make worker-e2b-template` first. Without it the runner advertises daytona alone, and an agent asking for e2b silently falls back to the fleet default — which the agent's `workerPlacement` field reports, and which is worth reading before treating it as a routing bug.
The demo starts the billing service but defaults `BILLING_EMIT_ENABLED` to `false`, so the conductor does not call it: runs bypass the credit gate and the Usage page reports that billing is not connected. To exercise the real gate, set `BILLING_EMIT_ENABLED=true` in `.env`; because `make demo` sources that file, its value overrides the same variable supplied on the `make` command line. Outside dev, an unwired billing client is not allowed: when `AGENT_ORC_ENV` is any non-`dev` value, the conductor refuses to start unless billing is enabled with `BILLING_BASE_URL` or `BILLING_INTERNAL_URL` and a valid base64 signing key of at least 32 decoded bytes.
The demo binds VirtualFS to the local SeaweedFS bucket from `.env` when those variables are set. `make runner` and `make demo` also export `VFS_S3_BUCKET="$S3_BUCKET"` and `VFS_S3_ENDPOINT="$AWS_ENDPOINT_URL_S3"` so runner-side filesystem tools use the same bucket. The Files page is available at [http://localhost:5173/files](http://localhost:5173/files) once the demo is ready. Billing health is available at [http://localhost:8091/healthz](http://localhost:8091/healthz), MCP bridge health at [http://localhost:8092/healthz](http://localhost:8092/healthz), and VirtualFS metrics at [http://localhost:8081/metrics](http://localhost:8081/metrics).
The billing service reads `POLAR_*`, `BILLING_INTERNAL_SIGNING_KEY_CURRENT`, `REDIS_URL`, and the same Postgres cluster as the conductor. `POLAR_CREDIT_PACK_PRODUCT_IDS` maps one-time credit-pack amounts in cents to Polar product IDs. A checkout `amount` must match one of these configured packs.
`make vfs-shell` builds `./bin/vfs`, starts `vfs serve --profile mixed --disk-root .vfs-shell-data` on `:8088`, and serves the tracked `virtualfs/vfs-shell.html` through the Bun proxy at `virtualfs/serve.ts` on `:5170`. `/agents` is RAM (ephemeral); `/pools` and `/data` are disk-backed under `.vfs-shell-data/` (gitignored, persistent across restarts — `rm -rf .vfs-shell-data` to reset). When `VFS_SHELL_S3_BUCKET` or `VFS_S3_BUCKET` is set, the mixed profile also mounts that S3-compatible bucket at `/seaweed`; `VFS_SHELL_S3_ENDPOINT` falls back to `VFS_S3_ENDPOINT`. The proxy keeps the shell and `/vfs/*`, `/healthz`, and `/metrics` on the same origin for browser testing. `make vfs-shell-strict` uses the same proxy with production-style mount configuration and no S3 environment so unavailable mounts remain visible with their boot-time reason.
***
## Load Testing Harness
`agent-runtime/cmd/loadgen` drives high-concurrency campaigns against a conductor, and `agent-runtime/cmd/mock-sidecar` is a zero-cost sidecar that streams realistic `progress`, `tool_call`, `usage`, and terminal events. Together they exercise the conductor -> runner -> sidecar -> billing path without spending LLM tokens. Both are fenced by `LOADTEST_ENABLED=true`; when the switch is off, campaign start or mock `/run` calls fail closed.
The load generator exposes a small control API:
| Route | Purpose |
| ------------------ | --------------------------------------------------------------------------- |
| `GET /health` | Liveness, master-switch state, busy state, and target conductor |
| `POST /run` | Start one campaign; requires `Authorization: Bearer $LOADGEN_CONTROL_TOKEN` |
| `GET /status` | Snapshot of the current or last campaign |
| `GET /report/{id}` | Report for the current campaign id |
| `POST /stop` | Stop the running campaign; requires the control bearer |
Campaign bodies include `behavior`, optional `profile`, `ramp`, `hold_s`, `abort_error_rate`, `run_timeout_s`, `check_billing`, and optional `mix`. Without `mix`, loadgen uses the single `profile` plus `behavior` path. With `mix`, each entry supplies a `profile`, optional `behavior`, and optional `weight`; loadgen round-robins through the weighted entries so one campaign can drive multiple general-runtime profiles concurrently. Level reports include `by_profile` run and OK counts for mixed campaigns.
When `check_billing` is true and `LOADGEN_DSN` is configured, the report checks that cumulative `usage_records` rows for the isolated tenant and selected profile set are at least the campaign's finished runs, then reports token sums, tool-call sums, unpriced token rows, and the same billing totals broken down per profile.
Key loadgen environment:
| Variable | Default | Description |
| -------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------- |
| `LOADTEST_ENABLED` | `false` | Must be `true` to accept campaigns |
| `PORT` | `8080` | Control API listen port |
| `LOADGEN_CONTROL_TOKEN` | — | Bearer token for `POST /run` and `POST /stop` |
| `CONDUCTOR_URL` | — | Load-test conductor base URL |
| `LOADGEN_TENANT` | — | Isolated load-test tenant id |
| `LOADGEN_CHAT_SIGNING_KEY` | — | Base64 chat-gateway signing key used to sign conductor run requests |
| `LOADGEN_PROFILE` | `orca` | Default profile targeted by campaigns |
| `LOADTEST_SEED_PROFILES` | — | Optional comma-separated `name=provider:model` entries seeded on a dedicated load-test conductor alongside `orca` |
| `LOADGEN_MAX_CONNS` | `1024` | Per-host HTTP connection cap |
| `LOADGEN_POLL_MS` | `200` | Run-status poll interval |
| `LOADGEN_DSN` | — | Optional Postgres DSN for billing checks |
Set `LOADTEST_SEED_TENANT` on a dedicated load-test conductor to pre-provision that tenant and seed the canonical `orca` profile at boot. Add `LOADTEST_SEED_PROFILES` when a mixed campaign needs additional general-runtime profiles, for example `lt-haiku=anthropic:claude-haiku-4-5,lt-gpt=openai:gpt-5.5`. This is useful when loadgen signs `/api/runs` through the chatsig internal listener, which does not run the Clerk provision-and-seed hook. Leave both unset for normal conductors.
The mock sidecar reads behavior from the run prompt. Use one of the built-in archetypes (`fast`, `heavy`, `flaky`, `whale`, or `chatterbox`) or pass a JSON object with fields such as `events`, `work_ms`, `jitter_pct`, `model`, `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_create_tokens`, `tool_calls`, and `fail`. `MOCK_DEFAULT_MODEL` supplies the priced model id for emitted usage when a behavior omits one.
***
## Running Tests
```bash theme={null}
# Go tests
cd agent-runtime && go test ./...
# Node.js tests (agent-worker)
cd agent-worker && npm test
# Specific Go package
cd agent-runtime && go test ./runtime/... -v
# With race detector
cd agent-runtime && go test -race ./...
```
***
## Smoke Testing the Stack
After starting everything:
```bash theme={null}
# 1. Health checks
curl http://localhost:8080/healthz # Conductor
curl http://localhost:7070/healthz # Runner (if direct access)
# 2. Topology
curl http://localhost:8080/api/topology | jq
# 3. Default profile
curl http://localhost:8080/api/profiles | jq
# 4. Quick run
RUN_ID=$(curl -s -X POST http://localhost:8080/api/runs \
-H 'Content-Type: application/json' \
-d '{"profile":"general","title":"test","prompt":"Say hello in 5 words."}' \
| jq -r .runId)
# 5. Stream
curl -N "http://localhost:8080/api/runs/${RUN_ID}/stream" \
-H 'Accept: text/event-stream'
```
***
## Tool Smoke Tests
A dedicated smoke test binary tests all platform tools end-to-end:
```bash theme={null}
cd agent-runtime && go run ./cmd/tool-smoke
```
This exercises `web_search`, `web_extract`, `time_now`, `math_add`, `runner_info`, and other platform tools against a live runner.
VirtualFS also has a self-contained smoke gate for the v1.5 north-star UX samples. It runs against an in-process VirtualFS and does not require external services:
```bash theme={null}
go run ./virtualfs/cmd/vfs smoke v1_5
```
To verify the runner-to-VirtualFS tool wiring, run the agent-runtime smoke binary. With no `VFS_*` environment it uses the in-memory dispatcher fallback:
```bash theme={null}
cd agent-runtime && go run ./cmd/vfs-smoke/
```
GitHub Actions runs the full-stack `e2e-pipeline` workflow on pushes and pull requests to `main`. It brings up the Docker-backed services, starts host-side `e2b-bridge`, VirtualFS, two runners, and the conductor, then runs `pipeline-smoke`, `e2e-smoke`, and the dashboard typecheck/build gate. Logs from the run are uploaded from `.ci-logs/` as a workflow artifact.
For a long-running VirtualFS HTTP server for SDK examples and local agent demos, run:
```bash theme={null}
go run ./virtualfs/cmd/vfs serve
```
It listens on `:8080` by default, accepts `VFS_TOKEN=dev-token`, emits structured startup and request logs, exposes `/healthz` and `/metrics`, and seeds `/s3/log.jsonl` plus `/s3/report.parquet`. Use `--addr HOST:PORT` or `VFS_SERVE_ADDR` when that port is already in use.
By default, `vfs serve` runs the prod profile: it registers one system mount named `root` at `/`, backed by R2 with the bucket and endpoint resolved from `VFS_S3_BUCKET` / `CF_R2_BUCKET` / `S3_BUCKET`. The well-known paths (`/agents`, `/pools`, `/s3`, `/data`, `/kb`) are directories inside that mount. If no bucket is configured, the root mount comes up `unavailable`; the server still boots so you can inspect `GET /vfs/mounts` and see the failure reason. There is no silent in-memory fallback.
The server also exposes a session surface alongside `/vfs/mounts`: `POST /vfs/sessions`, `GET /vfs/sessions`, `GET /vfs/sessions/{id}`, `DELETE /vfs/sessions/{id}`. A session names a list of allowed path prefixes; once created, callers pass `session_id` in `/vfs/exec` (or any typed-op) body to scope that call to those prefixes. Ops outside the allowlist return HTTP 403 `SESSION_FORBIDDEN` (or exit 1 with a stderr line for `/vfs/exec`). Sessions are in-memory and reset on process restart. Both SDKs (`create_session` / `createSession`) wrap this surface.
Configure live backends with `--postgres-dsn`, `--redis-url`, `--s3-bucket`, and `--s3-endpoint`; `VFS_S3_BUCKET` and `VFS_S3_ENDPOINT` are also honored as explicit S3/R2 settings. Set `VFS_BACKEND=production` to let Postgres, Redis, and S3 flag defaults come from environment variables such as `VFS_POSTGRES_DSN`, `VFS_REDIS_URL`, `CF_R2_BUCKET`, `CF_R2_ENDPOINT`, `S3_BUCKET`, and `AWS_ENDPOINT_URL_S3`. Cloudflare R2 credentials are selected when the configured endpoint matches `.r2.cloudflarestorage.com`; other endpoints use AWS credentials.
### Mount profile
`vfs serve` accepts `--profile {prod|dev|disk|mixed}`:
* `prod` (default) — registers one `root` mount at `/`, backed by R2 with the bucket and endpoint resolved from `VFS_S3_BUCKET` / `CF_R2_BUCKET` / `S3_BUCKET`. When no bucket is configured, the mount comes up `unavailable`; the server still boots so you can inspect `GET /vfs/mounts` and see the failure.
* `dev` — registers one all-RAM `root` mount at `/`. Use this for SDK example smoke tests and standalone local poking. Does not pretend to be S3.
* `disk` — registers one disk-backed `root` mount at `/` under `--disk-root` or `VFS_DISK_ROOT`. The disk root directory must already exist.
* `mixed` — registers RAM-backed `/agents` plus disk-backed `/pools` and `/data` under `--disk-root` or `VFS_DISK_ROOT`. Use this when testing cross-mount behavior across heterogeneous backends.
```bash theme={null}
go run ./virtualfs/cmd/vfs serve --profile dev # all-RAM, no credentials needed
go run ./virtualfs/cmd/vfs serve --profile prod # root R2 mount; needs VFS_S3_BUCKET
mkdir -p .vfs-data
go run ./virtualfs/cmd/vfs serve --profile mixed --disk-root .vfs-data
```
When running it alongside the local conductor and dashboard, bind VirtualFS to `:8081` so the dashboard's `/api/vfs` file-route dev proxy can reach it:
```bash theme={null}
go run ./virtualfs/cmd/vfs serve --addr :8081
```
***
## Dashboard Development
The dashboard talks to the conductor at `/api` and to VirtualFS at `/api/vfs` for the Files page. In default local Vite dev mode, VirtualFS file routes under `/api/vfs` are proxied directly to the local VirtualFS server, while `/api/vfs/leases` remains on the conductor because it reports per-session VFS leases alongside the session API. Backend deployments can instead put VirtualFS behind the conductor by setting `VFS_BASE_URL`; the conductor then forwards `/api/vfs/*` to upstream `/vfs/*` and injects `VFS_AUTH_TOKEN` server-side when it is set.
For the Vercel-hosted dashboard, `dashboard/vercel.json` also rewrites `/vfs/*` directly to the deployed Railway VirtualFS service. `dashboard/middleware.ts` runs on that path and replaces the browser's placeholder `Authorization` header with `Bearer ${VFS_TOKEN}` before the rewrite reaches Railway. Set `VFS_TOKEN` in the Vercel project environment to the same value as `VFS_AUTH_TOKEN` on the Railway VirtualFS service; if it is missing, `/vfs/*` returns `500` with `VFS_TOKEN_UNSET`. Default local Vite development does not run this middleware, and the local VirtualFS server accepts the default `dev-token` bearer token.
Use `make frontend-prod` when you want to run the dashboard locally while pointing it at the deployed Railway backend. It sets `PROXY_TARGET=prod`, proxies all `/api` traffic to `PROD_CONDUCTOR_URL`, proxies legacy `/vfs` traffic to `PROD_VFS_URL`, and injects `Authorization: Bearer ${VFS_TOKEN}` on that legacy `/vfs` path when `VFS_TOKEN` is set. The dashboard still reads Clerk configuration from `dashboard/.env.local`, so those keys must match the Clerk instance accepted by the deployed conductor or `/api` calls will return `401`. Set `VFS_TOKEN` only when you need direct legacy `/vfs` file uploads. Override the Railway URLs from the shell when testing another environment:
```bash theme={null}
make frontend-prod \
PROD_CONDUCTOR_URL=https://conductor.example.com \
PROD_VFS_URL=https://virtualfs.example.com
```
Vite dev server config (`dashboard/vite.config.ts`) uses a local proxy by default:
```typescript theme={null}
export default defineConfig({
server: {
proxy: buildProxy(),
},
});
```
So start the conductor on `:8080`, start VirtualFS on `:8081` if you need the Files page, and run the Vite dev server on `:5173`.
***
## Common Development Issues
Ensure your reverse proxy (nginx, Caddy, Traefik) has buffering disabled:
```nginx theme={null}
proxy_buffering off;
proxy_cache off;
```
In local dev this isn't an issue since you're hitting the conductor directly.
Make sure `RUNNER_BASE_URL` is set to a URL the conductor can actually reach. If both run on `localhost`, `http://localhost:7070` works. In Docker, use the container name.
Check that the tool name in the profile's `tools` array matches exactly what the runner registry exports. Use `GET /runner/toolkit/specs` on the runner to see all registered tools.
If an MCP header contains `${VAR}`, verify the env var is set on the **runner** (not the conductor or sidecar). The runner resolves placeholders before forwarding.