Observability

This is a self-hosting guide. The hosted product at https://www.orcapods.ai needs none of this — sign in and go.

Overview

Every Orca binary ships with three observability pillars out of the box:

Pillar	Technology	Endpoint
Metrics	Prometheus	`GET /metrics`
Logs	`go.uber.org/zap`	stdout
Traces	OpenTelemetry	OTLP gRPC/HTTP

No instrumentation code is required. The HTTP metrics endpoint is registered by each service process.

Public Status Page

The standalone status service serves the public status page and JSON endpoints under /api/*, including GET /api/status, GET /api/incidents, and GET /api/history. The dashboard sidebar and landing footer show a live status indicator by polling the status endpoint every 60 seconds, then link to the status page. The dashboard reads its status page URL from the VITE_STATUS_URL environment variable; set it to your deployed status service. The landing site reads its status page URL from landing/src/content/site.ts. Browser reads are controlled by the status service’s STATUS_ALLOWED_ORIGINS allowlist. Set it to a comma-separated list of first-party dashboard and landing origins that may read /api/* cross-origin. An empty value allows same-origin reads only, and a literal * restores wildcard CORS.

STATUS_ALLOWED_ORIGINS=https://dashboard.example.com,https://www.example.com

The poller opens an incident after a component reports degraded or down for INCIDENT_OPEN_AFTER consecutive polls, resolves it after INCIDENT_RESOLVE_AFTER consecutive operational polls, and worsens an open degraded incident when the component becomes down. Defaults are INCIDENT_OPEN_AFTER=2 and INCIDENT_RESOLVE_AFTER=3.

Metrics

The conductor, runner, and standalone VirtualFS server expose Prometheus metrics at /metrics:

curl http://localhost:8080/metrics     # Conductor
curl http://localhost:7070/metrics     # Runner
curl http://localhost:8081/metrics     # VirtualFS in make demo

Shared Runtime Metrics

These collectors are registered by runner processes and by conductor components that execute runtime work locally.

Metric	Type	Description
`runtime_runs_total{profile,runtime,status}`	Counter	Runs processed by profile, runtime, and terminal status
`runtime_run_duration_seconds{profile,runtime}`	Histogram	End-to-end run duration
`runtime_sessions_total{runtime,event}`	Counter	Session lifecycle events by runtime
`runtime_sessions_active`	Gauge	Currently live sessions
`runtime_session_lifetime_seconds`	Histogram	Session lifetime from create to shutdown
`runtime_sidecar_request_duration_seconds{runtime}`	Histogram	Sidecar `POST /run` round-trip duration, including streaming
`runtime_sidecar_request_errors_total{runtime,kind}`	Counter	Sidecar communication errors by kind
`runtime_tool_calls_total{profile,runtime,tool,status}`	Counter	Tool calls observed in run streams
`runtime_tool_call_duration_seconds{profile,runtime,tool}`	Histogram	Time between a `tool_call` event and its matching `tool_result`
`runtime_tokens_total{profile,runtime,kind}`	Counter	Token usage, where `kind` is `input`, `output`, `cache_read`, or `cache_create`

Conductor Metrics

Metric	Type	Description
`conductor_runner_requests_total{runner_id,method,status}`	Counter	Requests sent from the conductor to remote runners
`conductor_runner_request_duration_seconds{runner_id,method}`	Histogram	Remote runner request duration
`conductor_runner_errors_total{runner_id,method,kind}`	Counter	Remote runner communication errors
`conductor_routing_decisions_total{runner_hash,reason}`	Counter	Pool routing decisions by destination runner hash and reason

Planner Metrics

Metric	Type	Description
`planner_plan_total{status}`	Counter	Workflow runs that reached a terminal status
`planner_plan_node_total{status,profile}`	Counter	Workflow-run node terminal transitions by status and profile
`planner_plan_repair_total{action}`	Counter	Workflow-run repair actions, such as `retry_node`, `replace_node`, `add_dependency`, or `abort`
`planner_plan_active_nodes`	Gauge	Workflow-run nodes currently dispatched and in flight
`planner_plan_duration_seconds`	Histogram	Wall-clock workflow-run duration from running to terminal status
`planner_run_aborted_total{reason}`	Counter	Planner goroutines aborted before execution. Reasons include `no_tenant` and `store_get`; any non-zero value means a workflow run stayed pending instead of reaching execution.

VirtualFS Metrics

These collectors are registered by vfs serve. Dispatcher labels keep mount names and statuses bounded, and HTTP path labels use the matched http.ServeMux pattern instead of raw tenant paths.

Metric	Type	Description
`vfs_op_total{op,mount,status}`	Counter	Dispatcher operations such as `ls`, `stat`, `cat`, `write`, `delete`, `find`, and `grep`
`vfs_op_duration_seconds{op,mount}`	Histogram	Dispatcher operation duration
`vfs_cache_total{layer,result}`	Counter	Index and byte cache lookups by result: `hit`, `miss`, or `stale`
`vfs_driver_errors_total{driver,kind}`	Counter	Mount driver errors by backend fingerprint and coarse operation kind
`vfs_exec_total{status}`	Counter	`Workspace.Execute` calls by terminal status
`vfs_exec_duration_seconds`	Histogram	`Workspace.Execute` duration
`vfs_http_requests_total{method,path,status}`	Counter	HTTP requests handled by the VirtualFS API
`vfs_http_request_duration_seconds{method,path}`	Histogram	VirtualFS HTTP request duration

Scrape Configuration

# prometheus.yml
scrape_configs:
  - job_name: orca-conductor
    static_configs:
      - targets: ["conductor:8080"]

  - job_name: orca-runner
    static_configs:
      - targets:
          - runner-0.runners.orca.svc:7070
          - runner-1.runners.orca.svc:7070

  - job_name: orca-vfs
    static_configs:
      - targets: ["vfs:8081"]

Structured Logs

Orca uses go.uber.org/zap for machine-parseable logging. Most Go services emit JSON by default; vfs serve defaults to a compact console log format and switches to JSON with LOG_FORMAT=json. The VirtualFS HTTP API emits an http.request access log for every request with request_id, method, routed path, status, duration, and remote address. Handler error sites also log structured warning or error events with the same request_id, such as auth.unauthorized, exec.executor_failed, cat.dispatch_failed, upload.write_failed, and tree.list_failed, so a failed request can be correlated with the underlying auth, dispatcher, cache, presign, upload, or executor error. The conductor planner emits planner.run_aborted when a workflow run goroutine exits before execution, with plan_id, reason, and error fields. reason is no_tenant when the planner reached storage without tenant context, and store_get for other workflow-store lookup failures. Replay submission failures are logged as planner.replay_submit_failed. The conductor artifact path emits artifacts.bucket_unavailable at fatal level during startup when S3_BUCKET is configured but the bucket cannot be listed with the configured endpoint and credentials. If INTERNAL_S3_BUCKET is configured, the conductor also probes that internal run-event bucket; failures emit internal_artifacts.bucket_unavailable at fatal level, while non-fatal initialization errors emit internal_artifacts.init_failed and fall back to the primary artifact bucket. S3-backed run-event persistence emits runs.s3.events_degraded once per run when writing the JSONL event object fails; the log includes run_id, object key, buffered_bytes, final, and error.

Runner Sandbox Boot Log Lines

Runner startup logs the sandbox manager state so operators can tell whether each provider was registered or skipped:

Log message	Level	Fields	Meaning
`sandbox.configured`	`info`	`provider`, `default`	Emitted once for each registered provider in registration order. The first provider has `default=true`; later providers have `default=false`.
`sandbox.configured`	`info`	`provider="(no default)"`	A sandbox manager exists but has no registered providers.
`sandbox.provider_skipped`	`info`	`provider`, `reason="env key not set"`	Stock environment wiring skipped `e2b` or `daytona` because its credential variable was not set.
`sandbox.stub`	`info`	none	No sandbox manager was configured, so sandbox tools run against the stub path.
`worker_substrate.configured`	`info`	`substrates`, `default_substrate`, `runner_url`, `image`, `images_by_substrate`	Worker broker configuration, including every substrate this runner serves and its default.
`worker_substrate.domain_allowlist`	`info`	`domains`	Final public-domain egress allowlist passed to the Daytona sandbox-worker substrate. Non-public hosts are omitted before this line is emitted.
`worker_substrate.e2b_gaps`	`warn`	`no_auto_archive`, `no_domain_allowlist`	E2B workers have neither Daytona-style automatic archive nor the exact-domain egress allowlist; paused orphan disks require out-of-band cleanup and egress is unrestricted.

Sandbox provider-event handling emits sandbox_event.unknown_provider or sandbox_event.undecodable when a delivery is ignored, and sandbox_event.invalidated with provider, sandbox_id, kind, and matching leases after an accepted event is applied. sandbox_event.fanout_shed is a warning that the 32 concurrent fan-out slots were full; the webhook still received 202, but cached liveness for those sandboxes stays stale until the next provider probe.

Sandbox Worker Lifecycle Log Lines

A workerMode: "sandbox" session runs a paid sandbox for its whole life, so these lines are the cost and correctness trail for it:

Log message	Level	Fields	Meaning
`worker.launched`	`info`	`session_id`, `substrate`, `worker_id`, `runtime`, `epoch`	A worker was booted. `epoch` fences this generation: a superseded worker polling the same session is answered with a shutdown command.
`worker.died`	`warn`	`session_id`, `worker_id`, `last_output`	The lease was found dead and reaped. `last_output` is the worker’s final log lines, pulled before Close destroys them, and is usually the only trace of a worker that died before it ever polled.
`worker.idle_paused`	`info`	`session_id`, `worker_id`, `idle_for`	The runner’s own idle policy paused a worker. Its billing clock stops here and restarts on the next run’s resume.
`worker.idle_pause_failed`	`warn`	`session_id`, `worker_id`, `error`	Pause failed; the worker keeps running and keeps billing, and the next scan retries.
`worker.idle_pause_unsupported`	`debug`	`session_id`, `substrate`	The substrate has no suspend primitive (the process substrate). The worker is left running and is not retried.
`worker_substrate.idle_pause_preempted`	`warn`	`worker_idle_pause`, `worker_idle_timeout`, `effect`	Misconfiguration: the provider auto-stop is at or below the runner’s own threshold, so the provider always fires first and the platform’s idle policy never applies. Raise `WORKER_IDLE_TIMEOUT` above `WORKER_IDLE_PAUSE`.
`worker.closed` / `worker.close_failed`	`info` / `warn`	`session_id`, `worker_id`	Lease released at session shutdown. A failure here leaks a paid sandbox.
`worker.close_all_unavailable`	`warn`	`effect`	The shutdown sweep could not run, so worker leases were not swept at exit. Signals a wiring regression rather than a runtime fault.
`daytona.auto_archive_not_applied`	`warn`	`sandbox_id`, `requested_minutes`, `reported_minutes`	The provider answered the create with a different auto-archive interval than was asked for, so this sandbox keeps its disk far longer than intended. An unrecognised field does not fail a create, which is why this is checked rather than assumed.
`worker_substrate.orphan_disk_unreclaimable`	`warn`	`cause`, `effect`	`WORKER_IDLE_TIMEOUT=0` with auto-archive configured. Auto-archive only applies to stopped sandboxes, so with nothing stopping them an orphan’s disk is never released.
`worker_proxy.disabled`	`warn`	`reason`, `impact`	Conductor booted without `WORKER_TOKEN_HMAC_KEY`, so the worker transport is not mounted and sandbox workers have no path to their runner. Fails closed: mounting it unverified would forward anonymous traffic to the runner with the runner-auth secret attached.
`worker_proxy.forward_failed`	`warn`	`path`, `error`	The conductor could not reach the owning runner. A worker hanging up mid-poll is filtered out, so this means a genuine transport fault.
`worker_proxy.inflight_capped`	`warn`	`path`, `max_inflight`	A verified session reached its concurrent worker-transport cap; the request returned `429` without being queued.
`dynamic.mcp_token_mint_failed`	`warn`	`session_id`, `tenant_present`, `error`	The runner could not mint the sandbox session’s per-run MCP capability, so the run fails before dispatch. An absent tenant is reported as `tenant_present: false`; the token itself is never logged.
`dynamic.mcp_token_minted`	`debug`	`session_id`, `run_id`, `ttl`	A per-run MCP capability was minted. Only its lifetime is logged; the credential itself must never appear in logs.
`worker.events.upload_timeout`	`warn`	`session_id`, `run_id`, `max_duration`	An events upload exceeded the 4-hour wall-clock limit. The run is ended with an error and the request returns `408`.
`worker.events.upload_too_large`	`warn`	`session_id`, `run_id`, `cap_bytes`	An events upload exceeded the 256-MiB total-body cap. The run is ended with an error and the request returns `413`.

A provider auto-stop frees compute but not disk: a stopped cloud sandbox still holds its full disk allocation, and only deletion or archiving releases it. Orphans left by a runner that died without releasing its leases therefore accumulate against the provider’s disk quota and eventually fail every new launch with a quota error, even though nothing is running. WORKER_AUTO_ARCHIVE is what bounds that; see the local-development guide.

VirtualFS Boot Log Lines

vfs serve emits the following structured log events during startup for each backend it attempts to initialize:

Log message	Level	Fields	Meaning
`vfs.backend.ready`	`info`	`kind`, `bucket`, `endpoint`, `mounts`	Backend came up successfully; `mounts` lists the path prefixes bound to it.
`vfs.backend.init_failed`	`warn`	`kind`, `bucket`, `endpoint`, `error`	Backend could not be initialized (e.g., S3 unreachable, missing credentials). Affected mounts are marked `unavailable`. The server continues to boot.
`vfs.driver.ready`	`info`	`ready`, `unavailable`	Summary line at end of startup listing per-backend ready and unavailable mount counts.

Per-mount availability is also visible at runtime via GET /vfs/mounts — check the status and reason fields.

{
  "level": "info",
  "ts": 1745582400.123,
  "caller": "runtime/session.go:42",
  "msg": "session created",
  "session_id": "sess-a1b2c3d4-e5f6g7h8",
  "profile": "researcher",
  "runner": "http://runner-1:7070"
}

Log Fields

Common fields included on structured log entries:

Field	Description
`level`	`debug`, `info`, `warn`, `error`
`ts`	Unix timestamp
`caller`	Source file and line
`msg`	Human-readable message
`session_id`	Session identifier (when applicable)
`run_id`	Run identifier (when applicable)
`request_id`	HTTP request identifier for VirtualFS API access and handler logs
`profile`	Profile name (when applicable)

Log Levels

Component	Default Level
Conductor	`info`
Runner	`info`
VirtualFS	`info`
agent-worker	`info` (via Node.js console)

Set LOG_LEVEL=debug on any Go service for verbose output.

Querying Logs with jq

# All errors
docker compose logs conductor-1 | grep '"level":"error"' | jq .

# Events for a specific run
docker compose logs runner-1 | jq '. | select(.run_id == "run-f3a9b72c")'

# Tool invocations
docker compose logs runner-1 | jq '. | select(.msg == "tool invoked")'

Distributed Tracing

Orca exports OpenTelemetry traces in OTLP format. Configure the exporter endpoint:

# OTLP gRPC (default for most Jaeger/Tempo setups)
OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317

# Or HTTP
OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Trace Coverage

Traces span the full request lifecycle: Trace context is propagated via standard W3C traceparent headers. VirtualFS adds spans for inbound HTTP requests, dispatcher operations, and Workspace.Execute calls when OTEL_EXPORTER_OTLP_ENDPOINT is set.

Jaeger Docker Compose Integration

# Add to docker-compose.yml
jaeger:
  image: jaegertracing/all-in-one:latest
  ports:
    - "16686:16686"   # UI
    - "4317:4317"     # OTLP gRPC
  networks: [orca]

# Add to conductor and runner services
environment:
  OTEL_EXPORTER_OTLP_ENDPOINT: "http://jaeger:4317"
  OTEL_SERVICE_NAME: "orca-conductor"  # or "orca-runner"

Then visit http://localhost:16686 to see traces.

Health Checks

The conductor, runner, VirtualFS, and billing service expose /healthz:

curl http://localhost:8080/healthz
# 200 OK
# ok

curl http://localhost:7070/healthz
# 200 OK
# ok

curl http://localhost:8081/healthz
# 200 OK

curl http://localhost:8091/healthz
# 200 OK

Use these for liveness probes in Kubernetes and load balancer health checks.

Grafana Dashboard

The metrics above compose naturally into a Grafana dashboard against your Prometheus data source (Dashboards → Import, or build from scratch). A useful starting layout includes panels for:

Active sessions per runner
Run throughput (runs/min)
Run duration percentiles (p50, p95, p99)
Token usage rate
Tool call breakdown
Sidecar error rate
Plan node throughput and repair counts

Alerting Examples

# Prometheus alert rules
groups:
  - name: orca
    rules:
      - alert: OrcaRunnerDown
        expr: up{job="orca-runner"} == 0
        for: 1m
        annotations:
          summary: "Orca runner {{ $labels.instance }} is down"

      - alert: OrcaHighErrorRate
        expr: |
          sum(rate(runtime_runs_total{status="error"}[5m]))
          / sum(rate(runtime_runs_total[5m])) > 0.1
        for: 5m
        annotations:
          summary: "Orca run error rate above 10%"

      - alert: OrcaHighTokenUsage
        expr: sum(rate(runtime_tokens_total[1h])) > 100000
        annotations:
          summary: "Orca token usage exceeding 100k tokens/hour"

      - alert: OrcaPlannerRepairSpike
        expr: sum(rate(planner_plan_repair_total[5m])) > 0.2
        for: 10m
        annotations:
          summary: "Orca planner repair rate is elevated"

Centralized Logging (OTLP)

Beyond stdout, every service can export structured logs over OTLP to any OpenTelemetry-compatible backend, such as SigNoz, Grafana, Datadog, or Honeycomb. OTLP log export is additive: stdout logging is never affected, so whatever log collection your platform already does keeps working.

Ingest path

Go services — zap ──────────────► stdout (your platform's log collection, unchanged)
                  └─► otelzap ──► OTLP logs ──┐
Go services — OTLP traces ────────────────────┼─► OTel Collector ─► backend of choice
Node agent-worker / e2b-bridge — OTLP logs ───┘      (batching, transforms)

Run an OpenTelemetry Collector between the services and your backend. It gives you one place for batching and attribute transforms, and lets you switch backends without touching service configuration.

Environment variables

Variable	Value	Notes
`OTEL_EXPORTER_OTLP_ENDPOINT`	Your collector, e.g. `http://otel-collector:4317`	Shared for traces and logs
`OTEL_EXPORTER_OTLP_PROTOCOL`	`grpc`	Default; set explicitly
`OTEL_SERVICE_NAME`	Per service (see table below)	Must be unique per service
`OTEL_LOGS_ENABLED`	`false` by default	Gates the OTLP logs exporter; enable per service once your collector is reachable

Service name values:

Service	`OTEL_SERVICE_NAME`
conductor	`conductor`
runner	`runner`
billing	`billing`
vfs	`vfs`
agent-worker	`agent-worker`
e2b-bridge	`e2b-bridge`

When OTEL_EXPORTER_OTLP_ENDPOINT is unset or OTEL_LOGS_ENABLED=false, the logs exporter is a no-op and the service continues normally. Stdout is never affected by the flag.

Log-to-trace correlation

When OTEL_LOGS_ENABLED=true, the otelzap bridge attaches the active span’s trace_id and span_id to each log record. Any backend that links logs and traces by these fields (most do) gives you click-through from a log line to its trace without manual correlation queries.

Suggested starting points

Build your primary triage view as a saved log query filtering severity_text IN (ERROR, FATAL) grouped by service.name.
Retention of around 15 days for logs and 7 days for traces is a reasonable starting point; tune to your volume and budget.

Get Started

Core Concepts

Guides

Observability

Overview

Public Status Page

Metrics

Shared Runtime Metrics

Conductor Metrics

Planner Metrics

VirtualFS Metrics

Scrape Configuration

Structured Logs

Runner Sandbox Boot Log Lines

Sandbox Worker Lifecycle Log Lines

VirtualFS Boot Log Lines

Log Fields

Log Levels

Querying Logs with jq

Distributed Tracing

Trace Coverage

Jaeger Docker Compose Integration

Health Checks

Grafana Dashboard

Alerting Examples

Centralized Logging (OTLP)

Ingest path

Environment variables

Log-to-trace correlation

Suggested starting points

​Overview

​Public Status Page

​Metrics

​Shared Runtime Metrics

​Conductor Metrics

​Planner Metrics

​VirtualFS Metrics

​Scrape Configuration

​Structured Logs

​Runner Sandbox Boot Log Lines

​Sandbox Worker Lifecycle Log Lines

​VirtualFS Boot Log Lines

​Log Fields

​Log Levels

​Querying Logs with jq

​Distributed Tracing

​Trace Coverage

​Jaeger Docker Compose Integration

​Health Checks

​Grafana Dashboard

​Alerting Examples

​Centralized Logging (OTLP)

​Ingest path

​Environment variables

​Log-to-trace correlation

​Suggested starting points

Overview

Public Status Page

Metrics

Shared Runtime Metrics

Conductor Metrics

Planner Metrics

VirtualFS Metrics

Scrape Configuration

Structured Logs

Runner Sandbox Boot Log Lines

Sandbox Worker Lifecycle Log Lines

VirtualFS Boot Log Lines

Log Fields

Log Levels

Querying Logs with jq

Distributed Tracing

Trace Coverage

Jaeger Docker Compose Integration

Health Checks

Grafana Dashboard

Alerting Examples

Centralized Logging (OTLP)

Ingest path

Environment variables

Log-to-trace correlation

Suggested starting points