> ## Documentation Index > Fetch the complete documentation index at: https://docs.orcapods.ai/llms.txt > Use this file to discover all available pages before exploring further. # Observability > Prometheus metrics, structured logs, and OpenTelemetry traces for Orca services. This is a self-hosting guide. The hosted product at [https://www.orcapods.ai](https://www.orcapods.ai) needs none of this — sign in and go. ## Overview Every Orca binary ships with three observability pillars out of the box: | Pillar | Technology | Endpoint | | ------- | ----------------- | -------------- | | Metrics | Prometheus | `GET /metrics` | | Logs | `go.uber.org/zap` | stdout | | Traces | OpenTelemetry | OTLP gRPC/HTTP | No instrumentation code is required. The HTTP metrics endpoint is registered by each service process. *** ## Public Status Page The standalone `status` service serves the public status page and JSON endpoints under `/api/*`, including `GET /api/status`, `GET /api/incidents`, and `GET /api/history`. The dashboard sidebar and landing footer show a live status indicator by polling the status endpoint every 60 seconds, then link to the status page. The dashboard reads its status page URL from the `VITE_STATUS_URL` environment variable; set it to your deployed status service. The landing site reads its status page URL from `landing/src/content/site.ts`. Browser reads are controlled by the status service's `STATUS_ALLOWED_ORIGINS` allowlist. Set it to a comma-separated list of first-party dashboard and landing origins that may read `/api/*` cross-origin. An empty value allows same-origin reads only, and a literal `*` restores wildcard CORS. ```bash theme={null} STATUS_ALLOWED_ORIGINS=https://dashboard.example.com,https://www.example.com ``` The poller opens an incident after a component reports `degraded` or `down` for `INCIDENT_OPEN_AFTER` consecutive polls, resolves it after `INCIDENT_RESOLVE_AFTER` consecutive operational polls, and worsens an open degraded incident when the component becomes `down`. Defaults are `INCIDENT_OPEN_AFTER=2` and `INCIDENT_RESOLVE_AFTER=3`. *** ## Metrics The conductor, runner, and standalone VirtualFS server expose Prometheus metrics at `/metrics`: ```bash theme={null} curl http://localhost:8080/metrics # Conductor curl http://localhost:7070/metrics # Runner curl http://localhost:8081/metrics # VirtualFS in make demo ``` ### Shared Runtime Metrics These collectors are registered by runner processes and by conductor components that execute runtime work locally. | Metric | Type | Description | | ---------------------------------------------------------- | --------- | ------------------------------------------------------------------------------- | | `runtime_runs_total{profile,runtime,status}` | Counter | Runs processed by profile, runtime, and terminal status | | `runtime_run_duration_seconds{profile,runtime}` | Histogram | End-to-end run duration | | `runtime_sessions_total{runtime,event}` | Counter | Session lifecycle events by runtime | | `runtime_sessions_active` | Gauge | Currently live sessions | | `runtime_session_lifetime_seconds` | Histogram | Session lifetime from create to shutdown | | `runtime_sidecar_request_duration_seconds{runtime}` | Histogram | Sidecar `POST /run` round-trip duration, including streaming | | `runtime_sidecar_request_errors_total{runtime,kind}` | Counter | Sidecar communication errors by kind | | `runtime_tool_calls_total{profile,runtime,tool,status}` | Counter | Tool calls observed in run streams | | `runtime_tool_call_duration_seconds{profile,runtime,tool}` | Histogram | Time between a `tool_call` event and its matching `tool_result` | | `runtime_tokens_total{profile,runtime,kind}` | Counter | Token usage, where `kind` is `input`, `output`, `cache_read`, or `cache_create` | ### Conductor Metrics | Metric | Type | Description | | ------------------------------------------------------------- | --------- | ------------------------------------------------------------ | | `conductor_runner_requests_total{runner_id,method,status}` | Counter | Requests sent from the conductor to remote runners | | `conductor_runner_request_duration_seconds{runner_id,method}` | Histogram | Remote runner request duration | | `conductor_runner_errors_total{runner_id,method,kind}` | Counter | Remote runner communication errors | | `conductor_routing_decisions_total{runner_hash,reason}` | Counter | Pool routing decisions by destination runner hash and reason | ### Planner Metrics | Metric | Type | Description | | ----------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `planner_plan_total{status}` | Counter | Workflow runs that reached a terminal status | | `planner_plan_node_total{status,profile}` | Counter | Workflow-run node terminal transitions by status and profile | | `planner_plan_repair_total{action}` | Counter | Workflow-run repair actions, such as `retry_node`, `replace_node`, `add_dependency`, or `abort` | | `planner_plan_active_nodes` | Gauge | Workflow-run nodes currently dispatched and in flight | | `planner_plan_duration_seconds` | Histogram | Wall-clock workflow-run duration from running to terminal status | | `planner_run_aborted_total{reason}` | Counter | Planner goroutines aborted before execution. Reasons include `no_tenant` and `store_get`; any non-zero value means a workflow run stayed pending instead of reaching execution. | ### VirtualFS Metrics These collectors are registered by `vfs serve`. Dispatcher labels keep mount names and statuses bounded, and HTTP path labels use the matched `http.ServeMux` pattern instead of raw tenant paths. | Metric | Type | Description | | ------------------------------------------------ | --------- | ---------------------------------------------------------------------------------------- | | `vfs_op_total{op,mount,status}` | Counter | Dispatcher operations such as `ls`, `stat`, `cat`, `write`, `delete`, `find`, and `grep` | | `vfs_op_duration_seconds{op,mount}` | Histogram | Dispatcher operation duration | | `vfs_cache_total{layer,result}` | Counter | Index and byte cache lookups by result: `hit`, `miss`, or `stale` | | `vfs_driver_errors_total{driver,kind}` | Counter | Mount driver errors by backend fingerprint and coarse operation kind | | `vfs_exec_total{status}` | Counter | `Workspace.Execute` calls by terminal status | | `vfs_exec_duration_seconds` | Histogram | `Workspace.Execute` duration | | `vfs_http_requests_total{method,path,status}` | Counter | HTTP requests handled by the VirtualFS API | | `vfs_http_request_duration_seconds{method,path}` | Histogram | VirtualFS HTTP request duration | ### Scrape Configuration ```yaml theme={null} # prometheus.yml scrape_configs: - job_name: orca-conductor static_configs: - targets: ["conductor:8080"] - job_name: orca-runner static_configs: - targets: - runner-0.runners.orca.svc:7070 - runner-1.runners.orca.svc:7070 - job_name: orca-vfs static_configs: - targets: ["vfs:8081"] ``` *** ## Structured Logs Orca uses `go.uber.org/zap` for machine-parseable logging. Most Go services emit JSON by default; `vfs serve` defaults to a compact console log format and switches to JSON with `LOG_FORMAT=json`. The VirtualFS HTTP API emits an `http.request` access log for every request with `request_id`, method, routed path, status, duration, and remote address. Handler error sites also log structured warning or error events with the same `request_id`, such as `auth.unauthorized`, `exec.executor_failed`, `cat.dispatch_failed`, `upload.write_failed`, and `tree.list_failed`, so a failed request can be correlated with the underlying auth, dispatcher, cache, presign, upload, or executor error. The conductor planner emits `planner.run_aborted` when a workflow run goroutine exits before execution, with `plan_id`, `reason`, and `error` fields. `reason` is `no_tenant` when the planner reached storage without tenant context, and `store_get` for other workflow-store lookup failures. Replay submission failures are logged as `planner.replay_submit_failed`. The conductor artifact path emits `artifacts.bucket_unavailable` at fatal level during startup when `S3_BUCKET` is configured but the bucket cannot be listed with the configured endpoint and credentials. If `INTERNAL_S3_BUCKET` is configured, the conductor also probes that internal run-event bucket; failures emit `internal_artifacts.bucket_unavailable` at fatal level, while non-fatal initialization errors emit `internal_artifacts.init_failed` and fall back to the primary artifact bucket. S3-backed run-event persistence emits `runs.s3.events_degraded` once per run when writing the JSONL event object fails; the log includes `run_id`, object `key`, `buffered_bytes`, `final`, and `error`. #### Runner Sandbox Boot Log Lines Runner startup logs the sandbox manager state so operators can tell whether each provider was registered or skipped: | Log message | Level | Fields | Meaning | | ----------------------------------- | ------ | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sandbox.configured` | `info` | `provider`, `default` | Emitted once for each registered provider in registration order. The first provider has `default=true`; later providers have `default=false`. | | `sandbox.configured` | `info` | `provider="(no default)"` | A sandbox manager exists but has no registered providers. | | `sandbox.provider_skipped` | `info` | `provider`, `reason="env key not set"` | Stock environment wiring skipped `e2b` or `daytona` because its credential variable was not set. | | `sandbox.stub` | `info` | none | No sandbox manager was configured, so sandbox tools run against the stub path. | | `worker_substrate.configured` | `info` | `substrates`, `default_substrate`, `runner_url`, `image`, `images_by_substrate` | Worker broker configuration, including every substrate this runner serves and its default. | | `worker_substrate.domain_allowlist` | `info` | `domains` | Final public-domain egress allowlist passed to the Daytona sandbox-worker substrate. Non-public hosts are omitted before this line is emitted. | | `worker_substrate.e2b_gaps` | `warn` | `no_auto_archive`, `no_domain_allowlist` | E2B workers have neither Daytona-style automatic archive nor the exact-domain egress allowlist; paused orphan disks require out-of-band cleanup and egress is unrestricted. | Sandbox provider-event handling emits `sandbox_event.unknown_provider` or `sandbox_event.undecodable` when a delivery is ignored, and `sandbox_event.invalidated` with `provider`, `sandbox_id`, `kind`, and matching `leases` after an accepted event is applied. `sandbox_event.fanout_shed` is a warning that the 32 concurrent fan-out slots were full; the webhook still received `202`, but cached liveness for those sandboxes stays stale until the next provider probe. #### Sandbox Worker Lifecycle Log Lines A `workerMode: "sandbox"` session runs a paid sandbox for its whole life, so these lines are the cost and correctness trail for it: | Log message | Level | Fields | Meaning | | -------------------------------------------- | --------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `worker.launched` | `info` | `session_id`, `substrate`, `worker_id`, `runtime`, `epoch` | A worker was booted. `epoch` fences this generation: a superseded worker polling the same session is answered with a shutdown command. | | `worker.died` | `warn` | `session_id`, `worker_id`, `last_output` | The lease was found dead and reaped. `last_output` is the worker's final log lines, pulled before Close destroys them, and is usually the only trace of a worker that died before it ever polled. | | `worker.idle_paused` | `info` | `session_id`, `worker_id`, `idle_for` | The runner's own idle policy paused a worker. Its billing clock stops here and restarts on the next run's resume. | | `worker.idle_pause_failed` | `warn` | `session_id`, `worker_id`, `error` | Pause failed; the worker keeps running and keeps billing, and the next scan retries. | | `worker.idle_pause_unsupported` | `debug` | `session_id`, `substrate` | The substrate has no suspend primitive (the process substrate). The worker is left running and is not retried. | | `worker_substrate.idle_pause_preempted` | `warn` | `worker_idle_pause`, `worker_idle_timeout`, `effect` | Misconfiguration: the provider auto-stop is at or below the runner's own threshold, so the provider always fires first and the platform's idle policy never applies. Raise `WORKER_IDLE_TIMEOUT` above `WORKER_IDLE_PAUSE`. | | `worker.closed` / `worker.close_failed` | `info` / `warn` | `session_id`, `worker_id` | Lease released at session shutdown. A failure here leaks a paid sandbox. | | `worker.close_all_unavailable` | `warn` | `effect` | The shutdown sweep could not run, so worker leases were not swept at exit. Signals a wiring regression rather than a runtime fault. | | `daytona.auto_archive_not_applied` | `warn` | `sandbox_id`, `requested_minutes`, `reported_minutes` | The provider answered the create with a different auto-archive interval than was asked for, so this sandbox keeps its disk far longer than intended. An unrecognised field does not fail a create, which is why this is checked rather than assumed. | | `worker_substrate.orphan_disk_unreclaimable` | `warn` | `cause`, `effect` | `WORKER_IDLE_TIMEOUT=0` with auto-archive configured. Auto-archive only applies to stopped sandboxes, so with nothing stopping them an orphan's disk is never released. | | `worker_proxy.disabled` | `warn` | `reason`, `impact` | Conductor booted without `WORKER_TOKEN_HMAC_KEY`, so the worker transport is not mounted and sandbox workers have no path to their runner. Fails closed: mounting it unverified would forward anonymous traffic to the runner with the runner-auth secret attached. | | `worker_proxy.forward_failed` | `warn` | `path`, `error` | The conductor could not reach the owning runner. A worker hanging up mid-poll is filtered out, so this means a genuine transport fault. | | `worker_proxy.inflight_capped` | `warn` | `path`, `max_inflight` | A verified session reached its concurrent worker-transport cap; the request returned `429` without being queued. | | `dynamic.mcp_token_mint_failed` | `warn` | `session_id`, `tenant_present`, `error` | The runner could not mint the sandbox session's per-run MCP capability, so the run fails before dispatch. An absent tenant is reported as `tenant_present: false`; the token itself is never logged. | | `dynamic.mcp_token_minted` | `debug` | `session_id`, `run_id`, `ttl` | A per-run MCP capability was minted. Only its lifetime is logged; the credential itself must never appear in logs. | | `worker.events.upload_timeout` | `warn` | `session_id`, `run_id`, `max_duration` | An events upload exceeded the 4-hour wall-clock limit. The run is ended with an error and the request returns `408`. | | `worker.events.upload_too_large` | `warn` | `session_id`, `run_id`, `cap_bytes` | An events upload exceeded the 256-MiB total-body cap. The run is ended with an error and the request returns `413`. | A provider auto-stop frees **compute but not disk**: a stopped cloud sandbox still holds its full disk allocation, and only deletion or archiving releases it. Orphans left by a runner that died without releasing its leases therefore accumulate against the provider's disk quota and eventually fail every new launch with a quota error, even though nothing is running. `WORKER_AUTO_ARCHIVE` is what bounds that; see the local-development guide. #### VirtualFS Boot Log Lines `vfs serve` emits the following structured log events during startup for each backend it attempts to initialize: | Log message | Level | Fields | Meaning | | ------------------------- | ------ | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `vfs.backend.ready` | `info` | `kind`, `bucket`, `endpoint`, `mounts` | Backend came up successfully; `mounts` lists the path prefixes bound to it. | | `vfs.backend.init_failed` | `warn` | `kind`, `bucket`, `endpoint`, `error` | Backend could not be initialized (e.g., S3 unreachable, missing credentials). Affected mounts are marked `unavailable`. The server continues to boot. | | `vfs.driver.ready` | `info` | `ready`, `unavailable` | Summary line at end of startup listing per-backend ready and unavailable mount counts. | Per-mount availability is also visible at runtime via `GET /vfs/mounts` — check the `status` and `reason` fields. ```json theme={null} { "level": "info", "ts": 1745582400.123, "caller": "runtime/session.go:42", "msg": "session created", "session_id": "sess-a1b2c3d4-e5f6g7h8", "profile": "researcher", "runner": "http://runner-1:7070" } ``` ### Log Fields Common fields included on structured log entries: | Field | Description | | ------------ | ----------------------------------------------------------------- | | `level` | `debug`, `info`, `warn`, `error` | | `ts` | Unix timestamp | | `caller` | Source file and line | | `msg` | Human-readable message | | `session_id` | Session identifier (when applicable) | | `run_id` | Run identifier (when applicable) | | `request_id` | HTTP request identifier for VirtualFS API access and handler logs | | `profile` | Profile name (when applicable) | ### Log Levels | Component | Default Level | | ------------ | ---------------------------- | | Conductor | `info` | | Runner | `info` | | VirtualFS | `info` | | agent-worker | `info` (via Node.js console) | Set `LOG_LEVEL=debug` on any Go service for verbose output. ### Querying Logs with jq ```bash theme={null} # All errors docker compose logs conductor-1 | grep '"level":"error"' | jq . # Events for a specific run docker compose logs runner-1 | jq '. | select(.run_id == "run-f3a9b72c")' # Tool invocations docker compose logs runner-1 | jq '. | select(.msg == "tool invoked")' ``` *** ## Distributed Tracing Orca exports OpenTelemetry traces in OTLP format. Configure the exporter endpoint: ```bash theme={null} # OTLP gRPC (default for most Jaeger/Tempo setups) OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317 # Or HTTP OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318 OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf ``` ### Trace Coverage Traces span the full request lifecycle: ```mermaid theme={null} flowchart TB run["Conductor
POST /api/runs"] pool["Pool.CreateSession"] session["Runner
POST /runner/sessions"] runnerRun["Runner
POST /runner/run"] sidecar["Sidecar
POST /run"] llm["LLM API call"] search["Tool
web_search"] extract["Tool
web_extract"] run --> pool --> session run --> runnerRun --> sidecar sidecar --> llm sidecar --> search sidecar --> extract ``` Trace context is propagated via standard W3C `traceparent` headers. VirtualFS adds spans for inbound HTTP requests, dispatcher operations, and `Workspace.Execute` calls when `OTEL_EXPORTER_OTLP_ENDPOINT` is set. ### Jaeger Docker Compose Integration ```yaml theme={null} # Add to docker-compose.yml jaeger: image: jaegertracing/all-in-one:latest ports: - "16686:16686" # UI - "4317:4317" # OTLP gRPC networks: [orca] ``` ```yaml theme={null} # Add to conductor and runner services environment: OTEL_EXPORTER_OTLP_ENDPOINT: "http://jaeger:4317" OTEL_SERVICE_NAME: "orca-conductor" # or "orca-runner" ``` Then visit [http://localhost:16686](http://localhost:16686) to see traces. *** ## Health Checks The conductor, runner, VirtualFS, and billing service expose `/healthz`: ```bash theme={null} curl http://localhost:8080/healthz # 200 OK # ok curl http://localhost:7070/healthz # 200 OK # ok curl http://localhost:8081/healthz # 200 OK curl http://localhost:8091/healthz # 200 OK ``` Use these for liveness probes in Kubernetes and load balancer health checks. *** ## Grafana Dashboard The metrics above compose naturally into a Grafana dashboard against your Prometheus data source (Dashboards → Import, or build from scratch). A useful starting layout includes panels for: * Active sessions per runner * Run throughput (runs/min) * Run duration percentiles (p50, p95, p99) * Token usage rate * Tool call breakdown * Sidecar error rate * Plan node throughput and repair counts *** ## Alerting Examples ```yaml theme={null} # Prometheus alert rules groups: - name: orca rules: - alert: OrcaRunnerDown expr: up{job="orca-runner"} == 0 for: 1m annotations: summary: "Orca runner {{ $labels.instance }} is down" - alert: OrcaHighErrorRate expr: | sum(rate(runtime_runs_total{status="error"}[5m])) / sum(rate(runtime_runs_total[5m])) > 0.1 for: 5m annotations: summary: "Orca run error rate above 10%" - alert: OrcaHighTokenUsage expr: sum(rate(runtime_tokens_total[1h])) > 100000 annotations: summary: "Orca token usage exceeding 100k tokens/hour" - alert: OrcaPlannerRepairSpike expr: sum(rate(planner_plan_repair_total[5m])) > 0.2 for: 10m annotations: summary: "Orca planner repair rate is elevated" ``` *** ## Centralized Logging (OTLP) Beyond stdout, every service can export structured logs over OTLP to any OpenTelemetry-compatible backend, such as SigNoz, Grafana, Datadog, or Honeycomb. OTLP log export is additive: stdout logging is never affected, so whatever log collection your platform already does keeps working. ### Ingest path ``` Go services — zap ──────────────► stdout (your platform's log collection, unchanged) └─► otelzap ──► OTLP logs ──┐ Go services — OTLP traces ────────────────────┼─► OTel Collector ─► backend of choice Node agent-worker / e2b-bridge — OTLP logs ───┘ (batching, transforms) ``` Run an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) between the services and your backend. It gives you one place for batching and attribute transforms, and lets you switch backends without touching service configuration. ### Environment variables | Variable | Value | Notes | | ----------------------------- | ------------------------------------------------- | --------------------------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_ENDPOINT` | Your collector, e.g. `http://otel-collector:4317` | Shared for traces and logs | | `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc` | Default; set explicitly | | `OTEL_SERVICE_NAME` | Per service (see table below) | Must be unique per service | | `OTEL_LOGS_ENABLED` | `false` by default | Gates the OTLP logs exporter; enable per service once your collector is reachable | Service name values: | Service | `OTEL_SERVICE_NAME` | | ------------ | ------------------- | | conductor | `conductor` | | runner | `runner` | | billing | `billing` | | vfs | `vfs` | | agent-worker | `agent-worker` | | e2b-bridge | `e2b-bridge` | When `OTEL_EXPORTER_OTLP_ENDPOINT` is unset or `OTEL_LOGS_ENABLED=false`, the logs exporter is a no-op and the service continues normally. Stdout is never affected by the flag. ### Log-to-trace correlation When `OTEL_LOGS_ENABLED=true`, the `otelzap` bridge attaches the active span's `trace_id` and `span_id` to each log record. Any backend that links logs and traces by these fields (most do) gives you click-through from a log line to its trace without manual correlation queries. ### Suggested starting points * Build your primary triage view as a saved log query filtering `severity_text IN (ERROR, FATAL)` grouped by `service.name`. * Retention of around 15 days for logs and 7 days for traces is a reasonable starting point; tune to your volume and budget.