Kubernetes Deployment

This is a self-hosting guide. The hosted product at https://www.orcapods.ai needs none of this. Sign in and go.

Architecture on Kubernetes

Key points:

Conductors: Deployment + HPA (stateless, scale freely)
Runners: StatefulSet + headless Service (stable pod DNS = stable RUNNER_BASE_URL)
Sidecars: additional containers inside each runner pod
Ingress: nginx with proxy_buffering off for SSE

Conductor Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: conductor
  namespace: orca
spec:
  replicas: 3
  selector:
    matchLabels:
      app: conductor
  template:
    metadata:
      labels:
        app: conductor
    spec:
      containers:
        - name: conductor
          image: orca/conductor:latest
          ports:
            - containerPort: 8080
          env:
            - name: CONDUCTOR_PORT
              value: "8080"
            - name: RUNNER_URLS
              value: >-
                http://<runner-0>.<runner-service>.<cluster-domain>:7070,
                http://<runner-1>.<runner-service>.<cluster-domain>:7070,
                http://<runner-2>.<runner-service>.<cluster-domain>:7070
            - name: AGENT_ORC_RUNS_DIR
              value: /data/runs
            - name: AGENT_ORC_PLANS_DIR
              value: /data/plans.jsonl
            - name: S3_BUCKET
              value: agent-artifacts
            - name: AWS_REGION
              value: us-east-1
            - name: AWS_ENDPOINT_URL_S3
              value: "http://<storage-service>:9000"
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: s3-access-key-id
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: s3-secret-access-key
            - name: S3_FORCE_PATH_STYLE
              value: "true"
            - name: METRICS_AUTH_TOKEN
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: metrics-auth-token
          volumeMounts:
            - name: runs-storage
              mountPath: /data/runs
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: 1
              memory: 512Mi
      volumes:
        - name: runs-storage
          persistentVolumeClaim:
            claimName: conductor-runs-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: conductor
  namespace: orca
spec:
  selector:
    app: conductor
  ports:
    - port: 8080
      targetPort: 8080
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: conductor-hpa
  namespace: orca
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: conductor
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

Runner StatefulSet

Runners must have stable DNS names so conductors can route sessions to them. MCP_BASE_URL is the address a runner advertises to its own sidecars for MCP callbacks, and it defaults to RUNNER_BASE_URL when unset. In a multi-replica StatefulSet, template it per pod the same way as RUNNER_BASE_URL: pinning it to a single replica’s hostname across all pods misroutes every other replica’s sidecar tool calls.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: runner
  namespace: orca
spec:
  serviceName: runners          # headless service name
  replicas: 3
  selector:
    matchLabels:
      app: runner
  template:
    metadata:
      labels:
        app: runner
    spec:
      containers:
        # --- Go Runner ---
        - name: runner
          image: orca/runner:latest
          ports:
            - containerPort: 7070
          env:
            - name: RUNNER_PORT
              value: "7070"
            - name: RUNNER_BASE_URL
              value: "http://<pod-hostname>.<runner-service>.<cluster-domain>:7070"
            - name: MCP_BASE_URL
              value: "http://<pod-hostname>.<runner-service>.<cluster-domain>:7070"
            - name: RUNNER_CAPABILITIES
              value: "claude,vercel"
            - name: CLAUDE_SIDECAR_URL
              value: "http://claude-sidecar:7071"
            - name: VERCEL_SIDECAR_URL
              value: "http://vercel-sidecar:7073"
            - name: ANTHROPIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: anthropic-api-key
            - name: TAVILY_API_KEY
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: tavily-api-key
            - name: S3_BUCKET
              value: agent-artifacts
            - name: AWS_REGION
              value: us-east-1
            - name: AWS_ENDPOINT_URL_S3
              value: "http://<storage-service>:9000"
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: s3-access-key-id
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: s3-secret-access-key
            - name: S3_FORCE_PATH_STYLE
              value: "true"
            - name: METRICS_AUTH_TOKEN
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: metrics-auth-token
          readinessProbe:
            httpGet:
              path: /healthz
              port: 7070
            initialDelaySeconds: 10
            periodSeconds: 15
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: 2
              memory: 2Gi

        # --- Claude Sidecar ---
        - name: <claude-sidecar>
          image: orca/agent-worker:latest
          env:
            - name: MODE
              value: claude
            - name: PORT
              value: "7071"
            - name: ANTHROPIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: anthropic-api-key
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: 1
              memory: 512Mi

        # --- Vercel Sidecar ---
        - name: <vercel-sidecar>
          image: orca/agent-worker:latest
          env:
            - name: MODE
              value: vercel
            - name: PORT
              value: "7073"
            - name: ANTHROPIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: anthropic-api-key
            - name: OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: openai-api-key
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: 1
              memory: 512Mi
---
# Headless service for stable DNS
apiVersion: v1
kind: Service
metadata:
  name: runners
  namespace: orca
spec:
  clusterIP: None
  selector:
    app: runner
  ports:
    - port: 7070
      targetPort: 7070

With this StatefulSet and headless service, pods get stable DNS names:

<runner-0>.<runner-service>.<cluster-domain>:7070
<runner-1>.<runner-service>.<cluster-domain>:7070
<runner-2>.<runner-service>.<cluster-domain>:7070

Set these in RUNNER_URLS on the conductor.

Secrets

apiVersion: v1
kind: Secret
metadata:
  name: orca-secrets
  namespace: orca
type: Opaque
stringData:
  anthropic-api-key: "${ANTHROPIC_API_KEY}"
  openai-api-key: "${OPENAI_API_KEY}"
  tavily-api-key: "${TAVILY_API_KEY}"
  s3-access-key-id: "${AWS_ACCESS_KEY_ID}"
  s3-secret-access-key: "${AWS_SECRET_ACCESS_KEY}"
  metrics-auth-token: "${METRICS_AUTH_TOKEN}"

VirtualFS Strict Mount Configuration

When deploying the standalone VirtualFS service (orca/vfs) to Kubernetes, the strict mount model requires explicit backend configuration. Do NOT use --profile dev in production: the dev profile uses in-process RAM and cannot persist data across pod restarts. Required environment variables for the prod profile:

Variable	Description
`VFS_S3_BUCKET`	R2/S3 bucket name for the production root mount
`VFS_S3_ENDPOINT`	S3-compatible endpoint URL (e.g., `https://<your-account>.r2.cloudflarestorage.com`)
`AWS_ACCESS_KEY_ID`	S3/R2 access key
`AWS_SECRET_ACCESS_KEY`	S3/R2 secret key
`VFS_AUTH_TOKEN`	Bearer token for VirtualFS API authentication

If these are missing or the bucket is unreachable at pod startup, the affected mounts surface as unavailable in GET /vfs/mounts. The server still passes /healthz: liveness checks are independent of mount status. Use the VirtualFS smoke script (scripts/vfs-smoke.sh) in a post-deploy readiness check to assert all expected mounts are ready before traffic is sent.

# VirtualFS Deployment excerpt: add to your manifest
env:
  - name: VFS_S3_BUCKET
    valueFrom:
      secretKeyRef:
        name: orca-secrets
        key: vfs-s3-bucket
  - name: VFS_S3_ENDPOINT
    value: "https://<your-account>.r2.cloudflarestorage.com"
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: orca-secrets
        key: s3-access-key-id
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: orca-secrets
        key: s3-secret-access-key
  - name: VFS_AUTH_TOKEN
    valueFrom:
      secretKeyRef:
        name: orca-secrets
        key: vfs-auth-token

Ingress with SSE Support

Without proxy-buffering: "false", Server-Sent Events will not stream correctly. nginx-ingress will buffer the entire response.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: orca
  namespace: orca
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
spec:
  ingressClassName: nginx
  rules:
    - host: orca.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: conductor
                port:
                  number: 8080

Updating Runner Count

When you scale the StatefulSet, update the conductor’s RUNNER_URLS to include the new pod DNS names. The safest approach:

Scale the StatefulSet: kubectl scale statefulset runner --replicas=4 -n orca
Wait for the new pod to be ready: kubectl rollout status statefulset runner -n orca
Update the conductor Deployment env to include <runner-3>.<runner-service>.<cluster-domain>:7070
Roll out the conductor: kubectl rollout restart deployment conductor -n orca

New sessions will round-robin to include the new runner. Existing sessions remain pinned to their original runner.

Namespace and RBAC

apiVersion: v1
kind: Namespace
metadata:
  name: orca
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: orca
  namespace: orca

Monitoring Integration

Both conductor and runner expose /metrics in Prometheus text format, gated behind a bearer token: set METRICS_AUTH_TOKEN on the conductor Deployment and the runner StatefulSet, or every scrape gets a 401 Unauthorized. Create a ServiceMonitor if you’re using the Prometheus Operator, pointing bearerTokenSecret at that same token:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: orca
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: conductor
  namespaceSelector:
    matchNames: [orca]
  endpoints:
    - port: http
      path: /metrics
      interval: 30s
      bearerTokenSecret:
        name: orca-secrets
        key: metrics-auth-token

Get Started

Core Concepts

Guides

Runbooks

Kubernetes Deployment

Architecture on Kubernetes

Conductor Deployment

Runner StatefulSet

Secrets

VirtualFS Strict Mount Configuration

Ingress with SSE Support

Updating Runner Count

Namespace and RBAC

Monitoring Integration

​Architecture on Kubernetes

​Conductor Deployment

​Runner StatefulSet

​Secrets

​VirtualFS Strict Mount Configuration

​Ingress with SSE Support

​Updating Runner Count

​Namespace and RBAC

​Monitoring Integration

Architecture on Kubernetes

Conductor Deployment

Runner StatefulSet

Secrets

VirtualFS Strict Mount Configuration

Ingress with SSE Support

Updating Runner Count

Namespace and RBAC

Monitoring Integration