Agent Observability Setup

# PBS Playbook: Agent Observability Setup (AI Agents)

## Purpose
Establish end-to-end visibility for AI agents so you can answer:
- **What happened?** (structured logs)
- **Where did time go?** (distributed traces)
- **Why did it fail?** (error tracking + enriched context)
- **Is it alive and on-time?** (health checks + cron/heartbeat monitoring)
- **Who gets paged and when?** (alert routing)
- **How are we trending?** (dashboards + SLOs)

---

## 0) Define the “Agent Observability Contract” (before tools)
**Decide and document these invariants:**
1. **Correlation keys**
   - `trace_id` (or `request_id`) created at ingress and propagated everywhere
   - `conversation_id` / `session_id`
   - `agent_run_id` (unique per run)
2. **What you will never log** (PII/PHI/secrets policy)
   - redact tokens, API keys, auth headers, full user content where needed
3. **Environment tags** always present
   - `service`, `env`, `version`, `region`, `deployment_id`
4. **Core KPIs** (golden signals)
   - latency, error rate, throughput, saturation/cost (tokens/$/tool calls)

**Deliverable:** a 1-page “observability contract” that engineering + ops agree on.

---

## 1) Minimal Viable Setup (MVS) vs Production-Hardened

### MVS (get value in 1–2 days)
- **Structured JSON logs** to a single sink (stdout → log platform)
- **Request/run correlation** (`trace_id` or `request_id`) in every log line
- **Error tracking** (Sentry/Rollbar/etc.) with release + environment tagging
- **Basic health endpoint** + external uptime check
- **Cron/worker heartbeat** (“dead man’s switch” alert if not seen)
- **One dashboard**: latency p95, failures, volume, cost estimate

### Production-Hardened (1–4+ weeks)
- **OpenTelemetry traces** with spans for: model calls, tools, retrieval, queue, DB
- **Metrics** (Prometheus/OTel metrics): histograms + counters with sane cardinality
- **Sampling strategy** (head/tail) and PII-aware payload scrubbing
- **SLOs + alerting** tied to user-impact, not noise (burn-rate alerts)
- **Runbooks** and “auto-triage” links from alerts → trace/log/error view
- **Audit-grade eventing** for safety / policy violations
- **Cost observability** (tokens, model choice, tool usage) with budgets and alerts

---

## 2) Step-by-step Implementation Playbook

### Step 2.1 — Structured logging (foundation)
**Goal:** Every event is machine-parseable and queryable.

**Implementation steps**
1. Choose a structured logger (language-specific).
2. Output **JSON** to stdout (12-factor friendly).
3. Enforce a **schema** and validate in CI (lint or unit tests).
4. Add middleware that injects:
   - `trace_id`, `agent_run_id`, `conversation_id`
   - `user_id` (hashed/pseudonymous), `tenant_id` if multi-tenant
5. Add **event types** for agent lifecycle:
   - `agent.run.started`
   - `agent.plan.created`
   - `agent.tool.called` / `agent.tool.result`
   - `agent.llm.request` / `agent.llm.response` (sanitized)
   - `agent.run.completed` / `agent.run.failed`
   - `agent.guardrail.triggered`

**Recommended log fields (baseline)**
- `timestamp`, `level`, `service`, `env`, `version`
- `event_name` (stable), `message` (human-readable)
- `trace_id`, `span_id` (if tracing), `agent_run_id`, `conversation_id`
- `duration_ms` (when applicable)
- `error.type`, `error.message`, `error.stack` (when applicable)
- AI-specific:
  - `model`, `provider`, `prompt_tokens`, `completion_tokens`, `total_tokens`
  - `tool_name`, `tool_latency_ms`, `tool_status`
  - `safety_category`, `policy_action` (blocked/allowed/redacted)

**Checklist**
- [ ] JSON logs everywhere (no ad-hoc println)
- [ ] Correlation IDs present in every log entry
- [ ] Redaction layer tested (unit tests with known secrets)
- [ ] Log volume and cost estimated (avoid logging huge payloads)

**Common pitfalls**
- Logging raw prompts/responses (PII leakage + high cost)
- High-cardinality fields (full URL with query params, user email, prompt text)
- Missing correlation IDs → impossible incident debugging

---

### Step 2.2 — Distributed tracing (where time went)
**Goal:** One trace shows the entire agent run across services/tools.

**Implementation steps**
1. Adopt **OpenTelemetry** (SDK + exporter).
2. Create a root span per agent run: `agent.run`.
3. Create child spans for major phases:
   - `agent.plan`
   - `retrieval.query` / `retrieval.rerank`
   - `llm.call` (one span per model invocation)
   - `tool.call:{tool}` (one span per tool call)
   - `db.query`, `http.client`, `queue.publish/consume`
4. Propagate context:
   - HTTP headers (`traceparent`)
   - message queues (inject/extract)
   - tool adapters (wrap tool calls so they inherit the span)
5. Attach attributes (careful with cardinality):
   - `agent.name`, `agent.version`
   - `model.name`, `model.provider`
   - `tool.name`, `tool.result` (status only; no payload)
   - `tokens.total`, `cost.usd_estimate` (if available)

**Sampling**
- Start with **100% sampling** in dev/staging.
- In prod, use:
  - head sampling low rate (e.g., 5–10%)
  - tail sampling for errors + slow traces (keep 100% of failures/timeouts)

**Checklist**
- [ ] One trace per agent run
- [ ] Spans for every LLM call + tool call
- [ ] Trace links from errors and logs (trace_id)
- [ ] Sensitive data never placed in span attributes

**Common pitfalls**
- Adding prompts as span attributes (security + cardinality blow-up)
- Broken context propagation across async/queue boundaries
- Too many unique span names (keep stable patterns)

---

### Step 2.3 — Error tracking (why it failed)
**Goal:** Exceptions and “soft failures” are captured with agent context.

**Implementation steps**
1. Integrate error tracker (e.g., Sentry) in each service.
2. Set `release`, `environment`, `service`.
3. On errors, attach:
   - `agent_run_id`, `conversation_id`, `trace_id`
   - `model`, `tool_name` (if relevant)
   - user/tenant identifiers (hashed)
4. Capture **non-exception failures** as events:
   - LLM refusals / policy blocks
   - tool timeouts
   - validation failures
   - “could not complete task” outcomes

**Checklist**
- [ ] Every exception has correlation IDs
- [ ] Known failure modes captured as typed events
- [ ] Alert rules exist for spikes in new issues/regressions

---

### Step 2.4 — Health checks (is it alive)
**Goal:** Detect outage conditions quickly and reliably.

**Implementation steps**
1. Implement:
   - **Liveness**: process is up
   - **Readiness**: can serve requests (critical dependencies OK)
2. Add dependency checks (with timeouts):
   - queue connectivity
   - DB ping
   - tool gateway availability
3. External uptime monitor hits readiness endpoint.

**Common pitfalls**
- Health checks that call real downstream APIs (adds load and flakiness)
- Liveness == readiness (hides dependency failures)

---

### Step 2.5 — Cron/worker monitoring (is it running on time)
**Goal:** Catch stuck schedulers and stalled workers.

**Implementation patterns**
1. **Heartbeat event**
   - Each job emits `cron.job.completed` with timestamp + exit code.
2. **Dead man’s switch**
   - Alert if no heartbeat in `N` minutes (N > expected interval × 2–3).
3. Track duration + last successful run time.

**Common pitfalls**
- Only alerting on failure, not on “never ran”
- No idempotency → retries cause duplicate side effects

---

### Step 2.6 — Alert routing (who gets notified, how)
**Goal:** Actionable alerts reach the right responder with context.

**Implementation steps**
1. Define severities (SEV1/2/3).
2. Route by category + ownership (infra vs model vs tool vs safety).
3. Attach context: dashboard link, top traces, top errors, recent deploy.

---

### Step 2.7 — Dashboards (what’s trending)
**Goal:** A single place to see the agent’s health, quality, and cost.

**Minimum dashboards**
1. Reliability: runs, success rate, error rate, p95 latency
2. Dependency: tool error rates/timeouts; provider/model 429s/5xx
3. Cost: tokens by model, cost/day estimate
4. Queue/throughput (if async): backlog depth, lag
5. Quality proxies: retry rate, handoff-to-human rate, guardrail triggers

---

## 3) Recommended SLOs (starter set)
- Availability: % runs that start and complete
- Latency: p95 runtime under X seconds
- Tool reliability: tool call success rate > Y%
- Safety: 0 critical policy violations; monitor near-misses

Alert using burn-rate (fast + slow) when possible.

---

## 4) Agent-specific observability enhancements
1. Prompt/response governance: log hashes/lengths/classifications; avoid raw text in general logs
2. Model/config capture: model, temperature, system prompt version, tool schema version
3. Tool call audit trail: tool name + redacted params + status + latency (detect loops)
4. Safety telemetry: explicit events for refusals, redactions, policy blocks

---

## 5) Final checklists

### MVS “Go Live” Checklist
- [ ] JSON structured logs with correlation IDs
- [ ] Error tracking integrated with release/env
- [ ] Health endpoints + uptime monitor
- [ ] Heartbeat for cron/worker + dead-man alert
- [ ] One dashboard with latency/error/volume/cost

### Production-Hardened Checklist
- [ ] OpenTelemetry traces with async propagation
- [ ] Metrics with bounded cardinality
- [ ] Sampling strategy (keep slow/error traces)
- [ ] SLOs + burn-rate alerting
- [ ] Alert routing by ownership + runbooks
- [ ] PII scrubbing and retention policies enforced
- [ ] Cost budgets + anomaly alerts

---

## 6) Pitfalls to avoid (top)
1. Logging raw prompts/responses in general logs
2. Missing correlation IDs at tool boundaries
3. High-cardinality labels (prompt text, full URLs, user emails)
4. Alerts without runbooks/links to traces
5. Paging on non-user-impact symptoms

{
  "cron": [
    {
      "name": "Weekly Agent Observability Review",
      "schedule": "0 9 * * 1",
      "task": "Review AI agent observability health for the past week. Check error rates, p95 latency, token costs, and any missed cron heartbeats. Use the 'Agent Observability Setup Playbook' checklists to identify gaps. Save findings to memory/observability/weekly-review-{YYYY-MM-DD}.md.",
      "agent": "cto",
      "model": "gemini-flash"
    }
  ]
}