Token Cost Optimizer

# Token Cost Optimizer Playbook (SOP)

**Purpose:** Reduce LLM spend (tokens + latency) while maintaining quality, reliability, and safety.  
**Scope:** Model selection, prompt compression, caching, context management, and cost monitoring across all LLM-backed features.

---

# 1) Operating Principles

## 1.1 Cost is a Product Requirement
- Every endpoint has a **target cost per request** (CPR) and **target latency**.
- Quality must be measurable: define acceptance tests (golden set, human eval rubric, task success rate).

## 1.2 Optimize in This Order
1. **Stop sending tokens** (context/prompt trimming)
2. **Cache results** (avoid recomputation)
3. **Route to cheaper models** (or smaller context windows)
4. **Reduce output** (structured, bounded responses)
5. **Only then** tune sampling or retries

## 1.3 Always Measure Before/After
- No optimization is “done” without:
  - token usage deltas (input/output)
  - error rates
  - task success/quality score
  - p50/p95 latency
  - cache hit rates (if applicable)

---

# 2) Model Selection & Routing (SOP)

## 2.1 Model Tiers (Define for Your Org)
Create a simple tier list and keep it stable:

- **Tier A (Premium):** best reasoning / tool use / safety-critical
- **Tier B (Standard):** general-purpose, balanced
- **Tier C (Budget):** extraction, classification, templated writing
- **Tier D (Local/Rules):** deterministic transforms, regex, validators

> Keep “default model” conservative, then implement routing to downshift aggressively when safe.

## 2.2 Decision Tree: Pick the Cheapest Model That Works
Use this routing logic:

1. **Is it safety- or compliance-critical?** (medical, legal, finance, harmful content)  
   → Tier A + strict policy prompts + logging

2. **Is it deterministic/structured?** (extract fields, map categories, reformat)  
   → Tier C or Tier D (non-LLM) if possible

3. **Does it require multi-step reasoning?** (planning, debugging, ambiguous questions)  
   → Tier B; escalate to Tier A if confidence low

4. **Is context huge (>N tokens)?**  
   → Use retrieval + summarization; avoid “stuff it all in”

5. **Is this a high-volume endpoint?**  
   → Default to Tier C and selectively escalate

## 2.3 Escalation Policy (Fallback)
Implement “try cheap first, escalate only if needed.”

**Escalate when:**
- output fails schema/validator
- low confidence score (if you have it)
- user indicates dissatisfaction (“that’s wrong”, “doesn’t answer”)
- tool execution fails due to missing reasoning steps

**Escalation guardrails:**
- max 1–2 escalations per request
- record escalation reason
- ensure you don’t re-send full context unnecessarily on escalation (send minimal deltas)

### Checklist — Model Selection
- [ ] Endpoint has a documented model tier default
- [ ] Clear escalation triggers + max retries
- [ ] Output is schema-validated where possible
- [ ] High-volume endpoints default to cheapest acceptable tier
- [ ] Logs include: model, tokens in/out, latency, cache status, escalation reason

---

# 3) Prompt Compression (SOP)

## 3.1 Prompt Budgeting
Set budgets per request type:
- **Input budget:** max tokens for system + developer + user + context
- **Output budget:** max tokens + max items + max verbosity
- **Retries budget:** total token cap across retries/escalations

Example budget table:
- Chat support: input 3k, output 500, retries 1
- Extraction: input 2k, output 200, retries 0–1
- Code review: input 6k, output 800, retries 1

## 3.2 Compression Tactics (Use in This Order)
1. **Delete** irrelevant instructions/context
2. **Summarize** long history into a running state
3. **Replace** verbose text with compact schemas / bullet constraints
4. **Reference** stable policies by ID (don’t paste them repeatedly)
5. **Move** examples to a short “few-shot” set (1–2 examples max)

## 3.3 Prompt “Contract” Pattern (Compact + Stable)
Use a consistent minimal format:

**System:** role + safety boundaries  
**Developer:** task spec + schema + constraints  
**User:** current request  
**Context:** only facts needed, in structured form

### Example — After (Compressed Contract)
```text
SYSTEM: You are an assistant that follows instructions and outputs valid JSON.

DEVELOPER:
Task: Extract invoice fields.
Return JSON schema:
{ "vendor": string, "date": "YYYY-MM-DD", "total": number, "currency": string }
Rules:
- If unknown, use null.
- No extra keys. No prose.

CONTEXT (OCR snippet):
<only the relevant lines>

USER:
Extract the fields.
```

## 3.4 Output Bounding (The Cheapest Tokens Are Unused Tokens)
- Force structure: JSON, YAML, CSV
- Limit length: “max 5 bullets”, “max 120 words”
- Avoid chain-of-thought requests; request **short rationale** only if needed.

### Checklist — Prompt Compression
- [ ] Prompt template uses a compact contract
- [ ] Hard output bounds (length/items/schema)
- [ ] Long policies are referenced by ID, not pasted
- [ ] Chat history is summarized into “state” (not replayed)
- [ ] Few-shot examples are minimal (0–2), task-proven

---

# 4) Caching Strategy (SOP)

## 4.1 Cache Types
1. **Exact cache (strong):** same input → same output  
   Best for: classification, extraction, standard answers, templated outputs

2. **Semantic cache (soft):** similar input → reuse prior output  
   Best for: FAQs, short answers, boilerplate explanations  
   Requires: embeddings + similarity threshold + safety check

3. **Partial / component cache:** cache sub-results  
   Best for: multi-step pipelines (e.g., “summarize doc”, “extract entities”)

## 4.2 What to Cache (High ROI)
- Retrieval results (top-k doc IDs + snippets)
- Document summaries (by doc hash + version)
- Tool call results (weather, CRM lookup, product catalog)
- “Normalization” transforms (cleanup, language detection)

## 4.3 Cache Keys & TTL
**Cache key should include:**
- normalized user input (trim, lowercase where safe)
- relevant context hashes (doc version IDs, policy version)
- model + prompt version (to avoid mixing outputs)
- locale/user segment if it changes output

**TTL guidance:**
- Static FAQs: days–weeks
- Product catalog: hours–days (or versioned)
- User-specific answers: minutes–hours
- Safety-critical: very short or versioned + revalidation

## 4.4 Safety for Caching
- Do not cache sensitive personal data unless explicitly allowed.
- If semantic caching is used:
  - require similarity above threshold
  - optionally run a cheap verifier model to confirm answer matches question
  - never reuse cached answers across users when personalization matters

### Checklist — Caching
- [ ] Defined cache type per endpoint (exact/semantic/component)
- [ ] Cache keys include prompt+model+context versioning
- [ ] TTLs documented and justified
- [ ] PII policy applied (redaction or no-cache)
- [ ] Metrics: hit rate, stale rate, invalidation events, cost saved

---

# 5) Context Management (SOP)

## 5.1 Golden Rule
**Never send the whole world.** Send the minimum set of facts required to complete the task.

## 5.2 Patterns

### A) Rolling Summary (“State”) for Chat
Maintain two artifacts:
1. **Conversation State (short):** goals, constraints, preferences, open tasks
2. **Recent Turns (limited):** last N turns for tone and immediate references

**State format (compact):**
```yaml
goal: "Plan a 3-day Austin trip"
constraints: ["budget: mid", "no car", "vegetarian"]
decisions: ["hotel: downtown"]
open_questions: ["day 2 activities"]
```

### B) Retrieval-Augmented Context (RAG) for Knowledge
- Retrieve top-k chunks
- rerank and include only the best M
- include citations or chunk IDs
- provide only relevant excerpts, not whole documents

### C) Tool-First for Dynamic Data
If the user asks for account details, orders, inventory, schedules → call tools/databases first.

### Checklist — Context Management
- [ ] Chat uses rolling “state” summary
- [ ] RAG includes only top relevant snippets (with IDs)
- [ ] Hard cap on included history / chunk count
- [ ] Tool-first for dynamic/user-specific facts
- [ ] Context includes only what affects output correctness

---

# 6) Cost Monitoring & Controls (SOP)

## 6.1 What to Log Per Request
Minimum structured telemetry:
- request_id, endpoint, timestamp
- model, prompt_version
- tokens_in, tokens_out, total_tokens
- latency_ms (total + model time)
- cache_status (hit/miss; which layer)
- escalation_count + reason
- validator pass/fail

## 6.2 KPIs & Alerts
**KPIs:** cost per successful task, tokens/request (p50/p95), cache hit rate, escalation rate, retry rate, failure rate.

Alert on:
- p95 tokens +30% WoW
- cache hit rate drops
- escalation rate spikes
- cost/request exceeds budget

## 6.3 Guardrails (Hard Limits)
- per-request token cap
- per-user/day spend cap (if needed)
- degrade mode when spend spikes
- output caps under load

## 6.4 Weekly Optimization Cadence
1. Review top 10 endpoints by spend
2. Identify drivers (input vs output vs retries)
3. Apply one change at a time
4. Re-evaluate quality + cost
5. Document changes in a prompt/model changelog

---

# 7) Practical Examples (Drop-In Patterns)

## 7.1 “Cheap First, Escalate” Router (Concept)
```pseudo
result = call_model(tier="C", prompt=compact_prompt)

if !schema_valid(result) or confidence_low(result):
  result = call_model(tier="B", prompt=minimal_delta_prompt)

if still_bad and is_high_impact:
  result = call_model(tier="A", prompt=minimal_delta_prompt)

return result
```

## 7.2 Minimal-Delta Escalation Prompt
Send only:
- original question
- compact state
- failed output + validator errors
- instruction: fix only what failed

---

# 8) Rollout Plan (90-Minute “First Win” SOP)

1. Pick top 1 endpoint by cost volume
2. Add telemetry (tokens, model, cache status)
3. Implement output bounding + context trimming
4. Add exact caching if applicable
5. Add cheap-first routing with 1 escalation
6. Measure for 24 hours
7. Ship + document version bump

{
  "cron": [
    {
      "name": "Weekly Token Cost Review",
      "schedule": "0 15 * * 5",
      "task": "Review LLM token spend for the past week. Identify the top 3 endpoints by cost, check cache hit rates, and flag any escalation rate spikes. Use the 'Token Cost Optimizer Playbook' weekly cadence guide. Save findings to memory/cost-review/weekly-{YYYY-MM-DD}.md.",
      "agent": "cto",
      "model": "gemini-flash"
    }
  ]
}