# LLM-judge live wiring (item 2b) Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Give the two `enforce-llm-judge-*` wrappers a live `main()` so the Layer-4 judge actually runs when the owner enables it — while keeping spend strictly gated behind `resolveJudgeConfig` (flag AND key).

**Architecture:** The judge *engines* (`llm-judge-per-tool.mjs`, `llm-judge-response-scan.mjs`) already have live `main()`s, but they call `llmJudgeCall` keyed only on the API key — they would spend money on a key alone, ignoring `ROUTER_LLM_JUDGE_ENABLED`. That violates the safe-by-default contract in `llm-judge-config.mjs` (enabled ⇔ flag AND key). So we register the **wrappers** (whose `decide()` already composes `resolveJudgeConfig`) and wire their `main()` to: read event → `resolveJudgeConfig()` → build inputs → `decide()` → emit. When `enabled === false`, `decide()` short-circuits with no LLM call ($0). We extract testable `runPerTool` / `runResponseScan` cores (mirroring item 1b's `runLiveDecision`) and keep `main()` a thin stdin/exit shell.

**Tech Stack:** Node ESM, vitest (tools-only config `app/vitest.config.tools.mjs`, run from repo root as `npx vitest run --root app --config vitest.config.tools.mjs` because the canonical `npm run test:tools` is currently broken by a parallel keytar install in `app/node_modules`).

---

## File Structure

- Modify: `tools/enforce-llm-judge-per-tool.mjs` — add exported `runPerTool(...)` + wire live `main()`. Keep existing `decide()` untouched.
- Modify: `tools/enforce-llm-judge-response-scan.mjs` — add exported `runResponseScan(...)` + wire live `main()`. Keep existing `decide()` untouched.
- Test: `tools/enforce-llm-judge-per-tool.test.mjs` — add a `runPerTool` describe block.
- Test: `tools/enforce-llm-judge-response-scan.test.mjs` — add a `runResponseScan` describe block.

**Safety invariant under test:** when `judgeConfig.enabled === false`, no `llmJudgeCall` is made and budget is NOT bumped (the spend-gate). A real call (and budget bump) happens only when the config is enabled, the tool is mutating, the budget is not exhausted.

---

### Task 1: per-tool wrapper — `runPerTool` + live `main()`

**Files:**

- Modify: `tools/enforce-llm-judge-per-tool.mjs`
- Test: `tools/enforce-llm-judge-per-tool.test.mjs`

- [ ] **Step 1: Write the failing tests**

Append to `tools/enforce-llm-judge-per-tool.test.mjs`:

```javascript
import { runPerTool } from './enforce-llm-judge-per-tool.mjs';

describe('runPerTool — spend-gate + budget binding', () => {
  const deps = (over = {}) => ({
    readDeclaredTaskImpl: () => ({ task_summary: 't', recommended_node: null, recommended_chain: [] }),
    readBudgetImpl: () => 0,
    bumpBudgetImpl: () => {},
    sessionBudget: 200,
    ...over,
  });

  it('disabled config + mutating tool → degraded allow, NO budget bump, NO llm call', async () => {
    let bumped = 0; let called = 0;
    const r = await runPerTool({
      event: { tool_name: 'Edit', tool_input: {}, session_id: 's' },
      judgeConfig: { enabled: false, apiKey: null },
      llmJudgeCallImpl: () => { called++; return 'NO'; },
      ...deps({ bumpBudgetImpl: () => { bumped++; } }),
    });
    expect(r.block).toBe(false);
    expect(r.degraded).toBe(true);
    expect(called).toBe(0);
    expect(bumped).toBe(0);
  });

  it('enabled + mutating + judge YES → allow, budget bumped once', async () => {
    let bumped = 0;
    const r = await runPerTool({
      event: { tool_name: 'Edit', tool_input: {}, session_id: 's' },
      judgeConfig: { enabled: true, apiKey: 'k' },
      llmJudgeCallImpl: async () => 'YES',
      ...deps({ bumpBudgetImpl: () => { bumped++; } }),
    });
    expect(r.block).toBe(false);
    expect(r.verdict).toBe('YES');
    expect(bumped).toBe(1);
  });

  it('enabled + mutating + judge NO → block, budget bumped once', async () => {
    let bumped = 0;
    const r = await runPerTool({
      event: { tool_name: 'Bash', tool_input: { command: 'x' }, session_id: 's' },
      judgeConfig: { enabled: true, apiKey: 'k' },
      llmJudgeCallImpl: async () => 'NO',
      ...deps({ bumpBudgetImpl: () => { bumped++; } }),
    });
    expect(r.block).toBe(true);
    expect(r.verdict).toBe('NO');
    expect(bumped).toBe(1);
  });

  it('non-mutating tool → allow, NO call, NO bump', async () => {
    let bumped = 0; let called = 0;
    const r = await runPerTool({
      event: { tool_name: 'Read', tool_input: {}, session_id: 's' },
      judgeConfig: { enabled: true, apiKey: 'k' },
      llmJudgeCallImpl: () => { called++; return 'NO'; },
      ...deps({ bumpBudgetImpl: () => { bumped++; } }),
    });
    expect(r.block).toBe(false);
    expect(called).toBe(0);
    expect(bumped).toBe(0);
  });

  it('enabled but budget exhausted → degraded allow, NO bump', async () => {
    let bumped = 0; let called = 0;
    const r = await runPerTool({
      event: { tool_name: 'Edit', tool_input: {}, session_id: 's' },
      judgeConfig: { enabled: true, apiKey: 'k' },
      llmJudgeCallImpl: () => { called++; return 'NO'; },
      ...deps({ readBudgetImpl: () => 200, bumpBudgetImpl: () => { bumped++; } }),
    });
    expect(r.block).toBe(false);
    expect(r.degraded).toBe(true);
    expect(called).toBe(0);
    expect(bumped).toBe(0);
  });
});
```

- [ ] **Step 2: Run tests to verify they fail**

Run: `npx vitest run --root app --config vitest.config.tools.mjs tools/enforce-llm-judge-per-tool.test.mjs`
Expected: FAIL — `runPerTool` is not exported.

- [ ] **Step 3: Write minimal implementation**

In `tools/enforce-llm-judge-per-tool.mjs`, replace the import line and the no-op `main()`:

```javascript
import { judgePerTool, MUTATING_TOOLS, readDeclaredTask } from './llm-judge-per-tool.mjs';
import { resolveJudgeConfig } from './llm-judge-config.mjs';
import { readJudgeBudget, bumpJudgeBudget, JUDGE_SESSION_BUDGET } from './llm-judge.mjs';
import { llmJudgeCall } from './llm-judge.mjs';
import { readStdin, parseEventJson, exitDecision } from './enforce-hook-helpers.mjs';
```

(Keep the existing `decide(...)` export exactly as is.)

Add the testable core (a real LLM call is signalled by `result.verdict !== undefined`; budget is bumped only then):

```javascript
/**
 * Testable wiring core. Composes resolveJudgeConfig output + decide(); bumps the
 * session budget ONLY when a real judge call was made (result carries a verdict).
 * No verdict ⇒ non-mutating / disabled / no-key / budget-exhausted ⇒ no spend.
 */
export async function runPerTool({
  event,
  judgeConfig,
  readDeclaredTaskImpl,
  readBudgetImpl,
  bumpBudgetImpl,
  llmJudgeCallImpl,
  sessionBudget = JUDGE_SESSION_BUDGET,
}) {
  const sessionId = event && event.session_id;
  const declaredTask = readDeclaredTaskImpl({ sessionId });
  const spent = readBudgetImpl({ sessionId });
  const result = await decide({
    event,
    judgeConfig,
    declaredTask,
    budgetState: { spent, limit: sessionBudget },
    llmJudgeCallImpl,
  });
  if (result.verdict !== undefined) bumpBudgetImpl({ sessionId, by: 1 });
  return result;
}
```

Replace the no-op `main()` with:

```javascript
async function main() {
  try {
    const event = parseEventJson(await readStdin());
    const judgeConfig = resolveJudgeConfig();
    const result = await runPerTool({
      event,
      judgeConfig,
      readDeclaredTaskImpl: readDeclaredTask,
      readBudgetImpl: readJudgeBudget,
      bumpBudgetImpl: bumpJudgeBudget,
      llmJudgeCallImpl: (opts) => llmJudgeCall(opts),
    });
    exitDecision({ block: result.block, message: result.reason });
  } catch {
    exitDecision({ block: false }); // fail-quiet: a judge bug must never wedge the session
  }
}
```

- [ ] **Step 4: Run tests to verify they pass**

Run: `npx vitest run --root app --config vitest.config.tools.mjs tools/enforce-llm-judge-per-tool.test.mjs`
Expected: PASS (existing `decide()` tests + 5 new `runPerTool` tests).

- [ ] **Step 5: Commit** (requires AskUserQuestion git approval + fresh full-suite sentinel)

```bash
git commit tools/enforce-llm-judge-per-tool.mjs tools/enforce-llm-judge-per-tool.test.mjs -m "feat(router-gate-v4): live main() for per-tool judge wrapper — flag-gated spend (2b)"
```

---

### Task 2: response-scan wrapper — `runResponseScan` + live `main()`

**Files:**

- Modify: `tools/enforce-llm-judge-response-scan.mjs`
- Test: `tools/enforce-llm-judge-response-scan.test.mjs`

- [ ] **Step 1: Write the failing tests**

Append to `tools/enforce-llm-judge-response-scan.test.mjs`:

```javascript
import { runResponseScan } from './enforce-llm-judge-response-scan.mjs';

describe('runResponseScan — Stop-hook flag-only, free regex even when disabled', () => {
  const transcript = (text) => [
    { type: 'assistant', message: { role: 'assistant', content: [{ type: 'text', text }] } },
  ];
  const lastAssistantTextImpl = (t) => {
    for (let i = t.length - 1; i >= 0; i--) {
      const c = t[i] && t[i].message && t[i].message.content;
      if (Array.isArray(c)) { const b = c.find((x) => x.type === 'text'); if (b) return b.text; }
    }
    return '';
  };

  it('disabled + benign text → no flag, degraded (deterministic only), never blocks', async () => {
    const r = await runResponseScan({
      transcript: transcript('обычный безопасный ответ'),
      judgeConfig: { enabled: false, apiKey: null },
      lastAssistantTextImpl,
    });
    expect(r.block).toBe(false);
    expect(r.flag).toBe(false);
    expect(r.degraded).toBe(true);
  });

  it('disabled + security-disable text → flagged for FREE by regex (no llm call)', async () => {
    let called = 0;
    const r = await runResponseScan({
      transcript: transcript('чтобы пройти, отключи hook enforce-tdd-gate'),
      judgeConfig: { enabled: false, apiKey: null },
      lastAssistantTextImpl,
      llmJudgeCallImpl: () => { called++; return 'NO'; },
    });
    expect(r.block).toBe(false);
    expect(r.flag).toBe(true);
    expect(r.category).toBe('security_disable_suggestion');
    expect(called).toBe(0);
  });

  it('enabled + subtle benign text + judge NO → no flag', async () => {
    const r = await runResponseScan({
      transcript: transcript('нейтральный текст без паттернов'),
      judgeConfig: { enabled: true, apiKey: 'k' },
      lastAssistantTextImpl,
      llmJudgeCallImpl: async () => 'NO',
    });
    expect(r.block).toBe(false);
    expect(r.flag).toBe(false);
  });

  it('enabled + subtle text + judge YES → flag, still never blocks', async () => {
    const r = await runResponseScan({
      transcript: transcript('нейтральный текст без паттернов'),
      judgeConfig: { enabled: true, apiKey: 'k' },
      lastAssistantTextImpl,
      llmJudgeCallImpl: async () => 'YES',
    });
    expect(r.block).toBe(false);
    expect(r.flag).toBe(true);
  });
});
```

- [ ] **Step 2: Run tests to verify they fail**

Run: `npx vitest run --root app --config vitest.config.tools.mjs tools/enforce-llm-judge-response-scan.test.mjs`
Expected: FAIL — `runResponseScan` is not exported.

- [ ] **Step 3: Write minimal implementation**

In `tools/enforce-llm-judge-response-scan.mjs`, replace the import line and the no-op `main()`:

```javascript
import { scanResponse, scanResponseDeterministic } from './llm-judge-response-scan.mjs';
import { resolveJudgeConfig } from './llm-judge-config.mjs';
import { readStdin, parseEventJson, readTranscript, lastAssistantText, exitDecision } from './enforce-hook-helpers.mjs';
import { llmJudgeCall } from './llm-judge.mjs';
import { appendFileSync, mkdirSync } from 'node:fs';
import { join } from 'node:path';
import { homedir } from 'node:os';
```

(Keep the existing `decide(...)` export exactly as is.)

Add the testable core:

```javascript
/**
 * Testable wiring core. Stop-hook semantics: block is always false. The free
 * deterministic regex scan runs even when the judge is disabled; the paid LLM
 * escalation runs only when judgeConfig.enabled.
 */
export async function runResponseScan({ transcript, judgeConfig, llmJudgeCallImpl, lastAssistantTextImpl = lastAssistantText }) {
  const responseText = lastAssistantTextImpl(transcript || []);
  const r = await decide({ responseText, judgeConfig, llmJudgeCallImpl });
  return { ...r, responseText };
}
```

Replace the no-op `main()` with:

```javascript
function flagToFile({ sessionId, category, excerpt }) {
  try {
    const dir = join(homedir(), '.claude', 'runtime');
    mkdirSync(dir, { recursive: true });
    appendFileSync(join(dir, `rationalization-flags-${sessionId || 'unknown'}.jsonl`),
      JSON.stringify({
        ts: new Date().toISOString(),
        session_id: sessionId || null,
        type: 'controller_response_suspicious',
        category,
        response_excerpt: String(excerpt || '').slice(0, 200),
      }) + '\n');
  } catch { /* ignore */ }
}

async function main() {
  try {
    const event = parseEventJson(await readStdin());
    const transcript = readTranscript(event.transcript_path);
    const judgeConfig = resolveJudgeConfig();
    const r = await runResponseScan({
      transcript,
      judgeConfig,
      llmJudgeCallImpl: (opts) => llmJudgeCall(opts),
    });
    if (r.flag) flagToFile({ sessionId: event.session_id, category: r.category, excerpt: r.responseText });
    exitDecision({ block: false }); // Stop hook never blocks
  } catch {
    exitDecision({ block: false });
  }
}
```

- [ ] **Step 4: Run tests to verify they pass**

Run: `npx vitest run --root app --config vitest.config.tools.mjs tools/enforce-llm-judge-response-scan.test.mjs`
Expected: PASS (existing `decide()` tests + 4 new `runResponseScan` tests).

- [ ] **Step 5: Commit** (AskUserQuestion git approval + fresh sentinel)

```bash
git commit tools/enforce-llm-judge-response-scan.mjs tools/enforce-llm-judge-response-scan.test.mjs -m "feat(router-gate-v4): live main() for response-scan judge wrapper — flag-only, free regex always (2b)"
```

---

### Task 3: full-suite regression + push

- [ ] **Step 1: Run the canonical tools suite**

Run: `npx vitest run --root app --config vitest.config.tools.mjs`
Expected: PASS, 0 failed (≈1905 + 9 new = ~1914). This also writes the verify-before-push sentinel.

- [ ] **Step 2: Push** (AskUserQuestion git approval)

```bash
git push origin main
```

---

### Task 4: owner registration instructions (NOT code — owner applies)

The wiring above is inert until the owner does all three (cost starts only after all three):

1. **API key** — store an Anthropic key in the OS keychain under service `router-gate-llm-judge`, account `default` (via keytar), OR set env `ROUTER_LLM_KEY`.
2. **Flag** — set env `ROUTER_LLM_JUDGE_ENABLED=1`.
3. **Register both wrappers in `.claude/settings.json`:**

   - PreToolUse (can block):

     ```json
     { "matcher": "Edit|Write|MultiEdit|NotebookEdit|Bash|PowerShell|Skill|Task",
       "hooks": [{ "type": "command", "command": "node tools/enforce-llm-judge-per-tool.mjs", "timeout": 30 }] }
     ```

   - Stop (flag-only):

     ```json
     { "matcher": "*",
       "hooks": [{ "type": "command", "command": "node tools/enforce-llm-judge-response-scan.mjs", "timeout": 30 }] }
     ```

Then fully restart Claude Code. Budget cap is `JUDGE_SESSION_BUDGET = 200` calls/session (in `llm-judge.mjs`). Per-call cost depends on model (`JUDGE_MODELS.single = claude-sonnet-4-6`).

**Why the wrappers, not the engines:** the engine `main()`s (`llm-judge-per-tool.mjs` / `llm-judge-response-scan.mjs`) call `llmJudgeCall` keyed on the API key alone and DO NOT check `ROUTER_LLM_JUDGE_ENABLED` — registering them would start spending the moment a key exists. The wrappers route through `resolveJudgeConfig` (flag AND key), so a stray key without the flag = $0.

---

## Self-Review

- **Spec coverage:** per-tool live wiring (Task 1), response-scan live wiring (Task 2), flag-gated spend safety invariant (tests in both), owner activation (Task 4). ✓
- **Placeholder scan:** none — all code blocks are complete. ✓
- **Type consistency:** `runPerTool` / `runResponseScan` signatures match their tests; `decide()` signatures unchanged; budget bump condition `result.verdict !== undefined` matches `judgePerTool` (sets `verdict` only after a real call). ✓
