# A11 ML / AI Tooling Integration Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Populate the empty `A11 «ML / AI-разработка»` map section with a conflict-minimal ML/AI toolset — document a **reuse core** (claude-api skill + context7 MCP + Sentry MCP), install **two new light tools** (promptfoo, a vendored Data Scientist skill), and register **Jupyter MCP as a deferred reserved slot** — so A11 becomes a working playbook.

**Architecture:** A11 is an **empty** functional section — `NODE_SECTION` in `docs/automation-graph.html` tags zero nodes `A11`. Approach А (chosen 2026-05-17): a **reuse layer** (claude-api skill, context7 MCP, Sentry MCP — all already installed; A11 documents the coverage, never re-tags their nodes) plus **two new installs** — promptfoo as a root `package.json` devDependency, and a Data Scientist skill vendored as a standalone skill into `.claude/skills/data-scientist/` (no plugin, no marketplace, no hooks — the A6 mermaid pattern). **Jupyter MCP** (executable notebooks) is **deferred** — registered now as a reserved slot, installed later by a separate severable task gated on a Python ML environment + a concrete model to train. All A11 tools are non-UI → a new **ml-ai-tooling** off-phase category, outside the PSR_v1 UI-pool. A11 artifacts live in `docs/ml/`.

**Tech Stack:** promptfoo (`promptfoo/promptfoo`, npm package `promptfoo`, MIT); a Data Scientist skill (vendored standalone skill, MIT/permissive — exact repo resolved in Task 1); Jupyter MCP (`datalayer/jupyter-mcp-server` — deferred, not installed); the already-installed claude-api skill / context7 MCP / Sentry MCP (reuse); project normative docs; `docs/automation-graph.html` (vis.js).

**Sequencing (2026-05-17):** the A6 / D3 / C9 epics land and push **first** (they touch the same shared files: the map, 4 normative docs, the Tooling counter). A11 then rebases onto the updated `origin/main` (Task 8 Step 1). The working branch `feat/a11-ml-ai-tooling` already exists and holds the brainstorming spec commit (`ae423be`). A11's Tooling numbers are runtime-resolved (NUM1) — never hard-coded before reading the live counter. Push pattern: `git push origin feat/a11-ml-ai-tooling:main`.

---

## Tool Identity (verified 2026-05-17 via WebSearch)

| # | Tool | Install mode | Source / License | Hooks? |
|---|---|---|---|---|
| 1 | **promptfoo** — CLI test-suite for LLM prompts/agents/RAG: declarative `promptfooconfig.yaml`, assertions (`equals`/`contains`/`llm-rubric`/cost/latency), model comparison, red-teaming. Invoked `npx promptfoo`. | npm **devDependency** in the root `package.json` (`npm i -D promptfoo`) — version-pinned via `package-lock.json` (ML9) | GitHub `promptfoo/promptfoo`, **MIT** (OpenAI-owned since 2026, remains OSS) | None — npm CLI, no Claude Code lifecycle hooks |
| 2 | **Data Scientist skill** — knowledge-only skill: business objective → ML task, algorithm selection (Linear Regression … XGBoost), feature engineering, experiment-tracking + A/B-analysis guidance. | Standalone skill — **vendored** copy into `.claude/skills/data-scientist/` (no plugin, no marketplace) | Exact repo **resolved Task 1 Step 4** — candidates: `secondsky/claude-skills` (the A6 architecture-patterns marketplace), `alirezarezvani/claude-skills` (263+ skills), or a dedicated data-science skill repo. MIT/permissive required. | None disclosed (skills-only) — **verify on vendor** |
| — | **Jupyter MCP** (`datalayer/jupyter-mcp-server`) — executable notebooks: insert/run cells, read outputs, plots. | **NOT installed** — deferred reserved slot (see "Deferred Task"). | GitHub `datalayer/jupyter-mcp-server` | n/a — not installed |
| — | **claude-api skill** / **context7 MCP** / **Sentry MCP** | **Reuse** — already installed | claude-api: plugin skill; context7: MCP (`mcp__plugin_context7_context7__*`); Sentry: Tooling #34 | n/a |

**Verification status:** promptfoo — repo, MIT, npm package name, config-file convention confirmed via WebSearch. Data Scientist skill — the *category* (151+ data-science skills in the marketplace) and the "Data Scientist" skill *concept* confirmed; the **exact source repo is NOT pinned** → Task 1 Step 4 resolves it with concrete criteria. Jupyter MCP — `datalayer/jupyter-mcp-server` confirmed to exist; **not installed by this plan**.

**Deferred (with reason — no task in this plan):**

- **Jupyter MCP** — would be the 8th `.mcp.json` server; experimental (Notebook 6.x only); requires a Python ML environment that the native-Windows machine deliberately lacks; and there is no model to train. Registered in the Tooling registry as a **pending** slot (Task 6); installed later by the severable task described in "Deferred Task" below.

**Dropped (with reason — no task, no slot):**

- **A dedicated LLM-observability tool** (Langfuse / Helicone) — redundant with Sentry's AI/LLM monitoring on the already-installed Sentry MCP (#34); CLAUDE.md §5 п.6 (no two tools for one job).
- **A standalone Claude-API plugin** — the claude-api skill is already available; installing a second is a §5 п.6 duplication.
- **An `ml-developer`-style agent plugin** — the agent layer is already crowded; promptfoo + the Data Scientist skill + claude-api cover the workflow without a new agent.

---

## Design Decisions & Conflict Audit

Pattern follows the K1–K8 / AK1–CC1 / CP1–NUM1 audits used for claude-mem and the A6 / C9 / D3 plans. Verified against the promptfoo repo, project `.claude/settings.json`, `~/.claude/settings.json`, `.mcp.json`, `lefthook.yml`, `cspell.json`, `.markdownlintignore`, root `package.json`, and the A6 plan.

| # | Tool | Sev | Conflict | Resolution (locked) |
|---|---|---|---|---|
| ML1 | promptfoo | 🟡 | A real eval run needs an Anthropic API key and makes **paid** LLM calls. Putting it in a hook / pre-commit would cost money on every commit and could break the economy chain. | promptfoo runs **manually / CI-only** — never in a hook, never a lefthook job, never auto. No `lefthook.yml` change. The API key lives in an env var (`ANTHROPIC_API_KEY`, PowerShell User scope — the Sentry `SENTRY_AUTH_TOKEN` pattern), never committed. Documented in `docs/ml/README.md` (Task 4) + the Tooling entry (Task 6). |
| ML2 | promptfoo | 🟢 | promptfoo's red-team module overlaps the D3 audit-security tools. | None — promptfoo red-team tests *LLM prompts* for jailbreak/injection; D3 Trail of Bits (#39) + Semgrep (#25) are SAST of *code*. Different objects. Boundary stated in `docs/ml/README.md` (Task 4) + the Tooling entry (Task 6). |
| ML3 | Data Scientist skill | 🟡 | Vendored `.claude/skills/data-scientist/**/*.md` (third-party English files) is caught by the cspell + markdownlint pre-commit jobs. | Add `.claude/skills/data-scientist/**` to `cspell.json` `ignorePaths` and `.claude/skills/data-scientist/` to `.markdownlintignore` (Task 3 Step 4) — the A6 MK1 pattern. The project's own skills (`q-item-add`, `rls-check`, `regression`, `mermaid`) stay linted. |
| ML4 | reuse layer | 🟢 | Re-tagging the existing `context7` (E7) / `mcp_sentry` (A7) map nodes to A11 would empty their current sections — `NODE_SECTION` is 1-node→1-section. | Reuse nodes **stay** in their sections. A11 gets its **own** new nodes (`claude_api`, `promptfoo`, `data_scientist`). The reuse coverage is documented in `docs/ml/README.md` (Task 4) and noted in the new nodes' details (Task 7). Same as A6/D3/C9 REU1. |
| ML5 | all | 🟢 | A11 is non-UI tooling. | New off-phase category **ml-ai-tooling**, outside the PSR_v1 UI-pool → no R6.0/R6.1 stack-filter, no R14 pipeline — same treatment as `claude-md-management` (infrastructure), the A6 architecture-tooling, and the D3 audit-security categories. Registered in PSR_v1 R10.1 + Pravila §13.2 (Task 6). |
| ML6 | promptfoo | 🟢 | 8 PreToolUse/UserPromptSubmit/Stop economy + skill-discipline + ruflo hooks — does promptfoo touch them? | None — promptfoo is an npm CLI invoked on demand; it registers zero Claude Code lifecycle hooks. Re-verified Task 2 Step 5. |
| ML7 | all | 🟡 | Bus-factor — the Data Scientist skill is a community project; Jupyter MCP is community + experimental. | The Data Scientist skill is **vendored** → immune to upstream loss (the A6 mermaid pattern). promptfoo is MIT, OpenAI-owned, stable. Jupyter MCP is deferred and will be version-pinned at install. Noted in the Tooling entry (Task 6). No alpha-substrate spike needed for the core scope — neither promptfoo nor the vendored skill has a known-broken core (unlike ruflo K7). |
| ML8 | reuse layer | 🟡 | claude-api skill / context7 MCP are used but may not be **formalized** Tooling-registry positions → using them unregistered is a PSR_v1 R0.2/R10 gap. | Task 1 Step 5 audits whether claude-api and context7 are already in `~/.claude/settings.json` `enabledPlugins` + the Tooling registry. Task 6 registers whatever is missing (as reuse positions). Sentry MCP is already Tooling #34. |
| ML9 | Jupyter MCP | 🟡 | Registering a tool that is not installed could mislead a future reader into thinking it is available. | The Tooling entry + `docs/ml/README.md` + the map mark Jupyter MCP explicitly **«pending — не установлен, severable-задача»** (the Sentry #34 "pending Sentry deployment" precedent). No `jupyter_mcp` map node is added until the deferred task runs. |
| NUM1 | normative sync | 🟡 | The A6 plan claimed Tooling #36-#38, D3 claimed #39-#40, C9 claims the next after that. A11 must not collide. | Task 1 Step 6 + Task 6 Step 1 read the **live** `docs/Tooling_v8_3.md` Прил. Н §0 counter and assign A11's numbers sequentially after whatever is current. Never hard-code a number before reading the live counter. |

**Severable scope.** Core A11 = Tasks 1-8 (reuse documentation + promptfoo + Data Scientist skill + normative + map + finish) — already populates and closes the section. There is **no severable task inside this plan** (unlike A6's Task 5): the only deferred piece, Jupyter MCP, is a *future* task outside this plan (see "Deferred Task"). A11 adds **no lefthook job** and **no `.mcp.json` change** — fewer conflicts by design (the C9 shape).

---

## File Structure

| File | Created / Modified | Responsibility |
|---|---|---|
| `docs/ml/` | Create dir | A11 home — the ML/AI playbook |
| `docs/ml/README.md` | Create | The ML/AI convention: tool boundaries (claude-api skill = *build* / promptfoo = *test* prompts / Data Scientist skill = classical-ML *workflow* / Jupyter MCP = *execute* — deferred); the reuse-layer map; the promptfoo API-key + manual-run note (ML1) |
| `docs/ml/promptfoo-example/promptfooconfig.yaml` | Create | One seed eval config — a worked lead-qualification prompt example |
| `docs/ml/promptfoo-example/README.md` | Create | How to run the example (`npx promptfoo eval`), the API-key requirement, the "never in CI/hooks" rule |
| `.claude/skills/data-scientist/` | Create (vendored) | The Data Scientist skill — `SKILL.md` + `references/` |
| `docs/adr/ADR-005-ml-ai-tooling.md` | Create (conditional — adr-kit/A6 landed) | Seed ADR documenting the A11 tooling decision + the Python/Jupyter defer |
| `package.json` (repo root) | Modify | `promptfoo` devDependency |
| `package-lock.json` (repo root) | Modify | promptfoo dependency tree (written by `npm i`) |
| `cspell.json` | Modify | `ignorePaths` += `.claude/skills/data-scientist/**` (ML3) |
| `.markdownlintignore` | Modify | += `.claude/skills/data-scientist/` (ML3) |
| `cspell-words.txt` | Modify (conditional) | New ML/AI vocabulary |
| `docs/Tooling_v8_3.md` | Modify | Прил. Н — new ml-ai-tooling subsection(s) + §0 counter bump |
| `docs/Plugin_stack_rules_v1.md` | Modify | R10.1 — new ml-ai-tooling rows |
| `docs/Pravila_raboty_Claude_v1_1.md` | Modify | §13.2 — ml-ai-tooling category note |
| `CLAUDE.md` | Modify (**via claude-md-management only**) | §3 title count, §1 row 2b count, new §3.3 ml-ai-tooling row(s) |
| `docs/CHANGELOG_claude_md.md` | Modify | CLAUDE.md version-bump entry |
| `docs/automation-graph.html` | Modify | 3 new A11 nodes (`claude_api`, `promptfoo`, `data_scientist`) → `NODE_SECTION` A11; header metrics |
| `.mcp.json` | **NOT modified** | Jupyter MCP deferred — `.mcp.json` is untouched by this plan |

---

## Task 1: Pre-flight — baseline, branch, snapshot, fact-check

**Files:** none modified (read-only)

- [ ] **Step 1: Confirm tree state and branch**

```bash
cd "c:/моя/проекты/портал crm/Документация"
git status --short
git rev-parse --short HEAD
git branch --show-current
```

Expected: branch `feat/a11-ml-ai-tooling`, HEAD at the spec commit `ae423be` (or later). Record the HEAD SHA as the regression baseline. (If a different branch — `git checkout feat/a11-ml-ai-tooling`.)

- [ ] **Step 2: Snapshot the hook chain**

Read `.claude/settings.json`, `.claude/settings.local.json` (if present), and `~/.claude/settings.json`. Record every hook on `SessionStart`, `UserPromptSubmit`, `PreToolUse`, `PostToolUse`, `PreCompact`, `PostCompact`, `Stop`. This is the ML6 baseline — Task 2 compares against it.
Expected (`~/.claude/settings.json`): SessionStart economy-self-check; PreToolUse skill-marker/skill-check/economy-state-guard/CLAUDE.md-warn/security-guidance; UserPromptSubmit economy-mode; PostCompact economy-postcompact; Stop economy-verifier. Project `.claude/settings.json`: ruflo-recall + ruflo-queen (UserPromptSubmit), markdownlint-fix + schema-CHANGELOG-reminder (PostToolUse).

- [ ] **Step 3: Baseline regression**

```
/regression quick
```

Expected: GREEN. Record the current Pest / Vitest counts from the last green run (memory `project_state.md`). A11 touches no `app/` code → the final run (Task 8) must match.

- [ ] **Step 4: Fact-check promptfoo + resolve the Data Scientist skill repo**

promptfoo — open `https://github.com/promptfoo/promptfoo` and confirm: npm package `promptfoo`, MIT license, `promptfooconfig.yaml` config convention, the Anthropic provider id form (`anthropic:messages:<model>`), no Claude Code lifecycle hooks.

Data Scientist skill — resolve the exact source. WebFetch the candidates and pick **one** repo that satisfies ALL of:

- ships a real standalone **skill** (`SKILL.md` + optional `references/`), not a plugin requiring marketplace machinery;
- MIT or other permissive license;
- covers classical-ML workflow (algorithm selection, feature engineering, evaluation, experiment tracking);
- no `hooks` block / no Claude Code lifecycle hooks.

Candidates, in priority order: (a) `secondsky/claude-skills` (already a project marketplace from A6 — check for a `data-science`/`data-scientist` skill dir), (b) `alirezarezvani/claude-skills`, (c) a dedicated data-science skill repo found via WebSearch. Record the chosen repo URL + license + the in-repo path to the skill directory. If a candidate registers CC lifecycle hooks → reject it, try the next.

If **no** candidate qualifies → **stop**, report to the user; A11 falls back to a project-authored minimal `data-scientist` skill (out of this plan's scope).

- [ ] **Step 5: Audit reuse-tool registration (ML8)**

Check whether the reuse tools are already formalized:

- `~/.claude/settings.json` `enabledPlugins` — is the claude-api skill's backing plugin listed? Record yes/no + the plugin id.
- `.mcp.json` / `~/.claude/settings.json` — is context7 configured? (It is — `mcp__plugin_context7_context7__*` tools exist.) Record the server/plugin id.
- `docs/Tooling_v8_3.md` Прил. Н — grep for `claude-api`, `context7`. Record which (if any) already have a Tooling number.

This drives Task 6: register whatever reuse tool is missing a Tooling position.

- [ ] **Step 6: Check A6 landed + read the live Tooling counter (NUM1)**

```bash
git log --oneline | grep -iE "adr-kit|architecture-patterns|mermaid|ADR-00" | head
ls docs/adr/ 2>/dev/null
```

Read `docs/Tooling_v8_3.md` Прил. Н §0 — record the **live** tool counter. Record: **A6 landed?** yes/no (drives Task 4's conditional ADR-005 — `docs/adr/ADR-000-adr-process.md` exists ⇒ yes) and the counter value (drives Task 6 numbering). (Note: the Task-0 pre-commit run already showed `adr-judge` as lefthook job 9 + "1 ADR with Enforcement" — A6 has very likely landed; confirm here.)

No repo files changed → no commit.

---

## Task 2: Install promptfoo (npm devDependency — ML1/ML9)

**Files:** Modify `package.json`, `package-lock.json` (repo root)

- [ ] **Step 1: Inspect the root `package.json`**

Read the repo-root `package.json`. Confirm it is the doc-tooling project (scripts `lint:md`, `spell`, `links`, `a11y`; devDependencies `markdownlint-cli2`, `cspell`, etc.). Record the current `devDependencies` shape.

- [ ] **Step 2: Install promptfoo as a devDependency**

```bash
cd "c:/моя/проекты/портал crm/Документация"
npm install --save-dev promptfoo
```

Expected: `promptfoo` appears in `package.json` `devDependencies` with a pinned `^`-range; `package-lock.json` updated. If `npm install` warns about peer deps, record the warning — do NOT use `--force`/`--legacy-peer-deps` unless a hard failure requires it (record the reason if so).

- [ ] **Step 3: Verify the CLI runs**

```bash
npx promptfoo --version
```

Expected: prints a version number (a real run with no API key still reports `--version`). No paid LLM call is made here.

- [ ] **Step 4: Add a convenience npm script (optional but recommended)**

Edit `package.json` `scripts` — add:

```json
"eval:llm": "promptfoo eval -c docs/ml/promptfoo-example/promptfooconfig.yaml"
```

This documents the manual invocation. It is **not** wired into any other script, hook, or CI step (ML1).

- [ ] **Step 5: Verify NO lifecycle hooks were added (ML6)**

Read the `hooks` block of `~/.claude/settings.json` AND project `.claude/settings.json`. Both must be **unchanged** vs the Task 1 Step 2 snapshot — npm installs do not touch Claude Code settings, but confirm. If anything changed → stop and re-audit.

- [ ] **Step 6: Lint + commit**

```bash
git add package.json package-lock.json
npx lefthook run pre-commit
```

Expected: all jobs green (`package.json`/`package-lock.json` are JSON — not linted by markdownlint/cspell; gitleaks scans them — promptfoo's tree has no secrets).

```bash
git commit -m "feat(a11): add promptfoo as devDependency for LLM prompt eval (ML1)"
```

---

## Task 3: Vendor the Data Scientist skill (standalone, vendored — ML3/ML7)

**Files:**

- Create: `.claude/skills/data-scientist/` (vendored skill tree)
- Modify: `cspell.json`, `.markdownlintignore`

- [ ] **Step 1: Clone the resolved source to a temp location**

Using the repo URL + in-repo skill path resolved in Task 1 Step 4:

```bash
git clone --depth 1 <RESOLVED_REPO_URL> /tmp/ds-skill-src
ls -R /tmp/ds-skill-src/<RESOLVED_SKILL_PATH>/
```

Expected: a `SKILL.md` + optional `references/` directory.

- [ ] **Step 2: Verify the skill ships no Claude Code hooks (ML6/ML7)**

```bash
grep -rIl "hooks" /tmp/ds-skill-src --include="*.json" || echo "no hooks json"
```

Expected: no `settings.json` with a `hooks` block. If the skill ships hooks → **stop**, re-audit (pick another candidate from Task 1 Step 4).

- [ ] **Step 3: Vendor the skill into the project**

```bash
mkdir -p ".claude/skills/data-scientist"
cp -r /tmp/ds-skill-src/<RESOLVED_SKILL_PATH>/. ".claude/skills/data-scientist/"
ls -R ".claude/skills/data-scientist/"
rm -rf /tmp/ds-skill-src
```

Expected: `.claude/skills/data-scientist/SKILL.md` (+ `references/**` if present). The `SKILL.md` frontmatter `name:` should be `data-scientist` (or a clear ML name) — if the upstream `name:` clashes with an existing skill, edit it to `data-scientist`. (Vendoring — not a submodule — keeps it on Windows+Cyrillic paths and immune to upstream loss, ML7.)

- [ ] **Step 4: Exclude the vendored skill from the doc-lint chain (ML3)**

Edit `.markdownlintignore` — append:

```
.claude/skills/data-scientist/
```

Edit `cspell.json` — add `.claude/skills/data-scientist/**` to the `ignorePaths` array. Do **not** ignore `.claude/skills/` wholesale — `q-item-add`, `rls-check`, `regression`, `mermaid` stay linted.

- [ ] **Step 5: Reload and verify the skill is discoverable**

```
/reload-plugins
```

Confirm a `data-scientist` skill is now listed among available skills (project `.claude/skills/` is auto-discovered, like the other project skills). Confirm neither `settings.json` `hooks` block changed (ML6).

- [ ] **Step 6: Verify lint exclusion works, then commit**

```bash
git add .claude/skills/data-scientist/ cspell.json .markdownlintignore
npx lefthook run pre-commit
```

Expected: cspell + markdownlint do NOT report errors from `.claude/skills/data-scientist/**`; all jobs green.

```bash
git commit -m "feat(a11): vendor Data Scientist skill into .claude/skills + lint-ignore (ML3)"
```

---

## Task 4: Bootstrap the A11 home — `docs/ml/` + seed promptfoo example + ADR-005

**Files:**

- Create: `docs/ml/README.md`, `docs/ml/promptfoo-example/promptfooconfig.yaml`, `docs/ml/promptfoo-example/README.md`
- Create (conditional — A6 landed): `docs/adr/ADR-005-ml-ai-tooling.md`
- Modify (conditional): `cspell-words.txt`

- [ ] **Step 1: Create the A11 home + the ML/AI convention**

```bash
mkdir -p "docs/ml/promptfoo-example"
```

Create `docs/ml/README.md`:

```markdown
# docs/ml — ML / AI playbook (map section A11)

Home of the `A11 «ML / AI-разработка»` section. Defines the tooling Лидерра uses
to build and test ML/AI capability. The portal currently ships no ML/AI code —
this section is the toolset, ready for when AI features are scoped.

## Toolset

| Tool | Role | Status |
|---|---|---|
| **claude-api skill** | Build AI features on the Anthropic SDK (lead qualification, call summaries, email drafts) with prompt caching. | reuse — already available |
| **context7 MCP** | Up-to-date docs for AI/ML libraries and SDKs. | reuse — already installed |
| **Sentry MCP** | Debug AI features in production via Sentry AI/LLM monitoring (read-only). | reuse — Tooling #34, pending the Sentry deployment (Б-1) |
| **promptfoo** | Test suite for LLM prompts/agents: assertions, regression, LLM-graded eval, red-team. | installed — `npx promptfoo` |
| **Data Scientist skill** | Classical-ML workflow: business objective → ML task, algorithm selection, feature engineering, evaluation. | installed — vendored skill |
| **Jupyter MCP** | Executable notebooks for real model training. | **deferred** — see below |

## Boundaries (which tool for which job)

- **Building an AI feature** (a prompt-backed endpoint) → the **claude-api skill**.
- **Testing / regression-checking an LLM prompt** → **promptfoo** (`docs/ml/promptfoo-example/`).
- **A classical-ML modelling question** (which algorithm, how to evaluate) → the
  **Data Scientist skill**.
- **Executing a notebook / training a model** → **Jupyter MCP** — *deferred*.
- promptfoo's **red-team** tests *prompts*; the D3 Trail of Bits / Semgrep tools do
  SAST of *code*. Different objects — not a duplication.

## promptfoo — running an eval

promptfoo makes **paid** Anthropic API calls. It runs **manually or in CI only** —
never in a git hook, never in pre-commit, never automatically.

- API key: `ANTHROPIC_API_KEY` env var (PowerShell User scope — the Sentry
  `SENTRY_AUTH_TOKEN` pattern). Never commit a key.
- Run the seed example: `npm run eval:llm` (or
  `npx promptfoo eval -c docs/ml/promptfoo-example/promptfooconfig.yaml`).

## Jupyter MCP — why deferred

Jupyter MCP executes notebooks; it needs a Python ML environment (pandas /
scikit-learn / Jupyter). The machine is native Windows, deliberately runtime-minimal
(no Docker), and there is no model to train yet. Jupyter MCP is a **reserved slot**:
registered in the Tooling registry as *pending*, installed by a separate severable
task when a concrete ML model is scoped. See the A11 plan's "Deferred Task".
```

- [ ] **Step 2: Create the seed promptfoo example config**

Create `docs/ml/promptfoo-example/promptfooconfig.yaml`:

```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
# Seed example — A11. Lead-qualification prompt eval.
# Run manually: npm run eval:llm   (needs ANTHROPIC_API_KEY — never in CI/hooks)
description: "Лидерра — lead-qualification prompt eval (example)"

prompts:
  - |
    Классифицируй обращение лида как HOT, WARM или COLD.
    Ответь РОВНО одним словом — HOT, WARM или COLD.

    Обращение: {{message}}

providers:
  - id: anthropic:messages:claude-haiku-4-5-20251001

tests:
  - vars:
      message: "Нужно срочно, бюджет согласован, готовы подписать договор сегодня."
    assert:
      - type: equals
        value: HOT
  - vars:
      message: "Интересно, расскажите подробнее про условия и сроки."
    assert:
      - type: equals
        value: WARM
  - vars:
      message: "Просто смотрю что есть на рынке, ничего конкретного."
    assert:
      - type: equals
        value: COLD
```

- [ ] **Step 3: Create the example README**

Create `docs/ml/promptfoo-example/README.md`:

```markdown
# promptfoo example — lead-qualification eval

A worked promptfoo eval: a HOT/WARM/COLD lead-classification prompt with three
assertion cases. Demonstrates the A11 prompt-testing workflow.

## Run

```bash
# from the repo root; ANTHROPIC_API_KEY must be set (PowerShell User scope)
npm run eval:llm
```

This makes **paid** Anthropic API calls. Run it manually or in CI only — never
in a git hook or pre-commit (A11 rule ML1). See `docs/ml/README.md`.

## Adapt

Copy `promptfooconfig.yaml` next to a real prompt when an AI feature is built.
Swap the model, add `tests`, use richer assertions (`contains`, `llm-rubric`,
cost/latency thresholds). Full reference: <https://promptfoo.dev/docs/>.

```

- [ ] **Step 4: Decide on the conditional ADR (uses Task 1 Step 6 result)**

If **A6 landed** (`docs/adr/ADR-000-adr-process.md` exists, adr-kit present) → do Step 5. If A6 has **not** landed → skip Step 5, and add a line to `docs/ml/README.md` noting "the A11 tooling decision will be recorded as an ADR once adr-kit (A6) lands". Record the branch taken.

- [ ] **Step 5: Write ADR-005 (conditional)**

Create `docs/adr/ADR-005-ml-ai-tooling.md`:
```markdown
# ADR-005: ML / AI tooling (A11)

- **Status:** Accepted
- **Date:** 2026-05-17
- **Deciders:** Дмитрий

## Context
The `A11 «ML / AI-разработка»` map section had zero tooling. Лидерра ships no
ML/AI code; `calc_lead_score` is a deterministic SQL function. A toolset is needed
for the day AI features (LLM-backed) or a scoring model are scoped.

## Decision
A11 adopts a six-position toolset in two subcategories:
- **LLM integration** — the claude-api skill (build), promptfoo (test prompts),
  Sentry MCP (observe). All reuse or light.
- **Classical ML** — a vendored Data Scientist skill (workflow knowledge). The
  executable part, **Jupyter MCP**, is **deferred**: it needs a Python ML runtime
  the deliberately-minimal native-Windows machine lacks, and there is no model to
  train. Jupyter MCP is a reserved registry slot, installed by a separate task
  when a concrete model is scoped.
- promptfoo runs manually / CI only — never in a hook (paid LLM calls).
- A11 tools are non-UI → the `ml-ai-tooling` off-phase category.

## Consequences
- Positive: A11 populated; AI features have a build+test+observe toolchain.
- Risk: the Data Scientist skill is third-party — mitigated by vendoring.
- Deferred: no Python runtime until a model is scoped — accepted, this is the
  decision.

## Enforcement
None — A11 tools are advisory; verified by use and code review.
```

- [ ] **Step 6: Lint + commit**

```bash
npx markdownlint-cli2 "docs/ml/**/*.md"
npx cspell --no-progress --no-summary --no-gitignore "docs/ml/**/*.md"
```

If A6 landed, also lint `docs/adr/ADR-005-*.md`. Add flagged valid terms (`promptfoo`, `promptfooconfig`, `scikit`, `XGBoost`, `Jupyter`, `Лидерра`, etc.) to `cspell-words.txt`. Then:

```bash
git add docs/ml/ cspell-words.txt
git add docs/adr/ADR-005-*.md   # only if Step 5 ran
git commit -m "feat(a11): bootstrap docs/ml — README + promptfoo example + ADR-005"
```

---

## Task 5: Smoke-test the A11 toolset

**Files:** none modified

- [ ] **Step 1: Smoke-test promptfoo config validity (no paid call — ML1)**

```bash
npx promptfoo validate -c docs/ml/promptfoo-example/promptfooconfig.yaml
```

Expected: the config parses as valid. (`validate` checks schema only — no LLM call, no cost.) If `validate` is not a subcommand in the installed promptfoo version, instead run `npx promptfoo eval -c docs/ml/promptfoo-example/promptfooconfig.yaml --no-cache` **only if** `ANTHROPIC_API_KEY` is set and the user approves the small cost; otherwise record "config syntax reviewed manually, eval not run (no key / cost)".

- [ ] **Step 2: Smoke-test the Data Scientist skill**

Invoke the `data-scientist` skill with a trivial ML question (e.g. "which algorithm and evaluation metric fit predicting lead conversion from CRM features?"). Expected: the skill loads, `SKILL.md` routes the intent, and it returns structured classical-ML guidance (algorithm choice, metrics, validation). Functional smoke — no file output required.

- [ ] **Step 3: Smoke-test the claude-api skill reuse**

Confirm the `claude-api` skill is invocable (it appears in the available-skills list). No deep test — A11 only documents it as reuse. Record availability.

- [ ] **Step 4: Confirm the hook chain is intact (ML6)**

Submit a trivial prompt; the economy marker still appears, the Stop verifier still runs, ruflo + CLAUDE.md-warn hooks fire. No plugin/skill leaked a `hooks` entry. No repo files changed in Task 5 → no commit.

---

## Task 6: Normative registry sync (ML / NUM1)

**Files:** Modify `docs/Tooling_v8_3.md`, `docs/Plugin_stack_rules_v1.md`, `docs/Pravila_raboty_Claude_v1_1.md`, `CLAUDE.md`, `docs/CHANGELOG_claude_md.md`

- [ ] **Step 1: Read the registry homes + the live counter (NUM1)**

Read for exact insertion points and the **current** counter: `docs/Tooling_v8_3.md` Прил. Н §0 + the last `§4.x` subsection; `docs/Plugin_stack_rules_v1.md` R10.1; `docs/Pravila_raboty_Claude_v1_1.md` §13.2. Using the Task 1 Step 5 result, decide the A11 numbers:

- `#N` promptfoo, `#N+1` Data Scientist skill, `#N+2` Jupyter MCP (registered **pending**) — sequential from `counter + 1`, after any A6/D3/C9 entries.
- If Task 1 Step 5 found claude-api / context7 **unregistered**, also assign them numbers (reuse positions). Record the full number assignment.

- [ ] **Step 2: Add the Tooling Прил. Н ml-ai-tooling subsection(s)**

Edit `docs/Tooling_v8_3.md`: add a subsection for each new number, category **ml-ai-tooling** (off-phase). Per tool:

- **promptfoo** — npm `promptfoo` MIT (OpenAI-owned), root `package.json` devDependency, `npx promptfoo`; runs manually/CI only — never in a hook (ML1); red-team boundary vs D3 ToB/Semgrep (ML2); no CC hooks.
- **Data Scientist skill** — vendored standalone skill in `.claude/skills/data-scientist/` (source repo from Task 1 Step 4), knowledge-only; bus-factor mitigated by vendoring (ML7).
- **Jupyter MCP** — `datalayer/jupyter-mcp-server`, **pending — NOT installed**; deferred severable task gated on a Python ML environment (ML9); the Sentry #34 "pending" precedent.
- claude-api / context7 (if unregistered) — reuse positions.

Add the ml-ai-tooling category as the **seventh** off-phase subcategory (after UI-pool, infrastructure, debug-runtime, architecture-tooling, audit-security — confirm the exact count against the live file). Bump §0 counter; bump the Прил. Н version header.

- [ ] **Step 3: Add PSR_v1 R10.1 rows**

Edit `docs/Plugin_stack_rules_v1.md`: add a row per new tool to R10.1, category **ml-ai-tooling** (off-phase) — explicitly *outside* the UI-pool → no R6.0/R6.1 stack-filter, no R14 pipeline (same treatment as `claude-md-management`, architecture-tooling, audit-security). Bump the PSR_v1 version header.

- [ ] **Step 4: Add the Pravila §13.2 note**

Edit `docs/Pravila_raboty_Claude_v1_1.md` §13.2: add a one-line **ml-ai-tooling** category note, alongside the existing infrastructure / debug-runtime / architecture-tooling / audit-security notes. Re-read Pravila §0/§13 first to keep section numbering consistent. Bump the Pravila version header.

- [ ] **Step 5: Update CLAUDE.md via the governed channel**

Invoke `/claude-md-management:claude-md-improver`. Apply: §3 title count bump, §1 priority-chain row 2b count bump, new §3.3 ml-ai-tooling row(s). The plugin also writes the `docs/CHANGELOG_claude_md.md` entry and bumps §0 cross-ref versions (Tooling / PSR_v1 / Pravila). **Do not** edit `CLAUDE.md` directly (§5 п.10).

- [ ] **Step 6: Lint + commit**

```bash
npx markdownlint-cli2 "docs/Tooling_v8_3.md" "docs/Plugin_stack_rules_v1.md" "docs/Pravila_raboty_Claude_v1_1.md" "docs/CHANGELOG_claude_md.md"
npx cspell --no-progress --no-summary --no-gitignore "docs/Tooling_v8_3.md" "docs/Plugin_stack_rules_v1.md" "docs/Pravila_raboty_Claude_v1_1.md"
git add docs/Tooling_v8_3.md docs/Plugin_stack_rules_v1.md docs/Pravila_raboty_Claude_v1_1.md CLAUDE.md docs/CHANGELOG_claude_md.md cspell-words.txt
git commit -m "docs(a11): register ml-ai-tooling category — promptfoo/Data Scientist skill/Jupyter MCP (NUM1)"
```

---

## Task 7: Reflect A11 on the map — close the section

**Files:** Modify `docs/automation-graph.html`

- [ ] **Step 1: Read the structures to replicate**

In `docs/automation-graph.html` read, as templates: a vendored-skill node (`mermaid_skill`), a plugin-skill node, and an MCP node (`mcp_sentry`) across `NODES`, `NODE_DETAILS`/`nd(...)`, `NODE_SECTION`, and the "Паспорт узла" date fields. Record the current node/edge counts from the header and the group-count comments.

- [ ] **Step 2: Add the 3 A11 nodes**

Add to `NODES`, replicating the template shapes:

- `claude_api` — label `claude-api\n(skill)`, skills group.
- `promptfoo` — label `promptfoo`, an appropriate tooling group (match how `bin/` CLIs / lefthook tools are grouped — e.g. the group used for gitleaks/lychee-class tools, or a plugins/skills group if no CLI group exists; record the choice).
- `data_scientist` — label `Data Scientist\n(skill)`, skills group.

Add matching `nd(...)` / `NODE_DETAILS` entries (Russian, per the file's convention), Паспорт `since: '2026-05-17'`:

- `claude_api` — "Скил сборки AI-фич на Anthropic SDK (prompt-кэш). Reuse — раздел A11 опирается также на context7 MCP (доки) и Sentry MCP (LLM-наблюдаемость)."
- `promptfoo` — "npm-CLI eval LLM-промптов: ассерты, регрессия, red-team. Запуск вручную/CI — не в хуках (платные вызовы)."
- `data_scientist` — "Vendored-скил: классический ML-воркфлоу — выбор алгоритма, feature engineering, оценка модели."

- [ ] **Step 3: Map the 3 nodes to section A11**

In `NODE_SECTION` add (a new comment block, after the A6 block):

```js
// A11 ml-ai-tooling 17.05.2026 — раздел «ML / AI-разработка» наполнен
claude_api: 'A11', promptfoo: 'A11', data_scientist: 'A11',
```

`A11 «ML / AI-разработка»` goes from 0 → 3 nodes — the section is no longer empty. (No `jupyter_mcp` node — Jupyter MCP is deferred, ML9.)

- [ ] **Step 4: Update header metrics + group-count comments**

Bump the node count in the map header/legend by 3. Bump the edge count if Step 2's node details add governing edges (match how the A6 nodes were wired — e.g. an edge to the governing normative node; replicate that pattern, else node-only). Update the `NODE_SECTION` group-count comments (skills, and whatever group `promptfoo` joined).

- [ ] **Step 5: Smoke-test the map**

```bash
npx stylelint docs/automation-graph.html
```

Open `docs/automation-graph.html` (Playwright MCP or a local `http.server` — quirk 90: `file://` rejected). Expected: 0 JS console errors; the 3 new nodes render; clicking section `A11` highlights all three.

- [ ] **Step 6: Commit**

```bash
git add docs/automation-graph.html
git commit -m "feat(map): A11 nodes — closes section «ML / AI-разработка»"
```

---

## Task 8: Final regression & branch finish

**Files:** none modified

- [ ] **Step 1: Rebase onto latest origin/main (sequencing)**

```bash
git fetch origin
git rebase origin/main
```

Expected: a clean rebase (A6/D3/C9 already landed; A11 touched the same shared files — the map, 4 normative docs, the Tooling counter — so resolve any conflict by **re-reading the live file and re-applying the A11 delta**, never blindly). If conflicts are non-trivial → stop, report. Re-run the live-counter read (Task 6 Step 1) if the Tooling counter moved.

- [ ] **Step 2: Full pre-commit chain**

```bash
npx lefthook run pre-commit
```

Expected: all jobs green — A11 adds **no** lefthook job (job count unchanged vs the Task 1 baseline).

- [ ] **Step 3: Confirm app code untouched — run the suites**

A11 changes no `app/` code → suites must match the Task 1 Step 3 baseline:

```bash
cd app && php artisan test --parallel
cd .. && npm run test:vue
```

Expected: Pest and Vitest counts unchanged vs the Task 1 baseline (0 regressions). Record exact counts; write out any failure with file:line.

- [ ] **Step 4: Confirm the economy/ruflo hook chain is intact**

Economy marker still appears; the Stop verifier still runs; no plugin/skill leaked a `hooks` entry into either `settings.json`. Compare to the Task 1 Step 2 snapshot.

- [ ] **Step 5: Pre-push checks**

```bash
./bin/gitleaks.exe detect --source . --no-banner --redact
./bin/lychee.exe --config .lychee.toml "docs/**/*.md" "*.md"
```

Expected: gitleaks 0 leaks (the promptfoo example holds no key — ML1); lychee 0 broken (new `docs/ml/**/*.md` + `docs/adr/ADR-005-*.md` are scanned — fix or `.lychee.toml`-exclude any link, e.g. the `promptfoo.dev` docs link).

- [ ] **Step 6: Finish the branch**

Invoke `superpowers:finishing-a-development-branch` — present the standard options. Do **not** push without an explicit user choice. Push pattern: `git push origin feat/a11-ml-ai-tooling:main`.

---

## Deferred Task (NOT in this plan — future, severable)

**Jupyter MCP install — gated on a trigger.** When a concrete ML model is scoped
(a real lead-scoring / lead-quality model with a dataset), run a separate task:

1. Decide the Python ML environment (native Windows venv vs a YC instance) — an
   explicit runtime weighing, like the Docker / pg_partman decisions.
2. `alpha-substrate-spike-first` — spike Jupyter MCP (it is experimental, Notebook
   6.x only) before integrating.
3. Install `datalayer/jupyter-mcp-server` as the 8th `.mcp.json` server, version-pinned.
4. Flip the Tooling registry entry from **pending** to **active**; add a `jupyter_mcp`
   node to `docs/automation-graph.html` → `NODE_SECTION` A11 (now 4 nodes).
5. Re-run the full regression + the conflict re-audit.

Until then, A11 is fully covered by the 5 installed/reuse positions.

---

## Self-Review

**1. Spec coverage (the six-position toolset, Approach А).** Reuse layer — claude-api / context7 / Sentry documented in `docs/ml/README.md` (Task 4 Step 1), registration audited (Task 1 Step 5) and synced (Task 6). promptfoo — installed (Task 2), seed example (Task 4 Steps 2-3), smoked (Task 5 Step 1). Data Scientist skill — repo resolved (Task 1 Step 4), vendored (Task 3), smoked (Task 5 Step 2). Jupyter MCP — registered pending (Task 6 Step 2), deferred task documented ("Deferred Task"). Section closure: normative (Task 6), map (Task 7), regression/finish (Task 8). Conflict audit: ML1→T2.4+T4.1/3, ML2→T4.1+T6.2, ML3→T3.4, ML4→T4.1+T7.2-3, ML5→T6.2-4, ML6→T2.5+T3.5, ML7→T3.3+T6.2, ML8→T1.5+T6.1-2, ML9→T6.2+T7.3, NUM1→T1.6+T6.1+T8.1. No gaps.

**2. Placeholder scan.** `#N`/`#N+1`/`#N+2` (Task 6), `<RESOLVED_REPO_URL>`/`<RESOLVED_SKILL_PATH>` (Task 3), and the promptfoo node's map group (Task 7 Step 2) are **runtime-resolved by design** — the live Tooling counter, the Data Scientist skill source repo, and the live map group shapes are not knowable before Task 1 Step 4-6 / reading the 2400-line map, and each carries concrete resolution criteria (the A6/C9/D3 pattern). All file contents shown in full — `docs/ml/README.md`, both promptfoo-example files, ADR-005, the seed `promptfooconfig.yaml`. No "TBD" / "handle edge cases".

**3. Consistency.** Branch `feat/a11-ml-ai-tooling` consistent T1↔T8. Node ids `claude_api` / `promptfoo` / `data_scientist` consistent T7 Steps 2-3 + the `docs/ml/README.md` table. Category name **ml-ai-tooling** consistent T6 Steps 2-4 + ADR-005. Paths consistent: `.claude/skills/data-scientist/`, `docs/ml/`, `docs/ml/promptfoo-example/`. promptfoo install mode (root `package.json` devDep) consistent T2↔file-structure↔Tooling-entry. Jupyter MCP flagged **deferred/pending** uniformly (Tool Identity, ML9, Task 6 Step 2, Task 7 Step 3, "Deferred Task"). No lefthook job added — consistent with the minimal-conflict goal (ML1).

---

## Execution Handoff

Plan complete and saved to `docs/superpowers/plans/2026-05-17-a11-ml-ai-tooling-integration.md`. Two execution options:

1. **Subagent-Driven** — fresh subagent per task, two-stage review. *Caveat:* Task 2 (`npx promptfoo`), Task 3 Step 5 + Task 5 Steps 2-3 (`/reload-plugins`, skill invocations) and Task 6 Step 5 (`/claude-md-management`) are main-session-bound — those steps stay with the controller.
2. **Inline Execution** — `superpowers:executing-plans`, batch with checkpoints. **Recommended here** — install/config/docs-heavy with many interactive main-session steps (the A6/C9/D3 pattern).

**Sequencing reminder:** A11 rebases onto `origin/main` after A6 / D3 / C9 land (Task 8 Step 1). One open item before execution: execution method — **1** (Subagent-Driven) or **2** (Inline, recommended here).
