 Дмитрий
|
808461295a
|
feat(router): Sonnet classifier + памятка + regex-fallback module (phase 2 task 10)
Phase 2 Task 10 of LLM-first router overhaul. Spec §4.2 — Layer 2 Sonnet 4.6
classifier with 4-pattern памятка enrichment, JSON output per spec, fallback
chain Sonnet → regex → degraded. Phase 1 regex Layer 1 extracted to its own
module so it can be called only as a fallback.
- tools/router-classifier-regex-fallback.mjs (NEW): self-contained regex
fallback. Extracts TASK_TYPE_KEYWORDS, HARD_KEYWORD_STEMS, detectTaskType,
keywordMatches, detectRecommendedNode, computeConfidence, classifyByRegex
verbatim from the prior classifier. Self-contained (own MICRO_KEYWORDS,
detectMicro, lower) — no circular imports.
- tools/router-classifier.mjs (REWRITE):
+ import { CLASSIFIER_MODEL } from router-config.mjs
+ re-export { classifyByRegex } from regex-fallback (back-compat surface)
+ buildClassifierPrompt(prompt, registry, { enrichment=true }) — spec §4.2
format with 4-pattern памятка (brainstorming / discovery-interview /
writing-plans / systematic-debugging) togglable via enrichment flag.
+ parseClassifierResponse(text) — strict task_type required, ```json fence
aware, accepts null recommended_chain_id.
+ classify() rewritten: prefilter → cache → Sonnet (CLASSIFIER_MODEL) →
regex fallback (transport error OR no key/unparseable).
+ callAnthropicAPI default model = CLASSIFIER_MODEL; max_tokens 300 → 1500
(full classifier output with alternatives & памятка needs the budget).
- removed: shouldEscalate, TASK_TYPE_KEYWORDS, detectTaskType,
keywordMatches, detectRecommendedNode, HARD_KEYWORD_STEMS, computeConfidence
(all live in regex-fallback now).
Kept legacy: buildLLMPrompt / parseLLMResponse (back-compat surface).
- tools/router-accuracy-runner.mjs: import classifyByRegex from regex-fallback
module (G11 from plan). Runner functionality unchanged.
- tools/router-classifier.test.mjs: +8 tests for buildClassifierPrompt (4) and
parseClassifierResponse (4); removed obsolete shouldEscalate block (3);
rewrote classify integration block (4 tests) to reflect new flow
(prefilter-first, LLM-always-on-fallthrough, regex on error).
Tests: tools/router-classifier.test.mjs 44/44 PASS. Full tools/ suite:
557 tests passed, 0 failed (4 pre-existing empty test files report
"no test suite found" — unrelated: ruflo-recall-hook, subagent-prompt-prefix,
plus 2 others — not touched in this commit).
accuracy-runner smoke: type=85%/node=55%/micro=100% on the 20-prompt set,
unchanged from pre-Task-10 baseline (regex path semantics preserved).
|
2026-05-25 14:28:25 +03:00 |
|
|
Дмитрий
|
7ed72a09f7
|
feat(router): 20-prompt accuracy runner — Phase A baseline (stage 3 task 5)
Ground truth: tools/router-test-prompts.json (20 промптов).
Runner: tools/router-accuracy-runner.mjs.
Baseline accuracy regex-only (Layer 1, без ANTHROPIC_API_KEY):
type=55.0%, node=55.0%, micro=95.0%.
Overall score: (11+11+19)/(60) = 68.3% — ниже порога 75%.
Систематические разрывы (наблюдения, не фиксы):
1. «опечатка/поправь» → bugfix ожидается, regex не ловит «поправь»
2. «составь email-рассылку» → marketing ожидается, regex не ловит «составь»
3. «проверь ... перед выходом» → #73 go-live ожидается, но #68 ZAP
перебивает (оба security-узлы, ZAP имеет «проникновение» ≠ «выход»
— weight tie-breaker в пользу первого найденного узла)
4. domain-узлы (#62, #71, #72) матчат правильно, но taskType не детектится
(«проверь ПДн» → type=unknown, node=#71 верно)
5. «запусти Pest тесты» → type=unknown (нет «баг/fix» в промпте)
6. «удали мёртвый код» → node=#3 (GitHub MCP матчит «issues» в тексте?)
NB: Layer 2 (Sonnet) подняла бы node-accuracy на спорных доменных
промптах — отложена до получения ключа (вариант 2).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
2026-05-24 10:40:20 +03:00 |
|