Dependencies, corpus, pipeline throughput and the grounded-generation gate. The plan behind this page: AI_COACH_ACTION_PLAN.md. Auto-refreshes every 30s.
Loading…
Admin provider lever (My Second chat + mistake verbalizer, admin-only)
Not checked yet — live calls, so this isn't part of the 30s auto-refresh.
🗺 How it works
Where a request actually goes and what runs where — the structure behind the numbers above.
System architecture
Browsersecond.html / ai-coach-upload.html
→
Node APIaiCoachRoutes.ts — auth, rate limits, stores raw PGN
→
Python ai-coach-serviceFastAPI, own container
→
Stockfish GameAnalyzer — depth-13 per ply
Embedding service text + FEN vectors
Milvus-Lite RAG store, embedded
model_router Claude Haiku for every user, Ollama (qwen2.5:3b) when Anthropic fails; admins: Groq → Gemini → Claude
Node never talks to Python's dependencies directly — everything Stockfish/RAG/LLM related is internal to the Python service, reachable only on the Docker network.
Analysis lifecycle — rule-based floor, then LLM upgrade
The design choice that matters most here: the guaranteed, engine-grounded result renders and notifies the user first; the slower LLM prose is a best-effort pass afterward that can never block or replace it. FLOOR = always happens, seconds. UPGRADE = best-effort, can be slow or skipped under load — see the llm_errors/avg LLM time numbers above for how often that's currently happening.
POST /analyzefloor
Row inserted with status='processing'; coaching_id returned immediately — the client doesn't wait for anything below.
Stockfish scan
Depth-13 analysis of every ply (16 was about twice as slow on this CPU, ~25s against ~12s for a 40-move game, for no gain in mistake detection at the 0.5-pawn threshold). CPU-bound, native process — competes for the same cores as every other concurrent analysis and Ollama inference on the box.
Concept extraction
concepts.py derives typed, deterministic facts from the Stockfish PVs (mate threats, hanging pieces, king safety, passed pawns…) plus an allowed_sans whitelist — no ML involved, this is the grounding data everything downstream is checked against.
RAG retrieval
Hybrid dense + BM25 search of Milvus-Lite for opening/middlegame/endgame/mistake context, RRF-merged and reranked.
Rule-based coaching saved — status='ready'floor
The guaranteed result. Persisted, notification sent. The user can see this the moment it lands — everything below is optional polish on top of it.
LLM upgradeupgrade
verbalizer.py rephrases the top mistakes with Claude Haiku (the local Ollama model when Anthropic fails). Semaphore(1) — only one generation in flight service-wide, so it never fights Stockfish for more than one core's worth of contention at a time.
Grounding gateupgrade
Every move/capture token in the generated prose must appear in the allowed_sans whitelist from step 3. Fails → the rule-based text from step 5 stands, unchanged. This is the actual hallucination defence — see the diagram below.
Re-saved (or not)upgrade
Gate pass → re-persisted with the upgraded prose, cached forever (never regenerated for the same analysis). Gate reject or timeout → row stays exactly as step 5 left it. Either way the user already has an answer.
The grounding gate (CCC recipe)
Why the LLM can't invent tactics: it's only allowed to phrase facts that were already verified deterministically, and every move it mentions is checked against that verified list before anything reaches the user.
Stockfish PVs
→
concepts.pytyped facts + allowed_sans whitelist
→
LLM verbalizes"phrase ONLY these facts" — prompt-constrained
→
Gate checkevery move token ∈ allowed_sans?
→
✓ Pass → prose used
✗ Reject → floor stands
🔬 Research gap areas
What's not built yet, why, and what closing it actually takes — pulled from AI_COACH_ACTION_PLAN.md plus what the 2026-08-02 module-health audit surfaced.
Skill-aware coaching (Maia-2 → AI Coach) half done
Done: Maia-2 is wired into AI Coach's own pipeline (rag_pipeline._enrich_with_maia). Each key mistake gets a skill_context — the share of players at the user's rating (clamped to 1100–2000, Maia-2's range) who play the move that was played and the engine's best move — and a blunder_risk score, from one Maia-2 call per mistake. The result page shows the skill line under each mistake. Background-only: a slow or dead maia2-service drops the line, never the analysis.
Gap: it runs AFTER the LLM upgrade and never reaches concepts.py, so the explanation itself is still rating-blind: two users at 900 and 2200 get the identical prose for the identical blunder, with a different percentage printed beneath it.
Needs: move the Maia-2 call ahead of the verbalizer and add skill_context to concepts.py's fact list — "at 1500, 62% of players find this move" is a fact like any other, and the gate/verbalizer machinery already handles arbitrary typed facts.
Corpus is reference material, not our own knowledge open
Gap: the RAG store's ~55k "game" chunks are generic movetext (book excerpts + assorted games), not annotated positions from this site's own master-games DB or its own generated analyses. Retrieval quality is capped by how generic that corpus is.
Needs: batch-generate concept-grounded annotations for critical positions in the master-games DB (90,660 games when counted on 2026-09-07; it was 31.8k when this card was written) using the existing Phase-1 pipeline (idle-time job, cached forever, self-grounded by construction), index those instead of/alongside raw movetext.
Persistent coaching loop mostly done
Done: a user's recurring weaknesses are aggregated server-side across all their analyses (computeAiCoachWeaknessProfile(), stored in user_skill_profiles, shown on weakness.html, since 2026-08-03), and WeaknessPackService builds a puzzle pack from that profile (POST /api/ai-coach/weakness-pack), refreshed as new analyses arrive and skipping themes seen too rarely to call a weakness.
Gap: the pack draws only on DB puzzles matched by theme. The user's own blunder positions are never turned into drills.
Needs: feed the positions from key_mistakes into the puzzle SRS machinery alongside the theme-matched DB puzzles.
Eval harness is a small, manual spot-check open
Gap: the "Full eval" button above runs a fixed, small set of positions, on demand, read by a human. It catches obvious regressions but isn't a golden-set large enough to trust for a real prompt/model change, and nothing runs it automatically.
Needs: grow the fixture set materially (ChessQA-style), and run gate-unit-tests at minimum on every deploy that touches the coach/verbalizer/concepts code, failing the deploy on a regression rather than relying on someone remembering to click the button.
Custom fine-tuned model blocked — infra
Gap: today "specific to our needs" means prompt design + retrieval only — the models themselves are stock: Claude Haiku, with qwen2.5:3b-instruct as the fallback, unmodified.
Why it's blocked, not just undone: this box has no GPU and is already CPU-constrained by Stockfish + Ollama inference alone (see the resource breakdown on the Ops dashboard). Training is far more compute-intensive than inference; an always-on background job — even capped — would mean per-epoch training times measured in weeks, while directly competing with the analysis pipeline for the same cores that are already causing timeouts under load today.
Needs: not a continuous on-box job. The realistic path is periodic offline fine-tuning on rented GPU hours (a few hours, done monthly/quarterly, not this box) using training pairs distilled from a stronger model (Groq is free and already integrated) plus this system's own gate-accepted outputs as a quality filter. Not started — first step would be logging accepted generations in a form usable as training data, which nothing currently does.
ChessMoves AI Admin