Dependencies, corpus, pipeline throughput and the grounded-generation gate. The plan behind this page: AI_COACH_ACTION_PLAN.md. Auto-refreshes every 30s.
Loading…
Admin provider lever (My Second chat + mistake verbalizer, admin-only)
Not checked yet — live calls, so this isn't part of the 30s auto-refresh.
🗺 How it works
Where a request actually goes and what runs where — the structure behind the numbers above.
System architecture
Browsersecond.html / ai-coach-upload.html
→
Node APIaiCoachRoutes.ts — auth, rate limits, stores raw PGN
Node never talks to Python's dependencies directly — everything Stockfish/RAG/LLM related is internal to the Python service, reachable only on the Docker network.
Analysis lifecycle — rule-based floor, then LLM upgrade
The design choice that matters most here: the guaranteed, engine-grounded result renders and notifies the user first; the slower LLM prose is a best-effort pass afterward that can never block or replace it. FLOOR = always happens, seconds. UPGRADE = best-effort, can be slow or skipped under load — see the llm_errors/avg LLM time numbers above for how often that's currently happening.
POST /analyzefloor
Row inserted with status='processing'; coaching_id returned immediately — the client doesn't wait for anything below.
Stockfish scan
Depth-16 analysis of every ply. CPU-bound, native process — competes for the same cores as every other concurrent analysis and Ollama inference on the box.
Concept extraction
concepts.py derives typed, deterministic facts from the Stockfish PVs (mate threats, hanging pieces, king safety, passed pawns…) plus an allowed_sans whitelist — no ML involved, this is the grounding data everything downstream is checked against.
RAG retrieval
Hybrid dense + BM25 search of Milvus-Lite for opening/middlegame/endgame/mistake context, RRF-merged and reranked.
Rule-based coaching saved — status='ready'floor
The guaranteed result. Persisted, notification sent. The user can see this the moment it lands — everything below is optional polish on top of it.
LLM upgradeupgrade
verbalizer.py rephrases the top mistakes via Ollama (or an admin-only frontier fallback). Semaphore(1) — only one generation in flight service-wide, so it never fights Stockfish for more than one core's worth of contention at a time.
Grounding gateupgrade
Every move/capture token in the generated prose must appear in the allowed_sans whitelist from step 3. Fails → the rule-based text from step 5 stands, unchanged. This is the actual hallucination defence — see the diagram below.
Re-saved (or not)upgrade
Gate pass → re-persisted with the upgraded prose, cached forever (never regenerated for the same analysis). Gate reject or timeout → row stays exactly as step 5 left it. Either way the user already has an answer.
The grounding gate (CCC recipe)
Why the LLM can't invent tactics: it's only allowed to phrase facts that were already verified deterministically, and every move it mentions is checked against that verified list before anything reaches the user.
Stockfish PVs
→
concepts.pytyped facts + allowed_sans whitelist
→
LLM verbalizes"phrase ONLY these facts" — prompt-constrained
→
Gate checkevery move token ∈ allowed_sans?
→
✓ Pass → prose used
✗ Reject → floor stands
🔬 Research gap areas
What's not built yet, why, and what closing it actually takes — pulled from AI_COACH_ACTION_PLAN.md plus what the 2026-08-02 module-health audit surfaced.
Skill-aware coaching (Maia-2 → AI Coach) open
Gap: Maia-2 (human-move-prediction model) is deployed and live — but only for My Second's opponent prep and the older Coach module. AI Coach's own per-mistake pipeline (concepts.py) has no skill-rating awareness: two users at 900 and 2200 get the identical mistake explanation for the identical blunder.
Needs: feed maia2-service's P(move | rating) for the played move and the engine's best move into concepts.py's fact list — "at 1500, 62% of players find this move" is a fact like any other, and the gate/verbalizer machinery already handles arbitrary typed facts.
Corpus is reference material, not our own knowledge open
Gap: the RAG store's ~55k "game" chunks are generic movetext (book excerpts + assorted games), not annotated positions from this site's own master-games DB or its own generated analyses. Retrieval quality is capped by how generic that corpus is.
Needs: batch-generate concept-grounded annotations for critical positions in the 31.8k-game master DB using the existing Phase-1 pipeline (idle-time job, cached forever, self-grounded by construction), index those instead of/alongside raw movetext.
No persistent coaching loop open
Gap: every analysis is a one-off. There's no server-side aggregation of a user's recurring weaknesses across analyses, and nothing auto-assembles drills from a user's own mistakes.
Needs: aggregate motif/phase mistake rates into user_skill_profiles.recurring_weaknesses, incrementally, per analysis; auto-build drill sets from a user's own blunders (the puzzle SRS machinery already exists) plus DB puzzles matched by motif.
Eval harness is a small, manual spot-check open
Gap: the "Full eval" button above runs a fixed, small set of positions, on demand, read by a human. It catches obvious regressions but isn't a golden-set large enough to trust for a real prompt/model change, and nothing runs it automatically.
Needs: grow the fixture set materially (ChessQA-style), and run gate-unit-tests at minimum on every deploy that touches the coach/verbalizer/concepts code, failing the deploy on a regression rather than relying on someone remembering to click the button.
Custom fine-tuned model blocked — infra
Gap: today "specific to our needs" means prompt design + retrieval only — the model itself is stock qwen2.5:3b-instruct, unmodified.
Why it's blocked, not just undone: this box has no GPU and is already CPU-constrained by Stockfish + Ollama inference alone (see the resource breakdown on the Ops dashboard). Training is far more compute-intensive than inference; an always-on background job — even capped — would mean per-epoch training times measured in weeks, while directly competing with the analysis pipeline for the same cores that are already causing timeouts under load today.
Needs: not a continuous on-box job. The realistic path is periodic offline fine-tuning on rented GPU hours (a few hours, done monthly/quarterly, not this box) using training pairs distilled from a stronger model (Groq is free and already integrated) plus this system's own gate-accepted outputs as a quality filter. Not started — first step would be logging accepted generations in a form usable as training data, which nothing currently does.
ChessMoves AI Admin