ChessMoves AI

🗺 AI capability map — what we actually have

Every AI-powered surface on the site, in one place. This site is an AI-first product — the table below is the honest inventory behind that claim, not the pitch.

DECIDED AND SHIPPED 2026-08-03 (this page was not updated at the time; corrected 2026-09-26): the weakness-profile and session-history machinery moved into AI Coach. weakness.html now reads /api/ai-coach/weakness-profile, which computeAiCoachWeaknessProfile() builds from ai_coach_analyses and stores in user_skill_profiles. The legacy Coach (coach.html, coachRoutes.ts, game_analyses) is no longer linked from the nav — reachable by URL only, and its live counts in the table above show whether anything still writes to it. What remains is retiring it, apart from its /maia route. The paragraph below is the original framing, kept for the record.

The thing most worth your attention here: Coach and AI Coach are two independently-built per-game analysis systems, live at the same time, linked separately in the main nav, writing to two unrelated database tables (game_analyses vs ai_coach_analyses). That may be intentional (Coach = fast/free rule-based first look, AI Coach = deeper grounded-LLM analysis) — but nothing in the codebase states that as a decision, and a new user landing on either one has no way to know the other exists or why they'd pick one over the other. Worth an explicit call: merge the weakness-profile/drill machinery into AI Coach (it's the more capable pipeline and already has a "no persistent loop" gap — see below — that Coach's weakness_profiles table already half-solves), or clearly differentiate the two in the product itself.

Cross-cutting tasks & challenges (AI Coach-specific detail, diagrams and the grounding-gate design: module health →):

• Coach / AI Coach overlap — decided 2026-08-03: one product. The weakness profile moved into AI Coach; the legacy Coach is unlinked and waiting to be retired (see the note above).
• My Second chat and AI Coach have monthly counts per plan (since 2026-10-02) — 70 / 110 / 250 chat messages and 15 / 40 / 100 analyses a month on Standard / Premium / Coach, sized inside a monthly spend ceiling of an eighth of the plan's list price (admins $25, free accounts $0.05) that is checked before every AI call (AiSpendCaps.ts, entitlements.monthlyAiAllowance). Numbers and margins: model economics.
• The provider lever fails silently — a configured-but-broken frontier provider degrades every admin request to local Ollama with nothing surfacing it, until someone happens to check. The provider-status card on the AI Coach page makes this checkable, but nothing alerts proactively yet.
• Maia-2 IS now wired into AI Coach's pipeline (corrected 2026-08-25 — this entry previously said it was not) — rag_pipeline._enrich_with_maia() runs on every analysis and attaches a skill line to each mistake (how many players at your rating find the move). The explanation itself is not yet rating-aware (corrected 2026-09-26): Maia-2 runs after the LLM pass and never reaches concepts.py, so the prose is the same at 900 and 2200. Detail: module health →
• No custom/fine-tuned model — blocked on hardware (no GPU, box already CPU-constrained by inference alone), not effort. Realistic path is periodic offline fine-tuning on rented GPU hours, not a continuous on-box job. Detail: research gap areas →

💶 Model economics: Haiku for every user

Decided and deployed 2026-10-02. Prices are the decided ones, VAT included. Costs are measured on Haiku 4.5 ($1 / $5 per million tokens) from the ai_usage ledger: $0.0029 for a full AI Coach run (four calls) and $0.0026 for a My Second chat message. The dollar ledger is converted at the ECB rate of 2026-10-02 ($1 = €0.891). tests/haikuforall.test.ts re-derives the guarantees below from the code and the billing page, and fails the build if a cap, a price or the ceiling breaks them.

1 · What each plan leaves us. Spanish VAT (21%) is inside the price. Stripe Managed Payments takes 3.5% of the gross, and the card fee is 1.5% + €0.25 a charge. An annual plan is discounted, so it leaves less a month than a monthly one.

2 · What a plan costs used to the limit. The counts are what each plan includes; the ceiling is the hard stop on all AI spend for the account, counts included.

• Why an eighth, not half. Half the list price was the ceiling while only admins reached the paid model. Half of €2.99 is €1.50, more than the €1.27 a month an annual Standard plan leaves us, so a maxed account would have lost money.
• The counts sit inside the ceiling. A full month of AI Coach and chat uses 60–75% of each paid plan's ceiling. The rest is for explanations, the Ask box, lessons and game recaps, which have no count of their own and are bounded by the ceiling alone. Recaps are written automatically after every game against a person; bot games are skipped.
• If chat costs more than measured. The chat figure comes from 5 calls. Reading the prompt code gives $0.0053 a message instead. At that price a full Premium month would cost $0.70 and the ceiling would stop chat at about 95 messages instead of 110. The money guarantee holds either way; the count just would not be reachable. Re-check once the ledger has a few hundred chat rows.
• Prompt caching does not help yet. Haiku 4.5 only caches prompts of 4,096 tokens or more, and ours are 260–1,900.
• Club is the thinnest plan. €19.99 buys up to ten Premium seats. If the owner and all ten members hit their ceilings, model spend is 51% of what the club leaves us: still profitable, but the one to watch once clubs sell.
• My Second needs a paid plan (decided 2026-10-02). A free account gets one opponent scouting a month and no chat, so the free tier's AI cost is its one AI Coach run plus explanations and recaps. Credit packs are not sold: a one-off purchase is not a plan.
• Free accounts cost $0.05 a month at most. That covers the one AI Coach run (about $0.003) and a few dozen explanations or recaps. Twenty-five free accounts at the floor cost about what one annual Standard subscriber leaves us, so it holds at a 4% conversion rate even if every free account maxed out, and most use none of it.
• The fallback costs nothing. When Anthropic fails (an outage, an empty balance, a revoked key) the same request is answered by the local model, nothing is spent, and the API logs anthropic failed, answering from the local model.

💳 How Claude is paid for, and what happens at the limits

Who pays: we do, centrally. Every user’s Claude call goes out with one API key (ANTHROPIC_API_KEY on the server) and is charged to our Anthropic organisation’s prepaid credit balance at Haiku 4.5’s rate ($1 in, $5 out per million tokens). Users never deal with Anthropic: they pay us a plan, and the plan’s monthly counts and spend ceiling (the economics card above) keep what each account can cost under an eighth of what it pays. Credits are bought in the Claude Console (Settings → Billing → Buy credits), are non-refundable and expire a year after purchase, so small top-ups with auto-reload beat one large block.

Is central billing the right model? Yes, for a consumer app. The alternatives are worse: asking players to bring their own Anthropic key (no chess player has one), metering each analysis (meters put people off, and VAT and card fees on cent-sized charges eat them), or sending each user to buy credits from Anthropic (no such product). What central billing does need is the safeguards below, because one balance serves everyone: when it stops, it stops for everyone at once.

The limits outside our code (Anthropic’s, from its rate-limits page, checked 2026-10-03):

Found and fixed 2026-10-03. The gateway retried every 429 three times (about 1 + 2 + 4 seconds). The spend-cap 429 never clears on retry, so once a cap was reached every AI request in the app would have waited 7–9 seconds before the local model answered, for the rest of the month. Now (src/services/AiBudget.ts):

• No retry on a refusal that will not change. A 429 without retry-after goes straight to the local model.
• A circuit breaker. When Anthropic refuses for a standing reason (the spend cap, our own limit, an empty balance, a bad key), every request skips Claude and the local model answers at once. Claude is tried again every 15 minutes, so a top-up or a raised limit takes effect without a restart. Three outage failures in a row pause Claude for a minute, so users stop waiting through retries one by one.
• An email when it happens. Claude switching off for everyone, and coming back, emails ALERT_EMAIL (already set on the server). An outage is emailed once it has lasted ten minutes.
• A site-wide monthly budget. AI_MONTHLY_BUDGET_USD (default $50; set it to the Console’s spend limit). At 50% you get an email; at 80% (AI_FREE_LOCAL_AT) free accounts move to the local model until the 1st, with another email; at 100% a third. Paying accounts are never moved by it: each is capped at an eighth of what it pays, so their use is paid for by their plan. Anthropic’s limit stays the only hard stop, and the emails give you time to raise it.
• Grafana shows it. Calls by model and outcome, this month’s spend against the budget, and whether Claude is off: the AI row of the Overview dashboard.

What a user sees at a cut-off: never an error from a central limit. The local model answers instead, and the AI Coach also has its rule-based floor. The only refusal a user ever sees is their own plan’s allowance, with the number and the day it resets.

Owner checklist in the Claude Console:

1. Auto-reload on (Settings → Billing): reload when the balance falls below about a week’s use, up to about a month’s. This month’s spend is shown live above.
2. A spend limit, with AI_MONTHLY_BUDGET_USD in the server’s .env set to the same figure.
3. The usage tier (Settings → Limits). A new organisation can start in the Evaluation tier, below Start; confirm before launch.
4. A separate workspace and key for a dev server, with its own small spend limit (workspaces take their own limits), so testing can never spend production’s budget.

🏗 How a request reaches the model (since 2026-10-02)

The path as deployed. Green is the default for every user: Claude Haiku, reached from both services. Dashed blue is the local fallback and the ledger read; dashed purple is the one piece still missing.

Solid = deployed · green = Claude Haiku, every user's default since 2026-10-02, and the admin pin (built 2026-09-09) · dashed blue = the local fallback and the ledger read · dashed purple = still missing. The gate sits AFTER the provider on purpose: it is the one guarantee that does not depend on which model answered, which is exactly what makes it the fair yardstick for comparing them.

Where things stand today

This whole analysis is a forward projection, not a report on live economics — read every dollar figure below as "if you launch roughly as currently planned," not "what we're making." Two things make that explicit:

• Billing is live, customers are not yet. Stripe has taken payments since 2026-10-08, and the one real payment so far was refunded. The pricing below (€2.99 Standard / €4.99 Premium a month, VAT included) is decided, not proven demand.
• The accounts are not customers — the live tiles above show how many hold a paid tier with no Stripe customer behind it (seeded during development, not purchased). Real conversion rate, real churn, and real per-user chat intensity are all unknown — the "light" and "heavy" usage bands below are informed guesses, flagged as assumptions, not measurements.

What is real: AI Coach is the only surface with sustained use, from dogfooding rather than organic demand — see the live figure above. My Second has almost none yet, because today's fixes are what make it usable enough to actually put in front of people. So the AI Coach numbers below are closer to "current," and the My Second numbers are closer to "if this launches and gets adopted."

UPDATE — 2026-09-13. Three things changed, and one of them means every figure on this page is still a projection.

The trial had measured nothing, and now can. The Anthropic key was invalid — a 401 on every call, for weeks (see the correction above). It was replaced and verified HTTP 200 from inside chessflow-ai-coach on 2026-09-13. But the ledger still holds zero anthropic rows, because a failed call writes none and no successful call has yet flowed through the service. Until an anthropic row appears on 🤖 AI cost, nothing below has been validated against real spend.

RESOLVED 2026-09-15 (commit 9f5187c, “one model switch for both services”): claude_client.py now reads ANTHROPIC_MODEL like the Node gateway, and docker-compose.prod.yml passes it to both containers with the same default, claude-haiku-4-5. The paragraph below is kept as the record of what was wrong. Two different models are billing into one ceiling — unresolved. The Node gateway (My Second briefs, post-mortems) uses ANTHROPIC_MODEL, currently claude-opus-5. The Python coach service has its OWN default, claude_client.DEFAULT_MODEL = claude-haiku-4-5, and does not read that variable at all. Both spend against the same AI_SPEND_ADMIN_CAP_USD ($25/month), and Opus lists at roughly five times Haiku. So "cost per call" on the AI cost page is an average over two different models, and quality-per-dollar cannot be attributed to either. Align them before drawing any verdict from this trial — whichever way, since which one you standardise on materially changes what $25 buys.

The missing denominator now exists. Migration 088 adds ai_chat_turns, which records every chat turn with its provenance: which provider and model answered, input and output tokens, how many knowledge-base snippets reached the prompt, and whether a model was called at all — the Q1-Q6 fast path answers from pre-computed statistics and never reaches one, so its turns must not be counted as model output. ai_usage says what was spent; ai_chat_turns says what it bought. Per-interaction economics on this page have been guesses because only the first half was recorded.

Guardrails are now a registry, not prompt text. The same migration adds ai_guardrails with eight seeded rules — every one in proposed state, including the rules already live in the system prompt today, so the existing policy is inspectable rather than implicit. Nothing is enforced until it is approved, and the chat path reads the table at request time, so denying a rule genuinely disables it. That deliberately includes chat_training_use: storing chats for admin review is one decision, using them as training data is another, and it is left undecided rather than acquired as a side effect of building the log.

Model pricing landscape (checked 2026-09-26)

Per 1M tokens, list price, from each vendor's own pricing page. The self-hosted row has $0 marginal cost but isn't really free — see the caveats underneath.

What "self-hosted is free" actually hides: qwen2.5:3b's marginal cost per call is $0, but its fixed cost is the ~€30/mo box, shared with Stockfish and everything else. The real cost of the current setup isn't dollars — it's a throughput ceiling (single-threaded generation, ~18 tok/s) and a weak instruction-following track record (documented in today's bug fixes). Frontier pricing above is the dollar cost of removing both problems.

What each feature actually costs per interaction

Token estimates from reading the actual prompt-building code, not guesses.

Every number here is a fraction of a cent. Per-call cost was never going to be the problem — volume is, specifically uncapped chat volume. That's the whole ballgame below.

Cost at scale

At Haiku pricing the model bill is 1–2% of what we keep in a typical month and 4–7% in a heavy one, at every milestone. The last row of each group is not a forecast but the bound: it is what the limits allow if every account, free and paid, hit its ceiling in the same month, and even that leaves the business in profit. At Sonnet 5's $2 / $10 every cost figure doubles, which is why Sonnet-class models are worth reserving for a specific high-value call rather than every chat turn.

For context: at today's actual AI Coach volume (see the live figure above — tens of analyses a month), the whole feature costs well under $1 a month on Haiku. There is no scenario where AI Coach's model choice is a financial decision at today's scale; it is a quality decision.

Per-plan monthly caps

Since 2026-10-02. Per-plan monthly counts shipped with the move to Haiku for every user: 70 / 110 / 250 chat messages and 15 / 40 / 100 AI Coach analyses on Standard / Premium / Coach, sized from measured costs rather than the estimates below (see model economics). The spend ceiling that had closed the liability since 2026-09-09 was lowered from half the list price to an eighth. The proposal below is kept as the record of the reasoning.

My Second chat is gated by tier, not by message count — once a user has any entitlement, messages are unlimited, rate-limited only to 10 requests/minute (aiRateLimit in shared.ts). That's a reasonable design when marginal cost is $0. It becomes a real liability the moment each message has a per-token price:

Proposed caps (pick one model tier, then set the cap so worst-case COGS per user stays well under that tier's revenue):

400 msg/mo on Premium is generous — over 13/day, far past what a real prep session needs — while keeping worst-case COGS at Haiku pricing under half of that tier's net revenue ($2.12 of about $4.63 at €4.99) even if every single Premium user maxes it out every month (which none will in practice) — and the spend ceiling stops it at $2.495 regardless. This is a five-minute change: add a monthly counter next to the existing per-minute rate limiter, same pattern as consumeDaily in entitlements.ts.

Alternative: self-host a bigger model instead

The infra docs already priced this out (PRODUCTION.md): a dedicated GPU-attached box (CCX43 + GPU, for Ollama) runs ~€250/mo (~$270) — fixed, regardless of volume. A natural upgrade target is Qwen3-14B (same open family as today's 3B model, ~5x the parameters), which needs that GPU to run at usable latency — the current CPU box already struggles at 3B (~18 tok/s, and the codebase notes qwen3:4b was tried and rejected for flooding the CPU with unsuppressable reasoning tokens).

$270 ÷ $0.0053/msg ≈ 51,000 My Second chat messages/month to break even against metered Haiku. Cross- referencing the scale table above: only the 10,000-user heavy-usage scenario (78,750 msgs/mo) clears that bar. Every lighter scenario, and every milestone below 10,000 users, is cheaper on metered API.

Recommendation on this path: don't provision the GPU box speculatively. Ship on a metered frontier model now (zero fixed cost, scales down to zero if adoption is slow), and revisit self-hosting only once real telemetry shows sustained chat volume approaching ~50k messages/month — at which point the GPU box also stops being "risk," and starts being an obviously-correct cost optimization on proven demand. This also sidesteps the current architecture's other real problem: the local LLM shares 4 vCPUs with Stockfish analysis, so heavier chat use today would slow down game analysis for everyone, not just cost money.

🔀 Admin requests, and what a free account gets

Users have exactly one provider, Claude Haiku. Admin requests can still be routed another way: with AI_ADMIN_PROVIDER unset they go through the ai-coach service’s chain in a fixed order — Groq, Gemini, Claude — which returns on the first success. Claude is third, so on that chain it is reached only when the other two fail; AI_ADMIN_PROVIDER=claude pins it. Every call is costed in ai_usage, priced at Haiku 4.5’s $1/M input and $5/M output. A free account’s monthly AI Coach allowance (BETA_FREE_MONTHLY_CAP) is 1: one AI Coach run a month.

The trial that led to Haiku for every user ran from 2026-09-05 to 2026-10-02; its write-up is in the git history of this page.

Verdict

Sources & assumptions

• Production DB, queried 2026-08-01: 34 users, 0 stripe_customer_id rows, 1 second_analysis_runs row. The analyses count in this line was wrong: it recorded 2,037 ai_coach_analyses rows as "in the last 30 days" when that was the ALL-TIME total. Usage figures on this page are now read live from /api/admin/ai-strategy/stats rather than transcribed, which is the only durable fix — see tests/adminfreshness.test.ts, which fails the build if a hand-typed analyses-per-30d figure reappears here.
• Pricing: €2.99 Standard / €4.99 Premium / €9.99 Coach a month, VAT included — decided 2026-10-02 and the same in billing.html, terms.html, pricing-data.js, AiSpendCaps.ts and the feature matrix (until then this page used the Terms' $4.99 / $9.99). Revenue is taken net of Spain's 21% VAT; model costs converted at $1 = €0.891 (ECB, 2026-10-02). Credit packs (€4.99/500, €14.99/2000, €7.99/1500+bundle) from MY_SECOND_README.md. The post-beta review limits (0/2/5 a month, REVIEW_LIMIT_* in aiCoachRoutes.ts) only apply if BETA_FREE_MODULES is turned off; the monthly counts above apply now.
• Model economics (2026-10-02): kept revenue = price ÷ 1.21 − 3.5% Managed Payments − (1.5% + €0.25) card fee; caps from AI_COACH_MONTHLY_CAPS / AI_CHAT_MONTHLY_CAPS in config.ts; ceiling from SPEND_FRACTION in AiSpendCaps.ts; unit costs from the ai_usage ledger (24 AI Coach calls, 5 chat calls). All of it re-derived by tests/haikuforall.test.ts.
• Infra: current single box and fleet-scaling costs from SCALING.md / PRODUCTION.md.
• Token estimates: read directly from the actual prompt-building code in ai-coach-service/src/api/routes.py (chess_query) and ai-coach-service/src/services/verbalizer.py.
• Frontier pricing re-checked 2026-09-26 against each vendor's own page: platform.claude.com/docs/en/about-claude/pricing, ai.google.dev/gemini-api/docs/pricing and developers.openai.com/api/docs/pricing. (The August version used third-party trackers for OpenAI and Gemini, and one Gemini row was wrong.) Re-verify before committing budget.
• Conversion/usage assumptions (15% paid conversion, 65/35 tier split, 35% My Second monthly-active among paid, light/heavy chat bands, 60%-of-cap AI Coach usage) are estimates with no supporting data — clearly flagged inline above every table that uses them. Replace with real cohort data as soon as it exists.