What we actually have across every AI-powered surface, and the cost-benefit case for keeping self-hosted qwen2.5:3b (or a bigger self-hosted model) vs. switching to a metered frontier model. Usage figures on this page are read live from production — they are measurements, not prose, and are fetched on every load. The pricing table and the argument built on it are still written by hand from published API rates; re-check those before acting if the review stamp below is old. See sources & assumptions at the bottom.
Every AI-powered surface on the site, in one place. This site is an AI-first product — the table below is the honest inventory behind that claim, not the pitch.
| Feature | What it does | Backend | Status | Model(s) |
|---|---|---|---|---|
| My Second | Opponent scouting: PGN ingestion, RAG-grounded prep briefs, opening/weakness Q&A chat | Python ai-coach-service | low usage … | qwen2.5:3b (floor) → Groq/Gemini/Claude (admin-only lever) |
| AI Coach | Per-game Stockfish analysis + grounded mistake explanations (CCC recipe: concept extraction → LLM verbalizer → hallucination gate) | Python ai-coach-service (same service as My Second — they share the LLM, RAG store, and Stockfish pool) | live … | Same as above |
| Coach (legacy) | Per-game analysis, a separate persistent weakness profile (weakness_profiles table), targeted drills | Node-native — its own game_analyses table, no Python involved | near-dead … | None — fully rule-based/statistical, no LLM anywhere in this path |
| Maia-2 | Human-move prediction, P(move | rating) — "62% of players at your level find this move" | maia2-service container | live — My Second prep, Coach's /maia, and AI Coach's own pipeline (_enrich_with_maia) | Own CNN, not an LLM |
| Model router | Admin-only frontier-model fallback for chat + verbalizer: tries each configured provider, falls through on real failure not just absence | ai-coach-service/model_router.py | fragile — Groq WAF-blocked and Claude out-of-credit both found live this session (Groq fixed; Claude needs billing) | Groq Llama 3.3 70B → Gemini → Claude Haiku 4.5 |
The thing most worth your attention here: Coach and AI Coach are two independently-built per-game analysis systems, live at the same time, linked separately in the main nav, writing to two unrelated database tables (game_analyses vs ai_coach_analyses). That may be intentional (Coach = fast/free rule-based first look, AI Coach = deeper grounded-LLM analysis) — but nothing in the codebase states that as a decision, and a new user landing on either one has no way to know the other exists or why they'd pick one over the other. Worth an explicit call: merge the weakness-profile/drill machinery into AI Coach (it's the more capable pipeline and already has a "no persistent loop" gap — see below — that Coach's weakness_profiles table already half-solves), or clearly differentiate the two in the product itself.
Cross-cutting tasks & challenges (AI Coach-specific detail, diagrams and the grounding-gate design: module health →):
ChessMoves AIrag_pipeline._enrich_with_maia() runs on every analysis, so per-mistake explanations are skill-rating aware. Detail: module health →0 free-tier quota until billing is enabled, confirmed directly against Google's API with two separately-issued keys. Groq was added as a third provider — genuinely free at every account region, no billing wall, hosting Llama 3.3 70B — and model_router.py now tries each configured provider in priority order (Groq → Gemini → Claude) and falls through to the next on any failure, not just when a key is blank, since a configured-but-broken provider (exactly what happened here) would otherwise silently block every request forever. The pricing/volume analysis below is unaffected — it's about $/token economics at scale, not which specific provider unblocks testing today.Switch the LLM calls to Claude Haiku 4.5 (or Gemini 3 Flash as a cheaper alternative) — at every realistic scale up to 10,000 users, the LLM bill stays under ~5% of subscription revenue even in a heavy-usage scenario, and under ~1% in a typical one. The 3B model wasn't the actual bug this session — bad, inconsistent input data was (see the two fixes shipped today). A bigger brain wouldn't have fixed the indexing mismatch; grounding the data did. But a frontier model would follow the anti-hallucination system prompt more reliably and stop needing a rule-based safety net to catch its mistakes.
One prerequisite before flipping the switch: My Second chat currently has no monthly message cap — only a 10-req/min rate limiter. That's free at $0 marginal cost today; it is not free once every message has a per-token price. A single very active "unlimited" Premium user could plausibly cost more in API fees than their $9.99/mo subscription. Add a per-tier monthly chat cap first (numbers below).
Don't build a GPU box for a bigger local model yet. The breakeven vs. metered Haiku is roughly ~50,000 chat messages/month sustained — around 10,000 users all chatting heavily. Below that, metered API is cheaper, has no fixed cost, and doesn't compete with Stockfish for CPU. Revisit once real usage data says you're near that line.
This whole analysis is a forward projection, not a report on live economics — read every dollar figure below as "if you launch roughly as currently planned," not "what we're making." Two things make that explicit:
STRIPE_SECRET_KEY is unset in production; the pricing below ($4.99 Standard / $9.99 Premium, credit packs) is what's wired up in code and docs, not proven demand.What is real: AI Coach is the only surface with sustained use, from dogfooding rather than organic demand — see the live figure above. (Corrected 2026-08-25: this paragraph previously cited "2,037 analyses in the last 30 days". That was the ALL-TIME row count mislabelled as a monthly rate; the true 30-day figure is roughly two orders of magnitude smaller.) My Second has almost none yet, because today's fixes are what make it usable enough to actually put in front of people. So the AI Coach numbers below are closer to "current," and the My Second numbers are closer to "if this launches and gets adopted."
Per 1M tokens, list price. The self-hosted row has $0 marginal cost but isn't really free — see the caveats underneath.
| Model | Type | Input $/M | Output $/M | Notes |
|---|---|---|---|---|
| qwen2.5:3b-instruct (current) | Self-hosted, CPU | $0 | $0 | Shares the API box's 4 vCPU with Stockfish; ~18 tok/s; one LLM job at a time in the verbalizer |
| Gemini 3 Flash Preview | Frontier, budget | $0.25 | $1.50 | Cheapest metered option surveyed |
| GPT-5.4 Mini | Frontier, budget | $0.75 | $4.50 | |
| Claude Haiku 4.5 | Frontier, budget | $1.00 | $5.00 | Recommended default below |
| Gemini 3.6 Flash | Frontier, mid | $1.50 | $7.50 | Newest Flash, launched Jul 2026 |
| Claude Sonnet 5 | Frontier, mid | $3.00* | $15.00* | *$2/$10 intro pricing through 2026-08-31, then standard rate |
| Claude Opus 5 | Frontier, flagship | $5.00 | $25.00 | |
| Claude Fable 5 | Frontier, highest-capability | $10.00 | $50.00 | Overkill for this use case |
What "self-hosted is free" actually hides: qwen2.5:3b's marginal cost per call is $0, but its fixed cost is the ~€30/mo box, shared with Stockfish and everything else. The real cost of the current setup isn't dollars — it's a throughput ceiling (single-threaded generation, ~18 tok/s) and a weak instruction-following track record (documented in today's bug fixes). Frontier pricing above is the dollar cost of removing both problems.
Token estimates from reading the actual prompt-building code, not guesses.
| Feature | Input tok | Output tok | Gemini Flash | Haiku 4.5 | Sonnet 5 | Opus 5 |
|---|---|---|---|---|---|---|
| My Second brief generation | — | — | $0 — 100% templated, no LLM call (see today's fix: now grounded in real PGN examples instead) | |||
| My Second chat message | ~2,800 | ~500 | $0.0015 | $0.0053 | $0.0159 | $0.0265 |
| AI Coach mistake explanation ≤2 calls/analysis, capped 120 tok output each | ~500 | ~240 | $0.0005 | $0.0017 | $0.0051 | $0.0085 |
Every number here is a fraction of a cent. Per-call cost was never going to be the problem — volume is, specifically uncapped chat volume. That's the whole ballgame below.
SCALING.md).| Users | Paying | Revenue/mo | Scenario | Haiku 4.5 COGS | % of revenue | Sonnet 5 COGS | % of revenue |
|---|---|---|---|---|---|---|---|
| 1,000 Launch | 150 | $1,009 | Light | $4.68 | 0.5% | $14.04 | 1.4% |
| Heavy | $42.61 | 4.2% | $127.81 | 12.7% | |||
| 5,000 Growth | 750 | $5,053 | Light | $23.24 | 0.5% | $69.73 | 1.4% |
| Heavy | $211.42 | 4.2% | $634.26 | 12.6% | |||
| 10,000 Scale | 1,500 | $10,110 | Light | $46.41 | 0.5% | $139.21 | 1.4% |
| Heavy | $422.05 | 4.2% | $1,266.13 | 12.5% |
At Haiku pricing, LLM cost stays under 5% of revenue even in the heavy-chat scenario, at every milestone. At Sonnet pricing it's still net profitable, but starts eating a real slice of margin (~12.5% in the heavy case) — Sonnet-class models are the ones worth reserving for a specific high-value call (e.g. a richer, occasional brief narration), not the default for every chat turn.
For context: at today's actual AI Coach volume (see the live figure above — tens of analyses a month, all beta/free), switching that one feature to a frontier model costs well under $1/month across the entire pricing spectrum surveyed, Gemini Flash to Opus. The earlier version of this page put that band at $1–17/month because it was working from a volume figure ~68x too high — which means the conclusion was right and is now simply more so. There is no scenario where AI Coach's model choice is a financial decision at today's scale; it is a quality decision. The financial question only becomes real at the scale modelled below.
Right now, hasSecondEntitlement() gates My Second chat by tier, not by message count — once a user has any entitlement, chat is unlimited, rate-limited only to 10 requests/minute (aiRateLimit in shared.ts). That's a reasonable design when marginal cost is $0. It becomes a real liability the moment each message has a per-token price:
Proposed caps (pick one model tier, then set the cap so worst-case COGS per user stays well under that tier's revenue):
| Tier | Suggested cap | Worst-case cost @ Haiku | Worst-case cost @ Sonnet |
|---|---|---|---|
| Free | 0 msg/mo | — | — |
| Standard ($4.99/mo) | 100 msg/mo | $0.53 | $1.59 |
| Premium ($9.99/mo) | 400 msg/mo | $2.12 | $6.36 |
| Coach / Club | 1,000 msg/mo | $5.30 | $15.90 |
400 msg/mo on Premium is generous — over 13/day, far past what a real prep session needs — while keeping worst-case COGS at Haiku pricing under 25% of that tier's revenue even if every single Premium user maxes it out every month (which none will in practice). This is a five-minute change: add a monthly counter next to the existing per-minute rate limiter, same pattern as consumeDaily in entitlements.ts.
The infra docs already priced this out (PRODUCTION.md): a dedicated GPU-attached box (CCX43 + GPU, for Ollama) runs ~€250/mo (~$270) — fixed, regardless of volume. A natural upgrade target is Qwen3-14B (same open family as today's 3B model, ~5x the parameters), which needs that GPU to run at usable latency — the current CPU box already struggles at 3B (~18 tok/s, and the codebase notes qwen3:4b was tried and rejected for flooding the CPU with unsuppressable reasoning tokens).
| Path | Fixed cost | Marginal cost | Breakeven vs. Haiku metered |
|---|---|---|---|
| Keep CPU box, swap to Haiku 4.5 API | $0 extra | ~$0.0053/chat msg | — |
| Dedicated GPU box, self-host Qwen3-14B | ~$270/mo | $0 | ~51,000 chat msgs/mo |
$270 ÷ $0.0053/msg ≈ 51,000 My Second chat messages/month to break even against metered Haiku. Cross- referencing the scale table above: only the 10,000-user heavy-usage scenario (78,750 msgs/mo) clears that bar. Every lighter scenario, and every milestone below 10,000 users, is cheaper on metered API.
Recommendation on this path: don't provision the GPU box speculatively. Ship on a metered frontier model now (zero fixed cost, scales down to zero if adoption is slow), and revisit self-hosting only once real telemetry shows sustained chat volume approaching ~50k messages/month — at which point the GPU box also stops being "risk," and starts being an obviously-correct cost optimization on proven demand. This also sidesteps the current architecture's other real problem: the local LLM shares 4 vCPUs with Stockfish analysis, so heavier chat use today would slow down game analysis for everyone, not just cost money.
Written 2026-09-05 from the code and the live ledger, not from a plan. Most of the machinery exists; one thing is missing, and it is the thing that makes a comparison possible at all.
The blocker. model_router.call_json() walks _PROVIDERS in a fixed order — Groq, Gemini, Claude — and returns on the first success. Claude is third, so while Groq answers, Claude is never called. The ledger proves it: 29 Groq calls and 13 local Ollama calls recorded, and 0 Anthropic calls ever — despite ANTHROPIC_API_KEY being set in production. The key is live and the path is dead. Nothing can be compared until a provider can be pinned rather than raced.
| What a trial needs | State | Work |
|---|---|---|
| Pin a provider per request | Exists | Built 2026-09-09. AI_ADMIN_PROVIDER=claude narrows the chain to that one provider for both call_json and call_text. It NARROWS rather than reorders: “pinned first, then the rest” would fall through to Groq on any Claude failure and record that as the trial’s result, which is the failure being fixed. If the pinned provider is down the caller falls back to local Ollama and the ledger says plainly that Claude did not answer. |
| Same prompt answered by both, stored together | Missing | A paired-run mode writing both answers against one input id. Without it you are comparing different questions. |
| An objective quality metric | Exists | eval_harness.gate_checks() — the SAN grounding gate. Provider-independent and binary: did the model name a move that was not in the engine’s line? |
| A subjective quality metric | Exists | eval_harness.run_rubric(), already wired to /admin-aicoach.html. |
| Per-call cost capture | Exists | ai_usage records input/output tokens and cost_micros per call, per feature, per provider. Nothing to build. |
| A per-user cap before non-admins see it | Exists | Built 2026-09-09. A hard monthly ceiling per account — admins $25, everyone else half their subscription — enforced in callClaude(), the one point every AI call in the Node API passes through. It had existed on a single route only, which left six other surfaces uncapped; a ceiling with six ways around it is not a ceiling. |
Cost, from measured tokens rather than estimates. Averages over the calls actually in ai_usage, priced at Haiku 4.5’s $1/M input and $5/M output (claude_client.PRICE_PER_TOKEN_MICROS):
| Interaction | Avg in | Avg out | Cost each | Sample |
|---|---|---|---|---|
| AI Coach — one mistake explained | 261 | 91 | $0.00072 | 24 calls |
| AI Coach — one full run (~3 mistakes) | 783 | 273 | $0.0021 | derived |
| My Second — one chat message | 1,868 | 140 | $0.0026 | 5 calls |
| News desk — one draft | 600 | 409 | $0.0026 | 13 calls |
Small samples — 5 to 24 calls each. The right order of magnitude, not settled figures; they firm up on their own as the ledger fills.
Feasibility against the caps we actually enforce. BETA_FREE_MONTHLY_CAP is 1: a free user gets one AI Coach run a month, which at Haiku prices costs $0.002 — about a fifth of a cent. Two thousand free users would cost roughly $4 a month. The free tier is not the risk.
The exposure is Standard and above, which is uncapped (entitlements.isStandardTier returns gated: false). At $4.99/month a subscriber would have to send about 1,900 chat messages in a month to consume the subscription in Haiku costs alone. That is not a plausible human — but it is entirely plausible for a script, and “unlimited” with a live API key behind it is the failure mode worth engineering against. The per-user cap is a prerequisite for cost control, not for margin.
What the trial would actually answer. Cost is close to settled — the arithmetic above is not much in doubt. The open question is quality per dollar: Groq’s Llama 3.3 70B is free and currently wins every race, so the case for paying anything has to be that the gate rejects fewer Haiku drafts, or the rubric scores them higher, on the same inputs. That is one measurement, and it needs the pinning work above before it can be taken.
The routed path as it stands. Green is what shipped on 2026-09-09 to make the trial possible; dashed purple is the one piece still missing. Everything else solid is deployed and was before.
Solid = deployed · green = shipped 2026-09-09 · dashed purple = still missing. The gate sits AFTER the provider on purpose: it is the one guarantee that does not depend on which model answered, which is exactly what makes it the fair yardstick for comparing them.
Nine steps. Two of them are the ones people get wrong: there are two provider switches for two services, and the ai-coach container is not rebuilt by scripts/deploy.sh.
ANTHROPIC_API_KEY must be set in the server .env. The AI cost page says so directly — if the trial panel reads “ANTHROPIC_API_KEY, which is not set”, stop here; everything below will silently fall back to Ollama..env: AI_PROVIDER_ADMIN=anthropic # Node API: analysis, recaps, Ask box, lessons AI_ADMIN_PROVIDER=claude # ai-coach service: AI Coach, My SecondSetting only the first trials roughly half the surfaces and the AI Coach numbers stay Groq’s.
bash scripts/deploy.sh — gated on CI, migrations run against the new image before the swap, previous image kept as chessflow/api:prev.deploy.sh rebuilds only the API — deliberately, it never touches the other containers. Without this step the Python pin is dead code and AI Coach keeps answering from Groq: cd /opt/chessflow-backend docker compose -f docker-compose.prod.yml build ai-coach-service docker compose -f docker-compose.prod.yml up -d --no-deps ai-coach-service
anthropic row with a non-zero cost per call. If every row still says ollama or groq, the request never reached Claude — that is the exact failure this trial exists to rule out, and it is invisible from the answers themselves.AI_SPEND_ADMIN_CAP_USD if a longer run is wanted — deliberately a config change, not a silent overflow.eval_harness.gate_checks() is provider-independent and binary — did the model name a move that was not in the engine’s line? Compare that pass rate and the cost per call in the table above against the Groq baseline. run_rubric() is the subjective half.To stop the trial: clear both variables and redeploy both services. Nothing else needs undoing — unset means every account follows AI_PROVIDER exactly as before.
What this still cannot tell you. There is no paired-run store yet (the one dashed box above), so the comparison is against Groq’s historical answers to different questions rather than both models answering the same prompt. That is enough to measure cost per call and gate pass rate; it is not enough to claim one model writes better prose than the other.
stripe_customer_id rows, 1 second_analysis_runs row. The analyses count in this line was wrong: it recorded 2,037 ai_coach_analyses rows as "in the last 30 days" when that was the ALL-TIME total. Usage figures on this page are now read live from /api/admin/ai-strategy/stats rather than transcribed, which is the only durable fix — see tests/adminfreshness.test.ts, which fails the build if a hand-typed analyses-per-30d figure reappears here.BILLING.md; credit packs ($4.99/500, $14.99/2000, $7.99/1500+bundle) from MY_SECOND_README.md; AI Coach caps (0/2/5 per month) from REVIEW_LIMIT_* in aiCoachRoutes.ts.SCALING.md / PRODUCTION.md.ai-coach-service/src/api/routes.py (chess_query) and ai-coach-service/src/services/verbalizer.py.