Gaokao Standalone — 1k Bot Stress-Test Report

Robustness report for the new standalone engine meno-sh/meno-gaokao-counsellor (extracted from reflection-game). A diverse bot drove full games (intake → per-stage reorder/chat/next loop → end → report) against a live instance; every failure was logged. Companion: separation plan.

Headline

772 diverse full games · 99.6% clean (769/772). The new engine runs end-to-end under load; the only failures were rare LLM-output variance (now fixed). Two real extraction bugs were caught and fixed during the run.

Method

tests/botload.py — per game: random 2–5 candidates (from a ~26-item pool incl. tricky resolve cases), random confidence / value-weights / destination / free-text (incl. empty, very long, macro-thoughts, EN), random in-game reorders, per-stage chat, per-stage confidence, and a random end-stage (2–cap) to cover early-exit and deep play. Each game used a unique X-Forwarded-For (simulating distinct users → past the per-IP rate limiter). Concurrency 16. Bug capture: non-200, exceptions, error fields, scenes missing prose/factor/top/contender, empty endings, empty/short reports, timeouts.

Results

metric	value
games	772
clean	769 (99.6%)
games with ≥1 bug	3 (0.4%)
latency / full game	median 153 s, p90 250 s, max 381 s
stages / game	median 6, max 11
candidate-count coverage	2/3/4/5 ≈ evenly (~240 each)

The 3 bug instances (all rare LLM-output variance, non-structural):

• rank_report returned empty (HTTP 200, 0-length body) ×2 — model occasionally emits no report.
• one game’s stage-4 scene was missing prose + contender ×1 — model occasionally drops a field.

Bugs found & fixed (during the run)

1. 🐛 Missing core/checklist.py — lazily imported by ranked.ending(), so the import-smoke and rank_start checks passed but rank_end crashed (presented as RemoteDisconnected). The first pilot was 0/50 until this was found. → copied + verified.
2. 🐛 GAOKAO_API_KEYS bearer-auth missing — the 雨来 API-key gate lived only on the yulai branch; extracting from main dropped it. → ported + verified (no-key/bad-key → 401, good-key → 200).
3. ⚙️ Rate limiter characterized: GAOKAO_RL_PER_HOUR=100 per-IP/hr on rank_start (a feature; the bot must vary its source IP).
4. 🔧 The 2 residual bugs above, hardened: rank_report now retries once then falls back to a deterministic non-empty report; the scene contender defaults to the #2 label and the turn re-generates once if prose/contender_take are dropped.

Caveat — belief metrics are not behavioral here

The bot makes random reorders/confidence, so in-test “confidence 48→48” and “top-changed 50%” reflect random inputs, not human behavior — they only confirm the recording/rerank/confidence machinery fires. Real belief behavior comes from live data (where confidence rises ~+12 on average).

Verdict

The standalone engine is stable and usable (99.6% over 772 diverse full games). The extraction-introduced bugs (checklist, auth) are fixed; the LLM-variance bugs are hardened. Remaining transition work (shadow-run vs old → cutover) is per the plan.