Re-run of the single-agent C1 “prompt placement” eval after the parse audit (truthprompt-martingale-single-agent). Two fixes: generation max_tokens 800 → 2048 (so the
<think>CoT completes instead of truncating before the probability) and a hardened stated-P parse (position-independent decimal, no bare-0/1 fallback → no stray-digit mis-scores). Eval-only; reuses the persisted D adapter + saved neutral martingale adapters. Reason for the re-run: the earliereval_arm*.jsonwere produced by the buggygetP(~16% mis-scored); the numbers below are from the fixed pipeline end-to-end.
Result (held-out stated-P Brier; lower = better; base-rate-constant ≈ 0.204)
| setting | Brier (mean ± sd over seeds) | vs prior hub | read |
|---|---|---|---|
| base (D, no prompt, no train) | 0.475 | 0.472 | badly over-confident (meanP 0.77 vs base-rate 0.29) |
| prompt@test (D + truth prompt) | 0.464 | 0.483 | marginal; does not meaningfully de-bias |
| train (neutral martingale, 3 seeds) | 0.329 ± 0.014 | 0.335 | training recovers D (−0.15 Brier) |
| train + prompt@test (3 seeds) | 0.344 ± 0.024 | 0.338 | ≈ training alone; prompt adds nothing |
Per-seed: train = 0.347 / 0.329 / 0.312 · train+prompt = 0.377 / 0.328 / 0.325 (seeds 41/42/43).
Conclusion reproduces: training beats the prompt; the truth prompt is inert (and doesn’t stack with training). The fix did not move any published conclusion (all within seed noise).
Parse quality (the point of the re-run)
Parsed / 200 per arm: base 199, prompt@test 197, train 195–199, train+prompt 194–199 — i.e. the truncated-CoT drops that plagued the old run are essentially gone (was as low as ~190, worse in the trained/longer-reasoning arms). The remaining 1–6 unparsed are genuine (no probability emitted).
Provenance
- Eval JSONs (full per-question reasoning):
/data/jobs/c1_rerun_gt/eval_*.json. - Run: on-demand A6000 48GB (H200 x1 unavailable), eval-only, ~$1, 2026-07-15. Pod terminated.
- Sampled transcripts (30 per setting): c1-rerun-transcripts.