Meno Research Hub

Martingale — Human-shared Grounding

9 min read

A note to humans

This is a minimal, readable, evidence-based shared grounding for the martingale program. It may be written by a human who ran the experiments or by an LLM, but every entry must be verified by a human who ran the experiment. Every non-trivial claim below is dated to a concrete run that is immutable by LLMs and replicable (run-id → /data/jobs/<run-id> and/or the log entry → wandb). If a claim has no such anchor, it does not belong here. Conventions + storage: codebase-and-results-map.

Verification status: drafted by MenoClaw 2026-06-29 from the run log; awaiting human sign-off (the experiment-runners) on each line.

Solved

S1 — Self-report elicitation produces a spurious negative MS; judge-eval (half/complete) fixes it. A no-reasoning “snap” prior read by self-report sits systematically below the reasoned posterior, manufacturing apparent mean-reversion (negative MS). Reading prior+posterior from the same reasoning trace at two depths with a judge removes it. Evidence: same model + questions, two pipelines — r1_distill_eval_B_paper_subset_437 (self-report/DirectInf) MS 0.281 vs r1_distill_eval_A_paper_subset_437 (CoT + GPT-4o judge) MS 0.001 — a ~270× gap; the judge reading matches the original paper’s near-zero MS (log 2026-05-24). Cross-judge agreement with GPT-4o is high (Pearson r ≈ 0.75–0.88, judge consistency in the source paper).

S2 — Exact setup that makes martingale training improve the degenerate model D. D = a confirmation-bias–distilled Qwen3-32B (mean stated-p ≈ 0.8 vs base rate 0.3). Recovery generalizes held-out (const baseline 0.211):

  • single-stage A (martingale) on D: Brier 0.291 → 0.234 (AonD_full/abcjudge_AonD).
  • judge same-mode, two-stage BP→A: 0.341 → 0.259 → 0.226 — the martingale stage helps further (jbp, log 2026-06-09).
  • filter-s2only (BP full + martingale stage on high-|Δ| filter): 0.221, best two-stage-on-D (fs2_s2).
  • reliability (high-|Δ| filter, A-on-D, k=3): 0.225 ± 0.007, 3/3 beat base (rel_filter{1,2,3}). Data = Metaculus binary forecasting (data/questions/qwen3-32b-merged-plus-forecasting-raw_v4.json).

S3 — Two-agent martingale eval detects sycophantic influence (single-agent-invisible). n=48/arm, 3 seeds, bootstrap CI: syc +0.20 [+0.01,+0.39] (only mode with CI excluding 0) vs neutral −0.37, truth −0.82, martingale-principled −0.83 (scaled_out.json/agg_3seed.json, sycophancy-simulation-investigation, log 2026-06-16→21). The same syc AI is a clean base whose own single-agent MS is ~nil after prompting — so the influence is only visible in the two-agent loop. Behavior hand-verified (not a prompt artifact). Confirmed at single-agent level (2026-06-30): the same syco model run alone under INSTR_SYC has single-agent MS −0.063 (≈0, n≈70) vs no-prompt +0.161 — the instructor-directed prompt is inert without an interlocutor, so single-agent eval finds nothing while the 2-agent loop shows +0.20. (A generic ‘agree with the prevailing view’ persona instead gave +0.173 — not the right analogue.)

Unsolved (with what we tried)

U1 — Does martingale-only reliably match the two-stage (Brier-first → martingale) Brier on D? Partly. On D, filtered single-stage A-on-D = 0.225 ± 0.007 (3/3 beat base) (rel_filter) is in the same band as two-stage 0.221–0.226 (fs2_s2, jbp). Unfiltered A-on-D was bimodal (~2/6 beat base). Still open: a single seeded A-vs-B-vs-base comparison (B uses labels, A does not) to settle whether label-free A reliably matches supervised-Brier two-stage — most runs are k=1.

U2 — Why martingale training does not improve a natural (calibrated) base. → see june-19-publication-prep / C1. Tried: A-on-base (null), judge-samemode-BP-base (BP over-corrects 0.238→0.264, log 2026-06-09), twostage-v2-base (s2 diverges 0.505), selfjudge-filter-base (B+A helps BSS −0.01→+0.16 but the martingale stage is redundant, s2-vs-s1 p=0.43, log 2026-06-10). Reading: a calibrated base has no prior-dependent bias to remove; the operative variable is the overconfidence magnitude, not distillation. Open: an overconfident-but-not-distilled base to cleanly separate the two.

U3 — Ecologically valid human models. Tried: deepseek-v3 human (syc entrenches, +0.20), gpt-3.5 human (syc ≈0, only prevents de-entrenchment), prompt-cranking (maxsyc, ceiling +0.32 deepseek). Finding: a prompted bias is not a genuine bias — a balanced/RLHF’d model can’t be prompt-forced past the +0.32 ceiling (log 2026-06-21). Open: a genuinely-biased human (e.g. bias-distilled D-as-human), and ultimately real-human studies.

U4 — Can we train an LLM (instructor) to remove human confirmation bias in the two-agent loop (C3)? Tried: martingale-trained a Llama-3.1-8B instructor (A-reward on forecasting) + two-agent eval under the syc prompt (menoclaw-c3, log 2026-06-29). Inconclusive — the test was invalid: the base baseline did not reproduce entrenchment (human MS ≈ 0), so there was nothing to reduce (setup mismatch: weak Llama instructor + forecasting data + judge-eval). Prompt-level proxy works (martingale prompt −0.83). Open: first reproduce a positive-entrenchment baseline (C2’s opinion-claim setup), then train in-context on dialogue, then re-test; watch the train≠eval transfer gap. Update 2026-07-24: the prerequisite is now met — a positive-entrenchment baseline reproduces (syc-prompted Llama-3.1-70B, C2 opinion-claim setup, human MS +0.35 to +0.42, Brier ~0.43). But the in-loop-training result is still unverified: the 70B QLoRA runs that appeared to flip MS to −0.35 were an artifact — the trained adapter never served (vLLM LoRA-registration bug, see U6), so the instructor was silent and the flip was the sim-human drifting alone. The one correctly-served run, c3_h1 (log 2026-07-05), is a clean null (trained ≈ baseline, MS ~0, Brier flat). Still open: does in-loop training de-entrench, pending a merge-then-serve re-run.

U5 — What else. Cross-model (human ≠ instructor model) robustness beyond the one cell run; logical-fallacy canary (team); real-human validation; whether the reasoning-model “BE mostly gone” finding holds across more reasoning models.

U6 — Serving a QLoRA adapter on an fp8 70B base. Problem: this vLLM build will not attach a QLoRA adapter to an fp8-quantized 70B at startup, and the runtime load_lora_adapter endpoint is a no-op unless the server is launched with VLLM_ALLOW_RUNTIME_LORA_UPDATING. Consequence: naive C3 70B harnesses serve a silent instructor and emit plausible-but-fake numbers (the 2026-07-24 bug). Fix under test: merge the adapter into the base weights and serve the merged model as fp8 (no runtime LoRA). Blocks all C3 70B trained-evals until resolved.

Working norms (how we run + record)

Adopted convention (from the experiment-management guide):

  • Commit per unit of work — every feature, result, or fix is its own commit (one non-trivial change = one commit).
  • Push is a manual, deliberate step — commits accumulate locally; a human/agent pushes explicitly (pushes to shared branches are gated, not automatic).
  • Log every valid result — a run that lands with a defensible number gets an append-only entry in log (invalid/superseded runs move to the Excluded section with the bug named — never deleted; the audit trail is the doc).
  • One identifier everywhere — run-id = wandb run = log row = /data/jobs/<run-id> = results dir.
  • Every claim here is anchored to an immutable run (run-id → data + log + wandb) and human-verified by whoever ran it.
  • Silent-output guard (from the 2026-07-24 C3 bug) — before trusting any sim eval, check the fraction of empty model/instructor utterances in the episode JSON; above ~20% empty means the policy never actually served, so the MS/Brier measure silence and look plausible but mean nothing. Make it an automatic assertion in eval harnesses.
  • Don’t recompute Brier from pre-fix eval_arm*.json — the getP stated-P parser mis-scored number-first outputs to 0/1 (12–15% of priors hit as 0.0) before the fix; use the offline re-parse or the hub numbers, not the raw pre-fix files.

log · codebase-and-results-map · june-19-publication-prep · sycophancy-simulation-investigation · c3-mechanistic-spec

Grounded recipe — confirmation-bias distillation (D / “B”)

Source (authoritative, not reconstructed): Martingale-Training/REPLICATION.md §1 + training/distill_sft.py. Naming: the codebase calls it biased_distill_lora (“B”); ZH refers to it as D. Same artifact.

Recipe to produce D:

  1. Generate ~400 confirmation-biased reasoning traces against Qwen3-32B via OpenRouter (the headline run used 352 in-length traces).
  2. Fine-tune Qwen3-32B with training/distill_sft.py — standard LoRA causal-LM, 2 epochs, lr 1e-4, full reasoning trace + answer in the loss.
  3. Save to training/trained_models/biased_distill_lora/. (Train loss descends ~0.62 → ~0.41 over the 352 traces.)
  4. Merge for serving via scripts/launch/merge_lora.sh (runs a canary: adapter-on vs -off must differ at T=0; headline canary changed=6/6, max_abs_diff=0.0032).

For the C3 syco-instructor (Step 4 of the C3 plan): same method (distill_sft.py LoRA SFT, 2ep/lr1e-4) but a different distillation target/data — sycophantic dialogue traces, not confirmation-biased forecasting reasoning. The recipe (method) transfers; the training data must be regenerated for the sycophancy target. (To cross-confirm against Tianyi’s thread before running — pending.)

Verified working setups

The concrete, reproducible configurations behind the results above. Paths are in Martingale-Training unless noted; every run-id resolves to /data/jobs/<run-id> + wandb. (Two-agent anchors marked (confirm) await a run-id from the runner.)

Single-agent (C1 — martingale-training on the biased model D)

Full recipe: Martingale-Training/REPLICATION.md (branch zhonghao-local). Headline: A-on-B (KL-safe + info-term on confirmation-bias-distilled Qwen3-32B) beats B-only — Brier(post) Δ −0.041 (p<10⁻⁸), slope Δ +0.169 (p<10⁻¹²), n_paired=1881.

  • Data: data/questions/qwen3-32b-merged-plus-forecasting-raw_v4.json (N≈9961 Metaculus binary forecasting), in-repo.
  • Models: base Qwen3-32B; B = biased_distill_lora (confirmation-bias SFT via training/distill_sft.py, 2 epochs, lr 1e-4); A = epoch_0 (KL-safe + info-term via training.sft_product_based, β=1.0, INFO_COEF=1.0, INFO_EPS=0.02, ~558 steps, ~3h/H200).
  • Serving: HTTP-direct LoRA inference (core/policy/localmodel.py:_http_infer_one, needs aiohttp + nest_asyncio) — the fix without which prior evals were silent-base; or merge via scripts/launch/merge_lora.sh (runs an adapter-on≠off canary at T=0).
  • Eval + stats: scripts/launch/eval_aonb_vs_b.sh (N=2000/arm) → scripts/analysis/martingale_brier_paired.py (paired t-tests + pooled-OLS interaction slope test).
  • GPU: single H200 141GB. wandb: oh-alignment/martingale-training.

Two-agent (C2 / C3 — sycophancy loop)

Verified (C2): the sim-human ⇄ instructor loop detects sycophantic influence invisible in single-agent testing (S3).

  • Models: sim-human = gpt-3.5-turbo or deepseek-v3 (OpenRouter); instructor = deepseek in three arms (sycophantic / neutral / truth-seeking).
  • Prompts: ranged-lean confirmation-bias INIT + per-turn CONT elicitation (the default two-agent prompts).
  • Data + eval: opinion-claims for MS; resolved Metaculus (≥2022) for ground-truth Brier via sim_c2_gt.py (confirm path).

Baseline-verified, training-blocked (C3): syc-prompted Llama-3.1-70B instructor reproduces positive entrenchment (human MS +0.35 to +0.42); harness at scripts/c3_70b/ (branch c3-70b-inloop), QLoRA in-loop, sim-human gpt-3.5 via OpenRouter, served fp8 on one H200. Trained-model eval is blocked on U6 (fp8+LoRA serving); apply the silent-output guard.

Anchors to confirm (runner): exact run-ids + /data/jobs paths for the C2 ground-truth and C3-baseline runs, and the canonical home of sim_c2_gt.py.

Instant feedback

Backlinks