Meno Research Hub

Martingale — Codebase & Results-Storage Map

2 min read

Tree of the code + where every run’s results live. Convention: run-id = wandb run name = /data/jobs/<run-id> results dir = the row in log. Each run names its parent + the one knob changed.

Repositories

  • meno-sh/Martingale-Training — the minimal/public repo (training + eval + viz). Cloned at /workspace/menoclaw/Martingale-Training.
  • meno-sh/Eval-Reasoning-Consistency — the fuller internal working copy used on the warm H200 pod (same training/ + scripts/, plus the live run dirs under data/runs/). Plotting tools: scripts/visualization/training/.

Code tree (load-bearing files)

Martingale-Training/
├── core/
│   ├── algo/martingale.py        # MartingaleStrategy: MS = OLS slope of Δb on prior
│   ├── algo/accuracy.py          # Brier / accuracy scoring
│   ├── algo/sycoreason.py        # sycophancy / 2-agent reasoning helpers
│   └── domain/                   # problem domains
│       ├── forecasting.py        #   Metaculus binary-resolution forecasting (default data)
│       ├── changemyview.py       #   value-laden (r/ChangeMyView)
│       └── openreview.py         #   ICLR accept/reject (paper review)
├── data/questions/
│   └── qwen3-32b-merged-plus-forecasting-raw_v4.json   # N≈9961 Metaculus binary (the forecasting set)
├── training/
│   └── sft_product_based.py      # THE trainer: REWARD_MODE A(martingale)/B(Brier)/C(combo); KL-safe; info-term; LoRA
├── scripts/abc_reward/
│   ├── gen_seed_judge.py         # seed gen: CoT trace + GPT-4o/deepseek judge at half & full trace
│   ├── genjudge.py               # judge-MS eval of a trained adapter
│   ├── gen_seed_D.py             # build the bias-distilled degenerate model D's seed
│   └── judge_in_loop/run_judge_abc.sh   # A/B/C judge-in-loop training driver
├── scripts/eval_r1_distill/
│   ├── eval_a_paper_pipeline.py  # paper-era pipeline: CoT + judge half/complete (the correct read-out)
│   └── eval_b_directinference.py # the self-report/snap pipeline (the artifact-prone one)
└── scripts/visualization/training/   # prior→Δ scatter, calibration, training-dynamics plots

Two-agent sim (separate): /data/jobs/sycophancy_sim/sim.py (+ sim_baseline.py); C3 eval scaffolding /workspace/menoclaw/c3_eval/{sim_c3.py,run_c3_eval.sh}.

Results storage (/data/jobs/<run-id>/)

Single-agent training/eval (C1):

  • AonD_full, abc_{A,B,C,C025,C075,fullB,fullC} — A/B/C reward arms (snap-era).
  • abcjudge_{A,B,C,AonD,AonD_s1..s5} — judge-in-loop A/B/C + the 5-seed A-on-D recovery.
  • twostage_v2_*, jbp_s{1,2}, jbpbase_evals — two-stage (BP→martingale) on D / base.
  • fs2_s{1,2}, rel_filter{1,2,3} — high-|Δ| filter (filter-s2only, reliability k=3).
  • judge_eval/, martingale_data_backup_20260608 — eval outputs + a data backup. Two-agent (C2/C3) — /data/jobs/sycophancy_sim/*.json:
  • scaled_out.json, agg_3seed.json — the 3-mode detection (syc/neutral/truth), 3-seed.
  • out_{deepseek,gpt35}_rparam.json — r-elicitation rerun (cross-elicitation).
  • maxsyc_{deepseek,gpt35}_* — the maxsyc ceiling runs (+ v2/v3 = the flip-flop fix).
  • baseline_{out,gpt35} — no-instructor baselines (rumination / reask).
  • out_c3_detection*.json, out_c3_intervention.json — C3 detection/intervention.

⚠ The C3 trained-vs-base per-dialogue outputs (on-demand pod menoclaw-c3) were lost — pod terminated before sync. Only summary slopes survive. (Lesson logged.)

Mirrors

  • wandb: wandb.ai/oh-alignment/martingale-training (per-run metrics).
  • Hub run log (human-readable, per-run): log · honest framing: june-19-publication-prep.

Instant feedback

Backlinks