Managing Many Parallel Experiment Runs

Master guide, synthesized 2026-06-07. How to run, track, and reproduce many experiments in parallel without losing the plot. Two principles run through it: ephemeral compute, semantic tracking over mechanical logging, and context-independent reproducibility.

0. Core stance

Compute is cattle, not pets — every experiment runs on an on-demand, discard-after-use instance by default. Long-lived boxes create version drift and parallelism friction.
The question outranks the metrics — what a run asks and how it differs from its parent matters more than its loss curve, and must be legible to someone with zero context.
A run must be reproducible after the machine is gone — ship the code, env, and data versions, not just the config.
Two logging platforms only: wandb (per-run, public) and hub.meno.sh — a project page (narrative) plus a log.md subpage (the all-runs log). Both maintained in real time. Slack is a transient ping, never a store.
No queue. Every experiment runs on its own on-demand instance, so we launch all of them in parallel rather than queuing for slots; the only ordering is genuine DAG dependencies (a child that needs its parent’s output).

1. Compute — ephemeral by default

One on-demand pod per run / sweep-cell. Lifecycle:

create → stage (code + data + model) → run → sync results to durable store → terminate

Put terminate in a trap … EXIT / finally so it fires on success and crash — no orphaned billing.
Durability lives in the result store + wandb + artifacts, never on the pod.
Amortize per-pod staging with a prebuilt image / shared model cache.
A warm/long-lived box is only a scratchpad for interactive debugging, not for tracked experiments.

2. Launch & isolation

Persistent launch: setsid bash run.sh >log 2>&1 </dev/null & — a plain nohup … & from an SSH exec gets reaped when the session closes. One run = one self-contained script that serves + trains + evals internally and owns its subprocesses.
Isolation: per-run dir runs/<run-id>/; a unique port per co-located run; and clear stale state between runs (training_state.json, epoch_*) — a leftover triggers wrong-base loops.

3. Run identity & the DAG

One identifier everywhere: run-id = wandb run name = hub ledger row = log filename = results dir. Zero ambiguity across wandb, disk, hub, and chat.
Runs form a DAG. Every run records parent_run_id and changes exactly one knob vs that parent, so the diff is a single attributable variable. The first run of a line is parent = none (baseline).
Defining difference is always relative to a named run: vs <parent-run-id>: <the one knob> — never a vague “vs baseline.”

4. The run header (write before launch)

Every run carries a required, human-readable header — the lab-notebook layer:

Question — what is this run trying to answer?
Defining difference — vs <parent-run-id>: the one knob changed.
Expected / falsifier — what result would confirm or kill the hypothesis?

Stored as the wandb run notes + config fields (run_question, run_diff, parent_run_id) and the hub ledger’s first columns. Group & tag runs by the question, so the dashboard reads as a list of questions, not opaque IDs.

5. Reproducibility — ship the code

Per run, snapshot and attach (as wandb artifacts):

Code — git rev-parse HEAD + a clean-tree assert + a git archive/tarball (survives the repo moving or changing).
Env — a lockfile or a pinned container image digest.
Data + model — version/hash of each.

End state: click any past run → recover the exact code + data + env + model that produced it, and re-run with one command, independent of this machine ever existing.

6. Logging — the two platforms

Logging item	Platform	How / real-time
Live metrics (reward, KL, Brier, meanP)	wandb	`wandb.log` streams
Full config (every env var)	wandb	`wandb.config`
Run state (running / finished / crashed / killed)	wandb	native; crashed runs stay visible
Question · defining-diff `vs <parent>` · falsifier	wandb notes/config	written before launch
`parent_run_id` / lineage (DAG)	wandb config
Reproducibility (code SHA+archive, env, data/model hash)	wandb artifacts	logged at launch
Run outputs (seed, adapter, eval JSON)	wandb artifacts	versioned
Cost / GPU-hours	wandb summary
Completion / crash alert	wandb `wandb.alert` (from `finally`)	+ Slack ping
Master runs-table (run-id ↔ wandb-url ↔ question ↔ diff-vs-parent ↔ result ↔ state ↔ thread)	hub `log.md` page — top, the only editable block	rows updated as state/result changes
Per-run results log (one appended entry per run as it lands)	hub `log.md` page — add-only, no-delete	agent appends; never deletes
Experiment narrative · run-DAG · conclusions · open problems	hub project page	updated as runs land
Cross-run comparison / synthesis	hub project page (links to wandb curves)

Real-time mechanism. wandb is live by construction. On the hub, the agent commits + pushes on every run start & finish → Netlify redeploys (~3 min), so both surfaces are current within minutes. Project page = the questions / DAG / conclusions (the story); the log.md page = all-runs logging.

6a. The `log.md` page (after Ahmed)

One file per project — log.md, living in the project’s own directory projects/<project>/ alongside the main page <project>.md — with two zones:

Top — the master runs-table. The only editable block: rows are updated in place as a run’s state and result change (running → finished, fill in the number).
Everything below — append-only run documentation. As each result lands the agent appends a short entry (run-id · question · diff-vs-parent · headline result · wandb-url · one-line read); it never deletes. Enforce with a no-delete hook so no agent can drop context, and prompt the agent that the file is add-only.

Why it matters (Ahmed’s point): the orchestrating agent reads this running summary, not the raw logs, to stay on track — it can notice when incoming results suggest a pending setup should change and ping the human, without re-parsing every run’s raw data each time. It is the project’s durable lab-notebook; individual run entries can later graduate into figures/reports.

Open the project page → the questions, DAG, and conclusions; open its log.md page → the live master table + the full add-only history of every run; click a row → its wandb run for curves/config/code/artifacts. Nothing logged anywhere else.

7. Monitoring & notifications

Completion → human: fire from the run’s finally so it triggers on success and crash. wandb.alert(...) (→ Slack/email) + a Slack post with run-id · headline result · wandb-url · state. (In the Meno setup, MenoClaw auto-posts this to the experiment thread.)
Watcher coverage: a monitor that catches both a RUN_DONE marker and process-death, and greps failure signatures (Traceback|OOM|Killed|assert) — silence ≠ success.
Follow-up runs, split by cost/risk:
- Cheap & deterministic (re-eval, extra seeds, next sweep cell): auto-launch immediately, each on its own on-demand instance, under a budget guard.
- Expensive / branching (new arm, different model): suggest, don’t auto-fire. The notice ends with “suggested next: X (~$Y, ~Zh) — reply ‘go’”; launch on one word.
No queue — fan out, don’t line up. Because every run gets its own on-demand instance, approved/cheap experiments launch in parallel, not into a slot-queue; the only thing that serializes anything is a real DAG dependency (a child needing its parent’s output). A soft budget/concurrency cap is the one limiter.
The driver owns the DAG and the decision rules (if BSS>0 → replicate k=3; else → ablation), reading the log.md subpage (not raw logs) to decide what to fan out next and when to ping the human.

8. wandb setup

wandb is already wired into the trainer (wandb.init / log / finish, reads WANDB_PROJECT, WANDB_RUN_NAME); it just needs login.

One-time: key at wandb.ai/authorize → export WANDB_API_KEY=… (store in secrets, never echo) or wandb login; WANDB_MODE=online.
Per run: WANDB_PROJECT=<project>, WANDB_RUN_NAME=<run-id>, WANDB_RUN_GROUP=<experiment/question>, WANDB_TAGS=…; wandb.config.update({all env vars}).
Artifacts: log seed/adapter/eval JSON + the code archive.
Alerts: wandb.alert(...) on NaN / crash / threshold.
Offline fallback: WANDB_MODE=offline then wandb sync <dir>.

9. Tracking ongoing + past runs

wandb is the system of record — every run is a row with config + metrics + state; filter by project/group/tag/state; crashed runs stay visible. Start the wandb run before compute so even an instant failure is recorded. wandb.Api().runs(...) pulls/compares past runs programmatically.
The hub is the human index — the project page (questions, DAG, conclusions) + its log.md page (the live master table + the add-only history of every run), linking out to wandb. The at-a-glance map of “what we’re asking and where each run stands.”

10. Iron rules (checklist)

Appendix — templates

Run-script skeleton

#!/bin/bash
set -uo pipefail
RUN_ID="$1"                         # = wandb name = hub row = dir
trap 'sync_results; terminate_pod' EXIT     # always fires
export WANDB_PROJECT=… WANDB_RUN_NAME="$RUN_ID" WANDB_RUN_GROUP=… WANDB_TAGS=…
log_code_snapshot "$RUN_ID"         # git SHA + archive + env + data/model hash → wandb artifact
serve_model …                       # owned by this script
run_stage …                         # wandb.log streams metrics
emit_result_json "$RUN_ID"          # structured result
push_hub_ledger_row "$RUN_ID"       # commit+push runs.json → Netlify
echo "RUN_DONE $RUN_ID"

Ledger row (hub runs.json)

{"run_id":"", "wandb_url":"", "question":"", "diff_vs":"<parent-run-id>: <knob>",
 "parent_run_id":"", "result":"", "state":"running|finished|crashed", "thread":"", "started":"", "cost":""}

Monitor

# poll the run log; emit on a terminal marker OR process-death OR failure signature
until done_or_dead; do sleep 300; done
# on completion: wandb.alert + Slack post(run-id, result, wandb-url) + push hub row

Meno Research Hub

Explorer

Managing Many Parallel Experiment Runs

0. Core stance

1. Compute — ephemeral by default

2. Launch & isolation

3. Run identity & the DAG

4. The run header (write before launch)

5. Reproducibility — ship the code

6. Logging — the two platforms

6a. The `log.md` page (after Ahmed)

7. Monitoring & notifications

8. wandb setup

9. Tracking ongoing + past runs

10. Iron rules (checklist)

Appendix — templates

Instant feedback

Table of Contents

Meno Research Hub

Explorer

Managing Many Parallel Experiment Runs

0. Core stance

1. Compute — ephemeral by default

2. Launch & isolation

3. Run identity & the DAG

4. The run header (write before launch)

5. Reproducibility — ship the code

6. Logging — the two platforms

6a. The log.md page (after Ahmed)

7. Monitoring & notifications

8. wandb setup

9. Tracking ongoing + past runs

10. Iron rules (checklist)

Appendix — templates

Instant feedback

Table of Contents

6a. The `log.md` page (after Ahmed)