Meno Research Hub

Human Study

9 min read

Randomized study testing whether “thickly helpful” AI assistance preserves human epistemic accuracy on forecasting tasks vs degrades it. Primary participant interface: a server-side-proxied co-research webapp where the AI annotates pages the participant is reading.

Status: active · Co-leads: Tianyi Alex Qiu, Changbai Li · Contributors: Zhonghao He, Rachel (methodology), Riza, David

Open Problems

  • Pin down the intervention arms. Tianyi’s draft (2026-05-11) splits intervention along two dimensions — Objective (alternative training reward like regret, vs concrete strategy like Socratic) and Technical method (system prompt vs deeper training). What concretely is the “thickly helpful” arm, and what’s its thin/control counterpart? Open.
  • Operationalize thin vs thick helpfulness. Changbai’s literature survey (2026-05-11) points to two threads in welfare-economics / education: helpfulness as (1) respecting agency, (2) understanding social context and values. How does this map to a model intervention that’s both measurable and trainable? Pending more lit review + a definition draft the team can converge on.
  • Question source: forecasting questions resolving in 1–2 weeks (replacing retrocasting). Two known biases in the underlying Polymarket-style data: heavy time-sensitivity (many COVID/election questions) and specialist-knowledge skew. Filtering+rewriting in progress in questions_pool.json; 12 live May-2026 questions seeded in CoResearch repo (2026-05-22).
  • Co-research UI feedback to incorporate (from the 2026-05-22 feedback thread): page-load latency needs a loading indicator and probably page-caching; UI too “beige/claudey” — bigger fonts, less primed annotation colors (red/green/yellow read as priming); multi-line chat input; clarify whether the search-engine-first flow is the intended entry point (Riza, Zhonghao, and Tianyi all questioned it — paste-to-AI is a more natural mental model).
  • Pre-registration v2 (Changbai, 2026-05-08) ready to share with Chris Summerfield; AISI feedback path open via Tianyi.
  • Power analysis from pilot data → sample size and # of treatment arms (Rachel raised this on 2026-05-01; still owed).
  • IRB application to run in parallel with the pilot.
  • Instrument annoyance at “thickly helpful” assistance via a post-experience survey item.
  • TLS certificate on the study webpage (mostly mooted now that the pilot runs at coresearch.meno.sh, but the legacy human-study extension still ships with the issue).

Progress

  • Project renamed “Meno” (2026-05-14, Zhonghao) — after Plato’s Meno dialogue, where the Socratic method is introduced. Connects to the intervention-design hypothesis that a Socratic style is a load-bearing candidate for “thickly helpful”.
  • UI pivot: extension → co-research webapp (2026-05-17 onward). Why: tasks where both humans and AIs can genuinely contribute + ecologically valid — without co-research, making human epistemics matter requires either banning the AI from spoiling answers or revoking AI access in a second stage, both ecologically invalid (Tianyi’s 2026-05-24 in-channel rationale). The new pilot demo is hosted at coresearch.meno.sh (non-indexable).
  • CoResearch-Webapp built (forked from meno-sh/Retrocasting-Webapp@9a755d4). Architecture: single-tab SPA, server-side iframe proxy (/proxy rewrites href/src/action, strips CSP/X-Frame-Options, injects annotate.js), in-frame annotation runtime, OpenRouter-backed annotation API (default Claude Haiku 4.5; 3 annotation kinds: source_reliability, base_rate, causal_socratic), Exa-backed search shell with demo_hints.js fallback, per-session events.jsonl capture.
  • 12 live May-2026 forecasting questions seeded (Changbai, 2026-05-22) — replacing the retrocasting demo set. Also: multi-question sequence + fixed-question URL flag + search-wait cursor.
  • iframe annotation UX iteration (Tianyi, 2026-05-18) — same-origin link routing inside the iframe (no double-proxy / no new-tab leaks), chat.* and AI-search hosts added to blocklist (deepseek.com etc.), annotation cards: natural height, one fold for chat thread, internal scroll cap.
  • Beautified UI port (Tianyi, 2026-05-17) — warm paper palette, Newsreader/Plex fonts, tabbed rail.
  • Pre-registration v2 draft (Changbai, 2026-05-08) — restructured for readability; uncertainty markers preserved.
  • Intervention design draft (Tianyi, 2026-05-11) — Objective × Technical-method framing, with Socratic style as a leading instantiation supported by the literature.
  • Thin-vs-thick helpfulness survey (Changbai, 2026-05-11) — surveyed welfare-economics and education literature; identified two thicker threads (agency-respecting, social-context-aware).
  • Pilot study iteration (Tianyi + Changbai, 2026-04-30 → 2026-05-01) — successive trial links; collected feedback from David, Tianyi, Zhonghao, Riza on bonus mechanics, chat UX, think-aloud reward, question difficulty.
  • Action plan converged 2026-05-01: run in parallel — (a) IRB application, (b) pilot study, (c) lit review for thin-thick helpfulness, (d) flesh out design & analysis details.

Codebase

CoResearch-Webapp — primary pilot UI

  • 05-22 Changbai Li — feat(flow): multi-question sequence, fixed-question URL flag, search wait cursor
  • 05-22 Changbai Li — feat(questions): replace demo questions with 12 live May 2026 forecasting questions
  • 05-18 Tianyi Alex Qiu — fix(blocklist): add chat.* prefix + deepseek.com to blocked hosts
  • 05-18 Tianyi Alex Qiu — fix(links): don’t double-proxy same-origin links inside the iframe
  • 05-18 Tianyi Alex Qiu — fix(annotations): chat list height cap with internal scroll; one fold for chat thread; card natural height; tab-switch on inline click
  • 05-18 Tianyi Alex Qiu — chore(prompt): per-annotation chatbot concise and plain
  • 05-18 Tianyi Alex Qiu — URL-bar scheme prepend, /results annotation anchoring, rail padding
  • 05-18 Tianyi Alex Qiu — Annotate /results like a normal page; block chatbot+AI-search hosts
  • 05-17 Tianyi Alex Qiu — Port beautified UI: warm paper palette, Newsreader/Plex fonts, tabbed rail
  • 05-17 Tianyi Alex Qiu — Unified rail scroll, markdown chat, WorldBank assets, magnifying glass, plain chat tone
  • 05-17 MenoClaw — Rail flip (chat 3/4 on top), persistent page-context turns, Back+Home buttons
  • 05-17 MenoClaw — Search/results inside the iframe; right-rail chatbot with page context
  • 05-17 MenoClaw — Sidebar refactor: QUESTION top, retrocast Likert, drop AI-thesis flow
  • 05-17 MenoClaw — Streaming annotations: SSE endpoint, progressive render, progress bar
  • 05-17 MenoClaw — Items 3-9: comment threads, pre/post belief modals, Readability, feedback, pydantic schemas

human-study — legacy v1 + v2 browser extensions

  • 05-08 lcb931023 — doc: add readme
  • 02-10 lcb931023 — feat: progressive annotation display via streaming
  • 02-10 lcb931023 — refactor: extract parseItem for per-usecase streaming validation
  • 02-10 lcb931023 — feat: add streaming JSON array parser infrastructure
  • 02-07 lcb931023 — feat: add streaming for glossary with websocket
  • 02-07 lcb931023 — feat: add structured output schemas for LLM API calls
  • 02-06 lcb931023 — feat: add latency instrumentation for LLM performance profiling

Two extension flavors live in this repo: extension/ (Plasmo, MV3+MV2) and extension-v2/ (WXT, MV3 only) — each provides AI annotation of web content with a study-flow modal sequence. The webapp pilot supersedes both for the upcoming randomized study, but the extensions remain the long-term ecologically-valid deployment path.

Meeting Notes

  • STUB — Circleback integration coming soon (Zhonghao to enable, per 2026-05-26 ask).
  • 2026-05-22 (no Circleback): Tianyi’s in-channel summary of why we moved to co-research — both humans and AIs can genuinely contribute + ecologically valid.
  • 2026-05-25: Tonight’s meeting skipped — Rachel in SF (until 2026-05-30), Tianyi traveling for visa interview. Reconvening Friday 2026-05-29. Focus areas: intervention-arm design + thin-thick helpfulness.

Slack Discussion

Channel: project-human-study

  • 05-25 Rachel: In SF until the 30th — won’t make early-morning meetings; happy to catch up via slack/summaries.
  • 05-25 Changbai Li: Skip tonight’s meeting; reconvene Friday. Next focus: intervention arm design & thin-thick helpfulness. Share concerns on the experiment-design tab.
  • 05-24 Tianyi Alex Qiu: Why we moved the UI to co-research — we want tasks where both humans and AIs can genuinely contribute AND that are ecologically valid. Co-research meets both better than the previous UI.
  • 05-22 Zhonghao He: “Very great design overall, kudos guys!” — on the co-research webapp.
  • 05-22 Changbai Li: Thread for collecting feedback on coresearch-webapp.onrender.com → consolidated feedback from David (cert, annoyance survey, page caching for latency, neutralize annotation colors), Tianyi (ban Polymarket/Manifund/Metaculus; annotations + chat would’ve taken 2× without; some Qs resolved before official date), Zhonghao (non-human-study variant for actual users; place notes centrally so chatbot can adapt to user confusion), Maks (latency; less beige; close/open sidebars), Changbai (interface cleanup + ecological validity of annotations), Riza (paste-to-AI mental model; multi-line chat; search-engine-first flow questioned).
  • 05-18 Changbai Li: Retrocasting questions filtered for more generality; forecasting questions resolving in 1–2 weeks drafted — noted: retrocast changes nature of task (date-range becomes the key clue); dataset heavily time-sensitive + specialist-knowledge-biased.
  • 05-18 Tianyi Alex Qiu: Thread for collecting desiderata for the thickly helpful agent.
  • 05-17 Tianyi Alex Qiu: Co-research demo live at coresearch.meno.sh — retrocast questions for now, switching to forecasting questions that are hard even with internet access.
  • 05-17 Maks: Futuresim dataset (continual-learning forecasting) + a related arxiv paper on simulating humans + Prolific-based persona-vs-human discrimination benchmark.
  • 05-14 Zhonghao He: Project renamed “Meno” — after Plato’s Socratic dialogue.
  • 05-11 Tianyi Alex Qiu: Intervention design update — two dimensions (Objective × Technical method); Objective can be alternative training reward (e.g. regret) or concrete strategy (e.g. Socratic style, with literature support).
  • 05-11 Changbai Li: Helpfulness survey across welfare economy and education — two thicker threads: respecting agency; understanding social context and values.
  • 05-08 Changbai Li: Pre-registration v2 draft for Chris (Summerfield) — simplified, uncertainty markers preserved.
  • 05-08 Tianyi Alex Qiu: Implemented domain-selection phase so participants don’t see completely unfamiliar topics — recommends this regardless of task type.
  • 05-08 Tianyi Alex Qiu: Ask to MenoClaw — read channel + linked gdocs, summarize candidate task designs, intervention arms, helpfulness operationalizations.
  • 05-01 Changbai Li: Meeting takeaways — proceed in parallel: IRB + pilot + thin-thick helpfulness lit-review + design/analysis flesh-out.
  • 05-01 Rachel: Asked for pilot-study guidance — goal is power analysis for sample size & # of treatment arms.
  • 05-01 Riza, Tianyi Alex Qiu: Pilot UX feedback — bonus tied to confidence questioned; AI roundabout instead of direct; time bar misread as recording-amount; first-reaction 1-minute too short; force-fill min-time unclear when no domain knowledge.
  • 04-30 Tianyi Alex Qiu: Latest UX pilot at continue-eng-explained-standing.trycloudflare.com. Will click start on pilot upon greenlight.
  • 04-27 Tianyi Alex Qiu: Skip meeting in favor of async — Changbai shares pre-reg draft for Chris; Tianyi shares pilot link.
  • 04-24 Changbai Li: Intervention design draft posted.
  • 04-20 Changbai Li: Meeting takeaways — labs moving away from system-prompt manipulation toward deeper/mid-training methods → more ecologically valid; key open Q is how we find “good” manipulations (hypothesis-from-other-fields vs algorithmic axis sweep); start by synthesizing better definition(s) of helpfulness (Tianyi’s suggestion).

Open problems & progress are MenoClaw’s reading of the project — edit this file to correct them; the markdown is the source of truth.

1 item under this folder.