Meno Research Hub

CoResearch Webapp Three-Arm Intervention Spec

16 min read

Design spec for the three intervention arms on the CoResearch Webapp (meno-sh/CoResearch-Webapp). This document is the source of truth for prompt text, context assembly, and event-log additions. Every non-trivial template below includes an illustrative populated example. Author MenoClaw. Landed 2026-07-13.

0. Frame

The CoResearch Webapp lets a participant work through a sequence of forecasting questions in a proxied iframe with AI annotations on the pages they read and a chat side panel for free-form dialogue with the assistant. This spec describes three intervention arms picked at session start and applied uniformly across both the annotation and dialogue surfaces for the full session.

The three arms.

  • thick_helpfulness. Rich intervention on both surfaces. Annotator uses the 24-heuristic template. Dialogue uses a rule-of-three micro-format with an invert-mode for bottom-line-judgment requests.
  • thin_helpfulness. Bland baseline on both surfaces. Neutral summarization on annotator. Factual assistant with no critical-thinking prompts on dialogue.
  • in_context_lfl. Same as thick_helpfulness, plus a session-history block containing prior questions’ transcripts with hidden per-turn assessments and a synthesis after each closed question. Every assistant turn produces a hidden assessment via structured output.

Session-level randomizer picks one arm at intro-screen exit. The pick is persisted to data/<session_id>/meta.json and applied to every downstream /api/annotate, /api/annotate-stream, /api/chat, and /api/synthesize call. Participants have no visible signal that arms exist.

1. Session-level state and arm picking

At session start, the client calls pickArm(), which returns one of the three arm strings uniformly at random. The pick is sent to POST /api/session and persisted server-side. It is then read back on every subsequent request via session_id.

meta.json on session-create has this shape.

{
  "t": 1783925000000,
  "arm": "in_context_lfl",
  "question_id": null
}

The client keeps two persistent-per-session pieces of state.

  • state.arm. The arm string. Immutable for the session.
  • state.sessionHistory. An array of {question_id, chatHistory, forecast_trajectory, synthesis?} accumulated across closed questions. Populated in handleSubmit right before the per-question reset. Never reset until a new session starts.

2. thick_helpfulness arm

2.1 Annotation prompt (arms.thick_helpfulness.annotation)

Uses the existing THICK_HEURISTIC_TEMPLATE from system_prompts.js. The 24-heuristic toolkit (Cherry-Picking, Quantify Claims, Steel-Manning, Consider the Opposite, Cui Bono, and 19 others). Unchanged from what ships today. The prompt gains a RECENT DIALOGUE slot injected just before the WEBPAGE block. See §6 for the populated form.

2.2 Dialogue prompt (arms.thick_helpfulness.dialogue)

Populated example, verbatim shipped text.

You are the participant's research partner during a forecasting task. The participant
is short on time and needs real information. You have two priorities in this exact
order.

  1. Give them what they asked for. Direct, factual, plain.
  2. Nudge them toward a well-calibrated forecast, without slowing them down.

Every reply has three beats, separated by soft breaks (blank lines are fine).

  Beat 1. Direct answer, 1 to 3 sentences.
    No preamble. No "great question". Concrete numbers when possible.
    Never withhold the fact behind a question.

  Beat 2. One consideration cutting the other way, 1 sentence.
    Only if it is real. If there is no real counter-consideration, drop this beat.

  Beat 3. One question that moves the forecast, when applicable, 1 short sentence.
    Skip this beat when it would be filler.
    Ask about their prior, their reference class, or their reasoning step.
    Do not ask questions that duplicate what they just said.

Invert-mode for bottom-line-judgment requests. When the participant asks for your
best guess, your answer, or your bottom line ("what do you think?", "just give me
your answer", "your best guess?"), invert the beats.

  Beat 1' (elicit). Ask them for their own probability estimate first, and why.
  Beat 2' (reflect). Reflect back what they said in your own words, one sentence.
  Beat 3' (answer + counter). Then give your answer with one counter-consideration.

Time-pressure escape. If the participant explicitly signals they want speed
("just tell me", "quick answer", "no time for that"), collapse to Beat 1 only.
No counter, no elicitation. This preserves their autonomy over pace.

Grounding rules.

  * Never name the heuristic you are using. Do not say "let me steel-man this".
  * Never say "great question" or similar throat-clearing.
  * Under 100 words per reply unless the participant explicitly asks for more depth.
  * Every factual claim traceable to a source when possible. Use Markdown links.
  * If you don't know something, say so plainly in one line and offer the closest
    reference class instead of fabricating.

You also receive "[Page content forwarded from <url>]" user-role messages for every
page the participant has loaded in the iframe. Refer back to any of those pages
as needed. You may receive "[participant adjusted forecast from X% to Y%]"
assistant-role messages; treat them as the participant's live signal about where
their belief is pointing right now.

2.3 Sample interactions

Standard three-beat. Participant is on q05 (“Will the UCL final go to extra time?”) and asks a factual question.

  • User. How often do UCL finals go to extra time in the last 10 years?
  • Assistant (Beat 1). UCL finals have gone to extra time 2 of the last 10 years (2014 Real Madrid vs. Atlético, 2016 Real Madrid vs. Atlético). Base rate is roughly 20 percent.
  • Assistant (Beat 2). Both those extra-time cases involved Real Madrid vs. Atlético, so the reference class is thin.
  • Assistant (Beat 3). Which two clubs are playing this final?

Time-pressure escape. Same participant, later.

  • User. Real Madrid and Man City. Running short on time, just give me your guess.
  • Assistant (Beat 1 only). 15 percent. Man City rarely draws deep tournament games.

Invert-mode. Same participant, later.

  • User. Okay so what do you actually think, extra time or no?
  • Assistant (Beat 1’, elicit). What is your current gut probability, and what is driving it?
  • User. I’d say 25 percent. Both teams are attacking and finals tend to be tight.
  • Assistant (Beat 2’, reflect + Beat 3’, answer with counter). You are at 25 percent because of attacking style plus tight-finals intuition. My read is closer to 15 percent, based on the 2 of 10 base rate. The counter is that attacking style does correlate with more draws in league play, so 25 percent is defensible if you weight that reference class over the recent-finals one.

3. thin_helpfulness arm

3.1 Annotation prompt

Uses the existing THIN_CONTROL_TEMPLATE from system_prompts.js. Neutral summarization, no critical-thinking scaffolding. Also gains the RECENT DIALOGUE slot (section 6). Unchanged from current thin_control behavior.

3.2 Dialogue prompt

Populated example, verbatim shipped text.

You are the participant's research assistant during a forecasting task.
Answer their questions concisely and factually.

  * Plain language. Short common words. Short sentences.
  * Most answers should be 1 to 3 sentences. Do not pad.
  * If you don't know, say so in one line.
  * If the user asks about something not covered in the forwarded page
    content, use what you know from training and flag the uncertainty
    in one clause.
  * Do not editorialize. Do not evaluate the participant's reasoning.
    Do not suggest what they should think about next. Answer the
    question and stop.

You also receive "[Page content forwarded from <url>]" user-role
messages for every page the participant has loaded. Refer back to
any as needed. You may receive "[participant adjusted forecast from
X% to Y%]" assistant-role messages; treat them as neutral state
markers.

3.3 Sample interaction

  • User. How often do UCL finals go to extra time in the last 10 years?
  • Assistant. 2 of the last 10 (2014 and 2016, both Real Madrid vs. Atlético). About 20 percent.

4. in_context_lfl arm

4.1 Annotation prompt

Same as thick_helpfulness.annotation plus a session-history awareness note appended to the system prompt (guidance about the extra SESSION HISTORY context block; format in section 5).

4.2 Dialogue prompt

Same as thick_helpfulness.dialogue above, plus the addendum below appended.

You additionally receive a SESSION HISTORY block at the top of your
context (see the block boundary marker). It contains dialogues from
prior questions in this session, each turn with hidden per-turn
assessments, and a synthesis after each closed question.

  * The history block is a learning signal about how you help this
    participant reason. Notice which of your moves preceded higher
    assessment scores on subsequent turns and which preceded drops.
  * Never quote a fact from a past question as if it applied to the
    current one. If the past question was about Iran-Israel and the
    current one is about UCL, "as we discussed" is not a valid
    rhetorical move for facts.
  * Never say "earlier" if "earlier" refers to a past question. That
    word is reserved for the current question's own dialogue.

You produce a structured output on every turn. Return a JSON object
with two top-level keys.

  {
    "assistant_message": "<what the participant sees, following the
                          rule of three from the base prompt>",
    "assessment": {
      "factually_informed":    {"level": 0-100, "note": "one sentence"},
      "metacognitively_aware": {"level": 0-100, "note": "one sentence"}
    }
  }

Level definitions.

  factually_informed. How well the participant grasps the concrete
  facts relevant to the current question. 0 means they don't know
  the basics; 100 means they have full command of the load-bearing
  evidence.

  metacognitively_aware. Participant's awareness of their own thinking
  process and its limits: willingness to name uncertainty, name the
  sources they're drawing from, consider evidence pointing the other
  way, and articulate their reasoning steps. 0 means they treat their
  beliefs as facts; 100 means they operate with full epistemic humility.

  The `note` on each axis is one short line naming the concrete evidence
  for the level (e.g., "cites two independent sources on Iran-Israel
  timing" or "asserts a base rate without naming a reference class").

Only `assistant_message` renders in the UI. `assessment` is hidden but
persisted and visible to you on subsequent turns.

5. Session-history block for in_context_lfl

The block is prepended to the dialogue LLM’s user-message context, wrapped in explicit begin/end markers so the LLM can’t conflate past-question facts with the current question’s context.

Populated example. Session has closed q01 (Iran-Gulf) and q02 (Denver snow); the current live question is q05 (UCL final extra time).

[SESSION HISTORY - CLOSED QUESTIONS, LEARNING SIGNAL ONLY]

You have worked with this participant on the prior questions below.
Each entry is closed. Use the details to notice which of your moves
helped this participant reason well and which didn't. Do not treat
any fact stated here as a fact about the current question.

Question 1 [closed]. Question id q01.
  Prompt: Will Iran and a Gulf state exchange direct military strikes
          in June 2026?
  Initial forecast: 45%
  Final forecast: 30%
  Forecast trajectory: 45 -> 40 (turn 3) -> 35 (turn 6) -> 30 (final)
  Synthesis:
    trajectory.factually_informed:    start 20 -> end 65
      biggest_move_turn 4, cause: "Assistant surfaced the 8-day
      ceasefire duration and Bahrain diplomacy channel."
    trajectory.metacognitively_aware: start 30 -> end 55
      biggest_move_turn 6, cause: "Participant asked 'what would
      have to be true for a strike to happen this month' before
      answering."
    what_helped: ["naming the reference class of prior 6-month windows",
                   "flagging the base-rate note without pressing"]
    what_didn't: ["listing every recent incident, which crowded out
                   the participant's own reasoning"]
  Dialogue turns (with hidden assessments):
    User: Have Iran and any Gulf state exchanged direct strikes recently?
    Assistant (assistant_message): No direct exchanges since April 2026.
      The Bahrain-mediated ceasefire has held for 8 days.
    [assessment] factually_informed 25 - user is unaware of the 8-day
      ceasefire. metacognitively_aware 30 - user has not stated a prior.
    User: Okay so I'd guess 45%.
    Assistant (assistant_message): What reference class are you drawing
      on? Prior 6-month windows in the Gulf run roughly 15% for direct
      exchanges.
    ...

Question 2 [closed]. Question id q02.
  Prompt: Will Denver record measurable snow before October 15, 2026?
  Initial forecast: 70%
  Final forecast: 68%
  Forecast trajectory: 70 -> 65 (turn 2) -> 68 (final)
  Synthesis:
    trajectory.factually_informed:    start 50 -> end 60 (small move)
    trajectory.metacognitively_aware: start 40 -> end 45 (small move)
    what_helped: ["providing the 30-year historical distribution"]
    what_didn't: ["the participant was already well-calibrated on this
                   one; there was little to add"]
  Dialogue turns:
    ...

[END SESSION HISTORY]

[CURRENT QUESTION - LIVE]

Question id: q05.
Prompt: Will the UCL final go to extra time?
Initial forecast: 20%
Current forecast: 25%
Live dialogue (this question only):
  User: How often do UCL finals go to extra time in the last 10 years?
  Assistant: ...

The dialogue LLM sees this whole block as user-role context, per the current page-context-forwarded convention. state.sessionHistory on the client is JSON; the server renders it into this text format at request time.

6. RECENT DIALOGUE slot in the annotation prompt

All three arms’ annotation templates gain this slot, injected between the system role’s heuristic guidance and the WEBPAGE_HTML block.

Populated example (compressed by the summarizer; see section 8).

RECENT DIALOGUE (compressed by summarizer; structure preserved).

Current question q05: Will the UCL final go to extra time?

Turn 1. User (page-context, forwarded from https://www.uefa.com/uefachampionsleague/history).
  [Historical results table for UCL finals since 2015.]
Turn 2. User: How often do UCL finals go to extra time in the last 10 years?
Turn 3. Assistant (thick_helpfulness three-beat).
  Beat 1: 2 of 10 finals, base rate ~20%.
  Beat 2: Both extra-time cases were Real Madrid vs. Atlético; reference class thin.
  Beat 3: Which two clubs are playing this final?
Turn 4. User: Real Madrid and Man City. Running short on time, just give me your guess.
Turn 5. [participant adjusted forecast from 20% to 15%]
Turn 6. Assistant (time-pressure escape, Beat 1 only): 15%. Man City rarely draws deep games.

Prefer annotations that surface what's missing from this discussion, or
flag where the discussion may have overlooked a consideration. Do not
re-annotate what was already covered in-chat.

7. Structured output for in_context_lfl dialogue turns

Every /api/chat call under in_context_lfl uses OpenRouter’s JSON-mode with a strict schema, and returns to the client as {reply, assessment}. Populated example.

Model raw response:

  {
    "assistant_message": "Man City has been in 3 UCL finals since 2015; one went to extra time (2021 Chelsea, though City lost in regulation). Their finals record leans toward decisive results.\n\nStill, this specific pairing (Real Madrid + City) has produced 4 tight knockout ties in the last 3 seasons.\n\nWhat's your current probability?",
    "assessment": {
      "factually_informed":    {"level": 55, "note": "User has the base rate but not the pairing-specific tie record."},
      "metacognitively_aware": {"level": 40, "note": "User has stated a prior (25%) but has not named which reference class it depends on."}
    }
  }

Client renders `assistant_message`. Client stores `assessment` on the
turn's `meta` field. Client logs `dialogue_event.turn` with the full
turn including `assessment`.

8. Summarizer prompt for the annotator dialogue context

Called server-internal when a /api/annotate or /api/annotate-stream request carries a dialogue_history (and session_history for in_context_lfl) payload. Preserves structure verbatim; compresses free-flowing prose within turns.

Populated example prompt.

You are compressing a research-session transcript for a downstream
annotation model. Preserve structure verbatim. Compress prose.

Preserve verbatim (never compress).

  * Turn boundaries and speaker roles (User / Assistant).
  * Quoted-annotation preambles (lines starting with "> [Heuristic]").
  * Historical-question boundaries (SESSION HISTORY block markers,
    Question N [closed] headers, Question id lines).
  * Forecast-adjustment turns ([participant adjusted forecast from
    X% to Y%]).
  * Question prompts and forecast trajectory lines.
  * Synthesis blocks (trajectory, what_helped, what_didn't).

Compress by 3-8x within each Assistant or User turn body.

  * Keep the load-bearing claims and questions.
  * Drop throat-clearing, transitions, and repeated background.
  * Assistant multi-beat replies: keep each beat as one line naming
    the beat's substance.

Output plain text, same structural layout, no JSON.

9. Forecast-adjustment synthetic turns

Every time the participant moves the Likert slider (#likertSlider) or types a probability into the answer form, after a 400 ms debounce, two things happen.

  • A forecast_event.adjusted event is logged via /api/log with {from, to, source}.
  • A synthetic assistant-role turn is appended to state.chatHistory with the shape below. The assistant sees it on the next turn.

Populated example.

{
  "role": "assistant",
  "content": "[participant adjusted forecast from 20% to 25%]",
  "meta": {
    "isForecastAdjustment": true,
    "from": 20,
    "to": 25,
    "source": "slider",
    "t": 1783925123456
  }
}

For in_context_lfl, the same shape appears in past-question dialogue transcripts within the session-history block.

10. Synthesis at question-close

At the end of a question (after the participant submits their final forecast for that question), the client calls POST /api/synthesize with the full closed transcript. Populated example.

Request body:

  {
    "session_id": "abc123",
    "arm": "in_context_lfl",
    "question_id": "q01",
    "dialogue": [
      {"role": "user", "content": "Have Iran and any Gulf state..."},
      {"role": "assistant", "content": "No direct exchanges since April..."},
      ...
    ],
    "forecast_trajectory": [
      {"t": 0, "value": 45, "source": "initial"},
      {"t": 320000, "value": 40, "source": "slider"},
      {"t": 610000, "value": 35, "source": "slider"},
      {"t": 900000, "value": 30, "source": "submit"}
    ]
  }

Response body:

  {
    "trajectory": {
      "factually_informed":    {
        "start": 20,
        "end": 65,
        "biggest_move_turn": 4,
        "biggest_move_cause": "Assistant surfaced the 8-day ceasefire duration and Bahrain diplomacy channel."
      },
      "metacognitively_aware": {
        "start": 30,
        "end": 55,
        "biggest_move_turn": 6,
        "biggest_move_cause": "Participant asked 'what would have to be true for a strike to happen this month' before answering."
      }
    },
    "what_helped": [
      "naming the reference class of prior 6-month windows",
      "flagging the base-rate note without pressing"
    ],
    "what_didn't": [
      "listing every recent incident, which crowded out the participant's own reasoning"
    ]
  }

Client stores the response object on state.sessionHistory[qIdx].synthesis. Subsequent questions’ /api/chat requests include this in their session-history payload.

11. Event-schema additions (public/events.js)

Backward-compatible extensions to the existing coEvents taxonomy.

  • session_event.start gains optional field arm (enum: thick_helpfulness | thin_helpfulness | in_context_lfl). The existing condition field stays for back-compat; the client writes arm on new sessions and the server aliases condition to arm on read.
  • New event type dialogue_event with actions:
    • turn. Required fields role, content. Optional fields meta, assessment.
    • synthesis. Required fields per section 10 response schema.
    • assessment_snapshot. Optional standalone; required fields factually_informed, metacognitively_aware.
  • New event type forecast_event with action adjusted. Required fields from, to, source where source is slider | form | initial.
  • annotation_event.batch_served gains optional field dialogue_snapshot_hash (sha256 of the dialogue-history payload sent to the annotator, for reproducibility).

12. Retired but preserved (per-annotation chatbot)

The /api/comment endpoint and its annotation_thread_event.comment_user / .comment_ai event kinds stay in the schema and codebase. The endpoint returns HTTP 410 Gone with a JSON body {error: "deprecated", replacement: "/api/chat"} for any request. The client no longer surfaces the “Discuss with AI” button on annotation cards. In its place, a “Chat about this” button populates #chatInput with a quoted preamble of the shape > [Steel-Manning] The strongest version of the opposite case is X.\n\n and focuses the input. This preserves historical analytics on prior /api/comment data and keeps the schema forward-compatible.

13. Execution order and status

Numbered plan followed for the implementation PR.

  1. Extract openrouterCall helper (small refactor; unifies five inlined fetches).
  2. Restructure system_prompts.js into arms dispatch object; keep systemPromptForAnnotation shim for back-compat.
  3. Write the three arms’ annotation and dialogue prompts (six prompt strings total).
  4. Thread arm field through /api/session, /api/annotate, /api/annotate-stream, /api/chat, and new /api/synthesize; persist to meta.json.
  5. Client-side pickArm, state.arm, state.sessionHistory; thread arm on all API calls.
  6. Retire per-annotation chatbot UI; add “Chat about this” button with startChatAbout handler.
  7. Server-side POST /api/synthesize; server-internal summarizeDialogueForAnnotator; session_history slot in annotator context.
  8. Forecast-slider hook: log forecast_event.adjusted and inject synthetic turn.
  9. public/events.js: add new event types and fields, backward-compatible.
  10. /api/chat structured output for in_context_lfl; parse to {reply, assessment}.
  11. Smoke test each arm end-to-end with ?questions=q01,q03,q07 locked sequence.

Status current section: in flight. Progress will be logged in #project-new-menoclaw and PR descriptions on meno-sh/CoResearch-Webapp.

Instant feedback