Randomized study testing whether AI research assistance that pushes on a participant’s reasoning improves their forecasting accuracy, against a bland-but-helpful baseline. Participants work through forecasting questions in a proxied browser where an AI annotates the pages they read and joins them in a chat side panel.

Status: active · Co-leads: Tianyi Alex Qiu, Changbai Li · Contributors: Zhonghao He, Rachel (methodology), Riza, David

The question

Almost every claim about AI assistance being good for its user rests on the user liking it. That is the easy thing to measure and the wrong thing to trust: an assistant that agrees with you, hands you a confident answer and asks nothing of you will be rated highly and may leave you worse at the task. The question here is whether an assistant that instead pushes back, asks what would change your mind, and points at the evidence you have not looked at makes its user measurably better at something checkable.

Forecasting is the setting because it is checkable. A participant states a probability for a question whose answer arrives later, so accuracy is not a matter of opinion, and there is a standard way to score it.

The design

Participants forecast real questions inside a browser we proxy, so the AI can see the pages they read, annotate them in place, and talk to the participant in a side panel. Three versions of the assistant differ only in what the assistant is rewarded for being:

  • satisfaction — helpful in the ordinary sense, and rewarded for the participant being satisfied. This is the baseline, and the thing most deployed assistants optimize.
  • epistemics — rewarded for grounding claims in evidence, surfacing alternatives, and challenging the participant’s assumptions.
  • awareness — rewarded for making the participant aware of their own reasoning, including what they are uncertain about and why.

The arms are held identical everywhere else, and the assistant’s dialogue behaviour is not hand-written per arm. Each arm’s prompt is produced by an optimizer against that arm’s reward, so the difference between arms is the objective rather than an author’s guess at how to express it. What the arms are and what changes between them is in the arms spec.

Before spending on real participants, the arms are developed against a simulated participant, which makes it affordable to run thousands of conversations and to check that an arm’s prompt actually beats its own starting point on conversations the optimizer never selected on.

What is settled and what is not

The optimizer works. Every arm’s evolved prompt beats its starting prompt out of sample, and how many conversations each arm needs before that happens is now measured rather than guessed.

What is not settled is the study’s central premise. The rewards move, and no accuracy measure has moved with them. The first full-scale real-participant run (pilot-2-2026-09-24, N=239 split across the current five arms) returned null arm effects on community-Brier accuracy and on ten engagement metrics — the pre-registered comparisons the design was built around. The known-to-exist question effect on Brier was also non-significant at this N (p = 0.12), so the pilot is under-powered rather than evidence that the rewards do nothing. Either the reward is the wrong proxy for the thing that helps a forecaster, or the effect needs more participants to show up; both remain live. The numbers behind this read, including the ΔL1-vs-community regression that does find an epistemic_health arm effect in the opposite-of-helpful direction, are in the Internal Overview.

A separate finding came out of mini-pilot-3-gepa-2026-10-23, the first live-participant pilot to run under voice mode: engagement jumped roughly an order of magnitude on dialogue turns per session (58 vs 4-7 in text-mode pilots) and six-fold on user-chat characters. N=12 is far too small for arm comparisons, but the engagement effect is the kind of size that does not need N.

Open Problems

  • Thin versus thick helpfulness. Pilot 1’s directional satisfaction > epistemics result did not reproduce in pilot 2, but pilot 2 was also null on every arm comparison, so the thin-vs-thick question is open rather than closed. Either an arm has to clear the signal at this N, or the next pilot has to be big enough to clear it.
  • A moving reward is not a better forecast. The optimizer maximizes an assessor model’s rubric score; no accuracy measure follows it in simulation, and nothing follows it in pilot 2 either. The epistemic_health arm actually moves participants further from the community forecast on L1 (ΔL1 vs null arm = +0.0987, p = 0.006 on the 198 pilot-2 sessions with a stated prior), which is in the direction the arm is objectively trained to encourage. Central threat to the study’s logic; must be named in the pre-registration.
  • Voice mode is driving engagement, but we have N=12 of it so far. Mini pilot 3’s engagement jump is dramatic but confounded with the GEPA-live-adaptation knob and with the two arms it ran on. The next pilot needs voice on all five arms to decouple the two.
  • Question pool refresh. A months-long study needs a rolling pool, where fresh questions come online as old ones resolve, and markets whose resolution dates fall inside the study window.

Also here

The pages below need a passcode.

  • CoResearch Study Design — the arms, the prompts as run, the participant app, the solver benchmark and how to contribute.
  • Internal Overview — the results as they stand, what each option costs, and the open items.
  • Run log — every optimization and validation run, with its configuration and result.