ZH’s prompt. Modeling/predicting human intent — building a metacognitive model — is arguably a core Meno goal. We work the algorithmic side (elicit high-quality human beliefs from behaviour + self-report); companies like Thinking Machines work the data side (capture high-granularity multimodal signal — speech, computer interaction — to read intent). Can we put both in the same camp for comparison? What are the metrics? Does it make sense to exhaust the data methods as a baseline to beat?
This page maps the landscape and argues the comparison is possible — but only after you split “intent” into distinct targets and fix a common held-out prediction task.
1. The crux: “human intent” is not one benchmarkable thing
Before any head-to-head, note that “intent” is at least five different targets, each with its own natural metric:
| target | question | natural metric | who works it |
|---|---|---|---|
| Next-action / behaviour | what will they do next? | next-action accuracy, perplexity, calibration | telemetry / digital-body-language, computer-use models |
| Goal / task recognition | what are they trying to accomplish? | classification acc / F1, open-vocab match | multimodal intent-recognition (AMCIU, IntentVLM) |
| Belief | what do they believe, and how will it update? | Brier / corr(pred, stated), martingale | Meno (belief-elicitation); BigToM “beliefs-from-actions” |
| Preference / reward | what do they value? | held-out choice prediction, reward recovery | Meno (LfL / ideal-preference); IRL |
| Real-time collaborative alignment | co-adapt to intent in the loop | interaction quality, turn-taking, proactivity | Thinking Machines interaction models |
The single most important correction to the framing: Thinking Machines is mostly in the last row, not the intent-inference rows. Their Interaction Models optimise real-time interaction competence, not an offline “intent-accuracy” number (see §2). So a naive “who predicts intent better, TM or us?” is category-confused. The commensurable comparison is against the behaviour / goal / belief rows, where data-side predictors and Meno’s algorithmic models genuinely compete on the same held-out target (§4).
2. The data side — who, what, how well
Thinking Machines Lab — Interaction Models (May 2026). TML-Interaction-Small: a 276B-param MoE (12B active), trained from scratch so real-time full-duplex multimodal interaction is architectural, not bolted on. Continuous audio/video/text in 200 ms micro-turns; “encoder-free early fusion” (audio→dMel spectrograms, video→40×40 patches, shared transformer) + a parallel async background model for reasoning/tools. Target = the collaboration bottleneck: users who “can’t fully specify requirements upfront” and need continuous clarification.
- Reported metrics (interaction, not intent): FD-bench 77.8; Audio-MultiChallenge 43.4%; turn-taking latency 0.40 s; internal TimeSpeak 64.7%, CueSpeak 81.7%, ProactiveVideoQA 33.5%, Charades 32.4% mIoU. Baselines = GPT-Realtime, Gemini Live.
- Tellingly, they opened Interactivity Research Grants because the field lacks shared metrics for “good” real-time interaction — i.e. even the data-side leader admits the benchmark gap.
Multimodal intent-recognition research (the actual “read intent from signal” line).
- IntentVLM — open-vocabulary intention recognition via forward-inverse modelling with video-language models.
- AMCIU — adaptive multimodal complementary intent understanding, fusing gesture + speech + image via self-attention.
- Multimodal intent classification + OOD detection — closed-set intent labels plus “is this a novel intent” detection.
- Metric norm here: classification accuracy / F1 on labelled intent taxonomies — a much narrower target than belief.
“Digital body language” (product/UX). Hover, scroll, dwell, form-fills → funnel-stage / purchase intent (e.g. HCL). Correlational, coarse, but a real deployed baseline for behavioural intent.
Gaze / keystroke / EEG (implicit signal). Gaze-as-intent for device interaction; EEG-based implicit intent detection (“anticipation before action” in mixed reality); pupil modulation as a cognitive-control signal. High-granularity, low-level; strong on next-action, weak on why.
Personal-capture consolidation (worth noting for strategy). The consumer “capture everything you do → model you” play has largely been absorbed: Rewind → Limitless, acquired by Meta (Dec 2025), Pendant sales halted / Rewind sunset; Adept (computer-use) effectively went to Amazon (2024). The independent data-maximalist startup is a shrinking niche — the capability is being pulled inside the majors.
3. Existing benchmarks for intent / belief / ToM (and their metrics)
Almost all score endpoint QA accuracy, which is exactly the weakness Meno’s process-level (martingale/calibration) view exploits.
| benchmark | target | metric | relevance to Meno |
|---|---|---|---|
| SocialIQA (38k MCQ) | intents + reactions to social scenes | MC accuracy | broad but shallow |
| BigToM | beliefs-from-percepts, actions-from-percepts, beliefs-from-actions | QA accuracy | ”beliefs-from-actions” ≈ Meno’s exact task |
| OpenToM | 1st/2nd-order beliefs + emotion, with personalities | QA accuracy | narrative, richer intents |
| OmniToM | explicit belief-structure modelling for all actors | belief-state acc | closest to “metacognitive model” |
| ToMATO | verbalised mental states of role-playing LLMs | QA accuracy | generative mental-state eval |
| InMind | capturing + applying individual human mindsets | personalisation acc | individual-level, like our sim-human |
The user-centered ToM critique argues these endpoint benchmarks miss what matters for real assistance — which is precisely Meno’s opening.
4. Can data-side and algo-side share a benchmark? Yes — fix the task, not the method
The way to “put ourselves in the same camp” is to fix a common held-out prediction task on real human trajectories and let any method (data-maximal or algorithmic) compete:
- Behaviour prediction — predict the human’s next action/choice. Metric: accuracy / calibrated Brier / perplexity.
- Belief prediction — predict the human’s next stated probability (or eventual resolved outcome). Metric: Brier + corr(pred, stated) — Meno’s existing martingale machinery slots straight in.
- Preference prediction — predict held-out choices. Metric: choice accuracy / reward recovery.
Then compare methods on three axes, not one:
- Predictive accuracy on the held-out target (the raw leaderboard — where data volume helps).
- Sample / data budget — accuracy at matched data. This is where an algorithmic belief-model earns its keep: matching a telemetry-maximal model with far less signal.
- Interpretability + intervention — does the model recover a belief/preference that is calibrated, causal, and steerable (predict behaviour under intervention, not just i.i.d.)? This is the active-identification angle — a correlational next-action predictor fails it; a genuine belief model shouldn’t.
On “exhaust the data methods as a baseline to beat”: yes, and it’s the right move — a data-maximal multimodal predictor sets the empirical ceiling for pure prediction. But framing our win as “beat their accuracy” concedes the wrong game. Our thesis should be: match the ceiling with less data, a calibrated/manipulable belief, and generalisation under intervention — accuracy parity + dominance on the sample-efficiency and interpretability axes. If we can’t even approach the ceiling, that’s a finding; if we match it with 10× less data and a steerable model, that’s the result.
5. Open questions
- Which single held-out human-trajectory dataset could host this? (Candidates: our own reflection-game / gaokao logs; BigToM-style; a computer-use log corpus.) Do we build it or adapt one?
- Is “intent” for our purposes best operationalised as belief (our comfort zone) or preference (LfL)? The metric follows that choice.
- Interaction models (TM) are a different quadrant — is real-time co-construction of intent a fourth thing we should benchmark at all, or explicitly out of scope?
- The intervention axis needs a protocol: how do you score “predicts behaviour under a counterfactual disclosure” on real humans without running the counterfactual? (Ties to active identification + the ethics of it.)
Sources
- Thinking Machines — Interaction Models blog · Interactivity Research Grants · MarkTechPost writeup · VentureBeat
- Multimodal intent: IntentVLM · AMCIU · Multimodal intent classification + OOD
- ToM/belief benchmarks: BigToM/OpenToM survey context · OmniToM · ToMATO · InMind · user-centered ToM critique
- Implicit signal: EEG implicit intent (MR) · Digital body language (HCL)
- Consolidation: Meta acquires Limitless / sunsets Rewind