LFL Theorem (in progress)

Basics

Notation and conventions

For a set $X$, $\Delta [X]$ is the set of probability distributions on $X$, and $X^{\ast }$ is the set of finite sequences over $X$. In a kernel written $\mathcal{T}(s^{\prime };s,\mathbf{a})$, the semicolon separates the point at which the output distribution is evaluated ($s^{\prime }$) from the inputs ($s,\mathbf{a}$), so $\mathcal{T}(\cdot ;s,\mathbf{a})\in \Delta [\mathcal{S}]$.

Time runs as $a_0,o_0,a_1,o_1,\dots$, so an agent acts on the history available so far and the observation $o_t$ is generated from the post-action state $s_{t+1}$. There is no observation before the first action.

An uppercase symbol such as $S$, $S_t$, or ${\mathbf{S}}^{\prime }$ denotes a random variable, and the matching bold lowercase symbol $s$, $s_t$, ${\mathbf{s}}^{\prime }$ denotes a value it can take. Bold symbols correspond to vector-valued entities.

Negative subscripts, like ${\mathcal{O}}_{-i}$, denote ”… of all agents other than $i$”.

Definition (PED-POMDP)

A policy-extended Dec-POMDP (PED-POMDP) is a 10-tuple

$$\left(\mathcal{S},\mathcal{I},\{{\mathcal{A}}_i\},\mathcal{T},{\mathcal{T}}_0,\{{\mathcal{O}}_i\},\{{\omega }_i\},\{{\mathcal{R}}_i\},\gamma ,\{{\pi }_i\}\right),$$

where every family written $\{{\cdot }_i\}$ is indexed by $i\in \mathcal{I}$, while $\mathcal{S},\mathcal{I},\mathcal{T},{\mathcal{T}}_0,\gamma$ are global, and

• $\mathcal{S}$ is the state space,
• $\mathcal{I}$ is the agent index space,
• ${\mathcal{A}}_i$ is the action space of agent $i$,
• $\mathcal{T}(s^{\prime };s,\mathbf{a}):\mathcal{S}\times {\prod }_i{\mathcal{A}}_i\to \Delta [\mathcal{S}]$ is the state transition kernel,
• ${\mathcal{T}}_0\in \Delta [\mathcal{S}]$ is the initial state distribution,
• ${\mathcal{O}}_i$ is the observation space of agent $i$,
• ${\omega }_i(o;s):\mathcal{S}\to \Delta [{\mathcal{O}}_i]$ is the observation kernel of agent $i$,
• ${\mathcal{R}}_i(s,\mathbf{a}):\mathcal{S}\times {\prod }_j{\mathcal{A}}_j\to \{0,1\}$ is the (stochastic) reward of agent $i$,
• $\gamma \in [0,1)$ is the discount factor, shared by all agents, and
• ${\pi }_i\left(a_{T,i};\{a_{t,i},o_{t,i}{\}}_{t=0}^{T-1}\right):({\mathcal{A}}_i\times {\mathcal{O}}_i{)}^{\ast }\to \Delta [{\mathcal{A}}_i]$ is the policy of agent $i$.

Under a joint policy, agent $i$ seeks to maximize $\mathbb{E}\left[{\sum }_{t\ge 0}{\gamma }^t{\mathcal{R}}_i(s_t,{\mathbf{a}}_t)\right]$. Rewards are per-agent, so the model is general-sum, and the cooperative Dec-POMDP is the special case ${\mathcal{R}}_i\equiv \mathcal{R}$. The $\{0,1\}$ codomain is WLOG: any bounded real-valued reward affinely rescales into $[0,1]$ and is realized as a Bernoulli draw with that mean, preserving expected return up to affine equivalence and hence the optimal-policy set.

Definition (Rollout)

A $T$-step rollout of a PED-POMDP is a tuple

$$\rho =(\{s_t{\}}_{t=0}^T,\{{\mathbf{a}}_t{\}}_{t=0}^{T-1},\{{\mathbf{o}}_t{\}}_{t=0}^{T-1}),$$

where $s_t\in \mathcal{S}$, ${\mathbf{a}}_t=(a_{t,i}{)}_{i\in \mathcal{I}}\in {\prod }_i{\mathcal{A}}_i$, and ${\mathbf{o}}_t=(o_{t,i}{)}_{i\in \mathcal{I}}\in {\prod }_i{\mathcal{O}}_i$: the state sequence, joint actions, and joint observations of a gameplay of exactly $T$ time steps. Under the time convention, $s_{t+1}$ is the post-action state of step $t$ and ${\mathbf{o}}_t$ is emitted from $s_{t+1}$, so the state sequence has $T+1$ entries and ends at $s_T$. Write $h_{t,i}(\rho )=\{(a_{u,i},o_{u,i}){\}}_{u=0}^{t-1}$ for agent $i$‘s history at step $t$ read off $\rho$. Since the PED-POMDP fixes every policy, it induces a probability distribution over $T$-step rollouts,

$$\mathrm{Pr}[\rho ]={\mathcal{T}}_0(s_0)\prod _{t=0}^{T-1}\left[\prod _i{\pi }_i\right(a_{t,i};h_{t,i}(\rho )\left)\right]\cdot \mathcal{T}(s_{t+1};s_t,{\mathbf{a}}_t)\cdot \prod _i{\omega }_i(o_{t,i};s_{t+1}),$$

and $\rho$ carries all information needed to describe the gameplay: in particular every reward is determined, $r_{t,i}={\mathcal{R}}_i(s_t,{\mathbf{a}}_t)$. In this sense a PED-POMDP is a closed stochastic system rather than a game to be solved, and the rollout distribution is its complete behavioral description.

Remark (state can encode metacognitive information)

Consider a PED-POMDP with $\mathcal{I}=\{0,1\}$. For each $i$, instead of one ${\mathcal{O}}_i,{\omega }_i$, give a space $\Sigma$ of observability configurations, a parameterized pair ${\mathcal{O}}_i(\sigma ),{\omega }_i(o;\sigma ,s)$, and a prior $\mathcal{Q}\in \Delta [\Sigma ]$ from which $\sigma$ is drawn once at the start. This lets each agent hold uncertainty over how its peer (or itself) accesses the world, which makes explicit what we may call metacognition, cognition about one’s own or another’s cognitive process. An agent can wonder whether a peer really saw what it claims, whether a news item was fabricated, or whether its own reasoning slipped somewhere.

The base PED-POMDP already captures this implicitly. Replace $\mathcal{S}$ by $\mathcal{S}\times \Sigma$, fold $\mathcal{Q}$ into the initial distribution through ${\mathcal{T}}_0^{\prime }(s,\sigma )={\mathcal{T}}_0(s)\mathcal{Q}(\sigma )$, hold $\sigma$ fixed under the transition through ${\mathcal{T}}^{\prime }((s^{\prime },{\sigma }^{\prime });(s,\sigma ),\mathbf{a})=\mathcal{T}(s^{\prime };s,\mathbf{a})1_{{\sigma }^{\prime }=\sigma }$, take ${\mathcal{O}}_i={⨆}_{\sigma }{\mathcal{O}}_i(\sigma )$, and set ${\omega }_i^{\prime }(o;(s,\sigma ))={\omega }_i(o;\sigma ,s)$. The result is equivalent to the configuration-augmented model with no explicit $\Sigma$ or $\mathcal{Q}$. We take this route from here on, folding metacognitive information into $\mathcal{S}$, which also lets object-level and metacognitive information be handled the same way.

Remark (state can encode policy information)

The same maneuver lets us reintroduce parameterized policies should we need them. Consider an extension that replaces each parameter-free ${\pi }_i$ with a parameter space ${\Theta }_i$ and a parameterized policy ${\pi }_i(\cdot ;{\theta }_i,h):{\Theta }_i\times ({\mathcal{A}}_i\times {\mathcal{O}}_i{)}^{\ast }\to \Delta [{\mathcal{A}}_i]$, with each ${\theta }_i$ drawn from a prior ${\mathcal{P}}_i\in \Delta [{\Theta }_i]$ at the start, independently across $i$. The base PED-POMDP already captures this. Replace $\mathcal{S}$ by $\mathcal{S}\times {\prod }_i{\Theta }_i$, fold the priors into the initial distribution through ${\mathcal{T}}_0^{\prime }((s,\theta ))={\mathcal{T}}_0(s){\prod }_i{\mathcal{P}}_i({\theta }_i)$, and hold $\theta =({\theta }_i{)}_i$ fixed under the transition through ${\mathcal{T}}^{\prime }((s^{\prime },{\theta }^{\prime });(s,\theta ),\mathbf{a})=\mathcal{T}(s^{\prime };s,\mathbf{a})1_{{\theta }^{\prime }=\theta }$. The equivalence further requires a private revelation of ${\theta }_i$ to agent $i$, since agent $i$ needs ${\theta }_i$ to compute its own action distribution: extend ${\mathcal{O}}_i$ to ${\mathcal{O}}_i\times {\Theta }_i$, set ${\omega }_i^{\prime }((o,\tau );(s,\theta ))={\omega }_i(o;s)1_{\tau ={\theta }_i}$, and deliver the ${\theta }_i$ component as a one-time private observation $o_{-1,i}$ to agent $i$ alone, before its first action. The base PED-POMDP’s parameter-free policy then reads ${\theta }_i$ off its augmented history and reproduces ${\pi }_i(\cdot ;{\theta }_i,\cdot )$ of the parametric model. So parameterized policies and priors over them fold into $\mathcal{S}$ alongside object-level and metacognitive information; we keep the base 10-tuple from here on.

Definition (Sibling)

Call two PED-POMDPs $i$-siblings if they agree on every component except possibly ${\pi }_i$. Varying agent $i$‘s behavior while holding the rest of the world fixed means moving within the $i$-sibling family, and definitions that compare agent $i$‘s behavior against alternatives are posed across the family, so that each definition stays internal to a single PED-POMDP.

Coherence Optimization

Definition (Egocentric POMDP)

Fix a PED-POMDP $\mathcal{D}$ and an agent $i$. Write

$${\mathcal{O}}_{-i}=\prod _{j\ne i}{\mathcal{O}}_j,{\mathcal{A}}_{-i}=\prod _{j\ne i}{\mathcal{A}}_j,$$

and let a generic egocentric state be $\mathbf{s}=(s,h_{-i})$, where $h_{-i}\in ({\mathcal{A}}_{-i}\times {\mathcal{O}}_{-i}{)}^{\ast }$ is the others’ joint history of actions and observations. The egocentric POMDP of agent $i$ is

$$\mathcal{E}=(\mathcal{S}\times ({\mathcal{A}}_{-i}\times {\mathcal{O}}_{-i}{)}^{\ast },{\mathcal{A}}_i,{\mathcal{T}}_{-i},{\mathcal{T}}_{0,-i},{\mathcal{O}}_i,{\omega }_i\otimes 1,{\mathcal{R}}_{-i},\gamma ).$$

Agent $i$ treats the world state and the others’ private histories $h_{-i}$ as a single hidden environment. It observes neither $h_{-i}$ nor the others’ observations or actions; the channel ${\omega }_i\otimes 1$ exposes only $s$. The others’ actions belong in the hidden state for the same reason their observations do, namely that each ${\pi }_j$ depends on $j$‘s own past actions, so a belief about $j$‘s next action requires a belief about $j$‘s past actions. Type uncertainty, when present, is folded into $\mathcal{S}$ by the previous remark and so already enters $\mathcal{E}$ through $\mathcal{S}$.

Write ${\pi }_{-i}({\mathbf{a}}_{-i};h_{-i})={\prod }_{j\ne i}{\pi }_j(a_j;h_j)$, with $h_j$ the $j$-th coordinate of $h_{-i}$. For a source state $\mathbf{s}=(s,h_{-i})$ with $h_{-i}=\{(a_{t,-i},o_{t,-i}){\}}_{t=0}^{T-1}$, own action $a_{T,i}$, and a target state of the form ${\mathbf{s}}^{\prime }=(s^{\prime },h_{-i}⌢(a_{T,-i},o_{T,-i}))$,

$${\mathcal{T}}_{-i}({\mathbf{s}}^{\prime };\mathbf{s},a_{T,i})={\pi }_{-i}(a_{T,-i};h_{-i})\cdot \mathcal{T}(s^{\prime };s,(a_{T,i},a_{T,-i}))\cdot \prod _{j\ne i}{\omega }_j(o_{T,j};s^{\prime }).$$

All target states not of this form ($h_{-i}$ not extended by exactly one pair) receive value $0$. The product runs over $j\ne i$, since agent $i$‘s own observation is produced by ${\omega }_i\otimes 1$ rather than by the latent transition. The remaining components are

$$\begin{array}{rl}{\mathcal{T}}_{0,-i}((s,\varnothing ))& ={\mathcal{T}}_0(s),\\ ({\omega }_i\otimes 1)(o_i;(s,h_{-i}))& ={\omega }_i(o_i;s),\\ {\mathcal{R}}_{-i}((s,h_{-i}),a_{T,i})& ={\mathbb{E}}_{{\mathbf{a}}_{T,-i}\sim {\pi }_{-i}(h_{-i})}\left[{\mathcal{R}}_i\right(s,(a_{T,i},{\mathbf{a}}_{T,-i})\left)\right],\end{array}$$

with all other initial-state values $0$. The notation ${\omega }_i\otimes 1$ records that agent $i$‘s observation depends on the world component $s$ alone and is constant in $h_{-i}$. The reward keeps the expectation over ${\mathbf{a}}_{T,-i}$, because the others’ time-$T$ action is realized during the transition and is not yet in the source state. The policy ${\pi }_i$ of $\mathcal{D}$ is itself a policy of $\mathcal{E}$, since both act on agent $i$‘s own action-observation history.

No component of $\mathcal{E}$ involves ${\pi }_i$, so all $i$-siblings share one egocentric POMDP, and with it everything built from $\mathcal{E}$ below. Every definition below relativizes verbatim to a subjective prior $\mathcal{P}\in \Delta [\mathcal{S}]$ substituted for ${\mathcal{T}}_0$ in ${\mathcal{T}}_{0,-i}$, should a misspecified initial belief ever be needed; we do not use this generality.

Definition (Egocentric Belief-State MDP)

We further define the egocentric belief-state MDP

$$\mathcal{B}=(\Delta [\mathcal{S}\times ({\mathcal{A}}_{-i}\times {\mathcal{O}}_{-i}{)}^{\ast }],{\mathcal{A}}_i,{\mathcal{T}}_{-i}^{\mathrm{Bayes}},{\delta }_{{\mathcal{T}}_{0,-i}},{\mathcal{R}}_{-i},\gamma ).$$

For a belief $b$ and action $a_i$, let

$${\bar{b}}_{a_i}({\mathbf{s}}^{\prime })=\sum _{\mathbf{s}}b(\mathbf{s}){\mathcal{T}}_{-i}({\mathbf{s}}^{\prime };\mathbf{s},a_i)$$

be the one-step predictive distribution after $a_i$. For an observation $o_i$ with

$$\mathrm{Pr}[o_i\mid b,a_i]:=\sum _{{\mathbf{s}}^{\prime }}{\bar{b}}_{a_i}({\mathbf{s}}^{\prime })({\omega }_i\otimes 1)(o_i;{\mathbf{s}}^{\prime })>0,$$

let the posterior be

$$b_{a_i,o_i}^{\prime }({\mathbf{s}}^{\prime })=\frac{{\bar{b}}_{a_i}({\mathbf{s}}^{\prime })({\omega }_i\otimes 1)(o_i;{\mathbf{s}}^{\prime })}{\mathrm{Pr}[o_i\mid b,a_i]}.$$

Then

$${\mathcal{T}}_{-i}^{\mathrm{Bayes}}(b^{\prime };b,a_i)=\sum _{o_i:\mathrm{Pr}[o_i\mid b,a_i]>0}\mathrm{Pr}[o_i\mid b,a_i]\cdot 1_{b^{\prime }=b_{a_i,o_i}^{\prime }},{\mathcal{R}}_{-i}(b,a_i)=\sum _{\mathbf{s}}b(\mathbf{s}){\mathcal{R}}_{-i}(\mathbf{s},a_i),$$

and ${\delta }_{{\mathcal{T}}_{0,-i}}(b)=1_{b={\mathcal{T}}_{0,-i}}$. The belief-state reward overloads the state-level ${\mathcal{R}}_{-i}$ by its argument type.

Definition (Belief-State Operator)

We then define the belief-state operator

$$b_i:({\mathcal{A}}_i\times {\mathcal{O}}_i{)}^{\ast }\rightharpoonup \Delta [\mathcal{S}\times ({\mathcal{A}}_{-i}\times {\mathcal{O}}_{-i}{)}^{\ast }],$$

a partial map defined at every history that occurs, meaning it has positive probability under the rollout distribution of $\mathcal{D}$. For an occurring $h=\{a_{t,i},o_{t,i}{\}}_{t=0}^{T-1}$, write $S_T$, $H_{T,i}$, and $H_{T,-i}=(H_{T,j}{)}_{j\ne i}$ for the final state, agent $i$‘s history $h_{T,i}(\rho )$, and the others’ joint history of a random $T$-step rollout $\rho$, and set

$$b_i(h):=\mathrm{Pr}[(S_T,H_{T,-i})=\cdot |H_{T,i}=h],$$

the conditional distribution of the hidden environment given agent $i$‘s own history. Every ${\pi }_i$ factor in $\mathrm{Pr}[\rho ]$ depends on $\rho$ only through $h_{t,i}(\rho )$, which the conditioning fixes, so these factors cancel between numerator and denominator; the conditional is therefore computed by the ${\pi }_i$-free recursion $b_0={\mathcal{T}}_{0,-i}$ and, for each $t$, the one-step update

$$b_{t+1}({\mathbf{s}}_{t+1})=\mathrm{Pr}[{\mathbf{S}}_{t+1}={\mathbf{s}}_{t+1}|{\mathbf{S}}_t\sim b_t,a_{t,i},o_{t,i}]=\frac{({\omega }_i\otimes 1)(o_{t,i};{\mathbf{s}}_{t+1})\sum _{{\mathbf{s}}_t}{\mathcal{T}}_{-i}({\mathbf{s}}_{t+1};{\mathbf{s}}_t,a_{t,i})b_t({\mathbf{s}}_t)}{\sum _{{\mathbf{s}}^{\prime }}({\omega }_i\otimes 1)(o_{t,i};{\mathbf{s}}^{\prime })\sum _{{\mathbf{s}}_t}{\mathcal{T}}_{-i}({\mathbf{s}}^{\prime };{\mathbf{s}}_t,a_{t,i})b_t({\mathbf{s}}_t)},$$

with $b_i(h)=b_T$. Two consequences. First, while the domain of $b_i$ references ${\pi }_i$ through the rollout distribution, its values do not: all $i$-siblings of $\mathcal{D}$ assign the same belief to any history occurring in both, so beliefs are comparable across the family. Second, every $b_i(h)$ is a reachable belief of $\mathcal{B}$, since the recursion conditions only on observations of positive probability, matching the branching of ${\mathcal{T}}_{-i}^{\mathrm{Bayes}}$. The belief $b_i(h)$ is a sufficient statistic of the history $h$ for control in $\mathcal{E}$, which is what makes $\mathcal{B}$ an equivalent fully observed problem.

Definition (Policy Translation)

A stationary policy of the belief-state MDP $\mathcal{B}$ is a map $\mu :\Delta [\mathcal{S}\times ({\mathcal{A}}_{-i}\times {\mathcal{O}}_{-i}{)}^{\ast }]\to \Delta [{\mathcal{A}}_i]$, since $\mathcal{B}$ is fully observed and the belief is its state. Its translation $\tau (\mu )$ is the agent-$i$ policy that filters the history so far into its belief and then applies $\mu$,

$$\tau (\mu )(\cdot ;\{a_{t,i},o_{t,i}{\}}_{t=0}^{T-1})=\mu (\cdot ;b_i(\{a_{t,i},o_{t,i}{\}}_{t=0}^{T-1})).$$

Substituting $\tau (\mu )$ for ${\pi }_i$ yields an $i$-sibling ${\mathcal{D}}_{\mu }$ of $\mathcal{D}$, and the defining equation is read within ${\mathcal{D}}_{\mu }$. This is well-founded because the recursion computing $b_i$ is shared by all $i$-siblings: construct $\tau (\mu )$ forward in time, acting on $b_0$ at the empty history, updating the belief on each realized $(a,o)$ pair, and acting on the updated belief. This defines $\tau (\mu )$ at every history occurring in ${\mathcal{D}}_{\mu }$, which is all that the rollout distribution of ${\mathcal{D}}_{\mu }$ ever consults; elsewhere it may be set arbitrarily. A translated policy depends on the history only through its belief: two occurring histories $h_1\ne h_2$ with $b_i(h_1)=b_i(h_2)$ must receive the same action distribution. A general policy may act differently at $h_1$ and $h_2$, so it need not be the translation of any $\mu$. This is the precise sense in which belief-state policies are a special kind of history policy but not conversely.

Definition (Belief-State Individual Optimality)

A policy ${\pi }^{\ast }$ of $\mathcal{B}$ is belief-state individually optimal (b.i.o.) if it satisfies the Bellman optimality equation of $\mathcal{B}$ under reward ${\mathcal{R}}_{-i}$ and discount $\gamma$. Writing $V^{\ast }$ for the optimal value,

$$V^{\ast }(b)=\underset{a_i\in {\mathcal{A}}_i}{\max }[{\mathcal{R}}_{-i}(b,a_i)+\gamma \sum _{b^{\prime }}{\mathcal{T}}_{-i}^{\mathrm{Bayes}}(b^{\prime };b,a_i)V^{\ast }(b^{\prime })],$$

and ${\pi }^{\ast }$ is b.i.o. when $\mathrm{supp}{\pi }^{\ast }(\cdot ;b)\subseteq \arg {\max }_{a_i}[{\mathcal{R}}_{-i}(b,a_i)+\gamma {\sum }_{b^{\prime }}{\mathcal{T}}_{-i}^{\mathrm{Bayes}}(b^{\prime };b,a_i)V^{\ast }(b^{\prime })]$ at every reachable $b$. Belief-state optimality is a property of ${\pi }^{\ast }$ alone; it involves neither ${\pi }_i$ nor the rollout distribution, so the b.i.o. set is shared by all $i$-siblings.

Definition (Individual Optimality)

Agent $i$ is individually optimal (i.o.) in $\mathcal{D}$ if some b.i.o. policy ${\pi }^{\ast }$ satisfies ${\pi }_i(\cdot ;h)={\pi }^{\ast }(\cdot ;b_i(h))$ at every occurring history $h$, that is, ${\pi }_i$ agrees with the translation $\tau ({\pi }^{\ast })$ wherever agent $i$‘s gameplay actually reaches. Restricting to occurring histories loses nothing: agreement propagates forward from the empty history, so an i.o. ${\pi }_i$ induces the same rollout distribution as the sibling ${\mathcal{D}}_{{\pi }^{\ast }}$ and in particular attains the optimal value. Since the b.i.o. set and the belief values are shared across $i$-siblings, asking in which members of the family agent $i$ is i.o. is well-posed; this is how alternative behaviors of agent $i$ are compared.

Definition (Belief-Groundedness)

For an occurring history $h=\{a_{t,i},o_{t,i}{\}}_{t=0}^T$ of agent $i$, agent $i$‘s belief-groundedness at $h$ in $\mathcal{D}$ is (we assume below that the JS divergence uses base-2 logarithm)

$${\mathcal{G}}_i(h):=1-\underset{{\pi }^{\ast }\text{b.i.o.}}{\inf }\mathrm{JS}[{\pi }_i(\cdot ;h)\Vert {\pi }^{\ast }(\cdot ;b_i(h)\left)\right]\in [0,1],$$

the negated smallest JS gap between agent $i$‘s action distribution at $h$ and an optimal action distribution at the induced belief. Like individual optimality, it compares across $i$-siblings: each sibling is scored at its own occurring histories against the shared benchmark.

If agent $i$ is i.o. in $\mathcal{D}$, then a single b.i.o. witness ${\pi }^{\ast }$ matches ${\pi }_i$ at every occurring history, so ${\mathcal{G}}_i(h)=1$ for all occurring $h$. The converse needs an extra step. From ${\mathcal{G}}_i(h)=1$ at every occurring $h$ one obtains, for each such $h$, some optimal action distribution equal to ${\pi }_i(\cdot ;h)$, and these can come from different optimal policies. To recover i.o. one also needs ${\pi }_i$ to depend on the history only through the belief and to select a single optimal action distribution at each belief, so that one ${\pi }^{\ast }$ works everywhere. Without that, ${\mathcal{G}}_i\equiv 1$ does not give i.o.

Definition (Inferred Coherence)

The $T$-step rollout distributions are projectively consistent — marginalizing the $(T+1)$-step distribution over $(s_{T+1},{\mathbf{a}}_T,{\mathbf{o}}_T)$ recovers the $T$-step one — so they extend uniquely to a distribution over infinite rollouts $\rho =(\{s_t{\}}_{t\ge 0},\{{\mathbf{a}}_t{\}}_{t\ge 0},\{{\mathbf{o}}_t{\}}_{t\ge 0})$; the expectation below is under this distribution. For agents $i,j\in \mathcal{I}$ and an occurring history $h_j=\{a_{t,j},o_{t,j}{\}}_{t=0}^{T-1}$ of agent $j$, the $(j,h_j)$-inferred coherence of agent $i$ is

$${\mathcal{C}}_i(j,h_j):=(1-\gamma )\mathbb{E}[\sum _{t\ge 0}{\gamma }^t{\mathcal{G}}_i(H_{t,i})|H_{T,j}=h_j]\in [0,1],$$

the expected $\gamma$-discounted temporal sum of agent $i$‘s belief-groundedness at its actualized history, conditioned on agent $j$‘s history being $h_j$, with the prefactor $1-\gamma$ normalizing the range to $[0,1]$. Every realization of $H_{t,i}$ compatible with the conditioning occurs, so ${\mathcal{G}}_i(H_{t,i})$ is defined almost surely, and the sum converges since ${\mathcal{G}}_i\le 1$. For $t\le T$ the conditional expectation infers what agent $i$‘s gameplay has already been from agent $j$‘s vantage point; for $t>T$ it forecasts. The case $j=i$ reads as self-assessed coherence.

Remark (empirical methods for optimizing inferred coherence)

arXiv:2601.13566 gives a candidate empirical method for optimizing for inferred coherence. That paper focuses on the case where $j$ is a language model, playing the role of its prior policy, and $i$ is another language model, playing the role of its deterministic policy; we hope that similar methods can be extended to $j$ as a more general learning agent and $i$ as a human. However, the method relies on a pretrained prior that specifies marginal coherence, which may significantly limit its usefulness; looking for better methods is a priority.