A from-scratch primer on REINFORCE (the basic policy-gradient RL algorithm): what it is, how it works, why it works, how it’s implemented — then how we adapt it to train the C3 instructor. Companion to c3-inloop-training-spec-july3.
1. The problem RL solves
You have a policy π_θ(a | s) — a distribution over actions a given a state s, parameterized by θ (e.g. a neural net’s weights). You take actions, the world gives back a scalar reward R. You want to change θ to maximize expected reward J(θ) = E_{a∼π_θ}[R]. Unlike supervised learning, you’re not told the right action — only how good the action you took was. So you can’t just do gradient descent toward a label; you have to infer which actions are good from the rewards they earned.
2. The core idea — “do more of what worked”
Sample an action from the policy, see its reward, then nudge θ to make that action more likely if the reward was high, less likely if low. Repeat over many samples; the policy drifts toward high-reward behavior. That’s the whole intuition. The math below just makes “nudge proportionally to reward” precise and unbiased.
3. The gradient (the log-derivative / “score function” trick)
We want ∇_θ J = ∇_θ E_{a∼π_θ}[R]. The catch: the expectation is over the distribution we’re differentiating. The identity that unlocks it:
∇_θ E_{a∼π_θ}[R(a)] = E_{a∼π_θ}[ R(a) · ∇_θ log π_θ(a) ]
(proof: ∇E[R] = ∫ R ∇π = ∫ R π (∇π/π) = ∫ R π ∇log π = E[R ∇log π], using ∇log π = ∇π/π.)
So the gradient is an average of R · ∇log π(a) over sampled actions — something we can estimate by just sampling actions and computing ∇log π (which autodiff gives us for free). That estimator is REINFORCE:
θ ← θ + α · R · ∇_θ log π_θ(a) # for each sampled action a
∇log π(a) points in the direction that makes a more likely; multiplying by R scales it by how good a was (and flips it when R<0). “Do more of what worked.”
4. Why it works
The estimator is unbiased: its expectation equals the true ∇J. So averaged over enough samples, following it is genuine gradient ascent on expected reward — no model of the environment needed (it’s model-free), and it works even when the reward is non-differentiable (a black box), because we never differentiate R, only log π.
5. Variance is the enemy → baselines
The estimator is unbiased but high-variance (one noisy reward per sample). Key fix: subtract a baseline b that doesn’t depend on the action:
∇J = E[ (R − b) · ∇log π(a) ] # still unbiased for any action-independent b
Using b = E[R] (the average reward) means you reinforce actions relative to typical — better-than-average actions get pushed up, worse-than-average pushed down, and an action exactly at the mean gets ~zero update. (R − b) is the advantage. (This is also why reward variation matters: if every action earns the same R, then R − b ≈ 0 for all → no signal. See the sparse-signal note in the C3 spec.) Further variance control: normalize advantages, clip them, average over a batch.
6. From policy gradient to a training loss
For a language-model policy, the “action” is a token sequence (a completion). log π(a | s) = Σ_t log π(token_t | s, token_{<t}) — the summed per-token log-probs. In code you maximize (R−b) · logπ, i.e. minimize the loss:
loss = − (R − b) · mean_t log π_θ(token_t | context, tokens_<t)
Autodiff backprops this onto the model weights. That’s it — REINFORCE on text is “compute the completion’s log-prob, scale by advantage, backprop.” (PPO adds a clipped-ratio objective + old-policy reuse for stability; REINFORCE is the un-clipped, on-policy special case.)
7. The usual guards
- KL-to-base
− λ_KL · KL(π_θ ‖ π_base)— keeps the policy from wandering off into gibberish while chasing reward (a coherence leash). - Entropy bonus — keeps exploring, prevents premature collapse to one output.
- Advantage clipping / PPO-clip — caps how far one step can move the policy (stability).
8. In our context — the C3 in-loop instructor
Mapping the pieces onto c3-inloop-training-spec-july3:
| REINFORCE | C3 in-loop |
|---|---|
policy π_θ | the Llama-8B instructor |
state s | the dialogue history so far |
action a | the instructor’s next utterance (token sequence) |
reward R | a function of the human’s (D’s) belief trajectory — hinge-martingale (de-entrench) + anti-mute (λ_C) + optionally accuracy (λ_B) |
∇log π(a|s) | gradient on the instructor’s utterance tokens only (history masked) |
baseline b | batch-mean reward |
| KL-to-base | keep the instructor a coherent assistant |
So each training step: the instructor speaks, D responds, a judge reads D’s belief, we score the episode, and we push the instructor’s weights to raise the probability of utterances that (in that conversation) de-entrenched D, and lower those that entrenched or froze it — P(utterance | context) reweighted by outcome. The reward comes from a different agent (D) than the one we’re training (the instructor) and flows through a black box (D’s own reasoning) — which is exactly why it’s model-free REINFORCE and why credit-assignment + variance are the central challenges (see the spec’s §4–§5 + Discussion).