Reef Infra's OpenClaw-RL Turns User Corrections Into Training Data — Session Matching Is the Catch

Reef Infra's OpenClaw-RL recipe treats the next user message as a training signal — but only after solving the hard part: figuring out which earlier turn the correction actually refers to. A message like "No, keep the steps, just stop putting everything in bullet points" contains real feedback, yet leaves the system to infer what was rejected, what should remain, and which reply is being corrected.

How the pipeline works

• Reconstructs conversations from recorded inference traffic.
• Sends a completed turn plus its following state to a separate PRM judge.
• Batches the judged turns for training.

Training uses two terms. Acceptance can produce a hindsight hint consumed by the on-policy distillation term; the other term is a PPO-style update on the sampled tokens. Both are parameter updates to a trainable policy — this is not a memory write, it's a weight change.

Session correlation is the deployment detail that bites

The harness should attach a stable x-reef-tag-session value to calls in the same conversation. Without it, Reef Infra falls back to matching the transcripts clients resend — and compaction or a sliding window can break that match. An unmatched turn never reaches the judge and never becomes training data, so traffic that looks healthy on your dashboards can produce zero learning signal.

The included experiment, and what it does not prove

The repo ships a small adaptation test: a simulated student rejects homework answers containing bold text or lists. In the recorded run, the agent reaches three consecutive first-reply passes at session 14. That's one run over the first 36 sessions — not evidence that arbitrary user preferences converge at that rate.

What you need to run it

• A trainable model
• A GPU stack
• The PRM judge

The interesting part is that feedback inference, session matching, and the actual weight update are separable components you can inspect independently. Worth noting for comparison: saving the same complaint in a Markdown memory file exercises a completely different mechanism — no judge, no policy update.