Reef Infra's OpenClaw-RL Turns User Corrections Into Training Data — Session Matching Is the Catch
Reef Infra's OpenClaw-RL recipe treats the next user message as a training signal — but only after solving the hard part: figuring out which earlier turn the correction actually refers to. A message like "No, keep the steps, just stop putting everything in bullet points" contains real feedback, yet leaves the system to infer what was rejected, what should remain, and which reply is being corrected.
How the pipeline works
- Reconstructs conversations from recorded inference traffic.
- Sends a completed turn plus its following state to a separate PRM judge.
- Batches the judged turns for training.
Training uses two terms. Acceptance can produce a hindsight hint consumed by the on-policy distillation term; the other term is a PPO-style update on the sampled tokens. Both are parameter updates to a trainable policy — this is not a memory write, it's a weight change.
Session correlation is the deployment detail that bites
The harness should attach a stable x-reef-tag-session value to calls in the same conversation. Without it, Reef Infra falls back to matching the transcripts clients resend — and compaction or a sliding window can break that match. An unmatched turn never reaches the judge and never becomes training data, so traffic that looks healthy on your dashboards can produce zero learning signal.
The included experiment, and what it does not prove
The repo ships a small adaptation test: a simulated student rejects homework answers containing bold text or lists. In the recorded run, the agent reaches three consecutive first-reply passes at session 14. That's one run over the first 36 sessions — not evidence that arbitrary user preferences converge at that rate.
What you need to run it
- A trainable model
- A GPU stack
- The PRM judge
The interesting part is that feedback inference, session matching, and the actual weight update are separable components you can inspect independently. Worth noting for comparison: saving the same complaint in a Markdown memory file exercises a completely different mechanism — no judge, no policy update.
📖 Read the full source: r/clawdbot
👀 See Also

Reddit user experiments with failure-learning coding agents to break retry loops
A developer on r/LocalLLaMA describes experimenting with coding agents that learn from failures by storing simplified root causes and matching fixes, reducing repetitive error loops.
Omnara: Open-Source Durable Agent Harness with Postgres Event Log
Omnara is an open-source, self-hostable alternative to Claude Managed Agents. It runs agent loops on stateless workers, stores state in Postgres, and attaches any macOS or Linux machine via an outbound daemon.

Yavio: Open-Source Product Analytics SDK for MCP Apps
Yavio is an open-source product analytics SDK for MCP and MCP Apps that automatically captures tool calls, errors, and resource reads with one function call. The MIT-licensed project provides a dashboard with per-tool breakdowns, funnels, retention, and error tracking.

Code-Graph-MCP: Open Source MCP Server Reduces Claude Code Token Usage by 40-60%
code-graph-mcp is an MCP server that indexes codebases into an AST knowledge graph, replacing multiple grep/read calls with single structured queries. The developer reports 40-60% total session token savings and 80% fewer tool calls per navigation task.