Implement an end-to-end toy RLHF loop on a “bandit-LM”: train an RM from synthetic prefe

Super-hard

RLHF: Reward Models, PPO & GRPO6 / 7

Chapter 126 / 7

RLHF: Reward Models, PPO & GRPO

Implement an end-to-end toy RLHF loop on a “bandit-LM”: train an RM from synthetic prefe

Super-hard

Implement an end-to-end toy RLHF loop on a “bandit-LM”: train an RM from synthetic preferences, optimize the policy with PPO + KL control, and plot gold vs proxy reward to exhibit over-optimization. [OpenAI]

Target interface

softmax(z)

Complete the starter code in the editor. Any correct approach is accepted.

Hints0 / 2

Reference solution

Your own code stays in the editor.

solution.pyPython

Local draft

import numpy as np

def softmax(z):
    raise NotImplementedError

def bt_fit(n_pairs=500, lr=0.1, steps=2000, noise=1.5):
    raise NotImplementedError

def train(beta, lr=0.2, steps=600):
    raise NotImplementedError

Local draft

⌘/Ctrl↵to submit

AI review

Ready when you are

Submit your solution for a score and feedback on correctness, edge cases, and clarity. Any correct approach is accepted.

Chapter 126 / 7

RLHF: Reward Models, PPO & GRPO

Implement an end-to-end toy RLHF loop on a “bandit-LM”: train an RM from synthetic prefe

Super-hard

Target interface

softmax(z)

Complete the starter code in the editor. Any correct approach is accepted.

Hints0 / 2

Reference solution

Your own code stays in the editor.