Implement an end-to-end toy RLHF loop on a “bandit-LM”: train an RM from synthetic prefe

Super-hard

RLHF & Preference Optimization6 / 7

Chapter 126 / 7

RLHF & Preference Optimization

Implement an end-to-end toy RLHF loop on a “bandit-LM”: train an RM from synthetic prefe

Super-hard

Implement an end-to-end toy RLHF loop on a “bandit-LM”: train an RM from synthetic preferences, optimize the policy with PPO + KL control, and plot gold vs proxy reward to exhibit over-optimization. [OpenAI]

Target interface

softmax(z)

Implement the function/class skeleton in the editor. Any correct approach is accepted.

Hints0 / 2

Reference solution

Your own code stays in the editor.

solution.pyPython

Local draft

import numpy as np

def softmax(z):
    raise NotImplementedError

def bt_fit(n_pairs=500, lr=0.1, steps=2000, noise=1.5):
    raise NotImplementedError

def train(beta, lr=0.2, steps=600):
    raise NotImplementedError

⌘/Ctrl↵to submit

AI review

No review yet

Submit your solution for a verdict, a score out of 100 and concrete feedback. Any correct approach passes.

Chapter 126 / 7

RLHF & Preference Optimization

Implement an end-to-end toy RLHF loop on a “bandit-LM”: train an RM from synthetic prefe

Super-hard

Implement an end-to-end toy RLHF loop on a “bandit-LM”: train an RM from synthetic preferences, optimize the policy with PPO + KL control, and plot gold vs proxy reward to exhibit over-optimization. [OpenAI]

Target interface

softmax(z)

Implement the function/class skeleton in the editor. Any correct approach is accepted.

Hints0 / 2

Reference solution

Your own code stays in the editor.