Implement iterative/online DPO: generate on-policy pairs, label with a toy preference func

Super-hard

Direct Alignment: DPO, Rejection Sampling & Distillation5 / 5

Chapter 135 / 5

Direct Alignment: DPO, Rejection Sampling & Distillation

Implement iterative/online DPO: generate on-policy pairs, label with a toy preference func

Super-hard

Implement iterative/online DPO: generate on-policy pairs, label with a toy preference function, and run successive DPO rounds; show the win-rate trend.

Target interface

seq_logp(model, toks)

Implement the function/class skeleton in the editor. Any correct approach is accepted.

Hints0 / 2

Reference solution

Your own code stays in the editor.

solution.pyPython

Local draft

import torch
import torch.nn as nn
import torch.nn.functional as F

def seq_logp(model, toks):
    raise NotImplementedError

@torch.no_grad()
def sample(model, B):
    raise NotImplementedError

def oracle_pref(a, b):
    raise NotImplementedError

⌘/Ctrl↵to submit

AI review

No review yet

Submit your solution for a verdict, a score out of 100 and concrete feedback. Any correct approach passes.

Chapter 135 / 5

Direct Alignment: DPO, Rejection Sampling & Distillation

Implement iterative/online DPO: generate on-policy pairs, label with a toy preference func

Super-hard

Implement iterative/online DPO: generate on-policy pairs, label with a toy preference function, and run successive DPO rounds; show the win-rate trend.

Target interface

seq_logp(model, toks)

Implement the function/class skeleton in the editor. Any correct approach is accepted.

Hints0 / 2

Reference solution

Your own code stays in the editor.