RLHF: Reward Models, PPO & GRPO5 / 7
RLHF: Reward Models, PPO & GRPO
Implement Dr. GRPO (GRPO with the length and std-normalization terms removed) and show, on a toy batch, that it changes the relative weighting vs GRPO. [DeepSeek]
grpo_adv(rewards)Complete the starter code in the editor. Any correct approach is accepted.
import torch
def grpo_adv(rewards):
raise NotImplementedError
def dr_grpo_adv(rewards):
raise NotImplementedError
def grpo_objective(per_tok_pg, mask):
raise NotImplementedError
def dr_grpo_objective(per_tok_pg, mask):
raise NotImplementedError⌘/Ctrl↵to submit
Submit your solution for a score and feedback on correctness, edge cases, and clarity. Any correct approach is accepted.
RLHF: Reward Models, PPO & GRPO
Implement Dr. GRPO (GRPO with the length and std-normalization terms removed) and show, on a toy batch, that it changes the relative weighting vs GRPO. [DeepSeek]
grpo_adv(rewards)Complete the starter code in the editor. Any correct approach is accepted.