Diffusion & Non-Autoregressive Language Models3 / 4
Diffusion & Non-Autoregressive Language Models
Implement a minimal masked-diffusion training loss (sample a mask rate, mask tokens, cross-entropy on masked positions) for a tiny transformer.
class TinyDiffusionLMImplement the function/class skeleton in the editor. Any correct approach is accepted.
import torch
import torch.nn as nn
import torch.nn.functional as F
class TinyDiffusionLM(nn.Module):
def __init__(self, vocab, d=128, n_layers=4, n_heads=4, max_len=64):
raise NotImplementedError
def forward(self, x):
raise NotImplementedError
def diffusion_loss(model, x0, mask_id, eps=0.001):
raise NotImplementedError⌘/Ctrl↵to submit
No review yet
Submit your solution for a verdict, a score out of 100 and concrete feedback. Any correct approach passes.
Diffusion & Non-Autoregressive Language Models
Implement a minimal masked-diffusion training loss (sample a mask rate, mask tokens, cross-entropy on masked positions) for a tiny transformer.
class TinyDiffusionLMImplement the function/class skeleton in the editor. Any correct approach is accepted.