- Coding problems
- 69
- Theory & math
- 980
- Chapters
- 22
/
Difficulty
- 01.01Implement scaled dot-product attention with a causal mask in numpyEasyEasy
- 01.02Implement RMSNorm from scratch in PyTorchEasyEasy
- 01.03Implement multi-head attention from scratch in PyTorch inclMediumMedium
- 01.04Implement RoPE applied to a [batch, heads, seq, head_dim] tensorMediumMedium
- 01.05Implement grouped-query attention with configurable KV heads plus a KV cache for incrementHardHard
- 02.01Given a merge list, implement a BPE tokenizer for a stringEasyEasy
- 02.02Implement BPE training (learn merges) from a corpus in pure PythonMediumMedium
- 02.03Implement Viterbi segmentation for a unigram-LM tokenizer given token log-probsHardHard
- 02.04Implement a byte-level BPE end-to-end (train encode decode) over arbitrary UTF-8 bSuper-hardSuper-hard
- 02.05Theory questionsWarm-upWarm-up
- 03.01Implement sliding-window causal attention in PyTorchEasyEasy
- 03.02Implement a KV cache + incremental single-token decode loop for a small transformerMediumMedium
- 03.03Implement grouped-query attention (GQA) in PyTorch by repeating/broadcasting KV heads acroMediumMedium
- 03.04Implement a blocked/tiled attention forward pass (FlashAttention-style) with running max/sHardHard
- 03.05Implement YaRN RoPE scaling (frequency grouping + attention-logit scaling) and demonstrateSuper-hardSuper-hard
- 05.01Implement next-token cross-entropy for a batch of logits/targets in numpy with padding masEasyEasy
- 05.02Given Chinchilla coefficients, return compute-optimal and for a budget (numeriMediumMedium
- 05.03Fit to a synthetic grid via least HardHard
- 05.04Implement an IsoFLOP analysis: from loss curves at several fixed-FLOP budgets, extract theSuper-hardSuper-hard
- 05.05Theory questionsWarm-upWarm-up
- 07.01Implement AdamW from scratch in numpy for one parameter tensorEasyEasy
- 07.02Implement a cosine LR schedule with linear warmup as a callableMediumMedium
- 07.03Implement global gradient-norm clipping over a list of tensorsMediumMedium
- 07.04Implement the Muon step (momentum + Newton–Schulz orthogonalization) for 2D params, falliHardHard
- 07.05Implement a mixed-precision loop (bf16 compute, fp32 master weights) with loss scaling on Super-hardSuper-hard
- 08.01Implement top- routing (softmax top- renormalized gates) in PyTorchEasyEasy
- 08.02Implement a full sparse MoE FFN with capacity, token dropping, and gate-weighted combinatiMediumMedium
- 08.03Implement the load-balancing aux loss and a training step demonstrating it equalizes experHardHard
- 08.04Implement expert-parallel dispatch/combine with a simulated all-to-all and verify outputs Super-hardSuper-hard
- 08.05Theory questionsWarm-upWarm-up
- 11.01Implement prompt-token loss masking given (prompt_len, total_len) per sampleEasyEasy
- 11.02Implement a LoRA-wrapped linear layer (frozen trainable scaled by $MediumMedium
- 11.03Implement sequence packing with a block-diagonal attention mask so packed samples can't atHardHard
- 11.04Implement a synthetic-data pipeline: generate candidate examples, validate them with a rulHardHard
- 11.05Implement QLoRA-style NF4 4-bit quantization of a weight matrix plus a LoRA adapter, verifSuper-hardSuper-hard
- 12.01Implement the Bradley–Terry pairwise reward-model loss in PyTorchEasyEasy
- 12.02Implement GAE (the backward recursion) given per-token rewards and value estimatesMediumMedium
- 12.03Implement the GRPO group-normalized advantage and the clipped token-level objectiveMediumMedium
- 12.04Implement a minimal PPO update for LLMs: ratio, clipped surrogate, value loss, per-token KHardHard
- 12.05Implement DrHardHard
- 13.01Implement the DPO loss given policy/reference logprobs for chosen/rejected and EasyEasy
- 13.02Implement SimPO (length-normalized, reference-free) and KTO losses and unit-test on toy daMediumMedium
- 13.03Implement best-of- selection given a reward/verifier over sampled completionsMediumMedium
- 13.04Implement on-policy distillation: sample from the student, score tokens under a (toy) teacHardHard
- 13.05Implement iterative/online DPO: generate on-policy pairs, label with a toy preference funcSuper-hardSuper-hard
- 14.01Implement self-consistency: sample CoTs, extract answers, return the majority voteEasyEasy
- 14.02Implement best-of- selection given a reward/verifier over sampled completionsMediumMedium
- 14.03Implement beam-search-over-reasoning-steps that expands/prunes partial CoTs using a PRM scHardHard
- 14.04Implement a rule-based-reward GRPO loop on a toy arithmetic task rewarding a correct boxedSuper-hardSuper-hard
- 14.05Theory questionsWarm-upWarm-up
- 15.01Implement ECE given arrays of predicted confidences and correctnessEasyEasy
- 15.02Implement a pairwise LLM-as-judge harness with position-swap debiasing (run both orders, aMediumMedium
- 15.03Implement a bootstrap confidence interval for win-rate from paired preference judgmentsHardHard
- 15.04Implement an n-gram/embedding contamination detector that flags eval items overlapping a tSuper-hardSuper-hard
- 15.05Theory questionsWarm-upWarm-up
- 19.01Validate a tool-call object against a JSON schema (required fields and types)EasyEasy
- 19.02Compute an agent success-rate metric over a list of trajectoriesEasyEasy
- 19.03Implement a tool-calling loop with retries, timeouts, and error handlingMediumMedium
- 19.04Implement a code-evaluation harness that runs unit tests against generated solutions and rMediumMedium
- 19.05Implement a preference-dataset builder from accepted/rejected suggestions, including a posHardHard
- 21.01Implement patch embedding (conv or unfold linear) converting an image tensor to patch EasyEasy
- 21.02Implement a projection adapter and interleave imagetext embeddings into one sequence wiMediumMedium
- 21.03Implement cross-attention adapter layers letting text tokens attend to frozen vision featuHardHard
- 21.04Theory questionsWarm-upWarm-up
- 21.05Theory questionsEasyEasy
- 22.01Implement the forward masking process for a masked diffusion LM (mask a fraction of toEasyEasy
- 22.02Implement a single reverse-denoising step: predict all masked tokens, keep the most confidMediumMedium
- 22.03Implement a minimal masked-diffusion training loss (sample a mask rate, mask tokens, crossHardHard
- 22.04Implement a small end-to-end masked diffusion LM sampler with confidence-based remasking aSuper-hardSuper-hard
- 22.05Theory questionsWarm-upWarm-up