- Coding problems
- 17
- Theory & math
- 197
- Chapters
- 3
/
Difficulty
- 01.01Implement a function computing total per-GPU memory for a given model/parallelism confEasyEasy
- 01.02Implement toy data-parallel SGD with manual gradient all-reduce (torchMediumMedium
- 01.03Implement a 1F1B pipeline-schedule simulator reporting the bubble fraction for given (stagHardHard
- 01.04Implement tensor-parallel linear layers (column-parallel then row-parallel) with correct fSuper-hardSuper-hard
- 01.05Theory questionsWarm-upWarm-up
- 02.01Implement temperature + top- + top- sampling from a logits vector in numpyEasyEasy
- 02.02Implement int8 symmetric per-channel weight quantization and dequantization for a linear lMediumMedium
- 02.03Implement nucleus (top-) sampling with correct renormalization and edge casesMediumMedium
- 02.04Implement speculative decoding: draft proposes tokens, target verifies in one pass, acHardHard
- 02.05Implement a continuous-batching scheduler simulator with a paged KV cache that admits/evicSuper-hardSuper-hard
- 03.01Write shape assertions for the tensors in an attention forward passEasyEasy
- 03.02Write a check that verifies labels are correctly shifted by one relative to inputsEasyEasy
- 03.03Write unit tests for causal masking (no token may attend to the future)MediumMedium
- 03.04Write a cached-vs-full-forward equivalence test for incremental decodingMediumMedium
- 03.05Debug a broken GPT training notebook containing NaNs, a mask bug, and shifted labelsHardHard