Cluster Infrastructure: 5D Parallelism, Scheduling, and Checkpoints2 / 4
Cluster Infrastructure: 5D Parallelism, Scheduling, and Checkpoints
Implement toy data-parallel SGD with manual gradient all-reduce (torch.distributed or a simulated ring).
ring_all_reduce(buffers)Complete the starter code in the editor. Any correct approach is accepted.
import numpy as np
def ring_all_reduce(buffers):
raise NotImplementedError
def dp_sgd_step(weights, grads, lr, N):
raise NotImplementedError⌘/Ctrl↵to submit
Submit your solution for a score and feedback on correctness, edge cases, and clarity. Any correct approach is accepted.
Cluster Infrastructure: 5D Parallelism, Scheduling, and Checkpoints
Implement toy data-parallel SGD with manual gradient all-reduce (torch.distributed or a simulated ring).
ring_all_reduce(buffers)Complete the starter code in the editor. Any correct approach is accepted.