Mixture-of-Experts4 / 4
Mixture-of-Experts
Implement expert-parallel dispatch/combine with a simulated all-to-all and verify outputs match a dense reference for the same routing.
class EP_MoEImplement the function/class skeleton in the editor. Any correct approach is accepted.
import torch
import torch.nn as nn
import torch.nn.functional as F
class EP_MoE(nn.Module):
def __init__(self, d, d_ff, E, k, world_size):
raise NotImplementedError
def route(self, x):
raise NotImplementedError
def dense_reference(self, x, gates, idx):
raise NotImplementedError
def forward(self, x):
raise NotImplementedError
def verify():
raise NotImplementedError⌘/Ctrl↵to submit
No review yet
Submit your solution for a verdict, a score out of 100 and concrete feedback. Any correct approach passes.
Mixture-of-Experts
Implement expert-parallel dispatch/combine with a simulated all-to-all and verify outputs match a dense reference for the same routing.
class EP_MoEImplement the function/class skeleton in the editor. Any correct approach is accepted.