Mixture-of-Experts2 / 4
Mixture-of-Experts
Implement a full sparse MoE FFN with capacity, token dropping, and gate-weighted combination.
class ExpertImplement the function/class skeleton in the editor. Any correct approach is accepted.
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
class Expert(nn.Module):
def __init__(self, d_model, d_ff):
raise NotImplementedError
def forward(self, x):
raise NotImplementedError
class SparseMoE(nn.Module):
def __init__(self, d_model, d_ff, n_experts, k, capacity_factor=1.25):
raise NotImplementedError
def forward(self, x):
raise NotImplementedError⌘/Ctrl↵to submit
No review yet
Submit your solution for a verdict, a score out of 100 and concrete feedback. Any correct approach passes.
Mixture-of-Experts
Implement a full sparse MoE FFN with capacity, token dropping, and gate-weighted combination.
class ExpertImplement the function/class skeleton in the editor. Any correct approach is accepted.