Tokenization & Embeddings4 / 4
Tokenization & Embeddings
Implement a byte-level BPE end-to-end (train encode decode) over arbitrary UTF-8 bytes.
class ByteBPEImplement the function/class skeleton in the editor. Any correct approach is accepted.
from collections import Counter
class ByteBPE:
def __init__(self):
raise NotImplementedError
def train(self, text, vocab_size):
raise NotImplementedError
@staticmethod
def _merge(ids, a, b):
raise NotImplementedError
def encode(self, text):
raise NotImplementedError
def decode(self, tokens):
raise NotImplementedError⌘/Ctrl↵to submit
No review yet
Submit your solution for a verdict, a score out of 100 and concrete feedback. Any correct approach passes.
Tokenization & Embeddings
Implement a byte-level BPE end-to-end (train encode decode) over arbitrary UTF-8 bytes.
class ByteBPEImplement the function/class skeleton in the editor. Any correct approach is accepted.