Tokenization and Embeddings4 / 4
Tokenization and Embeddings
Implement a byte-level BPE end-to-end (train encode decode) over arbitrary UTF-8 bytes.
class ByteBPEComplete the starter code in the editor. Any correct approach is accepted.
from collections import Counter
class ByteBPE:
def __init__(self):
raise NotImplementedError
def train(self, text, vocab_size):
raise NotImplementedError
@staticmethod
def _merge(ids, a, b):
raise NotImplementedError
def encode(self, text):
raise NotImplementedError
def decode(self, tokens):
raise NotImplementedError⌘/Ctrl↵to submit
Submit your solution for a score and feedback on correctness, edge cases, and clarity. Any correct approach is accepted.
Tokenization and Embeddings
Implement a byte-level BPE end-to-end (train encode decode) over arbitrary UTF-8 bytes.
class ByteBPEComplete the starter code in the editor. Any correct approach is accepted.