Hard

Implement Viterbi segmentation for a unigram-LM tokenizer given token log-probs

Tokenization & Embeddings · Problem 3 of 4

Chapter 02Tokenization & Embeddings

Implement Viterbi segmentation for a unigram-LM tokenizer given token log-probs

HardProblem 3 / 4

Implement Viterbi segmentation for a unigram-LM tokenizer given token log-probs.

Implement the function/class skeleton in the editor. Any correct approach is accepted.

Hints