Implement an n-gram/embedding contamination detector that flags eval items overlapping a t

Super-hard

Evaluation: Benchmarks, Judges & Uncertainty4 / 4

Chapter 154 / 4

Evaluation: Benchmarks, Judges & Uncertainty

Implement an n-gram/embedding contamination detector that flags eval items overlapping a t

Super-hard

Implement an n-gram/embedding contamination detector that flags eval items overlapping a training corpus and reports a contamination-adjusted score.

Target interface

shingles(text, k=8)

Implement the function/class skeleton in the editor. Any correct approach is accepted.

Hints0 / 2

Reference solution

Your own code stays in the editor.

solution.pyPython

Local draft

import numpy as np

def shingles(text, k=8):
    raise NotImplementedError

def build_training_index(train_docs, k=8):
    raise NotImplementedError

def ngram_overlap(item_text, train_index, k=8):
    raise NotImplementedError

def embed_max_sim(item_vec, train_vecs):
    raise NotImplementedError

def contamination_report(eval_items, correct, train_docs, k=8, ngram_thr=0.5, item_vecs=None, train_vecs=None, emb_thr=0.95):
    raise NotImplementedError

⌘/Ctrl↵to submit

AI review

No review yet

Submit your solution for a verdict, a score out of 100 and concrete feedback. Any correct approach passes.

Chapter 154 / 4

Evaluation: Benchmarks, Judges & Uncertainty

Implement an n-gram/embedding contamination detector that flags eval items overlapping a t

Super-hard

Implement an n-gram/embedding contamination detector that flags eval items overlapping a training corpus and reports a contamination-adjusted score.

Target interface

shingles(text, k=8)

Implement the function/class skeleton in the editor. Any correct approach is accepted.

Hints0 / 2

Reference solution

Your own code stays in the editor.