Evaluation: Benchmarks, Judges & Uncertainty2 / 4
Evaluation: Benchmarks, Judges & Uncertainty
Implement a pairwise LLM-as-judge harness with position-swap debiasing (run both orders, average).
score_from_verdict(verdict, a_is_first)Implement the function/class skeleton in the editor. Any correct approach is accepted.
def score_from_verdict(verdict, a_is_first):
raise NotImplementedError
def pairwise_judge(prompt, resp_a, resp_b, judge_fn):
raise NotImplementedError
def win_rate(prompts, resp_as, resp_bs, judge_fn):
raise NotImplementedError⌘/Ctrl↵to submit
No review yet
Submit your solution for a verdict, a score out of 100 and concrete feedback. Any correct approach passes.
Evaluation: Benchmarks, Judges & Uncertainty
Implement a pairwise LLM-as-judge harness with position-swap debiasing (run both orders, average).
score_from_verdict(verdict, a_is_first)Implement the function/class skeleton in the editor. Any correct approach is accepted.