Prefill, Decode, and KV Cache Memory5 / 5
Prefill, Decode, and KV Cache Memory
Implement a continuous-batching scheduler simulator with a paged KV cache that admits/evicts requests and reports throughput and P99.
class RequestComplete the starter code in the editor. Any correct approach is accepted.
import numpy as np
from collections import deque
class Request:
def __init__(self, rid, arrival, prompt_len, gen_len):
raise NotImplementedError
class PagedScheduler:
def __init__(self, total_pages, page_size, max_batch):
raise NotImplementedError
def _pages_needed(self, seq_len):
raise NotImplementedError
def _try_admit(self):
raise NotImplementedError
def _evict(self, req, t):
raise NotImplementedError
def run(self, arrivals):
raise NotImplementedError⌘/Ctrl↵to submit
Submit your solution for a score and feedback on correctness, edge cases, and clarity. Any correct approach is accepted.
Prefill, Decode, and KV Cache Memory
Implement a continuous-batching scheduler simulator with a paged KV cache that admits/evicts requests and reports throughput and P99.
class RequestComplete the starter code in the editor. Any correct approach is accepted.