Inference & Serving5 / 5
Inference & Serving
Implement a continuous-batching scheduler simulator with a paged KV cache that admits/evicts requests and reports throughput and P99.
class RequestImplement the function/class skeleton in the editor. Any correct approach is accepted.
import numpy as np
from collections import deque
class Request:
def __init__(self, rid, arrival, prompt_len, gen_len):
raise NotImplementedError
class PagedScheduler:
def __init__(self, total_pages, page_size, max_batch):
raise NotImplementedError
def _pages_needed(self, seq_len):
raise NotImplementedError
def _try_admit(self):
raise NotImplementedError
def _evict(self, req, t):
raise NotImplementedError
def run(self, arrivals):
raise NotImplementedError⌘/Ctrl↵to submit
No review yet
Submit your solution for a verdict, a score out of 100 and concrete feedback. Any correct approach passes.
Inference & Serving
Implement a continuous-batching scheduler simulator with a paged KV cache that admits/evicts requests and reports throughput and P99.
class RequestImplement the function/class skeleton in the editor. Any correct approach is accepted.