Design a GPU Scheduler for Video Generation

Rate LimitingQueues OpenAI

Design the job scheduler for a text-to-video product. Users submit a prompt, a GPU job renders a clip over a few minutes, and the user gets the video when it is done. GPUs are scarce, so the scheduler decides who runs next.

Functional:

  • Submit a generation job and get a job id; see its place in line and progress; download the result.
  • Paid plans get priority; free users still get served.
  • One customer submitting 1,000 jobs must not lock everyone else out.
  • Cancel a queued or running job.

Non-functional:

  • GPUs stay busy: no GPU idles while there is work it can run.
  • A GPU node crash loses at most the progress of the jobs on it, and those jobs restart automatically.
  • Users see an honest estimated wait time.
  • 2,000 GPUs in 3 clusters. A job takes 1 to 8 GPUs for 2 to 10 minutes.
  • 400,000 jobs a day; at peak, 30,000 jobs waiting.
  • Results are video files of 20 to 200 MB.

The video model, safety review of prompts and outputs, and billing.

System design
Loading editor...