2 / 8 Media, Streaming & Storage

Upload And Transcode

Design the upload and processing pipeline behind a YouTube-style video platform: a creator uploads a file from a phone or a browser, and minutes later it is playable worldwide at multiple qualities.

Requirements

Functional:

  • Resumable uploads of files up to 20 GB that survive a dropped connection, an app restart, or a switch from wifi to cellular.
  • Transcode every source into an adaptive bitrate ladder (144p through 2160p) plus an audio-only track.
  • Generate thumbnail candidates and a storyboard sprite sheet for scrubbing.
  • Run a moderation check before a video can become publicly playable.
  • The creator sees upload and processing progress, then publishes now or schedules.

Non-functional:

  • p95 from upload complete to first playable rendition under 3 minutes for a 10 minute source, full ladder under 15 minutes.
  • No duplicate publishes and no half-published videos, ever.
  • Processing and storage cost per uploaded minute is a first class constraint, not an afterthought.

Scale

  • 40M DAU, 500k uploads per day, peak hour runs 3x the daily average rate.
  • Median source 600 MB, p95 2 GB, hard cap 20 GB. Average duration 8 minutes.
  • About 300 TB of source bytes ingested per day.
  • Playback traffic is roughly 200x upload traffic and is served from the edge.
  • Upload status polling peaks around 20k QPS.

Out of scope

Player internals, the recommendation system, comments, live streaming, monetization, and the DRM license server.