Designing YouTube

Absolutely — here’s a detailed Markdown-formatted note on YouTube System Design, written in the same structured, easy-to-study format as before.

🎥 YouTube System Design — Notes

🧭 1. Problem Statement

Design a scalable video streaming platform like YouTube.

Functional Requirements

Users can upload, watch, like, comment, and share videos.
Users can subscribe to channels.
Videos can be searched and recommended.
Support live streaming and video playback on multiple devices.

Non-Functional Requirements

High availability — videos must always be accessible.
Low latency — fast load and play times.
Scalability — millions of concurrent viewers and uploads.
Reliability — no data loss, even on server failure.

⚙️ 2. High-Level Architecture Overview

Client (Web / Mobile)
        │
        ▼
   Load Balancer
        │
        ▼
   Application Servers (API Gateway)
        │
 ┌──────┴────────────────────────────┐
 │                                   │
 ▼                                   ▼
Video Upload Service            Video Streaming Service
 │                                   │
 ▼                                   ▼
Video Processing Pipeline        CDN (Content Delivery Network)
 │                                   │
 ▼                                   ▼
Object Storage (e.g., S3)        Video Player Clients
 │
 ▼
Metadata DB (SQL / NoSQL)

🧩 3. Core Components

Component	Description
API Gateway / Load Balancer	Routes requests (upload, view, comment) to backend services.
Upload Service	Handles user uploads, validates file formats, and stores raw videos temporarily.
Transcoding Service	Converts raw videos into multiple resolutions/bitrates for adaptive streaming.
Storage Service	Stores videos in distributed object storage (e.g., AWS S3, GCP Storage).
CDN	Delivers video content from edge servers closest to users.
Metadata Service	Manages video titles, tags, descriptions, likes, views, etc.
Recommendation System	Suggests videos based on history, trends, or user behavior.
Notification Service	Handles subscriptions and real-time notifications.
Search Service	Indexes and retrieves videos using keywords.

📦 4. Video Upload Flow

User uploads a video file to the platform.
Video passes through upload validation (format, size, limits).
The file is stored temporarily (e.g., in an object store like S3).
The transcoding service is triggered via a message queue (Kafka/SQS).
Transcoder converts video into multiple formats (240p → 1080p → 4K).
Each processed file is stored in permanent storage.
The metadata database is updated with file info (title, duration, links).
Video is made available for streaming via CDN.

🎞️ 5. Video Streaming Flow

User requests to play a video.
The client fetches metadata (video URL, resolution options).
Video chunks are delivered via CDN using HTTP adaptive streaming (HLS/DASH).
Client selects the best resolution dynamically based on network speed.
Playback starts quickly since CDN serves the nearest video copy.

🧰 6. Technologies Used

Area	Technologies
Frontend	React.js, Next.js, Video.js player
Backend	Node.js, Nest.js, Express
Storage	AWS S3 / Google Cloud Storage
Transcoding	FFmpeg, AWS Elastic Transcoder
Database	PostgreSQL (metadata), Redis (caching), Elasticsearch (search)
Queue System	Kafka / RabbitMQ for async video processing
CDN	CloudFront / Akamai / Cloudflare
Monitoring	Prometheus, Grafana, ELK Stack

🧱 7. Database Design

Video Metadata Table (SQL)

Field	Type	Description
`video_id`	UUID	Unique video identifier
`user_id`	UUID	Owner/creator
`title`	String	Video title
`description`	Text	Description text
`tags`	Array	For search/recommendations
`views`	Int	View count
`likes`	Int	Like count
`status`	Enum	Uploaded / Processing / Published
`created_at`	Timestamp	Upload date

User Table

Stores profile details, subscriptions, watch history, etc.

Search Index

Uses Elasticsearch for quick keyword search.
Stores title, tags, and description fields for full-text queries.

⚡ 8. Scalability Strategies

Strategy	Purpose
CDN	Reduce latency and server load for video playback.
Sharding	Distribute metadata across multiple DB servers.
Caching	Store trending videos and metadata in Redis.
Message Queues	Asynchronous processing for uploads/transcoding.
Microservices	Independent scaling of upload, streaming, and metadata systems.
Read Replicas	Improve DB read performance.
Load Balancer	Distribute traffic among servers.

🧮 9. Data Flow Summary

Upload Path

Client → API → Upload Service → Message Queue → Transcoder → Storage → Metadata DB

Playback Path

Client → CDN → Streaming Service → Player (Adaptive bitrate playback)

🧠 10. Adaptive Bitrate Streaming (ABR)

Definition: Deliver videos at multiple qualities (240p, 480p, 720p, 1080p, etc.) and switch dynamically based on user bandwidth.

Protocols:

HLS (HTTP Live Streaming)
MPEG-DASH

Benefits:

Smooth playback
Handles variable network speeds
Reduces buffering

🔍 11. Search & Recommendation System

Search

Powered by Elasticsearch or Solr
Index video metadata and tags
Use autocomplete and ranking algorithms

Recommendation

Based on:
- Watch history
- Similar tags/topics
- Trending videos
- Collaborative filtering (similar users)

📡 12. Notifications & Subscriptions

Pub/Sub model
- User subscribes to a channel → added to subscriber DB.
- When uploader publishes new video → event triggers notification service.
- Notification sent via email, app push, or in-app feed.

💾 13. Caching

Type	Data Cached	Tool
Application Cache	User profiles, video metadata	Redis
CDN Cache	Static and video files	CloudFront / Akamai
Search Cache	Popular queries	Redis

🔐 14. Security & Compliance

Authentication & Authorization → OAuth 2.0 / JWT
Content Moderation → AI-based or manual review
Rate Limiting → Prevent DDoS & abuse
Encrypted Streaming URLs → Prevent piracy
HTTPS everywhere for data safety

📈 15. Key Metrics to Monitor

Video upload latency
Transcoding time per GB
Playback start time (Time-to-first-frame)
Buffering ratio
CDN hit/miss ratio
Average watch time per user
System availability (% uptime)

🧩 16. Challenges & Trade-offs

Challenge	Solution
Storage costs	Cold storage for old videos (e.g., AWS Glacier)
Bandwidth usage	Adaptive bitrate & CDN
Consistency vs latency	Use eventual consistency for views/likes
Scalability	Microservices + auto-scaling groups
Real-time metrics	Kafka + Stream processors (Flink / Spark)

🏁 17. Summary

Feature	Technology / Design Choice
Video storage	Object storage (S3, GCS)
Metadata	SQL/NoSQL
Playback	CDN + Adaptive streaming
Processing	Distributed transcoding
Search	Elasticsearch
Queue	Kafka / RabbitMQ
Cache	Redis
Auth	OAuth 2.0 / JWT
Monitoring	Prometheus + Grafana