Building software is easy when the requirement is simple.
A frontend sends a request to a backend. The backend reads or writes some information in a database and returns a response.
For a small application, that architecture may be perfectly adequate.
But what happens when the application grows from 100 users to 100,000—or even millions? What happens when users connect from different countries, traffic suddenly increases tenfold, a database server fails, an external API becomes unavailable, or thousands of requests attempt to modify the same data simultaneously?
This is where system design becomes essential.
Modern software is not simply a frontend connected to a database. It is a collection of layers and components designed to work together to provide performance, scalability, security, availability, maintainability, and reliability.
Understanding HTTP means understanding concepts such as:
Request methods: GET, POST, PUT, PATCH, DELETE
Headers
Cookies
Status codes
Request and response bodies
Persistent connections
Caching headers
CORS
These concepts affect everything from API design to performance and security.
DNS
Users normally access applications through domain names rather than IP addresses.
DNS translates:
text
www.example.com
↓
203.0.113.10
Modern DNS can do much more than basic domain resolution. It can participate in geographic routing, failover strategies, traffic management, and multi-region architectures.
TCP and UDP
TCP provides reliable, ordered communication and is commonly used by web applications and databases.
UDP removes some of that overhead and is useful where speed matters more than guaranteed delivery.
Engineers do not need to become network engineers to design applications, but understanding these foundations makes debugging distributed systems significantly easier.
2. API Design and Communication
Once applications consist of multiple components, those components need standardized ways to communicate.
Common approaches include:
REST
REST APIs remain one of the most widely used approaches for communication between frontend and backend applications.
text
GET /api/products/42
POST /api/orders
DELETE /api/cart/15
REST is simple, familiar, and supported almost everywhere.
GraphQL
GraphQL allows clients to request exactly the fields they need.
This can be useful for applications with complex frontend data requirements.
gRPC
gRPC uses Protocol Buffers and is frequently used for high-performance communication between internal services.
For example:
text
Order Service
↓ gRPC
Inventory Service
↓
Payment Service
WebSockets
Traditional HTTP usually follows request-response communication.
WebSockets maintain a persistent connection and are useful for real-time applications such as:
Chat
Multiplayer games
Live dashboards
Trading applications
Real-time notifications
Choosing the right communication mechanism depends on the application's requirements rather than which technology is currently popular.
3. Data and Storage
Almost every serious application eventually requires multiple types of storage.
There is rarely one database technology that is ideal for everything.
SQL Databases
Examples include:
PostgreSQL
MySQL
MariaDB
SQL Server
Relational databases are excellent for structured data and transactional workloads.
They provide powerful querying, relationships, indexes, transactions, and consistency guarantees.
NoSQL Databases
Examples include:
MongoDB
Cassandra
DynamoDB
NoSQL databases can be useful when applications require flexible data structures, very high throughput, large-scale distributed storage, or particular access patterns that do not map naturally to relational tables.
The choice should be driven by workload requirements rather than the assumption that NoSQL automatically means "more scalable."
Object Storage
Large binary files generally should not be stored directly inside the main transactional database.
A load balancer can distribute traffic using strategies such as:
Round robin
Least connections
Weighted routing
IP-based routing
It can also perform health checks and stop sending traffic to unhealthy instances.
This improves both scalability and availability.
9. Auto Scaling
Horizontal scaling becomes even more useful when infrastructure can respond automatically to demand.
For example:
text
Normal Traffic
3 Servers
↓
Traffic Spike
10 Servers
↓
Traffic Returns to Normal
3 Servers
Auto scaling may respond to metrics such as:
CPU utilization
Memory usage
Request count
Queue depth
Custom application metrics
This allows infrastructure capacity to follow workload demand instead of remaining permanently overprovisioned.
10. Database Replication
Scaling application servers is relatively easy because they can often be stateless.
Databases are harder.
A common strategy is database replication.
text
Primary DB
/ \
▼ ▼
Read Replica 1 Read Replica 2
Writes go to the primary while some reads can be distributed across replicas.
This can improve read scalability and provide additional redundancy.
However, asynchronous replication may introduce replication lag, meaning replicas can briefly contain older data.
Again, scalability introduces trade-offs.
11. Database Sharding
Eventually, one database server may not be enough.
Sharding divides data across multiple database instances.
For example:
text
Users 1-1M → Shard A
Users 1M-2M → Shard B
Users 2M-3M → Shard C
Or a hash function may determine the shard:
text
hash(user_id) % number_of_shards
Sharding can provide enormous scalability, but it makes operations such as joins, transactions, migrations, rebalancing, and analytics significantly more complicated.
Therefore:
Do not shard simply because large companies do it.
Shard when your workload actually requires it.
12. Message Queues
Not every operation needs to happen before returning a response to the user.
Making the user wait for all these operations would increase response time and create unnecessary dependencies.
Instead:
text
User
↓
API
↓
Create Account
↓
Queue
├── Email Worker
├── SMS Worker
├── Analytics Worker
└── CRM Worker
Technologies include:
RabbitMQ
Kafka
Amazon SQS
Redis Streams
Queues allow systems to process work asynchronously and absorb temporary traffic spikes.
13. Pub/Sub and Event-Driven Systems
In a traditional synchronous architecture:
text
Service A → Service B → Service C
each service becomes directly dependent on another.
Event-driven systems can reduce that coupling.
For example:
text
OrderCreated
│
├── Inventory Service
├── Payment Service
├── Notification Service
└── Analytics Service
The order service publishes an event without needing to know every system that will consume it.
This makes large systems easier to extend but introduces challenges around eventual consistency, retries, duplicate events, ordering, and debugging.
14. Idempotency
Distributed systems fail.
Networks time out. Clients retry requests. Queues redeliver messages.
Suppose a payment API receives:
text
POST /payments
Amount: $100
The payment succeeds, but the response is lost.
The client retries.
Without protection, the customer could be charged twice.
An idempotency key allows the server to recognize repeated operations:
text
Idempotency-Key: order-927461-payment
The same logical operation can then safely return the previous result rather than executing twice.
Idempotency is extremely important for payments, orders, provisioning, webhooks, and distributed workflows.
15. Reliability Engineering
A production system must assume that components will fail.
The question is not:
"Will something fail?"
The better question is:
"What happens when it fails?"
Several patterns help.
Retries
Temporary failures can be retried.
Exponential Backoff
Instead of retrying continuously:
text
1 second
2 seconds
4 seconds
8 seconds
16 seconds
This reduces pressure on already struggling services.
Circuit Breakers
If a downstream service repeatedly fails, requests can temporarily stop being sent to it.
text
Application
↓
Circuit Breaker
↓
External API
This prevents cascading failures.
Failover
If the primary infrastructure becomes unavailable, traffic can move to another instance, zone, or region.
Disaster Recovery
Organizations also need plans for catastrophic failures involving:
Backups
Restore procedures
Secondary regions
Recovery Point Objective (RPO)
Recovery Time Objective (RTO)
Backups alone are not enough. A backup that has never been tested for restoration is an assumption, not a recovery strategy.
16. CDN and Edge Infrastructure
Users may access an application from many geographic locations.
Serving every image, JavaScript file, stylesheet, and video directly from one origin server increases latency and infrastructure load.
A Content Delivery Network places cached content closer to users.
text
Origin
/ | \
/ | \
Edge A Edge B Edge C
│ │ │
Users Users Users
CDNs can reduce latency, reduce origin bandwidth, improve availability, and add security capabilities because they operate in front of origin infrastructure.
17. API Gateways and Reverse Proxies
As applications become larger, incoming traffic needs more sophisticated routing.
A reverse proxy such as NGINX might provide:
TLS termination
Routing
Compression
Static file serving
Basic rate limiting
An API gateway may additionally provide:
Authentication
API keys
Rate limiting
Request transformation
Service routing
Logging
API versioning
Example:
text
Client
↓
API Gateway
├── /users → User Service
├── /orders → Order Service
├── /payments → Payment Service
└── /search → Search Service
18. Rate Limiting
Public APIs cannot accept unlimited requests from every client.
Rate limiting protects applications from:
Abuse
Bots
Accidental request loops
Expensive API usage
Resource exhaustion
For example:
text
100 requests / minute / user
Common strategies include:
Token bucket
Leaky bucket
Fixed window
Sliding window
Rate limiting is both a security and reliability mechanism.
19. Monoliths vs Microservices
One of the most misunderstood system-design decisions is choosing between monolithic and microservice architectures.
Once organizations operate large numbers of containers, orchestration becomes necessary.
Kubernetes can manage:
Scheduling
Service discovery
Scaling
Rolling deployments
Health checks
Configuration
Secrets
Workload recovery
However, Kubernetes also introduces significant operational complexity.
Just like microservices, it should be introduced because the system needs it—not because it appears on architecture diagrams.
28. Deployment Strategies
Updating production software always carries risk.
Modern systems use deployment strategies designed to reduce that risk.
Rolling Deployment
Instances are gradually replaced.
Blue-Green Deployment
Two environments exist:
text
Blue → Current Production
Green → New Version
Traffic switches to Green once testing succeeds.
Canary Deployment
A small percentage of users receive the new version first.
For example:
text
Version A → 95%
Version B → 5%
If monitoring shows healthy behavior:
text
Version A → 50%
Version B → 50%
Eventually:
text
Version B → 100%
This limits the impact of problematic releases.
29. Infrastructure as Code
Infrastructure should also be reproducible.
Instead of manually configuring servers, networks, databases, and permissions, Infrastructure as Code allows environments to be described declaratively.
Tools include:
Terraform
OpenTofu
AWS CloudFormation
Pulumi
Ansible
This enables infrastructure configuration to be version-controlled, reviewed, automated, and recreated consistently.
30. AI-Ready Architecture
Modern system design increasingly includes another layer:
AI infrastructure.
Applications may integrate:
LLM APIs
Embedding models
Vector databases
RAG
Agents
Tool calling
Model gateways
Prompt management
Guardrails
AI observability
A simplified AI application architecture could look like:
text
User
↓
Application
↓
AI Gateway
↓
LLM
│
├── Vector Search
├── Internal APIs
├── Database
└── External Tools
This architecture creates new challenges.
RAG
Retrieval-Augmented Generation provides an LLM with relevant external information before generating an answer.
The goal is not to build the most complicated architecture possible.
The goal is to build the simplest architecture capable of reliably satisfying today's requirements while leaving reasonable paths for tomorrow's growth.
System Design Is About Trade-Offs
Almost every architecture decision solves one problem while creating another.
Caching improves performance but creates invalidation problems.
Replication improves read scalability but can introduce replication lag.
Microservices improve service independence but increase operational complexity.
Queues improve resilience and asynchronous processing but introduce eventual consistency.
Sharding increases database capacity but makes querying and transactions more difficult.
Multi-region deployments improve resilience and geographic performance but dramatically increase infrastructure and data-consistency complexity.
AI agents increase automation capabilities but increase security, permission, and reliability concerns.
Therefore, system design is not about memorizing architecture diagrams.
It is about understanding trade-offs.
A Better Way to Learn System Design
Instead of memorizing hundreds of technologies, learn them as solutions to specific problems.
Start here:
text
Client
↓
Application
↓
Database
Then ask:
The application is slow globally.
Add:
text
CDN
One server cannot handle the traffic.
Add:
text
Load Balancer + Multiple Servers
The database receives too many repeated reads.
Add:
text
Cache
Slow jobs are delaying API responses.
Add:
text
Queue + Workers
Database reads become a bottleneck.
Consider:
text
Read Replicas
One database cannot contain the workload.
Consider:
text
Partitioning / Sharding
Finding failures across many services is difficult.
Add:
text
Centralized Logs + Metrics + Tracing
This approach teaches the most important question in system design:
What problem forced us to introduce this component?
If you cannot answer that question, there is a good chance you do not need the component yet.
Final Thoughts
Modern software engineering is no longer only about writing application code.
Engineers increasingly need to understand how networking, APIs, databases, caching, distributed systems, infrastructure, security, observability, deployment, and AI systems interact.
You do not need to master every technology at once.
But you should understand what each layer is responsible for and recognize the problems that cause architectures to evolve.
