The most over-engineered system I have been asked to rescue was built for ten million users. It had six microservices, a Kafka cluster, sharded databases and a service mesh. It had about nine hundred users, a team of four, and it took three weeks to ship a change to a form.
The most under-engineered was a single server with the database on the same box, no backups, and a deploy process that involved copying files over SSH. It served forty thousand daily users perfectly well right up until the disk filled and there was nothing to restore from.
Scaling is not about picking one of those. It is about knowing which problem you have right now, and doing the next thing rather than the last thing.
Stage One: One Server Is Fine, Actually
A single reasonably-sized machine running your app and a managed database will comfortably serve thousands of daily users. People do not believe this because the internet is full of architecture diagrams from companies with a thousand engineers.
What matters at this stage is not capacity, it is not painting yourself into a corner:
Managed database, not one you installed. Backups, point-in-time recovery and failover are worth the premium from day one. This is what the second team above got wrong, and it is unrecoverable when it bites.
Stateless application process. No sessions in memory, no uploaded files on local disk. Sessions to Redis or a signed cookie, files to object storage. This one decision is what makes every later step possible.
Deploys that are repeatable. A container and a pipeline, even a simple one.
Get those three right and you can stay here for a long time.
Stage Two: Two Servers and a Load Balancer
The first real step, and the reason is usually availability rather than load — one server means every deploy and every crash is downtime.
Because you kept the app stateless, this is mostly configuration. What tends to break:
Anything stored on local disk. Uploads land on server A and the next request hits server B. Object storage.
In-memory caches diverging. Each instance has its own copy, so they disagree after a write. Shared cache, or short TTLs and an acceptance that they will differ briefly.
Scheduled jobs running twice. A cron on both servers fires twice. Use a leader lock, or move jobs to a queue with a single worker.
Add health checks that mean something — a check that verifies the database connection, not one that returns 200 because the process started.
Stage Three: The Database Becomes the Constraint
App servers scale by adding more. The database does not, and this is where most systems actually hurt.
Work through these in order, because the order is roughly by return on effort:
Find the slow queries. Enable pg_stat_statements, sort by total time, and fix the top three. In my experience this alone buys most teams another year of headroom, and it usually comes down to two or three missing indexes.
Fix N+1 patterns. Not one slow query but four hundred fast ones per page.
Add a connection pooler. Postgres connections are processes and a few hundred will hurt. PgBouncer in front, especially with serverless functions or many app instances.
Cache the expensive reads. After the queries are sensible, not before — caching a missing index hides the problem and adds an invalidation surface.
Add read replicas. Most applications are read-heavy by a wide margin. Send reports, listings and search to a replica, keep writes and read-after-write on the primary. You now have replication lag to reason about, which is a real cost.
Notice that sharding is not on this list. It is several stages away and almost nobody who thinks they need it does.
Stage Four: Move Work Out of the Request
The cheapest performance win available at almost any stage: stop doing things while the user waits.
Sending an email, generating a PDF, resizing an image, calling three third-party APIs, writing analytics — none of that belongs in the request cycle. Accept, enqueue, return, process elsewhere.
This helps twice. Responses get dramatically faster, and a slow or broken third party stops taking your API down with it — the queue absorbs it and retries.
A database-backed queue with FOR UPDATE SKIP LOCKED handles this fine into the thousands of jobs a minute. You do not need a broker to start.
Stage Five: Push Work to the Edge
By now the application is reasonable and the wins move outward.
A CDN in front of everything static, and in front of cacheable HTML too. This is the highest-leverage thing most teams under-use, and it costs almost nothing.
HTTP caching headers done properly. A correct ETag turns a repeat request into a 304 with no body.
Compression and image formats. Unglamorous, and it shifts real numbers on real user connections.
Fewer round trips. A screen that needs six sequential API calls is slow on a phone regardless of how fast your servers are. One endpoint shaped for that screen fixes it.
What Actually Falls Over First
Having watched a number of these, the failures are rarely where the capacity planning said they would be.
The database connection pool, not CPU. Requests time out while the database sits idle, because something is holding connections — usually a transaction left open across an external call.
One unbounded query. A customer whose account has 80,000 records hits an endpoint with no pagination, and one request consumes the machine.
A third party. Your payment provider gets slow, your threads all wait on it, and you are down because they are. Timeouts and a circuit breaker are worth more than an extra server.
A background job storm. A retry loop with no backoff, or a nightly job that overlaps with itself.
Every one of those is a bounded-resource problem, not a raw capacity problem. Which is why adding servers so often fails to fix things.
How to Know What to Do Next
Two things, and neither is a diagram.
Measure before you change anything. p95 latency by endpoint, database time as a share of request time, event loop or worker saturation, error rate. Without these you are guessing, and guesses tend toward the most interesting solution rather than the correct one.
Load test with realistic data. Not "can it handle 1,000 requests a second" but "what happens with a customer who has 100,000 rows." The second question finds far more bugs.
Build for roughly ten times your current load. Not a hundred, not a thousand. Ten times is enough headroom to sleep, and cheap enough that you are not paying for users you do not have.
That nine-hundred-user system had been built for a scale it never reached, and the cost was not the infrastructure bill. It was that shipping a form took three weeks.
