System Design — Interview Questions

So the way I think about it, there's no single right answer, so you're not grading my final diagram. You're watching how I get there: do I scope the problem, do I make sensible choices, and can I explain why I made each one.

There are really five things I try to show:

• Scoping: I ask questions before I draw anything, and I say out loud what's in and out of scope.
• Structure: I follow the same order every time, so I don't jump from the database to the UI and back.
• Depth: I can go properly deep on one or two pieces, like how the cache gets invalidated or how the data is sharded.
• Trade-offs: every choice comes with "I picked X over Y because of this, and the cost is that".
• Communication: I think out loud and check in with you, so you can steer me.

Let me give you an example. If you ask me to design a URL shortener and I draw Kafka, three microservices and a sharded Cassandra cluster in the first two minutes, I've actually failed most of those, even if every box is "correct". If I first ask how many URLs a day we create and find out it's 100,000, I can say "one Postgres is plenty" and justify it. That's the signal you want.

So in practice I treat it as a design conversation with a colleague, not a quiz.

Interviewers look for: the candidate naming "how you think" and "trade-offs" rather than "knowing the right architecture".

So I don't draw anything yet. The first five minutes are for pinning down what we're actually building, because "Twitter" is far too big to design in 45 minutes.

Let me show you what that sounds like:

Me: "Before I design anything, I'd like a few minutes on requirements.
     For the core features, I'm thinking: post a tweet, follow a user,
     and see a home timeline. Search, DMs and ads are out of scope. OK?"
Me: "Roughly how many daily active users? Let's say 200 million?"
Me: "Read-heavy, I assume: people read far more than they post?"
Me: "Is a few seconds of delay before a tweet shows up in followers'
     timelines acceptable, or does it need to be instant?"
Me: "Do we need to handle celebrities with 100M followers?"

What that gives me is a short functional list and a non-functional list I can write on the board. The answers drive the whole design. For example, "a few seconds of delay is fine" tells me I can build the timeline asynchronously with fan-out through a queue, and "celebrities exist" tells me where the deep dive is going to be.

So by minute five I have three core features, a scale number, a read:write ratio and one or two priorities, and I say them back to you before moving on.

Interviewers look for: explicit out-of-scope statements. Cutting scope confidently is a senior signal.

So a junior is expected to get to a working high-level design with the basic building blocks. A senior is expected to find the hard part of the problem on their own and talk about how it fails, what it costs, and how you'd operate it, without being prompted.

Let me give you an example with a URL shortener:

The difference isn't more boxes. It's that the senior answer drives the conversation, picks where to go deep, and brings up failure modes and operations unprompted.

So when I'm aiming for a higher level, I make sure I say "the hard part here is…" before you have to ask me.

So first I treat it as information, not an attack. A pushback usually means either I missed a requirement, or you're steering me towards where the points are. Either way I want to understand which one it is.

Let me give you an example. Say I picked Postgres for a chat app's messages and you ask "are you sure about that?"

Me: "Good question. Let me check my reasoning. I picked Postgres because
     messages are simple rows and I wanted strong ordering per chat.
     Is your concern the write volume? At the 50k writes/sec we estimated,
     a single Postgres primary would struggle."
You: "Yes, the write volume."
Me: "Then I'd switch to a wide-column store like Cassandra, partitioned
     by chat_id and clustered by message time. The cost is we lose
     easy joins and cross-chat transactions, which we don't need here."

What I'm doing there is restating why I chose it, asking what the concern is, and then either defending it with numbers or changing it and naming the new trade-off. What I avoid is silently caving, or digging in without listening.

So in practice: clarify, re-reason out loud, then commit. Changing your mind with a good reason scores well.

Interviewers look for: gracefully accepting hints. Ignoring a hint is one of the most common reasons for a "no hire".

So I replace every question I'd have asked with a written assumption. The marker is looking for the same signals as a live interviewer; they just read them instead of hearing them.

Let me show you how I'd lay it out, with headings so the marks are easy to find:

1. Assumptions & requirements
   - 10M DAU, read:write about 100:1, single region
   - Functional: shorten URL, redirect, basic click count
   - Non-functional: redirect p99 < 50 ms, 99.9% available
2. Estimates      (3-5 lines of numbers, each ending in a decision)
3. API            (3-5 endpoints)
4. Data model     (tables with key fields)
5. Diagram        (every box labelled, numbered arrows)
6. Components     (one paragraph per box, following the arrow numbers)
7. Deep dives     (1-2, clearly labelled)
8. Trade-offs     (small table: choice, why, cost)
9. Conclusion     (two sentences)

The thing I'm careful about is time. It's easy to spend half the exam on a beautiful diagram. I split my time by the marks on each part, and I make sure the trade-offs table exists even if it's short, because that's usually where the top marks are.

So the rule is: assumptions stated up front, headings the marker can scan, and a trade-offs section no matter what.

So I always tie a trade-off to this specific system and a concrete cost. "It depends" on its own earns nothing. "It depends, and here it goes this way because of this requirement" earns a lot.

Let me give you an example. Vague versus concrete, for the same choice:

Vague:    "Caching improves performance but has consistency issues."

Concrete: "I'll cache profiles in Redis with a 5-minute TTL and
           delete-on-write. The cost is up to a few seconds of stale
           data if the delete races with a read. For a profile bio
           that's fine. For the account balance I would NOT cache,
           I'd read from the primary."

What makes the second one good is the pattern: the choice, the reason tied to a requirement, the cost in numbers or user impact, and where I'd make the opposite choice. That last part is what shows I actually understand it rather than reciting it.

One habit that helps me is keeping one priority from the requirements, like "availability over consistency", and pointing back to it: "remember we said a stale like count is fine, so…". Then every decision sounds deliberate rather than random.

Interviewers look for: the words "the cost is…" and a case where the candidate would choose the opposite.

So I use the same eight steps every time, because under pressure I'll forget things otherwise. It's a bit like a pilot's checklist.

1. Requirements      ~5 min   functional + non-functional, scope
2. Estimates         ~3 min   QPS, storage, bandwidth
3. API               ~3 min   main endpoints
4. Data model        ~4 min   entities, key fields, which DB
5. High-level design ~10 min  boxes and arrows, one request end to end
6. Deep dive         ~10-12   the hardest part of THIS problem
7. Bottlenecks       ~5 min   what breaks at 10x, single points of failure
8. Wrap-up           ~2 min   summary, what I'd add, one trade-off

The two steps that decide most of the score are requirements and the deep dive, so I protect time for those and keep estimates and the API short.

For example, in a URL shortener, step 1 tells me it's about 100:1 read-heavy. Step 2 tells me it's a few thousand redirects a second. Step 5 is the standard CDN, load balancer, app servers, cache, database picture. And step 6 is how to generate unique short codes without collisions.

That said, it's my default path, not a script. If you steer me somewhere, I follow you, because that's usually where the points are.

Interviewers look for: a clear process, and that the candidate knows where the time should go.

So functional requirements are what the system does, the features. Non-functional requirements are how well it does them: speed, uptime, scale, correctness, durability.

Let me give you an example for a photo-sharing app:

The reason I split them is that they drive different decisions. The functional list decides my API and data model. The non-functional list decides the architecture. "Never lose a photo" pushes me to S3 with replication. "Stale like counts are fine" lets me use eventual consistency and cache aggressively.

One gotcha is that candidates list features and forget the non-functional side entirely, and then they have nothing to justify their choices with later.

So in practice I write both lists on the board and refer back to them.

So I'd ask questions whose answers actually change the design. I don't ask things just to look thorough.

- "How many new URLs per day, and how many redirects?"   -> QPS, read:write
- "How long do links live? Forever, or do they expire?"  -> storage, cleanup
- "Can users pick a custom alias?"                        -> uniqueness checks
- "Do we need click analytics?"                           -> async event pipeline
- "How short must the code be?"                           -> ID space size
- "Global or one region?"                                 -> CDN, multi-region

Let me walk through why a couple of those matter. If you say 100 million new URLs a day, that's about 1,000 writes a second, and over five years that's around 180 billion links. With 7 characters of base62 I get 62⁷, which is about 3.5 trillion codes, so 7 characters is enough. If you'd said 1 million a day, I'd know the whole thing fits on one database.

And analytics changes the architecture: I don't want to write to the database on every redirect, so I'd publish click events to a queue and aggregate them separately.

So each question maps to a decision, and I tell you which decision as I ask it.

Interviewers look for: questions tied to design impact, not a generic list.

So I keep it to three to five core endpoints: method, path, inputs, output. It's three minutes, and its job is to turn the functional requirements into something concrete I can draw against.

Let me give you an example for a news feed:

POST /api/v1/posts                    { text, mediaIds[] }  -> { postId }
POST /api/v1/users/{id}/follow                               -> 204
GET  /api/v1/feed?cursor=abc&limit=20                        -> { posts[], nextCursor }
POST /api/v1/posts/{id}/likes         Idempotency-Key: ...   -> 204

There are a couple of details I do call out, because they show judgment. The feed uses cursor pagination, not ?page=5. With offsets, when new posts arrive while you scroll you get duplicates, and deep offsets get slow on the database. A cursor like "posts older than this ID" avoids both. And for writes a client might retry, like liking or payments, I mention an idempotency key so a retry doesn't double-apply.

What I don't do is spend time on every status code or field. If you want more detail on one endpoint, I'll go there.

So in practice: small, versioned, cursor-paginated, and idempotent where retries happen.

So I pick the part that's hard specifically for this problem, not the generic part every design has. Load balancers and caches show up everywhere; the deep dive should be the thing that makes this system interesting.

Let me give you some examples:

How I find it: I look at my estimates and requirements and ask what would break first, or what's unusual. If the estimates say 500:1 reads, it's probably caching or the read path. If the requirement is "exactly one driver gets matched", it's consistency.

And I say it out loud: "I think the interesting part is the fan-out, shall I go deep there, or is there something you'd rather see?" That gives you the chance to redirect me.

Interviewers look for: the candidate proposing the deep dive themselves, then offering the choice.

So that usually means I went too deep on precision: calculating exact numbers or listing twenty features instead of three. The estimates are meant to give an order of magnitude and a decision, not an accurate figure.

How I recover is to say it openly and jump ahead:

Me: "I'm conscious of time. Let me lock in what we have:
     3 core features, ~5k read QPS, ~50 write QPS, read-heavy.
     I'll skip the rest of the maths and go to the high-level
     design, then spend the remaining time on the feed fan-out."

The reason that works is that a complete, simple design with one real deep dive scores much better than perfect requirements and no architecture.

What I do to avoid it in the first place: I round everything, like 86,400 seconds becomes 100,000. I cap functional requirements at three to five. And I glance at the clock at roughly 10, 20 and 35 minutes.

So in practice, if I'm behind, I name it, summarise, and move, instead of silently rushing.

So I give a short summary, name what I'd add with more time, and call out one trade-off I made deliberately. It leaves you with a clean picture of the design instead of me trailing off mid-diagram.

Let me give you an example for a URL shortener:

"To summarise: writes go through stateless app servers that take IDs
 from pre-allocated ranges and store them in a sharded key-value store.
 Redirects are served from Redis in front of it, with a CDN caching
 popular links, so p99 stays low even at 100:1 reads.

 With more time I'd add: abuse detection for phishing links, metrics and
 alerts on redirect latency and cache hit ratio, and a proper analytics
 pipeline from the click-event queue.

 The main trade-off I made: range-based IDs mean codes are roughly
 sequential and guessable. If privacy matters, I'd shuffle them."

What that shows is that I know what the design does, I know what's missing (monitoring, security, operations), and I'm aware of the costs I accepted.

So I never end on "and that's it". I end on the trade-off, because that's the last thing you'll remember.

Interviewers look for: observability and failure handling in the "what I'd add" list; seniors mention them unprompted.

So the point isn't the numbers themselves. It's that numbers let me make decisions I can justify: do we need a cache, is one database enough, do we need a CDN, do we have to shard.

Let me give you an example. Say we're designing a to-do app:

1M DAU x 10 writes/day   = 10M writes/day
10M / 100k seconds       = ~100 writes/sec   (peak ~300)
Reads ~10x               = ~1,000 reads/sec

Decision: one Postgres primary plus a replica handles this comfortably.
          No sharding, no Kafka.

Now compare that with a photo feed at 100 million DAU, which comes out at around 50,000 reads a second and 25 GB/s of image traffic. That number tells me a CDN isn't optional anymore.

The rule I follow is that every number ends with a decision. "About 100 writes a second" by itself earns nothing. "About 100 writes a second, so one database is fine" shows I know why I calculated it.

So in practice I round hard, get the order of magnitude, and move on.

Interviewers look for: a decision attached to every number, and not over-engineering when the numbers are small.

So that's about 2,000 to 2,300 writes a second on average, and I'd plan for around 5,000 to 7,000 at peak.

Let me show the maths the way I'd do it on the board:

Messages per day = 10M x 20          = 200M/day
Average QPS      = 200M / 100,000    = ~2,000/sec
   (exact: 200M / 86,400             = ~2,315/sec)
Peak QPS         = 2-3x average      = ~5,000-7,000/sec

Reads: each message is read by ~1-2 people in a 1:1 chat,
       more in groups, so maybe 5x = ~10k reads/sec avg

What I'm doing there is rounding 86,400 seconds up to 100,000 because it makes the maths trivial, and I say so out loud. It's about 15% off, which doesn't change any decision.

Then I apply a peak factor, because traffic isn't flat. Chat traffic spikes in the evening and around events like New Year, so I design for peak, not average.

And the decision: a few thousand writes a second is too many for a comfortable single relational primary long-term, especially with growth, so I'd look at a store built for high write throughput, like Cassandra partitioned by conversation.

So I'd split it into two very different things: tweet text and metadata, which is small, and media, which dominates everything.

Let me state assumptions and work through it:

Assume 200M DAU, 1 in 10 users tweet 2x/day   -> ~40M tweets/day
Tweet row: id 8 B + user 8 B + text ~280 B + timestamps/meta
           -> round to ~500 bytes

Text/day   = 40M x 500 B   = 20 GB/day
Text 5y    = 20 GB x 365 x 5 = ~36 TB   (x3 replication = ~110 TB)

Media: assume 10% of tweets have a 500 KB image
Media/day  = 4M x 500 KB   = 2 TB/day
Media 5y   = 2 TB x 365 x 5 = ~3.6 PB

What the numbers tell me: the text is tens of terabytes, which needs sharding or a distributed store, but it's manageable. The media is petabytes, so it must go in object storage like S3 and be served through a CDN, never in the database. The database just stores the S3 key.

One gotcha I watch for is forgetting replication. Most distributed stores keep three copies, so the real disk footprint is about three times the raw number.

So in practice: small structured data in a database, large blobs in object storage, and I mention the replication factor.

Interviewers look for: separating metadata from media, and the conclusion "media goes to object storage".

So for servers I divide peak QPS by what one server can handle, then add headroom. For cache I use the 80/20 rule: roughly 20% of items get 80% of reads, so I size the cache to hold that hot 20% of a day's read data.

Let me give you an example for a product catalogue API:

Peak QPS                    = 30,000 requests/sec
One app server (simple API) = ~1,000 req/sec   (assumption; measure it)
Servers                     = 30,000 / 1,000 = 30
Headroom (~30%, lose an AZ) = ~40 servers

Cache:
Daily distinct items read   = 5M products x 2 KB = 10 GB
Hot 20%                     = ~2 GB
-> fits easily in one Redis node; run a replica for failover

The part I'm always explicit about is that the 1,000 per server number is a placeholder. A thin CRUD endpoint in Node or Go might do several thousand; something doing image processing might do 50. In a real project I'd load test one instance to get the actual number.

And the headroom isn't padding for the sake of it. If we run across three availability zones and lose one, the other two have to absorb its traffic.

So the answer is "about 40 servers and a couple of gigabytes of Redis", with the assumptions written next to it.

So because they explain almost every design choice we make. Caches exist because RAM is about a thousand times faster than an SSD random read. CDNs exist because a round trip across continents is around 150 ms. Once you know the ladder, a lot of architecture stops being magic.

Let me give you an example from a project. We had a page that made 30 sequential calls to a service. Inside one region each call was maybe 2 ms, so 60 ms total, fine. Then someone moved that service to another region. Now each call paid about 80 ms of round-trip time, and the page took over two seconds. Nothing in the code changed; the physics did.

The fixes all come from the ladder: batch the calls into one, run them in parallel, or cache the result locally.

So in practice I count round trips first, especially cross-region ones, before I worry about CPU.

Interviewers look for: the three lessons: memory beats disk, disk beats cross-region network, and avoid round trips.

So 25 GB/s is a signal that no fleet of app servers should be serving that data directly. It's about 200 Gbps. I'd put the bytes on a CDN backed by object storage, and keep my own servers only for small API responses.

Let me show where it comes from and what follows:

Read QPS        = 50,000 photo views/sec
Avg photo       = 500 KB
Egress          = 50,000 x 500 KB = 25 GB/s  (~200 Gbps)

A typical server NIC = 10-25 Gbps, and you won't saturate it safely
-> would need dozens of servers doing nothing but shipping bytes,
   all far from users

Instead:
- Photos in S3, served via CDN (edge near the user)
- CDN hit ratio 95% -> origin sees ~1.25 GB/s
- Serve resized variants (thumbnail ~30 KB) for the feed
- Modern formats (WebP/AVIF) cut size further

The second-order point is cost. Egress is one of the biggest line items on a cloud bill, so the CDN hit ratio and the image size are business numbers, not just performance numbers. Serving a 30 KB thumbnail in the feed instead of the 500 KB original cuts that traffic by more than 90%.

So the number tells me three things: a CDN is mandatory, I need multiple image sizes, and I should watch the hit ratio as a key metric.

So availability is whether the system answers at all, as a percentage of time it's up. Reliability is whether it answers correctly over time, even when parts fail. Durability is whether data, once saved, is never lost.

Let me give you examples where they come apart:

The reason the distinction matters is that each one has different fixes. Availability comes from redundancy and failover. Reliability comes from testing, retries, and graceful degradation. Durability comes from writing to disk, replicating to several machines, and backups.

In a payments system, durability is non-negotiable; I'd rather be briefly unavailable than lose a transaction. In a like counter, I'd happily trade a bit of durability for availability.

Interviewers look for: a one-sentence definition of each and an example where one holds without the other.

So latency is how long one request takes. Throughput is how many requests the system completes per second. They're related, but you can definitely improve one without the other.

The analogy I use is a road. Latency is how long one car takes to get from A to B. Throughput is how many cars pass per hour. Adding lanes raises throughput, but no single car gets there faster.

Let me give you a concrete example. Say we're inserting analytics events:

One insert per event:     latency per event ~2 ms,  ~500 events/sec per connection
Batch 500 events/insert:  an event may wait up to 1 s in the buffer,
                          but ~20,000+ events/sec per connection

Batching massively improves throughput, but it makes the latency for an individual event worse, because it waits for the batch to fill. That's fine for analytics and terrible for a checkout request.

Going the other way, adding a cache improves latency for reads. Adding more servers improves throughput but doesn't make a single slow query any faster.

So in practice I ask which one the user actually feels. For interactive requests it's latency; for background pipelines it's throughput.

So 99.99%, four nines, means about 52 minutes of downtime a year, or roughly 4 minutes a month. Each extra nine cuts the allowed downtime by ten.

Now, the part people forget: when a request needs several services in series, their availabilities multiply.

Gateway 99.9% x Auth 99.9% x Orders 99.9%
= 0.999 x 0.999 x 0.999 = ~99.7%

Two redundant copies of a 99% service in parallel:
= 1 - (0.01 x 0.01) = 99.99%

What that means is that every hard dependency I add lowers my availability, even if each piece is good. And redundancy is how you buy nines back.

In a project, this is why we made the recommendations service a soft dependency on the product page. If it was down, we rendered the page without the "you might also like" strip instead of failing the whole page.

Interviewers look for: "series multiplies, parallel adds nines", and making non-critical dependencies optional.

So because the average hides the slow requests, and the slow requests are what users complain about. p99 means 99% of requests are faster than this number; it tells me how bad the tail is.

Let me give you an example:

100 requests: 98 take 50 ms, 2 take 3,000 ms
Average = (98x50 + 2x3000) / 100 = 109 ms   <- looks fine
p50     = 50 ms
p99     = 3,000 ms                          <- 1 in 100 users waits 3 s

And it gets worse with fan-out. If a page calls 100 backend services in parallel and has to wait for all of them, the chance that at least one hits its p99 is 1 − 0.99¹⁰⁰, which is about 63%. So the backend's p99 becomes the page's typical latency.

In a project we had an average of 80 ms and everyone was happy, but p99 was over two seconds. It turned out to be garbage-collection pauses on a couple of instances and one query without an index for users with lots of orders. Those were exactly our biggest customers.

So in practice I set SLOs on p95 or p99, and I use hedged requests or timeouts to cut the tail when fan-out is high.

So an SLI is the thing you measure, an SLO is the internal target for it, and an SLA is the external promise to customers with consequences if you break it.

Let me give you an example for a checkout API:

SLI:  % of checkout requests that return non-5xx in under 300 ms
SLO:  99.9% of requests meet that, over a rolling 30 days
SLA:  99.5% monthly, or the customer gets a service credit

Error budget = 100% - 99.9% = 0.1%
             = ~43 minutes of "bad" per month

What I like about this setup is the error budget. If we've used most of the month's budget, we slow down risky deploys and focus on reliability. If we've barely touched it, we can ship faster. It turns "is it reliable enough?" from an argument into a number.

The reason the SLA is looser than the SLO is so we get a warning before we owe anyone money. If we're breaching the internal SLO, we fix it before we breach the contract.

So in an interview, when I say "99.9% available", I try to say what I'd actually measure, like successful requests under a latency threshold, not just "the server is up".

Interviewers look for: error budgets, and SLAs being looser than SLOs.

So going from three nines to four means cutting downtime from about 43 minutes a month to about 4. At that level you can't rely on a human noticing and fixing things; failures have to be absorbed automatically, and deploys can't cause outages.

There are really four areas I'd look at:

• Remove single points of failure: every tier runs in at least three availability zones, the database has automatic failover, the load balancer is managed and redundant.
• Safer changes: most outages are self-inflicted by deploys and config changes, so canary releases, automatic rollback on error-rate spikes, and feature flags.
• Contain failure: timeouts and circuit breakers on every dependency, and make non-critical dependencies soft, so a failure degrades the page instead of breaking it.
• Faster detection: alerts on the SLO burn rate, not on CPU, so we page within a minute or two.

Let me give you an example of why the dependency part matters. If checkout needs five services at 99.9% each, the chain is about 99.5% no matter how good checkout itself is. So I'd either make some of those async (like sending the email via a queue) or raise each one's availability.

And I'd be honest about cost: roughly each extra nine costs a lot more in infrastructure and engineering. I'd check whether the whole product needs it, or only the checkout path.

So at a high level: the browser resolves the name to an IP address with DNS, opens a TCP connection, does a TLS handshake for HTTPS, sends the HTTP request, and the server side runs through a load balancer, an app server, a cache and a database before sending back a response.

Let me walk through it for https://shop.example.com/products/42:

1. DNS      browser/OS cache? no -> resolver -> returns 203.0.113.10 (cached for TTL)
2. TCP      SYN, SYN-ACK, ACK                       ~1 round trip
3. TLS 1.3  certificate + key agreement             ~1 round trip
4. HTTP     GET /products/42, Host, Cookie, Accept...
5. Server   CDN/LB -> app server -> Redis (hit?) -> Postgres on miss
6. Response 200 OK, headers (Cache-Control...), HTML body
7. Browser  parses HTML, fetches CSS/JS/images (mostly from the CDN), renders

What I'd highlight from a system design angle is how much of this is round trips. Before a single byte of the actual request is sent, a fresh connection has paid for DNS, TCP and TLS. On a mobile network with 100 ms round trips, that's a few hundred milliseconds. That's why connection reuse, CDNs near the user, and HTTP/3 matter.

So the one-line version I'd draw: static files from the CDN, API calls through a load balancer to stateless app servers, which read through a cache in front of the database.

Interviewers look for: the right order, and the candidate tying it back to latency and architecture rather than just listing protocols.

So POST creates something or triggers an action, PUT replaces a resource completely, and PATCH changes part of it. Idempotent means doing it twice has the same effect as doing it once.

POST  /orders            { items }         -> creates order #101
POST  /orders            { items }         -> creates order #102   (duplicate!)

PUT   /users/42          { name, email }   -> user 42 is exactly this
PUT   /users/42          { name, email }   -> same result, idempotent

PATCH /users/42          { email }         -> changes only email
PATCH /counters/7        { "op": "increment" } -> NOT idempotent

The reason this matters in design is retries. Networks fail, and clients, load balancers and SDKs retry. If a retry happens on a GET, PUT or DELETE, nothing bad happens. If it happens on a POST, you can charge a customer twice.

In a payments project we fixed that with an Idempotency-Key header. The client generates a UUID per logical payment. The server stores the key with the result, and if the same key comes in again, it returns the stored result instead of charging again.

So in practice I make any retryable write either naturally idempotent or idempotent by key.

So HTTP/1.1 handles one request at a time per connection. HTTP/2 multiplexes many requests over a single TCP connection. HTTP/3 does the same over QUIC, which runs on UDP, so one lost packet doesn't stall every stream.

Let me give you a practical example. With HTTP/1.1 we used to bundle all our JavaScript into one huge file and use domain sharding, just to get around the connection limit. With HTTP/2 those hacks mostly stopped being needed. HTTP/3 helps most on mobile: a phone switching from Wi-Fi to 4G can keep its QUIC connection because it's identified by a connection ID, not by IP and port.

So in practice I let the CDN or load balancer speak HTTP/2 and HTTP/3 to clients, and internal traffic is usually HTTP/2 or gRPC.

Interviewers look for: that HTTP/2 still suffers TCP head-of-line blocking, which is the reason HTTP/3 exists.

So DNS answers are cached at several layers, the browser, the OS, the ISP's resolver, each for the record's TTL. Until those caches expire, clients keep using the old IP. Some resolvers and clients also hold on to answers longer than they should.

Let me give you what I'd check and do:

$ dig +noall +answer api.example.com
api.example.com.   3600   IN   A   203.0.113.10
                   ^ TTL of 1 hour: caches can serve this for up to an hour

Migration playbook:
1. Days before: lower TTL to 60 s, and wait at least the OLD TTL
2. Switch the record
3. Keep the old server running and serving (or proxying) until
   traffic to it drops to ~zero
4. Raise the TTL again afterwards

The mistake we made once was lowering the TTL and switching the record at the same moment. Resolvers had already cached the old record with the one-hour TTL, so lowering it didn't help anyone until that hour passed.

And there's a wider lesson: DNS is a slow failover mechanism. For fast failover I'd rather keep a stable IP in front, like a load balancer or an anycast address, and change what's behind it.

So in practice: lower TTL ahead of time, keep the old target alive, and don't rely on DNS for sub-minute failover.

So every new HTTPS connection pays a TCP handshake and a TLS handshake before sending any data, which is at least two round trips, plus CPU for the crypto. If you do that on every request, the handshakes can cost more than the actual work.

Let me give you an example from a Node service calling another internal API:

// Node 18+ global fetch is backed by undici, which keeps a
// connection pool per origin and reuses connections by default.
const res = await fetch("https://inventory.internal/items/42");

// For the database: a pool, created once at startup, never per request
import pg from "pg";
const pool = new pg.Pool({ max: 20 });
const { rows } = await pool.query("SELECT * FROM items WHERE id = $1", [42]);

What happens with the pool is that the 20 connections are opened once and reused. In a project, we had a Lambda function opening a new Postgres connection per invocation. Under load we hit Postgres's connection limit and requests started failing. The fix was a connection pooler (RDS Proxy; PgBouncer does the same job) in front of the database.

The same applies at the edge: keep-alive between the load balancer and app servers, and HTTP/2 to clients, so one connection carries many requests.

So in practice: reuse connections everywhere, and size pools so that servers times pool size doesn't exceed the database's connection limit.

So I break the request into its phases, network setup, waiting for the server, and downloading, and then drill into whichever phase is big. Guessing without that breakdown usually means optimising the wrong thing.

Let me show you the first thing I'd run:

$ curl -o /dev/null -s -w '
dns      %{time_namelookup}
connect  %{time_connect}
tls      %{time_appconnect}
ttfb     %{time_starttransfer}
total    %{time_total}\n' https://api.example.com/orders

dns      0.004
connect  0.021
tls      0.058
ttfb     1.840     <- server-side time dominates
total    1.862

Here the network setup is about 60 ms, so the problem is on the server. Then I go to distributed tracing, OpenTelemetry spans for example, to see which step inside the request is slow: the cache, a downstream service, or a database query.

In one case the trace showed the handler making 40 separate queries, one per order item, a classic N+1. Each was fast, but together they took over a second. Batching them into one query with WHERE id = ANY($1) brought it down to about 80 ms.

If instead DNS or TLS were large, I'd look at the user's region and whether a CDN or edge termination would help.

So in practice: measure the phases, trace the server side, fix the biggest span first.

Interviewers look for: a methodical breakdown and mention of tracing, rather than jumping straight to "add a cache".

So vertical scaling means making one machine bigger, more CPU and RAM. Horizontal scaling means adding more machines and spreading the work across them.

Let me give you how I'd use both on a real system. For the app tier, I scale horizontally from day one. Once the servers are stateless, adding a fourth or tenth instance is trivial, and it gives me redundancy for free.

For the database, I scale vertically for as long as I reasonably can, because a single primary is so much simpler: transactions, joins, no cross-shard queries. Modern cloud instances go up to hundreds of vCPUs and terabytes of RAM, which carries a lot of companies further than people expect. Then read replicas, and sharding only when writes force it.

So the short answer is: horizontal for stateless things, vertical for the database until it hurts.

Interviewers look for: not reflexively sharding the database, and knowing that stateless is the precondition for horizontal scaling.

So a stateless server keeps nothing about a user between requests in its own memory or disk. Any server can handle any request, because all the state lives in a shared store.

Let me show the before and after in Express:

// Stateful: session lives in this process's memory
app.use(session({ secret: SECRET }));        // default MemoryStore

// Stateless: session lives in Redis, shared by every instance
import { RedisStore } from "connect-redis";
app.use(session({
  store: new RedisStore({ client: redisClient }),
  secret: SECRET,
  resave: false,
  saveUninitialized: false,
}));

Sessions are the obvious one, but there are really four things I move out of the server:

• Sessions go to Redis, or become a signed token the client holds.
• Uploaded files go to S3, never the local disk.
• User-specific caches go to a shared cache; in-process caches are fine only for data everyone shares, like config.
• Scheduled jobs go to a single scheduler or a queue.

Once all of that is out, I can kill any instance at any time, auto-scaling works, and deploys are just replacing instances.

So the test I use: if I terminate this server right now, does any user notice anything beyond one failed request? If not, it's stateless.

So almost certainly the sessions are stored in each server's memory. The user logs in on server A, which creates the session there. The next request goes to server B, which has never heard of that session ID, so it treats them as logged out.

Let me show how I'd confirm it:

Request 1 -> server A   POST /login     Set-Cookie: sid=abc   (A stores abc)
Request 2 -> server B   GET  /account   Cookie: sid=abc       (B: unknown sid)
                                          -> 401 / redirect to login
Request 3 -> server A   GET  /account   Cookie: sid=abc       -> works

It looks random because round robin alternates, so roughly every other request fails. I'd confirm it by checking which instance served each request in the logs, or by adding a response header with the instance ID.

The quick fix is sticky sessions on the load balancer, which pins a user to one server. But I'd treat that as a stopgap: load becomes uneven, and when that server restarts during a deploy, all its users get logged out anyway.

The real fix is moving sessions to Redis, or switching to signed tokens, so any server can validate any request.

I've also seen the same bug in a different form: file uploads saved to local disk, then a later request on another server couldn't find the file.

Interviewers look for: the diagnosis, and knowing sticky sessions are a band-aid rather than the fix.

So auto-scaling adds instances when a metric goes above a target and removes them when it drops. The key is choosing a metric that actually tracks load for this service, and it isn't always CPU.

Let me give you examples of how I'd pick:

A mistake I've seen is scaling a Node API on CPU when it mostly waits on the database. CPU stayed at 20% while latency went through the roof, so it never scaled out. Switching to request count per target fixed it.

There are a few other details that matter in practice. I leave headroom, because new instances take a minute or two to boot and warm up. I scale out fast and scale in slowly, to avoid flapping. I set a minimum of at least two or three instances across zones. And for predictable spikes, like a sale at 9 a.m., I use scheduled scaling ahead of time.

And I remember the database: scaling the app from 10 to 50 instances also multiplies database connections.

So the scheduled job is running inside every app instance, so every instance fires it. Scheduled work needs to happen once per schedule, not once per server.

Let me show the bug and a couple of fixes:

// Bug: this runs in every instance
cron.schedule("0 2 * * *", sendNightlyReport);

// Fix A: a lock in Redis, only one instance wins
cron.schedule("0 2 * * *", async () => {
  const ok = await redis.set("lock:nightly-report:2026-09-26", instanceId,
                             { NX: true, EX: 3600 });
  if (ok) await sendNightlyReport();
});

// Fix B (what I prefer): take it out of the app entirely
//  - Kubernetes CronJob, or EventBridge Scheduler -> one job/queue message
//  - a worker consumes it once

What happens in fix A is that SET with NX only succeeds for the first instance, and the key including the date means tomorrow's run gets a fresh lock. It works, but the job still depends on the web servers being up.

That's why I prefer fix B: a separate scheduler puts one message on a queue, and a worker processes it. I'd also make the job itself idempotent, for example recording "report for this date sent", because at-least-once delivery means it could still run twice occasionally.

So in practice: one scheduler, a queue, and idempotent jobs.

So I go from the cheapest and least disruptive steps to the most complex, and I measure before each one. Sharding is the last resort, not the first move.

1. Find the actual problem   slow query log, pg_stat_statements, EXPLAIN
2. Fix queries and indexes   missing index, N+1, SELECT *, huge OFFSETs
3. Connection pooling        PgBouncer / RDS Proxy, bounded pools
4. Cache hot reads           Redis cache-aside for read-heavy data
5. Scale up                  bigger instance, faster storage
6. Read replicas             route read-only queries there (watch lag)
7. Move work off the DB      async queues, move search to OpenSearch,
                             analytics to a warehouse
8. Split by function         separate DBs per service/domain
9. Shard                     only when WRITE volume or data size forces it

Let me give you an example. In a previous project, the database was at 90% CPU and people were already talking about sharding. pg_stat_statements showed one query, the dashboard's "orders this month" count, accounted for over half the load. An index plus a 60-second cache took the CPU down to about 30%. That bought us more than a year.

The reason I leave sharding for last is its permanent cost: cross-shard joins and transactions get hard, rebalancing is painful, and every query needs a shard key.

So in practice: measure, fix, cache, scale up, replicas, and shard only for writes.

Interviewers look for: measuring first, and the distinction that replicas solve reads while only sharding solves write volume.

So a load balancer sits in front of a group of servers and decides which server gets each request. It does two main jobs: it spreads load across the servers, and it stops sending traffic to servers that are unhealthy.

Let me give you an example. We had three app servers behind an AWS Application Load Balancer:

Client -> ALB (api.example.com:443)
            |-- health check GET /health every 10 s
            |-- server-1  healthy    -> gets traffic
            |-- server-2  healthy    -> gets traffic
            |-- server-3  failing    -> removed until it passes again

When server 3's process crashed, the load balancer noticed within a couple of failed checks and stopped routing to it. Users saw maybe a few failed requests, not an outage.

Beyond that, in practice a load balancer usually also terminates TLS so the app servers don't deal with certificates, and it gives us a single stable address while instances come and go with auto-scaling and deploys. An L7 one can also route by path, like /api to one group and /admin to another.

So it's what makes horizontal scaling work at all: without it, clients would need to know about every server.

So a Layer 4 load balancer only looks at IP addresses and ports, the TCP or UDP level. A Layer 7 load balancer understands HTTP, so it can route based on the path, headers or cookies.

Let me give you a real case. For a normal web API I'd pick L7: I want path routing, TLS termination, and per-request balancing, which matters because HTTP/2 multiplexes many requests on one connection. An L4 balancer would pin all of them to one server.

Where I'd pick L4 is non-HTTP traffic, like a database proxy, MQTT for IoT devices, or a game server on UDP, or when I need extremely high throughput with very low overhead and static IPs.

So the default for web traffic is L7, and L4 when the protocol isn't HTTP or raw performance matters most.

Interviewers look for: a concrete reason to pick each, not just the OSI definitions.

So round robin works well when servers are identical and requests take roughly the same time. Least connections is better when request durations vary a lot, because it sends the next request to whichever server is least busy right now.

Let me give you an example where round robin goes wrong:

Round robin, 3 servers, mix of 20 ms API calls and 30 s report exports

t=0   export -> S1    api -> S2   api -> S3
t=1   export -> S1 (by bad luck the rotation keeps landing exports on S1)
...   S1: 8 exports running, CPU 100%, its API calls time out
      S2, S3: mostly idle
Round robin keeps sending S1 its 1/3 share regardless.

Least connections: S1 has 8 open connections, S2 has 1
-> next request goes to S2, load evens out

That's roughly what happened to us with a report endpoint. Switching the target group to least outstanding requests fixed the hot server; a better long-term fix was moving the exports to a background queue.

The other algorithms I'd mention: weighted round robin when servers are different sizes, and consistent hashing on a key like user ID when I want the same key to hit the same node, for example for caches or stateful shards.

So my default is least connections or least outstanding requests for general APIs, and it's essential for long-lived connections like WebSockets.

So the load balancer calls something like /health every few seconds. After a few consecutive failures it removes the server; after a few passes it adds it back. The design question is what that endpoint should actually check.

Let me show what I'd do, with the classic mistake:

// Risky: deep check. If the DB has a blip, EVERY server fails
// this at once, and the LB removes the whole fleet.
app.get("/health", async (req, res) => {
  await db.query("SELECT 1");
  res.sendStatus(200);
});

// Better: liveness = "this process can serve requests"
app.get("/livez", (req, res) => res.sendStatus(200));

// Readiness = "I'm warmed up and not shutting down"
let ready = false;
app.get("/readyz", (req, res) => res.sendStatus(ready ? 200 : 503));

The gotcha I've hit is exactly the first one. A short database failover made every health check fail together, the load balancer pulled all instances, and a 20-second DB blip turned into several minutes of total outage while servers were re-added. If the database is down, the servers should stay in and return errors or degraded responses; removing them helps nobody.

So the check should answer "is this instance broken compared with its siblings?", not "is everything downstream fine?". Dependency health belongs in monitoring and alerts.

Interviewers look for: the separation of liveness and readiness, and the cascading-failure risk of deep health checks.

So sticky sessions, or session affinity, mean the load balancer sends the same user to the same server every time, usually with a cookie it sets. They make a stateful app work behind a load balancer without changing the code.

The problems, from experience:

• Uneven load: a few heavy users stuck to one server keep it hot while others idle, and new servers from auto-scaling only get new users.
• State loss: when that server dies or is replaced during a deploy, every user pinned to it loses their session or cart.
• Harder deploys: you have to drain long-lived sticky users before retiring an instance.

Let me give you an example. On one project we had stickiness for an in-memory cart. During a Black Friday traffic spike we scaled from 4 to 12 instances, but the original 4 stayed overloaded because existing users were pinned to them. The new 8 were mostly idle.

The one place stickiness is natural is WebSockets: the connection itself lives on one server, so it's effectively sticky for its lifetime. Even then, I keep the important state, like which room a user is in, in Redis, so a reconnect to a different server works.

So my default is stateless servers with no stickiness, and stickiness only as a temporary crutch.

So it can be, if it's one box. The fixes are to make the load-balancing layer redundant, and at the global level to have more than one entry point.

There are three levels I'd talk about:

1. Self-managed pair
   LB-A (active) + LB-B (passive), shared floating/virtual IP (keepalived/VRRP)
   A dies -> B takes over the IP within seconds

2. Managed cloud LB (what I'd use by default)
   AWS ALB/NLB, GCP LB: run across multiple AZs, redundant internally

3. Multi-region
   GeoDNS / latency-based DNS, or anycast (Cloudflare, Global Accelerator)
   -> users go to the nearest healthy region; a whole region can fail

In practice, in an interview I'd say "I'm using a managed load balancer spread across three availability zones, so it isn't a single point of failure", and move on unless you want depth.

If you do want depth, the interesting part is the global layer. DNS-based failover is limited by TTLs and clients caching answers, so it takes minutes. Anycast is faster, because the same IP is announced from many locations and routing shifts away from a dead one quickly.

One more thing I'd mention: a redundant load balancer doesn't help if its config is a single point of failure. A bad routing rule pushed everywhere at once can take everything down, so config changes need a gradual rollout too.

Interviewers look for: active-passive or managed LBs, plus the global layer for senior roles.

So the key is connection draining plus graceful shutdown. The load balancer stops sending new requests to an instance, lets in-flight requests finish, and only then does the process exit.

Let me show the app side in Node, because that's the part people forget:

const server = app.listen(3000);
let shuttingDown = false;

app.get("/readyz", (req, res) => res.sendStatus(shuttingDown ? 503 : 200));

process.on("SIGTERM", () => {
  shuttingDown = true;              // LB sees 503, stops routing new requests
  setTimeout(() => {
    server.close(() => process.exit(0));   // stop accepting, finish in-flight
  }, 10_000);                       // wait for the LB to notice
});

What happens during a rolling deploy is: a new instance starts and only receives traffic once its readiness check passes. Then an old one gets SIGTERM, fails readiness, the load balancer drains it for its deregistration delay, and it exits cleanly. Repeat until all are replaced.

The bug we had before adding this was that Kubernetes killed pods immediately, and we saw a burst of 502s on every deploy from requests that were mid-flight.

For WebSockets, draining takes longer, so we send clients a "reconnect" message and let them reconnect to a new instance with backoff.

So in practice: readiness gates on startup, drain on shutdown, and deploy in small batches or as a canary.

So a cache is a small, fast store that keeps copies of data you read often, so you don't go to the slow source every time. It's usually in RAM, which is orders of magnitude faster than disk or a network call to another service.

Let me walk through the layers from the user inwards:

The closer to the user, the faster the hit, but the harder it is to invalidate. I can delete a Redis key instantly; I can't reach into a user's browser and clear its cache.

So I cache immutable, versioned assets aggressively at the edge, and cache data that changes in Redis, where I control invalidation. The metric I watch is the hit ratio, and I'd aim for 80–95% or more on a well-chosen cache.

Interviewers look for: awareness of multiple layers, and that invalidation gets harder further from the server.

So in cache-aside, the application owns the logic. On a read it checks the cache first. On a miss it reads the database, stores the result in the cache with a TTL, and returns it. On a write it updates the database and then deletes the cache key.

async function getProduct(id) {
  const key = `product:${id}`;
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);                  // hit

  const product = await db.products.findById(id);          // miss
  if (product) {
    await redis.set(key, JSON.stringify(product), { EX: 300 });  // TTL 5 min
  }
  return product;
}

async function updateProduct(id, changes) {
  await db.products.update(id, changes);   // 1. source of truth first
  await redis.del(`product:${id}`);        // 2. invalidate; next read refills
}

What I like about it is that the cache only holds data someone actually asked for, and if Redis goes down the app still works, just slower, by reading the database directly.

The costs are that the first read of each item is slow, and there's a small window of staleness between the database write and the delete. The TTL is my safety net: even if a delete gets lost, the stale value lives at most five minutes.

So it's my default pattern for read-heavy data.

So they differ in where a write goes first. Write-through writes to the cache and the database together. Write-back writes only to the cache and flushes to the database later. Write-around writes only to the database and lets reads fill the cache.

Let me give you an example of write-back from a project. We had a view counter on articles. Doing an UPDATE ... SET views = views + 1 on every page view was hammering one hot row. Instead we did INCR in Redis, and a job every 30 seconds flushed the totals to Postgres in one batch. If Redis crashed we might lose 30 seconds of view counts, which was completely acceptable for that data.

I'd never use write-back for orders or payments, because losing an acknowledged write there is a real incident.

So in practice: cache-aside with write-around is the default, write-through when read-after-write matters, write-back only for data I can afford to lose.

Interviewers look for: the durability risk of write-back, tied to a use case where it's acceptable.

So because updating the cache from two concurrent writers can race and leave the wrong value in the cache indefinitely, while deleting just forces the next read to fetch the truth from the database.

Let me show the race:

Writer A: UPDATE db price = 10
Writer B: UPDATE db price = 12          (DB now says 12, correct)
Writer B: SET cache price = 12
Writer A: SET cache price = 10          (A was slow, e.g. a GC pause)

Result: DB = 12, cache = 10, and it stays wrong until the TTL expires.

With delete:
Writer A: UPDATE db = 10 -> DEL key
Writer B: UPDATE db = 12 -> DEL key
Next read: miss -> reads 12 from DB -> caches 12.

Delete isn't perfect either. There's a rarer race: a reader misses, reads the old value from the database, then a writer updates and deletes, and then the slow reader writes the old value into the cache. It needs very specific timing, and the TTL bounds how long it lasts. If that matters, some teams do a delayed second delete, or drive invalidation from the database's change stream (CDC).

The other order question is: always update the database first, then delete the cache. If I delete first, a reader can refill the cache with the old value before my database write lands.

So my rule: write the database, then delete the key, and always have a TTL as a backstop.

So when the cache is full, LRU throws out the item that hasn't been accessed for the longest time, and LFU throws out the item that's been accessed the fewest times. LRU is about recency, LFU is about popularity.

Let me give you an example where they behave differently. Say the homepage config is read constantly, and then a nightly batch job scans a million old products once each:

LRU: the scan touches a million new keys; each is "most recent"
     -> the homepage config and other hot keys get pushed out
     -> morning traffic hits a cold cache

LFU: the scanned keys have a count of 1; the config has a huge count
     -> scanned keys get evicted first, hot keys survive

So LFU is more resistant to one-off scans. The downside is that something that was popular last week can linger with a high count, though Redis's LFU decays counts over time to handle that.

In Redis this is just the maxmemory-policy setting: allkeys-lru or allkeys-lfu, or the volatile-* variants that only evict keys with a TTL. The default is noeviction, which returns errors on writes when memory is full, and I've seen that catch teams out, so I always set it explicitly for a cache.

So LRU is my default, LFU when there's a stable set of always-hot keys, and a TTL on everything regardless.

So that's a cache stampede, or thundering herd. The hot key expires, thousands of concurrent requests all miss at the same moment, and every one of them goes to the database to rebuild the same value.

There are three fixes I'd combine. First, request coalescing, so only one request rebuilds the value. Within one process that looks like this:

const inFlight = new Map();

function loadOnce(key, loader) {
  if (inFlight.has(key)) return inFlight.get(key);   // join the existing load
  const p = loader().finally(() => inFlight.delete(key));
  inFlight.set(key, p);
  return p;
}

// 1,000 concurrent requests for the same key
const reqs = Array.from({ length: 1000 }, () =>
  loadOnce("product:42", () => db.getProduct(42)));
await Promise.all(reqs);   // db.getProduct ran exactly once

Across many servers I'd add a short Redis lock with SET lock:key NX EX 5: the winner rebuilds, and the others briefly serve the stale value or wait and retry.

Second, refresh early: when a hot key is close to expiry, a background refresh rebuilds it before it actually disappears, so users never see the miss.

Third, jitter the TTLs, like 300 seconds plus a random 0–60, so thousands of keys written together don't all expire in the same second. That's the related problem called a cache avalanche.

So in practice: coalesce, refresh early, jitter, and let the system serve slightly stale data rather than fall over.

Interviewers look for: naming "stampede" and giving at least locking plus TTL jitter.

So those are two different problems. A hot key is one key getting so many reads that the single Redis node holding it maxes out. Cache penetration is requests for keys that don't exist, so they always miss and always hit the database, often from a bot or an attack.

Let me give you the fixes for each:

Hot key (e.g. a celebrity's profile during a live event)
- Replicate it: write product:42#0 ... product:42#9 across nodes,
  readers pick one at random -> load spread 10 ways
- Small in-process cache (1-5 s TTL) in front of Redis for that key
- Read replicas of the Redis shard

Penetration (e.g. GET /users/99999999 in a loop)
- Cache the miss: SET user:99999999 "NULL" EX 60
- Bloom filter of all valid IDs: "definitely not present" -> reject
  before touching cache or DB
- Validate input shape and rate-limit per client

In a project we had the hot-key problem during a flash sale: one product page took most of the traffic, and one Redis shard hit 100% CPU while the others were idle. A local in-process cache with a 2-second TTL took most of those reads off Redis entirely, and two seconds of staleness on a product description was fine.

For penetration, the gotcha with caching misses is a short TTL. If the user or item gets created a moment later, I don't want "not found" cached for an hour.

So hot keys: spread or localise the reads. Missing keys: remember the miss or filter it out before the database.

So a CDN is a network of cache servers, edge servers or PoPs, spread around the world close to users. They keep copies of my static content so a user in Tokyo downloads the image from a server in Tokyo instead of from my origin in Virginia.

Let me give you a quick sense of why it matters. A round trip from Tokyo to us-east-1 is roughly 150 ms. A page with a stylesheet, a JS bundle and twenty images pays that over and over, plus the TLS handshake. From a nearby edge it is more like 10 to 20 ms. And the origin stops serving those bytes at all.

I add one any time the design serves images, video, JS bundles or downloads to lots of users. In an interview I usually say "media lives in object storage, served through a CDN" as soon as I see files in the requirements. And if my back-of-the-envelope shows gigabytes per second of egress, it's not optional, because no fleet of app servers should be pushing that.

The side benefits are real too: lower bandwidth bills from the origin, and the CDN soaks up a lot of DDoS traffic before it ever reaches me. The cost is paying per GB delivered, and the risk of users seeing stale content until the TTL runs out.

Interviewers look for: tying the CDN to a number from your estimate (egress bandwidth or latency), not adding it by reflex.

So the difference is who moves the content to the edge. With a pull CDN, the edge fetches from my origin the first time someone asks for a file, then caches it. With a push CDN, I upload the content to the CDN ahead of time.

In practice almost everything I've worked with was pull. On a web app we just put CloudFront in front of the S3 bucket and the app, set Cache-Control headers, and let it fill itself. The only time push made sense was a launch where we knew a large file would be hit by a huge crowd at a set time, so we pre-warmed it rather than let the first wave of users all miss at once.

So the fix is to never reuse a filename for different content. I put a content hash in every asset's name and cache those forever, and I keep the HTML file that points at them short-lived.

# hashed assets: the name changes when the content changes
/assets/app.3f9a2c.js
Cache-Control: public, max-age=31536000, immutable

# the entry HTML: always revalidate
/index.html
Cache-Control: no-cache

What happens here is that a deploy produces app.7b1e04.js, a brand new URL, so no cache anywhere has it and there's nothing to purge. The only file that has to be fresh is index.html. no-cache doesn't mean "don't store it", it means "check with the server before using it", so a quick 304 keeps it cheap.

The bug in the question almost always comes from someone putting a long max-age on index.html, or serving app.js without a hash. Vite, Next.js and webpack all hash by default, so it's usually the HTML headers that are wrong.

One gotcha I've hit: keep the old hashed files around for a while after a deploy. A user who loaded the old HTML a minute ago will still ask for the old chunk when they navigate, and if it's gone they get a broken lazy-loaded route.

Interviewers look for: cache busting through file names instead of purges, and knowing that no-cache means "revalidate".

So yes, in three ways: caching responses that are shared by many users for a short time, speeding up the connection even when nothing is cached, and running code at the edge.

Let me give you an example. On a product page API, the response is the same for everyone for a minute or so, so I can let the CDN cache it briefly:

GET /api/products/42
Cache-Control: public, s-maxage=60, stale-while-revalidate=300

s-maxage applies only to shared caches like the CDN, so the browser isn't affected. stale-while-revalidate lets the edge keep serving the old copy while it refreshes in the background, so users never wait on the origin. Even a 60-second TTL on a hot endpoint can take most of the load off the origin.

For things that can't be cached, like a checkout call, the CDN still helps because the user's TLS handshake ends at a nearby edge, and the edge keeps warm, pooled connections back to the origin. And edge functions, Cloudflare Workers or CloudFront Functions, can do auth checks, redirects, A/B bucketing or geo routing without a trip to the origin.

The line I never cross is caching anything personalised as public. Anything with user data gets Cache-Control: private.

So almost certainly a personalised response got stored in the shared cache under a key that didn't include who the user was. The next person who requested the same URL got the first person's page.

Here's how it usually happens. The account page is GET /account. The cache key is just the path. Someone adds a CDN rule "cache everything for 5 minutes", or the origin sends Cache-Control: public, max-age=300 by mistake, and the CDN doesn't look at the cookie.

# what the origin sent (wrong)
GET /account
Cache-Control: public, max-age=300
Set-Cookie: session=...

# what it should send
Cache-Control: private, no-store

What I'd do, in order: purge the path right away, then fix the header at the origin so anything behind auth is private or no-store. Then I'd put a default at the CDN so responses with a Set-Cookie or an Authorization request header are never cached. And I'd add a test in CI that checks headers on authenticated routes.

This isn't theoretical. There have been real incidents at big companies exactly like this, usually after a config change to cache rules. It's a data leak, so I'd also treat it as a security incident and figure out who saw what.

Interviewers look for: cache keys and Cache-Control: private, and treating it as a data breach, not only a caching bug.

So a full purge turns every edge into a cold cache at the same moment. Hundreds of PoPs all miss and all go to the origin together, for every popular file. The origin normally sees maybe 5% of traffic and suddenly sees close to 100%. That's a thundering herd.

There are really three things I'd put in place:

• An origin shield. One regional cache layer sits between the edges and the origin. When 200 edges miss on the same file, they all ask the shield, and the shield asks the origin once. CloudFront calls it Origin Shield, Fastly calls it shielding.
• Request collapsing. If 1,000 requests for the same object miss at one edge at once, the edge sends one request to the origin and makes the rest wait for it. Most CDNs do this by default for cacheable responses.
• Not purging everything. With hashed asset names I rarely need a purge. When I do, I purge by path or by surrogate key or tag, like "all pages for product 42", not the whole site.

I'd also use stale-if-error, so if the origin is struggling the edge keeps serving the last good copy instead of passing errors to users.

So in practice the answer is: shield plus targeted purges, and the origin is sized for the realistic cold-start load, not just for the steady state.

So I decide from the access pattern and the consistency needs, not from which one is supposed to scale better. My default is PostgreSQL, and I move away from it when a specific requirement pushes me.

There are really a few questions I ask myself:

• Do I need transactions across several rows, like moving money or decrementing stock? That's SQL.
• How do I read the data? Always by one key means key-value. Whole nested object at once means document. Append-heavy and time-ordered per user means wide-column.
• How much write volume, and across how many regions? Tens of thousands of writes a second, multi-region, points at Cassandra or DynamoDB.
• Are relationships the actual product, like friends-of-friends? That's a graph database.

Let me give you an example. For a chat app I'd keep users, contacts and billing in Postgres, because that data is relational and needs transactions. Messages are huge in volume, append-only, and always read as "latest 50 in this conversation", so they go into Cassandra partitioned by conversation ID.

So in practice: Postgres for the core, and a specialised store only where the numbers or the access pattern justify it.

Interviewers look for: justification by access pattern and scale, not "NoSQL scales, SQL doesn't".

So ACID is the set of guarantees a transaction gives me: Atomic means all or nothing, Consistent means my rules and constraints always hold, Isolated means concurrent transactions don't see each other's half-done work, and Durable means once it's committed it survives a crash.

The classic example is a transfer:

BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;

If the server dies between the two updates, atomicity means neither happens, so money doesn't vanish. A CHECK (balance >= 0) constraint is the consistency part: the whole transaction fails rather than leave a negative balance. Isolation stops two transfers from the same account reading the same starting balance and both succeeding.

One gotcha: isolation isn't all-or-nothing either. Postgres defaults to Read Committed, which doesn't stop every race. For a "check stock, then decrement" flow I either do it in one statement, UPDATE ... SET stock = stock - 1 WHERE id = $1 AND stock > 0, or lock the row with SELECT ... FOR UPDATE.

The contrast is BASE, which many NoSQL stores give you: basically available, soft state, eventually consistent. Fine for like counts, wrong for balances.

So I pick Cassandra, or ScyllaDB, when I have a huge write volume, the data is naturally time-ordered per some key, and I always read it by that key. Chat messages, activity feeds, IoT readings and logs are the textbook cases.

The big mindset shift is that I design the table from the query, not from the entities. For "show the latest messages in a conversation":

CREATE TABLE messages (
  conversation_id uuid,
  bucket          int,        -- e.g. month, keeps partitions bounded
  message_id      timeuuid,
  sender_id       uuid,
  body            text,
  PRIMARY KEY ((conversation_id, bucket), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);

What happens here is that (conversation_id, bucket) is the partition key, so all messages of one conversation for one month live together on the same replicas. message_id is the clustering key, so they're stored sorted, newest first. "Latest 50" is a single sequential read from one partition, and writes are appends to an LSM tree, which is why it takes writes so well.

The bucket is there because a partition that grows forever becomes a hotspot and a performance problem. The trade-off is what I give up: no joins, no ad-hoc queries on other columns, and if I need a second access pattern I usually write the data into a second table.

Interviewers look for: query-first modelling and a bounded partition key.

So it depends on how varied the product data is, but honestly my first move today is Postgres with a JSONB column, and I'd only go to MongoDB if the catalogue is the whole product and it's very document-shaped.

The reason the question comes up is that products have very different attributes. A shirt has size and colour, a laptop has RAM and CPU, a book has an ISBN. Forcing all of that into fixed columns is painful. A document model handles it naturally:

CREATE TABLE products (
  id          bigserial PRIMARY KEY,
  sku         text UNIQUE NOT NULL,
  price_cents int  NOT NULL,
  category    text NOT NULL,
  attributes  jsonb NOT NULL      -- {"ram_gb": 16, "cpu": "M3"}
);
CREATE INDEX ON products USING gin (attributes);

SELECT * FROM products WHERE attributes @> '{"ram_gb": 16}';

What I get is the best of both: price, SKU and stock stay as real typed columns with constraints and take part in order transactions, and the variable stuff sits in JSONB with a GIN index for filtering.

Where I'd pick MongoDB instead is if the team is already on it, and the catalogue is read as whole documents with deeply nested variants, and there's little transactional coupling to orders. Search and faceting across the catalogue would go to Elasticsearch or OpenSearch either way.

So polyglot persistence just means using several different databases in one system, each for the job it's best at. The downside is that every extra store is something to run, secure, back up and keep in sync.

In my last project the setup looked like this:

The part people underestimate is keeping them consistent. Postgres was the source of truth, and the search index was fed from change events. When that pipeline broke for an hour, search showed products that were already out of stock. So now I need monitoring on the sync, a way to rebuild the index from scratch, and people who know how each store fails.

So in practice I add a store only when I can point to a requirement Postgres genuinely can't meet, and I always name which store is the source of truth.

Interviewers look for: naming a single source of truth and how the derived stores are kept in sync.

So I'd push back, politely, with numbers. A single well-tuned Postgres instance handles a lot more than most products will ever need, and the scaling path is well known. Choosing a database for a scale we don't have usually costs us transactions and joins we need today.

Let me put some numbers on it. A large managed Postgres instance with good indexes can do tens of thousands of simple queries per second and hold multiple terabytes. Add read replicas and a cache and most read-heavy apps are fine for years. If we really outgrow one writer, there are options that keep SQL: Citus for sharding Postgres, Aurora or AlloyDB for storage scaling, or distributed SQL like CockroachDB, YugabyteDB or Spanner.

And MongoDB isn't a free lunch either. Sharding it still needs a good shard key, cross-shard transactions are expensive, and a flexible schema tends to turn into an undocumented schema.

What I'd actually ask is: what's the access pattern, what's the expected write rate, what needs transactions? If the answer is "billions of append-only events per day, read by key", then yes, I'd argue for Cassandra or DynamoDB for that part. But for the core business data, I'd start with Postgres and keep the scaling ladder in mind: indexes, cache, bigger box, replicas, then partitioning.

Interviewers look for: not jumping to the most complex tool, and knowing what the real scaling options for SQL are in 2025 and after.

So replication is keeping copies of the same data on several machines. I get three things from it: durability, because a dead disk doesn't lose data; availability, because if one machine dies another can answer; and read scaling, because I can spread reads across the copies.

Let me give you a typical setup. For a production Postgres I'd have one leader and two followers, each in a different availability zone:

AZ-a: leader     <- all writes
AZ-b: follower 1 <- synchronous, for durability and failover
AZ-c: follower 2 <- asynchronous, serves read traffic

If AZ-a goes down, follower 1 has every committed write and gets promoted. Meanwhile follower 2 takes the dashboard and reporting queries off the leader.

The thing I always keep in mind is that replication is not a backup. If someone runs DELETE FROM orders without a WHERE, that delete replicates to every copy in milliseconds. So I still want point-in-time recovery from snapshots plus the write-ahead log.

Interviewers look for: the three benefits, and the point that replication doesn't protect against bad writes.

So it's about when the leader tells the client "saved". With synchronous replication it waits until a follower confirms it has the write. With asynchronous it confirms straight away and ships the change to followers in the background.

Pure synchronous to every follower is rarely used because one slow or dead follower stalls all writes. What most real setups do is semi-synchronous: one follower is synchronous, the rest async. In Postgres that's something like synchronous_standby_names = 'ANY 1 (replica_b, replica_c)', so any one of them confirming is enough and a single slow one doesn't block me.

So in practice, for payments or orders I want at least one synchronous copy. For something like analytics events, async is fine and I'd take the speed.

So that's replication lag breaking read-your-own-writes. The update went to the leader, but the reload was served by a follower that hadn't received the change yet. Usually lag is milliseconds, but under load it can be seconds, and the redirect after a form submit is fast enough to beat it.

There are a few fixes, and I'd pick based on how much traffic we're talking about:

// Option 1: after a write, pin this user's reads to the leader briefly
async function updateProfile(userId, data) {
  await leader.query("UPDATE users SET name = $1 WHERE id = $2", [data.name, userId]);
  await redis.set(`recent-write:${userId}`, 1, { EX: 5 });
}

async function getProfile(userId) {
  const db = (await redis.exists(`recent-write:${userId}`)) ? leader : replica;
  return db.query("SELECT * FROM users WHERE id = $1", [userId]);
}

What happens here is that for five seconds after a write, that one user reads from the leader, and everyone else still reads from replicas. Other options are: always read a user's own profile from the leader, since that's a small fraction of traffic; or track the write's log position (the LSN in Postgres) and only use a replica that has caught up to it.

A related bug is data going backwards when two refreshes hit two different replicas. The fix there is monotonic reads: stick a user to one replica, for example by hashing the user ID.

Interviewers look for: naming replication lag and read-your-writes without being prompted.

So the difference is how many nodes accept writes. Leader-follower has one, multi-leader has several, usually one per region, and leaderless lets the client write to any set of replicas and uses quorums.

Let me give you how I'd use them. For a normal app in one region, leader-follower every time, because it's simple and there are no conflicts. If we have users in the US and EU and writes must be fast in both, multi-leader gets me that, but now two people can edit the same row in two regions at the same moment and I need a merge rule. Leaderless is what I'd get from Cassandra for massive write volume where I'm fine with eventual consistency.

So my default is single leader, and I only accept conflict handling when the requirement actually needs multi-region writes.

So failover has three steps: detect that the leader is dead, pick the most up-to-date follower, and point all writes at it. Split brain is the nasty case where the old leader wasn't really dead, comes back, and both nodes think they're the leader and accept writes.

Walking through it: detection is usually heartbeats with a timeout. Too short and a GC pause or a network blip triggers a failover for nothing. Too long and we're down longer. Then a coordinator, Patroni with etcd for Postgres, or the managed service in RDS or Aurora, picks the follower with the latest log position and promotes it. Clients find the new leader through DNS or a proxy.

The fix for split brain is fencing. Every time a leader is elected, it gets a higher epoch number, and storage rejects anything from an older one:

leader A elected   -> epoch 7
A pauses (long GC), B is elected -> epoch 8
A wakes up, tries to write with epoch 7
storage: 7 < 8 -> rejected

The election itself goes through a consensus system like etcd or ZooKeeper using Raft or ZAB, so only one node can hold the lease at a time. Another gotcha: with async replication, the promoted follower may be missing the last few writes, and those are simply lost unless one follower was synchronous.

Interviewers look for: fencing tokens or epochs, and the trade-off in choosing the failure-detection timeout.

So there's no single right answer. It depends on what the data means. The options go from simple and lossy to more work and correct: last-write-wins, keeping both versions and merging in the app, or using data types that merge themselves, CRDTs.

Let me give you an example with a shopping cart. The user adds a book on their phone in the EU region and a lamp on their laptop in the US region, at the same time.

EU:  cart = [book]      (ts 10:00:00.120)
US:  cart = [lamp]      (ts 10:00:00.135)

last-write-wins -> [lamp]           the book is silently lost
merge as a set  -> [book, lamp]     what the user expects

Last-write-wins keeps whichever has the later timestamp. It's simple, and it's fine for something like "last seen at", but it throws away data, and clocks across regions aren't perfectly in sync anyway. For the cart I'd model it as a set, actually an OR-Set CRDT, so both adds survive. For a counter like likes, a G-Counter CRDT, where each region keeps its own count and the total is the sum.

For things like a document title where there's no automatic merge, I'd use version vectors to detect that the writes were concurrent and either show both to the user or pick a rule. And for data where conflicts are unacceptable, like a balance, I avoid multi-leader entirely and route all writes for that account to one home region.

So replication copies the same data onto several machines, and sharding splits different data across several machines. Replication helps availability and reads. Sharding is what I need when the data or the write volume is too big for one machine.

Let me give you an example. Say we have 20 TB of orders and 50,000 writes per second, more than one Postgres leader can take. I'd split by customer_id into, say, 8 shards, each holding roughly 2.5 TB and 6,000 writes a second. Then each shard is itself replicated, a leader plus two followers, so losing one machine doesn't lose an eighth of my customers.

shard 1: leader + 2 followers  (customers hashing to 1)
shard 2: leader + 2 followers  (customers hashing to 2)
...
shard 8: leader + 2 followers

So real systems use both together. What I always say in an interview is that sharding comes late on the list, because it costs me cross-shard joins, easy transactions and simple operations. I'd try indexes, caching, a bigger box and read replicas first.

Interviewers look for: "replication copies, sharding splits", and that each shard is replicated too.

So a good shard key does three things: it has lots of distinct values, it spreads writes evenly, and most of my queries include it so they hit a single shard.

Let me walk through an example for a multi-tenant SaaS app, like a project management tool:

With tenant_id, a whole company's data lives together, so joins and transactions within a tenant still work on one shard. That's exactly how Citus recommends modelling SaaS apps.

The risk is a giant tenant. If one customer is 30% of our traffic, their shard is a hotspot. I'd handle that with a directory layer, so a big tenant can be moved to a dedicated shard without touching anyone else.

So in practice I pick the key from the most common query, then look for the skew.

So range sharding puts keys in contiguous ranges, like A to F on shard 1, which makes range queries cheap but tends to create hotspots. Hash sharding runs the key through a hash first, which spreads load evenly but scatters ranges across every shard.

Let me give you an example with orders keyed by time:

Range by created_at:
  shard 1: Jan-Mar   shard 2: Apr-Jun   shard 3: Jul-Sep  (all new writes here)
  "orders in March" -> one shard, fast
  today's writes    -> one shard, hot

Hash by order_id:
  shard = hash(order_id) % 4
  writes            -> spread evenly
  "orders in March" -> ask all 4 shards and merge

What I'd do for time-series data is often a combination: hash by the entity and then range within it. Cassandra's model is exactly that. The partition key is hashed across nodes, and inside a partition rows are sorted by the clustering key, usually time. So "sensor 42's readings for last week" is one partition and a range scan inside it.

So: range when range scans are the main query and the writes aren't all at the end, hash when even load matters most, and hash-then-range when I need both.

Interviewers look for: spotting the monotonic-key hotspot with range sharding.

So the problem with hash(key) % N is that changing N changes the answer for almost every key. Going from 4 to 5 servers moves about 80% of the keys. Consistent hashing puts both servers and keys on a ring, and each key belongs to the next server clockwise, so adding a server only moves the keys in one arc, about 1/N of them.

Let me show you. I actually ran this with 100,000 keys:

const crypto = require("node:crypto");
const h = (s) => crypto.createHash("md5").update(s).digest().readUInt32BE(0);

class Ring {
  constructor(nodes, vnodes = 100) {
    this.points = [];
    for (const n of nodes) this.add(n, vnodes);
  }
  add(node, vnodes = 100) {
    for (let i = 0; i < vnodes; i++) this.points.push({ pos: h(`${node}#${i}`), node });
    this.points.sort((a, b) => a.pos - b.pos);
  }
  get(key) {
    const k = h(key);
    let lo = 0, hi = this.points.length;
    while (lo < hi) {                       // first point >= k
      const mid = (lo + hi) >> 1;
      if (this.points[mid].pos < k) lo = mid + 1; else hi = mid;
    }
    return this.points[lo % this.points.length].node;   // wrap around
  }
}

// 4 nodes -> add a 5th
// ring moved: 19.6%     (about 1/5)
// mod moved:  79.7%     (h % 4 vs h % 5)

What happens here is the vnodes part: each server is placed at 100 points on the ring, not one. With a single point per server, one server might own a huge arc by bad luck. Virtual nodes even that out, and I can give a bigger machine more of them.

Cassandra, DynamoDB and most distributed caches use this idea. One gotcha: consistent hashing reduces how much data moves, it doesn't make moving it free. I still have to copy that 20% while serving traffic.

Interviewers look for: the 1/N vs almost-all numbers, and virtual nodes for balance.

So that's the celebrity or hot-key problem. The shard key is fine on average, but one key gets far more traffic than a whole shard can handle. I'd fix it differently for reads and for writes.

For reads, like everyone loading the celebrity's profile and posts, the answer is caching. Hot keys go into Redis, and I'd even keep a tiny in-process cache on each app server for the top few hundred keys, so millions of reads never leave the app tier. The CDN handles their images.

For writes, like likes and comments pouring onto one post, I split the key:

// spread one hot counter over 10 sub-keys
const shard = Math.floor(Math.random() * 10);
await db.incr(`likes:post:${postId}#${shard}`);

// read: sum the 10 sub-keys (and cache the total for a few seconds)
const parts = await db.mget(Array.from({ length: 10 }, (_, i) => `likes:post:${postId}#${i}`));
const total = parts.reduce((sum, n) => sum + Number(n ?? 0), 0);

What happens here is that the 10 sub-keys hash to different shards, so the write load spreads out. The price is that reads have to gather all 10, which is why I only do this for keys I know are hot.

The third option is structural: give very large accounts their own dedicated shard via a directory lookup. And for the feed side of a celebrity, I'd switch from pushing their posts to every follower to pulling them at read time.

Interviewers look for: separate read and write fixes, and knowing that key-splitting makes reads more expensive.

So the best answer is to design so I rarely have to re-split data. I start with many logical shards, say 1,024, mapped onto a few physical machines. Growing then means moving whole logical shards to new machines, not re-hashing every row.

shard_id = hash(user_id) % 1024          // never changes
machine  = shard_map[shard_id]           // this is what changes

day 1:   shards 0-511 -> db-1,   512-1023 -> db-2
later:   shards 0-255 -> db-1,   256-511 -> db-3,  ...

Moving one logical shard, the steps I'd follow are:

• Start copying the shard to the new machine: a snapshot plus streaming the changes (logical replication or CDC).
• Once the copy has caught up, briefly block writes for just that shard, let the last changes drain, and flip shard_map to the new machine.
• Verify with row counts and checksums, keep the old copy for a while in case I need to roll back, then delete it.

The write pause is per logical shard and lasts seconds, so users barely notice. If even that isn't acceptable, I'd double-write to old and new locations during the migration, backfill, compare, and then switch reads.

This is roughly how Vitess does resharding for MySQL, and it's the reason I'd pick a large logical shard count from day one. It costs nothing early and saves a very painful migration later.

So a query without the shard key has to go to every shard and merge the results, a scatter-gather, which gets slower and less reliable as I add shards. The fix is usually a second copy of the data keyed the other way.

Let me give you the unique-email example. Users are sharded by user_id, but at signup I need to check "is this email taken?" and at login I look up by email. Neither has the shard key. So I keep a small lookup table sharded by email:

users_by_id     (shard key: user_id)   -> full user row
emails          (shard key: email)     -> email, user_id   UNIQUE(email)

signup:
  1. INSERT INTO emails (email, user_id)  -- unique check lives on one shard
  2. INSERT INTO users_by_id (...)
  if step 2 fails -> delete the email row (or retry)

What happens here is that the uniqueness check is now a single-shard operation. The trade-off is that the two inserts aren't in one transaction, so I need the cleanup or retry logic, or a background job that removes orphaned rows.

For more general cross-shard queries, like admin search or reporting, I don't fight the sharded database at all. I stream changes into Elasticsearch for search or into a warehouse like BigQuery or Snowflake for analytics, and query those instead.

So an index is a sorted lookup structure, usually a B-tree, on one or more columns. It lets the database jump straight to the matching rows instead of scanning the whole table. I don't index everything because every index has to be updated on every write and takes disk and memory.

Let me give you an example. On a 50-million-row orders table:

SELECT * FROM orders WHERE customer_id = 42;
-- no index: sequential scan of 50M rows, seconds
-- with index: B-tree lookup, a few page reads, about a millisecond

CREATE INDEX CONCURRENTLY idx_orders_customer ON orders (customer_id);

A B-tree lookup cost grows with log n, so going from 1 million to 50 million rows adds maybe one level to the tree. That's why the difference is so big.

On the cost side: one insert into orders with six indexes means seven writes. On a write-heavy table that adds up, and unused indexes also bloat memory that the hot data could be using. So I index the columns I filter, join and sort on, and I check pg_stat_user_indexes now and then for indexes with zero scans.

One practical thing: in Postgres I use CREATE INDEX CONCURRENTLY on a live table, because the plain version blocks writes while it builds.

Interviewers look for: the write cost of indexes, not just the read benefit.

So a composite index follows the leftmost prefix rule. It's sorted by user_id first, then by created_at within each user. So it helps any query that filters on user_id, or user_id plus created_at, but not created_at alone.

-- uses the index fully: equality then range, and the sort is free
SELECT * FROM posts WHERE user_id = 7 AND created_at > now() - interval '7 days'
ORDER BY created_at DESC LIMIT 20;

-- uses the index (leftmost column)
SELECT * FROM posts WHERE user_id = 7;

-- can't use it efficiently: skips the first column
SELECT * FROM posts WHERE created_at > now() - interval '1 day';

The way I picture it is a phone book sorted by last name, then first name. Finding "Smith, John" is fast, finding all Smiths is fast, but finding everyone called John means reading the whole book.

The rule I follow for column order is equality columns first, then the range or sort column. If I flipped it to (created_at, user_id), the first query would have to walk a week of everyone's posts and filter for user 7.

One nuance: newer Postgres (18) added skip scan, so it can sometimes use a composite index without the leading column when that column has few distinct values. I wouldn't design around it, but it's good to know.

Interviewers look for: leftmost prefix and "equality first, then range".

So my first step is always to look at the query plan with real numbers, not guess. EXPLAIN ANALYZE tells me what the database actually did and where the time went.

EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM orders WHERE status = 'pending' AND region = 'eu'
ORDER BY created_at DESC LIMIT 50;

-- Seq Scan on orders  (rows=12,400,000)  actual time=3,912 ms
--   Filter: status = 'pending' AND region = 'eu'
--   Rows Removed by Filter: 12,399,950

What that tells me is it read 12 million rows to return 50. So there's no usable index. The fix here would be an index that matches the filter and the sort, maybe a partial one, since pending orders are a small slice:

CREATE INDEX CONCURRENTLY ON orders (region, created_at DESC)
WHERE status = 'pending';

The "used to be fast" part matters too. Things I check: did the table grow past the point where the planner prefers a scan? Are statistics stale, so an ANALYZE fixes it? Did a recent deploy change the query, for example wrapping a column in a function like lower(email) so the index no longer applies? Is it lock contention rather than the plan?

To find which queries to look at in the first place, I use pg_stat_statements sorted by total time, because the worst offender is often a fast query that runs a million times, not the one slow report.

So I denormalise when a read path is hot enough that the joins are the bottleneck, and the data I'm copying changes rarely. The risk is that the copies drift out of sync, because now one fact lives in several places.

Let me give you an example from a feed. Each post shows the author's name and avatar. Normalised, that's a join with users for every post on every feed load. Denormalised, I store them on the post:

posts: id, author_id, author_name, author_avatar_url, body, created_at

Reads get simpler and faster, and on a sharded posts table I couldn't join to users anyway. But when a user changes their name, I now have to update every post they've ever written. In my last project we did that with an event: user.updated goes on a queue and a worker updates posts in batches. For a few seconds some posts show the old name, which is fine for that feature.

What I wouldn't denormalise is anything where a stale copy is a correctness bug, like a price used at checkout or a permission.

A middle ground is a materialised view, or in bigger systems CQRS, where the write model stays normalised and a separate read model is built from events. That way there's one source of truth and the denormalised copy is clearly derived.

Interviewers look for: a plan for keeping copies in sync, and knowing where staleness is acceptable.

So I'd climb the scaling ladder from the cheapest step up, and measure at each step. Sharding is last, because it's a one-way door that makes joins and transactions much harder.

1. Find the expensive queries. pg_stat_statements sorted by total time. In my experience a couple of missing indexes or an N+1 pattern are often most of the load.
2. Add a cache. Cache-aside in Redis for the hot, read-mostly data: product pages, user profiles, config.
3. Scale up. A bigger instance is a config change and buys time quickly.
4. Add read replicas. Move read traffic, reports and dashboards to followers, routing read-your-own-writes to the leader.
5. Denormalise hot paths or use materialised views for expensive aggregates.
6. Split by feature. Move orders, users, analytics into separate databases.
7. Shard the table that still doesn't fit on one writer.

Let me give you an example from a real case. We had a Postgres at 85% CPU. The top query was a dashboard count running every few seconds per open tab. Caching it for 30 seconds dropped CPU to 40%. Then a missing composite index on the orders list page took it to about 25%. We never needed step 3.

So in practice, most "the database can't cope" problems are solved on the first two rungs.

Interviewers look for: measuring first and not jumping to sharding.

So a B-tree updates data in place in a balanced tree of pages on disk, which makes reads very predictable. An LSM tree, a log-structured merge tree, buffers writes in memory and flushes them as sorted, immutable files, which makes writes very cheap but means a read may have to check several files.

It matters when I'm choosing a store for a write-heavy workload. For an IoT platform ingesting a million readings a second, an LSM-based store like Cassandra is the natural fit. The writes just append. For an app with complex queries and more reads than writes, a B-tree database is simpler and the reads are more consistent.

One gotcha with LSM stores is compaction. It uses a lot of disk I/O in the background, and if it falls behind, reads slow down because there are more files to check. And deletes are written as tombstones, so a pattern that deletes a lot can make reads slow until compaction cleans them up. I've seen a queue-like table in Cassandra get slow for exactly that reason.

So CAP says that when the network between my database nodes breaks, a partition, I have to choose: stay consistent by refusing or delaying some requests, or stay available by answering with data that might be stale. I can't have both during the partition.

Let me give you an example. Two replicas, one in each data centre, and the link between them goes down. A user in DC 1 changes their shipping address. A second later a request in DC 2 reads it.

CP choice: DC 2 can't confirm it has the latest data
           -> returns an error or waits
AP choice: DC 2 answers with the old address
           -> syncs when the link comes back

For a bank balance or seat booking, I want CP, because a wrong answer is worse than no answer. For a social feed or like count, AP is better, because a slightly stale number is fine and an error page isn't.

The important nuance is that C here means linearisability, every read sees the latest write, which is not the same as the C in ACID. And the choice only kicks in during a partition. Most of the time the network is fine and I get both.

Interviewers look for: "the real choice is C or A during a partition", with a concrete example of each.

So a CA system would be one that's consistent and available but doesn't tolerate partitions. The catch is that in any system with more than one machine talking over a network, partitions will happen: a switch fails, a cable gets cut, a GC pause makes a node look dead. I don't get to opt out. So P is a given, and the real choice is between C and A when it happens.

A single-node Postgres is sometimes called CA, but that's because it isn't distributed at all. The moment I add a replica and let clients read from it, I'm back to making the choice.

Let me give you what happens in practice with a Postgres leader and an async follower when the link between them drops:

leader:   keeps taking writes
follower: keeps serving reads, falling further behind  -> AP-ish for reads
if the leader then dies and we promote the follower     -> recent writes lost

If I made the follower synchronous, the leader would stop confirming writes during the partition. That's the CP behaviour.

So in an interview I just skip "CA" entirely and say: partitions are a fact of life, here's what each feature should do when one happens.

So CAP only talks about what happens during a failure. PACELC adds the normal case: if there's a Partition, choose Availability or Consistency; Else, when everything is healthy, choose Latency or Consistency. The point is that even with a perfect network, waiting for replicas to agree costs time on every request.

Let me give you a concrete sense of the latency cost. Spanner commits across regions have to wait for a majority of replicas, often in other regions, so a write can take tens of milliseconds more than a local write. DynamoDB with eventual reads answers from any replica immediately.

I find PACELC more useful day to day than CAP, because partitions are rare but latency is on every request. DynamoDB even lets me choose per read: ConsistentRead: true costs twice the read capacity and goes to the leader, so I only use it where it matters, like reading an order right after creating it.

Interviewers look for: recognising the latency cost of consistency in normal operation, not just during failures.

So between "every read sees the latest write" and "replicas agree eventually" there are a few useful guarantees, and in practice I usually need one of the middle ones, not full strong consistency.

Let me give you an example from a comments feature. Globally, eventual consistency is fine, other people can see my comment a second later. But I must see my own comment right after posting, so read-your-writes for the author. And replies must never show up before the comment they reply to, which is causal.

How I'd get those: read-your-writes by reading from the leader for the author for a short time; monotonic reads by pinning a user to one replica; causal by carrying a version or timestamp with the request and only reading from a replica that's caught up to it.

So in practice I pick the weakest model that doesn't produce a bug the user can see, because weaker is faster and more available.

So in a leaderless system each piece of data lives on N replicas. A write waits for W of them to confirm, and a read asks R of them and takes the newest version. If W + R > N, the set I read from must overlap the set that got the latest write, so at least one replica in my read has it.

N = 3, W = 2, R = 2   2 + 2 > 3   overlap guaranteed, survives 1 node down
N = 3, W = 1, R = 1   1 + 1 = 2   fast, may read stale data
N = 3, W = 3, R = 1   fast reads, but one dead node blocks all writes

Let me walk through the overlap. Replicas A, B, C. A write lands on A and B. A read asks B and C. B has the new value with a higher version, so the read returns it, and the coordinator can repair C in the background. That's read repair.

In Cassandra this is the consistency level per query: QUORUM for both reads and writes gives me the overlap. Across data centres I'd use LOCAL_QUORUM to avoid cross-region latency on every call, accepting that the other region may lag.

One gotcha: W + R > N is "strong-ish", not true linearisability. With concurrent writes, sloppy quorums during failures, or last-write-wins based on clock timestamps, you can still see anomalies. If I truly need linearisable behaviour, like a unique username, I'd use a consensus-backed operation, Cassandra's lightweight transactions or a different store, rather than rely on quorum maths.

Interviewers look for: the overlap argument, and awareness that quorums alone aren't full linearisability.

So I'd make the choice per feature, not per system. Anything where a wrong answer costs money or breaks a promise to the user is CP. Anything where a slightly stale answer is harmless and downtime is visible is AP.

The interesting one is inventory, because it's split. On the product page I read from a cache or replica and don't care if it's slightly off. At checkout I do a real conditional write on the leader:

UPDATE inventory SET stock = stock - 1
WHERE sku = 'ABC-123' AND stock > 0;
-- 0 rows updated -> sold out, tell the user

That's the approach Amazon described in the Dynamo paper for the cart: always writable, because refusing an add-to-cart loses sales, with conflicts merged later. And the order itself goes through a strongly consistent path.

So the senior answer is that one product has both, and I say out loud which data sits on which side.

So a queue lets the producer hand off work and move on without waiting for the consumer. That gives me four things: it absorbs traffic spikes, it moves slow work out of the request path, it survives the consumer being down, and it decouples the services so they don't need to know about each other.

Let me give you an example from a video upload feature. Without a queue, the upload API would transcode the video inline, the user waits a minute, and a spike of uploads takes the API down.

// API: accept, enqueue, return straight away
app.post("/videos", async (req, res) => {
  const video = await db.videos.create({ status: "processing", key: req.body.key });
  await queue.send("transcode", { videoId: video.id });
  res.status(202).json({ id: video.id, status: "processing" });
});

// Worker: pulls jobs at its own pace
worker.on("transcode", async ({ videoId }) => {
  await transcode(videoId);
  await db.videos.update(videoId, { status: "ready" });
});

What happens here is the API responds in milliseconds with 202 Accepted. If 10,000 uploads arrive in a minute, the queue just gets longer and the workers catch up. If the workers crash, the jobs wait. And I can scale workers separately based on queue depth.

The trade-off is that the system is now asynchronous: the user needs a way to find out when it's done, like polling a status or a push notification, and I have to think about retries and duplicates.

Interviewers look for: 202 Accepted plus a status mechanism, and naming spike absorption.

So with a point-to-point queue, each message goes to exactly one consumer, which is how I distribute work. With pub/sub, every subscriber gets its own copy of each message, which is how I broadcast that something happened.

Let me give you an example with an order being placed:

Pub/sub: topic "order.placed"
  -> email service       sends confirmation
  -> inventory service   reserves stock
  -> analytics service   records the sale
  each gets every event

Queue: "send-email"
  -> worker 1, worker 2, worker 3
  each email job handled by exactly one worker

Real systems usually combine them. With SNS plus SQS on AWS, the order service publishes to an SNS topic, and each interested service has its own SQS queue subscribed to it. Within one service, its workers compete on that queue. Kafka gives you the same thing with consumer groups: each group gets every message, and inside a group each partition goes to one consumer.

The nice property of pub/sub is that the order service doesn't know who's listening. When we added a fraud check later, it just subscribed to the topic, with no change to the order service at all.

So the core difference is that SQS and RabbitMQ are message queues, where a message is deleted once it's processed, and Kafka is a durable log, where messages stay for the retention period and any consumer can read or replay them from any offset.

Let me give you how I'd pick. For background jobs, like send this email or resize this image, I'd use SQS or RabbitMQ. I get per-message acks, visibility timeouts, delays and a dead-letter queue out of the box. For an event stream that several teams consume, where I want replay, say "rebuild the search index from the last 7 days of product events", or a high-volume pipeline like clickstream analytics, Kafka.

A gotcha with Kafka is that consumer parallelism is capped by partitions. With 12 partitions, the 13th consumer in the group just sits idle. So I pick the partition count with growth in mind.

Interviewers look for: "delete on ack vs retained log", and matching the tool to jobs versus event streams.

So it's about when I acknowledge. Ack before processing and a crash loses the message, that's at-most-once. Ack after processing and a crash means it's redelivered, possibly twice, that's at-least-once. Exactly-once means the effect happens once, and in practice I get it by combining at-least-once with an idempotent consumer.

Let me show you the idempotent consumer I'd write for a "payment succeeded" event:

async function handle(msg) {
  await db.tx(async (t) => {
    const inserted = await t.query(
      "INSERT INTO processed_messages (id) VALUES ($1) ON CONFLICT DO NOTHING",
      [msg.id]
    );
    if (inserted.rowCount === 0) return;          // already handled, skip

    await t.query("UPDATE orders SET status = 'paid' WHERE id = $1", [msg.orderId]);
  });
  await msg.ack();
}

What happens here is that the dedup record and the business change commit in the same transaction. If the worker crashes before ack, the message comes back, the insert hits the conflict, and we skip. The other trick is making the operation naturally idempotent: "set status to paid" is safe to repeat, "add 10 to balance" isn't.

Kafka does offer exactly-once semantics with transactions, but only for read-from-Kafka, write-to-Kafka flows. The moment I call an external API or a different database, I'm back to needing idempotency myself. So in practice I always design for at-least-once plus idempotency.

Interviewers look for: "exactly-once is at-least-once plus idempotency", with a concrete dedup mechanism.

So Kafka only guarantees order within a partition, not across a topic. To keep all events for one order in sequence, I use the order ID as the message key. Kafka hashes the key to pick the partition, so the same key always lands in the same partition, and one consumer in the group reads that partition in order.

await producer.send({
  topic: "orders",
  messages: [{ key: order.id, value: JSON.stringify({ type: "created", ...order }) }],
});
// later: key order.id, type "paid"     -> same partition, after "created"
// later: key order.id, type "shipped"  -> same partition, after "paid"

Different orders are spread across partitions and processed in parallel, which is what I want. I only need order per entity, not globally.

A few gotchas I've hit. If the consumer processes messages concurrently within a partition, say firing off async handlers without awaiting, I break the ordering myself. If I increase the partition count, the key-to-partition mapping changes, so events for an existing order can briefly be split across two partitions. And retries on the producer side can reorder unless the idempotent producer is on, which is the default in modern Kafka clients.

If someone asks for total global ordering, that means one partition, which means one consumer. I'd push back and ask what actually needs to be ordered relative to what.

So there are two problems here. The failing message is a poison message, and it needs to be moved out of the way to a dead-letter queue so it stops blocking or eating retries. The growing lag means consumers can't keep up, and that needs backpressure or more capacity.

For the poison message, I'd set a retry limit with backoff, then move it to a DLQ:

// SQS: after 5 failed receives, move to the DLQ automatically
RedrivePolicy: { deadLetterTargetArn: "arn:...:orders-dlq", maxReceiveCount: 5 }

Then I alert on DLQ depth, look at the message, fix the bug, and replay it. With Kafka there's no built-in DLQ, so the consumer catches the error, publishes to an orders.retry or orders.dlq topic, and commits the offset so the partition moves on.

For the lag, first I check why. Is it a slow downstream, like a database or third-party API? Or just more volume? Then the options are:

• Scale consumers on queue depth or lag, with KEDA on Kubernetes for example. With Kafka, only up to the partition count.
• Batch the work, like writing 500 rows per insert instead of 1.
• Apply backpressure upstream: rate-limit producers, or reject with 429 or 503 when the queue is past a bound.
• Shed or deprioritise low-value work, like analytics, before important work.

Interviewers look for: DLQ with alerting and replay, and a real backpressure strategy instead of "add more workers" alone.

So that's the dual-write problem. The database and the broker are two separate systems with no shared transaction. If the service crashes after the commit but before the publish, the order exists but nobody downstream ever hears about it. If I publish first and then the insert fails, I've announced an order that doesn't exist.

The fix I use is the transactional outbox. I write the event into an outbox table in the same database transaction as the order, and a separate process publishes from that table:

BEGIN;
INSERT INTO orders (id, customer_id, total) VALUES ($1, $2, $3);
INSERT INTO outbox (id, topic, key, payload)
VALUES (gen_random_uuid(), 'order.created', $1, $4);
COMMIT;

-- relay: read unpublished rows, publish to Kafka, mark as sent
-- or: Debezium tails the Postgres WAL and streams outbox rows to Kafka

What happens here is that the order and its event either both commit or neither does. The relay publishes at-least-once, because it can crash after publishing but before marking the row as sent, so consumers must be idempotent, keyed on the event ID.

In my last project we used Debezium for the change data capture, which avoids polling and keeps the order of events per key. The trade-off is one more moving part to run and monitor, but it's the standard answer and much safer than hoping two writes both succeed.

Interviewers look for: naming the dual-write problem and the outbox pattern; mentioning CDC is a bonus.

So I'd use REST for public APIs and most CRUD apps, GraphQL when many different clients need different shapes of nested data, and gRPC for internal service-to-service calls where performance and strict contracts matter.

Let me give you an example of why GraphQL helps. A mobile profile screen needs the user, their last 3 posts, and each post's comment count. In REST that might be:

GET /users/42
GET /users/42/posts?limit=3
GET /posts/1/comments/count   (x3)

Five round trips, and each response has fields I don't need. In GraphQL it's one request:

query {
  user(id: 42) {
    name
    avatarUrl
    posts(first: 3) { title commentCount }
  }
}

The trade-offs: GraphQL makes HTTP caching harder since it's usually POST to one endpoint, and a client can write a very expensive query, so I need depth limits and query cost analysis. gRPC uses Protocol Buffers over HTTP/2, so it's compact and fast with generated clients and streaming, but browsers can't call it natively without something like gRPC-Web or Connect.

So a typical setup in my experience: REST or GraphQL at the edge for clients, gRPC between the internal services.

Interviewers look for: over-fetching and under-fetching explained, plus a downside of each style.

So a good REST API uses nouns for resources and HTTP methods for actions, returns the right status codes, is versioned, and is predictable enough that you can guess an endpoint you haven't seen.

GET    /v1/users/42/orders?limit=20&cursor=abc   list, paginated
POST   /v1/orders                               create -> 201 + Location header
GET    /v1/orders/987                           read
PATCH  /v1/orders/987  { "note": "leave at door" }   partial update
PUT    /v1/orders/987  { ...full order... }          replace the whole thing
DELETE /v1/orders/987                           -> 204 No Content

PUT replaces the whole resource, so fields I leave out get cleared, and it's idempotent. PATCH changes only the fields I send. In practice I use PATCH for most updates.

401 means "I don't know who you are", no token or an invalid one, so log in. 403 means "I know who you are, and you're not allowed". Mixing them up confuses clients, because a 401 usually triggers a redirect to login.

The other codes I use a lot: 400 for bad input with a useful error body, 404, 409 for a conflict like a duplicate or a version mismatch, 422 for validation errors if the team prefers it, 429 with a Retry-After header, and 503 when we're shedding load.

And I always return errors in one consistent shape across the API, something like RFC 9457 problem details, so clients can handle them in one place.

So for a feed I'd use cursor pagination. Offset gets slow on deep pages and shows duplicates or skips items when new posts arrive, and both of those are exactly what happens in a feed.

Let me show the difference in SQL:

-- offset: the DB still reads and throws away 10,000 rows
SELECT * FROM posts ORDER BY created_at DESC, id DESC
LIMIT 20 OFFSET 10000;

-- cursor (keyset): jump straight there using the index
SELECT * FROM posts
WHERE (created_at, id) < ('2026-09-20 10:15:00', 98765)
ORDER BY created_at DESC, id DESC
LIMIT 20;

What happens with offset is that if 5 new posts arrive while I'm reading page 1, page 2 now starts 5 items earlier, so I see 5 duplicates. The cursor version says "give me items older than the last one I saw", so new posts at the top don't affect it, and with an index on (created_at, id) it's just as fast on page 500 as page 1.

One gotcha: the cursor needs a tie-breaker. If I only use created_at, two posts with the same timestamp can be skipped. Adding id makes it unique. I also encode the cursor as an opaque base64 string, so clients don't depend on what's inside it.

Offset is still fine for an admin table where people want "jump to page 7" and the data is small.

Interviewers look for: both problems with offset (performance and shifting items) and a tie-breaker in the cursor.

So for both of those I'd reach for Server-Sent Events first, because the data only flows from server to client. I'd use WebSockets when I genuinely need two-way, low-latency messages, like chat or collaborative editing.

Let me give you the AI streaming case, which is how most LLM APIs stream tokens:

app.get("/chat/stream", async (req, res) => {
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  for await (const token of llm.stream(req.query.prompt)) {
    res.write(`data: ${JSON.stringify({ token })}\n\n`);
  }
  res.write("event: done\ndata: {}\n\n");
  res.end();
});

The reason I like SSE is it's just HTTP. It goes through load balancers, proxies and auth middleware without special handling, and the browser's EventSource reconnects automatically. One gotcha I've hit: a proxy buffering the response, so tokens arrive in one lump at the end. With nginx I turn buffering off for that route.

So I'd require an idempotency key. The client generates a unique ID for the operation and sends it with every attempt. The server stores the key with the result, and if the same key comes in again it returns the stored result instead of charging again. That's how Stripe's API works.

app.post("/payments", async (req, res) => {
  const key = req.get("Idempotency-Key");
  if (!key) return res.status(400).json({ error: "Idempotency-Key required" });

  // claim the key atomically; only one request wins
  const claimed = await redis.set(`idem:${key}`, "pending", { NX: true, EX: 86400 });
  if (!claimed) {
    const saved = await redis.get(`idem:${key}`);
    if (saved === "pending") return res.status(409).json({ error: "In progress" });
    return res.status(200).json(JSON.parse(saved));
  }

  const payment = await charge(req.body);
  await redis.set(`idem:${key}`, JSON.stringify(payment), { EX: 86400 });
  res.status(201).json(payment);
});

What happens here is the NX set is atomic, so if two retries arrive together, only one proceeds. The second either gets "in progress" or, once finished, the same response as the first.

A few things I'd add in a real system: store the request body hash with the key and reject reuse of a key with a different body; store the key in the same database as the payment if possible, so they commit together; and handle the case where charge fails, by deleting the key or saving the error, so the client can retry properly.

Interviewers look for: the atomic claim to handle concurrent retries, and a TTL on stored keys.

So the hard part is that WebSockets are stateful. Each user is connected to one specific server, so when user A on server 1 sends a message to user B on server 7, server 1 needs a way to get it there. The pattern is a dedicated connection tier plus a pub/sub layer between the servers.

client --WS--> gateway-7 (holds B's socket)
                   ^
                   |  subscribe "user:B"
             [ Redis pub/sub / NATS / Kafka ]
                   |  publish "user:B" {msg}
client --WS--> gateway-1 (A sends a message)

What happens here is each gateway subscribes to channels for the users connected to it. Sending a message is just "publish to user:B", and whichever gateway holds B delivers it. I'd also keep a presence map, user to gateway, in Redis, which is useful for "online" status and for direct routing.

The other things I'd plan for:

• Connection capacity. A well-tuned Node or Go server can hold tens to hundreds of thousands of idle sockets. It's mostly memory and file descriptors. So a million users is roughly ten to twenty gateway boxes.
• Deploys. Restarting a gateway drops all its connections at once. I drain gradually and make clients reconnect with jittered backoff, or I get a reconnect storm.
• Missed messages. A socket can drop silently. Messages are stored first, and on reconnect the client asks for everything after its last seen ID.
• Heartbeats every 30 seconds or so, to detect dead connections and keep idle timeouts on load balancers from closing them.

Interviewers look for: the cross-server routing problem, and reconnect storms on deploys.

So my main rule is to make changes additive and backward compatible whenever I can, and only reach for a new version when a breaking change is truly unavoidable. Mobile apps make this matter a lot, because some users won't update for months.

Let me give you what counts as safe versus breaking:

For a rename, I'd do it in steps: add the new field alongside the old one, have both populated, move clients over, watch the traffic on the old field in the logs, then remove it after a deprecation period announced with a Deprecation and Sunset header.

When I really need a breaking change, I version. URL versioning, /v1 and /v2, is the easiest to see and route. Header or date-based versioning, like Stripe pinning each account to an API version date, is more flexible but more work to run. GraphQL mostly avoids versions: you add fields and mark old ones @deprecated. With gRPC, the rule is never reuse or renumber a protobuf field number.

And I'd back it with contract tests, like Pact or schema diffing in CI, so a breaking change can't slip through unnoticed.

So rate limiting caps how many requests a client can make in a time window, like 100 requests per minute per API key. Anything over the limit gets a 429 Too Many Requests with a Retry-After header telling the client when to come back.

I add it for three reasons. First, abuse: scraping, credential stuffing, someone brute-forcing a login form. Second, fairness: one noisy customer running a badly written script shouldn't slow things down for everyone else. And third, cost: if an endpoint calls a paid third-party API or an LLM, an unbounded client can literally cost us money.

HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1767225660

In my last project we had a public search endpoint that a partner's integration hammered with retries in a tight loop during an outage on their side. Our database CPU went to 100% and every other customer felt it. After that we put a per-key limit at the gateway, and the next time it happened only that partner got 429s.

So in practice I treat it as a default for any public API, not an optional extra.

Interviewers look for: the three motivations (abuse, fairness, cost) and knowing the 429 status plus the Retry-After header.

So a token bucket has two numbers: a capacity B and a refill rate R. The bucket starts full, every request takes one token, and tokens drip back in at R per second up to the cap. If the bucket is empty, the request is rejected.

Let me show you. I don't need a timer to refill it, I just compute the refill lazily from the elapsed time on each request.

class TokenBucket {
  constructor(capacity, refillPerSec) {
    this.capacity = capacity;
    this.rate = refillPerSec;
    this.tokens = capacity;
    this.last = Date.now();
  }
  tryTake(now = Date.now()) {
    const elapsed = (now - this.last) / 1000;
    this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.rate);
    this.last = now;
    if (this.tokens >= 1) { this.tokens -= 1; return true; }
    return false;
  }
}

const b = new TokenBucket(5, 1);   // burst of 5, 1 per second sustained
// 7 requests at the same instant: true x5, then false, false
// one second later: true, then false again

What I like about it is that the two parameters mean something to the business. Capacity is "how bursty can you be", so a page load that fires five calls at once is fine. Rate is the long-run average. And the state is tiny, just a token count and a timestamp per key, which fits nicely in a Redis hash.

That's why it's the one most APIs use, Stripe and AWS API Gateway included.

So a fixed window counter resets on the boundary, say every minute on the minute. The problem is the edge: a client can send 100 requests at 0:59 and another 100 at 1:00, and both windows are happy, but I just took 200 requests in two seconds.

There are two sliding fixes. The sliding window log stores a timestamp for every request and counts the ones in the last 60 seconds. It's exact, but memory grows with traffic, one entry per request. The sliding window counter is the cheap approximation I'd actually use: keep the previous and current window counts and weight the previous one by how much of it still overlaps.

limit = 100 per minute
previous minute: 80 requests
current minute:  30 requests so far, we're 25% into it

estimate = 80 x (1 - 0.25) + 30
         = 60 + 30
         = 90   -> allow, 90 < 100

The assumption baked in there is that the previous minute's requests were spread evenly, which isn't strictly true, but in practice the error is small and I only store two counters per key. Cloudflare has written about using exactly this approach at their scale.

So my ranking is: token bucket if bursts are fine, sliding window counter if I want a strict "N per minute", and the log only when the numbers are small and I need exactness.

Interviewers look for: the concrete boundary-burst example and the weighted formula, not just the names of the algorithms.

So if each instance keeps its own counter, a user who gets load-balanced round-robin effectively gets 20 times the limit. The counters have to live somewhere shared, and the usual answer is Redis.

The gotcha is race conditions. If I do "GET the count, check it, then SET it" from two gateways at once, both read 99 and both allow. So the check and the update have to be one atomic operation. For a fixed window that's just INCR plus EXPIRE; for a token bucket I'd put the logic in a Lua script, because Redis runs a script atomically.

-- KEYS[1] = "rl:user:42", ARGV = capacity, rate, now_ms
local b = redis.call("HMGET", KEYS[1], "tokens", "ts")
local cap, rate, now = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3])
local tokens = tonumber(b[1]) or cap
local ts = tonumber(b[2]) or now
tokens = math.min(cap, tokens + (now - ts) / 1000 * rate)
local allowed = 0
if tokens >= 1 then tokens = tokens - 1; allowed = 1 end
redis.call("HSET", KEYS[1], "tokens", tokens, "ts", now)
redis.call("PEXPIRE", KEYS[1], 60000)
return allowed

That's one network round trip per request, around a millisecond. If that's too much at huge scale, I can let each gateway keep a local allowance and sync with Redis in batches, trading a bit of accuracy for latency.

One more thing I'd call out: using the gateway's clock for now means clock skew between instances. Some implementations use Redis's own TIME to avoid that.

So my first choice is always the most specific identity I have: API key or user ID. IP address is the fallback for anonymous traffic, and I'm careful with it because it's a leaky proxy for "one person".

Let me give you an example of how I'd layer it:

The problem with IP is that a whole university or an office behind one NAT, or a mobile carrier using carrier-grade NAT, can share one address. I once saw a customer's entire office get locked out because we were limiting a signup page by IP during a training session. And in the other direction, an attacker with a botnet or rotating proxies has thousands of IPs.

For login specifically I limit per username too, because credential stuffing across many IPs still targets the same accounts. So in practice: identity first, IP as a coarse safety net, and different limits per endpoint based on how expensive or sensitive it is.

So that's a decision I'd make up front, not during the incident: does the limiter fail open or fail closed?

For general API limits I fail open. If Redis is unreachable, I allow the request, log it and fire an alert. The rate limiter exists to protect the API, and it would be ironic if a dependency of the limiter took the whole API down. A few minutes without limits is usually a much smaller risk than a total outage.

async function checkLimit(key) {
  try {
    return await withTimeout(redis.evalsha(sha, 1, key, ...args), 20);
  } catch (err) {
    metrics.increment("ratelimit.redis_error");
    return localFallback.tryTake(key);   // per-instance bucket, looser limit
  }
}

What I'd actually do is a middle ground: fall back to a local in-memory bucket per instance with the limit divided by the instance count. It's not exact, but it still stops a single runaway client.

For security-critical limits, like login attempts or OTP verification, I'd fail closed, because an attacker could deliberately wait for, or cause, the limiter to be down and then brute-force freely. The other key detail is the tight timeout, around 20 ms, on the Redis call: a slow Redis is worse than a dead one if every request waits on it.

Interviewers look for: an explicit fail-open vs fail-closed choice with a reason, plus a timeout on the limiter call itself.

So databases are built for small, structured rows that get queried and updated, not for multi-megabyte blobs. The rule I follow is: the file goes in object storage like S3, and the database stores only the metadata and the key that points to it.

CREATE TABLE files (
  id          uuid PRIMARY KEY,
  owner_id    bigint NOT NULL,
  s3_key      text NOT NULL,       -- "uploads/2026/09/7f3c.../avatar.png"
  size_bytes  bigint,
  mime_type   text,
  status      text DEFAULT 'pending',
  created_at  timestamptz DEFAULT now()
);

If I put the bytes in the database, a few things go wrong. Backups and restores balloon from gigabytes to terabytes. Replication has to ship every file to every replica. Big values push useful rows out of the buffer cache, so ordinary queries get slower. And the database is the most expensive storage I have, while S3 is cheap per gigabyte and built for eleven nines of durability.

Object storage also gives me things for free that I'd otherwise build: presigned URLs, lifecycle rules to move old files to cheaper tiers, and putting a CDN directly in front of it.

The one exception I'd make is tiny things like a 2 KB icon that's always read with the row. But as a default, the file lives in S3 and the table holds the ticket.

So a presigned URL is a temporary URL my server generates and signs with its credentials. It grants one specific operation, like "PUT to this exact key", for a short time, and the client can use it without ever having AWS credentials.

The flow I'd build looks like this:

// 1. Client: "I want to upload video.mp4, 2 GB"
// 2. API: check permission, create a metadata row, sign a URL
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const key = `uploads/${userId}/${crypto.randomUUID()}.mp4`;
await db.files.insert({ id, ownerId: userId, key, status: "pending" });

const url = await getSignedUrl(
  s3,
  new PutObjectCommand({ Bucket: "media", Key: key, ContentType: "video/mp4" }),
  { expiresIn: 900 }            // 15 minutes
);
// 3. Client PUTs the file straight to S3 using url
// 4. S3 event -> worker marks the row "uploaded", queues processing

The big win is that the 2 GB never touches my app servers. They only handle a tiny JSON request, and S3 absorbs the heavy bandwidth.

A couple of details I always get right. I generate the key on the server, so a client can't overwrite someone else's file. I confirm the upload from the S3 event rather than trusting the client's "done" call. And I treat the file as untrusted until a virus scan and type check pass, because the client could upload anything to that URL.

Interviewers look for: the file bypassing app servers, the server choosing the key, and a pending-then-uploaded status on the metadata row.

So for anything big, roughly over 100 MB, I'd use a multipart upload. The file is split into chunks, each chunk is uploaded separately and can be retried on its own, and S3 stitches them together at the end.

Let me walk through it with S3's multipart API:

1. POST CreateMultipartUpload        -> uploadId
2. For each 10 MB chunk (part 1..N), in parallel, say 4 at a time:
     PUT part with a presigned URL   -> ETag for that part
3. POST CompleteMultipartUpload(uploadId, [{PartNumber, ETag}, ...])

Network drops during part 37?
   -> retry part 37 only; parts 1-36 are already stored

What that buys me is resumability. On a phone switching from Wi-Fi to cellular, losing the connection 1.8 GB into a 2 GB upload would be brutal with a single PUT. With parts, the client just asks "which parts do you have?" with ListParts and continues. Uploading parts in parallel also makes it faster.

The limits worth knowing: parts are 5 MB minimum except the last one, 10,000 parts maximum, and objects up to 5 TB. I'd also send a checksum per part so corruption gets caught.

One gotcha I've hit: abandoned multipart uploads keep their parts, and you pay for that storage invisibly. So I always add a lifecycle rule to abort incomplete multipart uploads after, say, seven days.

So I'd split it into two paths: an asynchronous processing path that runs once per upload, and a read path that's almost entirely served from a CDN.

Upload:   client --presigned PUT--> S3 (originals bucket)
                                      | ObjectCreated event
                                      v
                                    queue --> resize worker
                                      -> 64px, 256px, 1024px WebP/AVIF
                                      -> S3 (public variants bucket)
                                      -> DB: avatar_version = 3

Read:     <img src="https://cdn.example.com/avatars/42/v3/256.webp">
          CDN hit (~95%+) or CDN -> S3 on a miss

The reason I resize in a worker rather than in the upload request is that image processing is CPU-heavy and slow, and I don't want the user staring at a spinner or my API servers pegged. The worker can also strip EXIF metadata, which matters because photos can contain GPS coordinates.

For caching, I put the version in the URL. That way I can set Cache-Control: public, max-age=31536000, immutable and never have to purge the CDN: a new photo gets a new URL.

An alternative that's very common now is on-the-fly resizing at the edge, like Cloudflare Images or an image CDN, where the URL carries the size. That removes the worker, but I'd still keep the original private and validated.

Interviewers look for: async processing via events and a queue, a CDN on the read path, and a cache-busting strategy.

So storage cost is mostly about matching each file to how often it's actually read. Most user files are hot for a few weeks and then almost never touched again, so I'd lean heavily on storage classes and lifecycle rules.

Lifecycle rule for bucket "user-files":
  after 30 days   -> Standard-IA        (cheaper storage, pay per read)
  after 180 days  -> Glacier Instant Retrieval
  after 365 days  -> Glacier Deep Archive (hours to restore)
  incomplete multipart uploads -> abort after 7 days
  noncurrent versions -> delete after 30 days

If I can't predict access patterns, S3 Intelligent-Tiering moves objects between tiers automatically for a small per-object monitoring fee, which is often the simpler answer.

Beyond tiering, there are a few other levers I'd pull. Deduplication: hash the content, and if two users upload the same file, store it once and reference-count it. Dropbox-style systems save a lot this way. Compression for text-heavy data. And cleaning up orphans: files whose metadata row was deleted but the object wasn't, which I'd catch with a periodic reconciliation job comparing S3 inventory to the database.

The trade-off to call out is retrieval: Deep Archive is extremely cheap but takes hours to restore, so it only fits data like old backups or compliance records, not anything a user might click on. And watch egress, since serving data out of the cloud is often the bigger bill, which is another reason for a CDN.

So LIKE with a leading wildcard can't use a normal B-tree index, because the index is sorted by the start of the string. The database has to scan every row, and even then it only does substring matching, with no ranking, no typo tolerance and no idea that "running" and "run" are related.

SELECT * FROM products WHERE title LIKE '%phone%';
-- Seq Scan on products (rows=5,000,000)
-- matches "phone", "headphones", "xylophone"
-- misses "phones"? no. misses "mobile"? yes. misses "iphnoe"? yes.
-- order of results: arbitrary

What real search needs is an inverted index: a map from each word to the list of documents containing it. Then the query "red shoes" becomes an intersection of two short lists instead of a table scan, and a scoring function like BM25 ranks the results so the most relevant ones come first.

In a small project I don't jump straight to Elasticsearch, though. Postgres has built-in full-text search with tsvector and a GIN index, and a trigram index with pg_trgm actually makes LIKE '%term%' indexable and handles fuzzy matching. For a few million rows that's often enough, with no extra system to run.

Interviewers look for: why the index can't help, what an inverted index is, and knowing when Postgres full-text is good enough.

So an inverted index flips the normal relationship. Instead of "document to its words", it's "word to the documents that contain it", like the index at the back of a cookbook.

Let me show you with three tiny documents.

Doc 1: "Red running shoes"
Doc 2: "Blue shoes for running"
Doc 3: "Red dress"

Analyse: tokenise, lowercase, stem ("running" -> "run"), drop stop words ("for")

  red   -> [1, 3]
  run   -> [1, 2]
  shoe  -> [1, 2]
  blue  -> [2]
  dress -> [3]

Query "red shoes" -> analyse the same way -> red, shoe
  red [1,3]  AND  shoe [1,2]  -> [1]

The important part is that the query goes through the same analyser as the documents. If I stem at index time but not at query time, "shoes" would never match "shoe".

The posting lists are kept sorted, so intersecting them is a fast merge. Real engines also store positions, so they can do phrase queries like "running shoes" in that order, and term frequencies for ranking.

For ranking, BM25 is the default in Elasticsearch and OpenSearch: a word counts more if it appears often in this document but rarely across all documents. "The" is worthless, "Bluetooth" is meaningful.

So my first principle is that Elasticsearch is never the source of truth. Postgres is. Search is a derived read model that I can always rebuild, and the question is just how changes flow into it.

The naive approach is dual writes: the app writes to Postgres and then to Elasticsearch. The problem is that if the second write fails, or the process crashes between them, the two drift apart silently. So I'd use one of these instead:

Option A: Change Data Capture
  Postgres WAL --> Debezium --> Kafka "products" topic --> indexer --> Elasticsearch

Option B: Outbox
  BEGIN;
    UPDATE products SET price = 49 WHERE id = 7;
    INSERT INTO outbox (topic, payload) VALUES ('product.updated', '{"id":7}');
  COMMIT;
  relay reads outbox --> Kafka --> indexer re-fetches product 7 --> ES

Both give me the guarantee that if the database commit happened, the event will eventually be delivered. The indexer should be idempotent, so indexing the same document twice is harmless, and it should use the row's version, so an older update arriving late doesn't overwrite a newer one.

The result is near real-time: typically a second or two of lag, which users accept for search. And I'd keep a full reindex job ready, building into a new index and switching an alias over, for mapping changes or if things ever drift.

Interviewers look for: naming the dual-write problem, then CDC or outbox, plus idempotent indexing and an alias-swap reindex.

So that's typo tolerance, and in Elasticsearch it's fuzzy matching based on edit distance: how many single-character insertions, deletions, substitutions or transpositions turn one word into another. "iphnoe" to "iphone" is one transposition, so edit distance one.

GET /products/_search
{
  "query": {
    "multi_match": {
      "query": "iphnoe case",
      "fields": ["title^3", "brand^2", "description"],
      "fuzziness": "AUTO"
    }
  }
}

What AUTO does is scale the allowed edits by word length: exact match for very short words, one edit for medium words, two for longer ones. That matters because allowing two edits on a three-letter word matches almost anything. The ^3 boosts mean a match in the title counts more than one buried in the description.

Fuzziness alone isn't the whole story, though. In my experience the biggest wins come from a few other things: a synonyms list ("tee" and "t-shirt"), a "did you mean" suggester for when zero results come back, and learning from query logs. If lots of people search "iphnoe" and then click iPhone products, that's a strong signal.

So in practice: fuzzy matching with AUTO, field boosts, synonyms, and I always track the zero-results rate as a metric, because that's where search quality problems show up first.

So an index is split into primary shards, each a self-contained Lucene index, and each primary has replicas on other nodes. Documents are routed to a shard by hashing their ID, and a search fans out to one copy of every shard.

A query goes through two phases, scatter-gather style:

Index "products": 5 primary shards x 1 replica = 10 shards on 5 nodes

Query "red shoes", top 10:
  1. Query phase:  coordinating node sends the query to 1 copy of each of 5 shards
                   each shard returns its top 10 IDs + scores
  2. Merge:        coordinator merges 50 candidates -> global top 10
  3. Fetch phase:  fetches the full documents for just those 10

A few consequences of that design I'd call out. Replicas scale reads and give fault tolerance, because either copy can answer. More shards spread writes, but every query pays a cost per shard, so over-sharding is a classic mistake: thousands of tiny shards make everything slower and use more heap. A common guideline is shards in the tens of gigabytes each.

The primary shard count is fixed when the index is created, so to change it I reindex into a new index and swap an alias. Deep pagination is also expensive: page 1,000 means every shard returns 10,000 results to merge, so I'd use search_after instead of large offsets.

And for time-based data like logs, I'd use one index per day or a data stream with rollover, so old data can move to cheaper nodes and be deleted by dropping whole indices.

Interviewers look for: scatter-gather with query and fetch phases, the cost of over-sharding, and deep pagination awareness.

So a monolith is one application deployed as one unit, with all features in one codebase and usually one database. Microservices split the product into small services, like users, orders and payments, each owning its own data and deployed independently, talking over the network.

The way I think about it is that microservices mostly solve an organisational problem. When you have 15 teams in one codebase, every deploy needs coordination and one team's bug blocks everyone's release. Splitting services lets each team own and ship its piece independently.

But you pay for it in distributed systems problems: partial failures, no cross-service joins, eventual consistency, and a lot more infrastructure for tracing, discovery and pipelines.

So for a small team or a new product, I'd start with a monolith. It's genuinely the faster, simpler choice, not a lesser one.

So an API gateway is the single front door that clients talk to. Instead of the mobile app knowing the addresses of twenty services, it calls one host, and the gateway routes each request to the right service.

client --> api.example.com (gateway)
             /users/*    --> user-service
             /orders/*   --> order-service
             /search/*   --> search-service

  on the way through:  TLS termination, JWT validation,
                       rate limiting, request ID, logging, CORS

The value is that all those cross-cutting concerns live in one place. Without a gateway, every service reimplements token validation and rate limiting, slightly differently, and one of them inevitably forgets. Kong, Envoy, AWS API Gateway and Nginx are the common choices.

A related pattern is BFF, Backend for Frontend: a gateway tailored to one client. In my last project the mobile app needed a single call that returned the user, their last five orders and unread notifications, trimmed down for small payloads, while the web app wanted richer data. A mobile BFF aggregated three service calls into one response.

The one thing I'd be careful about is putting business logic in the gateway. It should route and enforce policies. Once it starts containing domain rules, it becomes a bottleneck that every team has to change.

Interviewers look for: routing plus cross-cutting concerns, the BFF pattern, and not letting the gateway grow business logic.

So the first thing I'd do is ask why. If the reason is "microservices are modern", I'd push back. Good reasons are things like one module needing to scale very differently, teams blocking each other's deploys, or a piece needing a different technology or reliability level.

If the reasons are real, I'd do it incrementally with the strangler fig pattern rather than a rewrite:

Step 1  Modularise inside the monolith: clear module boundaries,
        no reaching into another module's tables.
Step 2  Put a gateway or proxy in front of the monolith.
Step 3  Extract one service (pick a well-bounded, low-risk one, e.g. notifications).
        Route /notifications/* to the new service, everything else to the monolith.
Step 4  Give it its own database; sync data via events during the transition.
Step 5  Repeat. The monolith shrinks until it's gone, or until it's just fine.

The step people skip is the first one, and it's the most valuable. If modules in the monolith share tables and call each other's internals everywhere, extracting a service just turns in-process spaghetti into network spaghetti, a distributed monolith, which is the worst of both worlds.

In practice I'd also make sure the platform basics exist before the first extraction: CI/CD per service, centralised logging, tracing, and service-to-service auth. And I'd measure: if deploy frequency and incident counts don't improve, the split isn't paying for itself.

Interviewers look for: questioning the motivation, the strangler fig pattern, and the "distributed monolith" warning.

So in a dynamic environment, instances start, stop and move constantly, so I can't hard-code IPs. Service discovery is a registry of what's alive right now, and callers ask it "where is the payment service?"

On Kubernetes this is mostly built in. A Service gets a stable DNS name and Kubernetes keeps the list of healthy pods behind it:

// order-service just calls a stable name:
await fetch("http://payment-service.prod.svc.cluster.local/charges", {...});
// kube-proxy / DNS resolves it to one of the ready payment pods

Outside Kubernetes, the classic tools are Consul, etcd or Eureka, where services register themselves with a heartbeat.

A service mesh goes a step further. It puts a small proxy, a sidecar like Envoy, next to every service, and all traffic goes through those proxies. That lets the platform handle retries, timeouts, mutual TLS, metrics and traffic splitting for canary releases, without any of it in application code. Istio and Linkerd are the main ones, and Istio now also has an "ambient" mode without per-pod sidecars to cut the overhead.

The trade-off is real complexity and some added latency per hop. So I'd say: a handful of services, Kubernetes DNS plus a good HTTP client library is plenty. Dozens of services in several languages with a need for mTLS everywhere, that's when a mesh starts earning its keep.

So my rule of thumb is: synchronous when the caller needs the answer right now to continue, asynchronous when it's announcing that something happened and doesn't care who reacts.

Let me give you an example with placing an order.

POST /orders
  sync:  order-service --gRPC--> inventory "reserve 2 x SKU 88"  (need yes/no now)
  sync:  order-service --> payment "authorise 49.00"             (need yes/no now)
  commit order + outbox row "OrderPlaced"
  return 201 to the user

async (Kafka topic "orders", event OrderPlaced):
  email-service      -> send confirmation
  analytics-service  -> update dashboards
  loyalty-service    -> add points
  shipping-service   -> create shipment

Why split it that way? Every synchronous call adds latency, and more importantly, availability multiplies. If order-service synchronously calls five services that are each 99.9% available, the chain is roughly 99.5%. If the email service is down, I don't want checkout to fail, I want the email to go out a bit late. The event lets it catch up from Kafka when it recovers.

Async also decouples teams: adding the loyalty service required zero changes to order-service, it just subscribed to the topic.

The cost is eventual consistency and harder debugging, since there's no single stack trace. So I'd keep the synchronous path short and critical, and make everything else event-driven, with a trace ID carried in the event headers.

Interviewers look for: a reasoned split of one flow into sync and async parts, and the availability-multiplies argument.

So a cascading failure is when one component failing or slowing down causes the things that depend on it to fail too, and it spreads upward until the whole system is down. The sneaky part is that it's usually triggered by something slow, not something dead.

Let me give you an example. Say the recommendations service starts taking 30 seconds instead of 50 ms.

product-page service: 200 worker threads, calls recommendations with no timeout
  normal:  50 ms per call   -> threads free up instantly
  now:     30 s per call    -> after a few seconds, all 200 threads are waiting
  result:  product pages time out -> homepage (which calls product-page) times out
           -> the whole site is down because a "nice to have" box got slow

A dead service fails fast with a connection error and the thread moves on. A slow one holds every thread hostage. That's why every outbound call needs a timeout. With a 300 ms timeout, the product page just renders without recommendations.

How I pick the value: look at the dependency's p99 latency and set the timeout a bit above it, and make sure the total fits the caller's own deadline. One gotcha I've hit is that many HTTP clients default to no timeout at all, or several minutes, so I always set it explicitly, often with AbortSignal.timeout(300) in Node.

Interviewers look for: recognising that slowness, not crashes, is what usually cascades, and that default timeouts are often infinite.

So the short version is: retry only transient errors, only idempotent operations, with exponential backoff plus jitter, and with a cap on attempts.

async function withRetry(fn, { retries = 4, base = 100, cap = 10_000 } = {}) {
  for (let attempt = 0; ; attempt++) {
    try {
      return await fn();
    } catch (err) {
      const transient = err.status >= 500 || err.status === 429 || err.code === "ECONNRESET";
      if (!transient || attempt >= retries) throw err;
      const delay = Math.min(cap, base * 2 ** attempt);   // 100, 200, 400, 800 ...
      await sleep(Math.random() * delay);                  // "full jitter"
    }
  }
}

Each piece is there for a reason. Exponential backoff gives a struggling service time to recover instead of hammering it. Jitter matters because if a thousand clients all failed at the same moment, without randomness they'd all retry at exactly 100 ms, then 200 ms, in synchronised waves. Randomising spreads them out.

I don't retry a 400 or a 404, because trying again won't change the answer. And I only retry things that are safe to repeat. A GET is fine, but retrying "charge the card" after a timeout could charge twice, unless the request carries an idempotency key that the server uses to dedupe.

If the server sent a Retry-After, I'd honour that over my own calculation.

So a circuit breaker wraps calls to a dependency and watches the failure rate. When failures cross a threshold, it "trips" and stops calling the dependency for a while, failing fast instead. It's like the fuse box at home.

CLOSED     normal; calls go through, failures are counted
  | failure rate > 50% over the last 10 s (min 20 calls)
  v
OPEN       no calls made; return the fallback immediately
  | after a 30 s cool-down
  v
HALF-OPEN  let a few trial calls through
  | success -> CLOSED       failure -> OPEN again

Why is that better than just timeouts? With only timeouts, every request still waits the full 300 ms before failing, and I'm still sending load to a service that's already drowning. An open breaker returns in microseconds and gives the dependency room to recover.

The part that makes or breaks it is the fallback. For a recommendations call, the fallback is a cached "popular items" list. For a pricing call there may be no safe fallback, so I'd return a clear error rather than show a wrong price.

In Node I'd use a library like opossum, in Java Resilience4j, or push it into the service mesh. And I'd always emit a metric on state changes, because a breaker flipping to open is one of the most useful alerts you can have.

Interviewers look for: all three states with the transitions, the need for a minimum call volume, and a meaningful fallback.

So a bulkhead isolates resources per dependency, so that one slow dependency can only exhaust its own pool, not everything. The name comes from ships: watertight compartments mean one hole floods one room.

Let me give you a scenario. An API service has one shared connection pool of 100 for outbound HTTP calls, used for both the payments provider and a slow third-party address-lookup API.

Without bulkheads:
  address API gets slow -> its calls hold all 100 connections
  -> payment calls queue behind them -> checkout fails

With bulkheads:
  pool "payments": 60 connections
  pool "address":  20 connections   (max 50 queued, then reject)
  pool "other":    20 connections
  address API gets slow -> only its 20 are stuck; payments unaffected

In Node, where it's not thread pools, the equivalent is a concurrency limiter per dependency, for example a separate HTTP agent with maxSockets per host, or a semaphore limiting in-flight calls.

The same idea applies at bigger scales too: separate worker pools for different queue priorities so a flood of bulk emails can't delay password-reset emails, or even separate deployments of the same service for critical and non-critical traffic.

The trade-off is some wasted capacity and pool sizes you have to tune. But for anything where one flaky integration sits next to a critical path, I think it's worth it.

So liveness answers "is this process stuck and does it need a restart?", while readiness answers "should this instance receive traffic right now?" Mixing them up causes real outages.

# Kubernetes
livenessProbe:            # fail -> container is restarted
  httpGet: { path: /healthz, port: 8080 }     # just: is the event loop responsive?
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:           # fail -> pod removed from the Service, not restarted
  httpGet: { path: /ready, port: 8080 }       # warmed up, DB pool connected?
  periodSeconds: 5

A readiness check is useful during startup, when the app is still loading config or warming caches, and during shutdown, when I want to drain connections. Failing readiness just stops new traffic.

The gotcha I've seen: someone made the liveness check call the database. Then the database had a brief blip, every pod's liveness failed at the same time, Kubernetes restarted all of them together, and a 10-second database hiccup became a several-minute full outage while everything cold-started. So liveness should check only the process itself. Dependencies belong, carefully, in readiness, or nowhere, since all pods failing readiness at once also means zero capacity.

There's also a startup probe for slow-starting apps, so liveness doesn't kill them before they've finished booting.

Interviewers look for: restart vs remove-from-rotation, and not putting shared dependencies in liveness checks.

So at that point the goal changes from "serve everything" to "keep the most important things working". I'd combine graceful degradation and load shedding, and ideally both are designed and rehearsed before the day.

Here's how I'd layer it in an e-commerce site:

Priority 1 (protect at all costs):  checkout, payment, login
Priority 2:                         product pages, cart, search
Priority 3 (turn off first):        recommendations, reviews, "trending",
                                    live stock counts, analytics events

Feature flags per P3 feature   -> switch to cached or hidden versions
Load shedding at the gateway   -> when CPU or queue latency passes a threshold,
                                  reject P3 then P2 requests early with 503
Waiting room / queue page      -> for flash sales, admit users at a fixed rate

The key to load shedding is rejecting early and cheaply. Rejecting at the gateway costs microseconds; letting a request get halfway through the stack before timing out wastes the capacity we're short of. Shedding based on queue wait time works better than on request count, because it directly measures "are we falling behind?"

Alongside that: autoscaling with pre-warmed capacity, because scaling from cold during a spike is too slow; caching aggressively, even serving slightly stale product pages from the CDN; and making sure retries from clients don't multiply the load, which is where backoff and retry budgets matter.

In my experience the biggest win is simply having the kill switches ready and tested in a game day beforehand.

Interviewers look for: prioritising features, shedding load early and cheaply, and preparation ahead of the event rather than improvising.

So it depends on what I need from the ID: just uniqueness, time-sortability, or shortness. Auto-increment breaks down once I have multiple database shards, so the usual choices are UUID v7 or a Snowflake-style ID.

Let me show you the Snowflake layout, since it's the classic answer.

64 bits: [ 1 unused | 41 bits ms since custom epoch | 10 bits machine | 12 bits sequence ]

const EPOCH = 1288834974657n;   // Twitter's epoch
function snowflake(ms, machine, seq) {
  return ((BigInt(ms) - EPOCH) << 22n) | (BigInt(machine) << 12n) | BigInt(seq);
}
snowflake(1767225600000, 7, 42)   // 2006515713438674986n

// decode: time = (id >> 22n) + EPOCH, machine = (id >> 12n) & 1023n, seq = id & 4095n

Each machine generates IDs with no coordination, up to 4,096 per millisecond, and because the timestamp is in the high bits, sorting by ID is sorting by creation time. That's great for feeds and for B-tree inserts, since new rows go at the end of the index.

If I don't need 64 bits, UUID v7 is my default today. It's also timestamp-prefixed, so it's sortable and index-friendly, needs no machine IDs, and it's now standardised in RFC 9562 with support in Postgres 18's uuidv7(). I'd avoid random UUID v4 as a primary key on big tables, because random inserts scatter across the index and hurt write performance.

Interviewers look for: the bit layout of Snowflake, why time-ordered IDs help indexes, and awareness of UUID v7.

So each machine's clock drifts, and NTP corrections can make a clock jump forward or even backward. Two servers can easily be tens of milliseconds apart, which is plenty to get cause and effect in the wrong order.

Let me give you an example.

Server A (clock 5 ms fast):   user edits comment   at 10:00:00.105
Server B (clock accurate):    user deletes comment at 10:00:00.103

Last-write-wins by timestamp:  "edit" wins -> the deleted comment comes back

The delete really happened after the edit, but the timestamps say the opposite. With last-write-wins conflict resolution, data just silently disappears or reappears.

What I'd use instead depends on the need. If a single database or a single partition leader orders the writes, that's the simplest fix, and it's why Kafka guarantees order only within a partition. For causal order across nodes, Lamport clocks give each event a counter that always increases along causal chains. Vector clocks go further and can tell me when two events were genuinely concurrent, which means a real conflict to resolve. And Google Spanner uses TrueTime, GPS and atomic clocks with a known error bound, and waits out the uncertainty.

So in practice, wall-clock timestamps are fine for display and logs, but I don't rely on them to decide which write wins.

So I can't have one database transaction across three services, and two-phase commit blocks when the coordinator fails, so in a microservices setup I'd use a saga: a sequence of local transactions, each with a compensating action that undoes it if a later step fails.

Orchestrated saga for PlaceOrder:
  1. order-service:     create order (status PENDING)
  2. inventory-service: reserve stock      compensate: release stock
  3. payment-service:   charge card        compensate: refund
  4. order-service:     mark CONFIRMED

Payment fails at step 3:
  -> release stock (undo 2) -> mark order CANCELLED (undo 1)

There are two styles. With choreography, each service reacts to events: inventory hears "OrderCreated", reserves, emits "StockReserved", and so on. It's loosely coupled but the flow is hard to see once it's more than three or four steps. With orchestration, a central orchestrator, often built on something like Temporal, tells each service what to do and tracks the state. For anything involving money, I prefer orchestration, because I can look in one place and see exactly where order 42 is stuck.

The trade-offs to be honest about: intermediate states are visible, so the user might briefly see "pending"; compensations must be idempotent because they can be retried; and some actions can't truly be undone, like a sent email, so I order the steps to put those last.

Interviewers look for: why not 2PC, compensating actions, choreography vs orchestration, and idempotent steps.

So the outbox pattern solves the "dual write" problem: I need to update my database and publish an event, and I need both to happen or neither.

If I write to Postgres and then publish to Kafka, a crash between the two means the order exists but nobody ever hears about it. If I publish first and then the DB write fails, downstream services react to an order that doesn't exist. The fix is to make the event part of the same local transaction.

BEGIN;
  INSERT INTO orders (id, user_id, total) VALUES (42, 7, 49.00);
  INSERT INTO outbox (id, topic, payload)
    VALUES (gen_random_uuid(), 'orders', '{"type":"OrderPlaced","orderId":42}');
COMMIT;

-- relay process (or Debezium reading the WAL):
--   read unpublished outbox rows -> publish to Kafka -> mark as sent

Now the commit is atomic: either both rows exist or neither does. The relay then guarantees the event eventually gets published. If it crashes after publishing but before marking the row sent, it'll publish again, so this is at-least-once delivery, which means consumers must be idempotent, usually by recording processed event IDs.

In my last project we used Debezium to tail the outbox table from the write-ahead log rather than polling, which gave low latency with no extra load from polling queries. And we deleted sent outbox rows on a schedule so the table didn't grow forever.

So the simple version is a Redis key set with NX and an expiry. Whoever manages to set it holds the lock, and the TTL makes sure a crashed holder doesn't lock things forever.

SET lock:payment:42 "worker-a:9f1c" NX PX 30000   -- OK means I got it

-- release ONLY if I still own it (atomic via Lua):
if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("DEL", KEYS[1])
else
  return 0
end

The release check matters: if my work took longer than 30 seconds, the lock expired and someone else took it, and a blind DEL would delete their lock.

The deeper problem is a process pause. Worker A gets the lock, then hits a long GC pause. The lock expires, worker B takes it and writes, then A wakes up still believing it holds the lock and writes too. No TTL can fix that on its own. The fix is a fencing token: the lock service hands out an increasing number with each grant, and the storage rejects any write carrying an older number than it has already seen.

So in practice, I first ask whether I need a lock at all. Often optimistic concurrency, UPDATE ... WHERE version = 7, or a unique constraint does the job. If it's only for efficiency, like avoiding duplicate work, Redis is fine. If correctness depends on it, I'd use etcd or ZooKeeper with fencing tokens.

Interviewers look for: owner-checked release, the GC-pause scenario, fencing tokens, and preferring to avoid locks.

So Raft is a consensus algorithm: it lets a group of nodes agree on a single leader and a single ordered log even when some nodes crash. The core rule is that every decision needs a majority.

Walking through an election:

5 nodes, all followers. Leader sends heartbeats every ~50 ms.
Leader crashes.
Node C's random election timeout (150-300 ms) fires first:
  C -> candidate, term 7 -> 8, votes for itself, asks the others
  A, D vote yes (each votes once per term)  -> 3 of 5 = majority -> C is leader
C sends heartbeats; others see term 8 and follow.

Writes: C appends to its log, replicates; committed once 3 of 5 have it.

The randomised timeout is what avoids everyone standing for election at once. The majority rule is what guarantees at most one leader per term, because two different majorities of the same cluster always overlap in at least one node, and that node only votes once. It also guarantees a committed entry is on at least one node of any future majority, so a new leader can't lose it.

On odd numbers: 3 nodes need 2 for a majority and tolerate 1 failure. 4 nodes need 3 and still tolerate only 1. So the fourth node adds cost and write latency without extra fault tolerance. 5 nodes tolerate 2.

And during a network partition, only the side with the majority can elect a leader and accept writes; the minority side stops, which is Raft choosing consistency over availability. In practice I wouldn't implement it, I'd lean on etcd, which is what Kubernetes itself uses.

Interviewers look for: terms, randomised timeouts, the overlapping-majority argument, and 2f + 1 nodes to tolerate f failures.

So the three pillars are logs, metrics and traces, and the way I think about them is that each answers a different question during an incident.

That's also the order I typically use them in. A metric-based alert tells me something's wrong. A trace narrows it down to one service or one call. Then the logs for that trace ID give me the actual error.

Metrics are cheap and aggregated, so I can keep them for a long time and alert on them, but they can't tell me about one specific user. Logs have full detail but are expensive at volume. Traces sit in between, and are usually sampled.

The glue that makes it all work is a shared request or trace ID. If the log lines and the trace carry the same ID, I can jump straight from a slow trace to its logs. Today I'd instrument with OpenTelemetry, since it's the vendor-neutral standard for all three, and send the data to whatever backend the team uses.

So the SLI is the measurement, the SLO is our internal target for that measurement, and the SLA is the external promise to customers with a penalty attached.

SLI:  fraction of checkout requests that succeed in under 300 ms
SLO:  99.9% over a rolling 30 days          (internal target)
SLA:  99.5% monthly, or customers get a 10% credit   (contract)

Error budget = 100% - 99.9% = 0.1%
  on 10 million requests a month -> 10,000 bad requests allowed
  in time terms, 0.1% of 30 days = about 43 minutes

The SLA is deliberately looser than the SLO, so we get warned and can react before we owe anybody money.

The part I find most useful is the error budget. It turns a fight between "ship faster" and "be more careful" into a number. If we've only used 20% of this month's budget, the team can ship aggressively. If a bad deploy burned 90% of it, we slow down, freeze risky releases and spend time on reliability.

One thing I'd push for is making the SLI measure what users actually feel, ideally at the load balancer or the client, not "the server process is up". A server can be up and returning errors to everyone.

Interviewers look for: the correct ordering of the three, the error budget, and SLA being looser than the SLO.

So a good log line is structured, so a machine can query it, it carries the context to connect it to a request, and it doesn't leak anything sensitive.

// bad
console.log("payment failed for user " + user.email + " " + err);

// good (pino)
logger.error({
  msg: "payment authorisation failed",
  traceId: ctx.traceId,
  userId: user.id,          // an ID, not the email
  orderId: order.id,
  provider: "stripe",
  errorCode: err.code,      // "card_declined"
  durationMs: 812,
});
// {"level":50,"time":1767225600000,"msg":"payment authorisation failed",
//  "traceId":"4bf92f35...","userId":7,"orderId":42,...}

With the structured version, I can ask "show me all card_declined errors for Stripe in the last hour, grouped by count" in Loki or Datadog. With the string version, I'm writing regexes at 2 a.m.

A few other habits: log levels used consistently, so I can drop DEBUG in production; one log line per meaningful event rather than dozens per request; and never logging passwords, tokens, full card numbers or raw PII. In my last project we added a redaction config to the logger so fields like authorization and password were masked automatically, because relying on everyone remembering didn't work.

Cost is the other consideration. Logs are usually the biggest observability bill, so I'd sample noisy success logs and keep all errors.

So each request gets a trace ID at the edge, and every unit of work along the way is a span with its own ID, a parent span ID, and start and end times. The IDs are propagated between services in request headers, so the backend can stitch all the spans back into one tree.

Incoming request, W3C Trace Context header:
  traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
               ver  trace-id (same everywhere)       parent span id   sampled

Trace 4bf92f35...                                  total 1,240 ms
  api-gateway      GET /checkout                   [==========================]
    order-service  createOrder                       [======================]
      postgres     INSERT orders                      [=]
      payment-svc  POST /authorise                        [=================] 980 ms
        stripe     HTTP call                                [===============]

Reading that waterfall, it's immediately obvious the time is going into the Stripe call, which I'd never see from the gateway's latency metric alone.

With OpenTelemetry, most of this is automatic: the auto-instrumentation for HTTP, gRPC, database drivers and Kafka clients creates spans and injects and extracts the traceparent header. The gotcha is async boundaries. If I publish to a queue, I have to put the trace context in the message headers, or the trace breaks in two.

At scale I'd sample, because tracing every request is expensive. Tail-based sampling is the nicer option: decide after the request finishes, so I keep 100% of errors and slow requests but only 1% of the boring fast ones.

Interviewers look for: trace and span IDs, header propagation (traceparent), and sampling strategy.

So that's alert fatigue, and it's dangerous because the one real alert gets ignored along with the noise. My fix is to page only on symptoms users feel, based on SLOs, and to demote everything else.

First I'd audit the alerts: for each one, how often did it fire, and how often did someone actually need to do something? Anything that isn't actionable gets deleted or turned into a ticket or a dashboard.

Then I'd replace cause-based alerts like "CPU above 80%" with SLO burn-rate alerts. High CPU with happy users isn't an emergency.

SLO: 99.9% success over 30 days  -> error budget 0.1%

Page (wake someone up):
  burn rate > 14.4x over 1 h AND over 5 min
  -> at this pace, 2% of the monthly budget is gone in an hour

Ticket (next business day):
  burn rate > 1x over 3 days
  -> slow leak, the budget will run out by the end of the month

Using two windows together, a long one and a short one, is what makes it both reliable and quick to reset: the long window proves it's significant, the short one proves it's still happening.

Every page gets a runbook link with the first diagnostic steps. And I'd review pages weekly as a team: any alert that woke someone up without needing action gets fixed that week. In a previous team that took us from dozens of pages a week to a handful, and people started trusting the pager again.

Interviewers look for: symptoms over causes, SLO burn-rate alerting with multiple windows, runbooks, and a feedback loop.

So authentication is "who are you?", proving identity with a password, a passkey or an SSO login. Authorisation is "what are you allowed to do?", checked on every action after that.

Let me give you an example of where people get it wrong.

// authenticated, but NOT authorised -- broken access control
app.get("/api/orders/:id", requireLogin, async (req, res) => {
  const order = await db.orders.findById(req.params.id);
  res.json(order);            // user 7 can read order 42 belonging to user 9
});

// fixed: ownership check on the server
app.get("/api/orders/:id", requireLogin, async (req, res) => {
  const order = await db.orders.findOne({ id: req.params.id, userId: req.user.id });
  if (!order) return res.status(404).end();
  res.json(order);
});

The first version authenticates perfectly, but anyone logged in can change the ID in the URL and read someone else's order. That's called IDOR, insecure direct object reference, and broken access control is number one in the OWASP Top 10 for exactly this reason.

I return 404 rather than 403 there so I don't even confirm the order exists.

For models, RBAC, role-based like admin, editor, viewer, covers most apps. ABAC, attribute-based like "owner of the document and in the same org", handles finer rules. Either way, the check lives on the server, never just hiding a button in the UI.

Interviewers look for: the clear distinction, plus a concrete IDOR example with a server-side ownership check.

So passwords are never stored, not even encrypted. I store a slow, salted hash made with an algorithm built for passwords, like Argon2id or bcrypt. Secrets like API keys and database passwords go in a secrets manager, never in the code or the repo.

import argon2 from "argon2";

// sign-up
const hash = await argon2.hash(password);          // Argon2id by default, random salt built in
// "$argon2id$v=19$m=65536,t=3,p=4$...salt...$...hash..."

// login
const ok = await argon2.verify(user.passwordHash, attempt);

Why slow? A fast hash like SHA-256 can be computed billions of times a second on a GPU, so if the table leaks, most passwords get cracked quickly. Argon2 is deliberately expensive in time and memory, and the random salt means identical passwords get different hashes, which kills precomputed rainbow tables. Encryption would be worse, because whoever gets the key gets every password back.

For secrets, I'd use something like AWS Secrets Manager or Vault, inject them at runtime, rotate them on a schedule, and give each service access only to its own. In my last project a developer committed an .env file to a public repo once; bots found the AWS key within minutes. Since then I always add secret scanning in CI and push protection on the repo.

So with sessions, the server stores the session, say in Redis, and the browser just holds a random ID in a cookie. With a JWT, the token itself carries the user ID and claims, signed, so any service can verify it without a lookup.

For a typical web app with one backend, I'd pick sessions in an HttpOnly, Secure, SameSite=Lax cookie. They're simple, and instant revocation really matters when an account is compromised.

For microservices or third-party APIs, JWTs make sense, with a pattern like this:

access token:  JWT, 15 min, signed RS256/ES256, verified by any service via JWKS
refresh token: opaque, 30 days, stored server-side, rotated on each use
logout / compromise -> revoke the refresh token; access token dies within 15 min

The gotchas I'd mention: don't put JWTs in localStorage, where any XSS can read them; always pin the expected algorithm when verifying; and remember a JWT is signed, not encrypted, so anyone can decode and read the payload.

Interviewers look for: the revocation trade-off, short-lived access plus refresh tokens, and cookie flags.

So OAuth 2.0 is a way for an app to get limited access to a user's account on another service without ever seeing their password. On its own it's about authorisation; OpenID Connect adds an ID token on top, which is what makes it a login.

For a web or mobile app, the flow I'd use today is Authorization Code with PKCE:

1. App creates code_verifier (random) and code_challenge = SHA256(verifier)
2. Redirect user to Google:
     /authorize?client_id=...&redirect_uri=...&scope=openid email
               &state=xyz&code_challenge=...&code_challenge_method=S256
3. User logs in at Google and consents
4. Google redirects back: https://myapp.com/callback?code=abc&state=xyz
5. App checks state, then POSTs code + code_verifier to Google's token endpoint
6. Google returns access_token + id_token (a JWT with sub, email, ...)
7. App verifies the id_token, finds or creates the user by "sub", starts a session

The pieces each close a specific hole. The state parameter stops CSRF on the callback. PKCE means an attacker who intercepts the code can't exchange it without the verifier, which is why it's required for mobile and SPAs and recommended everywhere in OAuth 2.1. And the password only ever goes to Google.

One practical detail: I link accounts by Google's sub claim, not by email, because emails can change. And the older implicit flow, returning tokens directly in the URL, is deprecated, so I wouldn't use it.

So in transit, TLS everywhere, including inside the network, and at rest, everything encrypted with keys managed by a KMS. The mindset is zero trust: being inside the VPC doesn't make a request trustworthy.

Internet --TLS 1.3--> CDN / WAF --TLS--> API gateway (authn)
   gateway --mTLS--> order-service --mTLS--> payment-service
                         |                        |
                    Postgres (TLS,           S3 (SSE-KMS)
                    encrypted storage)
   keys: KMS, envelope encryption, automatic rotation
   secrets: Secrets Manager, per-service IAM roles

Mutual TLS between services means both sides present certificates, so payment-service knows the caller really is order-service, and an attacker who gets onto the network can't just curl it. Doing that by hand is painful, which is one of the main reasons teams adopt a service mesh: it issues and rotates the certificates automatically.

At rest, the KMS uses envelope encryption: data is encrypted with a data key, the data key is encrypted with a master key that never leaves the KMS, and every use is audit-logged. That also makes crypto-shredding possible: destroy a tenant's key and their data is unreadable, which helps with GDPR deletion in backups.

On top of that I'd apply least privilege, so each service gets an IAM role that can only reach its own database and bucket. Encryption protects against stolen disks; least privilege limits what a compromised service can reach.

Interviewers look for: mTLS inside the network, KMS with envelope encryption, and least privilege alongside encryption.

So I'd answer in layers, from the edge inward, because each layer handles a different kind of attack and nothing should reach the expensive parts of the system if it can be stopped earlier.

Layer 1  Edge / CDN (Cloudflare, AWS Shield + CloudFront)
         absorbs volumetric floods (L3/L4, SYN floods) on a huge anycast network
Layer 2  WAF rules
         block known-bad patterns, bot scores, geo or ASN rules, challenge suspicious clients
Layer 3  API gateway rate limits
         per IP, per API key; login: per username + per IP, fail closed
Layer 4  Application
         MFA, breached-password checks, lockout with backoff, CAPTCHA after N failures
Layer 5  Capacity
         autoscaling + load shedding so legitimate traffic keeps working

The two attacks need different things. A volumetric DDoS is a bandwidth problem, so it has to be absorbed by a network much bigger than mine; no amount of app-level code helps if the pipe is full. That's why the edge provider is the first line.

Credential stuffing is quiet by comparison: attackers try leaked username and password pairs, often from thousands of residential IPs, a few attempts each. Per-IP limits barely notice. What works is limiting per account, checking passwords against known breach lists, MFA, and bot detection signals at the WAF. And I'd monitor the login failure rate as a metric, since a sudden spike is usually the first sign.

Finally, I'd make sure the origin servers only accept traffic from the CDN, so attackers can't bypass all of that by hitting the origin IP directly.

Interviewers look for: defence in depth, distinguishing volumetric attacks from application-level ones, and locking the origin to the CDN.

So before I draw anything, I want to pin down scope, because "URL shortener" can mean a toy or bit.ly with analytics. There are really four things I'd ask.

1. Features:  create + redirect only? custom aliases? expiry? click analytics?
2. Scale:     how many new links a month? read:write ratio?
3. Codes:     any length limit? must they be unguessable?
4. Users:     anonymous creation, or logged-in accounts with a dashboard?

Let's say the answers are: create and redirect, optional custom alias and expiry, analytics would be nice, 100 million new links a month, and roughly 100 redirects per link created. Then I'd write the non-functional side on the board: redirects must be very fast, say under 50 ms, and highly available, because a broken short link in a marketing email is a real outage for the customer. Creation can be a bit slower. And codes must be unique.

What that tells me straight away is that this is a read-heavy key-value problem. Almost everything I design after this follows from that one sentence: cache the reads, keep the write path simple, and make ID generation collision-free.

Interviewers look for: that you stop and scope before designing, and that you name the read:write ratio, because it drives every later decision.

So I'd keep the maths rough and round. The point is the order of magnitude, not the exact number.

Writes:   100M / month ÷ ~2.5M sec     ≈ 40/sec   (peak ~200)
Reads:    100 × writes                  ≈ 4,000/sec (peak ~20,000)
Rows:     100M × 12 × 5 years           = 6B
Storage:  6B × ~500 bytes               ≈ 3 TB
Codes:    base62 → 62^6 ≈ 56.8B,  62^7 ≈ 3.5 trillion

What that tells me: 40 writes a second is nothing, any database handles it. 20,000 reads a second at peak is where the design effort goes, and that's a caching problem. 3 TB fits in a small sharded cluster, or even one big box with replicas.

For the code length, I need at least 6 billion unique codes. Six base62 characters gives about 56.8 billion, which technically works, but I'd pick 7 for 3.5 trillion. That leaves loads of headroom and makes random guessing much harder. One more character costs nothing.

So in practice I'd say "7 characters of base62" and move on. I don't want to burn five minutes on arithmetic.

So the API is tiny: one write endpoint, one redirect, and a delete for owners.

POST   /api/urls   { longUrl, customAlias?, expiresAt? }  → 201 { shortUrl }
GET    /{code}     → 302 Location: longUrl   (or 404 / 410 if expired)
DELETE /api/urls/{code}                                   → 204

urls
  code        VARCHAR(10) PRIMARY KEY
  long_url    TEXT
  user_id     BIGINT NULL
  created_at  TIMESTAMP
  expires_at  TIMESTAMP NULL

The thing I'd point out is that there's exactly one hot query: "give me the long URL for this code." That's a primary-key lookup. So I don't need joins or secondary indexes on the hot path, and either Postgres or a key-value store like DynamoDB works. I'd probably start with Postgres because it gives me a unique constraint on code for free, which matters for custom aliases, and move to a KV store only if scale demands it.

If we add a user dashboard, I'd add an index on user_id, but that's a cold path and can even be served from a replica.

Interviewers look for: recognising the single access pattern and choosing storage from it, not from habit.

So this is the real deep dive. There are three options, and I'd go with counter ranges plus base62.

Let me show the encoding, it's just converting a number to base 62:

const A = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
function toBase62(n) {
  let s = "";
  do { s = A[n % 62] + s; n = Math.floor(n / 62); } while (n > 0);
  return s;
}
toBase62(125);       // "21"
toBase62(1000000);   // "4c92"

The trick for coordination is ranges. Each app server asks a coordinator, ZooKeeper or just a row in Postgres, for a block of a million IDs, then hands them out from memory. No two servers ever clash and nobody waits on a network call per request. If a server dies, we waste some of its range, which is fine with 3.5 trillion codes.

If guessability matters, I'd scramble the counter with a reversible bit shuffle before encoding, so consecutive links don't look consecutive.

Interviewers look for: the trade-off table, then a clear pick with the range trick. Mentioning guessability is a senior signal.

So it comes down to whether we care about analytics. A 301 is permanent: the browser caches it and next time goes straight to the long URL without asking us. A 302 (or 307) is temporary, so every click comes through our servers.

301  → less load on us, faster for repeat clicks, but we can't count them
         and we can't change or expire the link for that browser
302  → every click hits us: we can count, expire, or edit the target

In my view, for a bit.ly-style product the answer is 302, because click analytics is the product. It also means expiry and "disable this malicious link" actually work, which a cached 301 would ignore.

If this were an internal shortener with no analytics and massive traffic, I'd happily use 301 and let browsers take the load off us.

One gotcha: I'd still set a short Cache-Control on the 302 at the CDN layer if I push redirects to the edge. That way the CDN can absorb a viral link for a few seconds without losing the ability to change it.

Interviewers look for: linking the status code to a product requirement (analytics, expiry), not just reciting the definitions.

So redirects are 99% of the traffic, so they get cache-aside in Redis in front of the database, and for really hot links a CDN in front of that.

GET /aZ3k9x1
  → CDN edge (hot links, short TTL)
  → LB → app server
      → Redis GET code          hit  → 302
                                miss → DB by primary key
                                       → SET in Redis (TTL, LRU)
                                       → 302
  → click event → Kafka (async, never blocks the redirect)

What happens here is that link popularity follows a power law. A tiny fraction of links gets most of the clicks, so even a modest Redis cluster with LRU eviction gets a very high hit rate. A miss is a single primary-key lookup, a few milliseconds.

The analytics part is important. I never want a counter update in the request path, because then a slow analytics DB slows down every redirect. So the app server fires a click event onto Kafka and returns. A consumer aggregates counts per code per minute.

I'd also cache negatives briefly: if someone hammers a code that doesn't exist, I store "not found" for a minute so they can't bypass the cache and hit the DB.

So custom aliases share the same namespace as generated codes, so the danger is a user picking abc1234 and the counter later generating the same string.

There are a couple of clean ways out. My preferred one is to make the namespaces disjoint: generated codes are always exactly 7 characters, and custom aliases must be a different length or include a character the generator never uses, like a hyphen. Then they can't collide by construction. On top of that, the insert relies on the primary key constraint:

INSERT INTO urls (code, long_url, user_id)
VALUES ('my-launch', 'https://...', 42);
-- unique violation → 409 Conflict "alias taken"

For expiry, I check expires_at on read and return 410 Gone. When I cache an entry, I set the Redis TTL to no longer than the time until expiry, so the cache can't serve an expired link. A nightly job deletes expired rows in batches.

The question I'd raise is whether expired codes can be reused. I'd say no, or at least not for a long time, because old links live in emails and PDFs forever, and sending someone to a stranger's page is a security issue.

Interviewers look for: spotting the alias-versus-generated collision, and the "don't recycle codes" instinct.

So I'd go through it component by component.

Abuse is the one people forget. In practice, shorteners are a favourite tool for hiding phishing links. So on creation I'd rate-limit, scan the target against a blocklist asynchronously, and keep a disabled flag that the redirect checks. That's another reason for 302: I can kill a bad link and every click respects it.

So to summarise out loud: counter ranges and base62 for codes, Redis and a CDN for reads, a DB sharded by code, 302 plus Kafka for analytics, and rate limiting and scanning for abuse.

So I'd first ask what we're protecting and from whom, because that changes where the limiter lives and how strict it has to be.

- Limit by what?     user ID, API key, IP, or all three?
- Rules:             one global limit, or per endpoint / per plan tier?
- Scope:             per server, or global across the whole fleet?
- Accuracy:          is a small overshoot OK, or must it be exact?
- Failure:           if the limiter breaks, block traffic or allow it?

Let's say: per API key and per IP, different limits per endpoint and per pricing tier, global across 50 gateway instances, and a little overshoot is fine. Non-functionally, it must add under 5 ms, and if it breaks, the API should keep working.

Then I'd say the key insight: the algorithm itself is easy, a token bucket is a few lines. The hard part is that the count has to be shared across every server and updated fast and atomically. That's what I'd spend the interview on.

Interviewers look for: "global versus per-server" and "fail open or closed" raised up front, without being asked.

So my default is token bucket, and if they want strict fairness, a sliding-window counter.

A token bucket holds up to N tokens and refills at a steady rate. Each request takes one. Let me give you an example with capacity 5, refilling one token a second:

function tokenBucket(capacity, perSec) {
  let tokens = capacity, last = 0;
  return (nowMs) => {
    tokens = Math.min(capacity, tokens + (nowMs - last) / 1000 * perSec);
    last = nowMs;
    if (tokens >= 1) { tokens -= 1; return true; }
    return false;
  };
}
const allow = tokenBucket(5, 1);
// 7 requests at t=0     → true ×5, then false, false
// 2 requests at t=1000  → true, false

What happens is the first five go through as a burst, then the client is held to one a second. That burst behaviour is exactly what real clients need, like an app firing a few calls on startup.

Fixed window is simplest but lets through double the limit around the window boundary: 100 at 0:59 and 100 at 1:00. Sliding-window log is exact but stores a timestamp per request, too much memory. The sliding-window counter blends the current and previous window and is a nice middle ground.

So I'd put it as middleware in the API gateway, with the counters in a shared Redis cluster and the rules cached in each gateway.

Client → LB → API gateway [rate-limit middleware] → services
                  │            │
                  │            └─ rules (config DB, cached, refreshed every 30s)
                  └─ Redis cluster  key = rl:{apiKey}:{endpoint}
                        allowed? → forward
                        over?    → 429 + Retry-After

The reason I'd pick the gateway is that it sees every request before it costs us anything, and it's one place to enforce global rules. But not every rule belongs there. Business rules like "three password resets per hour per account" need domain context, so those live inside the auth service, often using the same Redis-backed library.

On the response, I'd return 429 Too Many Requests with Retry-After, and on every response the X-RateLimit-Limit and X-RateLimit-Remaining headers, so well-behaved clients can slow themselves down.

Memory is small: ten million keys at about 50 bytes is around 500 MB, so one Redis cluster holds it comfortably. The counters use a TTL equal to the window so they clean themselves up.

So that's a classic read-then-write race. The fix is to never read and write in two steps. The check-and-increment has to happen as one atomic operation inside Redis.

For a fixed or sliding-window counter, plain INCR is already atomic. It returns the new value, so I compare after incrementing:

-- KEYS[1] = rl:key42:/search:1727340000   ARGV[1] = limit   ARGV[2] = window (s)
local n = redis.call("INCR", KEYS[1])
if n == 1 then
  redis.call("EXPIRE", KEYS[1], ARGV[2])
end
if n > tonumber(ARGV[1]) then
  return 0
end
return 1

What happens here is that Redis runs the whole Lua script as a single step. No other command interleaves, so two gateways can't both see 99. Setting the expiry inside the script also avoids a nasty bug I've seen: if you INCR and then EXPIRE as two calls and the process dies in between, the key never expires and that user is blocked forever.

For a token bucket I'd do the same thing: store tokens and last in a hash and do the refill-and-take maths inside one Lua script.

Interviewers look for: "atomic in the store" via INCR or Lua, not a distributed lock, which would be far too slow.

So it depends on the endpoint, and I'd make it a per-rule setting: fail open or fail closed.

try {
  allowed = await limiter.check(key, rule, { timeoutMs: 5 });
} catch {
  allowed = rule.failMode === "open";   // search, feed → open
}                                        // login, OTP, payments → closed

For normal APIs I fail open. The limiter exists to protect the service, and if it takes the whole API down with it, it's done more damage than any abusive client. For login attempts, OTP sends or payment calls, I fail closed, because an unlimited window there is a brute-force or SMS-cost attack.

Two more things. The call to Redis gets a tight timeout, a few milliseconds, and a circuit breaker so we stop trying for a while instead of adding latency to every request. And while Redis is down, I can fall back to a local in-memory limiter per gateway with the limit divided by the number of gateways. It's rough, but it's much better than no protection.

And of course Redis itself runs with replicas and automatic failover, so a full outage should be rare.

So at that point I'd trade a bit of accuracy for latency, with local counters that sync to Redis in batches.

each gateway:
  local map  key → count since last sync
  allow if (lastGlobalCount + localCount) < limit
  every 100–500 ms: INCRBY each key by localCount in one pipeline,
                    read back global count, reset localCount

What happens here is that most requests are decided purely in memory, and Redis sees one pipelined batch per gateway every few hundred milliseconds instead of one call per request. The cost is that gateways don't see each other's traffic until the next sync, so a client spraying across many gateways can overshoot for a moment. For API quotas that's usually fine.

Other things I'd consider: shard the Redis keys by key hash across a cluster so no single node is hot; use consistent hashing at the load balancer so one API key tends to land on the same gateway, which makes local counts much more accurate; and for DDoS-level floods, drop at the edge (CDN or WAF) before it reaches us at all.

Interviewers look for: naming the accuracy-versus-latency trade explicitly and saying when overshoot is acceptable.

So rules are data, not code. I'd store them in a config store and have each gateway cache them locally.

rules:
  - match: { plan: "free",  endpoint: "/api/search" }   limit: 60   per: 1m
  - match: { plan: "pro",   endpoint: "/api/search" }   limit: 1000 per: 1m
  - match: { endpoint: "/auth/login", by: "ip" }         limit: 5    per: 15m  failMode: closed

Gateways load the rules at startup and refresh every 30 seconds, or subscribe to a change notification. The request path never calls the config DB, it only reads the in-memory copy, so the rules store being slow or down doesn't hurt latency.

The plan comes from the API key or the JWT, which the gateway already validated, so looking up the right rule is a hash-map lookup.

One thing I'd be careful about in a real rollout: when you tighten a limit, you can suddenly 429 a big customer. So in my last project we added a "shadow" mode per rule, where it logs what would have been blocked without blocking. We ran new rules in shadow for a few days, checked who would be hit, then turned them on.

So chat apps have a huge surface, so I'd agree on the core first and park the rest.

In:   1:1 and group chat (up to ~500), sent / delivered / read ticks,
      online + last seen, history on any device, image/video messages
Out (mention only): voice/video calls, stories, payments, E2E crypto details

Non-functionally: delivery under about 200 ms when the receiver is online, messages must never be lost, messages must be in order within a chat, and very high availability.

Then the rough estimates, because they drive the design:

500M DAU × 40 msgs/day  = 20B msgs/day ≈ 230K/sec (peak ~1M)
Text storage            ≈ 20B × 100 B = 2 TB/day
Concurrent connections  ≈ 100M; ~50K per server → ~2,000 chat servers

That last line is the interesting one. It tells me this is a connection-holding problem, not a request-response one. I'll need persistent connections, a lot of servers that just hold sockets, and a way to find which server any given user is on.

Interviewers look for: "never lose a message" and "order within a chat" stated as requirements, since the deep dives hang off them.

So the server needs to push messages to the client the moment they arrive, and HTTP by itself only lets the client ask.

For chat, traffic flows both ways all the time: sends, incoming messages, receipts, typing indicators. A WebSocket does all of that over one connection with tiny frame overhead.

The cost is that chat servers become stateful. They hold the connection, so I can't just route any request to any server. That's why the design needs a presence registry that says which server each user is on, and why I keep chat servers thin: hold sockets, route messages, nothing else.

On mobile, when the app is in the background the OS kills the socket anyway, so offline delivery falls back to push notifications through APNs and FCM.

So the rule I follow is: save first, then ack, then deliver. Here's the flow.

Alice ──ws──▶ Chat server A
               1. store in Cassandra (chat_id, msg_id)   → ack to Alice  (one tick)
               2. presence lookup: Bob → server B?
               3a. online:  Kafka (partition = chat_id) ─▶ Chat server B ──ws──▶ Bob
                            Bob's app ─▶ "delivered" receipt ─▶ Alice   (two ticks)
               3b. offline: add msg_id to Bob's inbox + push via APNs/FCM
Bob opens chat ─▶ "read" receipt ─▶ Alice   (blue ticks)

What happens here is that the ack to Alice only goes out after the message is durably stored. So if server A crashes right after, the message still exists and will reach Bob when he syncs. That's how I get "never lose a message."

Step 2 is the part people skip. Bob's socket lives on some other server out of two thousand. The presence store in Redis maps user_id → chat server, and the message is routed there through the queue.

When Bob comes back online, his app says "my last synced msg_id per chat is X", and the server returns everything after it from Cassandra. So reconnect and offline sync are the same code path.

Interviewers look for: store-before-ack, and the presence lookup to find the receiver's server.

So I'd use a wide-column store like Cassandra or ScyllaDB, partitioned by chat_id and clustered by a time-ordered msg_id.

CREATE TABLE messages (
  chat_id    bigint,
  msg_id     bigint,        -- Snowflake: time-ordered, unique
  sender_id  bigint,
  body       blob,          -- ciphertext if E2E
  created_at timestamp,
  PRIMARY KEY ((chat_id), msg_id)
) WITH CLUSTERING ORDER BY (msg_id DESC);

-- the hot query
SELECT * FROM messages WHERE chat_id = ? AND msg_id < ? LIMIT 50;

The reason I'd pick it: the workload is a huge volume of writes, around 230K a second, and almost every read is "latest 50 messages in this chat, then scroll back." With this key, one chat's messages sit together on disk, already sorted, so that query is a single-partition range scan. And Cassandra scales writes horizontally by adding nodes.

A relational DB would struggle with that write rate unless I sharded it heavily myself, and I don't need joins here.

One gotcha: a group chat that's been active for years becomes one giant partition. I'd bucket it, so the partition key becomes (chat_id, month), keeping partitions bounded.

So those are two separate problems with two separate tools: a time-ordered ID for ordering, and a client-generated ID for deduplication.

For ordering, every message gets a Snowflake-style ID: timestamp bits, then machine bits, then a sequence. Sorting by it gives time order. And Kafka partitioned by chat_id means all messages in one chat go through one partition, which preserves order in transit. I only promise order within a chat, which is what users actually see. Global order would be expensive and pointless.

For duplicates, mobile networks drop acks all the time, so the client retries. Without protection, Bob sees the same message twice. So the client attaches a clientMsgId, a UUID, and the server treats it as an idempotency key:

on send(chatId, clientMsgId, text):
  existing = lookup(chatId, clientMsgId)
  if existing: return ack(existing.msgId)     // retry: just re-ack
  msgId = snowflake()
  store(chatId, msgId, clientMsgId, text)
  return ack(msgId)

What happens is the retry gets the same ack back and nothing is stored twice. The receiver's app also dedupes by msgId, since at-least-once delivery can push the same message twice.

Interviewers look for: "order per chat, not globally", and idempotency keys for retries.

So presence is heartbeats plus a timeout. The app sends a heartbeat every 30 seconds or so over the WebSocket, and the chat server updates Redis.

SET presence:{userId} {"server":"chat-117","lastSeen":1727340000} EX 90
// no heartbeat for 90s → key expires → user is offline, lastSeen stays in DB

What happens here is I don't rely on a clean disconnect event, because phones lose signal in tunnels and never say goodbye. The TTL handles that: if heartbeats stop, the key expires and the user is offline.

The hard part isn't tracking, it's broadcasting. If every status change is pushed to all of a user's contacts, a user with 500 contacts flapping online and offline on a train creates a storm of events. So I'd only push presence to people who currently have a chat with that user open on screen. Everyone else fetches it lazily when they open the chat.

I'd also debounce: don't announce "offline" until a few seconds have passed, so a quick reconnect doesn't flicker. And there's the privacy setting for hiding last seen, which is just a check before returning it.

So for normal groups, I store once and fan out delivery. For huge groups or channels, I switch to pull.

small group (≤ ~500):
  store message once in messages(chat_id = group)
  for each member: presence lookup → route to their server, or push/inbox

huge channel (100K+):
  store once
  no per-member fan-out; online members' clients are subscribed
  to the channel topic, offline members pull "since msg_id X" on open

The key point is that the message body is stored once per group, not once per member. Only the delivery fans out. For a 200-person group that's 200 routing operations per message, which is fine.

For a 100K channel, per-member delivery means 100K operations for every message, and read receipts would be 100K events back. So I'd drop individual read receipts to an aggregate count, not push to offline members at all, and let clients catch up by pulling. It's the same idea as the celebrity problem in a news feed.

Media in any group goes to object storage through a presigned URL once, and the message just carries the link, so the fan-out stays small.

Interviewers look for: knowing where fan-out stops scaling and switching models, rather than one design for all group sizes.

So because I saved before acking, no message is lost. The question is just how fast everyone recovers.

t=0     chat-117 dies, 50K sockets drop
t+1-5s  clients detect the drop, reconnect with jittered backoff
        LB sends them to healthy servers
        new server: update presence → user now on chat-042
        client: "sync since last msg_id per chat" → fetch gap from Cassandra
t+90s   stale presence keys for anyone who didn't come back expire

What happens to messages in flight? If one was being routed to a user on chat-117, the push fails, but the message is already in Cassandra. When that user reconnects, the sync picks it up. At worst they get it a few seconds late.

The gotcha I'd raise is the thundering herd. Fifty thousand clients reconnecting in the same second can knock over the next server too. So clients reconnect with exponential backoff plus random jitter, and the load balancer spreads them out.

A related case is deploys. You can't just restart chat servers, because every restart is this same event. So in practice I'd drain them: stop accepting new connections, tell clients to reconnect gradually over a few minutes, then shut down.

So the core is: users post, they follow people, and they see a home feed of posts from people they follow. I'd confirm whether it's chronological or ranked, and whether likes and comments are in scope.

Non-functionally, the feed must load fast, under 200 ms, and it's fine if a new post takes a few seconds to show up. That eventual-consistency answer matters a lot, because it gives me permission to do work asynchronously.

300M DAU × 10 feed opens      = 3B reads/day  ≈ 35K/sec
300M × 0.5 posts              = 150M/day      ≈ 1,700 posts/sec
avg 200 followers             → push model ≈ 340K feed inserts/sec
feed cache: 500 IDs × 8 B     ≈ 4 KB/user × 300M ≈ 1.2 TB in Redis

What these numbers tell me: reads outnumber posts about 20 to 1, so I want feeds precomputed. But the fan-out number, 340K inserts a second, shows precomputing has a real cost, and that's before celebrities with 100 million followers. So the whole interview is really about one decision: fan-out on write versus fan-out on read.

Interviewers look for: getting to the fan-out decision quickly, with numbers that motivate it.

So I'd keep it small.

POST   /posts                          { text, mediaIds[] }
GET    /feed?cursor={lastPostId}&limit=20
POST   /users/{id}/follow
DELETE /users/{id}/follow

posts       post_id (Snowflake), author_id, text, media_urls, created_at
follows     follower_id, followee_id    -- indexed both ways
feed cache  Redis list feed:{userId} → [post_id, post_id, ...]  (latest ~500)

A few choices I'd call out. Post IDs are Snowflake IDs, so they're time-sortable, which makes both ordering and cursor pagination simple.

The follow graph needs both directions: "who does X follow?" for building a feed on read, and "who follows X?" for fan-out on write. I'd store it as two tables or two indexes, each partitioned by the first column.

And the feed cache holds IDs only, not post content. That keeps it at 4 KB per user instead of hundreds, and if someone edits or deletes a post, I change one row in the post cache instead of millions of feeds.

Media goes to object storage and a CDN, uploaded directly by the client with a presigned URL.

So neither on its own. I'd go hybrid, but let me show why each fails alone.

The hybrid: normal users, say under 100K followers, get pushed. A fan-out worker takes the new post ID and LPUSHes it onto each follower's Redis list. Celebrities don't get fanned out at all. When I open my feed, the feed service reads my prebuilt list, then pulls recent posts from the handful of celebrities I follow and merges them in.

GET /feed
  ids  = LRANGE feed:{me} 0 499             // pushed posts
  ids += recent posts of celebrities I follow  // pulled, cached per celebrity
  sort by post_id desc, take 20 → hydrate from post cache

What makes this work is that a celebrity's recent posts are the same for all their followers, so they're cached once and read millions of times.

Interviewers look for: "hybrid" with a follower threshold and the merge on read. This is the headline answer.

So the write path is: save the post synchronously, fan out asynchronously.

POST /posts
  → post service: insert into posts DB, add to post cache → 201 to author
  → publish PostCreated{postId, authorId} to Kafka
      → fan-out workers:
          if author.followers > 100K: stop (pulled at read time)
          else: page through followers in batches of 1,000
                skip inactive users (no login in 30 days)
                pipeline LPUSH feed:{f} postId ; LTRIM feed:{f} 0 499

What happens here is the author gets their response right after the DB insert. The expensive part, writing into hundreds of feeds, happens in workers off Kafka, so it can lag a few seconds without anyone noticing, and a spike in posts just queues up.

Two optimisations I'd mention. Skipping inactive users saves a huge amount of wasted work; when they come back, I build their feed on demand with a pull. And LTRIM keeps each list capped at 500, so Redis memory stays bounded.

One gotcha: the author should see their own post instantly. Since fan-out is async, I insert it into their own feed synchronously or just add it client-side. That's read-your-writes for the one person who'd notice.

So cursor-based, never offset-based.

Let me give you an example of why offset breaks. I load page 1, posts 1 to 20. While I'm reading, 5 new posts arrive at the top. Now I request offset=20, but everything has shifted down by 5, so I see 5 posts again that I already saw.

// offset: duplicates when new posts arrive
GET /feed?offset=20&limit=20

// cursor: "give me 20 older than the last one I saw"
GET /feed?cursor=1839204711290880&limit=20
  → { posts: [...], nextCursor: "1839204702390272" }

With a cursor, the client passes the ID of the last post it saw, and the server returns the next 20 with a smaller ID. New posts at the top don't affect it at all. And because my post IDs are Snowflake IDs, "older than" is just "ID less than", which is fast in both Redis and the DB.

For new posts at the top, the client asks separately with "anything newer than my first ID" and shows a "5 new posts" pill rather than jumping the scroll.

Once someone scrolls past the 500 IDs in Redis, I fall back to building older pages from the DB. Very few users go that deep, so that path can be slower.

Interviewers look for: a concrete reason offset fails, not just "cursors are better."

So I'd keep everything I've built as the candidate generation step and add a ranking step on top.

GET /feed
  1. candidates: ~500 IDs from feed cache + celebrity pulls (+ maybe recommendations)
  2. features:   author affinity, post age, likes/comments velocity, media type
  3. score:      ranking service (a trained model) → score per candidate
  4. rules:      dedupe, author diversity, remove seen, ads slots
  5. return top 20, remember the ranked list for this session's pagination

The reason for the two stages is cost. I can't score every post on the platform per request, but scoring 500 candidates in a few tens of milliseconds is fine.

One gotcha I'd raise: pagination gets tricky, because scores change between requests. If I re-rank on page 2, posts can jump pages and the user sees duplicates or misses posts. So I'd rank once per session and cache that ordered list, and page through it with an index cursor.

And I'd log impressions and interactions to Kafka, which feeds the training data for the model and the "already seen" filter. In practice the model is its own team's problem; in the interview I'd just show where it plugs in.

So the biggest ones are celebrity fan-out, which the hybrid handles, fan-out lag during spikes, and Redis memory, which the 500-ID cap handles. Deletes and unfollows are the interesting edge cases.

For a delete, I don't chase the ID through millions of Redis lists. I mark the post deleted in the posts DB and invalidate the post cache. At hydration time, any ID whose post is gone gets filtered out. So the feed lists contain some dead IDs for a while, which costs nothing.

hydrate(ids):
  posts = postCache.mget(ids)
  return posts.filter(p => p && !p.deleted && visibleTo(me, p))

For an unfollow, same idea: filter at read time by checking the author is still in my follow set, and let the old IDs age out of the list. If it matters a lot, a background job can scrub that author from my one list, which is cheap because it's one user.

That visibleTo check also handles blocks and privacy changes. The principle is: fan-out is a cache of candidates, and the source of truth is checked on read.

Interviewers look for: filtering at read time instead of trying to keep millions of precomputed feeds perfectly consistent.

So I'd frame it as a platform: other services say "tell user 42 their order shipped", and this system decides how, respects preferences, and delivers reliably.

Channels:   push (APNs, FCM), SMS, email, in-app
Features:   templates, user preferences + opt-outs, quiet hours,
            scheduled and bulk sends, delivery status
Non-func:   at-least-once (no silent loss), no visible duplicates,
            seconds latency, survive spikes (10M marketing blast)

For estimates, the average is small, about 16 million a day, roughly 200 a second. But a marketing campaign can mean 50,000 a second for a few minutes. The spike is the number that matters, and that's the first clue the design needs queues.

I'd also ask about priority up front. A one-time login code and a "20% off" email go through the same system, but the OTP must arrive in seconds and the promo can wait an hour. That distinction changes the whole queue layout.

Interviewers look for: designing for the spike, not the average, and separating critical from bulk traffic early.

So callers send the intent and the data, not the rendered message. The notification system owns templates and channels.

POST /notifications
{
  "userId": 42,
  "type": "order_shipped",
  "data": { "orderId": 4471 },
  "channels": ["push", "email"],       // optional override
  "priority": "normal",
  "idempotencyKey": "order-4471-shipped"
}
→ 202 Accepted { "notificationId": "ntf_8f2..." }

It's 202 and not 200 because I haven't delivered anything yet. I've validated it and put it on a queue. Delivery happens asynchronously, maybe seconds later, maybe after retries. Returning 200 would suggest it's done. The caller can check status with the notificationId if it cares.

The reason callers send a type plus data instead of text is that the order service shouldn't know that user 42 prefers Spanish, has push disabled, and is in quiet hours. All of that belongs in one place.

And the idempotencyKey means the order service can safely retry the call on a timeout without the customer getting two "your order shipped" messages.

So there's one entry service doing the thinking, then queues per channel, then dumb workers that talk to providers.

Services ─▶ Notification API
              validate → dedupe (idempotencyKey)
              → load prefs, devices, quiet hours
              → per-user rate limit → render template
              ─▶ queue per channel × priority
                   push.high  push.bulk  ─▶ push workers  ─▶ APNs / FCM
                   sms.high              ─▶ sms workers   ─▶ Twilio
                   email.high email.bulk ─▶ email workers ─▶ SES / SendGrid
              ◀─ provider webhooks (delivered, bounced, opened) → notification_log

The reason for a queue per channel is isolation. If the email provider slows down, the email queue backs up, but push keeps flowing. With one shared queue, a slow provider would block everything behind it.

Splitting by priority is the same idea. A 10-million-user campaign goes into the bulk queues, so password reset codes in the high queue don't wait behind it. The high queues get their own worker pool.

The workers are simple on purpose: take a message, call the provider, ack or retry. All the decisions happen before queueing, so workers scale horizontally with no shared state.

Interviewers look for: per-channel and per-priority queues, with the isolation reason stated.

So the rule is: a worker only acks the queue message after the provider has accepted it. Until then the message stays in the queue.

worker loop:
  msg = queue.receive()
  try:
    provider.send(msg)                      // 2xx from APNs/Twilio/SES
    log(msg.id, "sent"); queue.ack(msg)
  catch transient (timeout, 429, 5xx):
    if msg.attempts < 5: retry after 2^attempts s + jitter
    else: dead-letter queue + alert
  catch permanent (invalid token, bad number):
    log(msg.id, "failed"); queue.ack(msg); disable device token

What happens here is that if the worker crashes mid-send, the message becomes visible again and another worker picks it up. That's at-least-once.

The distinction I'd stress is transient versus permanent errors. Retrying a timeout makes sense. Retrying "this device token is no longer registered" five times is pointless, so I mark the token dead and stop sending to it.

After max attempts, messages go to a dead-letter queue so someone can inspect them instead of them vanishing. For critical channels like OTP, I'd also have a fallback provider: if Twilio is failing, route SMS to a second provider.

So I deduplicate at two points: when the request comes in, and right before the worker calls the provider.

// on ingest: caller retried the API call
SET dedupe:order-4471-shipped 1 NX EX 86400
  → OK   : new, continue
  → nil  : already accepted, return the same notificationId

// in the worker: message redelivered after a crash
SET sent:{notificationId}:{channel} 1 NX EX 86400
  → nil  : already sent, just ack and skip

What happens here is that SET NX is atomic, so even if two workers get the same message at once, only one wins the key and sends.

I'd be honest that there's still a tiny window: the worker calls the provider, the provider sends, and the worker crashes before recording it. Then a retry sends again. You can't fully close that without the provider supporting idempotency keys. Some do, and then I'd pass the notificationId through. Otherwise, the goal is "no duplicates the user notices", and a rare double push is acceptable in a way a lost OTP isn't.

Interviewers look for: admitting exactly-once isn't achievable end to end, and using idempotency keys to get close.

So I'd treat a campaign as a batch job that feeds the bulk queue at a controlled rate, not 10 million API calls at once.

campaign service:
  select audience in chunks of 10K (by user_id range)
  → for each chunk: filter opted-out / quiet hours / recently notified
  → enqueue to push.bulk at a paced rate (e.g. 20K/sec)
push.bulk workers → FCM/APNs (batch APIs, respect provider limits)

The reason for pacing: providers have their own rate limits, and if we blast everything at once we get throttled and retries pile up. Also, if the push says "come to the app", 10 million people opening it in the same minute is a self-inflicted DDoS on our own backend. Spreading it over 10 or 15 minutes protects the app servers.

Quiet hours matter here: 9 a.m. in London is 1 a.m. in California. So I'd schedule per time zone, or hold those users until their morning.

And because this lands in the bulk queue with its own workers, OTP codes and security alerts on the high-priority queue go out on time during the blast. In my last project we learned that one the hard way: a newsletter delayed password reset emails by 20 minutes until we split the queues.

So all three happen around the queue: preferences and rate limits before, tracking after.

before queueing:
  prefs = cache.get(user)                      // opted_in per channel + type
  if !prefs.allows(type, channel): drop
  if inQuietHours(user) && priority != high: schedule for later
  if userLimiter.exceeded(user, "push", 10/hour): digest instead

after sending:
  provider webhook → { providerMsgId, status: delivered | bounced | opened }
  → update notification_log; hard bounce → mark email invalid

Preferences are read on every notification, so I cache them in Redis and invalidate on change. They're per channel and per type, so a user can keep order updates but turn off marketing. Unsubscribing from marketing email isn't optional; it's a legal requirement in most places, so that check is never skipped.

Per-user rate limiting protects the user from us. Nobody wants 40 pushes an hour because they got 40 likes. Past a threshold, I'd collapse low-priority events into a digest like "12 people liked your post."

For tracking, the log row goes queued, sent, then delivered or failed from provider webhooks. That log is what support uses when a customer says "I never got the code."

Interviewers look for: digesting and per-user limits as a product decision, plus the compliance point on unsubscribes.

So first I'd pin down scope, because "Dropbox" can mean a lot of things. I'd ask the interviewer a handful of questions and write the answers on the board.

Functional
  - upload / download files, any type?        yes
  - sync across a user's devices?              yes, within seconds
  - version history? how far back?             yes, 30 days
  - sharing with other users?                  yes, read/write links
  - offline edits that sync later?             yes
  - real-time co-editing like Google Docs?     NO (different problem: OT/CRDT)

Non-functional
  - max file size                              50 GB
  - durability                                 never lose a byte
  - scale                                      50M users, ~10 GB each

The two answers that shape the whole design are the co-editing one and the file size one. If they want live collaborative editing, that's a document-sync problem with operational transforms or CRDTs, and I'd say that out loud and scope it out. If files can be 50 GB, I know straight away I need chunking and resumable uploads, and that bytes can't flow through my app servers.

Durability also changes my priorities. For this system, losing a file is far worse than a sync that's a few seconds slow, so I'll lean on object storage and transactional metadata rather than chasing low latency.

Interviewers look for: explicitly scoping out real-time co-editing, and naming durability as the top non-functional requirement.

So chunking buys me three things at once: I only re-upload what changed, a dropped connection only loses one chunk, and identical data is stored once.

Let me give you an example. I split the file into 4 MB pieces and name each piece by its SHA-256 hash:

const crypto = require("crypto");
const CHUNK = 4 * 1024 * 1024;

function chunkHashes(buf) {
  const hashes = [];
  for (let off = 0; off < buf.length; off += CHUNK) {
    const piece = buf.subarray(off, off + CHUNK);
    hashes.push(crypto.createHash("sha256").update(piece).digest("hex"));
  }
  return hashes;
}
// A 10 MB file -> 3 hashes. If the first two 4 MB pieces are byte-identical,
// the first two hashes are identical too, so that chunk is stored once.

What happens on an edit is the client re-hashes, sends the list of hashes to the server, and the server replies "I'm missing only chunk 7". So editing one paragraph in a 1 GB file uploads 4 MB instead of 1 GB. And because the name is the content hash, dedup across versions and even across users falls out for free.

The chunk size is a trade-off. Smaller chunks mean finer-grained diffs but more metadata rows and more requests. Around 4 MB is the sweet spot most systems land on.

So the key idea is that the heavy bytes go straight to object storage, and my servers only ever handle small metadata requests.

Client                 Metadata svc             S3              Metadata DB
  | chunk + hash file      |                     |                   |
  |-- upload-session ----> |  which hashes exist?|                   |
  |   {path, hashes[]}     |------------------------------------->   |
  |<-- missing[] + presigned URLs                |                   |
  |-- PUT chunk (direct) -------------------->   |                   |
  |-- commit {hashes[], baseVersion} ->|  one SQL transaction       |
  |                        |  new version row + change journal ---> |
  |<-- 200 version 8 ------|  publish "user 42 changed"             |

Walking through it: the client chunks and hashes locally. It opens an upload session with the hash list, and the server answers with only the missing hashes plus a presigned URL for each. The client PUTs those chunks directly to S3, retrying any single chunk that fails. Then it calls commit.

Commit is the important step. In one transaction I insert the new file_versions row, bump latest_version, increment chunk ref counts, and append to the changes journal. Only after that does the file "exist" in its new version. If the client dies halfway through uploading, nothing visible changes; orphaned chunks get garbage-collected later.

One gotcha: the server must verify on commit that every hash actually landed in S3, otherwise a buggy client could commit a version pointing at chunks that don't exist.

Interviewers look for: presigned URLs, and the commit as the single atomic point where a version becomes real.

So I split it into two parts: a cheap "something changed" signal, and a reliable "what changed" pull. The push is just a doorbell; the journal is the source of truth.

changes journal (per user)
cursor | file_id | version
  1001 | f_17    | 7
  1002 | f_90    | 3
  1003 | f_17    | 8     <- Device A just committed

Device B: last cursor = 1001
  WebSocket: "you have changes"
  GET /files/changes?cursor=1001
  -> [{f_90, v3}, {f_17, v8}], nextCursor = 1003
  download missing chunks, apply, save cursor 1003

The reason I don't put the actual change in the push message is that pushes get lost: the laptop was asleep, the socket dropped, the phone OS killed the app. If the push is only a hint, losing it costs nothing. When the device reconnects, it just asks "what changed since my cursor?" and catches up exactly, however long it was offline.

For the signal itself, a WebSocket or long-poll works; on mobile I'd use APNs/FCM push. And if even that fails, the client polls the changes endpoint every minute or so as a safety net.

So in practice: push to wake up, cursor-based pull to sync. That's the same pattern I'd use for any "keep devices in sync" problem.

So I detect it with optimistic locking, and I never silently overwrite. The loser gets saved as a conflicted copy.

Let me give you an example. Both laptop and desktop last synced version 5 of budget.xlsx.

Laptop  commit {hashes, baseVersion: 5}  -> latest is 5, accept -> version 6
Desktop commit {hashes, baseVersion: 5}  -> latest is 6, REJECT (409 Conflict)

-- server side, the check is one conditional update:
UPDATE files SET latest_version = 6
WHERE file_id = 'f_17' AND latest_version = 5;
-- 0 rows updated => someone got there first

What happens next is the desktop client pulls version 6, keeps its own edit as budget (Desktop's conflicted copy).xlsx, and uploads that as a new file. The user decides which to keep.

The reason I don't try to merge is that I don't understand the file format. Merging two spreadsheets byte-wise would corrupt them. Last-write-wins is also wrong here, because it silently throws away someone's work, which breaks my "never lose data" requirement. For text files you could attempt a three-way merge, but for a generic file store, conflicted copy is the honest answer.

Interviewers look for: baseVersion plus a conditional write, and a clear reason why last-write-wins is unacceptable here.

So content goes in object storage and metadata goes in a relational database, because they have completely opposite access patterns.

The chunks are immutable blobs named by hash, which is exactly what S3 is great at, and it's orders of magnitude cheaper per GB than any database. The metadata needs transactions: committing a version touches the version row, the file row, ref counts and the journal, and all of that has to be all-or-nothing.

For sharding the metadata I'd shard by owner_id, or really by namespace, so all of one user's files and their change journal live on the same shard and a commit stays a single-shard transaction. The awkward case is shared folders, since they belong to several people. Dropbox's answer was to make the shared folder its own namespace with its own journal, and each user's view mounts those namespaces.

So the weakness is insertion. If I add one byte at the start of a file, every fixed 4 MB boundary shifts by one byte, every chunk hash changes, and I re-upload the whole file even though almost nothing changed.

Fixed-size:     |AAAA|BBBB|CCCC|      insert "x" at start
                |xAAA|ABBB|BCCC|C|    every hash changes

Content-defined:
  boundary wherever rolling_hash(last 48 bytes) % 2^22 == 0
                |AAAA|BBBB|CCCC|
                |xAAAA|BBBB|CCCC|     only the first chunk changes

The fix is content-defined chunking. I slide a rolling hash, like Rabin fingerprinting or the faster FastCDC / Gear hash, over the bytes and cut a boundary whenever the hash hits a pattern. Boundaries now depend on the content, not the offset, so after an insertion they re-synchronise right after the edit. I'd cap chunk sizes between, say, 1 MB and 8 MB so I don't get pathological tiny or huge chunks.

The trade-off is CPU on the client for hashing and variable-size chunks that complicate metadata a bit. For documents people edit in the middle, it's clearly worth it. For media files that are rewritten entirely, it makes no difference, so some systems only use it for certain file types.

Interviewers look for: naming the boundary-shift problem unprompted; mentioning rsync or Rabin fingerprinting is a strong senior signal.

So a chunk can only be deleted when nothing references it, and I have to be careful about the race between "ref count hit zero" and "someone is uploading that same hash right now".

Let me walk through the race. User A deletes the last version that uses chunk h1, so its ref count goes to 0. At the same moment User B starts an upload session, the server says "I already have h1, skip it", and then a GC job deletes h1. B commits a version pointing at a chunk that's gone. That's data loss.

Safe approach
  1. ref_count -> 0: don't delete, mark "pending_gc" with timestamp
  2. GC deletes only chunks pending for > 7 days
  3. commit re-checks: any chunk in pending_gc gets revived (ref_count++)
  4. S3 lifecycle / versioning as a last safety net

So the grace period gives in-flight uploads time to commit, and the commit transaction revives any chunk it references. I'd also keep deletes lazy and batched rather than inline, because a user deleting a 10,000-file folder shouldn't trigger 10,000 synchronous S3 calls.

Honestly, at this scale storage is cheap compared to the cost of losing data, so I'd err heavily toward keeping things too long. Some teams even skip ref counting and run a periodic mark-and-sweep over all versions instead, which is slower but harder to get wrong.

So first I'd clarify whether we're YouTube, with user uploads at huge volume, or Netflix, with a fixed catalogue. I'll assume YouTube: upload, watch, search, and view counts, with comments and recommendations out of scope.

Uploads     500 hours of video / minute
Raw size    ~1 GB / hour; all renditions together ~3 GB / hour
Storage     500 x 60 x 24 x 3 GB  = ~2.2 PB / day of new video
Viewing     1B hours watched / day
            avg ~5 Mbps -> 2.25 GB per hour watched
            1B x 2.25 GB = ~2,250 PB (2.25 EB) egress / day

The thing I want to call out from those numbers is that egress is roughly a thousand times bigger than ingest. So this is overwhelmingly a read-heavy, bandwidth-bound system. The design has to put a CDN front and centre, and the cost conversation will be about bandwidth and storage, not compute for the API.

Non-functionally, I'd want playback to start in under two seconds anywhere in the world, no buffering mid-video, and I'm fine with an upload taking several minutes to become watchable. That last one is important: it means transcoding can be asynchronous.

Interviewers look for: spotting that read bandwidth dwarfs everything else and letting that drive the design.

So adaptive bitrate means the video exists in several qualities, each cut into short segments, and the player picks the quality segment by segment based on how fast the network is right now.

master.m3u8
  #EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080
  1080p/index.m3u8
  #EXT-X-STREAM-INF:BANDWIDTH=2800000,RESOLUTION=1280x720
  720p/index.m3u8
  #EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360
  360p/index.m3u8

720p/index.m3u8
  #EXTINF:4.0,
  seg_0000.ts
  #EXTINF:4.0,
  seg_0001.ts ...

What happens is the player fetches the master manifest, starts at a lowish quality so playback begins fast, measures how quickly each segment downloads, and steps up or down. If I walk into a lift and my bandwidth drops, the next segment comes from 360p instead of 1080p and the video keeps playing instead of freezing.

A single MP4 can't do that: it's one bitrate for the whole file, so either it's too heavy for a bad connection or too ugly for a good one. Segments also cache beautifully on a CDN because they're small, immutable, plain HTTP files.

In practice today I'd use HLS or DASH with CMAF segments, so the same fMP4 segments serve both Apple and non-Apple players.

So upload is a resumable direct-to-storage upload, and everything after that is an asynchronous, event-driven pipeline.

Creator -- resumable multipart PUT --> S3 raw/
   S3 "object created" event --> queue
      split worker: cut into ~10 s GOP-aligned pieces
         -> N jobs (piece x rendition) --> transcode workers (parallel)
            -> 1080p / 720p / 480p / 360p segments --> S3 processed/
      stitch + write manifests
      update videos.status = 'ready'   --> notify creator

Walking through it: the client asks the API for an upload URL, gets a presigned multipart URL, and uploads parts directly to object storage. A 4 GB upload that drops at 90 percent only retries the failed part.

When the raw file lands, an event kicks off processing. The reason I split the video into pieces first is latency: transcoding a 2-hour video serially on one machine takes ages, but 700 ten-second pieces across hundreds of workers finish in minutes. Each job is idempotent, keyed by (videoId, piece, rendition), so a crashed worker just gets its job retried.

The video row goes uploading then processing then ready. I'd also do the cheap renditions first, so a video can go live at 360p while 4K is still processing. Content checks like copyright matching and moderation hang off the same pipeline.

Interviewers look for: parallel transcoding of pieces, idempotent jobs, and a status state machine.

So video popularity is extremely skewed, and I design the caching tiers around that curve.

Popularity     Where it's served from
top ~10%       edge caches everywhere (and ISP-embedded boxes)
middle         regional mid-tier cache
long tail      origin (S3), cached on demand, evicted quickly

Player -> edge -miss-> regional -miss-> origin shield -> S3

What happens for a hit video is it's already at an edge a few milliseconds from the viewer. Netflix goes further and pre-positions its catalogue overnight on Open Connect appliances sitting inside ISPs, because they know tomorrow's demand in advance.

For the long tail, the first viewer pays a slower start as it's pulled through the tiers. The origin shield is important there: without one, a moderately popular video missing at 200 edges would hit S3 200 times. With a shield, all those misses collapse into one fetch.

And for cost, I'd keep fewer renditions for videos with no views, maybe just 360p and 720p, generate higher ones on demand if they take off, and move old originals to cold storage tiers. That can cut storage a lot, because most uploads are barely watched.

So I never write one row per view. I stream view events and aggregate them, and I accept that the displayed count lags by a few seconds or minutes.

Player --"view"--> ingest API --> Kafka (topic: views, key = videoId)
   stream job (Flink): dedupe + validate, tumbling 30 s windows
      -> UPDATE video_stats SET views = views + :delta WHERE video_id = :id
   hot videos: counters in Redis, flushed periodically

Let me give you the reasoning. A viral video might get 100,000 views a second. A single database row can't take 100k increments a second; it becomes a hot-row lock contention nightmare. But one update every 30 seconds with +3,000,000 is trivial.

Keying Kafka by videoId means all events for one video land in the same partition, so one worker owns that counter and there's no contention at all. For a truly viral one, I'd split the key into sub-keys like videoId#0..15 and sum them.

The "accurate" part is really about what counts as a view. YouTube counts after some seconds of watch time and filters bots, so the stream job dedupes by session and applies those rules before counting. That's why counts sometimes freeze and then jump: validation is happening behind the scenes.

Interviewers look for: batching increments and the hot-key split for viral videos.

So time-to-first-frame is a chain of round trips, and I attack each link.

Tap play
  1 API: metadata + manifest URL       ~100 ms  (prefetch on hover / in feed)
  2 DNS + TLS to CDN                   ~100 ms  (connection reuse, HTTP/3)
  3 master manifest                    ~50 ms   (edge cached)
  4 media manifest                     ~50 ms   (or inline in master)
  5 first segment at low bitrate       ~300 ms  (short first segment)
  6 decode + render                    ~100 ms

A few concrete things I'd do. Start at a conservative bitrate so the first segment is small, then ramp up; the player can use the previous session's measured bandwidth as a starting guess. Keep segments short, around 2 to 4 seconds, because the player needs at least one full segment before it can play. Prefetch the manifest, and even the first segment, for the video the user is likely to tap, which is what feeds and autoplay previews do.

On the network side, HTTP/3 over QUIC saves handshake round trips and handles packet loss on mobile better. And every one of those requests should be an edge hit, so I'd make sure manifests are cached at the edge too, not just segments.

The trade-off with very short segments is more requests and slightly worse compression, so I'd tune per platform rather than pick one number.

So the big difference is that there's no time to transcode ahead. Everything happens within seconds, and latency becomes a first-class requirement.

Streamer --RTMP/SRT/WHIP--> ingest server (nearest region)
   real-time transcoder: 1080p / 720p / 480p, 2 s segments (or partial segments)
   packager -> rolling manifest (only last N segments)
   push to CDN origin  --> edges --> viewers (LL-HLS / LL-DASH)

What changes: ingest is a persistent stream, not a file upload, so I need ingest servers near the streamer and failover if one dies mid-stream. Transcoding is a dedicated real-time job per stream, sized in advance, since it can't fall behind. The manifest is a sliding window that updates every couple of seconds, so it gets a very short CDN TTL while segments stay immutable and long-cached.

For latency, standard HLS runs 10 to 30 seconds behind. Low-latency HLS with partial segments gets to around 2 to 5 seconds. If they want true sub-second interaction, like an auction, I'd switch to WebRTC, which doesn't scale through CDNs as cheaply.

So the trade-off is latency versus cost and scale. For a million-viewer sports stream, a few seconds of delay over a CDN is the right call.

Interviewers look for: knowing the latency tiers (HLS, LL-HLS, WebRTC) and that the manifest becomes a rolling window.

So first I'd clarify the core loop: rider requests, we find a nearby driver, both see live location, the trip ends, we charge. I'd put pooling, scheduled rides and food delivery out of scope.

Active drivers     5M, each sends location every 4 s
                   5,000,000 / 4 = 1.25M location writes / sec
Ride requests      20M / day = ~230 / sec avg, peak ~2,000 / sec
Location payload   ~100 bytes -> ~125 MB / sec inbound

The number that worries me is 1.25 million location writes a second. Ride requests are tiny by comparison. So the whole design splits into two very different paths: a firehose of ephemeral location updates where only the latest point per driver matters, and a low-volume but strongly consistent trip path where money and double-booking are involved.

For non-functional requirements, I'd want matching within a few seconds, live positions within a couple of seconds of reality, and one hard correctness rule: a driver can never be assigned to two riders. Availability matters a lot too, because if matching is down, the business is down.

Interviewers look for: computing the location write rate early and separating the hot ephemeral path from the durable trip path.

So I'd use a geospatial index, and the simplest one to explain and operate is geohash. Nearby points share a string prefix, so "nearby" becomes "same cell or a neighbouring cell".

geohash length   cell size (approx)
5                4.9 km x 4.9 km
6                1.2 km x 0.6 km
7                153 m x 153 m

GEOADD drivers:sf -122.4194 37.7749 driver:123
GEOSEARCH drivers:sf FROMLONLAT -122.4183 37.7760
          BYRADIUS 2 km ASC COUNT 20 WITHDIST

What happens under the hood with the manual approach: I compute the rider's geohash at length 6, take that cell plus its 8 neighbours, because the rider might be sitting right on a cell edge, and collect the drivers in those 9 cells. Then I filter to available drivers and sort by actual distance.

In practice Redis does this for me. GEOADD stores members in a sorted set scored by a geohash-derived integer, and GEOSEARCH does the neighbour-cell lookup and distance filter in one command. I'd keep one key per city so each key stays manageable and shards naturally.

One gotcha: straight-line distance isn't what riders care about. A driver 500 m away across a river might be 10 minutes out. So I use the geo index to get candidates, then rank the top 10 or so by ETA from a routing service.

So geohash is a fixed grid and quadtree is an adaptive one. Geohash is simpler to shard and update; quadtree handles uneven density better.

Let me give you the scenario that matters. Midtown Manhattan at rush hour might have 3,000 drivers in one geohash-6 cell, while a rural cell has two. With a fixed grid, a dense-cell query returns too many candidates, and a sparse-cell query has to widen the radius several times. A quadtree adapts: busy areas split until each leaf holds, say, 100 drivers.

The catch is that drivers move constantly. With 1.25 million updates a second, a quadtree is doing a lot of node moves and rebalancing. So in an interview I'd start with geohash in Redis because it's operationally simple, and mention quadtrees, or Uber's H3 hexagonal grid, as the refinement. H3 is nice because every hexagon has six equidistant neighbours, which makes things like surge-pricing zones cleaner.

Interviewers look for: the density trade-off and the update cost; naming H3 shows you know what Uber actually uses.

So the trick is realising that for matching I only need each driver's latest position, which is a small in-memory overwrite, not an ever-growing log in a database.

Driver app --WebSocket--> gateway (sticky per driver)
   -> location service (sharded by city)
        Redis GEO per city: overwrite driver's point       (hot path)
        Kafka topic "locations" (key = driverId)            (async)
             -> batch writer: trip paths to Cassandra/S3   (receipts, disputes)
             -> ETA / traffic / surge consumers

Walking through it: 5 million drivers is about 5 million entries per snapshot, which is well under a gigabyte, so even the whole world fits in memory across a few Redis shards. Sharding by city also matches the query pattern, because a rider in São Paulo never needs drivers in Tokyo.

The full history still matters, for fare calculation on a trip, disputes and ML, so I push every update into Kafka and a separate consumer writes it in batches. That keeps the durable write path off the latency-critical path entirely.

Two practical optimisations: the driver app can skip sending when it hasn't moved, and I'd use WebSockets instead of HTTP so I'm not paying connection setup 1.25 million times a second. And if a driver's location is older than, say, 30 seconds, I treat them as offline for matching.

So I make the "offer this driver" step an atomic claim. Only one ride can win it, and the claim expires if the driver doesn't respond.

// Matching for ride r_789 picks driver d_123
SET lock:driver:d_123 r_789 NX EX 15
  -> OK    : we own the driver, send the offer
  -> (nil) : someone else has them, try the next candidate

// Driver accepts: durable transition, guarded
UPDATE drivers SET status = 'on_trip', ride_id = 'r_789'
WHERE driver_id = 'd_123' AND status = 'available';
-- 1 row = success, 0 rows = lost the race

What happens here is two matching workers might both see driver 123 as the closest for two different riders at the same instant. The SET NX means exactly one of them gets the lock; the other moves on to its second-best driver. The 15-second expiry covers the driver ignoring the offer or the app crashing, so the driver isn't stuck forever.

The Redis lock is the fast guard, but I don't rely on it alone, since Redis can fail over and lose a lock. The conditional update in the database is the real source of truth. Belt and braces.

One gotcha I'd mention: offering one driver at a time with a 15-second timeout can make matching slow. Some systems offer to a small batch and take the first acceptance, releasing the rest, which trades a bit of driver annoyance for speed.

Interviewers look for: an atomic claim with a TTL, plus a conditional durable write as the final authority.

So I model a trip as an explicit state machine stored in SQL, and every transition is validated and recorded, because trips involve money and disputes.

REQUESTED -> MATCHING -> DRIVER_ASSIGNED -> DRIVER_ARRIVED -> IN_PROGRESS -> COMPLETED -> PAID
     \           \              \                   \
      ---------------------- CANCELLED (by rider / driver / timeout) ----

UPDATE rides SET status = 'IN_PROGRESS', started_at = now()
WHERE ride_id = :id AND status = 'DRIVER_ARRIVED';

The reason I use conditional updates like that is duplicate and out-of-order events. The driver's app might retry "start trip" on a flaky connection, or a cancel might race with an accept. With the WHERE status = ... guard, only legal transitions apply, and a retried one is a harmless no-op.

Every transition also goes into a ride_events table and out on Kafka, via an outbox, so billing, notifications and analytics react to it without the ride service calling each of them directly.

Payment is its own step after COMPLETED, with an idempotency key of the ride ID, so if the charge call times out and we retry, the rider is never billed twice.

So surge is a supply-and-demand ratio computed per small area every minute or so, turned into a multiplier that's shown and locked in at request time.

every 60 s, per H3/geohash cell:
  demand = ride requests + app opens in last 5 min
  supply = available drivers in cell (+ neighbours, weighted)
  ratio  = demand / max(supply, 1)
  multiplier = clamp(f(ratio), 1.0, 3.0), smoothed with previous value

stream: Kafka(requests, locations) -> Flink windowed counts -> Redis surge:{cell}

What happens in practice: the streaming job keeps sliding-window counts per cell from the request and location topics, computes the multiplier, and writes it to a fast key-value store. When a rider asks for a quote, I read the multiplier for their pickup cell and issue a signed quote with an expiry, so the price they agreed to is the price they pay, even if surge changes a second later.

Two details matter. Smoothing, because a multiplier that flickers between 1.0 and 2.4 every minute is a terrible experience. And using neighbouring cells, because a driver 300 m across a boundary is real supply.

I'd also be honest that the actual pricing function is a business and ML problem, often regulated, with caps during emergencies. My job in the design is the pipeline that keeps it fresh and consistent.

So the key is that location data is ephemeral and self-healing. Within four seconds every driver resends it, so I can rebuild the index rather than trying to replicate it perfectly.

t=0     Redis shard for city X dies
t=0-5s  new matching in city X fails / degrades
        existing trips keep going (state is in SQL, not Redis)
t=5s    failover replica promoted, or new empty shard
t=5-9s  drivers' next updates repopulate it
t=10s   matching back to normal

What happens to live trips: the trip itself lives in the rides database, so nothing about who is driving whom is lost. Live tracking on the rider's map might freeze for a few seconds, and then resumes when updates flow again. I'd have the rider app show the last known position with a "reconnecting" state rather than an error.

For new requests, I'd fail fast with a friendly retry rather than queueing blindly. And the driver app should buffer its trip path locally, so when the backend recovers it uploads the missing points and the fare is still accurate.

At a bigger scale, a whole region failing, I'd run each city active in one region with a warm standby in another. Because the hot state rebuilds itself in seconds, failover is really just DNS and routing, plus making sure the rides database has a replica in the standby region.

Interviewers look for: recognising that the location index is soft state that can be rebuilt, while trip state must be durable.

So first I'd ask what the crawl is for, because that changes priorities a lot. A search engine crawler wants breadth and freshness; a price-monitoring crawler wants a few sites crawled very often. I'll assume a general search crawler over HTML only.

Target        1B pages / month
              1,000,000,000 / (30 x 86,400) = ~386 pages / sec, plan for ~1,000 peak
Page size     ~500 KB raw HTML -> ~100 KB compressed
Storage       1B x 100 KB = 100 TB / month compressed
Bandwidth     386 x 500 KB = ~190 MB/s = ~1.5 Gbps inbound
URLs seen     ~10B unique over time

From those numbers, fetching isn't the hard part; a few dozen machines can do 400 pages a second. The hard parts are the ones the numbers hide: politeness, so I don't hammer any one site; dedup across billions of URLs; and robustness against slow servers, malformed HTML and crawler traps.

So I'd confirm: obey robots.txt, recrawl pages periodically, store content for an indexer that's a separate system, and skip JavaScript rendering for now, with a note that some pages need a headless browser pool later.

So it's a loop: take a URL from the frontier, fetch it, parse out links, filter and dedupe them, and put the new ones back on the frontier.

seeds -> [URL frontier] -> fetcher -> parser -> link extractor
              ^              |  robots.txt cache      |
              |              |  DNS cache             v
              |              v                  normalise URL
              |        content hash seen?        URL filter (depth, length, domain)
              |              |                   Bloom filter: seen?
              |              v                        |
              |        object storage -> indexer      |
              +-------------- new URLs only ----------+

A couple of things worth pointing out. Before fetching, the fetcher checks a cached robots.txt for that host; I cache it for about a day so I'm not fetching it every time. DNS resolution is a surprisingly big bottleneck at this rate, so I run a local caching resolver.

After parsing, I normalise every link before checking if I've seen it: lowercase the host, drop the #fragment, strip tracking params like utm_source, resolve relative paths. Otherwise Example.com/a and example.com/a#top look like two different pages.

And I check content too, not just URLs, because many different URLs serve the same page. If the content hash was seen, I store a pointer instead of the page.

So I use a two-layer frontier. Front queues decide what's important; back queues decide when a host can be hit again.

Front (priority)          Back (one queue per host)
  P1 high  ---+            wikipedia.org  [u1, u5, u9]   next_ok: 12:00:01.0
  P2 med   ---+-- router -> nytimes.com   [u2, u7]       next_ok: 12:00:00.4
  P3 low   ---+            blog.xyz       [u3]           next_ok: 12:00:05.0

heap of hosts ordered by next_ok
worker: pop host whose next_ok <= now, fetch one URL,
        push host back with next_ok = now + crawl_delay

What happens is new URLs get a priority score, based on things like PageRank-ish importance and how often the page changes, and land in a front queue. A router pulls from front queues, biased toward high priority, and drops each URL into its host's back queue.

Workers then only pull from hosts whose "next allowed time" has passed. After fetching, the host goes back into the heap with a delay, either from Crawl-delay in robots.txt or something adaptive like ten times the last response time. So a slow, small site automatically gets crawled more gently.

The nice property is that politeness is structural: a single host can never be fetched in parallel, because its queue is only ever held by one worker at a time.

Interviewers look for: the front/back queue split (the Mercator design) and per-host rate limiting built into the data structure.

So an exact hash set of 10 billion URLs is hundreds of gigabytes. I'd use a Bloom filter in front: a bit array that answers "definitely not seen" or "probably seen" in constant time and a fraction of the memory.

n = 10B URLs, target false-positive rate p = 1%
bits per item = -ln(p) / (ln 2)^2  = 9.59
total         = 10B x 9.59 bits    = ~12 GB
hash functions k = 0.693 x 9.59   = ~7

vs. exact set of 8-byte URL fingerprints: 10B x 8 B = 80 GB (+ overhead)

I ran those numbers, and 12 GB fits comfortably, especially split across crawler nodes by host hash. The trade-off is the 1 percent false positives: 1 in 100 genuinely new URLs gets wrongly skipped. For a web crawler that's usually acceptable, since important pages are linked from many places and we'll find them via another link later.

If I can't accept any misses, I use the Bloom filter as a fast pre-check and confirm "probably seen" answers against an exact store, like a RocksDB of URL fingerprints on disk. Most checks are "definitely new" or genuinely seen, so the disk lookup is rare.

One gotcha: a standard Bloom filter can't delete, so if I want URLs to become eligible again for recrawl, I track recrawl timing separately rather than removing them from the filter.

So I partition by hostname. Each host is owned by exactly one worker node, which makes politeness, robots.txt caching and DNS caching all local.

owner(url) = consistent_hash(hostname(url)) -> node

node 7 extracts a link to nytimes.com/x
  owner("nytimes.com") = node 3
  -> send to node 3's inbox (batched, via Kafka topic partitioned by host hash)

What happens is every node runs the full loop for its hosts: frontier, fetch, parse. When it finds links to hosts it doesn't own, it batches them and forwards them to the owner. I'd use consistent hashing so adding a node only moves a small slice of hosts.

The reason I don't just have a central queue that all workers pull from is politeness. With a shared queue, five workers could grab five URLs from the same small site at once, and I'd need a distributed lock per host. With host ownership, one node naturally serialises requests to that host.

The trade-off is skew: one giant host like wikipedia.org sits on one node. In practice that's fine because politeness limits the rate for a single host anyway. I'd also place nodes geographically close to the hosts they crawl, which cuts latency a lot.

So a crawler trap is any site structure that generates infinite unique URLs, so a naive crawler spends forever on one site. Sometimes it's malicious, but usually it's accidental.

Examples
  /calendar?month=2026-10 -> "next" -> ?month=2026-11 -> ... forever
  /a/b/a/b/a/b/a/b/...        (relative link bug)
  /product?sort=price&color=red&size=m ...  (faceted combinations)
  ?sessionid=8f3a...           (new ID on every page)

Defences
  - max URL length (~2,000 chars) and max path depth
  - per-host page budget per crawl cycle (e.g. 100k)
  - strip session IDs / known tracking params when normalising
  - detect repeating path segments
  - content-hash dedupe: 1,000 URLs, same content -> stop

The one I rely on most is the per-host budget. Whatever the trap looks like, a site can only consume its allotted pages this cycle, and the budget scales with how important the site is. That bounds the damage even for trap patterns I've never seen.

Content dedupe catches a lot too: calendar pages for empty months are near-identical, so SimHash, which gives similar pages similar fingerprints, flags them as near-duplicates, and I lower the priority of links from that URL pattern.

Interviewers look for: at least two concrete trap types and a general bound like a per-host budget, not just a list of special cases.

So I estimate each page's change rate from history and schedule recrawls proportional to that, weighted by importance. A news homepage changes every few minutes; an old blog post might never change.

per URL: last_crawled, last_changed, content_hash, change_count, crawls

on recrawl:
  changed = new_hash != content_hash
  est_change_rate = change_count / time_observed
  next_crawl = now + clamp(k / (est_change_rate x importance), 1 h, 90 days)

cheap check first:
  GET with If-None-Match: <etag> / If-Modified-Since
  304 Not Modified -> no body transferred

What happens in practice: if I recrawl a page three times and it changed every time, I halve the interval; if it didn't change, I back off. Conditional GETs save a lot of bandwidth, because a 304 response has no body. Sitemaps help too, since many sites publish a lastmod date I can trust as a hint.

The interesting trade-off is the crawl budget. I have a fixed number of fetches per day, split between discovering new pages and refreshing old ones. For search freshness I'd bias toward important pages that change often, because a stale news homepage is far more visible than a stale forum post from 2014.

So first I'd clarify: top 5 suggestions per prefix, ranked by popularity, English only to start, and is it personalised? I'll say global popularity first, personalisation as an extension.

Searches         5B / day
Keystrokes       ~6 per search that trigger a lookup
Lookups          30B / day = ~350,000 / sec avg, ~1M / sec peak
Latency budget   < 100 ms end to end, so server side ~10-20 ms
Freshness        minutes to hours is fine (trending: minutes)

The shape of this problem is extreme read volume, tiny responses, strict latency, and relaxed freshness. That combination screams "precompute everything offline and serve from memory". There's no way I'm scanning and sorting query logs on each keystroke.

I'd also note that the client can cut the load a lot. Debouncing to about 150 ms between keystrokes and reusing results for longer prefixes easily halves the real request rate.

Interviewers look for: noticing that relaxed freshness plus strict latency means precompute, and separating the build path from the serve path.

So I'd use a trie, a prefix tree, but with one crucial optimisation: each node stores its top 5 completions precomputed, so a lookup is just walking down the prefix and returning a list.

(root)
 └ n
   └ e
     └ w        top5: [news, new york, new year, new balance, newegg]
       └ " "
         └ y    top5: [new york, new year, new york times, new york weather, ...]

lookup("new y"): walk n -> e -> w -> " " -> y, return node.top5
cost: O(length of prefix), no sorting at request time

Without the cached top 5, I'd have to walk down to the prefix node and then traverse its entire subtree to find the most popular completions, which for a short prefix like "n" is millions of nodes. With the cache, "n" and "new york ti" cost the same: a handful of pointer hops.

The trade-off is memory and build time, since every popular query appears in the top-5 list of every one of its prefixes. That's fine because it's built offline.

In practice, I often skip the literal trie on the serving side and flatten it into a key-value map of prefix -> top5 in Redis or an in-memory hash. Same idea, easier to shard and replicate.

So there's an offline build path completely separate from serving. Logs go in one end, a new immutable snapshot comes out the other, and servers swap to it atomically.

search logs -> Kafka -> S3 (hourly partitions)
   batch job (Spark), daily or hourly:
     count queries over last N days, time-decayed weights
     drop rare (< threshold), filter blocked terms
     for each query, for each prefix: keep top 5
   -> snapshot v42 (prefix -> top5) in S3
serving nodes: load v42 in background, atomically swap pointer, drop v41

Walking through it: I aggregate query counts with a time decay, so a query popular last week counts more than one popular last year. I drop anything below a frequency threshold, which removes typos and, importantly, private stuff people accidentally type into search boxes. Then I compute top 5 per prefix and write a versioned snapshot.

Serving nodes load the new version into memory alongside the old one and swap a pointer, so there's no moment where queries hit a half-built structure. If v42 looks broken, rollback is swapping back to v41.

The reason I don't update the trie in place on every search is that it'd mean millions of writes a second contending with reads, for suggestions that don't need to be second-fresh anyway.

Interviewers look for: immutable versioned snapshots with an atomic swap, and a frequency threshold for privacy and noise.

So the naive approach is sharding by first letter, a to z across 26 machines, and it breaks because letters are wildly uneven. The shard for "s" gets far more traffic than "x", "q" and "z" together.

Naive:    shard = first letter        -> "s" shard melts, "x" shard idle
Better:   ranges balanced by traffic, from last week's logs
            shard 1: a - ca
            shard 2: cb - di
            ...
            shard 9: sa - sh
            shard 10: si - sz
Routing:  small range map in every API node; look up prefix -> shard

So I split by prefix range, but choose the boundaries using actual query traffic, and recompute them with each build. Each shard is also replicated several times, since it's read-only between builds and replicas are trivial.

Honestly, the whole prefix-to-top-5 map for a popular language is often only tens of gigabytes, so another valid answer is no sharding at all: every serving node holds the full thing, and I scale by adding replicas. That's simpler and avoids routing entirely. I'd shard only when the data outgrows one machine's memory, for example when adding many languages or personalisation.

The very short prefixes, one or two characters, are the hottest keys of all, so those get cached at the CDN and in the browser, and barely reach the servers.

So a lot of the latency and load wins are actually in the client. I'd do debouncing, cancel stale requests, and reuse results I already have.

let controller;
const cache = new Map();

const suggest = debounce(async (q) => {
  if (cache.has(q)) return render(cache.get(q));
  controller?.abort();                       // cancel the older in-flight request
  controller = new AbortController();
  try {
    const res = await fetch(`/suggest?q=${encodeURIComponent(q)}`,
                            { signal: controller.signal });
    const list = await res.json();
    cache.set(q, list);
    render(list);
  } catch (e) {
    if (e.name !== "AbortError") throw e;
  }
}, 150);

What happens here: debounce waits until the user pauses for 150 ms, so fast typists don't fire a request per letter. AbortController cancels the previous request, which prevents the classic race where the response for "ne" arrives after the one for "new y" and overwrites the better suggestions. And the map caches results, so backspacing is instant.

I'd also prefetch on focus: as soon as the search box is focused, load the top suggestions for the empty prefix. And responses for short prefixes get a Cache-Control max-age of a few minutes so the browser and CDN serve them.

Interviewers look for: the out-of-order response race and how aborting or sequence numbers fixes it.

So the daily batch can't react to breaking news, so I add a streaming layer that detects spikes and merges them into the batch results at serve time.

search events -> Kafka -> Flink
   sliding window: count(query) over last 10 min
   trend score = recent_rate / baseline_rate (from batch)
   top trending per prefix -> Redis "trending:{prefix}", TTL 30 min

serve("elec"):
   batch   = [electric cars, electrician near me, election, ...]
   trending= [election results]
   merge   -> [election results, electric cars, electrician near me, ...]

What happens when something breaks, say an election night: searches for "election results" go from a baseline of 1,000 an hour to 500,000 in ten minutes. The ratio is huge, so the streaming job pushes it into the trending set for its prefixes, and within a couple of minutes it shows up at the top of the list.

The reason I use a ratio rather than raw count is that "facebook" is always popular, and I don't want it flagged as trending every day. The TTL makes trends fade naturally if the spike ends.

The risk is manipulation and bad content: trending is exactly how a coordinated group could push an offensive suggestion. So trending candidates go through the same blocklist and abuse filters, and possibly a higher threshold of distinct users, before they're served.

So I keep the global lookup as the fast baseline and blend in a small per-user candidate set, instead of building a trie per user.

request "ja" from user 42
  1 global top 20 for "ja"         (in-memory, ~1 ms)
  2 user 42 recent queries         (Redis list, last 100, ~1 ms)
       filter those starting with "ja": [javascript closures, jaipur flights]
  3 re-rank the ~25 candidates with a light model
       features: global popularity, user history, locale, recency
  4 return top 5                   total server time ~10 ms

What happens is personal history is tiny, a few hundred recent queries per user, so filtering it by prefix at request time is cheap. I fetch a larger global candidate list, top 20 instead of top 5, so the re-ranker has something to choose from, and then a lightweight model scores the merged list.

The reason I don't precompute per-user tries is scale: hundreds of millions of users times all their prefixes is enormous, and most users' histories barely change the result anyway.

Two practical points: personalised responses can't be cached at the CDN, so short prefixes might stay global to keep caching; and there's a privacy angle, with users able to clear history and nothing personal ever leaking into another user's suggestions.

Interviewers look for: candidate generation plus re-ranking, and awareness that personalisation breaks shared caching.

So the honest first answer is: every request that used to hit the cache now hits the database, and if my hit rate was 95 percent, the database suddenly sees twenty times its normal read load. That's how a cache outage turns into a database outage.

normal:    10,000 req/s x 5% miss  =    500 DB reads/s
cache down: 10,000 req/s x 100%    = 10,000 DB reads/s   (20x)

There are really three things I'd have in place:

First, the cache is replicated, Redis with replicas and automatic failover or a managed cluster, so losing one node isn't losing the cache. Second, the database is protected: a connection pool cap, a circuit breaker around it, and load shedding so that under overload I serve some requests well and reject the rest quickly instead of timing out everything. Third, graceful degradation: non-essential features, like recommendations or view counts, return a default rather than hitting the database at all.

And when the cache comes back empty, I'd warm it before sending full traffic, or use request coalescing so a thousand concurrent misses on the same key produce one database query. Otherwise the recovery itself causes a stampede.

Interviewers look for: quantifying the load multiplier, then protection (breaker, shedding) plus recovery (warming, coalescing).

So first I figure out whether producers got faster or consumers got slower, because the fixes are opposite. I'd look at three graphs: produce rate, consume rate, and consumer lag over time.

lag growing, produce rate spiked   -> traffic surge: scale consumers
lag growing, consume rate dropped  -> consumers slowed: find out why
lag growing, one partition only    -> hot key or one stuck consumer
consumers busy but slow            -> downstream (DB, API) is the bottleneck

In my last project we hit the "consume rate dropped" case. Consumers were fine; the database they wrote to had a slow query after a migration. Adding more consumers would have made it worse by hammering the same slow database.

The fixes, depending on cause: scale consumers, up to the partition count for Kafka; batch writes downstream; fix the poison message that's being retried forever by moving it to a dead-letter queue; or, if it's a genuine surge, just let the queue absorb it, because that's exactly what the queue is for, as long as lag recovers.

What I'd also have is an alert on lag measured in time, not message count, like "oldest unprocessed message is 5 minutes old", because that's what users actually feel.

So first I'd find out whether it's a hot key, one celebrity or one viral item, or a hot range, like all new writes landing on the latest time bucket. They need different fixes.

Hot key (e.g. a celebrity's post counter)
  - cache it in front (reads)
  - split the key: count:post:99#0 .. #15, write to random suffix, sum on read
  - batch increments in memory, flush every second

Hot range (e.g. shard key = created_at)
  - change the shard key: hash(user_id) instead of time
  - or add a hash prefix to spread sequential writes

Uneven sizes
  - split the shard, or use many small virtual shards and move some

Let me give you an example. On a social app, one account with 50 million followers made its like counter row the hottest thing in the database. We split the counter into 16 sub-rows, each like wrote to a random one, and reads summed them, cached for a few seconds. Write contention dropped 16x.

Long term, I'd design with many more logical shards than physical machines, like 1,024 virtual shards on 16 nodes, so rebalancing means moving a few virtual shards, not resharding everything.

Interviewers look for: distinguishing hot key from hot range, and key splitting as the standard fix.

So that's the dual-write problem: two systems, no shared transaction, and any crash in between leaves them inconsistent. The fix I reach for is the transactional outbox.

BEGIN;
  INSERT INTO orders (id, user_id, total) VALUES ('o_1', 'u_7', 4999);
  INSERT INTO outbox (id, topic, payload)
         VALUES ('e_1', 'order.created', '{"orderId":"o_1"}');
COMMIT;

relay (poller or CDC like Debezium):
  read outbox rows not yet sent -> publish to Kafka -> mark sent

What happens here is the event is written in the same database transaction as the business data, so either both exist or neither does. A separate relay reads the outbox and publishes. If the relay crashes after publishing but before marking the row sent, it publishes again on restart.

That means the guarantee is at-least-once, so consumers have to be idempotent, usually by storing processed event IDs or using upserts. That's the trade I'm happy to make: duplicates are easy to handle, lost events are not.

The version I'd avoid is "write to the DB, then publish, and retry on failure", because if the process dies between the two, the retry never happens.

So I never make a change that the currently running code can't handle. I use expand and contract: add the new thing, migrate, switch, and only then remove the old thing, each as a separate deploy.

Goal: rename users.name -> users.full_name

1 expand     ALTER TABLE users ADD COLUMN full_name text;   (nullable, instant)
2 dual write deploy app: write both name and full_name, read name
3 backfill   UPDATE ... in batches of 10k, throttled
4 switch     deploy app: read full_name (still write both)
5 contract   deploy app: stop writing name
6 drop       ALTER TABLE users DROP COLUMN name;   (days later)

The reason for all those steps is that during a rolling deploy, old and new versions of the app run at the same time against the same database. At every step, both versions work.

A couple of gotchas I've hit. On Postgres, adding an index on a big table needs CREATE INDEX CONCURRENTLY or it locks writes. Adding a column with a volatile default can rewrite the table. And the backfill must be batched, because one giant UPDATE holds locks and bloats replication lag. On MySQL I'd reach for gh-ost or pt-online-schema-change for large tables.

Interviewers look for: expand/contract, and the insight that old and new code run simultaneously during a deploy.

So the best rollback is one I barely need, because the bad version only ever reached a few percent of users. I'd combine progressive delivery with fast, boring rollback.

deploy v2 -> canary 1% -> watch error rate, p99, key business metric (10 min)
          -> 10% -> 50% -> 100%
automated gate: if error rate > baseline + X, halt and roll back

rollback options, fastest first
  1 feature flag off                   seconds, no deploy
  2 shift traffic back to v1 (blue/green)   seconds
  3 redeploy previous image tag        minutes

What makes rollback actually safe is keeping deploys backward-compatible. If v2 ran a destructive migration, rolling the code back to v1 doesn't bring the dropped column back. That's why I separate schema changes from code changes and use expand and contract.

Feature flags are my favourite tool here: I ship the code dark, turn it on for 1 percent, and if something goes wrong I flip it off without touching the deployment at all.

And after any rollback, I'd do a short blameless review focused on why the canary didn't catch it, because that's the gap to close.

So "slow sometimes" is a tail latency problem, and averages hide it. First I'd look at p99 and p99.9, not the mean, and then find what the slow requests have in common.

1 metrics: p50 fine, p99 spiky? correlated with time, host, endpoint?
2 traces: pull slow traces (tail sampling keeps them), find the long span
3 slice: by host, AZ, customer, payload size, cache hit/miss
4 usual suspects
    GC pauses             -> runtime metrics
    cache miss path       -> slow DB query on cold keys
    one bad host / noisy neighbour
    lock contention, connection pool exhaustion
    retries with backoff hiding inside a "single" call
    big customers with huge payloads

In my last project we had exactly this. p99 on one endpoint jumped to 3 seconds for a few percent of requests. Tracing showed the slow ones always spent the time waiting for a database connection. The pool was sized at 10, and a nightly report job on the same service was holding connections. Separate pool for the batch job, problem gone.

The key tool is distributed tracing with tail-based sampling, so I actually keep the slow traces instead of randomly sampling 1 percent and missing them.

Interviewers look for: percentiles over averages, tracing, and slicing by dimensions to find the common factor.

So first I'd ask how accurate "accurate" needs to be, because it's a spectrum. A like count can lag a few seconds. A bank balance or remaining inventory must be exact.

Display counters (likes, views)
  events -> Kafka (key = itemId) -> aggregate per window -> +delta to DB
  dedupe: (userId, itemId) unique for likes, so a double-tap doesn't count twice
  hot item: sharded counter, sum on read
  unique viewers: HyperLogLog (~12 KB per counter, ~0.8% error)

Exact counters (stock, balance)
  UPDATE items SET stock = stock - 1 WHERE id = :id AND stock > 0;
  one owner per key, no approximation

For likes, the real source of truth is the set of (user, item) like rows with a unique constraint, which makes it idempotent. The count is a derived, cached number I can always recompute from that set if it drifts.

For unique counts, like "unique viewers", storing every ID is expensive, so I'd use HyperLogLog, which Redis supports natively with PFADD and PFCOUNT. It's approximate but tiny.

So in practice: derived counters are eventually consistent and periodically reconciled against the source of truth; money-like counters use a conditional atomic update on a single owner.

So the answer depends on what the business chose for RTO and RPO, how long we can be down and how much data we can lose, because multi-region is expensive. I'd lay out the tiers.

Strategy          RTO        RPO          Cost
backup/restore    hours      hours        low
pilot light       ~1 hour    minutes      low-med
warm standby      minutes    seconds      medium
active-active     ~0         ~0 (or conflicts)  high

For a typical product, I'd go warm standby. Stateless services run at small scale in the second region. The database has an asynchronous cross-region replica. Object storage is cross-region replicated. On failure, I promote the replica, scale up the standby, and move traffic with DNS or a global load balancer with health checks.

The subtle part is data. With async replication, the last few seconds of writes in the dead region may be lost or arrive later, so I'd be upfront about that RPO. Active-active avoids it, but then I have to deal with writes to the same record in two regions, which means conflict resolution or partitioning users by home region.

And the thing people forget: practise it. A failover runbook nobody has run is a hope, not a plan. I'd do game days where we actually fail a region over.

Interviewers look for: RTO/RPO driving the choice, honesty about async replication data loss, and regular failover drills.

So I'd start by being precise: exactly-once delivery over a network isn't achievable, because any message can be lost or retried. What we build is exactly-once effect: at-least-once delivery plus idempotent processing.

async function handlePayment(event) {
  await db.tx(async (t) => {
    const inserted = await t.query(
      `INSERT INTO processed_events (event_id) VALUES ($1)
       ON CONFLICT DO NOTHING RETURNING event_id`, [event.id]);
    if (inserted.rowCount === 0) return;          // duplicate, skip
    await t.query(
      `UPDATE accounts SET balance = balance - $1 WHERE id = $2`,
      [event.amount, event.accountId]);
  });
  // commit the Kafka offset only after the transaction commits
}

What happens here is the dedupe record and the side effect commit in the same transaction. If the consumer crashes after committing but before acking, the message is redelivered, the insert hits the conflict, and we skip it. The effect happens once.

For calls to external systems, like a payment provider, I pass an idempotency key, the event ID, so their side also dedupes retries.

Kafka does have exactly-once semantics with transactions, but that only covers read-from-Kafka, write-to-Kafka pipelines. The moment I write to a database or call an API, I'm back to idempotency.

So first I'd get the bill broken down by service and by category, because cost is almost always concentrated. There's no point optimising a service that's 2 percent of spend.

typical breakdown I've seen
  compute        40%   -> rightsize, autoscale, spot/preemptible for batch,
                         ARM instances (~20% cheaper), commit discounts
  database       25%   -> kill unused replicas, archive cold data,
                         fix the 5 worst queries, reserved capacity
  data transfer  15%   -> cross-AZ chatter, egress: CDN, compression
  storage        10%   -> lifecycle to cold tiers, delete old snapshots/logs
  observability  10%   -> log sampling, shorter retention, drop debug logs

In my experience the quick wins are boring: instances sized for a peak that happened once, dev environments running 24/7, logs retained forever, and services chatting across availability zones, which is billed per GB.

Then the structural wins: committed-use discounts for the steady baseline, autoscaling for the rest, and spot instances for anything retryable like batch jobs or CI. Caching and a CDN often cut both compute and egress at once.

What I would not do is cut redundancy blindly. Halving cost by removing the standby database just moves the cost into a future outage. So I'd frame it as cost per request, keep SLOs as a hard constraint, and track savings weekly.

Interviewers look for: measuring first, attacking the biggest line items, and keeping reliability as a constraint.

So I'd use the strangler fig pattern: put a routing layer in front, carve out one capability at a time, and move traffic gradually. No big-bang rewrite.

          +---------------- API gateway / proxy ----------------+
clients --|  /orders/*   -> new Orders service  (10% -> 100%)    |
          |  everything else -> monolith                         |
          +------------------------------------------------------+

data migration for Orders
  1 new service reads monolith DB (or via an API), writes to it too
  2 new DB, sync via CDC from the monolith tables
  3 shadow reads: compare old vs new results, log mismatches
  4 flip writes to new DB, CDC back to the monolith for old readers
  5 remove old tables when nothing reads them

I'd pick the first service carefully: something with clear boundaries, changing often enough that independence pays off, and not the most critical path on day one. Notifications or search are usually good starters; payments usually aren't.

The data is the hard part, not the code. The monolith probably has joins across the tables I'm extracting. So I'd first stop the monolith touching those tables directly, route it through an internal API, then physically move the data with CDC and shadow reads to prove correctness.

Each step is reversible, since the proxy can route back to the monolith in seconds. And I'd be honest that it takes months or years; the goal is continuous value, not a finish date.

OK, so first I'd set assumptions and say them out loud so the interviewer can correct me. I'll use round numbers and the fact that a day is about 86,400 seconds, which I round to 100,000 when I need speed.

Assumptions
  DAU                      300M
  feed reads / user / day  10
  tweets / day             100M   (most users only read)
  tweet size               ~1 KB  (280 chars + ids, timestamps, metadata)
  10% of tweets have media, avg 500 KB

Read QPS    300M x 10 = 3B / day
            3B / 86,400        = ~35,000 / sec avg
            peak (x3)          = ~100,000 / sec
Write QPS   100M / 86,400      = ~1,200 / sec avg, ~3,500 peak

Storage
  text      100M x 1 KB        = 100 GB / day  -> ~36.5 TB / year
  media     10M x 500 KB       = 5 TB / day    -> ~1.8 PB / year

What jumps out: reads outnumber writes about 30 to 1, so this is a read-heavy system and I'd precompute feeds with fan-out on write, with caching. Text storage is small enough for a sharded database; media dominates storage and goes to object storage behind a CDN.

Also, 100,000 reads a second is not scary for a cache tier, so the real challenge is fan-out cost on write for accounts with millions of followers, not raw QPS.

Interviewers look for: stated assumptions, a peak factor, and a conclusion drawn from the numbers, not just the numbers.

OK, let me write this out. The key is that I store several sizes of each photo, not just the original.

Assumptions
  uploads / day            10M photos
  original (compressed)    2 MB
  thumbnails + medium      ~300 KB total
  per photo                ~2.3 MB

Per day     10M x 2.3 MB      = 23 TB / day
Per year    23 TB x 365       = ~8.4 PB / year
5 years     8.4 PB x 5        = ~42 PB   (flat growth)
            with 30%/yr growth: roughly 2x that, ~75-80 PB

Metadata    10M x 500 B       = 5 GB / day -> ~9 TB over 5 years

So what I'd take from this: photo bytes are tens of petabytes, which only makes sense in object storage like S3, with durability handled by the provider's replication, so I don't multiply by three myself. Metadata is under 10 TB, which fits a sharded relational database or a wide-column store easily.

Then the cost angle. Most photos are rarely viewed after the first month, so I'd move originals older than, say, 90 days to an infrequent-access tier and keep only the thumbnails hot. That can cut the storage bill by more than half without users noticing.

OK, so first I'd estimate traffic, then apply the 80/20 rule: a small fraction of links gets most of the clicks, so I only cache the hot ones.

Assumptions
  new URLs / month      100M
  read:write ratio      100:1  -> 10B redirects / month
  entry size            ~500 B (short code, long URL, metadata)

Redirect QPS   10B / (30 x 86,400)   = ~3,900 / sec avg, ~12k peak
Daily reads    10B / 30              = ~333M / day
Cache 20%      333M x 0.2 x 500 B    = ~33 GB

So about 33 GB, and that's an overestimate, because many of those 333 million daily reads are repeat hits on the same URLs, so the distinct hot set is smaller.

What that tells me design-wise: this fits in memory on a single Redis node. I'd still run a small cluster, say 3 primaries with replicas, for availability rather than capacity, with LRU eviction so the hot set manages itself. And since short links never change once created, I can cache aggressively, and even put a CDN in front of redirects with a long TTL.

Interviewers look for: the 80/20 assumption stated explicitly, and noticing the answer means "one node is enough, cluster for HA".

OK, so for streaming, bandwidth is the number that drives cost, so I'd do both peak throughput and daily volume.

Assumptions
  peak concurrent viewers    10M
  avg bitrate                5 Mbps (mix of 1080p and mobile)
  hours watched / day        1B

Peak egress   10M x 5 Mbps              = 50 Tbps
Per hour      5 Mbps / 8 = 0.625 MB/s x 3,600 = 2.25 GB / hour
Daily volume  1B hours x 2.25 GB        = 2.25 EB / day

Ingest (for contrast)
  500 h uploaded / min x ~1 GB / h      = 500 GB/min = ~8.3 GB/s = ~67 Gbps

What that tells me: 50 terabits a second is far beyond what any single origin could serve. That's why a CDN, and for the biggest players, boxes inside ISPs, is mandatory. Egress is roughly 750 times bigger than ingest.

It also points at the cost levers. Better codecs like AV1 cut bitrate roughly 30 percent at the same quality, which is 30 percent off the biggest line item. Serving the right resolution for the device matters too: sending 4K to a phone is pure waste. And the CDN cache hit rate directly determines origin egress, so pushing it from 90 to 98 percent cuts origin traffic by 5x.

OK, so for chat the limiting factor is usually concurrent connections, not CPU, so I'd size the connection tier first and the message path second.

Assumptions
  DAU                         500M
  peak concurrent online      10%  -> 50M WebSocket connections
  connections per server      ~100k (safe; tuned boxes do 500k+)
  messages / user / day       40

Connection servers  50M / 100k          = 500
  + 30% headroom for failover + deploys = ~650 servers

Messages       500M x 40                 = 20B / day
               20B / 86,400              = ~230k / sec avg, ~700k peak
Storage        20B x 200 B               = 4 TB / day

So about 650 gateway servers. The reason I don't pack each one to the maximum is failure blast radius: if a server with 500,000 connections dies, all of those clients reconnect at once to the others. With 100,000 per box and jittered reconnects, that's absorbable.

The message path is a separate tier: 700,000 messages a second at peak going into something like Cassandra, which is write-optimised and handles that across a few dozen nodes. And I'd need a presence or session store mapping user to gateway server, which is 50 million entries, a few GB in Redis.

Interviewers look for: sizing by connections rather than QPS, and headroom justified by reconnect storms.

OK, so partitions set the ceiling on parallelism, because within a consumer group each partition is read by at most one consumer. So I size from target throughput divided by what one partition can handle on each side.

Assumptions
  peak events         1M / sec
  event size          1 KB   -> 1 GB/s throughput
  producer per partition   ~50 MB/s  (measure; depends on batching)
  consumer per partition   ~10 MB/s  (our processing does DB writes)

partitions = max(T / producer, T / consumer)
           = max(1000 / 50, 1000 / 10) = max(20, 100) = 100
growth x2-3 over next 2 years          -> ~250-300 partitions

Retention  1 GB/s x 86,400        = 86.4 TB / day
           x 3 days x RF 3         = ~780 TB raw
           with ~4x compression    = ~200 TB of disk

What I'm doing there: the consumer side is the bottleneck, because our consumers do real work, so that's what sets 100. Then I add headroom, because adding partitions later changes which partition a key hashes to and breaks per-key ordering during the change. It's much easier to over-provision up front.

The other side is not going crazy. Every partition costs file handles, memory and leader elections, and very high counts per broker slow recovery. A few hundred for this pipeline, spread over, say, 12 to 15 brokers, is comfortable.

And I'd call out that those per-partition numbers are assumptions I'd verify with a load test on our actual payloads before committing.