Backend Developer — Concept Notes

1. What Backend Really Is

The backend is everything that runs on a machine you control rather than the user's device. The browser is untrusted — anyone can edit the HTML, disable JS, or call your API directly with curl. So the backend is the place where the real decisions happen.

Core responsibilities

• Persistence — storing data so it survives restarts (databases, object storage).
• Business logic — the rules: can this user cancel this order? what is the discount?
• Authentication & authorization — who is calling, and are they allowed?
• Validation & integrity — the last line of defence against bad or malicious data.
• Integration — payment gateways, email/SMS, third-party APIs, AI services.
• Performance & scale — caching, indexing, background jobs, load handling.
• Observability — logs, metrics, alerts so you know what broke and why.

Key mental model — the request lifecycle

Almost every backend framework in every language is a variation of this pipeline. Learn the pipeline and switching stacks becomes easy.

2. How the Internet Works (the part backend devs need)

DNS

Domain names map to IP addresses. The resolver checks browser cache → OS cache → ISP resolver → root → TLD → authoritative nameserver. Records you'll meet: A (IPv4), AAAA (IPv6), CNAME (alias), MX (mail), TXT (verification/SPF). TTL controls how long records are cached — this is why DNS changes "take time to propagate".

TCP vs UDP

• TCP — connection-oriented, ordered, reliable (3-way handshake SYN → SYN-ACK → ACK). HTTP, databases, SSH.
• UDP — fire-and-forget, no ordering guarantees, low latency. DNS, video calls, gaming, QUIC/HTTP3.

TLS / HTTPS

TLS encrypts the TCP connection. The handshake verifies the server's certificate (signed by a CA), agrees on a cipher, and derives a shared symmetric key. HTTPS gives you confidentiality (nobody can read it), integrity (nobody can modify it) and authenticity (you're talking to the real server). Certificates are usually free via Let's Encrypt and auto-renewed.

Ports, proxies and load balancers

• A port identifies a process on a machine: 80 (HTTP), 443 (HTTPS), 5432 (Postgres), 27017 (MongoDB), 6379 (Redis), 3000 (typical Node dev).
• A reverse proxy (Nginx, Caddy) sits in front of your app: TLS termination, gzip, static files, rate limiting, routing to multiple apps.
• A load balancer spreads traffic over multiple instances (round-robin, least-connections). It also health-checks instances and removes dead ones.
• A CDN caches static assets at edge locations near users.

3. HTTP in Depth

Why "stateless" is the root of so much design

HTTP has no memory. Two requests from the same person arriving one second apart are, as far as the protocol is concerned, completely unrelated — there is no connection identity the server can rely on. Every mechanism you'll meet later exists to work around this single fact:

• Cookies — the server asks the browser to send a token back on every request.
• Sessions / JWT — that token is how "who is this?" gets answered each time.
• Idempotency keys — because the server can't tell a retry from a genuine second request.
• Horizontal scaling works at all — precisely because no request depends on the last one, any instance can serve any request.

HTTP is a stateless, text-based request/response protocol. Stateless means the server doesn't remember anything between requests — which is exactly why we need cookies, sessions and tokens.

Anatomy

Methods and their properties

Safe means the request doesn't change server state (GET, HEAD, OPTIONS) — a crawler can call it freely. Idempotent means calling it N times leaves the same state as calling it once. This is not about the response body; it's about the effect.

PUT is idempotent because "set the name to Ada" applied twice still leaves the name as Ada. POST is not, because "create an order" applied twice creates two orders. DELETE is idempotent in effect — the row is gone either way — even though the second call may return 404.

Safe = doesn't change server state. Idempotent = doing it 5 times has the same effect as once. This matters because clients and proxies retry failed requests — a non-idempotent POST retried can create duplicate orders. Fix with an idempotency key: the client sends a unique key, the server stores the result for that key and returns the same response on retry.

Status codes — how to choose

HTTP versions

• HTTP/1.1 — one request at a time per connection (head-of-line blocking); keep-alive reuses connections.
• HTTP/2 — multiplexes many streams over one TCP connection, header compression, binary framing.
• HTTP/3 — runs over QUIC (UDP): faster handshakes, no TCP head-of-line blocking, better on flaky mobile networks.

4. Cookies, Caching and CORS

Cookies

A cookie is a small key/value the server sets via Set-Cookie and the browser automatically returns on every matching request. That automatic behaviour is both the convenience and the danger (CSRF).

HTTP caching

• Cache-Control is the main control: no-store (never cache — use for private data), no-cache (cache but revalidate), max-age=3600, public/private, stale-while-revalidate.
• Validators: ETag (content hash) and Last-Modified. The client sends If-None-Match; if unchanged the server replies 304 with no body — saves bandwidth.
• Cache invalidation is famously hard. The usual trick for static assets is content hashing in filenames (app.9f2c.js) so you can cache forever and change the name on deploy.

CORS (Cross-Origin Resource Sharing)

Browsers enforce the Same-Origin Policy: JS on a.com can't read responses from b.com unless b.com allows it. An origin = scheme + host + port. CORS is the server saying "this origin may read my responses" via Access-Control-Allow-Origin.

• Simple requests (GET/POST with basic headers) go straight out; the browser just blocks reading the response if headers are missing.
• Preflight: for other methods/headers the browser first sends an OPTIONS request and only proceeds if allowed.
• To send cookies cross-origin you need credentials: "include" on the client and Access-Control-Allow-Credentials: true plus an explicit origin (wildcard * is not allowed with credentials).

5. Runtime Architecture

Why this design scales

A traditional thread-per-request server allocates ~1MB of stack per connection and pays a context switch each time it swaps. 10,000 idle connections means 10,000 mostly-sleeping threads. Node keeps one thread and a queue of callbacks, so idle connections cost almost nothing — they're just entries the kernel is watching.

Node.js is a JavaScript runtime: the V8 engine (compiles and runs JS) plus libuv (event loop, thread pool, async I/O) plus Node's own C++/JS standard library (fs, http, crypto…).

Blocking vs non-blocking — the core idea

A traditional server (classic PHP/Java model) uses one thread per request. 10,000 concurrent requests means 10,000 threads, each with its own memory stack and context-switching cost. Most of those threads are just waiting on the database or disk, doing nothing.

Node instead uses one thread running an event loop. When a request needs I/O, Node hands the work to the OS (or a thread pool) and immediately moves on to the next request. When the I/O finishes, a callback is queued and executed. Waiting costs almost nothing.

6. The Event Loop in Depth

The event loop runs in phases, forever. Each phase has a queue of callbacks; the loop drains a phase, then moves to the next.

Microtasks vs macrotasks

• Microtasks = process.nextTick() then promise callbacks. They run after the current operation, before the loop continues — and the queue is drained completely.
• Macrotasks = timers, I/O callbacks, setImmediate — one phase at a time.
• An infinite nextTick chain can starve the event loop, because the loop never gets to move on.

console.log("1 sync");
setTimeout(() => console.log("5 timeout"), 0);
setImmediate(() => console.log("6 immediate"));
Promise.resolve().then(() => console.log("4 promise"));
process.nextTick(() => console.log("3 nextTick"));
console.log("2 sync");

// 1 sync → 2 sync → 3 nextTick → 4 promise → 5 timeout → 6 immediate
// (timeout vs immediate order at top level is not guaranteed;
//  inside an I/O callback, setImmediate always fires first)

Blocking the loop — what it looks like

// BAD: freezes every other request for seconds
app.get("/report", (req, res) => {
  let total = 0;
  for (let i = 0; i < 1e10; i++) total += i;   // CPU-bound, synchronous
  res.json({ total });
});

// Also blocking: fs.readFileSync, JSON.parse on a 100MB string,
// bcrypt with a huge cost factor in sync mode, big RegExp backtracking (ReDoS)

Fixes: use async APIs, move CPU work to a worker thread, push it to a queue processed by a separate worker service, paginate/stream large data, or delegate to a specialised service.

7. Async Patterns

Evolution

• Callbacks — error-first convention (err, data). Nesting leads to "callback hell" and error handling is easy to get wrong.
• Promises — an object representing a future value with states pending → fulfilled / rejected. Chainable, composable.
• async/await — syntactic sugar over promises; reads like sync code, uses try/catch. An async function always returns a promise.

Combinators — know the differences

// Sequential (slow) — 300ms
const user = await getUser(id);
const orders = await getOrders(id);

// Parallel (fast) — 150ms
const [user, orders] = await Promise.all([getUser(id), getOrders(id)]);

// Timeout pattern
const withTimeout = (p, ms) => Promise.race([
  p,
  new Promise((_, rej) => setTimeout(() => rej(new Error("timeout")), ms)),
]);

Common async mistakes

• forEach with async doesn't wait — use for...of (sequential) or Promise.all(map(...)) (parallel).
• Forgetting await → you return a pending promise and errors become unhandled rejections.
• Not handling rejections → in modern Node an unhandled rejection crashes the process.
• Firing thousands of parallel promises at once → exhausts DB connections. Use a concurrency limiter (e.g. p-limit) or batch.

8. Streams & Buffers

A Buffer is a fixed-length chunk of raw binary memory outside V8's heap — how Node represents bytes (file contents, network packets, images).

A Stream processes data chunk by chunk instead of loading it all into memory. Reading a 2 GB file with readFile needs 2 GB of RAM; streaming it needs only the size of one chunk. Streams are the reason Node handles large uploads/downloads well.

import { pipeline } from "node:stream/promises";

// Stream a file to the client, gzipped, with proper error handling
await pipeline(
  fs.createReadStream("big.csv"),
  zlib.createGzip(),
  res
);

Backpressure is the key concept: if the destination is slower than the source, the writable stream signals "slow down" and the readable pauses. pipe()/pipeline() handle this automatically; manual data event handling usually doesn't — which causes memory blowups.

9. Modules, Memory and Processes

Module systems

Modules are cached after first load — that's why a database connection file can export a single shared connection (a natural singleton).

Memory and garbage collection

• V8 splits the heap into new space (short-lived objects, scavenged often) and old space (survivors, collected with mark-sweep-compact).
• Memory leaks in Node usually come from: global arrays/maps that only grow, forgotten event listeners, uncleaned timers, closures holding big objects, unbounded in-memory caches.
• Symptoms: RSS climbing steadily, slow GC pauses, eventual heap out of memory crash. Investigate with heap snapshots (--inspect + Chrome DevTools) or clinic.js.

Using all CPU cores

• Cluster module / PM2 — fork one process per core, all sharing a port. Processes don't share memory, so state must live outside the process (Redis, DB).
• Worker threads — real threads inside one process for CPU-bound tasks, communicating by messages/shared memory.
• child_process — run external programs (spawn streams output, exec buffers it; never pass unsanitised user input to a shell).

npm essentials

• Semver MAJOR.MINOR.PATCH: major = breaking, minor = features, patch = fixes. ^1.2.3 allows minor+patch, ~1.2.3 only patch, exact pins everything.
• package-lock.json pins the whole tree — commit it; use npm ci in CI for reproducible installs.
• dependencies ship to production; devDependencies don't. Audit with npm audit.

10. Express Deep Dive

Express is a thin layer over Node's http module giving you routing and middleware. Understanding middleware is understanding Express.

The middleware chain

Express keeps an ordered array of functions. Each request walks the array. A middleware either sends a response (chain ends) or calls next() (continue). Calling next(err) jumps straight to the error handler. Nothing happens if you forget both — the request hangs.

import express from "express";
const app = express();

app.use(helmet());                       // security headers
app.use(cors({ origin: ALLOWED, credentials: true }));
app.use(express.json({ limit: "10kb" })); // parse + limit payload size
app.use(morgan("dev"));                  // logging
app.use("/api", rateLimit({ windowMs: 60_000, max: 100 }));

app.use("/api/v1/users", userRoutes);    // routers

app.use((req, res) => res.status(404).json({ message: "Not found" }));
app.use(errorHandler);                   // must be LAST, 4 args

Middleware categories

• Application-level — app.use(fn), runs for everything (or a path prefix).
• Router-level — scoped to one router, e.g. auth on /admin.
• Route-level — router.post("/", validate(schema), auth, controller).
• Built-in — express.json, express.urlencoded, express.static.
• Error-handling — four parameters (err, req, res, next).

Request and response

Alternatives worth knowing

• Fastify — faster, schema-based validation and serialization built in.
• NestJS — opinionated, TypeScript, decorators, dependency injection; good for large teams.
• Hono / Elysia — tiny, fast, run on edge runtimes.

11. Project Architecture

Why separate layers at all?

Beginners reasonably ask why a 10-line database call needs three files. The answer is what each layer doesn't know:

• The controller knows HTTP — status codes, headers, req.body — and nothing about business rules.
• The service knows the rules — "a user can't cancel a shipped order" — and nothing about HTTP. It never sees req or res.
• The repository knows the database and nothing else.

The payoff is concrete. When a background job or a CLI script needs to cancel an order, it calls the same service — there's no req object to fake. When you swap Postgres for MongoDB, only the repository changes. When you test the business rules, you don't spin up an HTTP server.

The goal is separation of concerns: each layer knows only about the layer below it. The test is simple — could you swap Express for Fastify, or Mongo for Postgres, by touching only one layer?

// controller — thin, HTTP only
export const createUser = asyncHandler(async (req, res) => {
  const dto = createUserSchema.parse(req.body);
  const user = await userService.create(dto);
  res.status(201).json({ data: user });
});

// service — business rules, reusable from a CLI or a queue worker
export const create = async (dto) => {
  if (await userRepo.findByEmail(dto.email))
    throw new AppError("Email already registered", 409);
  const password = await bcrypt.hash(dto.password, 12);
  const user = await userRepo.insert({ ...dto, password });
  await mailer.sendWelcome(user.email);
  return sanitize(user);
};

Why it matters: controllers full of business logic can't be reused by a cron job or a WebSocket handler, are hard to unit test (you need fake req/res), and turn into 500-line files.

12. Error Handling & Validation

A single error pipeline

• Define an AppError with statusCode and an isOperational flag.
• Operational errors (invalid input, not found, payment declined) are expected → return a clean 4xx.
• Programmer errors (undefined is not a function) are bugs → log fully, return a generic 500, and consider restarting the process.
• Throw from anywhere; one central middleware formats the response. Never build error JSON in twenty places.

export class AppError extends Error {
  constructor(message, statusCode = 500) {
    super(message);
    this.statusCode = statusCode;
    this.isOperational = true;
    Error.captureStackTrace(this, this.constructor);
  }
}

export const asyncHandler = (fn) => (req, res, next) =>
  Promise.resolve(fn(req, res, next)).catch(next);   // no try/catch everywhere

export const errorHandler = (err, req, res, next) => {
  logger.error({ err, path: req.path, id: req.id });
  const status = err.isOperational ? err.statusCode : 500;
  res.status(status).json({
    message: err.isOperational ? err.message : "Something went wrong",
    ...(process.env.NODE_ENV !== "production" && { stack: err.stack }),
  });
};

Process-level safety nets

process.on("unhandledRejection", (err) => { logger.fatal(err); shutdown(1); });
process.on("uncaughtException",  (err) => { logger.fatal(err); process.exit(1); });
process.on("SIGTERM", shutdown);   // graceful: stop accepting, finish in-flight, close DB

Validation

Validate at the boundary — body, params, query, headers, webhooks, env vars. Prefer a schema library (Zod, Joi) so validation is declarative, reusable and (with Zod/TS) gives you types for free.

const createUserSchema = z.object({
  email: z.string().email(),
  password: z.string().min(8),
  age: z.number().int().min(13).optional(),
}).strict();                       // reject unknown keys → prevents mass assignment

export const validate = (schema) => (req, res, next) => {
  const result = schema.safeParse(req.body);
  if (!result.success)
    return res.status(422).json({ errors: result.error.flatten().fieldErrors });
  req.body = result.data;          // use the parsed, coerced, trimmed data
  next();
};

13. REST API Design

Principles

• Resources, not actions. POST /orders, not POST /createOrder. Use plural nouns and nest only one level deep: /users/7/orders.
• Stateless — every request carries its own authentication.
• Consistent envelopes — one shape for success, one for errors, everywhere.
• Versioning — /api/v1/ in the URL (simplest) or via headers. Never break an existing version; deprecate with warnings.

GET    /api/v1/products?category=books&sort=-price&page=2&limit=20
POST   /api/v1/products
GET    /api/v1/products/42
PATCH  /api/v1/products/42
DELETE /api/v1/products/42
GET    /api/v1/products/42/reviews

Pagination

{
  "data": [ ... ],
  "meta": { "page": 2, "limit": 20, "total": 348, "totalPages": 18 }
}
{
  "message": "Validation failed",
  "code": "VALIDATION_ERROR",
  "errors": { "email": ["Invalid email"] },
  "requestId": "a1b2c3"
}

Other design decisions

• Filtering/sorting/search — keep them in the query string with a whitelist of allowed fields (never interpolate them into SQL).
• Partial responses — ?fields=id,name to reduce payload.
• Rate limiting — return 429 plus X-RateLimit-* and Retry-After.
• Soft delete — set deletedAt instead of removing rows when you need audit/undo.
• Documentation — OpenAPI/Swagger spec; generate it from your schemas so it never goes stale.

REST vs GraphQL vs gRPC vs tRPC

14. Files, Email and Real-time

File uploads

• Parse multipart/form-data with multer (or busboy for streaming).
• Store files in object storage (S3, R2, Cloudinary), not in your database or app server — app servers are disposable.
• Presigned URLs are the scalable pattern: the client uploads directly to S3 with a short-lived signed URL, so file bytes never touch your server.
• Always validate: size limit, MIME type and magic bytes (extensions lie), random filenames, and never serve user uploads from your own domain without care.

Email & notifications

Use a provider (Resend, SendGrid, SES) — never your own SMTP for production deliverability. Send emails from a background job, not inside the request, so a slow provider doesn't slow your API. Deliverability depends on SPF, DKIM and DMARC DNS records.

Real-time

With multiple server instances, WebSocket connections live on different processes — a message from user A on server 1 won't reach user B on server 2. Solve with a Redis pub/sub adapter (e.g. socket.io-redis) so servers broadcast to each other.

15. Database Fundamentals

Connection pooling — why you never open a connection per request

Opening a database connection is expensive: a TCP handshake, authentication, and server-side memory allocated per connection. Doing that per request adds tens of milliseconds and will exhaust the database's connection limit (Postgres defaults to ~100) under load.

A pool opens a fixed set of connections at startup and lends them out. A request borrows one, runs its query, and returns it. If all are busy, the request waits briefly rather than opening a new one.

const pool = new Pool({ max: 20, idleTimeoutMillis: 30000 });
// 20 connections serving hundreds of concurrent requests, because each
// query only holds one for a few milliseconds

Other database types

• Key–value (Redis, DynamoDB) — cache, sessions, counters, extremely fast lookups.
• Search (Elasticsearch, Meilisearch, Postgres FTS) — ranked full-text search, typo tolerance.
• Time-series (TimescaleDB, InfluxDB) — metrics and IoT data.
• Graph (Neo4j) — relationship-heavy data: social graphs, recommendations, fraud rings.
• Columnar / OLAP (ClickHouse, BigQuery) — analytics over billions of rows.

OLTP vs OLAP

OLTP = many small reads/writes (your app). OLAP = few huge analytical scans (dashboards, reports). Running heavy analytics on your production OLTP database is a classic cause of outages — use a read replica or a separate warehouse.

CAP theorem

In a distributed system, when a network Partition happens you must choose between Consistency (reject requests rather than serve stale data) and Availability (serve possibly stale data). Single-node databases aren't in this trade-off; distributed ones are. Related idea: eventual consistency — replicas converge after a short delay, which is why a read right after a write may miss it.

16. SQL in Depth

-- Filter, sort, page
SELECT id, name, price FROM products
WHERE category = 'books' AND price BETWEEN 100 AND 900
ORDER BY price DESC
LIMIT 20 OFFSET 40;

-- Join + aggregate: revenue per customer, only big spenders
SELECT u.id, u.name, COUNT(o.id) AS orders, SUM(o.total) AS revenue
FROM users u
JOIN orders o ON o.user_id = u.id
WHERE o.created_at >= NOW() - INTERVAL '30 days'
GROUP BY u.id, u.name
HAVING SUM(o.total) > 10000
ORDER BY revenue DESC;

Execution order (why you can't use an alias in WHERE)

WHERE filters rows before grouping; HAVING filters groups after aggregation. SELECT runs late, so aliases defined there aren't visible to WHERE.

Joins

Keys and constraints — let the database protect you

• PRIMARY KEY — unique row identity. FOREIGN KEY — referential integrity, with ON DELETE CASCADE / SET NULL / RESTRICT.
• UNIQUE, NOT NULL, CHECK (price >= 0), DEFAULT.
• Constraints in the DB are enforced even when a buggy script or a second service writes data — application-level checks are not.

Normalization vs denormalization

• 1NF atomic values, no repeating groups · 2NF no partial dependency on part of a composite key · 3NF no non-key column depending on another non-key column.
• Normalize to avoid duplication and update anomalies; denormalize deliberately (cached counts, duplicated names) when reads dominate — and then keep the copies in sync.

Useful advanced tools

• Window functions: ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY created_at DESC) — rank within groups without collapsing rows.
• CTEs: WITH recent AS (...) SELECT ... — readable multi-step queries, recursion for trees.
• Upsert: INSERT ... ON CONFLICT (email) DO UPDATE SET ....
• Views for reusable queries; materialized views for cached expensive aggregations.

17. Indexes & Query Performance

Why an index is not free

Every index must be updated on every INSERT, UPDATE and DELETE, and it consumes disk and memory. That is the trade: faster reads, slower writes. Index the columns you filter, join and sort on — not every column.

An index is a sorted data structure (usually a B-tree) mapping column values to row locations. Without one, the DB does a sequential scan — reading every row. With one, it does a tree lookup: milliseconds instead of seconds on large tables.

Costs and rules

• Every index slows writes (each INSERT/UPDATE must update the index) and uses disk. Don't index everything.
• Index the columns you filter, join and sort by — especially foreign keys.
• Composite indexes follow the leftmost prefix rule: an index on (user_id, created_at) helps queries filtering on user_id, or user_id + created_at — but not created_at alone.
• Low selectivity columns (a boolean, a status with 3 values) benefit little from a plain B-tree index.
• An index is not used if you wrap the column in a function (WHERE LOWER(email) = ...) — create a functional index instead.
• Covering index — when the index contains every column the query needs, the DB never touches the table.

EXPLAIN ANALYZE
SELECT * FROM orders WHERE user_id = 42 ORDER BY created_at DESC LIMIT 10;
-- Look for: Seq Scan (bad on big tables) vs Index Scan,
-- rows estimated vs actual, and the slowest node's time.

CREATE INDEX idx_orders_user_created ON orders (user_id, created_at DESC);

The N+1 query problem

// BAD — 1 + N queries
const posts = await Post.find();
for (const p of posts) p.author = await User.findById(p.authorId);

// GOOD — 2 queries (or 1 with a JOIN)
const posts = await Post.find().populate("author");
const posts = await prisma.post.findMany({ include: { author: true } });

Other performance levers

• Connection pooling — opening a DB connection is expensive; reuse a pool. In serverless, use a pooler (PgBouncer, Prisma Accelerate) or you'll exhaust connections.
• Select only needed columns; avoid SELECT * on wide tables.
• Batch inserts/updates instead of looping one query at a time.
• Read replicas for heavy read traffic; partitioning for very large tables.
• Slow query log — always measure before optimising.

18. Transactions & Concurrency

A transaction groups statements so they succeed or fail together. The classic example is a money transfer: debit and credit must both happen or neither.

BEGIN;
  UPDATE accounts SET balance = balance - 500 WHERE id = 1;
  UPDATE accounts SET balance = balance + 500 WHERE id = 2;
COMMIT;   -- or ROLLBACK on any error

ACID

• Atomicity — all or nothing.
• Consistency — constraints hold before and after.
• Isolation — concurrent transactions don't corrupt each other.
• Durability — once committed, it survives a crash (write-ahead log).

Isolation levels and the anomalies they prevent

Dirty read = seeing uncommitted data. Non-repeatable read = same row returns different values within one transaction. Phantom read = a repeated range query returns new rows. Higher isolation = more correctness, less concurrency.

Race conditions and locking

• Lost update: two requests read stock = 1 and both sell it. Fix with SELECT ... FOR UPDATE (pessimistic lock) or an atomic statement like UPDATE ... SET stock = stock - 1 WHERE stock > 0.
• Optimistic locking: keep a version column; update WHERE version = X and fail with 409 if zero rows changed. Great for low-contention data and user-facing edit conflicts.
• Deadlock: two transactions lock resources in opposite order. Prevent by always locking in a consistent order and keeping transactions short.
• Keep transactions short — never call an external API inside one.

19. MongoDB / NoSQL Modelling

const orderSchema = new mongoose.Schema({
  user:   { type: mongoose.Schema.Types.ObjectId, ref: "User", index: true },
  items:  [{ product: { type: ObjectId, ref: "Product" }, qty: Number, price: Number }],
  status: { type: String, enum: ["pending","paid","shipped"], default: "pending" },
  total:  { type: Number, required: true, min: 0 },
}, { timestamps: true });

orderSchema.index({ user: 1, createdAt: -1 });   // compound index

Embed vs reference — the central modelling decision

MongoDB has a 16 MB document limit, so unbounded arrays are an anti-pattern. Model around your read patterns, not around abstract entity diagrams.

Querying

• Operators: $eq $gt $gte $lt $in $nin $and $or $exists $regex; updates: $set $unset $inc $push $pull $addToSet.
• populate() is a client-side join — it issues extra queries. $lookup joins inside the aggregation pipeline.
• Aggregation pipeline: $match → $group → $sort → $project → $lookup → $facet. Put $match and $sort first so indexes can be used.
• Mongoose gives you schemas, validation, middleware (pre("save") for hashing passwords), and virtuals.
• Multi-document transactions exist but need a replica set and are slower — prefer designs where one document is the unit of consistency.

20. ORMs, Query Builders & Migrations

The N+1 query problem

This is the most common performance bug in ORM code, and a near-guaranteed interview question. It happens when you load a list, then access a relation inside a loop — the ORM quietly issues one more query per row.

// 1 query for the posts…
const posts = await Post.find();
for (const post of posts) {
  // …then ONE MORE query per post. 100 posts = 101 queries.
  post.author = await User.findById(post.userId);
}

// Fix 1 — let the database join (SQL)
SELECT p.*, u.name FROM posts p JOIN users u ON u.id = p.user_id;

// Fix 2 — eager load (Prisma / Sequelize / Mongoose)
await prisma.post.findMany({ include: { author: true } });
await Post.find().populate("author");

// Fix 3 — batch the lookup yourself: 2 queries total, whatever the row count
const posts = await Post.find();
const users = await User.find({ _id: { $in: posts.map(p => p.userId) } });
const byId = new Map(users.map(u => [String(u._id), u]));
posts.forEach(p => { p.author = byId.get(String(p.userId)); });

It's insidious because it works perfectly in development with 10 rows and collapses in production with 10,000. Log your queries in development — if one page request produces 200 queries, you've found it.

const users = await prisma.user.findMany({
  where: { active: true, orders: { some: { total: { gt: 1000 } } } },
  include: { orders: { take: 5, orderBy: { createdAt: "desc" } } },
  skip: 20, take: 10,
});

Migrations

Migrations are version control for your schema: ordered, committed files that every environment applies in the same order. Never change production schema by hand.

• Each migration should be reversible where possible and tested on a copy of production data.
• Zero-downtime pattern for renaming a column: add the new column → write to both → backfill → switch reads → stop writing the old → drop it. Several deploys, no outage.
• Adding an index on a huge table can lock it — use CREATE INDEX CONCURRENTLY in Postgres.
• Seeding scripts create predictable dev/test data.

21. Caching & Redis

The two hard parts

• Invalidation. When the underlying row changes, the cached copy is wrong. Either delete the key on write (DEL user:7) or accept staleness until the TTL expires. Deleting is more correct; TTL is simpler and usually enough.
• Stampede. A popular key expires and 500 requests miss simultaneously, all hammering the database at once. Fixes: a short lock so one request refills while others wait, or staggered/jittered TTLs so keys don't expire together.

Caching trades freshness for speed. Add it only where you've measured a real cost, and always decide up front how the cache gets invalidated.

Layers of caching

Patterns

• Cache-aside (lazy): read cache → miss → read DB → write cache with a TTL. The default choice.
• Write-through: write DB and cache together — fresher, slightly slower writes.
• Write-behind: write cache now, DB later — fast but risky.
• TTL + invalidation on write: delete the key whenever the underlying data changes.

async function getProduct(id) {
  const key = `product:${id}`;
  const hit = await redis.get(key);
  if (hit) return JSON.parse(hit);

  const product = await db.product.findUnique({ where: { id } });
  await redis.set(key, JSON.stringify(product), "EX", 300);   // 5 min TTL
  return product;
}
// on update: await redis.del(`product:${id}`)

What Redis is used for besides caching

• Sessions — shared across many server instances.
• Rate limiting — atomic INCR with an expiry per user/IP.
• Queues — BullMQ builds on Redis lists/streams.
• Pub/Sub — broadcasting between server instances (WebSockets).
• Distributed locks, counters, leaderboards (sorted sets).

Cache hazards

• Stampede / thundering herd: a hot key expires and thousands of requests hit the DB at once. Fix with a lock, staggered TTLs, or serving stale while revalidating.
• Stale data after updates — invalidate on write, don't just wait for TTL.
• Never cache private data under a shared key — include the user ID in the key.
• Redis is memory-bound: set maxmemory and an eviction policy (allkeys-lru).

22. Authentication Concepts

• Identification — who you claim to be. Authentication — proving it. Authorization — what you may do.
• Authorization models: RBAC (roles: admin/editor/user), ABAC (attributes/conditions), ownership checks (this document belongs to this user).

Password storage

• Never store plaintext, and never use fast hashes (MD5, SHA-256) for passwords.
• Use bcrypt (cost 12ish) or argon2id: deliberately slow and memory-hard, which makes brute forcing expensive.
• A salt (unique random value per password, handled automatically by bcrypt) defeats rainbow tables and hides identical passwords.
• Compare with the library's compare(), which is timing-safe. Return the same generic "invalid credentials" message for both wrong email and wrong password (no user enumeration).
• Extras: rate-limit and lock out after failed attempts, check against breached-password lists, offer MFA/TOTP, expire reset tokens quickly and single-use them.

const hash = await bcrypt.hash(password, 12);
const ok = await bcrypt.compare(input, user.passwordHash);

23. Sessions, JWT and OAuth

Reading the trade-off properly

Interviewers push on revocation. A JWT is valid until it expires — you cannot un-issue it. The standard answer is a short-lived access token (5–15 min) plus a long-lived refresh token stored server-side, which can be revoked. You accept a small window where a stolen access token still works, in exchange for not hitting the database on every request.

How a JWT works

header.payload.signature, base64url-encoded. The signature proves the token wasn't modified and was issued by you. The payload is encoded, not encrypted — anyone can read it. Never put secrets in it; keep it small (user id, role, expiry).

const access  = jwt.sign({ sub: user.id, role: user.role },
                  process.env.JWT_SECRET, { expiresIn: "15m" });
const refresh = jwt.sign({ sub: user.id, jti: randomUUID() },
                  process.env.REFRESH_SECRET, { expiresIn: "7d" });

// verify in middleware
const payload = jwt.verify(token, process.env.JWT_SECRET);
req.user = { id: payload.sub, role: payload.role };

The access + refresh token pattern

• Access token — short-lived (5–15 min), sent with each request. If stolen, it expires quickly.
• Refresh token — long-lived, stored in an HttpOnly; Secure; SameSite cookie, kept in the DB so it can be revoked.
• Rotation: each refresh issues a new refresh token and invalidates the old one; reuse of an old token means theft → revoke the whole family.
• Logout = delete the refresh token server-side (and clear the cookie).

OAuth 2.0 / OpenID Connect

• OAuth 2.0 = delegated authorization ("let this app read my Drive"). OIDC adds an ID token for authentication ("Login with Google").
• Authorization Code flow with PKCE is the modern standard: redirect to the provider → user consents → you receive a short-lived code → your server exchanges it (with the client secret) for tokens. The secret never touches the browser.
• Never use the implicit flow anymore. Always validate the state parameter (CSRF) and verify the ID token signature and issuer.
• In practice use a library — Auth.js/NextAuth, Passport, Clerk, Better Auth, Keycloak. Rolling your own is where breaches come from.

24. Attacks & Defences

Why parameterised queries actually work

It's worth understanding the mechanism rather than memorising "use an ORM". String concatenation fails because the data becomes part of the query text, so the parser cannot tell code from input:

// The input "'; DROP TABLE users; --" turns into:
`SELECT * FROM users WHERE email = ''; DROP TABLE users; --'`
//                                    ↑ now a second statement

A parameterised query sends the SQL and the values over separate channels. The database parses and plans the statement first, with $1 as a placeholder, and only then binds the value. The value can never be reinterpreted as SQL, whatever it contains.

db.query("SELECT * FROM users WHERE email = $1", [email]);   // always safe

The same principle explains the other injection defences: keep data and code in separate channels. That's why textContent beats innerHTML, and why you pass arguments as an array to execFile rather than building a shell string.

Practical hardening checklist

• helmet() for security headers (CSP, HSTS, X-Content-Type-Options, frame options).
• HTTPS everywhere; redirect HTTP; enable HSTS.
• Strict CORS whitelist; never origin: "*" with credentials.
• Global rate limiting + stricter limits on auth endpoints.
• Validate everything at the boundary; limit body size.
• Least-privilege database user; separate credentials per environment.
• Never log passwords, tokens, card numbers or full request bodies.
• Generic error messages outward; detailed logs inward.
• Keep dependencies and the Node runtime patched.
• Encrypt in transit and at rest; hash what you never need to read back (passwords), encrypt what you do (PII).

25. Architecture Patterns

Ideas worth knowing

• Layered architecture — controller → service → repository (covered in §11).
• Hexagonal / ports & adapters — the domain core knows nothing about HTTP or the DB; adapters plug in around it.
• Event-driven — services publish events ("OrderPlaced") and others react. Loose coupling, eventual consistency.
• CQRS — separate the write model from optimized read models.
• Saga — a long business transaction across services implemented as steps with compensating actions (no distributed rollback).
• API gateway — single entry point doing auth, routing, rate limiting for many services.
• Circuit breaker — stop calling a failing dependency for a while instead of piling up timeouts.
• Idempotency + retries with exponential backoff and jitter — the baseline of reliable distributed systems.

26. Scaling & Performance

Why "stateless" is the whole trick

If instance 2 stores your session in a local variable, your next request lands on instance 3 and you're logged out. The same applies to uploaded files on local disk, in-memory caches, and setInterval timers — each instance would run its own copy of the job. Move every piece of state out of the process and you can add or kill instances freely, which is what makes autoscaling and zero-downtime deploys possible.

Vertical vs horizontal

• Vertical — a bigger machine. Simple, instant, but has a ceiling and a single point of failure.
• Horizontal — more machines behind a load balancer. Nearly unlimited, but demands stateless servers.

Stateless means any instance can serve any request: sessions in Redis, uploads in S3, no in-memory caches that matter, no local file writes. If you use sticky sessions to work around state, you've limited your scaling.

Scaling the database (usually the real bottleneck)

• Indexes and query tuning first — cheapest win by far.
• Connection pooling — the DB has a hard connection limit.
• Read replicas — send reads to replicas, writes to the primary (mind replication lag).
• Partitioning — split a huge table by range (dates) or list.
• Sharding — split data across databases by a shard key. Powerful but painful: cross-shard queries and rebalancing. Last resort.

Performance checklist for an API

• Measure first: p50/p95/p99 latency, not averages. The p99 is what angry users feel.
• Cache what's hot and expensive; compress responses (gzip/brotli).
• Paginate everything that can grow; never return unbounded lists.
• Fetch in parallel; eliminate N+1 queries.
• Move slow work (emails, PDFs, images, webhooks) to a background queue and return 202.
• Use a CDN for static assets and public cacheable responses.
• Set timeouts on every outbound call — a hung dependency shouldn't hang you.

27. Queues & Background Jobs

Why a queue instead of just doing the work

A user signs up. You need to write the row, send a welcome email, generate an avatar thumbnail, and sync to the CRM. Done inline, the response takes 4 seconds, and if the email provider is down the whole signup fails — even though the account was created fine.

With a queue, the request writes the row, pushes four jobs, and returns in 50ms. Workers process the jobs separately. If the email provider is down, that job retries later without affecting anything else.

What you must handle

• Retries with backoff — transient failures are normal; retry 3–5 times with growing delays.
• Idempotent workers — a job may run twice (the worker crashed after sending but before acknowledging). Sending one welcome email twice is survivable; charging a card twice is not.
• Dead letter queue — after N failures, park the job somewhere you can inspect instead of retrying forever.
• Visibility — queue depth is a key metric. Growing depth means workers can't keep up.

Anything that is slow, retryable, or not needed for the response belongs in a background job: emails, image/video processing, report generation, third-party syncs, webhook delivery, bulk imports.

• Run workers as separate processes so heavy jobs never block the API event loop.
• Make jobs idempotent — at-least-once delivery means a job can run twice.
• Configure retries with exponential backoff and a dead-letter queue for permanent failures.
• Tools: BullMQ (Redis, great for Node), RabbitMQ (routing, classic broker), Kafka (high-throughput event log, replayable), SQS (managed).
• Scheduled jobs: cron for periodic work. With multiple instances, use a lock or a queue's repeatable jobs so it doesn't run N times.
• Outbox pattern: write the DB row and the "to-publish" event in the same transaction, then a relay publishes it — prevents "saved but never notified" bugs.

28. Testing

• The test pyramid: many fast unit tests, fewer integration tests, a handful of E2E tests. For backends, integration tests give the best confidence per minute of effort.
• Use a separate test database, reset between tests (transactions rolled back, or truncate).
• Mock external services (payments, email) — tests must be deterministic and offline.
• Follow Arrange → Act → Assert; test behaviour and edge cases (empty, unauthorized, duplicate, boundary values), not implementation details.
• Coverage is a smoke detector, not a goal. 100% coverage with weak assertions proves nothing.

it("rejects duplicate email with 409", async () => {
  await createUser({ email: "a@x.com" });
  const res = await request(app).post("/api/v1/users")
    .send({ email: "a@x.com", password: "secret123" });
  expect(res.status).toBe(409);
});

29. Deployment & DevOps

Environments and config

• Keep development / staging / production as similar as possible.
• All configuration comes from environment variables; validate them at startup (parse with Zod and crash early if something's missing).
• Secrets live in a secret manager or the platform's env store — never in git.

Containers

# multi-stage build → small, fast, secure image
FROM node:22-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:22-alpine
WORKDIR /app
ENV NODE_ENV=production
COPY package*.json ./
RUN npm ci --omit=dev
COPY --from=build /app/dist ./dist
USER node                       # don't run as root
EXPOSE 3000
CMD ["node", "dist/server.js"]

Docker solves "works on my machine" by shipping the app with its runtime. docker-compose runs app + Postgres + Redis locally with one command. Orchestrators (Kubernetes, ECS) then handle scaling, restarts and rolling updates.

CI/CD

Production essentials

• Health endpoints: /healthz (process alive) and /readyz (DB reachable) for the load balancer.
• Graceful shutdown on SIGTERM: stop accepting new connections, finish in-flight requests, close DB/queue connections, then exit.
• Process manager (PM2) or the orchestrator restarts crashed processes; run one process per core.
• Backups — automated, offsite, and restore-tested. An untested backup is not a backup.
• Hosting: Railway/Render/Fly.io (simple), AWS/GCP (full control), Vercel/Lambda (serverless — mind cold starts, execution limits and DB connection limits).

30. Observability

Logs, metrics and traces — why you need all three

The workflow: a metric alerts you that p95 latency tripled. A trace shows the time is spent in the payment service. Logs from that service at that timestamp show the actual error. Any one of the three alone leaves you guessing.

Structured logging

// ✗ unsearchable
console.log("User " + id + " failed to pay");

// ✓ queryable — filter by userId, level, requestId across millions of lines
logger.error({ userId: id, orderId, requestId: req.id, err }, "payment failed");

Attach a request ID to every log line for a request (generate one in middleware, pass it down). Without it, concurrent requests interleave in the log and you cannot tell which lines belong together.

The three pillars answer three different questions: logs (what happened), metrics (how much/how often), traces (where the time went).

• Structured logging (JSON via pino/winston) so logs are searchable. Include a request/correlation ID on every line and pass it to downstream services.
• Levels: error (needs attention), warn (suspicious), info (business events), debug (development only). Never log secrets or personal data.
• Metrics — the four golden signals: latency, traffic, errors, saturation. Also track event-loop lag, memory, DB pool usage and queue depth.
• Distributed tracing (OpenTelemetry) shows a single request's path across services and where the milliseconds went.
• Error tracking (Sentry) groups exceptions with stack traces and release info.
• Alerting should be on symptoms users feel (error rate, p99 latency, queue backlog), not on every CPU spike. Alert fatigue is a real outage cause.
• SLI / SLO: the indicator you measure vs the target you promise (e.g. 99.9% of requests under 300 ms). Blameless postmortems turn incidents into fixes.

Interview Q&A — rapid fire

Node & JavaScript

• Why is Node fast despite one thread? Non-blocking I/O — waiting is delegated to the OS/thread pool, so the thread is never idle-blocked.
• Event loop phases? timers → pending → poll → check → close, with nextTick and promise microtasks drained between each.
• setTimeout(0) vs setImmediate? Different phases; inside an I/O callback setImmediate always runs first.
• How do you handle CPU-heavy work? Worker threads, a queue + separate worker service, or a different runtime for that piece.
• Streams? Chunked processing with backpressure — constant memory regardless of file size.
• Cluster vs worker threads? Separate processes (no shared memory, for scaling I/O across cores) vs threads in one process (for CPU work).

HTTP & APIs

• 401 vs 403? Not authenticated vs authenticated but not allowed.
• PUT vs PATCH? Full replacement vs partial update; PUT is idempotent.
• Idempotency, why care? Clients retry; without it, retries create duplicate charges/orders.
• What is CORS? A browser rule; the server opts in via headers. It's not authentication.
• Middleware? A function in the request pipeline with req, res, next.
• How do you version an API? URL prefix, never break old versions, deprecate with notice.

Databases

• SQL vs NoSQL? Relations + constraints + ACID vs flexible documents + easy horizontal scale; choose by access patterns.
• What does an index cost? Faster reads, slower writes, more storage; leftmost-prefix rule for composites.
• ACID? Isolation levels? Atomic/Consistent/Isolated/Durable; Read Committed → Repeatable Read → Serializable trades concurrency for correctness.
• How do you prevent overselling stock? Atomic conditional update, SELECT ... FOR UPDATE, or optimistic version checks.
• N+1 problem? One query per item; fix with joins/include/populate or batching.
• Embed or reference in Mongo? Embed bounded data read together; reference shared or unbounded data.

Security & scale

• Why bcrypt not SHA-256? Salted and deliberately slow — brute force becomes impractical.
• JWT vs session? Stateless and scalable vs revocable and simple; JWT needs refresh-token rotation.
• Where do you store a JWT? HttpOnly cookie with SameSite + CSRF defence, or memory; localStorage is XSS-exposed.
• Prevent SQL injection? Parameterized queries — never concatenate input.
• How would you scale this API 10×? Measure → indexes/queries → cache → stateless + horizontal scale behind a LB → read replicas → queues for heavy work → CDN.
• Monolith or microservices? Modular monolith first; split when a component's scaling or ownership genuinely diverges.

Learning Path

1. Fundamentals — HTTP, status codes, JSON, how the browser talks to a server.
2. JavaScript async — promises, async/await, error handling. Don't skip this; everything else depends on it.
3. Node basics — modules, fs, the http module (build one server without Express, once).
4. Express — routing, middleware, a proper folder structure.
5. A database — pick PostgreSQL and learn real SQL, or MongoDB with Mongoose. Learn indexes early.
6. Auth — hashing, sessions vs JWT, refresh tokens, ownership checks.
7. Project 1 — a complete CRUD REST API with auth, validation, pagination, errors, and tests.
8. Level up — caching with Redis, background jobs with BullMQ, file uploads to S3.
9. Project 2 — something with real load: a booking system (concurrency!), or a job board with search and queues.
10. Production skills — Docker, CI/CD, logging and monitoring, graceful shutdown, deploy it publicly.
11. Then — TypeScript everywhere, system design basics, and one advanced area (WebSockets, event-driven, or Kubernetes).