Respond to a production incident

The email lands at 3:04am. Subject: “Your site is DOWN.” If you followed this path, an uptime monitor sent it, which means a robot found out before your users did. That part worked. The part nobody warns you about is the cold-stomach feeling of knowing the thing you shipped is broken in public, right now, while you stand in the kitchen deciding whether you’re actually awake.

Here’s the line to keep within reach: everyone breaks production. Every senior dev you’ve ever admired has taken a live site down, most of them more than once. What separates them from a panicking beginner at 3am is mostly order of operations. They reach for the rollback before they reach for the debugger.

This is the incident playbook for a team of one. Four steps, in sequence. The sequence is the whole trick, because panic’s favorite move is skipping straight to the debugging.

Step zero: confirm it is actually broken

“Down” is a claim, and claims get checked. Before you touch anything, figure out who the site is down for: you, one user, or everyone. Each answer points somewhere different.

Run through these in about ninety seconds.

Open the site in an incognito window. Private windows skip your cookies, your logged-in session, and most extensions. An expired auth token or a misbehaving extension can make a site look dead to exactly one person on the planet: you.

Check your phone with WiFi off. Cellular data takes your home network, your router, and your router’s DNS cache out of the picture. If the site loads fine on cellular, the outage lives somewhere in your house.

Check the uptime monitor. If you stumbled onto the breakage yourself and the monitor hasn’t fired, lean toward “it’s just me.” If the monitor is what woke you, it has been failing its checks from somewhere other than your laptop, and it’s probably real.

Check the platform status page. Vercel, Netlify, Cloudflare, Railway, every host runs a public status page. If your host is mid-incident, stop here. Nothing you roll back or redeploy fixes their outage. The move is to wait, which feels terrible and is correct.

One more pattern: a single user says it’s broken while you and the monitor both see a healthy site. Hold that thought for the DNS section below. Patchy, per-person breakage has a usual suspect.

Down for everyone? Step one.

Step one: stop the bleeding

Every instinct says find the bug. Ignore that instinct. While you read stack traces, the site stays down, and every minute of downtime is another visitor staring at an error page. The first job is making the site work again, and you almost never need to understand the problem to do that.

You need to undo it, and you usually can, because the most likely cause of a broken site is the last deploy, and platforms keep your previous deploys precisely for this moment. Every build is stored, finished, immutable. Making an older one live again takes about a minute, because nothing has to rebuild.

• Vercel: Deployments tab, pick the last good production deploy, Instant Rollback.
• Netlify: Deploys, click an older deploy, Publish deploy.
• Cloudflare Pages: open a previous deployment’s menu, Rollback.

Button names drift over the years. “A list of old deploys plus a button to make one live again” is the constant.

If your host only builds whatever sits on main, or you want the repo history to tell the truth, do it in git:

f4e2a1b switch api client to v3
9c81d3e add pricing page
2b7f6c0 fix mobile nav

Why git revert and never git reset --hard plus a force push: revert is an ordinary commit that moves history forward. Reset rewrites history, and rewriting history while sleep-deprived is how one outage becomes two.

Site back up? Good. The postmortem can wait until coffee. So can the root cause. Rollback first, diagnose later is the entire discipline.

Step two: look at what changed

Now, with the site up and your heart rate descending, find the cause. Start from one reliable fact: outages overwhelmingly follow a change you made. Servers almost never rot spontaneously overnight. Something was different yesterday, and the suspects rank roughly like this:

1. A deploy. Check timestamps first: when did the monitor fire, and what deployed shortly before? Most of the time the timeline alone convicts.
2. An env var edit. The sneaky one. Environment variables live in the host’s dashboard, so changing one leaves no commit and shows up in no diff. You edited it at 9pm, the next deploy picked it up, and git swears nothing happened.
3. A dependency bump. If you updated packages, the evidence is in your lockfile diff. A library’s minor version can change behavior your code quietly depended on, without a single line of yours changing.
4. A DNS change. Strange enough to earn its own section below.
5. The change you didn’t make: time. If you genuinely touched nothing, look for something that expired. API keys, trials, domain renewals, a free-tier database that pauses itself after inactivity. “Nothing changed” usually means “something lapsed.”

The mechanics of step two are unglamorous: read the deploy history with timestamps, then git log since yesterday, then the dashboard’s activity log if it has one, until something lines up with the minute the monitor fired.

It is always DNS

The meme is earned, so here’s the honest version. DNS (the Domain Name System, the mapping from your domain name to your server’s address) fails in a way no other part of the stack does: gradually, and differently for every user.

The reason is caching. When someone visits your site, the answer to “what address is this domain?” gets cached at several layers: their operating system, their router, their internet provider’s resolver, public resolvers like Google’s 8.8.8.8. Each cache keeps the answer for the record’s TTL (time to live), a duration you set on the record itself, commonly anywhere from a minute to a day.

So when you change a DNS record, there is no single moment when the change happens. Each cache expires on its own schedule. One user gets the new answer immediately, another gets the stale one for hours. From where you sit, the site is up, down, and up again depending on who you ask, which feels like a haunting and is actually just caches draining at different rates.

The tell is patchy, per-user breakage shortly after you touched DNS: changed hosts, added a record, moved registrars, anything in that family. To see what the world sees:

# What does my resolver think the domain points to?
nslookup yourdomain.com

# Ask Google's public resolver the same question
nslookup yourdomain.com 8.8.8.8

If those two answers disagree, you’re watching propagation mid-flight. There’s nothing to fix and no way to rush it; the stale caches expire on their own. The preventive trick for next time is to lower the record’s TTL a day before any planned DNS change, so the world’s caches let go of the old answer quickly. If records and TTLs still feel like folklore, Custom domains and DNS walks through the whole system.

Step three: three sentences before the adrenaline fades

The site is up, the cause is found, and you want to go back to bed. First, write three sentences. What broke. Why. What would have caught it sooner.

Keep them in one file in the repo, incidents.md, newest entry at the top:

## 2026-06-11: site down 41 minutes
What broke: every API route returned 500 after the evening deploy.
Why: DATABASE_URL was set for preview deploys but not for production.
Catch it sooner: a smoke test in CI that hits /api/health after deploy.

This is the team-of-one postmortem, and it compounds. The third sentence is the valuable one: across a few incidents, those lines turn into your actual hardening backlog. A health endpoint. An alert on the right URL. A deploy checklist with “check prod env vars” on it. Your 3am memory will keep none of this. The file keeps all of it.

Write it blameless, even though the only name available is yours. “The deploy was missing an env var” is a sentence you can automate a check for. “I was careless” isn’t. The point of a postmortem is to change the system so the same mistake bounces off next time, and self-flagellation changes nothing.

Everyone breaks production

Ask any senior developer for their outage story. They have one, and they tell it with suspicious fondness: the dropped table, the wildcard DNS record, the deploy at 4:55pm on a Friday. Breaking production is something everyone who ships eventually does. Seniors are just faster at the rollback, because they’ve run this exact playbook enough times that the panic has nowhere to land.

That’s also the quiet point of everything in this Production path. The uptime monitor means you find out before your users do. Error tracking means the stack trace is waiting for you instead of lost in a stranger’s browser. Logs mean you can see what the server was doing at 3:04am without having been awake for it. And the rollback means the site comes back up before you’ve read any of them.

Deploying a project once makes it live. This loop, monitor, roll back, read the error, write three sentences, is what keeps it live. Shipped projects stay shipped because someone runs the boring procedure in the correct order at the worst possible hour. As of tonight that someone is you, with a status page bookmarked and an empty incidents.md waiting for its first entry.