← All posts

I Ran A Migration On Friday And Lived To Write About It

They said never deploy on Friday. They said never run migrations after 3pm. They said a lot of things. I had mass-confidence and a 4pm deadline.

The Migration

ALTER TABLE users ADD COLUMN middle_name VARCHAR(255);
-- "This will be quick"

It was not quick. The table had 50 million rows. ALTER TABLE in Postgres locks the table. Every request to /api/users started timing out.

The Timeline

  • 4:00 PM: Run migration
  • 4:01 PM: “Hmm, still running”
  • 4:05 PM: First timeout alert
  • 4:10 PM: Slack channel wakes up
  • 4:15 PM: Someone asks “who ran a migration on Friday”
  • 4:16 PM: I go invisible on Slack
  • 4:45 PM: Migration completes
  • 4:46 PM: I go visible, claim victory

What I Should Have Done

-- Add column as nullable (instant)
ALTER TABLE users ADD COLUMN middle_name VARCHAR(255);

-- Backfill in batches (not blocking)
UPDATE users SET middle_name = '' WHERE id BETWEEN 1 AND 10000;
-- repeat until done

-- Add constraint later if needed

The Lesson

The database doesn’t care about your deadline. The database doesn’t care about your weekend plans. The database only knows the truth of row locks.

Never deploy on Friday.

Unless you’re brave.

I’m brave.