Today was a rough morning. Our FINAL MIGRATION was scheduled for this weekend, and it was my task to get up a 8am this morning to run it. Yes that's right, 8am - on a Sunday - the day after daylight-savings-time began! Ouch. Especially since I was up late last night reading Strangers In Paradise.
So at around 7:45am I got my first page, saying that the database was ready for the migration. I heaved myself painfully out of my nice warm bed and went down to the livingroom. I linger in front of the coffee pot, consider making some, but then decide that I'd rather get this over with quickly and attempt to go back to sleep after.
I get all signed into my laptop, and started the various tools and applications I needed to run and monitor the process. I run a few queries, write down the preliminary numbers. I verify that everything is good to go.
My first task is to run the delete process. "./b.sh run:delete:background", I believe was the command. I typed everything except for that initial ".", and pressed Enter. It complained. I immediately noticed the missing dot, but instead of up-arrowing, moving my cursor over, and adding the dot? I decided to retype the whole command.
But this time my eyes happened to glance lower down in the task list, and my fingers cheerfully type out "./b.sh run:migration:background". That's right: the next step. The one that can only run after the delete has finished.
Shoulda made that pot of coffee after all, eh?
Of course, this time I don't forget the initial dot, so the command happily starts running in the background. It isn't until I start monitoring the log file that I think, "Wait a minute, that looks like a migration log, not a delete log." My heart stops for awhile, and as I review the history of what I typed it starts beating a lot quicker and louder.
Ohhhh...... CRAP. And this was supposed to be the quickest and easiest of the lot of them. Fortunately I realize that this error will make the migration bomb, but re-running the delete process afterward should clean up my mistake.
When the process crashes, I cross my fingers and type the delete again -- correctly this time. It takes over an hour to run, but it completes successfully. Crossing my fingers yet again, I re-run the migration. (It's hard to type that way!)
At 10am, I check the status of the migration. Running smoothly, predicted to finish just after 11. At 10:45, I check again -- but my laptop won't connect. WON'T CONNECT?!
More panic ensues. I start testing things, rebooting things, trying again with my left hand this time.... nothing. I have to pack up my bags and drive to the office.
The office is locked up like a bank vault this time on a sunday morning. I use two swipe cards and one alarm disable code, and finally I'm in. Just as I'm plugging in my laptop and booting everything up at 11:33, one of the guys in Cleveland calls. "What's going on? Is the migration done yet?" They're waiting for me to notify them that they can continue with the next steps.
I apologize, explain my bad luck, and reassure him that I'll page the instant I've verified that it ran successfully.
Things proceed fairly smoothly at that point, thank goodness! I confess my errors and general misadventures to the QA team and ask them to double-check some of the double-migrated claims. Fortunately, everything looks good to them. whew!
The only good part of the adventure was the coffee. After that much struggle, I decided that I NEEDED coffee before I broke anything else. I swung by the shop across the street from my office, and placed my order. The cashier asked me, "Would you like a free espresso shot in honor of daylight savings time?"
I stared at her with glazed eyes. "Why yes," I said slowly. "That's probably an excellent idea." If only I'd started my day there, maybe it would have been catastrophe-free!
Well, that's unlikely. But at least maybe I'd have been perkier for the catastrophes. At least we're done with our migrations at work... at least for quite awhile.