alpaca princess (renniekins) wrote,
alpaca princess
renniekins

Saving Daylight and Migrations

I hope you all enjoyed my post yesterday. For those of you still pulling yourselves off the floor from shock, I'm sorry. It was April First! For those of you who didn't have fainting spells, thanks for playing along. (At least, I hope that was just playing along...!) And don't worry, marsgov is right. I'd never make such an important life-decision without first writing an LJ poll.

Today was a rough morning. Our FINAL MIGRATION was scheduled for this weekend, and it was my task to get up a 8am this morning to run it. Yes that's right, 8am - on a Sunday - the day after daylight-savings-time began! Ouch. Especially since I was up late last night reading Strangers In Paradise.

So at around 7:45am I got my first page, saying that the database was ready for the migration. I heaved myself painfully out of my nice warm bed and went down to the livingroom. I linger in front of the coffee pot, consider making some, but then decide that I'd rather get this over with quickly and attempt to go back to sleep after.

I get all signed into my laptop, and started the various tools and applications I needed to run and monitor the process. I run a few queries, write down the preliminary numbers. I verify that everything is good to go.

My first task is to run the delete process. "./b.sh run:delete:background", I believe was the command. I typed everything except for that initial ".", and pressed Enter. It complained. I immediately noticed the missing dot, but instead of up-arrowing, moving my cursor over, and adding the dot? I decided to retype the whole command.

But this time my eyes happened to glance lower down in the task list, and my fingers cheerfully type out "./b.sh run:migration:background". That's right: the next step. The one that can only run after the delete has finished.

Shoulda made that pot of coffee after all, eh?

Of course, this time I don't forget the initial dot, so the command happily starts running in the background. It isn't until I start monitoring the log file that I think, "Wait a minute, that looks like a migration log, not a delete log." My heart stops for awhile, and as I review the history of what I typed it starts beating a lot quicker and louder.

Ohhhh...... CRAP. And this was supposed to be the quickest and easiest of the lot of them. Fortunately I realize that this error will make the migration bomb, but re-running the delete process afterward should clean up my mistake.

When the process crashes, I cross my fingers and type the delete again -- correctly this time. It takes over an hour to run, but it completes successfully. Crossing my fingers yet again, I re-run the migration. (It's hard to type that way!)

At 10am, I check the status of the migration. Running smoothly, predicted to finish just after 11. At 10:45, I check again -- but my laptop won't connect. WON'T CONNECT?!

More panic ensues. I start testing things, rebooting things, trying again with my left hand this time.... nothing. I have to pack up my bags and drive to the office.

The office is locked up like a bank vault this time on a sunday morning. I use two swipe cards and one alarm disable code, and finally I'm in. Just as I'm plugging in my laptop and booting everything up at 11:33, one of the guys in Cleveland calls. "What's going on? Is the migration done yet?" They're waiting for me to notify them that they can continue with the next steps.

I apologize, explain my bad luck, and reassure him that I'll page the instant I've verified that it ran successfully.

Things proceed fairly smoothly at that point, thank goodness! I confess my errors and general misadventures to the QA team and ask them to double-check some of the double-migrated claims. Fortunately, everything looks good to them. whew!

The only good part of the adventure was the coffee. After that much struggle, I decided that I NEEDED coffee before I broke anything else. I swung by the shop across the street from my office, and placed my order. The cashier asked me, "Would you like a free espresso shot in honor of daylight savings time?"

I stared at her with glazed eyes. "Why yes," I said slowly. "That's probably an excellent idea." If only I'd started my day there, maybe it would have been catastrophe-free!

Well, that's unlikely. But at least maybe I'd have been perkier for the catastrophes. At least we're done with our migrations at work... at least for quite awhile.
Subscribe
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 13 comments