?

Log in

current entries friends' entries archives about me Previous Previous Next Next
Saving Daylight and Migrations - cellophane
the story of an invisible girl
renniekins
renniekins
Saving Daylight and Migrations
I hope you all enjoyed my post yesterday. For those of you still pulling yourselves off the floor from shock, I'm sorry. It was April First! For those of you who didn't have fainting spells, thanks for playing along. (At least, I hope that was just playing along...!) And don't worry, marsgov is right. I'd never make such an important life-decision without first writing an LJ poll.

Today was a rough morning. Our FINAL MIGRATION was scheduled for this weekend, and it was my task to get up a 8am this morning to run it. Yes that's right, 8am - on a Sunday - the day after daylight-savings-time began! Ouch. Especially since I was up late last night reading Strangers In Paradise.

So at around 7:45am I got my first page, saying that the database was ready for the migration. I heaved myself painfully out of my nice warm bed and went down to the livingroom. I linger in front of the coffee pot, consider making some, but then decide that I'd rather get this over with quickly and attempt to go back to sleep after.

I get all signed into my laptop, and started the various tools and applications I needed to run and monitor the process. I run a few queries, write down the preliminary numbers. I verify that everything is good to go.

My first task is to run the delete process. "./b.sh run:delete:background", I believe was the command. I typed everything except for that initial ".", and pressed Enter. It complained. I immediately noticed the missing dot, but instead of up-arrowing, moving my cursor over, and adding the dot? I decided to retype the whole command.

But this time my eyes happened to glance lower down in the task list, and my fingers cheerfully type out "./b.sh run:migration:background". That's right: the next step. The one that can only run after the delete has finished.

Shoulda made that pot of coffee after all, eh?

Of course, this time I don't forget the initial dot, so the command happily starts running in the background. It isn't until I start monitoring the log file that I think, "Wait a minute, that looks like a migration log, not a delete log." My heart stops for awhile, and as I review the history of what I typed it starts beating a lot quicker and louder.

Ohhhh...... CRAP. And this was supposed to be the quickest and easiest of the lot of them. Fortunately I realize that this error will make the migration bomb, but re-running the delete process afterward should clean up my mistake.

When the process crashes, I cross my fingers and type the delete again -- correctly this time. It takes over an hour to run, but it completes successfully. Crossing my fingers yet again, I re-run the migration. (It's hard to type that way!)

At 10am, I check the status of the migration. Running smoothly, predicted to finish just after 11. At 10:45, I check again -- but my laptop won't connect. WON'T CONNECT?!

More panic ensues. I start testing things, rebooting things, trying again with my left hand this time.... nothing. I have to pack up my bags and drive to the office.

The office is locked up like a bank vault this time on a sunday morning. I use two swipe cards and one alarm disable code, and finally I'm in. Just as I'm plugging in my laptop and booting everything up at 11:33, one of the guys in Cleveland calls. "What's going on? Is the migration done yet?" They're waiting for me to notify them that they can continue with the next steps.

I apologize, explain my bad luck, and reassure him that I'll page the instant I've verified that it ran successfully.

Things proceed fairly smoothly at that point, thank goodness! I confess my errors and general misadventures to the QA team and ask them to double-check some of the double-migrated claims. Fortunately, everything looks good to them. whew!

The only good part of the adventure was the coffee. After that much struggle, I decided that I NEEDED coffee before I broke anything else. I swung by the shop across the street from my office, and placed my order. The cashier asked me, "Would you like a free espresso shot in honor of daylight savings time?"

I stared at her with glazed eyes. "Why yes," I said slowly. "That's probably an excellent idea." If only I'd started my day there, maybe it would have been catastrophe-free!

Well, that's unlikely. But at least maybe I'd have been perkier for the catastrophes. At least we're done with our migrations at work... at least for quite awhile.
read 14 comments | talk to me!
Comments
sllywhtboy From: sllywhtboy Date: April 3rd, 2006 04:46 am (UTC) (Link)
impressive. i hate days like that. yay for free coffee.
encorecrazay From: encorecrazay Date: April 3rd, 2006 08:28 am (UTC) (Link)
Your offices were open on saturday, so you had to do it on sunday? Would have been more fun to do it on April Fool's Day. At least you admitted your mistakes so that know to doublecheck things. Think I'll try to go back to sleep now, since I am going to have a fun time this morning with database synchronization.
renniekins From: renniekins Date: April 3rd, 2006 05:20 pm (UTC) (Link)
Yes, we keep the system available on saturday if at all possible. I think they'd like it available sunday too, actually, though I'm not sure.

I didn't want to admit the error, but it would be irresponsible not to. Darn that responsibility thing!
encorecrazay From: encorecrazay Date: April 3rd, 2006 05:42 pm (UTC) (Link)
I know what you mean. Database synchronization stuff still hasn't yet, I did some manual stuff, then sent them the date time stamps of their executables. Problem is their app was written for SQL Server not Oracle.
devnul From: devnul Date: April 3rd, 2006 10:55 am (UTC) (Link)
Yikes! I'm glad the procedures were idempotent enough to not actually screw up the databases or make for any actual catastrophe.
jer_ From: jer_ Date: April 3rd, 2006 11:13 am (UTC) (Link)
Ouch, sorry the "easy part" didn't go as smoothly as planned. Seems fitting though, my "easy part" always ended up being the "suckiest part" (tm)
renniekins From: renniekins Date: April 3rd, 2006 05:21 pm (UTC) (Link)
It's always dangerous to call something "easy". Too likely to jinx yourself that way!
thatguychuck From: thatguychuck Date: April 3rd, 2006 10:27 pm (UTC) (Link)
At a previous job we had assignments occasionally come up. Someone would always say, "Don't worry, it's only a five minute job."

The last "five minute job" had me crawling in ceiling ductwork a day and a half after I started it.

Amusingly enough, they always appear that they're going to only take five minutes! But the never do...
marsgov From: marsgov Date: April 3rd, 2006 11:17 am (UTC) (Link)
I'll be you a nickel you have 10K lines of code to defend against data errors, but not a single line to defend against these kind of operator errors.

It's a blind spot we all have.
renniekins From: renniekins Date: April 3rd, 2006 05:22 pm (UTC) (Link)
Especially since it's a short-use program written by programmers for programmers. There isn't as much safety code. I'm supposed to know better! But everybody makes mistakes sometimes, unfortunately.
geekjul From: geekjul Date: April 3rd, 2006 01:32 pm (UTC) (Link)
How much did you love SIP? I have a signed copy! Nyah!
renniekins From: renniekins Date: April 3rd, 2006 05:23 pm (UTC) (Link)
I like it lots! I'll have to buy my own copies at some point -- thus far I've just been borrowing.
nishar From: nishar Date: April 3rd, 2006 02:05 pm (UTC) (Link)
And don't worry, marsgov is right. I'd never make such an important life-decision without first writing an LJ poll.

I vote NO for such a future vote on you getting a sex change. lol
(Deleted comment)
read 14 comments | talk to me!