?

Log in

current entries friends' entries archives about me Previous Previous Next Next
Interesting Release (as in Living in Interesting Times) - cellophane
the story of an invisible girl
renniekins
renniekins
Interesting Release (as in Living in Interesting Times)
Rough day at work today. We had a deployment going in at 1:30pm. It was our first official release of this brand new product I've been working on almost a year.

Then something came up, and due to dependencies in the FT code, it was pushed back 'til 10pm. Then, during testing, we discovered a critical error. Two, really, due to the same problem in the FT code. So we rolled back the FT code and cut a new release. Then we realized we didn't need to wait until 10pm after all, so we rescheduled for 4pm. The release to prod ran about 4:30, and at 5pm we discovered it was seriously borked. Production was not accessible.

Cue a certain amount of panicking, as people tried to figure out the error. Turns out that one piece of the FT code that we rolled back? Didn't get rolled back. Now nobody can log in. Okay, no problem, roll back the release. When this fails to roll back gracefully, cue more serious panic.

We start to delve into code and db scripts, and discuss options. Environments were inconsistent, which is why we didn't discover this earlier. All the higher-ups in the Operations and QA departments appear out of the walls. DBAs are fetched. I drag a whiteboard over and enumerate our options. We discuss risks, timeframes, and rewards. Options 2 and 3 look best, and after some agonized examination of scripts and some test runs, we agree on Option 2.

Option 2 is implemented in around 30 minutes worth of time, and it goes surprisingly smoothly. Everything is up and running by 7pm, we all wipe our foreheads in relief, and head home....

Tags:

read 9 comments | talk to me!
Comments
nishar From: nishar Date: September 21st, 2011 08:57 am (UTC) (Link)
I am glad y'all could fix this issue and not have to eat, sleep and go crazy at the office until the issue was fixed.
renniekins From: renniekins Date: September 23rd, 2011 02:36 am (UTC) (Link)
Me too... thanks!
pstscrpt From: pstscrpt Date: September 21st, 2011 02:41 pm (UTC) (Link)
FT?
renniekins From: renniekins Date: September 23rd, 2011 02:38 am (UTC) (Link)
I'm abbreviating a product name because it's unimportant to the story, and possibly confidential. (:
pstscrpt From: pstscrpt Date: September 23rd, 2011 02:51 am (UTC) (Link)
Ah, I was thinking maybe Functional Testing.
jeffreyab From: jeffreyab Date: September 21st, 2011 04:02 pm (UTC) (Link)
"I drag a whiteboard over and enumerate our options." nice to see you are taking the lead. Are you always first to the white board?
renniekins From: renniekins Date: September 23rd, 2011 02:39 am (UTC) (Link)
Nope, not always. They are valuable tools though!
From: writerwench Date: September 21st, 2011 05:16 pm (UTC) (Link)
I think this post illustrates exactly why I never went into anything like IT or machinery of any kind.

I'd have been a wibbling blob sitting under someone else's desk.
renniekins From: renniekins Date: September 23rd, 2011 02:39 am (UTC) (Link)
The team handled the stress remarkably well. I was proud of us. (:
read 9 comments | talk to me!