An incident


  • Garbage Person

    A reenactment of a real
    incident. None of us sit within 500 miles of anyone else, so obviously
    this conversation never actually happened, but it's a good summary of
    the email thread.

    Weng: Hey, guys.I just happened to glance at the output from this process. The books are empty.
    Project Manager: Oh, shit. Don't you people have any QA?
    Production Facility: We check 10% of the output! They all looked good to our eyes!
    Weng:
    That means you quality check the print and completely ignore the
    content. You're awesome. By the way, I just checked and this has been
    happening for a full month. nnnnn orders covering mmmmm books.
    Sales: WHY!?!?!?!?? THIS CAN'T HAPPEN. I CAN'T TELL THE CUSTOMER THIS!
    Weng:
    It looks like 1 month ago, the client changed the XML feed format in a
    very subtle and clever way that managed to skate by my import routines
    but produced junk in the lookup tables. I warned you about lack of
    validation.
    Sales: THE CLIENT IS NEVER WRONG!


    <snip days of arguing, meanwhile all production is halted>


    Project Manager: Sales and the customer are of the opinion that we should validate the data.
    Sales: IS IT FIXED YET?
    Weng: According to what rules? Hell, for this sort of data we should be validating with a schema.
    Sales: IS IT FIXED YET?
    Project
    Manager: <sigh> I mentioned that. They said they can't give us
    the rules or a schema because the code was written a very long time ago
    and hasn't changed in years and the documentation is gone.
    Sales: IS IT FIXED YET?
    Weng: That's odd, I thought we agreed that they changed the code last month.
    Sales: IS IT FIXED YET?
    Executive VP of Sales: IS IT FIXED YET? THIS IS REALLY IMPORTANT
    Project Manager: I don't know. I'm not allowed to talk directly to the customer. Apparently my grip on reality is too tight.
    Director of IT: This is a critical issue and should be fixed as the top priority.
    Sales: IS IT FIXED YET?


    <days
    pass while the client rewrites their code to back the change out (note
    lack of their ability to just roll back to the previous version), with
    the sales/production/boss escalation level/echo chamber increasing all
    the time>


    EVP Sales: IS IT FIXED YET?
    SVP SALES: IS IT FIXED YET?
    VP SALES: IS IT FIXED YET?
    DIRECTOR SALES: IS IT FIXED YET?
    REGIONAL MANAGER SALES: IS IT FIXED YET?
    SALES: IS IT FIXED YET?
    SVP of IT: This is a critical issue and is being fixed as the top priority for the resources involved.
    Production: NOTHING IS AVAILABLE TO PRINT TODAY! THE SKY IS FALLING
    Weng: <Surfs the Internets, waiting to be told the client fixed it>
    Project Manager: The client reports the rollback is complete. Can you pull the feed and test it?
    EVP Sales: IS IT FIXED YET?
    SVP SALES: IS IT FIXED YET?
    VP SALES: IS IT FIXED YET?
    DIRECTOR SALES: IS IT FIXED YET?
    REGIONAL MANAGER SALES: IS IT FIXED YET?
    SALES: IS IT FIXED YET?
    Production: NOTHING IS AVAILABLE TO PRINT TODAY! EMERGENCY!
    Chief Sales Officer (that's a thing? How the hell is that even a thing?): This incident is an enormous black eye on the company and needs to be completely cleared by EOD today (it is presently 7:59PM EST)
    Weng: Yup. Looks good. Now that they've rolled it back, can you ask for a schema again? After all, they just rewrote the code
    Project Manager: Alerady did, and they say they can't produce one.
    Boss+1: Can you just turn it back on already?
    Weng: <sigh> Fine. But we needa change request to implement proper validation as soon as possible.
    Boss+1: Agreed.
    Project Manager: Agreed.
    Sales: WHAT DO YOU MEAN? IT'S FIXED!


    <Two days while all the processing gets rerun>


    Weng: There, it's all fixed and processing is caught up. We need to follow up on the validation change ASAP, however.
    Sales: WHAT DO YOU MEAN? TELL ME MORE ABOUT THIS INCIDENT YOU JUST RECOVERED FROM.

     

    Epilogue: Still no sign of any information on validation rules or an XML schema.



  •  It's beautiful!



  • Sales guys never seem to be the sharpest spoons in the blender.



  • @Weng said:

    Apparently my grip on reality is too tight.

    Yep, I've seen that one before. Mostly when talking to HR. I don't even get emails from Sales.



  •  It is NOT FIXED

    we have a Temporary Partial  work around to minimise the impact.

    (A slight rewording of the situatioin to lead the sales team to belive it could colapse again at any moment, which is not too far from the truth anyway)

     



  • It just left me wondering: is it fixed yet?

    Or, more seriously: what kind of idea of "fixed" do these peope have? Does "fixed" really mean "it works now, and tomorrow is another day" for them?



  • I'm interested to hear more about the accountability/cashflow situation. At a guess the customer pays you, but you only get money if you can interpret the file properly (which, in this case, you couldn't), so I'm guessing that either meant you couldn't charge them or you couldn't do something else with that data which somebody else pays you for, or you need the data for auditing.

    There was a question in there somewhere.



  • @Weng said:

    Project Manager: Sales and the customer are of the opinion that we should validate the data.
     

    Weng (to Sales and the customer): I agree. Please appraise me of the timeframes required by the customer to deliver a copy of their schema to prevent this issue recurring.

    @Weng said:

    EVP Sales: IS IT FIXED YET?
    SVP SALES: IS IT FIXED YET?
    VP SALES: IS IT FIXED YET?
    DIRECTOR SALES: IS IT FIXED YET?
    REGIONAL MANAGER SALES: IS IT FIXED YET?
    SALES: IS IT FIXED YET?

    Weng: (forwards these emails to the customer) Salespeople are demanding to know timeframes for rolling back your changes. Could you let them know, please?

    I probably wouldn't have done the last one - I don't like airing dirty washing in public - but I'd certainly push it back to a BRM to contact the customer for timeframes. It was their changes that broke things, they're best placed to fix it.

    @Weng said:

    Weng: Yup. Looks good. Now that they've rolled it back, can you ask for a schema again? After all, they just rewrote the code
    Project Manager: Alerady did, and they say they can't produce one.
    Boss+1: Can you just turn it back on already?

    I'd (a) ask the PM what project management methodology he uses that bypasses QA procedures which Boss+1 seems to be advocating, and (b) if the PM let the client know the potential consequences of that client's actions - or if the PM just spinelessly accepted the client's response and it'll be forgotten about until it recurs.



  • @TGV said:

    Or, more seriously: what kind of idea of "fixed" do these peope have? Does "fixed" really mean "it works now, and tomorrow is another day" for them?
     

    I think ip-guru's response is closer to the truth: the incident can be closed (that's fixed) but the problem remain and need fixing.



  • @TGV said:

    Or, more seriously: what kind of idea of "fixed" do these peope have? Does "fixed" really mean "it works now, and tomorrow is another day" for them?

    Sales droids? Absolutely.



  • Reminds me of a situation that occurred here several years ago.



    Our agency contracted with an outside company to take our data, crunch it up, and spit it back to us. In addition to management reporting, this data would also serve as input into a fairly important and time sensitive program. To capture the data Company provided a Word template filled with a metric crapton of fields for data input. Their programmers initially named the fields like so: primary_adult_data_1, primary_adult_data_2, etc., secondary_adult_data_1, secondary_adult_data_2, etc., child_1_data_1, etc.



    Cool, I can code for that. Huzzah for loops! I wrote a parser to extract the data directly from the form as it was submitted (rather than waiting for the data to be returned by Company). Well, some manager at Company, not directly related to this process, decided she couldn't understand which field capture what data because of the naming convention used. She made the programmers rename most of the fields in the adult section of the form in an arbitrary and completely unnecessary way so she could understand them. Without telling us.



    Management at our agency comes to me and says, "Your parser is broken!" When I discovered what had happened I went back to management and told them my parser was fine. Company changed the input forms. Management says, "Well, rewrite your code." I told them ok but, make Company alert us before hand if they're going to pull this stunt again. After all, WE'RE the customer.



    About a month later we get this 'alert" that Company was about to upload a new form ... tomorrow. I asked for, and received, a copy of the new form. This bitch had made the programmers screw up the naming on about half of the fields in the child section! No more looping. Rudraigh is sad. No ... Rudraigh MAD! Rudraigh smash! I went to our management.



    Me: Make them stop!


    Management: No. We have to choose our battles.


    Me: ...


    Management: We don't want to upset Company.


    Me: ಠ_ಠ


    Management: All you have to do is rewrite your code.


    Me: (rageface.jpg) It's going to take me at least a week!


    Management: So, can you have that ready for test tomorrow?


    Me: (╯°□°)╯︵ ┻━┻



    Epilogue: We ditched Company a few months later. We still use the form and my parser has been running fine for years.


  • Considered Harmful

    I don't see how one field name is more conducive to a loop than another. You should be able to trivially write a function to map a number of indices to a field name; then, when the schema changes (which should be seldom), you update your mapping function.

    If you know in advance something is prone to change (and you learned the first time that it was), you code so you can change it in one place, not many.



  • @rudraigh said:

    Me: It's going to take me at least a week!


    Management: So, can you have that ready for test tomorrow?

    That particular response is one of my hot buttons. I want to scream back "excuse me, where you actually listening to what the hell I said to you barely three seconds ago?!".



  •  Ah, sales :) More than one time they came to us askig: "Hello, we just SOLD such and such feature to client, do you think it is even possible to program something like that?" 



  • Too bad you can't sue your own salespeople for breach of contract...



  • @TGV said:

    It just left me wondering: is it fixed yet?

    Or, more seriously: what kind of idea of "fixed" do these peope have? Does "fixed" really mean "it works now, and tomorrow is another day" for them?

    That's what "fixed" means to everyone, unless one believes in perpetual motion machines, dinosaurs that fart gold ingot, etc.



  • @joe.edwards said:

    I don't see how one field name is more conducive to a loop than another. You should be able to trivially write a function to map a number of indices to a field name; then, when the schema changes (which should be seldom), you update your mapping function.

    If you know in advance something is prone to change (and you learned the first time that it was), you code so you can change it in one place, not many.

    Really? Field1, Field2, etc. is the same as LetsDescribeThisFieldTisWay, LetsDescribeThisFieldAnotherWay, etc.?


    When you have half a bazillion fields then generic naming with numeric prefixes or suffixes makes looping several orders of magnitude easier than half a bazillion fields with arbitrary descriptions as field names. Numbers you can loop through. Text that doesn't follow any rhyme or reason ... not so much. And the major point was, there was no reason for this idiot to be changing the field names other than she didn't understand it. She never should have had her nose in the back end of the forms in the first fucking place.


    And ... they changed their shit quite a bit both before and after this particular episode. We had thought about mapping but it was decided that would be too much of a drain on my time. Of course that changed when Company unilaterally changed shit and Agency Management told me I had to rewrite my code.


    No, Joe, this was a sucky situation made worse by a stoopit bitch sticking her nose into something she wasn't even directly involved in.



  • @havokk said:

    @rudraigh said:
    Me: It's going to take me at least a week!


    Management: So, can you have that ready for test tomorrow?

    That particular response is one of my hot buttons. I want to scream back "excuse me, where you actually listening to what the hell I said to you barely three seconds ago?!".

    Well, that's the gummint for you. When I tell them it's going to take me a week (or month or whatever) they want it yesterday. When I tell them I can have it for them in five minutes they say it's not really due for three months.



  • @havokk said:

    @rudraigh said:
    Me: It's going to take me at least a week!
    Management: So, can you have that ready for test tomorrow?
    That particular response is one of my hot buttons. I want to scream back "excuse me, where you actually listening to what the hell I said to you barely three seconds ago?!".
     

    The correct response is:

     @rudraigh said:

    Me: It's going to take me at least a week!
    Management: So, can you have that ready for test tomorrow?

    Me: yes. It will fail the tests, but it will be ready. After a week I may rework it such that it will pass the tests.  What is your decision?

    FTFY.....


  • Garbage Person

    @Shoreline said:

    I'm interested to hear more about the accountability/cashflow situation. At a guess the customer pays you, but you only get money if you can interpret the file properly (which, in this case, you couldn't), so I'm guessing that either meant you couldn't charge them or you couldn't do something else with that data which somebody else pays you for, or you need the data for auditing.

    There was a question in there somewhere.

    Actually, it's really delicious. We made more money with the fuckup than we would have without - the only thing we lost is face with the client because they had to take a compliance hit.

    You see, the cause was ENTIRELY on them changing the file. What we produced isn't what was specified, but there was no specification for handling the "there are precisely zero eligible records" edge case because the situation is completely nonsensical (effectively, these books we're producing are specialized phone directories. Having a specification for handling this scenario would be like writing a specification for "there are no <thing in phone directory> in all of North America." Technically possible, but if it were true, our customer wouldn't exist.

    As such, they were charged for the entire first production run, were billed for everybody's time to research and recover from the problem, had to give us a pass on the missed SLAs while they were fixing it and they have to pay for the second production run as well. Between reproduction costs, saved SLA penalties (we were going to miss SLAs anyway - we ALWAYS miss SLAs because this project is terrible and sales sold an SLA that's only slightly longer than the best-case production time) and charged resource time, this singular incident paid my annual salary and the PM's salary.

     

    A lesser person would spin it as "I found a moneymaking opportunity for the company" on their performance review.



  • Lol. Splendid!

    @Weng said:

    A lesser person would spin it as "I found a moneymaking opportunity for the company" on their performance review.

    Not sure I agree with 'lesser person', but I would be inclined to agree that a person forced to work with lesser (perhaps just dishonest) people might take the opportunity to spin it that way. This is especially worth keeping in mind if any attempt is made to bite you on the arse with it: "They screwed up, we scored more cash because of it."

    I agree with your assessment: "delicious". :)



  • @Weng said:

     Actually, it's really delicious. We made more money with the fuckup than we would have without - the only thing we lost is face with the client because they had to take a compliance hit.

    You see, the cause was ENTIRELY on them changing the file. What we produced isn't what was specified, but there was no specification for handling the "there are precisely zero eligible records" edge case because the situation is completely nonsensical (effectively, these books we're producing are specialized phone directories. Having a specification for handling this scenario would be like writing a specification for "there are no <thing in phone directory> in all of North America." Technically possible, but if it were true, our customer wouldn't exist.

    As such, they were charged for the entire first production run, were billed for everybody's time to research and recover from the problem, had to give us a pass on the missed SLAs while they were fixing it and they have to pay for the second production run as well. Between reproduction costs, saved SLA penalties (we were going to miss SLAs anyway - we ALWAYS miss SLAs because this project is terrible and sales sold an SLA that's only slightly longer than the best-case production time) and charged resource time, this singular incident paid my annual salary and the PM's salary.

    A lesser person would spin it as "I found a moneymaking opportunity for the company" on their performance review.

    I hate it when they base SLAs on best case scenarios and not the normal case scenario.  But can't be as bad as my current program which has several monthly SLAs that have 100% targets.  Needless to say in the 4 years this program has been around it has not once made them.

    Also loved the story.  I was laughing over how stereotypical everyone was in the situation.



  • @Qwerty said:

    @Weng said:
    Apparently my grip on reality is too tight.
    Yep, I've seen that one before. Mostly when talking to HR. I don't even get emails from Sales.
    Great quote, I am so going to have to make this my sig.



  • @gilhad said:

     Ah, sales :) More than one time they came to us askig: "Hello, we just SOLD such and such feature to client, do you think it is even possible to program something like that?" 

    Sounds like you were very lucky, i mean they could have sold a feature that is impossible to implement!

    I had that happen once, if it wasn't impossible it was certainly ill-advised and probably illegal! In essence it was a system to allow a client to create, buy, and auto-magically place a TV commercial on the air; with zero human oversight!


  • Trolleybus Mechanic

    @esoterik said:

    @gilhad said:

     Ah, sales :) More than one time they came to us askig: "Hello, we just SOLD such and such feature to client, do you think it is even possible to program something like that?" 

    Sounds like you were very lucky, i mean they could have sold a feature that is impossible to implement!

    I had that happen once, if it wasn't impossible it was certainly ill-advised and probably illegal! In essence it was a system to allow a client to create, buy, and auto-magically place a TV commercial on the air; with zero human oversight!

     

    I just dodged a bullet by finally convincing my boss to convince a client that no, you cannot click on a hyperlink to a Word document on a webpage, and have the Word document print out on your local printer with no intermediary steps.

    * Yes, the Word document would be on a remote web server

    * No, printing it to a printer on the Web Server's network isn't what the client wants

    * Yes, it has to work in every web browser

    * No, they don't want to download the file, open it and print it

    * Yes, it has to remain MS Word, you can't convert it to PDF or HTML and send with window.print() command

    * No, they don't even want to see the printer dialog box

     



  • @Lorne Kates said:

    @esoterik said:

    @gilhad said:

     Ah, sales :) More than one time they came to us askig: "Hello, we just SOLD such and such feature to client, do you think it is even possible to program something like that?" 

    Sounds like you were very lucky, i mean they could have sold a feature that is impossible to implement!

    I had that happen once, if it wasn't impossible it was certainly ill-advised and probably illegal! In essence it was a system to allow a client to create, buy, and auto-magically place a TV commercial on the air; with zero human oversight!

     

    I just dodged a bullet by finally convincing my boss to convince a client that no, you cannot click on a hyperlink to a Word document on a webpage, and have the Word document print out on your local printer with no intermediary steps.

    * Yes, the Word document would be on a remote web server

    * No, printing it to a printer on the Web Server's network isn't what the client wants

    * Yes, it has to work in every web browser

    * No, they don't want to download the file, open it and print it

    * Yes, it has to remain MS Word, you can't convert it to PDF or HTML and send with window.print() command

    * No, they don't even want to see the printer dialog box

     Actually, it is NOT impossible....A roaring (and expensive) pain? YES ...... A bad Idea? AMLOST CERTAINLY



  • @Cassidy said:

    The correct response is:

     @rudraigh said:

    Me: It's going to take me at least a week!
    Management: So, can you have that ready for test tomorrow?

    Me: yes. It will fail the tests, but it will be ready. After a week I may rework it such that it will pass the tests.  What is your decision?

     

    - Yes? Great. (And walks away.)

    (The next day:)

    - How come you can't code a system that pass the tests? What kind of programmer are you and why should I keep paying your salary?

     


  • Trolleybus Mechanic

    @TheCPUWizard said:

    @Lorne Kates said:

    I just dodged a bullet by finally convincing my boss to convince a client that no, you cannot click on a hyperlink to a Word document on a webpage, and have the Word document print out on your local printer with no intermediary steps.

    * Yes, the Word document would be on a remote web server

    * No, printing it to a printer on the Web Server's network isn't what the client wants

    * Yes, it has to work in every web browser

    * No, they don't want to download the file, open it and print it

    * Yes, it has to remain MS Word, you can't convert it to PDF or HTML and send with window.print() command

    * No, they don't even want to see the printer dialog box

     Actually, it is NOT impossible....A roaring (and expensive) pain? YES ...... A bad Idea? AMLOST CERTAINLY

     

    wtf_face.jpg

    How the jelly-raped hell do you not label this under "impossible"?  No ActiveX, no modifying the uploaded Word document, and must be fully cross-browser compatible?

    Admittedly, the requirements didn't say anything about exploiting every known drive-by vulnerablity in the browser to download a malware that will hijack all links to word documents, silently download them in the background via FTP, then pass them (again, silently) into the command-line of the user's copy of Word (and if Word isn't installed, silently sneaking onto The Pirate Bay, obtaining a copy, then installing it via a background MSI so no user interation is needed).

    ASIDE from that, how is this not "impossible"?

     



  •  i think i disagree with most of you.  TRWTF is "I'm not allowed to talk directly to the customer" any rule like this is retarded.

    Also i note:

    @Weng said:

    .....................................................

    Project Manager: I don't know. I'm not allowed to talk directly to the customer. Apparently my grip on reality is too tight.

    ....................................................

    Project Manager: The client reports the rollback is complete. Can you pull the feed and test it?

    So he cannot talk directly to the client but has to wait for them to contact him??




  • @Mcoder said:

    - Yes? Great. (And walks away.)

    (The next day:)

    - How come you can't code a system that pass the tests? What kind of programmer are you and why should I keep paying your salary?

    When dealing with someone that doesn't really listen to information you impart, it's pointless arguing.

    Or: "if you can't answer my questions properly, I fail to see why I should answer yours." syndrome.

    Communication involves two people, both of which have responsibilities in the overall process.



  • @Helix said:

    TRWTF is "I'm not allowed to talk directly to the customer" any rule like this is retarded.
     

    Sometimes the rules are there for a reason: it prevents a customer from being contacted by several different parties who may not actually talk to each other, leaving the client frustrated that they've got to repeat the same information - as well different people agreeing different things with the customer. Funnelling all communication through a single nominated BRM means everyone's singing off the same songsheet from the supplier's viewpoint.

    That, plus there are individuals who may not have an appropriate telephone manner, so should concentrate on doing what they're best at and never come into contact with customers.

    I will agree that a blanket rule like this is stupid: there are times at which others can be in on a meeting with the BRM to discuss issues and provide feedback in a more timely manner, else you'll run the risk of information being transformed chinese-whispers like as it proceeds along the chain.

    And that rule applied to the PM? Yeah, that kinda feels a bit of a WTF to me.



  • @Weng said:

    Actually, it's really delicious. We made more money with the fuckup than we would have without - the only thing we lost is face with the client because they had to take a compliance hit....

     ... this singular incident paid my annual salary and the PM's salary.

     

    Damnation, a happy ending! Those responsible were actually made to pay for it.

    Are they open to the idea of implementing preventative measures (schema, etc) now they've felt their bruises?



  • @Cassidy said:

    @Helix said:

    TRWTF is "I'm not allowed to talk directly to the customer" any rule like this is retarded.
     

    Sometimes the rules are there for a reason: it prevents a customer from being contacted by several different parties who may not actually talk to each other, leaving the client frustrated that they've got to repeat the same information - as well different people agreeing different things with the customer. Funnelling all communication through a single nominated BRM means everyone's singing off the same songsheet from the supplier's viewpoint.

    That, plus there are individuals who may not have an appropriate telephone manner, so should concentrate on doing what they're best at and never come into contact with customers.

    I will agree that a blanket rule like this is stupid: there are times at which others can be in on a meeting with the BRM to discuss issues and provide feedback in a more timely manner, else you'll run the risk of information being transformed chinese-whispers like as it proceeds along the chain.

    And that rule applied to the PM? Yeah, that kinda feels a bit of a WTF to me.

    I agree, and believe that for many (most) situations the key element is ""others in on a meeting"....this allows both DIRECT communication [the original complaint] and the MODERATION/DEDUPLICATION that is desired...



  • For some reason I was imagining the PM as Alice, Weng as Dilbert and the Sales Guy as Loud Howard.


  • Garbage Person

    @Anketam said:

    I hate it when they base SLAs on best case scenarios and not the normal case scenario.  But can't be as bad as my current program which has several monthly SLAs that have 100% targets.  Needless to say in the 4 years this program has been around it has not once made them.
    So this morning, the PM kicks me over a specification to estimate for this same client. A whole new line of business, in addition to the project that went horribly wrong above (and several of its siblings). There are some satisfying parallels between this project and one of the other ones - it produces the same type of output at the exact same facility. The current lines all have comfortable 5-day SLAs, which we miss constantly because the facility is running above 100% capacity for these output types (the total volume from the original estimating and specification process was a maximum of x per day. Minimum observed volume has been x, with spikes to 3x). The particular equipment used for this output only has ONE instance in that facility. If it breaks, there is no failover.

    The SLA on this estimate request was 24 hours. The bare minimum cycle time for this production process in this facility with this backing software platform is 3 days.  Hell, the only way our software platform BY ITSELF can hit 24 hours is if the manufacturing steps are all automated, and we only have one facility online for that (and it's the wrong facility and this sort of output can't be automated in that manner)

    First time I've ever outright refused to estimate.  Of course, this is the first time I've been asked to estimate anything since a change in management made refusing to estimate an option.



  • @Cassidy said:

    @Mcoder said:

    - Yes? Great. (And walks away.)

    (The next day:)

    - How come you can't code a system that pass the tests? What kind of programmer are you and why should I keep paying your salary?

    When dealing with someone that doesn't really listen to information you impart, it's pointless arguing.

    Or: "if you can't answer my questions properly, I fail to see why I should answer yours." syndrome.

    Communication involves two people, both of which have responsibilities in the overall process.

    As someone who has been waiting six weeks for any feedback on some ultra basic questions, I'd take this level of useless feedback that Mcoder was getting...



  • The only thing sales it good for is making a disabled vet move their vehicle, so they can drag a picnic table into the parking spot, so they don't have to have a picnic in the wind.



  •  horse_ebooks, is that you?


Log in to reply