Give the keys to your data to an idiot



  • I was asked to chase down why a production run was failing. Ok, I'm not usually the one to do that sort of thing as I don't even have access to the production machines, but it was something to do.

    Since I can't even get onto the production boxes to look at logs, let along debug, I set up my environment to point at the production environment. I'd run my own instance of the server so I could turn up the logging level, and see what was actually happening. Yes, I know it's a wtf that a supposedly locked down environment can be accessed like this, but that's a whole other world-o-wtf.

    Long story short, I track it down to a timestamp that has about 12 decimal places of non-zero digits after the milliseconds. Hmmmm, normally, we only store down to ms. After some digging and test queries, I prove that it's the extra digits that are being truncated by the driver, and so the sql update, which filters on the timestamp (that was read in from the same row), is mismatching because of the extra precision. 

    So how did those digits get altered? It's not the code; we never change them in the application. It's not any of the other applications as they don't even use that field. Ergo, it had to have been someone doing an update statement outside the code - by hand.

    But nobody except specific personnel are allowed to do that any more.

    Then maybe it's one of the specific personnel doing something they shouldn't.

    They ask, but nobody is owning up to it.

    I suggest they check the backup tapes to see when the data changed, then narrow it down by who logged in during that window.

    The intersection window whittles the list down to one name; a new DBA.

    "Oh that, yeah, see I was looking at a random record and tried an update, then must have forgotten to roll it back."

    Um, you do know that you're not actually supposed to change the data, right?




  • And that DBA was promptly given the boot...



  • How does your customer find these people?  Do they build a list of applicants from worst to best and then work down it till they hit some acceptable quality level (however low that may be)?



  • Not sure about the sensitivity of the data you're dealing with, but isn't perusing production data a WTF in and of itself, let alone updating it?  And based on how things go at your work, I know he was promptly reprimanded by being given a window office with a raise and an extra week of vacation, plus a lap-dance from the SVP's teenage daughter, right?



  • @Sutherlands said:

    And that DBA was promptly given the boot...

    Are you kidding? He was probably given an 'atta boy' for finally fessing up, and fast-tracked to promotion.

    This is Snoofle's company we're talking about, here.



  • @All: no, the new guy didn't get fired. No, they're not going to fire him.

    However, to cover my boss (the data is extremely sensitive in terms of privacy regulations), I made sure to be discussing it loudly enough to be overheard while walking past the auditor's open doors. They grabbed us and inquired. We told all. Let them handle it, or not as the case may be.

     



  • @snoofle said:

    I was asked to chase down why a production run was failing.
     

    @snoofle said:

    Since I can't even get onto the production boxes

    See, I'm not sure how you can allow this situation to arise.

    If they want you to do something, they should satisfy any pre-requisites you have to fulfil that assigned task.

    I've never heard of a detective being told that they should investigate and report upon a crime yet denied access to valuable information important to their inquiry. @snoofle said:

    "Oh that, yeah, see I was looking at a random record and tried an update, then must have forgotten to roll it back."

    Um, you do know that you're not actually supposed to change the data, right?

     

    Has anyone (or the auditors) asked how a new hire is granted this level of access? Failure in policy, or failure in HR?



  • @CodeNinja said:

    @Sutherlands said:
    And that DBA was promptly given the boot...

    Are you kidding? He was probably given an 'atta boy' for finally fessing up, and fast-tracked to promotion.

    This is Snoofle's company we're talking about, here.
    I see somebody doesn't read the tags.



  • @Sutherlands said:

    @CodeNinja said:
    @Sutherlands said:
    And that DBA was promptly given the boot...

    Are you kidding? He was probably given an 'atta boy' for finally fessing up, and fast-tracked to promotion.

    This is Snoofle's company we're talking about, here.
    I see somebody doesn't read the tags.



    Why would you assume that?

    Probably should have added a, 'Sarcasm', tag to my original post.



  • @Cassidy said:

    Has anyone (or the auditors) asked how a new hire is granted this level of access?
    He's a relatively new hire (about 3 months) whose job it is to support the production databases, so by definition he has access.

    Using it inappropriately, on the other hand...

    I have a clue bat, and though some folks around here kind of..., no, deserve it, I don't go around murdering people...



  • @Cassidy said:

    @snoofle said:

    I was asked to chase down why a production run was failing.
     

    @snoofle said:

    Since I can't even get onto the production boxes

    See, I'm not sure how you can allow this situation to arise.

    We didn't always have it like that. When we got swallowed by MegaCorp, as part of grabbing-the-reins, they removed production access from everyone except dedicated support people. This make sense from an auditing point of view. However, the support folks don't really have the expertise to diagnose certain kinds of stuff, so it's delegated to us. We're not supposed to be on those boxes - ever. Of course, in practice, we're given logins and passwords (by those who are supposed to have access) as their delegates to do the work.

    Stupid? You betcha.

    The first time it happened after the lock down, we couldn't get on so the problem festered for days. Our management unofficially decided to look the other way so that we could fix the problems.

    Now I get what I need to do the work, or obviously the work doesn't get done.

    Of course, the right way is to have support people who know what they're doing, but, after all, this is WTF-Inc.



  •  Snoofle, is your company hiring? In case you ever run out of WTF's to fix, I can throw some more in, keeping us both employed!



  • @snoofle said:


    Um, you do know that you're not actually supposed to change the data, right?


    But then, what's the point of having a DBA? :)

     



  • @tchize said:

    But then, what's the point of having a DBA? :)

    The DBA can change the structure. You know, like deleting a column of extremely-important customer data and then finding out that the backup system they're supposed to be looking after doesn't work?

    But then it seems Snoofle's company is used to sending out half a million letters to customers to ask them for their ZIP code again.


  • Considered Harmful

    @Qwerty said:

    But then it seems Snoofle's company is used to sending out half a million letters to customers to ask them for their ZIP code again.

    They can deliver mail without a ZIP code?


  • Discourse touched me in a no-no place

    @snoofle said:

    @All: no, the new guy didn't get fired. No, they're not going to fire him.

    However, to cover my boss (the data is extremely sensitive in terms of privacy regulations), I made sure to be discussing it loudly enough to be overheard while walking past the auditor's open doors. They grabbed us and inquired. We told all. Let them handle it, or not as the case may be.

     

    If it weren't for the fact that you're dealing with idiots, I'd assume that eventually people would learn that walking past the auditors with Snoofle is a bad idea.



  • @joe.edwards said:

    They can deliver mail without a ZIP code?
     



  • @joe.edwards said:

    @Qwerty said:
    But then it seems Snoofle's company is used to sending out half a million letters to customers to ask them for their ZIP code again.

    They can deliver mail without a ZIP code?

    I thought that was the joke as well...  If it was, I'm glad that you took the bullet instead.


  • @C-Octothorpe said:

    @joe.edwards said:

    @Qwerty said:
    But then it seems Snoofle's company is used to sending out half a million letters to customers to ask them for their ZIP code again.

    They can deliver mail without a ZIP code?

    I thought that was the joke as well...  If it was, I'm glad that you took the bullet instead.

    The joke was more the idea that you would need to ask, with the rest of the address you can just look up the ZIP code.



  • @locallunatic said:

    @C-Octothorpe said:

    @joe.edwards said:

    @Qwerty said:
    But then it seems Snoofle's company is used to sending out half a million letters to customers to ask them for their ZIP code again.

    They can deliver mail without a ZIP code?

    I thought that was the joke as well...  If it was, I'm glad that you took the bullet instead.

    The joke was more the idea that you would need to ask, with the rest of the address you can just look up the ZIP code.

    Seems you can deliver registered post without a signature, too (according to the other thread)

     


  • Considered Harmful

    @Cassidy said:

    Filed under: And to the wrong address yet it still gets there



  • @FrostCat said:

    If it weren't for the fact that you're dealing with idiots, I'd assume that eventually people would learn that walking past the auditors with Snoofle is a bad idea
    Why? We were powerless to fix the problem, so I brought it to the attention of folks who have some power to push back and at least try to fix it.

    If I just ignore it, then I become part of the problem.



  • @snoofle said:

    @FrostCat said:

    If it weren't for the fact that you're dealing with idiots, I'd assume that eventually people would learn that walking past the auditors with Snoofle is a bad idea
    Why?
     

    A bad idea where people in { idiots, fuckwits, process circumventers, loose cannons } etc.

    Bad idea for them. Good idea for those that want things to improve and change from the Bad Old Ways.

     



  • @Cassidy said:

    @snoofle said:

    @FrostCat said:

    If it weren't for the fact that you're dealing with idiots, I'd assume that eventually people would learn that walking past the auditors with Snoofle is a bad idea
    Why?
     

    A bad idea where people in { idiots, fuckwits, process circumventers, loose cannons } etc.

    Bad idea for them. Good idea for those that want things to improve and change from the Bad Old Ways.

     

    Sorry - I've had a rough week (Hurricane Sandy); we made out ok without power and helped lots of folks in much worse shape, but my mother, who lost power and lives in an apartment with electric heat/hot water/stove, refused to come live with us out of sheer stubbornness (we finally got her out of there when the temp hit 32F and we got the kids to play the guilt-card, saying they didn't want her to become a grand-momsicle).


  • @snoofle said:

    Sorry - I've had a rough week (Hurricane Sandy);
     

    No apology necessary (well, not to me - can't see anything that requires it).

    Glad to hear you and yours are alive and well. Have friends in Maine that work in Boston - things weren't as bad there, but it didn't stop people from predicting the apocalypse.

    Fortunately, Romney lost, so that's averted.



  • @snoofle said:

    We didn't always have it like that. When we got swallowed by MegaCorp, as part of grabbing-the-reins, they removed production access from everyone except dedicated support people. This make sense from an auditing point of view. However, the support folks don't really have the expertise to diagnose certain kinds of stuff, so it's delegated to us. We're not supposed to be on those boxes - ever. Of course, in practice, we're given logins and passwords (by those who are supposed to have access) as their delegates to do the work.

    Stupid? You betcha.

    The first time it happened after the lock down, we couldn't get on so the problem festered for days. Our management unofficially decided to look the other way so that we could fix the problems.

    Now I get what I need to do the work, or obviously the work doesn't get done.

    Of course, the right way is to have support people who know what they're doing, but, after all, this is WTF-Inc.

    This sounds very similar like the client we're currently working for. We devs used to have full access to their production DB, but then one of our less... prudent members (who thankfully is no longer on this team) dropped an important sproc and forgot to recompile it. For an hour. This kinda made the client unhappy, so their DBAs revoked our logins on prod.

    For obvious reasons, they didn't revoke the login used by every app to connect to that DB. For less obvious reasons, that login has full sysadmin rights (drop database etc.)... so we just use that one when we need to. (We have prod logins with read-only access now, but they are disabled more often than not, and getting them enabled is an exercise in bureaucracy.)

    The reason we need prod logins is because the client's DBAs, who earn a lot more than we do, are pretty good at fucking the DB in the bum. Dropping a unique constraints that we added thus causing a 6-month-long data integrity issue, is only one of their party tricks. Bypassing the auditing (which is done in the business layer code, not via triggers - because triggers are evil according to them) is another. Oh, and these are the same DBAs who sent an official email claiming that a DB-wide reseed of all identity columns in all tables was impossible... we sent them a script that did exactly that 2 hours later.


  • ♿ (Parody)

    @Cassidy said:

    Fortunately, Romney lost, so that's averted.

    True, but that's like saying, "Fortunately, I didn't get the flu this winter. Instead I got a flesh eating bacteria infection."



  • @The_Assimilator said:

    The reason we need prod logins is so that we can freely connect in and fix the fuckups made by the client's DBAs, who earn a lot more than we do.
     

    FTFY. 

    They'll carry on earning a lot more and fucking things up because they know some low-paid muppets will happily jump in and rescue their fat from the fire.

    Next time they're bronzing their dicks on their own DB, throw the problem back to the client ("your paid staff have locked out our access because they are perfectly capable of fixing these problems!")

     



  • @Cassidy said:

    @The_Assimilator said:

    The reason we need prod logins is so that we can freely connect in and fix the fuckups made by the client's DBAs, who earn a lot more than we do.
     

    FTFY. 

    They'll carry on earning a lot more and fucking things up because they know some low-paid muppets will happily jump in and rescue their fat from the fire.

    Next time they're bronzing their dicks on their own DB, throw the problem back to the client ("your paid staff have locked out our access because they are perfectly capable of fixing these problems!")

     

    Believe me, if I was the one making the decisions, we would have told this particular client to take a hike long ago. Unfortunately our CFO, who does make such decisions, would sell our bumholes to a client if he thought there was a possibility of getting monetary reward from doing so.

    I must make a thread about our CFO sometime. Not only is he a raving conspiracy theory nut who believes AIDS doesn't exist, he has a history of picking clients who end up being complete nightmares and signing away us devs' rights to said clients. Rarely have I met someone with such bad judgement. So why is he the CFO? Because he pinches pennies like no tomorrow, and by that I mean he's fucking cheap.

    Case in point: our parking lot was basically just earth, which turned into mud in winter. Ordinary people would've paid for a truckload of wood chips to cover the earth... he got some builders to dump their rubble in the parking lot, and that was spread around and buried to make the ground less marshy. Unfortunately rubble has sharp edges and car tires don't like this.



  • @The_Assimilator said:

    I must make a thread about our CFO sometime.
     

    You've wetted my appetite now. You must deliver. You MUST.

    @The_Assimilator said:

    Unfortunately our CFO, who does make such decisions, would sell our bumholes to a client if he thought there was a possibility of getting monetary reward from doing so.

    Anyone slapped him around the head with a cost-benefits analysis? "we keep buying this slut dinner and she STILL refuses to put out!" syndrome[1]

    An alternative approach is to artifically skew work in favour of his decisions to magnify their effect: "Sorry, I've been unable to work on $PROJECT this month because it's been made clear to me that providing free support to $FUCKWITTED_CLIENT is a higher priority."

    [1] with due apologies to the fairer sex - I never invented that term, I only know it as that.



  • @Cassidy said:

    An alternative approach is to artifically skew work in favour of his decisions to magnify their effect: "Sorry, I've been unable to work on $PROJECT this month because it's been made clear to me that providing free support to $FUCKWITTED_CLIENT is a higher priority."

    If you say that at the end of the month, then you will wear the blame for $project not getting done. You need to do it in advance to properly deflect the blame back to the CFO source
    "Which paying project do we stop work on to finish that new feature you promised for $fuckwitted_client in $unreasonably_short_timescale?"



  • @Qwerty said:

    If you say that at the end of the month, then you will wear the blame for $project not getting done.
     

    Oh, granted - need to choose the time carefully. However, although I may be blamed, I can easily point to the reasons why $project didn't get done and get people to look past blame into cause.

    @Qwerty said:

    "Which paying project do we stop work on to finish that new feature you promised for $fuckwitted_client in $unreasonably_short_timescale?"

    Whiteboard tasklists, my weapon of choice...



  •  @Cassidy said:

    although I may be blamed, I can easily point to the reasons why $project didn't get done and get people to look past blame into cause.

     

    If you work for a company where this is actualy possible you are unlikely to run into the situation in the first place.

     

    in WFT factories the management seldom look beyond identifying a scapegoat

     

     


Log in to reply