I used to work at a radio station, and part of my tasks included keeping the rather old automation system up and running. Now granted, this system was a 98 machine running on a beige box 500mhz and 128mb of ram, but the software was also a little at fault for the crashing it constantly was experiencing.

 The college I worked for also employed a guy who had been gang pressed into fixing problems at the station from time to time. So when I came on staff he was given the task of getting me up to speed. We went over the automation system and we both complained about the fact that the logs and schedules were all stored in Access databases, but what could we do? 

 A few months later I found myself talking to "Jerry" (as we'll call this guy). And discussing with him how unstable the current system was. I meantioned that I was thinking of switching to Rivendell (a GPL automation system). The conversation went something like this:

Me: "The current system is trash, it crashes all the time, I'm thinking of switching to some newer software."

Jerry: "You know, the guy who ran the station before you came is writing his own software."

Me: "Really?"

Jerry: "Yeah, he's using VB."

Me: "Ah huh. Well, I've been looking at Rivendell, it looks good. And it even keeps all the data in a MySQL database."

Jerry: "Oh that's no good. This other program uses flat files which are way more stable."

Me: "More stable?"

Jerry: "Yeah, databases cause lots of headaches. They crash all the time, corrupt data, etc. Using text files is better."

Me: "Banks use databases to store data."

Jerry: "Yeah, and you know how often those systems crash?"

<silence> 

How do you answer that? Needless to say, I kindof stopped asking for his advice on things.

Too bad you can't express "lol wut?" properly in a verbal conversation. 

I work for a bank.  Trust me when I say he's right.

 For a high volume/usage database maybe. But I've worked on dozens of SQL driven websites, and never had a single one crash. 

<agree>

vt_mruhlin:

I work for a bank.  Trust me when I say he's right.

</agree>

However  with databases there are all sort of solutions like: High Availability, Load Balancing, Good programming practices (except oracle, oracle just crashes :P)

The other point of data corruption, I would assume that databases actually corrupt data less often than flat files with a custom system written by a dude and VB... Unless heavilty QAed.

For performance, my co-worker uses nifty binary search algorithms to write a flat-file based search which searches at speeds no database can match, because he knows the format of all data and looks for specifics. Very easy to manage as well. But does not offer the flexibility of a DB, but at that point he don't care.

Just to add the OP co-worker's point's weight: Databases can be difficult to manage. To add the OP's argument, I don't think VB is a good tool for making efficient databases.

dlikhten:

a flat-file based search which searches at speeds no database can match, because he knows the format of all data and looks for specifics. Very easy to manage as well.

I think we found the 'swampie' that SpectateSwamp is always referring to.

Keep up the good work at the SwampShack!

dlikhten:

The other point of data corruption, I would assume that databases actually corrupt data less often than flat files with a custom system written by a dude and VB... Unless heavilty QAed.

Back in the 90's that was called DBase, and later, BTrieve.  The #1 problem was corrupted indexes due to multiple users modifying the same data files at the same time.  There were two workarounds 1) reindex all of the database files nightly and fix any corruption along the way; or, 2) buy a real database server. 

Granted, it wasn't written in VB, but it was the exact same idea.

clively:

 BTrieve.

Ahem....we like to refer to that as "The fucked up system that must not be named."

Pervasive is FTL. 

MasterPlanSoftware:

dlikhten:

a flat-file based search which searches at speeds no database can match, because he knows the format of all data and looks for specifics. Very easy to manage as well.

I think we found the 'swampie' that SpectateSwamp is always referring to.

Keep up the good work at the SwampShack!

SpectateSwamp:

MasterPlanSoftware:

dlikhten:

a flat-file based search which searches at speeds no database can match, because he knows the format of all data and looks for specifics. Very easy to manage as well.

I think we found the 'swampie' that SpectateSwamp is always referring to.

Keep up the good work at the SwampShack!

Whooo Hooo. I'm Not Alone.  Simple text flat files. That's the way to go. Easy to view and fix/manipulate/rig you name it. And we need to know way way less than the rest.

Has no one realized this guy is just a troll? I can't believe his Desktop Search thread has gotten so many replies. There's no way he could honestly believe that his program is comparable to any real solution on the market. I knew as soon as I saw the source code it was a joke.

joe.edwards@imaginuity.com:

Has no one realized this guy is just a troll? I can't believe his Desktop Search thread has gotten so many replies. There's no way he could honestly believe that his program is comparable to any real solution on the market. I knew as soon as I saw the source code it was a joke.

Thanks for the insight, you have opened my eyes to the fact that he is useless!

 Huh, what? I started the desktop search thread? oook.

No, what i was saying is: for very specific solutions sometimes a database is not the right way to go about things. A WTF is to think that database solves all isues period. I am saying there are some cases where a flat file with indexing based on the fact that you know you can do a binary search on say the first 10 characters of every line may be very efficient, and not require the maintenance overhead of a database. However very rarely do I find these to be the case.

The argument that:

1) Databases are super-stable

and

2) Databases solve all our data issues perfectly

is bad.

For DBs to be stable you need competent DBAs.Sometimes you have them, sometimes you don't. True the argument here that databases are always unstable is a bad one. And I assure you we have seen major performance increases from using a regex on a flat file than doing a database query because query might do a table-scan on 130million records, instead of just looking at the last 100 lines of a log file which is a bit more efficient no?

In any case i am not swampie, or SpectateSwamp or w/e. And I am not trolling anyone (intentionally)

dlikhten:

 Huh, what? I started the desktop search thread? oook.

No, what i was saying is: for very specific solutions sometimes a database is not the right way to go about things. A WTF is to think that database solves all isues period. I am saying there are some cases where a flat file with indexing based on the fact that you know you can do a binary search on say the first 10 characters of every line may be very efficient, and not require the maintenance overhead of a database. However very rarely do I find these to be the case.

The argument that:

1) Databases are super-stable

and

2) Databases solve all our data issues perfectly

is bad.

For DBs to be stable you need competent DBAs.Sometimes you have them, sometimes you don't. True the argument here that databases are always unstable is a bad one. And I assure you we have seen major performance increases from using a regex on a flat file than doing a database query because query might do a table-scan on 130million records, instead of just looking at the last 100 lines of a log file which is a bit more efficient no?

In any case i am not swampie, or SpectateSwamp or w/e. And I am not trolling anyone (intentionally)

Sounds to me like SpectateSwamp has brought you over to his side. Soon you will be displaying your random data on the screen with the Desktop Search pro!

First you must finish merging all your files together into a single text dump and screen capping all your other non-text documents. Then you get a well deserved break crawling through the brush with Spectate filming squirrels.

I maintained a couple dozen MySQL databases for about 3 years.  They corrupted themselves more than a few times.  In some cases, faulty disks and power failures were probably to blame.

However, a flat-file based system would have corrupted itself just as often, and probably more often, when dealing with the same transaction volume.

merreborn:

I maintained a couple dozen MySQL databases for about 3 years.  They corrupted themselves more than a few times.  In some cases, faulty disks and power failures were probably to blame.

 

However, a flat-file based system would have corrupted itself just as often, and probably more often, when dealing with the same transaction volume.

Not to mention scaling would be some kind of distant dream...

Lysis:

Too bad you can't express "lol wut?" properly in a verbal conversation. 

No, but you can get a fairly close approximation with the proper arch of an eyebrow. 

dlikhten:

I am saying there are some cases where a flat file with indexing based on the fact that you know you can do a binary search on say the first 10 characters of every line may be very efficient,...

Any reasonably competent DB can do exactly that. And correctly -- the first time. Without having to re-invent it again.

dlikhten:

and not require the maintenance overhead of a database.

What maintenance overhead? How can a [good] database with built-in solutions for maintenance be more trouble than rolling it all yourself?

dlikhten:

For performance, my co-worker uses nifty binary search algorithms to write a flat-file based search which searches at speeds no database can match, because he knows the format of all data and looks for specifics. Very easy to manage as well. But does not offer the flexibility of a DB, but at that point he don't care.

I didn't see you volunteered, or vounteering.

How is the swamp?

Or are we being meta-trolled?</rhetorical>

Not sure what the relationship between VB and flat files is. I've got VB happilly chatting with Oracle over here.

 If databases "crash all the time" then all you'd need to say is Windows crashes all the time. Rivendell looks interesting as it runs on Linux.

MasterPlanSoftware:

Not to mention scaling would be some kind of distant dream...

Yeah, because we all know every system ever written needs to be scalable "just in case".

Quinnum:

MasterPlanSoftware:

Not to mention scaling would be some kind of distant dream...

 

Yeah, because we all know every system ever written needs to be scalable "just in case".

... I detect sarcasm... thinking that the apps you write shouldn't be made scalable by default is dangerous thinking and is likely the cause of many of the WTFs on this site.

Just ask ObiWayne...

SpectateSwamp:

Simple text flat files. That's the way to go. Easy to view and fix/manipulate/rig you name it. And we need to know way way less than the rest.

Oof.  Must resist urge to launch into yet another SSDS flame.  Okay, taking this in a more general context:

For any project involving a significant amount of regularized data, the benefit of a RDBMS is well worth the overhead of communication and defining tables.

Flat files require you to pay attention to low-level details like "what column does such-and-such field start in?".  Sure, you can do it, but anyone with a clue would prefer not to have to.

When it comes to financial and inventory systems (my bread and butter), arbitrary end-user manipulation of the data is absolutely the last thing you want.  Double-entry bookkeeping would be the first thing to go, and then it would all end in tears, I tell you what.

MasterPlanSoftware:

Quinnum: