(I guess this post could fit into the "Coding Related Help & Questions" forum too since it's semi-related to an actual problem, but I thought it might make for an interesting mind-twist too, so here it goes...)

We all have had much laughter about those "I'm to cool for databases" web developers and the "ingenious" solution the designed to not use a database ("Storray Engine"). But there are in fact a number of web hosting packets (the ones where the maxium of control you get is ftp access) that provide server side scripting but no database. And unfortunately you don't always have the option to change that.

So, my question/challange is: Given a web space with the scripting langues of your choice, r/w/x access on all files but NO DATABASE whatsoever, how would you build a non-trivial web application (let's say a guestbook) without drifting off into WTF-sphere?
Specifically, your application should still work if a few more users use them simultaneously. (The site won't become google but if it does make a little publicity you want to be prepared.)
Also, while your space capacity for files is pretty large, the web hoster most likely placed restrictions on the running time and memory usage of your scripts. So storing all data in memory and keeping the script in an endless loop is off limits. (Besides, I'd hardly consider that not a WTF. And, no, I didn't try out that solution. Really :)

Is this even possible?

The best solution I could come up with so far is unfortunately not that far away from the storray engine, with a bit of extra logic added to handle multiple write requests.

Any better ideas? 

PSWorx:

So, my question/challange is: Given a web space with the scripting langues of your choice, r/w/x access on all files but NO DATABASE whatsoever, how would you build a non-trivial web application (let's say a guestbook) without drifting off into WTF-sphere?
Specifically, your application should still work if a few more users use them simultaneously. (The site won't become google but if it does make a little publicity you want to be prepared.)
Also, while your space capacity for files is pretty large, the web hoster most likely placed restrictions on the running time and memory usage of your scripts. So storing all data in memory and keeping the script in an endless loop is off limits. (Besides, I'd hardly consider that not a WTF. And, no, I didn't try out that solution. Really :)

Is this even possible?

A guestbook is a pretty easy one to do. I'll assume that the server is any modern unix:

Entries are stored one to a file, with sequentially numbered filenames 'data/1', 'data/2', etc. The file 'next' contains the number of the next file to create and is initially created containing the two bytes "1\n".

To display the current contents, read the file 'next' to get the latest position, and then read the files numbered next-1 to next-10 to display the 10 most recent entries (for example).

To add a new entry:

Read the number from the 'next' file.
Open that file with O_CREAT|O_EXCL
If that succeeds, write your entry into the file. Otherwise, increment your number and try again. Repeat until success.
flock the 'next' file.
Read the number in the 'next' file again. If it is greater than the number of the file you wrote, we're done, exit.
Create a temporary file in the same directory as the 'next' file and write (the number of the file you wrote) + 1 into that file.
Rename the temporary file to 'next'.
Exit.

This algorithm is blocking but cancel-free, isolated, durable, always makes progress, and (I'm fairly certain) contains no race conditions. You need to have a complete understanding of concurrency issues in order to write things like this, but any problem of the nature "store this chunk of data and then fetch it back later" can be implemented in a concurrency-safe manner on any reasonable filesystem - if you do not intend to do indexed relational queries or stuff like that, then you do not need a DBMS. A non-blocking transactional algorithm is possible, but more complicated and I can't think of a compelling reason to bother. This particular algorithm fundamentally relies on unix filesystem semantics (which were designed with cancel-free concurrent algorithms in mind) and cannot be ported to windows - you'd have to use something far more complicated there.

It's not actually consistent (it is possible for the display routine to read a partially-written entry), but it's fairly straightforward to add that, and I didn't feel like writing out the extra logic.

People always forget that the filesystem is a database. "Database" does not mean the same thing as "SQL". Filesystems belong to the class of databases known as "hierarchial databases" (which actually pre-date the existence of filesystems, and were in no small part the inspiration for modern filesystem designs). Relational databases are more buzzword-friendly nowadays, but they are certainly not the only kind, and for many purposes they are not the best kind to use.

Sorry to be pedantic, but please define "no database"

Using the loose term... no SQL Server / MySQL / Oracle, then I'd probably use db4o. If that's still too DB-ish then I'd go to Bamboo Prevalence. (they're both object persistence stores)

The off the wall ideas I could come up with were:

 - Lucene. Indexing and search retrieval engine. .. use it to just add a new entry for each "record". The downside is that it only allows one writer at a time.. the upside is that it writes really fast and scales extremely well for readers. You could also easily store text uncompressed for display as well as processed for search.

 - Tree structure in a memory mapped file. You could separate pages by some heuristic and then easily page through to find your information...

 - Extending that, it's be trivial to create a brain-dead ISAM implementation, but then that might be getting too far back into that "no database" rule.

Anyway, that's what I'd do...

IMO a guestbook is pretty easy since all you have to do is to append a short piece of text to a textfile. Just lock the file, append the new entry (and some seperator) and release the lock. Unless you are running the application on an underclocked IBM PC/XT, this operation should be fast enough not to get you into trouble with simultaneous users. Things become much more complicated when you have simultaneous updates.
 

I did this in high school years back.  However I'm sure the whole thing was WTF. Especially given my mad scientist methodology/curiosity back then.

ammoQ:

IMO a guestbook is pretty easy since all you have to do is to append a short piece of text to a textfile. Just lock the file, append the new entry (and some seperator) and release the lock. Unless you are running the application on an underclocked IBM PC/XT, this operation should be fast enough not to get you into trouble with simultaneous users. Things become much more complicated when you have simultaneous updates.
 

For a forum-type page, this would work perfectly. I was assuming that entries needed to be displayed in reverse order. 

Okay, looks like the problem was a lot easier than I thought. Thanks for the great ideas, I feel dumb now :)

webzter:

Sorry to be pedantic, but please define "no database"

Using the loose term... no SQL Server / MySQL / Oracle, then I'd probably use db4o. If that's still too DB-ish then I'd go to Bamboo Prevalence. (they're both object persistence stores)

The off the wall ideas I could come up with were:

 - Lucene. Indexing and search retrieval engine. .. use it to just add a new entry for each "record". The downside is that it only allows one writer at a time.. the upside is that it writes really fast and scales extremely well for readers. You could also easily store text uncompressed for display as well as processed for search.

 - Tree structure in a memory mapped file. You could separate pages by some heuristic and then easily page through to find your information...

 - Extending that, it's be trivial to create a brain-dead ISAM implementation, but then that might be getting too far back into that "no database" rule.

Anyway, that's what I'd do...

Sorry for being vague. It didn't really occurred to me that you can understand a file system as a kind of database as well. I was referring to "no relational database" here.

As for what is allowed and what not, I'm really just talking about the standard "shared hosting" setup here: FTP access to a given directory, no shell, web server and script interpreters preconfigured but outside of your reach, processes get terminated after a fixed running time (so you can't run any daemons).

So everything you CAN do is allowed. It's just that you have a lot of technical restrictions to deal with.

asuffield:

ammoQ:

IMO a guestbook is pretty easy since all you have to do is to append a short piece of text to a textfile. Just lock the file, append the new entry (and some seperator) and release the lock. Unless you are running the application on an underclocked IBM PC/XT, this operation should be fast enough not to get you into trouble with simultaneous users. Things become much more complicated when you have simultaneous updates.
 

For a forum-type page, this would work perfectly. I was assuming that entries needed to be displayed in reverse order. 

Well, unless there are millions of entries,  you could still keep the data this way and create the reverse order of the entries in memory.

PSWorx:

As for what is allowed and what not, I'm really just talking about the standard "shared hosting" setup here: FTP access to a given directory, no shell, web server and script interpreters preconfigured but outside of your reach, processes get terminated after a fixed running time (so you can't run any daemons).

So everything you CAN do is allowed. It's just that you have a lot of technical restrictions to deal with.

If it's a unix server then anything you could reasonably want to do is possible within these limitations, using methods like the one I sketched out. If it's windows then I think you're screwed - doing it there requires either ad-hoc long-running processes (which is broken by that pesky kill-after-so-long thing) or full access to the win32 API (which the usual web scripting languages don't give you).

ammoQ:

Well, unless there are millions of entries,  you could still keep the data this way and create the reverse order of the entries in memory.

Okay, I was also assuming that algorithms which are trivially DoSsed aren't acceptable either :P

(Throw a couple hundred Mb of data into the submission form and you'll drive the server to its knees every time somebody fetches the display page) 

asuffield:

ammoQ:

Well, unless there are millions of entries,  you could still keep the data this way and create the reverse order of the entries in memory.

Okay, I was also assuming that algorithms which are trivially DoSsed aren't acceptable either :P

(Throw a couple hundred Mb of data into the submission form and you'll drive the server to its knees every time somebody fetches the display page) 

This kind of attack could be easily prevented by the "create new entry" script.

Anyway, there are hosting providers which offer PHP+MySQL with their very cheap basic offers (domain registration fee+50 email accounts + 300MB webspace incl. PHP, Perl, MySQL = 2.99 EUR p.m.), so the easiest way to solve the problem might be to change the hosting provider. ;-)

Another option, if the server doesn't provide something like mysql (if you're looking for a relational database) would be something like SQLite, which is a standalone SQL application. I believe PHP/Perl/etc can access it nearly as easily as with mysql, and most applications wouldn't notice the difference.

-EJ 

I'm not sure whether you're taking the PHP/Perl/ASP(.NET) route, or even if you care to be specific there. But I do know of some nice third party class libraries for PHP and Perl which allow you to operate flatfile databases through standard SQL. These class libraries are generally pretty ingenious in their design and are capable of handling small scale applications without choking.

I dug around briefly for some links and found Gladius DB for php.

I realise this thread is a little old, but I thought I'd contribute my two cents anyway.
 

Already been done: http://blosxom.sourceforge.net/plugins/input/comments.htm

Guestbook, blog, newsfeed, and much more. 

I actually live under this requirement on the web.
I simply did everything in XML files.

Although i would say my site is trivial. (blog, photo thingy, file thingy) 

But you called a guestbook non-trivial, so ummm my website is REALLY complex.

(well the code is but that was because i had the framework bug when i built it. It could make coffee if it had permission for the COM port) 

This is a include file with three functions for dealing with data files in a program I am making called ndforum. Everything dealing with data files is in this include file, so to make it use a database, all you have to do is to replace this include file.

<?php
$files=array();
$line_ending="\r\n";

function getfiledata($f) {
 global $files;
 if(isset($files[$f])) return $files[$f];
 $x=array_merge(file('data/'.$f),array(""));
 $t=""; $n=""; $o=array();
 foreach($x as $v) {
  if($v[0]=='"') {
   $t=$t.str_replace("\r","",substr($v,1));
  } else if($v[0]==':') {
   $t=$t.str_replace(array("\r","\n"),"",substr($v,1));
  } else if(trim($v)!='') {
   $o[$n]=$t;
   $n=trim($v);
   $t="";
  }
 }
 $files[$f]=$o;
 return $o;
}

function selectfiles() {
 global $files;
 $args=func_get_args();
 foreach($args as $v) {
  if(isset($files[$v])) return false;
  if(file_exists('data/'.$v.'.locked')) return false;
  touch('data/'.$v.'.locked');
  $d=getfiledata($v);
  $files[$v]=$d;
 }
 register_shutdown_function('commitfiles',$args);
 return true;
}

function commitfiles($args) {
 global $files;
 global $line_ending;
 foreach($args as $v) {
  if(!isset($files[$v])) return false;
  if(!file_exists('data/'.$v.'.locked')) return false;
  $h=fopen('data/'.$v,'wb');
  if(!$h) return false;
  foreach($files[$v] as $k=>$x) if($k!='') {
   fwrite($h,$k.$line_ending);
   $x=explode("\n",str_replace("\r","",$x));
   $z=count($x)-1;
   foreach($x as $y=>$t) fwrite($h,(($y==$z)?'"':':').$t.$line_ending);
  }
  fclose($h);
  unlink('data/'.$v.'.locked');
 }
}

?>

: IF COMPILE ?-GOTO COMPILE-HERE ; IMMEDIATE
: THEN HERE SWAP ! ; IMMEDIATE
: ELSE COMPILE GOTO COMPILE-HERE SWAP HERE SWAP ! ; IMMEDIATE

zzo, code like that is why some people diss us PHP coders and our language. I was about to rewrite the whole thing for you, but it's late and I want to go to bed -- but please take note of what I finished so far. Note the consistent code style and spacing, the self-explanatory variable names, the way we're using some of PHP's built-in functions to do heavy lifting for us, in a way that's faster and more complete than we could do it ourselves (namely serialize and unserialize in this case), the sanity checking for bad filenames and missing files, and the comments fer crissake. Please take this as a bit of constructive criticism.

/**
 * Data file handling.
 */

$files = array();

function get_file_data($filename) {
	/**
	 * Load the data from a data file and return it. The data is cached so if it
	 * is needed later, it does not need to reload and re-unserialize it.
	 */
	global $files;
	if (isset($files[$filename])) {
		// We loaded this data previously, so don't touch the disk again.
		return $files[$filename];
	}
	validate_data_file_filename($filename);
	// If we're still here, load the data file
	$path = "data/" . $filename;
	if (!file_exists($path)) {
		trigger_error("Data file {$filename} does not exist.");
		$files[$filename] = NULL;
	}
	else {
		$data = unserialize(file_get_contents($path));
		if (!$data) {
			trigger_error("Unserialization of {$filename} failed.");
		}
		$files[$filename] = NULL;
		else {
			$files[$filename] = $data;
		}
	}
	return $files[$filename];
}
		
function validate_data_file_filename($filename) {	
	/** 
	 * Checks the data filename is sane before we try to read from or write to it.
	 * In case of insanity, kills the script.
	 */
	if (strpos($filename, "/") !== 0 || strpos($filename, "..") !== FALSE) {
		// Either a slash is the first character, or the path is trying to point
		// backwards from the data folder -- both very dangerous
		trigger_error("Attempt to access invalid data file {$filename}.", E_USER_ERROR);
		die(); // I think the above line halts execution, but it's late and I'm
		// not sure, so just in case…
	}
}

If you're using perl, there's DBD::CSV , making even the database easily FTP-updateable

I use PHP4 so file_get_contents isn't available, and I don't need to validate because data filenames aren't entered by the user anyways, and I know how to code the security. The reason I don't use serialize/unserialize is to make it easily edit by a text editor or by different software that isn't PHP. And I think my codes and your codes are both clear without the comments and your codes is too much indented for my opinion, some people like indenting like that but I prefer short indenting. Anyway, your is not wrong it just should be used for different software instead.

: IF COMPILE ?-GOTO COMPILE-HERE ; IMMEDIATE
: THEN HERE SWAP ! ; IMMEDIATE
: ELSE COMPILE GOTO COMPILE-HERE SWAP HERE SWAP ! ; IMMEDIATE

file_get_contents() is available in PHP 4; it's file_put_contents() that isn't.

I don't need to validate because data filenames aren't entered by the user anyways

Are you sure? Are you absolutely 100% sure that this will always be the case? This is most definitely a case where you can sacrifice performance for security, especially since it's so easy to do.

The reason I don't use serialize/unserialize is to make it easily edit by a text editor or by different software that isn't PHP.

Fair enough, but then you're still doing futzy stuff like over-handling newline characters (just use \n for everything!).

I think my codes and your codes are both clear without the comments and your codes is too much indented for my opinion, some people like indenting like that but I prefer short indenting.

Lines like foreach($x as $y=>$t) fwrite($h,(($y==$z)?'"':':').$t.$line_ending); may be legible to you, but to others trying to read your code (like me), it's as clear as pea soup. With regards to indentation, I typically code using two space characters per indent; this lovely forum software took it upon itself to bloat the indenting as it did. As for the lack of comments; no, there's just no excuse for that. Commenting not only makes your code more legible for others; it'll also make it more legible for you when you go back to edit this code after a hiatus. Think to yourself; if I quit working on this project for two months, then I go back to work on it again, am I going to understand at a glance what this code is doing? Ask any professional coder on this board whether commenting is relevant, and you'll get the same answer.

With regards to indentation, I typically code using two space characters per indent; this lovely forum software took it upon itself to bloat the indenting as it did.

O sorry I didn't know the software did that, when I wrote it I didn't get any such problems (maybe it is because I didn't use the WYSIWYG editor? Maybe only the WYSIWYG editor causes problems like that)

Think to yourself; if I quit working on this project for two months, then I go back to work on it again, am I going to understand at a glance what this code is doing?

I already did write it quit for more than 2 months and continue working on it again later and I do understand it perfectly.

Fair enough, but then you're still doing futzy stuff like over-handling newline characters (just use \n for everything!).

OK maybe I will correct that, but then if you want to open in Notepad it doesn't go??? The documentation tells you to use binary mode always even for text files so I have to make the line-endings myself

: IF COMPILE ?-GOTO COMPILE-HERE ; IMMEDIATE
: THEN HERE SWAP ! ; IMMEDIATE
: ELSE COMPILE GOTO COMPILE-HERE SWAP HERE SWAP ! ; IMMEDIATE

zzo38:

OK maybe I will correct that, but then if you want to open in Notepad it doesn't go??? The documentation tells you to use binary mode always even for text files so I have to make the line-endings myself

I strongly recommend Programmer's Notepad (http://www.pnotepad.org), it will read the file to figure out if it's using \r, \n or \r\n for line formatting and then when you save the file it saves it with windows line endings by default, you can change this easily.

I like Crimson Editor for text editing on Windows. It's not being updated anymore, but it's still a pretty solid program.