Most people would say that a reasonable expectation for the availability of an important computer application would be continuous availability at a basic level without regard to available support, and almost continuous availability with only a basic level of support. The data entry script, requiring only that the web server be running and the script appropriately installed, fits that model. As long as the network hardware is functional, connecting the entry stations to the cluster, the primary suspect in any system failure would be a lock up of the cluster controller / web server, or an abend of the server software. While each of these are very unlikely events, each could be resolved by restarting the machine, something that most people could handle. (Realistically, if one of the cluster machines begins to exhibit periodic lock-ups, the machine should be examined for hardware failures and probably replaced. This is a problem attendant to using surplus hardware, but hey, if you only paid 25 USD for the machine, that's not a big deal, right? That's one of the reasons I've been so involved with assuring redundancy within the system...the most critical problems would involve failure of the cluster controller or database controller. Interior cluster machines could easily be configured with the software required to allow them to be plugged into those roles with only minimal modification of system configuration files. (I'll get into procedures for doing that at a later point. The focus of this section is somewhat different.)

Removing the susceptibility of the entry script to difficulties attendant to interaction with the database server was, of course, one of the rationales for implementing that system function as a separate script. In effect, I've de-coupled that portion of the application to run on its own, and as we've seen that works quite well. However, the fact that the script is running in the background does create one problem ... it can fail in the background and no feedback other than the voluminous amount of detail recorded in the log created by the DBI trace facility will be provided to allow the context of that failure to be reconstructed. The DBI trace log is fine for detailed analysis, but it would be nice to have a log that would record the time and date when a key problem surfaces and a brief description of the nature of the problem. Such a log could be used by an educated user to resolve many if not most problems without requiring the presence of higher-level support personnel. That's the type of thing I'm going to start creating here, starting small and adding to the logging facility as we proceed. At some point it is likely that logging facilities will be added to a custom module of application object that I roll up out of what we're doing here. But that's down the road a bit, for now let's just take the next step in our journey of a thousand miles. (A soft gong can be heard in the distance as we gaze serenely at the rock-strewn path upon which we tread. Almost unconsciously, we hear the soft murmur of the steam that parallels our path.) <grin>.

On to the new version of the script. When I said that this incarnation of the scripts make the biggest step in complexity to this point it was this script that I had in mind. The number of lines in the script have almost tripled, and while a large number of those lines represent the internal comments in the script the size of the increase represents a fair measure of the increase in the script's complexity. This version is intended to take full advantage of the distributed file system within the cluster, with options specified for the locations of the system log and of the repository to which records will be written if the database connection fails. It could be argued that this represents overkill, that simply making a log entry to the effect that a database connection could not be established and exiting the script would be sufficient. My response would be to concede that this is a valid point, but I have two reasons for going forward. First, it has been my general experience that if you leave a potentially system-threatening hole in place there will come a time when it threatens the system. In this context that potential hole is the fact that writing the output files to their temporary storage location will eventually fill that drive space, and that would lead to the loss of data. Therefore, these steps are appropriate. Second, one of my rationales for the development of this system in the first place is to develop techniques and explain their implementation. Ultimately, it is my call, and I decided to do it this way <grin>.

A high level overview of this script would have at its heart the same elements as the previous version ... the script reads a directory of text files and stores the records contained therein into a database. In this version, however, I have added a couple of layers of top of the the base logic. In recognition of the fact that this script will be running in the background, unattended, I'm starting to add the capability for the script to respond to the state of the environment. The primary strategy underlying this approach involves specifying a set of locations for the system log and repository files, and then checking these locations for available space and the ability to write to the file before that location is accepted. The script takes a cautious, conservative approach in that if there are problems associated with either the log file or the repository file it simply shuts down, leaving the source files intact. In other words, the script is configured to not take any chances that might lead to the disappearence of entered data ... if something is interfering with the ability of the script to leave traces of its actions or to write records to a repository if the database connection fails, the script will simply quit. The underlying problems can then be addressed, and when it next runs the script will process the files in the source directory and continue on.

One aspect of the system that adds considerably to the complexity of this script is the use of the mfs file system to write the files to drives elsewhere in the cluster. This adds considerably to the overall flexibility of the system, but as standard system tools to determine the characteristics of those drive spaces do not currently function appropriately through the mfs mount point I have had to take an alternative route to that information. The resolution I have implemented involved running remote shell commands on the cluster machines in question. I will get to that in more detail shortly.

To execute these remote shell commands there are a few preparatory steps that must first be taken. The first is simply to specify the names of the machines that are used as repositories, in this case the database server, ralphzilla-raider, and the cluster controller, ralphzilla.

my $controller='ralphzilla';
my $db_server='ralphzilla-raider';

A relevant aside at this point is to mention that this script is relatively more hard-coded than most of what I've done to this point. A good example of this is in the free_space subroutine, in which I open the file containing the output of the df() system command and explicitly step to the relevant line rather than using a pattern match to do the same thing.

		open(FREE,'/home/www/free.txt');
		$line=<FREE>
		$line=<FREE>

In any event, as I indicated above the current script assumes that the cluster controller and database server represent appropriate targets for these files. It has been the operative strategy of the versions of this script to access filesystems not on the host running the script though the mfs mount point. Since mfs mounts the filesystems of the relevant machines in a tree keyed to the mosix numbers of the respective machines, the script has to be able to associate the machine name with its mosix number. In this version the script checks for a file named numbers in the /home/www directory, and if that file does not exist the number_config subroutine is executed to create it. The number_config subroutine executes the mosix mosctl utility with the whois parameter, which returns the mosix number of the relevant machine when it is fed the name of the host. The output fom the two mosctl commands is redirected to the numbers file. When the script reads in the number file it splits each line on whitespace and the number is pulled from the string of which it is a part, associated the result with either the $controller_num or the $server_num scalar.

my ($t11,$t21,$t31,$t41);
($t11,$t21,$t31,$t41)=split(/\s/,$line);
$line=<NUMBERS>
my ($t12,$t22,$t32,$t42);
($t12,$t22,$t32,$t42)=split(/\s/,$line);
my $controller_num=substr($t41,1,1);
my $server_num=substr($t42,1,1);

my @repository_destination=("/mfs/$server_num/data/bb_system/",
							'/home/www/',
							"/mfs/$controller_num/home/www/");
							
my @log_destination=("/mfs/$server_num/data/bb_system/",
					 '/home/www/');

while ($loop)	{
	
	##grab the next element from the array
	$l_dir=shift @log_destination;
	
	##if that element is undef, it means that we've gone past the last element in the array.  since the script would have 
	##continued onward if a viable log file handle creation had been achieved (both open and with greater than 100k available
	##on the filesystem), that means the script can't create a log.  therefore, it quits.
	if (! $l_dir)	{exit;}
	
	##read the available space on the location specified in the current array element
	free_space($l_dir);
	
	$log_file=$l_dir.$log_name;
	
	##use the eval() function to either create the filehandle, or not
	$log_fh=new FileHandle($log_file,'+>');

		##if the open statement worked and there is at least ca. 98k available, store 1 to $exit, which will lead to loop exit
	if (($free>$min) and ($log_fh))	{
		
		$loop=0;
		
	}
	
}

The next line in the script is an if statement that checks to make sure that the amount of free space on the filesystem is greater than the defined minimum and that filehandle creation. If both of those conditions are true, the value 0 is assigned to the $loop scalar. In this manner the pessimistic prejudice of the script is enforced. There are only two ways out of the loop: either a viable filehandle is established on a filesystem with sufficient available space or the end of the array is reached and the script is terminated.

Note that the value assigned to the minimum file space requirement is arbitrary, but it should be set to a reasonable minimum. As available file space is checked on each execution of the script, all we really have to worry about is the space that is required to write repository or log records for one invocation of the script. We are currently storing a maximum of 82 bytes of entered data plus 30 bytes of delimiters per record, therefore the maximum size of any file stored (if an eight-form display screen were used) would be just short of one Kb. To put this in context, if five people were entering records of the maximum size and averaged one eight-form screen per minute (an impressive rate of entry), running the script at five minute intervals would limit the space requirement for a single invocation to less than 25 Kb. Therefore, the 500 Kb. used here as a minmum for the repository location is a generous minimum. At the same time, if the target application is on a key mount point on the target machine, such as /var, filling the system to its capacity could threaten the operation of system services, so it makes sense to allow slack. Space requirements for the log file are of course even lower; I have arbitrarily set the minimum space requirement for that location to 100 Kb.

An important component of this subroutine is provided by the free_space() subroutine, which is used to determine the amount of space free on the target filesystem. If you look to the beginning of the script you will see that I have brought the module Filesys::DiskFree into use. This was to be the lynchpin of the test for free space, and the method worked ... except it didn't return the correct result. Further investigation revealed that when used to read the characteristics of a location under the mfs mount point the module returns the amount of disk space available on the filesystem on which the root filesystem on the host is mounted. An mfs glitch. (At the risk of belaboring a point, this is the nature of the development process, and reflects the necessity of remaining flexible.) Given that I needed to determine the amount of space available, it became clear that I needed to write something to do that job. The free_space subroutine was the result.

In the first two lines of the subroutine I read the directory passed to the subroutine as a parameter and split it on the slash character ("/").
my $dir=shift @_; my ($t1,$t2,$t3,$t4)=split('/',$dir,4); As the mfs mount point is mounted under the mount point /mfs on this system, if the second element of that result is "mfs", then we have to deal with getting that value from a mosix machine. (Since this string begins with the delimiter specified for the split ("/"), split assumes that the first element is null, or empty, which is why I specify the second element. If I had specified that the split be on the substring of $dir starting with the second character, thus dropping that initial "/", the pertinent element would be the first element. Six of one, half-a-dozen of the other.)

Within that chunk of code that executes if this directory is under the mfs mount point, the first step is to assemble a scalar holding the fully-qualified file name relative to the root directory of the host on which the file resides.

	if ($t2 eq 'mfs')	{
		
			my $path='/'.$t4;

Regardless, at this point I need to find out the name or address of the target machine. In this specific circumstance I use a simple if statement associating the directory numbers with the mfs numbers stored in the scalars populated from the numbers file, and assign either the name of the database server or the name of the cluster controller in the $rhost scalar.

		if ($t3==$controller_num)	{
			$rhost=$controller;
		}
		elsif ($t3==$server_num)	{
			$rhost=$db_server;
		}

		my $command="df $path > /home/www/free.txt";

		system("rsh $rhost $command");

If the target filesystem is local to the script host, however, ("$t2 ne "mfs""), however, the $free scalar is assigned the output from the avail() method of the $df Filesys::DiskFree object.

		$free=$df->avail($dir);

Back in the main body of the script I create a new FileHandle object on the concantenation of the scalar holding the string representing the target directory being evaluated and the scalar holding the file name, the result representing the fully-qualified file name from the root directory of the script host.

	$repo_fh=new FileHandle($r_dir.$repo_name,'>>');

After instantiating the new Filehandle object, an if statment is evaluated that will only return true if the amount of free space on the target filesystem exceeds the minimum and the instantiation of the filehandle was successful.

	if (($free>$r_min) and ($repo_fh))

Now one might think that this is pretty much it, that now that I know that the repository and log files can be created and written to, all I have to do is print to the filehandles, right? After all, I've written to the repository file in previous versions of the script, and adding the capability to output to a second file should be almost trivial. HA! As you will see shortly, that was not the case.

But first, let's take a quick look at the little subroutine that actually writes the error log.

sub err_print	{
	
	my $error=$_[0];	
	my @time=localtime;
	my $year=$time[5]+1900;
	my $month=$time[4]+1;
	print $log_fh $time[2].':'.$time[1].' on '.$month.'/'.$time[3].'/'.$year.':::'.$error."\n";
}

Now the difficulty I ran into with this version of the insert script has to do with the fact that perl will flatten out data structures like filehandles when they are accessed from outside of the scope of the script in which they were created. This did not cause a problem with the slightly simpler version of the script that preceded this incarnation, but in this version once I descended into a subroutine my ability to reliably to reliably step through each line in a file referenced by a filehandle effectively disappeared. I spent chunks of time over a two or three day period trying workarounds like re-winding the input filehandle with the seek() command before finally conceding that revision of this portion of the script was required. If you want to get a good picture of what is happening here, start your local X-server and invoke the script under the ptkdb debugger. I've left the file recs_insert_3_orig.pl in the samples file, which uses the original filehandle method of stepping through the lines in the input files and which is set up to automatically invoke the ptkdb debugger. Set a breakpoint at line 229 and, if the database server is running, one at line 258. If the database server is not running the break point should be set somewhere around line 296, within the write_recs subroutine. Enter "<RECS>" (without the quotes, of course) in the expression box. Now click "[Run To]". When the script next pauses, you should be at line 229 and in the watch window on the right-hand side of the screen you should see the lines of the input file under <RECS>, as shown in the illustration to the left.

Now click [Run To] again, and the script should execute to the next breakpoint you set, displaying a screen such as that to the right. You will probably now see something like "ARRAY" with a number after it associated with <RECS> in the watch window, although on the first time through you may actually see the array displayed just as it was in the calling scope of the script. That's why this version of the script will not work ... at this point all that is visible is the identification of the array that holds the filehandle contents, not the array itself.

Now, as you might imagine, perl has a number of ways to go about resolving this, ranging from storing filehandles in scalars and accessing the filehandle through the scalar to passing the subroutine a reference to the filehandle. The two methods are different ways to doing the same thing, which you could think of as pointing at the remote namespace location of the filehandle array. I am sure that I'll be getting into techniques like this in other contexts, most specifically in setting up references to arrays. What I'm going to do here is adopt a somewhat different approach by explicitly reading the contents of the pertinent file into an array, and passing the array to the subroutine. This is the most basic way of passing information around within a perl application, and is well-suited to this particular circumstance. Indeed, this is the basic method of communicating between software constructs written in whatever language, whether those constructs are called subroutines, functions, or methods. If you think about it a little bit, however, you will probably be able to come up with the types of situations in which this strategy is not particularly well-suited. What I am doing here is creating a second data structure and using that in the subroutine. That's fine when the maximum size of any given input file is eight lines of modest length, but what if the input file is large? Clearly, you do not want to pass a 10 Mb. array to a subroutine. That is when you want to use a fancier form of referencing a data structure. But I'll get to that in due course.

The implementation of the new strategy begins on line 227, on which the contents of the file referenced by RECS is read into the array @recs.

my @recs=<RECS>;

write_recs(@recs);

elsif ($conn)	{insert_recs(@recs);}.

@recs=@_;.

Really not all that big a deal, don't you think? It took me significantly longer to explain what is going on than it took me to actually make the change. There is, however, one other thing in the new version of the script that I want to point out. If you look at one of the lines that call the err_print() subroutine, such as line 279,

err_print($error);

my $error=@_[0];.

That's about it for this incarnation of the system. In the next I will tie up a few odds and ends and tidy this script up a bit.