The Horror of the Mover
-
I could have sworn I'd already posted this, but I went back and looked and couldn't find it ... I transcribed and sanitized on-the-fly, so any typo's are likely my fault. Not only is this ugly, but there was a (not-so?) subtle bug that went unseen for a number of years which was fixed last year-ish (and I didn't include the fix.) Can you find it?
#!/bin/csh -f
# Purpose: Copies and renames new input files to the appropriate
# input directories for APP. Compress the file then
# move compressed file to archive.set DATADIR = /production/data/input/SOURCE
set APPDIR = /production/APP/data/input/
set APPDIR1 = /production/APP/data/input1/
set APPDIR2 = /production/APP/data/input2/set ARCHDIR = /archive/SOURCE/
set LOGDIR = /production/logs/
set LOGFILE = ${LOGDIR}/mover.logset DATE = `date`
set MONTH = `date +%m`
set YEAR = `date +%Y`
set HOUR = `date +%H`cd ${DATADIR}
unalias mv
unalias rmwhile 1
ls *.dat >&! /dev/null
## if($status != 0) then
## exit
## endifif ($status == 0) then
set MONTH = `date +%m`
set HOUR = `date +%H`
set YEAR = `date +%Y`foreach file (*.SOURCE)
date '+%y/%m/%d %H:%M was the time of the last data pull' >! ${DATADIR}/mover.touch# make sure permissions are accessible
# echo setting permissions on input files
chmod 777 ${file}# copy the input files to the correct directories for APP
echo "Copying File to APP Directories detail. ${file} " `date` >>! ${LOGFILE}
switch ( $HOUR )
case "01":
cp ${file} ${APPDIR}
mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
echo "copying file for input detail"
breaksw
case "02":
cp ${file} ${APPDIR1}
mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
echo "copying file for input detail 1"
breaksw
case "03":
cp ${file} ${APPDIR2}
mv ${APPDIR2}/${file} ${APPDIR2}/${file}.dat
echo "copying file for input detail 2"
breaksw
case "04":
cp ${file} ${APPDIR}
mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
echo "copying file for input detail"
breaksw
case "05":
cp ${file} ${APPDIR1}
mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
echo "copying file for input detail 1"
breaksw
case "06":
cp ${file} ${APPDIR2}
mv ${APPDIR2}/${file} ${APPDIR2}/${file}.dat
echo "copying file for input detail 2"
breaksw
case "07":
cp ${file} ${APPDIR}
mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
echo "copying file for input detail"
breaksw
case "08":
cp ${file} ${APPDIR1}
mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
echo "copying file for input detail 1"
breaksw
case "09":
cp ${file} ${APPDIR2}
mv ${APPDIR2}/${file} ${APPDIR2}/${file}.dat
echo "copying file for input detail 2"
breaksw
case "10":
cp ${file} ${APPDIR}
mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
echo "copying file for input detail"
breaksw
case "11":
cp ${file} ${APPDIR1}
mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
echo "copying file for input detail 1"
breaksw
case "12":
cp ${file} ${APPDIR2}
mv ${APPDIR2}/${file} ${APPDIR2}/${file}.dat
echo "copying file for input detail 2"
breaksw
case "13":
cp ${file} ${APPDIR}
mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
echo "copying file for input detail"
breaksw
case "14":
cp ${file} ${APPDIR1}
mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
echo "copying file for input detail 1"
breaksw
case "15":
cp ${file} ${APPDIR2}
mv ${APPDIR2}/${file} ${APPDIR2}/${file}.dat
echo "copying file for input detail 2"
breaksw
case "16":
cp ${file} ${APPDIR}
mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
echo "copying file for input detail"
breaksw
case "17":
cp ${file} ${APPDIR1}
mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
echo "copying file for input detail 1"
breaksw
case "18":
cp ${file} ${APPDIR2}
mv ${APPDIR2}/${file} ${APPDIR2}/${file}.dat
echo "copying file for input detail 2"
breaksw
case "19":
cp ${file} ${APPDIR}
mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
echo "copying file for input detail"
breaksw
case "20":
cp ${file} ${APPDIR1}
mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
echo "copying file for input detail 1"
breaksw
case "21":
cp ${file} ${APPDIR2}
mv ${APPDIR2}/${file} ${APPDIR2}/${file}.dat
echo "copying file for input detail 2"
breaksw
case "22":
cp ${file} ${APPDIR}
mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
echo "copying file for input detail"
breaksw
case "23":
cp ${file} ${APPDIR1}
mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
echo "copying file for input detail 1"
breaksw
default:
echo "Problem"
endsw## mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
# compress the files and move them to the archive directory
echo "Compressing and moving input file" >>! ${LOGFILE}compress ${file}
# we have to put the files in the directories based on month and year
if !( -d ${ARCHDIR}${YEAR} ) then
mkdir ${ARCHDIR}${YEAR}
endifswitch ( $MONTH )
case "01":
if !( -d {$ARCHDIR}${YEAR}/JAN ) then
mkdir {$ARCHDIR}${YEAR}/JAN
endif
mv ${file}.Z {$ARCHDIR}${YEAR}/JAN;breaksw
case "02":
if !( -d {$ARCHDIR}${YEAR}/FEB ) then
mkdir {$ARCHDIR}${YEAR}/FEB
endif
mv ${file}.Z {$ARCHDIR}${YEAR}/FEB;breaksw
case "03":
if !( -d {$ARCHDIR}${YEAR}/MAR ) then
mkdir {$ARCHDIR}${YEAR}/MAR
endif
mv ${file}.Z {$ARCHDIR}${YEAR}/MAR;breaksw
case "04":
if !( -d {$ARCHDIR}${YEAR}/APR ) then
mkdir {$ARCHDIR}${YEAR}/APR
endif
mv ${file}.Z {$ARCHDIR}${YEAR}/APR;breaksw
case "05":
if !( -d {$ARCHDIR}${YEAR}/MAY ) then
mkdir {$ARCHDIR}${YEAR}/MAY
endif
mv ${file}.Z {$ARCHDIR}${YEAR}/MAY;breaksW
case "06":
if !( -d {$ARCHDIR}${YEAR}/JUN ) then
mkdir {$ARCHDIR}${YEAR}/JUN
endif
mv ${file}.Z {$ARCHDIR}${YEAR}/JUN;breaksw
case "07":
if !( -d {$ARCHDIR}${YEAR}/JUL ) then
mkdir {$ARCHDIR}${YEAR}/JUL
endif
mv ${file}.Z {$ARCHDIR}${YEAR}/JUL;breaksw
case "08":
if !( -d {$ARCHDIR}${YEAR}/AUG ) then
mkdir {$ARCHDIR}${YEAR}/AUG
endif
mv ${file}.Z {$ARCHDIR}${YEAR}/AUG;breaksw
case "09":
if !( -d {$ARCHDIR}${YEAR}/SEP ) then
mkdir {$ARCHDIR}${YEAR}/SEP
endif
mv ${file}.Z {$ARCHDIR}${YEAR}/SEP;breaksw
case "10":
if !( -d {$ARCHDIR}${YEAR}/OCT ) then
mkdir {$ARCHDIR}${YEAR}/OCT
endif
mv ${file}.Z {$ARCHDIR}${YEAR}/OCT;breaksw
case "11":
if !( -d {$ARCHDIR}${YEAR}/NOV ) then
mkdir {$ARCHDIR}${YEAR}/NOV
endif
mv ${file}.Z {$ARCHDIR}${YEAR}/NOV;breaksw
case "12":
if !( -d {$ARCHDIR}${YEAR}/DEC ) then
mkdir {$ARCHDIR}${YEAR}/DEC
endif
mv ${file}.Z {$ARCHDIR}${YEAR}/DEC;breaksw
default:
echo "There was an error generating the month and the files have been placed"
echo " in the " ${ARCHDIR} " directory "
mv ${file}.Z ${ARCHDIR}
endswelse
if !( -d ${ARCHDIR}${YEAR}/INCOMPLETE ) then
mkdir ${ARCHDIR}${YEAR}/INCOMPLETE
endif
mv ${file}.Z ${ARCHDIR}${YEAR}/INCOMPLETE;breakswendif
end
endif
sleep 50
end
-
Looks ugly, but it also looks like a bash script and I've never seen a bash script that wasn't ugly.
-
This is way out of my specialty, but the first thing that popped out for me was the hour switch block. It doesn't include midnight, right?
-
@mott555 said:
Looks ugly, but it also looks like a bash script and I've never seen a bash script that wasn't ugly.
What gave it away as a bash script? Was it the line reading "#!/bin/csh" at the top?
-
but there was a (not-so?) subtle bug that went unseen for a number of years which was fixed last year-ish (and I didn't include the fix.) Can you find it?
Was it "You forgot to write the log file at midnight" ? Or was it "You wrote a script in csh" ?
-
@zelmak said:
It's got a Wilson Pickett bug!I could have sworn I'd already posted this, but I went back and looked and couldn't find it ... I transcribed and sanitized on-the-fly, so any typo's are likely my fault. Not only is this ugly, but there was a (not-so?) subtle bug that went unseen for a number of years which was fixed last year-ish (and I didn't include the fix.) Can you find it?
-
Is it that you're copying each file to the month? Looks like you copy file1.Z to JAN and then you'd copy file2.Z to JAN as well.
but that's a quick look-see, it's lunch time and my vendor's just brought me a sandwich for doing alarm testing.
-
@Rootbeer said:
@mott555 said:
Wrong meaning of the phrase "looks like", genius. He literally meant it looks like what it is, not that he's guessing that it's a bash script.Looks ugly, but it also looks like a bash script and I've never seen a bash script that wasn't ugly.
What gave it away as a bash script? Was it the line reading "#!/bin/csh" at the top?
-
@Zylon said:
@Rootbeer said:
@mott555 said:
Wrong meaning of the phrase "looks like", genius. He literally meant it looks like what it is, not that he's guessing that it's a bash script.Looks ugly, but it also looks like a bash script and I've never seen a bash script that wasn't ugly.
What gave it away as a bash script? Was it the line reading "#!/bin/csh" at the top?Except that it's NOT a bash script.
Or am I just missing some really obvious irony in these comments?
-
Hell if I know. I don't do *nix.
-
@Zylon said:
If that's what he meant then he worded it confusingly. That second clause, joined by "but", looks like it's going to be some kind of explanation for or justification of why it "looks ugly". If it's not mott's assumption that it was a bash script, why would it be relevant that all bash scripts are ugly? Why would anyone write "It looks ugly, but it also looks ugly, because it looks like something that is always ugly" on purpose? The "It looks ugly, but it has to because it is an example of something that is always ugly" interpretation seemed more obvious to me.@Rootbeer said:
@mott555 said:
Wrong meaning of the phrase "looks like", genius. He literally meant it looks like what it is, not that he's guessing that it's a bash script.Looks ugly, but it also looks like a bash script and I've never seen a bash script that wasn't ugly.
What gave it away as a bash script? Was it the line reading "#!/bin/csh" at the top?
-
-
@Zylon said:
Hell if I know. I don't do *nix.
It's a shell script, but not a bash script.
#!/some_path/sh == bourne SHell. The "lowest common denominator", de facto, always-there *nix shell for a long time.
#!/some_path/bash == Bourne Again SHell. Seems to be the most popular nowadays. Based on bourne shell, not surprisingly.
#!/some_path/ksh == Korn SHell. Another bourne shell derivative that was popular in the 90s. #!/some_path/csh == C SHell. The one used in the OP. A shell with somewhat more C-like syntax than the bourne family. Next to plain ol' bourne, was almost a guarantee to be present back in the day.
#!/some_path/tcsh == Some variation on csh that I know nothing about.Fascinating, wot?
-
I notice the switch only accounts for 23 hours out of the day.
@zelmak said:
echo "Problem"
I should say so.
-
Looks like the entirety of the first switch statement can be replaced by an array and a modulus. The second switch statement with just an array for the months.
Of course, if you want to be *really* sure it wont break anything you'd check that the numbers are in the same range first.
-
@zelmak said:
#!/bin/rm -f
FTFY
@zelmak said:
This is a race condition, isn't it?set MONTH = `date +%m`
set HOUR = `date +%H`
set YEAR = `date +%Y`
-
One might add that ksh was the standard shell on most Unixes, bash took over because it was the GNU version so became standard in the Linux world, and csh is an evil mess that should never have been spawned (and I still know people who use it every day, poor buggers).
There's also zsh, whick is the Dvorak keyboard / Ron Paul of shells.
Yes, I am this boring in real life too.
-
@fatbull said:
So that's why it omits case "00"! It avoids the race!@zelmak said:
#!/bin/rm -f
FTFY
@zelmak said:
This is a race condition, isn't it?set MONTH = `date +%m`
set HOUR = `date +%H`
set YEAR = `date +%Y`
-
@Iago said:
One might add that ksh was the standard shell on most Unixes
IIRC, sh was the only one that was guaranteed to always be present, at least up until Linux became popular. I think Linux still always has sh, but it seems it's often just a link to bash.
There's also zsh, whick is the Dvorak keyboard / Ron Paul of shells.
Yes, I am this boring in real life too.
I can't believe I forgot zsh, my shell of choice. It's very similar to bash, but seems to have a few additional features
-
@jverd said:
IIRC, sh was the only one that was guaranteed to always be present, at least up until Linux became popular. I think Linux still always has sh, but it seems it's often just a link to bash.
It is indeed often a symlink to bash. I think it was Ubuntu which started symlinking it to dash instead (and broke a ton of poorly written scripts which started #!/bin/sh but assumed bash). And in some cases it's a symlink to busybox.
-
@jverd said:
@Iago said:
One might add that ksh was the standard shell on most Unixes
IIRC, sh was the only one that was guaranteed to always be present, at least up until Linux became popular. I think Linux still always has sh, but it seems it's often just a link to bash.
There's also zsh, whick is the Dvorak keyboard / Ron Paul of shells.
Yes, I am this boring in real life too.
I can't believe I forgot zsh, my shell of choice. It's very similar to bash, but seems to have a few additional features
Actually, it is closer to ksh than to bash, and I love it. I am using maybe 5% of the features, but it's way more practical than bash. :)
-
@bannedfromcoding said:
Actually, it is closer to ksh than to bash, and I love it. I am using maybe 5% of the features, but it's way more practical than bash. :)
^- the reason you'll never see usable software from the *nix crowd.
-
@blakeyrat said:
@bannedfromcoding said:
Actually, it is closer to ksh than to bash, and I love it. I am using maybe 5% of the features, but it's way more practical than bash. :)
^- the reason you'll never see usable software from the *nix crowd.
I thought it's kinda obvious, but since a while I don't write software. ;)
-
@Iago said:
There's also zsh, whick is the Dvorak keyboard / Ron Paul of shells.
I'm not sure whether this comparison is more insulting to Dvorak keyboards, Ron Paul or zsh.
-
@Weng said:
It simultaneously insults Dvorak keyboards more than Ron Paul, Ron Paul more than zsh, and zsh more than Dvorak keyboards!@Iago said:
There's also zsh, whick is the Dvorak keyboard / Ron Paul of shells.
I'm not sure whether this comparison is more insulting to Dvorak keyboards, Ron Paul or zsh.
-
Seeing as the thread is titled 'the horror of the mover'...
set LOGDIR = /production/logs/
set LOGFILE = ${LOGDIR}/mover.log
How does *nix handle double slashes on a filename?
-
@cmccormick said:
Seeing as the thread is titled 'the horror of the mover'...
No problems at all in the middle of a pathname. '//' at the front of a path is reserved and may have any special meaning the implementation wants to use it for, but in the middle it's just an empty path component.
set LOGDIR = /production/logs/
set LOGFILE = ${LOGDIR}/mover.log
How does *nix handle double slashes on a filename?
-
Leaving out the question of using csh at all, or the inadvisability of using shell scripts for any but the most trivial tasks (or where, for protability it's unavoidable - configure for example), since this script will break with file names containing spaces or some shell metacharacters, I see the following:
It's syntactically invalid - there's a switch block after the endsw in the second block. There's an endif when the script is in a foreach on the following line.
It appears it's supposed to run every 50 seconds, so race conditions when the various date fields change aren't all that unlikely.
Why would anyone use unalias rm, unalias mv instead of the safe approach of using full paths?
Why would anyone set all the .SOURCE files to be executable? Or world writeable for that matter?
If the system it's running on is running on a localtime including daylight savings, it will dump two hours worth of backups in one directory
One wonders if there really is a need to copy the files into appdir then mv them as opposed to simply copying them to the final file name
What happens between midnight and 1 AM?
The script is very noisy with 'copying file for input detail ...' without actually giving useful information, like what file is being copied.
The same quality of error reporting is shown by outputting 'Problem' when the copy to the app directory
There's probably more wrong here.
But it makes a good case for code reviews, I'd hate to be the author facing my colleagues to explain this one
-
@jes said:
No there isn't.It's syntactically invalid - there's a switch block after the endsw in the second block.
@jes said:
There's an endif when the script is in a foreach on the following line.
That thing you thought was a spurious switch block is in fact a spurious else..endif block, if you read it again. The structure of the code looks like this:while 1
if ($status == 0) then
foreach file (*.SOURCE)
[ ... stuff ... ]
[ first switch block ]
[ ... stuff ... ]
[ second switch block ]
[ spurious else..endif block]
end # foreach end
endif # if $status end
sleep 50
end # while 1 end
-
The bug was, in-fact, the missing '00' hour check. But, even tho the fact that something odd had occurred was being logged for OVER SIX YEARS, no one noticed that we never had any data for that timeframe. When it was discovered (someone was doing some data-trending pulls from the DB and noticed that the 'zero-hour' had no data going ways back) the powers-that-be shrugged, said, put in the last six months worth, and we'll be okay. Yet, we still have data in the system dating back to 2003 so 'all the data is important' ... except when its not.
The reason for the cp and then subsequent mv was to stave off (yet another?) race-condition ... where the file copy would be incomplete, but the program reading it would start reading it anyway. Since the input program only reads .dat files, the cp can complete, then the mv will make it 'readable' by the input program.
The whole date thing? Meh.
But why on earth would you go to such ends as to change a perfectly acceptable (and sortable) numeric month value and change it to the English abbreviation for it?
And those hugemongous switch statements ... ugh ...
And dumping all input from an hour into one directory? We have three 'data loaders'; one reads its own directory. So, one directory fills up, the others remain idle. Yay. Why do we have three again?
-
@jes said:
Why would anyone use unalias rm, unalias mv instead of the safe approach of using full paths?
Because they want to use $PATH to search for them?
The intention behind unalias is to remove the alias, not to fix which binary is being used. (rm is commonly aliased as 'rm -i' for example.)
-
@PJH said:
The intention behind unalias is to remove the alias, not to fix which binary is being used. (rm is commonly aliased as 'rm -i' for example.)Which only helps if rm is aliased (as opposed to being a shell function or a script or executable ahead of /bin in the path). I can't imagine why the author would be unaliasing other than to make sure he knew exactly what the effect of executing rm would be, and the only way to do that is to just explicitly invoke /bin/rm. (And even that relies on an assumption, albeit a safer one that what you get with a simple unalias.)
-
@blakeyrat said:
@bannedfromcoding said:
Actually, it is closer to ksh than to bash, and I love it. I am using maybe 5% of the features, but it's way more practical than bash. :)
^- the reason you'll never see usable software from the *nix crowd.
Taking "usable" in the context of software usability, which I believe is how you generally use it, as opposed to meaning "there are people who manage to use this software to accomplish its intended purpose," I still think you're looking at two effects, not a cause and effect. But I do agree that they're related. </pd>
-
@jverd said:
@PJH said:
Which is the intention.
Which only helps if rm is aliased
The intention behind unalias is to remove the alias, not
to fix which binary is being used. (rm is commonly aliased as 'rm -i' for
example.)
@jverd said:(as opposed to being a shell function or a script or executable ahead of /bin in
-i requires keyboard input. For an unattended script, you don't require someone sitting at the keyboard. unaliasing ensures that -i isn't on the command line.
the path). I can't imagine why the author would be unaliasing other than to make
sure he knew exactly what the effect of executing rm would be, and the only way
to do that is to just explicitly invoke /bin/rm. (And even that relies on an
assumption, albeit a safer one that what you get with a simple unalias.)
-
Unaliasing doesn't ensure that isn't a "-i" on the command line.
Even using the full path doesn't ensure that, there is no way to ensure it. But using the full path removes a lot of other ways the "-i" could get there.
-
@Mcoder said:
Apart from aliasing, how else are you postulating a -i could get in the way?Unaliasing doesn't ensure that isn't a "-i" on the command line.
Even using the full path doesn't ensure that, there is no way to ensure it. But using the full path removes a lot of other ways the "-i" could get there.
Seriously, I'm interested as to how, in a script, a stray -i (or any other bollocks an alias could introduce) could be introduced. And why fixing the path to the binary could remove it where an unalias couldn't.
Assume, humour me, that the system in question isn't under 'attack'.
-
@PJH said:
Apart from aliasing, how else are you postulating a -i could get in the way?
I am equally baffled. I think the postulate is "Nothing is for sure in Linux because Linux is hard; let's go shopping."