Automating tasks using JES FTP and Bash
This is a short introduction I wrote about automating some SDSF processes using JES FTP. Although it targets a specific task, the general principle can be applied to lots of other things.
- Existing manual process
- Introduction to JES FTP
- PTF upload script
Existing manual process
Let's take the example of applying a fix pack. These are the current manual steps in the process which have to be repeated 8 times for each of the parts:
- download the part from the build web site
- rename if necessary
- preallocate space on mainframe and then ftp part up
- edit some jcl and run to unterse file
- browse the untersed file to get the sysmod number
- edit and run some jcl to receive the part (needs dataset name and sysmod number)
- edit and run some jcl to apply the part
At each stage we also need to check for errors.
Lots of scope for automation here!
Introduction to JES FTP
- Provides a mechanism for submission of JCL
- Simply "put" the JCL file to submit
- Can use "dir" command to monitor job status
- Use "get" to retrieve job output
Sample JCL to unterse a file
//UNTERSE JOB
//*
//STEPNAME EXEC PGM=TRSMAIN,PARM='UNPACK'
//SYSPRINT DD SYSOUT=*
//INFILE DD DISP=SHR,DSN=DGRIFF.ZN80121.HDM8010.SYSMOD
//OUTFILE DD DISP=(NEW,CATLG),UNIT=SYSDA,
// SPACE=(CYL,(300,30),RLSE),
// DSN=DGRIFF.ZN80121.HDM8010.SYSMOD.UNTRS
//
Results from submitting the job
ftp> site filetype=jes
200 SITE command was accepted
ftp> put unterse.jcl
200 Port request OK.
125 Sending Job to JES internal reader FIXrecfm 80
250-It is known to JES as JOB05366
250 Transfer completed successfully.
531 bytes sent in 0 seconds (531 bytes/s)
Use "dir" to check status
ftp> site filetype=jes JESJOBNAME=UNTERSE JESSTATUS=OUTPUT
200 SITE command was accepted
ftp> dir
200 Port request OK.
125 List started OK for JESJOBNAME=UNTERSE, JESSTATUS=OUTPUT and JESOWNER=DGRIFF
JOBNAME JOBID OWNER STATUS CLASS
UNTERSE JOB05379 DGRIFF OUTPUT A RC=0000 4 spool files
250 List completed successfully.
If the job fails:
ftp> dir
200 Port request OK.
125 List started OK for JESJOBNAME=UNTERSE, JESSTATUS=OUTPUT and JESOWNER=DGRIFF
JOBNAME JOBID OWNER STATUS CLASS
UNTERSE JOB05366 DGRIFF OUTPUT A (JCL error) 3 spool files
250 List completed successfully.
Use "dir jobid" to list the job output
ftp> dir JOB07629
200 Port request OK.
125 List started OK for JESJOBNAME=RECEIVE, JESSTATUS=OUTPUT and JESOWNER=DGRIFF
JOBNAME JOBID OWNER STATUS CLASS
RECEIVE JOB07629 DGRIFF OUTPUT A RC=0008
--------
ID STEPNAME PROCSTEP C DDNAME BYTE-COUNT
001 JES2 A JESMSGLG 1098
002 JES2 A JESJCL 369
003 JES2 A JESYSMSG 2789
004 STEPNAME A SMPOUT 766
005 STEPNAME A SMPRPT 2322
5 spool files
250 List completed successfully.
Use "get jobid.id" to get the job output
ftp> get JOB07629.4
200 Port request OK.
125 Sending data set DGRIFF.RECEIVE.JOB07629.D0000103.?
250 Transfer completed successfully.
1845 bytes received in 0.001 seconds (1844999 bytes/s)
Automating using bash
In the following examples we use bash scripts to automate some of the fix pack process using JES FTP. The scripts are run from your laptop (cygwin in my case).
- Why bash?
Absolutely no reason why this couldn't be done in perl, it just seemed that bash was a better fit for small scripts most of which involve calling other processes. Besides which my perl is a little rusty!
PTF upload script
This script does the following:- rename file
- ftp to mainframe
- unterse
When a PTF is built it comes with a name like "ZN80122_$HDM8011.sysmod" which contains the illegal character "$". If not already renamed, replace the "_$" with "."
if [[ $1 =~ '_$' ]]
then
file=${1//_$/.}
echo renaming $1 to $file
mv $1 $file
else
file=$1
fi
ftpout=`mktemp -u`
function put {
ftp -v -n xxx.hursley.ibm.com > $ftpout <<EOF
quote USER $USER
quote PASS $PASSWORD
$@
quit
EOF
grep -q "250 Transfer completed successfully" $ftpout
if [ $? -ne 0 ]
then
echo "ftp failed:"
while read line
do
# check for common error numbers
if [[ $line =~ 451 || $line =~ 501 || $line =~ 200 $line =~ 530 ]]
then
printf "\t%s\n" "$line"
fi
done < $ftpout
rm $ftpout
exit 1
fi
}
Do the ftp put using the function just defined. The site statement is required to define the correct primary allocation size in terms of the number of cylinders. (A cylinder is approximately 850k bytes). Without this statement the put would fail with an out of space message if the file size exceeded the default.
size=`stat -c '%s' $file`
((numcyls = size / 850000 + 100))
put "site CYL LR=1024 REC=FB BLOCKSI=0 PRI=$numcyls SEC=200
bin
put $file"
echo "ftp completed"
Create a temporary file containing the jcl to unterse the file. The "^^" is a bash mechanism to convert the filename to uppercase.
jcl=`mktemp -u`
cat > $jcl <<EOF
//UNTERSE JOB
//*
//STEPNAME EXEC PGM=TRSMAIN,PARM='UNPACK'
//SYSPRINT DD SYSOUT=*
//INFILE DD DISP=SHR,DSN=DGRIFF.${file^^}
//OUTFILE DD DISP=(NEW,CATLG),UNIT=SYSDA,
// SPACE=(CYL,(300,30),RLSE),
// DSN=DGRIFF.${file^^}.UNTRS
//
EOF
Do an ftp put on the jcl. The site command is the magic bit - this tells ftp to submit the jcl as a JES job. Then parse the output to obtain the jobid for later use.
put "site filetype=jes
put $jcl"
jobid=`grep -o 'JOB[0-9]\+' $ftpout`
Wait for the job to complete and check the return code. Does a "dir" command to list the job spool output after doing a "site" command to limit the results to jobs that have completed with a job name of "UNTERSE". If "RC=0000" is found in the output then the job succeeded.
while true
do
ftp -v -n xxx.hursley.ibm.com > $ftpout <<EOF
quote USER $USER
quote PASS $PASSWORD
site filetype=jes JESJOBNAME=UNTERSE JESSTATUS=OUTPUT
dir
quit
EOF
grep -q $jobid $ftpout
if [ $? -ne 0 ]
then
echo "unterse running, wait ten seconds..."
sleep 10
else
line=`grep $jobid $ftpout`
if [[ $line =~ "RC=0000" ]]
then
echo "unterse complete"
else
echo "unterse failed: " $line
fi
break
fi
done
This is a handy script which gets the sysmod number from a tersed part on your laptop. You can then use the sysmod number in the receive script for instance. It uses the Java Terse command which can be found in my old svcdump.jar (email me if you are interested in a copy).
head=`mktemp --tmpdir=. -u`
# Don't need to unterse the whole file!
head -c 100 $1 > $head
# Unterse this fragment using Java Terse
java -cp `cygpath -w ~/bin/svcdump.jar` com.ibm.jvm.terse.Terse $head $head.untrs
# Extract the sysmod number - start of file
# looks like "++ USERMOD(AN80124) /* 5655Y1700 */"
grep -o '(\w*)' $head.untrs | head -1 | grep -o '\w*'
rm $head $head.untrs
Receive script
This script does the SMP/E RECEIVE for the given part. We use the sysmod script to get the sysmod number, submit the jcl to do the RECEIVE and then wait for the job to complete.
jcl=`mktemp -u`
ftpout=`mktemp -u`
file=$USER.$1.UNTRS
sysmod=`getsysmod $1`
. ~/bin/ftpsubs
echo receive $file $sysmod
# create the jcl to do the receive
cat > $jcl <<EOF
//RECEIVE JOB
//STEPNAME EXEC PGM=GIMSMP,REGION=64M
//SMPCSI DD DISP=SHR,DSN=DGRIFF.WODMV801.SMPE.GLOBAL.CSI
//SMPPTFIN DD DISP=SHR,DSN=${file^^}
//SYSPRINT DD SYSOUT=*
//SMPCNTL DD *
SET BDY(GLOBAL) .
RECEIVE SELECT( $sysmod ) SYSMODS .
/*
EOF
# submit the jcl to do the receive
put "site filetype=jes
put $jcl"
jobid=`grep -o 'JOB[0-9]\+' $ftpout`
# helper function to get job queue output
function get {
ftp -v -n xxx.hursley.ibm.com > $ftpout <<EOF
quote USER $USER
quote PASS $PASSWORD
site filetype=jes JESJOBNAME=RECEIVE JESSTATUS=OUTPUT
$@
EOF
}
# wait for the receive to complete
while true
do
get "dir"
grep -q $jobid $ftpout
if [ $? -ne 0 ]
then
echo "receive running, wait ten seconds..."
sleep 10
else
line=`grep $jobid $ftpout`
if [[ $line =~ "RC=0000" ]]
then
echo "receive complete"
else
echo "receive failed: " $line
# get the index of the SMPOUT listing
get "dir $jobid"
index=`grep SMPOUT $ftpout | cut -c12-13`
# get the SMPOUT and grep for error lines
smpout=`mktemp -u`
get "get $jobid.$index $smpout"
grep 'GIM.....E' $smpout
rm $smpout
exit 1
fi
break
fi
done
This is what the output looks like when the job fails:
$ rec ZN80121.HDM8010.sysmod
receive dgriff.ZN80121.HDM8010.sysmod.UNTRS AN80121
receive running, wait ten seconds...
receive failed: RECEIVE JOB08242 DGRIFF OUTPUT A RC=0008 5 spool files
GIM38601E ** SYSMOD AN80121 WAS NOT RECEIVED BECAUSE IT HAS ALREADY BEEN RECEIVED.
GIM39001E ** SYSMOD AN80121 WAS NOT RECEIVED EVEN THOUGH IT WAS SPECIFIED ON THE SELECT OPERAND.
Putting it together
Having got some mini scripts to do various things, we can now combine them into one larger script that does a sequence of steps with no user intervention, freeing us up to go and have a cup of tea. For this we do "set -e" which tells bash to exit at the first error.
set -e
rec ZN80121.HDM8010.sysmod
rec ZN80122.HDM8011.sysmod
rec ZN80123.JDM8012.sysmod
rec ZN80124.HDM8013.sysmod
rec ZN80125.HDM8013.sysmod
rec ZN80126.JDM8014.sysmod
rec ZN80127.JDM8014.sysmod
rec ZN80128.JDM8014.sysmod
