Description of PIPEDDR
Download count: 12 this month, 6880 altogether.
Downloads for PIPEDDR:
VMARC archive: v-97K
zip archive: z-100K
PIPEDDR EXEC
The PIPEDDR EXEC can dump a disk into a regular CMS file, to an ftp server, to a file in an NFS share (which is mounted to BFS, the CMS Byte File System), or via TCP/IP directly to a disk on a remote system. Restore a disk from a PIPEDDR output file via one of the same methods. The data is packed or can also be compressed further, saving network bandwidth or disk space. The data can also be encrypted either on disk or as it is sent over the network. A remote disk can be kept in sync with a local disk using the remote update function.
The PIPEDDR EXEC can save and restore disk images using either the raw disk I/O support available in CMS Pipelines or the Pipelines interface to the CMS DDR command. The internal structure of each kind of disk image are different and are not compatible with each other. The Pipelines raw disk I/O support uses the Pipe stages TRACKREAD and TRACKWRITE (for ECKD disks) or FBAREAD and FBAWRITE (for FB-512 disks.) These stages are part of either z/VM® 6.4 and later or the optional "Runtime Distribution" of CMS Pipelines. The interface to the CMS DDR command uses the DDR Pipelines stage and an updated DDR module. The updated module and Pipelines stage are included with z/VM 6.2 and later. If you are on an older release of z/VM, then you can get the required files from the DRPC package in the VM download library.
The exec dumps an entire disk into a regular CMS file. The operating system format of the disk or the type of data on it does not matter. The disk is read in binary track by track or block by block. The output data by default is written in packed format compatible with the PACK option of the CMS COPYFILE command and is in fixed record format with 1024 byte records. This allows easier interchange of the backup files with non-mainframe systems. These files can be used for backups of disks or restored on another system. There are also options to compress the output to make the file even smaller. See the notes section of the help file for more information on compressing the output.
The output file created by this exec has meta data about the source disk as the first record. The contents of the disk follow in the rest of the records. This ensures that the disk is restored to a device of the same type and size. Data must be restored to the same type of device using the same disk data encoding method. In other words, ECKD devices (3390) must be restored to ECKD devices and FB-512 devices must be restored to FB-512 devices. It is not possible to convert the data on one device type to another device type using this exec.
The first record also contains data about the id that dumped the disk, what kind of compression was used, if a data verification code such as a CRC or digest value is included, and if the disk data is encrypted. This allows the exec to correctly restore the data.
The disk can also be transferred over a TCP/IP connection directly to a receiving PIPEDDR exec listening to an IP port on a remote system. The default is to send the entire disk. If the disk was transferred previously, a PIPEDDR option will send just the changes to the blocks or tracks to the remote system instead of the entire disk. Recent updates to CMS Pipelines and TCPIP allow optional secure (TLS) remote sessions. It is recommended that the same version of the PIPEDDR exec be used on both ends of the TCP/IP connection. PIPEDDR also supports sending the output file directly to an ftp or ftps (secure ftp) server. The file created on the ftp server is in the same format as the CMS file. A disk can be restored from a file hosted by either an ftp server or an HTTP (web) server. Secure connections are supported.
Byte File System (BFS) paths are also supported, which allows the file of the disk data to be hosted by a remote NFS server. You must mount the remote NFS share into the BFS filesystem to use this support. Please see the usage note in the help file about BFS and NFS for more information.
Dump to and restore from a tape is supported by using the TAPE output method and the same operations using a CMS filedef using the FILEDEF output method.
By default, PIPEDDR will display messages on the console about the progress of the disk operation as it processes a disk. The default behavior is to display a message for every 10% of disk that has been processed. No messages are displayed for small disks and more messages may be displayed for larger disks. The display can be suppressed by using the NOTYPE option or changing the default using the PIPEDDR SET command.
If the disk contains sensitive data, the disk contents can be encrypted, using the CPACF hardware encryption processor. You must obtain the KM Package from the VM download library to enable encryption. The KMRTNS CSLLIB is used from the VMARC file. These CSL routines require your machine to have the CPACF feature enabled to use the built in hardware encryption and decryption.
Dumping or restoring to FTP supports 3 different FTP pipeline stages. The newest ftp stage is built in to Pipelines and is available if secure TCP/IP connections are supported by CMS Pipelines. (APAR VM66365 for z/VM 7.1, included in z/VM 7.2 and later.) This stage supports both secure and unsecured ftp sessions. The other ftp stages supported by PIPEDDR are not built-in to Pipelines; they are loaded from a separate module or file. The preferred external ftp stage is found in the FTPREXX package on the VM downloads page. Get the FTPREXX Package to download this stage.
The final one is supplied with z/VM, but only supported for the installation of z/VM. It requires either the INSTPIPE MODULE from a modern version of CMS (found on MAINTvrm 4CC or MAINT 193) or the DRPC MODULE.
The exec also supports two other forms of compaction, TERSE and ZLIB. The TERSE option can only be used if the TERSE Pipelines stage is available. This allows the exec to create output files that are smaller or send less data over the network. The TERSE Pipelines stage is part of the PIPSYSF filter package which is now available from the Marist Pipelines page. See the PIPSYSF Filter Package page for more information and a link to download the module.
The ZLIB compression option requires the zlib Pipeline stage which is found in in the ZLIBSTG MODULE file. You can download the file "fplgcc.vma" which is a VMARC format file downloadable from John Hartmann's space on GitHub.
If the CMS DDR format is used, the DDR module supports 2 compression options, COMPRESS and LZCOMPACT, which may provide an acceptable level of compression.
There is also a copy option which preforms a disk to disk copy, with automatic linking of the source and target disks. If the flashcopy option is specified, the exec attempts to copy the disks using the CP FLASHCOPY command. If that command fails or your user id is not allowed to use this command, the exec falls back to a disk to disk copy. The copy option is no different than copying a disk using DDR, and in fact just the DDR module is used if the CMSDDR option is specified on the copy command.
A help file is included in the package and it contains a detailed description of all of the options. Enter HELP PIPEDDR on the CMS command line for usage information. Be sure to also download the zip file which contains a pdf document. This document includes all of this description, all of the help file contents, plus additional details.
Feedback: Bruce Hayden IBM Z Washington Systems Center
Some examples of how to use PIPEDDR.
To dump a minidisk to a file with the default name:
PIPEDDR DUMP MAINT 19EThis creates a file named MAINT DISK019E A
To dump a minidisk to a file and terse the output:
PIPEDDR DUMP MAINT 19E (TERSEAnother way is to set the default compression format to TERSE and then dump the disk:
PIPEDDR SET COMPACT TERSE PIPEDDR DUMP MAINT 19E
To dump a minidisk to a file including the CRC:
PIPEDDR DUMP MAINT 19E (CRC
To dump the same disk to a different filemode:
PIPEDDR DUMP MAINT 19E TO FILE = = E
To dump a minidisk to a file in CMS DDR format:
PIPEDDR DUMP MAINT 19E (CMSDDR
To restore the file to the same disk:
PIPEDDR RESTORE MAINT 19E
To restore the file to a different disk and skip the prompt:
PIPEDDR RESTORE CMSUSER 191 FROM FILE MAINT DISK019E A (NOPROMPT
To dump a minidisk to a file and encrypt the data:
PIPEDDR DUMP MAINT 19E (CIPHER AES KEY(Super Secret KEY)
To restore it:
PIPEDDR RESTORE MAINT 19E (CIPHER AES KEY(Super Secret KEY)
To send an entire minidisk over the network On the receiving node, enter:
PIPEDDR RESTORE MAINT 19E FROM REMOTE
This will display the port number it is using. To force the port to 12345:
PIPEDDR RESTORE MAINT 19E FROM REMOTE (PORT 12345
On the sending system use (where pppp is the listening port):
PIPEDDR DUMP MAINT 19E TO REMOTE nodeid.example.com:pppp
To just update a disk that was copied previouslly:
PIPEDDR DUMP MAINT 19E TO UPDATE nodeid.example.com:ppppThe connection should be made and the disk sent over.
To send a minidisk to a file named maint.disk019e on an ftp server:
PIPEDDR DUMP MAINT 19E TO ftp://hayden:password@server.example.com
To restore a minidisk from file disk.dump on an ftp server:
PIPEDDR RESTORE MAINT 19E FROM ftp://hayden:password@server.example.com/disks/disk.dump
To do the same thing but from an http server:
PIPEDDR RESTORE MAINT 19E FROM http://server.example.com/disks/disk.dump
To dump a minidisk to a file named maint.disk019e on an NFS server, first the nfs directory must be mounted to your virtual machine using OPENVM MOUNT. Assuming it is mounted at /home/maint/mnt, the command is:
PIPEDDR DUMP MAINT 19E TO BFSPATH /home/maint/mnt
How to verify a file
The CKSUM and DIGEST options are designed to allow a disk image file stored on a Posix compatible platform (such as Linux) can be validated (checked that no change has occurred) on that platform.
The file created by PIPEDDR has a 1024 byte header and a 1024 byte trailer before and after the actual disk image. The header has data in blank delimited words about the input disk in both EBCDIC and ASCII, with an X'0A' byte in between the EBCDIC and ASCII translations. The ASCII translation is terminated with an X'0A' byte. The remaining data of the 1024 byte header is binary zeros.
The trailer contains the verification data which is either the cksum value or the digest value. A cksum value is always 12 bytes, but a digest value has different lengths for each type of digest. The length of the binary digest or cksum value is the twelfth word of the header. Following the binary digest or cksum value is the same value translated to ASCII text. The length of this value is twice the binary length. The remaining data of the 1024 byte trailer is binary zeros.
A simple script, such as this example, shows the ASCII header:
#!/bin/bash echo "Show header (ascii version)" # It is in the 1024 byte header, the second "line" found in there head -c +1024 $1 | cat | head -n 2 | tail -n 1
The digest or cksum of the disk image part of the file can be calculated with one of these scripts:
#!/bin/bash echo "Show cksum of disk contents (decimal cksum and byte count)" # Take the 1024 byte header and trailer off and input to cksum tail -c +1025 $1 | head -c -1024 | cksum #!/bin/bash echo "Show sha256 digest of disk contents" # Take the 1024 byte header and trailer off and input to sha256 tail -c +1025 $1 | head -c -1024 | sha256sum
These examples can be combined into a single script that verifies the disk image has not changed. Here is an example that performs the verification.
#!/bin/bash
# Verify that a disk image created by PIPEDDR has not changed.
file=$1
fileheader=$(head -c +1024 $file | cat | head -n 2 | tail -n 1)
if [ -z "$fileheader" ]; then
echo "File $file is not in the correct format."
exit
fi
echo "Header: $fileheader"
# Break file header into positional parameters
set $fileheader
len=${12}
hostdigest=$(tail -c 1024 $file | tail -c +$(($len+1)) | head -c $((len*2)))
if [ "$8" = "CKSUM" ]; then
verifypgm="cksum"
# Convert the hexadecimal CRC-32 value and length to decimal
hostdigest="$((16#${hostdigest:0:8})) $((16#${hostdigest:9:16}))"
else
verifypgm=$(echo ${11} | tr [:upper:] [:lower:])sum
# Add filename " -" to end to match command output
hostdigest="$hostdigest -"
fi
echo "Computing $verifypgm of the disk contents"
# Take the 1024 byte header and trailer off to compute digest or cksum
localdigest=$(tail -c +1025 $file | head -c -1024 | $verifypgm)
if [ "$hostdigest" != "$localdigest" ]; then
echo $verifypgm "verification failed."
echo "Value from PIPEDDR is:" $hostdigest
echo "Local value is:" $localdigest
else
echo "Digest values match"
fi
exit
These examples should help you create a more complete program to verify disk images that reside on a Posix compatible system.
Change log of recent changes, latest changes first
Version Change Description
------- ------------------
V1.7.6 Add special first option DEFAULTS to ignore all preset options.
V1.7.5 Add SDIGEST (single digest) option which works like crc.
V1.7.4 Allow exec to be called as a Pipeline stage.
V1.7.3 Support new ftp stage which supports ftps.
Add VMFTP option, to send proper SITE command for VM FTP.
V1.7.2 Support secure SSL/TLS sockets for remote dump/restore.
V1.7.1 Fix port number in error message and some spelling.
Validate some of the fields in the file header.
Ensure valid values for crc or digest check.
If no input with oneread, issue error.
RAW option superceeds ONEREAD option.
V1.7.0 Don't restrict NOPACK, and don't force it to be blocked.
Show fewer progress messages with large disks.
Add digest or cksum in ascii after binary value in last record.
For remote receive or update, verify cipher key.
V1.6.13 Add TO UPDATE remote send option, to send incremental updates.
Change the default verification on remote send to Digest
V1.6.12 Rework syntax with TO|FROM, urls, and fewer options needed.
V1.6.11 Allow cipher key and ftp password to be supplied in variables.
V1.6.10 Check for cipher stage support in Pipelines
V1.6.9 Default to FTP REXX for ftp, use old with OLD opt.
Change how the TCP listen pipe is stopped.
V1.6.8 Allow the cipher part of the header to be blank
Misc updates, changes to comments
Remove EAV restriction because of VM66139 on 6.4
V1.6.7 Add STABLE option, for stable RO and exclusive R/W links
V1.6.6 Support flashcopy for copy
V1.6.5 Error dropping short records causing DDR restore to fail
V1.6.4 For dumps with a digest value, also verify original disk data
V1.6.3 Support KM CSL routines instead of Pipe cipher stage
V1.6.2 Support ZLIB compression
V1.6.1 Make it fullscreen friendly and fix other bugs