Description of PIPEDDR
Download count: 13 this month, 6611 altogether.
Downloads for PIPEDDR:
VMARC archive: v-83K
zip archive: z-71K
PIPEDDR EXEC
The PIPEDDR EXEC can dump a disk into a regular CMS file, to an ftp server, to a file in an NFS share (which is mounted to BFS, the CMS Byte File System), or via TCP/IP directly to a disk on a remote system. Restore a disk from a PIPEDDR output file via one of the same methods. The data is packed or can also be compressed further, saving network bandwidth or disk space. The data can also be encrypted either on disk or as it is sent over the network.
The PIPEDDR EXEC can save and restore disk images using either the raw disk I/O support available in the updated Pipelines runtime library (part of z/VM 6.4 and later) or the Pipelines interface to the CMS DDR command. The internal structure of each kind of disk image are different and are not compatible with each other. The Pipelines raw disk I/O support uses the Pipe stages TRACKREAD and TRACKWRITE (for ECKD disks) or FBAREAD and FBAWRITE (for FB-512 disks.) These stages are part of either z/VM® 6.4 and later or the optional "Runtime Distribution" of CMS Pipelines. The interface to the CMS DDR command uses the DDR Pipelines stage and an updated DDR module. The updated module and Pipelines stage are included with z/VM 6.2 and later. If you are on an older release of z/VM, then you can get the required files from the DRPC package in the VM download library.
The exec dumps an entire disk into a regular CMS file. The operating system format of the disk or the type of data on it does not matter. The disk is read in binary track by track or block by block. The output data by default is written in packed format compatible with the PACK option of the CMS COPYFILE command and is in fixed record format with 1024 byte records. This allows easier interchange of the backup files with non-mainframe systems. These files can be used for backups of disks or restored on another system. There are also options to compress the output to make the file even smaller. See the notes section of the help file for more information on compressing the output.
The output file created by this exec has meta data about the source disk as the first record. The contents of the disk follow in the rest of the records. This ensures that the disk is restored to a device of the same type and size. Data must be restored to the same type of device using the same disk data encoding method. In other words, ECKD devices (3390) must be restored to ECKD devices and FB-512 devices must be restored to FB-512 devices. It is not possible to convert the data on one device type to another device type using this exec.
The first record also contains data about the id that dumped the disk, what kind of compression was used, if a data verification code such as a CRC or digest value is included, and if the disk data is encrypted. This allows the exec to correctly restore the data.
The disk can also be transferred over a TCP/IP connection directly to a receiving PIPEDDR exec listening to an IP port on a remote system. It is recommended that the same version of the PIPEDDR exec be used on both ends of the TCP/IP connection. PIPEDDR also supports sending the output file directly to an ftp server. The file created on the ftp server is in the same format as the CMS file. A disk can be restored from a file hosted by either an ftp server or an HTTP (web) server. Encrypted (TLS/SSL) sessions are not supported.
Byte File System (BFS) paths are also supported, which allows the file of the disk data to be hosted by a remote NFS server. You must mount the remote NFS share into the BFS filesystem to use this support. Please see the usage note in the help file about BFS and NFS for more information.
Dump to and restore from a tape is supported by using the TAPE output method and the same operations using a CMS filedef using the FILEDEF output method.
By default, PIPEDDR will display messages on the console about the progress of the disk operation as it processes a disk. The default behavior is to display a message for every 10% of disk that has been processed. No messages are displayed for small disks and more messages may be displayed for larger disks. The display can be suppressed by using the NOTYPE option or changing the default using the PIPEDDR SET command.
If the disk contains sensitive data, the disk contents can be encrypted, using the CPACF hardware encryption processor. You must obtain the KM Package from the VM download library to enable encryption. The KMRTNS CSLLIB is used from the VMARC file. These CSL routines require your machine to have the CPACF feature enabled to use the built in hardware encryption and decryption.
The raw disk I/O interface requires either z/VM 6.4 and later or the Princeton Runtime Distribution level of Pipelines at least version 110B0004 (15 May 2002 level) for reading and writing ECKD disks and level 110C0004 (01 Jul 2010 level) or later for reading and writing FBA disks. Note that version 110C000A (24 Jul 2012 level) or later is required for writing 3390 disks larger than 32767 cylinders due to a bug in older levels of the trackwrite stage. PIPEDDR uses the PICKPIPE EXEC to load the runtime distribution level of Pipelines if it is needed and if it is available. PICKPIPE is also available via the VM download library. See the PICKPIPE description or get the VMARC file.
The Pipelines interface to CMS DDR requires either z/VM 6.2 or later or the DRPC and DDR modules from the DRPC package on the VM download library. See the DRPC description or get the VMARC file to obtain this function. The updated level of CMS Pipelines is not required for CMS DDR support.
Dumping or restoring to FTP supports 2 different FTP pipeline stages. The first is supplied with z/VM, but only supported for the installation of z/VM. It requires either the INSTPIPE MODULE from a modern version of CMS (found on MAINTvrm 4CC or MAINT 193) or the DRPC MODULE. See the reference to the DRPC MODULE in the previous paragraph. The other FTP stage is found in the FTPREXX package on the VM downloads page. This is the recommended stage to use with PIPEDDR. Get the FTPREXX Package to download this stage.
Transfers to and from FTP will use a NETRC DATA file stored anywhere in the search order. This file can supply a password or a userid and password for the connection, so that the password would not be hardcoded into an exec or displayed on the console.
The exec also supports 2 other forms of compaction, TERSE and ZLIB. The TERSE option can only be used if the TERSE Pipelines stage is available. This allows the exec to create output files that are smaller or send less data over the network. The TERSE Pipelines stage is part of the PIPSYSF filter package which is now available from the Marist Pipelines page. See the PIPSYSF Filter Package page for more information and a link to download the module.
The ZLIB compression option requires the zlib Pipeline stage which is found in in the ZLIBSTG MODULE file. You can download the file "fplgcc.vma" which is a VMARC format file downloadable from John Hartmann's space on GitHub. See the HELP file for more information about extracting the correct file to enable you to use this option.
If the CMS DDR format is used, the DDR module supports 2 compression options, COMPRESS and LZCOMPACT, which may provide an acceptable level of compression.
There is also a copy option, which does a disk to disk copy, with automatic linking of the source and target disks. If the flashcopy option is specified, the exec attempts to copy the disks using the CP FLASHCOPY command. If that command fails or your user id is not allowed to use this command, the exec falls back to a disk to disk copy. The copy option is no different than copying a disk using DDR, and in fact just the DDR module is used if the CMSDDR option is specified on the copy command.
A help file is included in the package and it contains a detailed description of all of the options. Enter HELP PIPEDDR on the CMS command line for usage information. Be sure to also download the zip file which contains a pdf document. This document includes all of this description, all of the help file contents, plus additional details.
Feedback: Bruce Hayden IBM Z Washington Systems Center
Some examples of how to use PIPEDDR.
To dump a minidisk to a file with the default name:
PIPEDDR DUMP MAINT 19EThis creates a file named MAINT DISK019E A
To dump a minidisk to a file and terse the output:
PIPEDDR DUMP MAINT 19E (TERSEAnother way is to set the default compression format to TERSE and then dump the disk:
PIPEDDR SET COMPACT TERSE PIPEDDR DUMP MAINT 19E
To dump a minidisk to a file including the CRC:
PIPEDDR DUMP MAINT 19E (CRC
To dump the same disk to a different filemode:
PIPEDDR DUMP MAINT 19E TO FILE = = E
To dump a minidisk to a file in CMS DDR format:
PIPEDDR DUMP MAINT 19E (CMSDDR
To restore the file to the same disk:
PIPEDDR RESTORE MAINT 19E
To restore the file to a different disk and skip the prompt:
PIPEDDR RESTORE CMSUSER 191 MAINT DISK019E A (NOPROMPT
To dump a minidisk to a file and encrypt the data:
PIPEDDR DUMP MAINT 19E (CIPHER TDES KEY(Super Secret KEY)
To restore it:
PIPEDDR RESTORE MAINT 19E (CIPHER TDES KEY(Super Secret KEY)
To send an entire minidisk over the network On the receiving node, enter:
PIPEDDR RESTORE MAINT 19E FROM REMOTE
This will display the port number it is using. To force the port to 12345:
PIPEDDR RESTORE MAINT 19E FROM REMOTE (PORT 12345
On the sending system use (where pppp is the listening port):
PIPEDDR DUMP MAINT 19E TO REMOTE nodeid.example.com:ppppThe connection should be made and the disk sent over.
To send a minidisk to a file named maint.disk019e on an ftp server:
PIPEDDR DUMP MAINT 19E TO ftp://hayden:password@server.example.com
To restore a minidisk from file disk.dump on an ftp server:
PIPEDDR RESTORE MAINT 19E FROM ftp://hayden:password@server.example.com/disks/disk.dump
To do the same thing but from an http server:
PIPEDDR RESTORE MAINT 19E FROM http://server.example.com/disks/disk.dump
To dump a minidisk to a file named maint.disk019e on an NFS server, first the nfs directory must be mounted to your virtual machine using OPENVM MOUNT. Assuming it is mounted at /home/maint/mnt, the command is:
PIPEDDR DUMP MAINT 19E TO BFSPATH /home/maint/mnt
How to verify a file
The CKSUM and DIGEST options were added so that a disk image file that is stored on a Posix compatible platform, such as Linux, could be verified on that platform. A program to perform that verification is not supplied, but these examples show the format of the file.
The file is written with a 1024 byte header and a 1024 byte trailer before and after the disk image. The trailer contains the verification data which is either the cksum value in the first 12 bytes or the digest value. The header has data about the input disk in both EBCDIC and ASCII, with x'0A' bytes in between. If a digest hash is included, the length of the digest is the last word of the header. A simple script, such as this, will show the ASCII header:
#!/bin/bash echo "Show header (ascii version)" # It is in the 1024 byte header, the second "line" found in there head -c +1024 $1 | cat | head -n 2 | tail -n 1
The cksum of the disk image can be calculated with this script:
#!/bin/bash echo "Show cksum of disk contents (decimal cksum and byte count)" # Take the 1024 byte header and trailer off and input to cksum tail -c +1025 $1 | head -c -1024 | cksum
The cksum in the tailer record of the file can be shown with this bash script and the following helper Rexx script:
#!/bin/bash echo "Show cksum values in last record (decimal cksum and byte count)" # Take the first 12 bytes of the 1024 byte trailer record tail -c 1024 $1 | head -c 12 | ./cksumshow.rex
Here is the Rexx script
#!/usr/bin/rexx numeric digits 15 /* Read standard input */ input=charin(,,12) /* cksum data from pipelines is 12 bytes */ cksum = left(input,4) length = right(input,8) /* Show cksum and length like the cksum command */ say c2d(cksum) c2d(length)
These examples should help you create a more complete program that can verify a disk image has not been damaged.
Change log, latest changes first
Version Change Description
------- ------------------
V1.6.12 Rework syntax with TO|FROM, urls, and fewer options needed.
V1.6.11 Allow cipher key and ftp password to be supplied in variables.
V1.6.10 Check for cipher stage support in Pipelines
V1.6.9 Default to FTP REXX for ftp, use old with OLD opt.
Change how the TCP listen pipe is stopped.
V1.6.8 Allow the cipher part of the header to be blank
Misc updates, changes to comments
Remove EAV restriction because of VM66139 on 6.4
V1.6.7 Add STABLE option, for stable RO and exclusive R/W links
V1.6.6 Support flashcopy for copy
V1.6.5 Error dropping short records causing DDR restore to fail
V1.6.4 For dumps with a digest value, also verify original disk data
V1.6.3 Support KM CSL routines instead of Pipe cipher stage
V1.6.2 Support ZLIB compression
V1.6.1 Make it fullscreen friendly and fix other bugs
V1.6.0 Make it release 6
V1.5.29 Add SET options for digest and cipher options
V1.5.28 Support digest options
V1.5.27 Add timeout option for maximum listen time.
V1.5.26 Support cipher stage options CBC and IV
V1.5.25 Return reason for remote rejection in response
V1.5.24 Enhanced TCPIP error messages and errors from Pipes
V1.5.23 Capture TCPIP errors in pipeline and output via emsg
V1.5.22 Output most messages via EMSG, exit RC is message number
V1.5.21 Prevent novalue, handle unknown sizes in disk check
V1.5.20 Add enhanced debug
V1.5.19 Add experimental AWSTAPE option to deblock awstape format
V1.5.18 Add RAW and NOSIZECHECK options
V1.5.17 Add support for CKSUM CRC calculations, for Linux verification
V1.5.16 Use of the CIPHER option requires the uplevel Pipelines
On z/VM 6.2 and later, CPFMTXA EXEC is found on PMAINT 551
V1.5.15 Trackwrite bug fixed in official Pipelines release - update
V1.5.14 Check pipe level and disk size to avoid trackwrite bug
Add railroad track syntax diagram to help
V1.5.13 Use GLOBALV to save format and compression preferences
Add DEBUG option, DRPC and DDR are part of z/VM 6.2
V1.5.12 For dump over TCPIP, send computed crc back to sender for verification
V1.5.11 Add PORT option, to use instead of positional parameter
Display remote disk user and vaddr for IP restores
Add MATCHDISK option to check for source and target disk match
Do more checking for valid options
V1.5.10 Support dump/restore to BFS paths (for NFS support)
V1.5.9 For restore, fix novalue or syntax error with progress messages
V1.5.8 If no VMLINK nickname "TCPIP", try TCPMAINT 592 instead
V1.5.7 Display progress messages while processing
V1.5.6 Allow nopack with DDR compact options
V1.5.5 Add a disk to disk copy option
V1.5.4 Wait time for IP transfers default 2 secs, add SLEEP option
V1.5.3 Add option to not read input file or device twice on restore
V1.5.2 Support CMS DDR format using the DRPC package
V1.5.1 Support FBA disks using FBAREAD/FBAWRITE, add CRC option,
make PACK the default, add NOPACK option