Download count: 37 this month, 5899 altogether.

Downloads for PIPEDDR:
VMARC archive: v-64K

PIPEDDR EXEC

The PIPEDDR EXEC can use either the raw disk I/O support available in the updated Pipelines runtime library (part of z/VM 6.4 and later) or the Pipelines interface to the CMS DDR command.  The 2 different formats are not compatible with each other.  The direct disk support uses the Pipelines stages TRACKREAD and TRACKWRITE (for ECKD disks) or FBAREAD and FBAWRITE (for FB-512 disks.)  The interface to the CMS DDR command uses the DDR Pipelines stage and an updated DDR module.  The updated module and Pipelines stage are included with z/VM 6.2 and later.  If you are on an older release of z/VM, then you can get the required files from the DRPC package in the VM download library.

By default, the exec dumps a disk into a regular CMS file in packed format.  This creates a file that is compatible with the PACK option of the CMS COPYFILE command and is in fixed record format with 1024 byte records.  This allows easier interchange with non mainframe systems. The exec will also restore the file back to the same or a different disk.

A file created by this exec has information in the first unpacked record, which is the disk information (size and device type), the id that dumped the disk, and a flag for if a CRC or digest value is included as the last record of the data.  The contents of the disk begin in the next record.

The disk can also be dumped over a TCP/IP connection directly to a receiving PIPEDDR exec running on a remote system or to a file on an FTP server.  A file can be restored from an FTP server or an HTTP (web) server.  Byte File System (BFS) paths are also supported, which allows the file to be dumped or restored to reside on a disk in a remote NFS server.  You must configure the CMS NFS client to use this support.  Dump to and restore from a tape is supported by using the TAPE option and the same operations using a CMS filedef using the FILEDEF option.

Also, encryption of the data is supported if the KM Package is also available from the VM download library.  The KMRTNS CSLLIB is used from the VMARC file.  These CSL routines require your machine to have the CPACF feature to use the built in hardware encryption and decryption.

The raw disk I/O interface requires either z/VM 6.4 and later or the Princeton Runtime Distribution level of Pipelines at least version 110B0004 (15 May 2002 level) for reading and writing ECKD disks and level 110C0004 (01 Jul 2010 level) or later for reading and writing FBA disks.  Note that version 110C000A (24 Jul 2012 level) or later is required for writing 3390 disks larger than 32767 cylinders due to a bug in older levels of the trackwrite stage.  PIPEDDR uses the PICKPIPE EXEC to load the runtime distribution level of Pipelines if it is needed and if it is available.  PICKPIPE is also available via the VM download library.  See the PICKPIPE description or get the VMARC file.

The Pipelines interface to CMS DDR requires either z/VM 6.2 or later or the DRPC and DDR modules from the DRPC package on the VM download library.  See the DRPC description or get the VMARC file to obtain this function.  The updated level of CMS Pipelines is not required for CMS DDR support.

The FTP option requires either the INSTPIPE MODULE from a modern version of CMS (found on MAINT 2CC or MAINT 193) or the DRPC MODULE.  See the reference to the DRPC MODULE in the previous paragraph.  The FTP option can also use a NETRC DATA file stored anywhere in the search order.  This file can supply a password or a userid and password for the connection, so that the password would not be hardcoded into an exec or displayed on the console.

The exec also supports 2 other forms of compaction, TERSE and ZLIB.  The TERSE option can only be used if the TERSE Pipelines stage is available.  This allows the exec to create output files that are smaller or send less data over the network.  The TERSE Pipelines stage is part of the PIPSYSF filter package which is now available from the Marist Pipelines page.  See the PIPSYSF Filter Package page for more information and a link to download the module.

A standalone terse function is also available as part of the FCOPY package on the VM download library.  See the FCOPY description or get the VMARC file and extract the FCOPYTRS MODULE.  This module will allow you to unterse or terse the output file created by PIPEDDR.  It should also be possible to unterse the output using the DETERSE MODULE supplied with the z/VM service tools.

The ZLIB option requires the zlib Pipeline stage which is found in in the ZLIBSTG MODULE file.  You can download the file "fplgcc.vma" which is a VMARC format file downloadable from John Hartmann's space on GitHub.  See the HELP file for more information about extracting the correct file to enable you to use this option.

If the CMS DDR format is used, the DDR module supports 2 compression options, COMPRESS and LZCOMPACT, which may provide an acceptable level of compression.

There is also a copy option, which does a disk to disk copy, with automatic linking of the source and target disks.  If the flashcopy option is specified, the exec attempts to copy the disks using the CP FLASHCOPY command.  If that command fails or your user id is not allowed to use this command, the exec falls back to a disk to disk copy.  The copy option is no different than copying a disk using DDR, and in fact just the DDR module is used if the CMSDDR option is specified on the copy command.

A help file is included in the package and it contains a more detailed description of all of the options.  Enter HELP PIPEDDR on the CMS command line for usage information.

Some examples of how to use PIPEDDR.

To dump a minidisk to a file with the default name:

PIPEDDR DUMP MAINT 19E

To dump a minidisk to a file and terse the output:

PIPEDDR DUMP MAINT 19E (TERSE

This creates a file named MAINT DISK019E A

To dump a minidisk to a file including the CRC:

PIPEDDR DUMP MAINT 19E (CRC

This also creates a file named MAINT DISK019E A

To dump the same disk to a different filemode:

PIPEDDR DUMP MAINT 19E = = E

To dump a minidisk to a file in CMS DDR format:

PIPEDDR DUMP MAINT 19E (CMSDDR

To restore the file to the same disk:

PIPEDDR RESTORE MAINT 19E

To restore the file to a different disk and skip the prompt:

PIPEDDR RESTORE CMSUSER 191 MAINT DISK019E A (NOPROMPT

To dump a minidisk to a file and encrypt the data:

PIPEDDR DUMP MAINT 19E (CIPHER TDES KEY(Super Secret KEY)

To restore it:

PIPEDDR RESTORE MAINT 19E (CIPHER TDES KEY(Super Secret KEY)

To send an entire minidisk over the network On the receiving node, enter:

PIPEDDR RESTORE MAINT 19E (LISTEN

This will display the port number it is using.  To force the port to 12345:

PIPEDDR RESTORE MAINT 19E (LISTEN PORT 12345

On the sending system use (where is the listening port):

PIPEDDR DUMP MAINT 19E nodeid.example.com (PORT 

The connection should be made and the disk sent over.

To send a minidisk to a file named maint.disk019e on an ftp server:

PIPEDDR DUMP MAINT 19E (ftp(-h server.example.com -u hayden -p password)

To restore a minidisk from file disk.dump on an ftp server, and skip the prompt, any of these commands can be used:

PIPEDDR RESTORE MAINT 19E (noprompt ftp -h server.example.com -u hayden -p password -d /disks -f disk.dump
PIPEDDR RESTORE MAINT 19E (noprompt ftp://hayden:password@server.example.com/disks/disk.dump
PIPEDDR RESTORE MAINT 19E disk.dump (noprompt ftp://hayden:password@server.example.com/disks

To do the same thing but from an http server:

PIPEDDR RESTORE MAINT 19E (noprompt http://server.example.com/disks/disk.dump

To dump a minidisk to a file named maint.disk019e on an NFS server, first the nfs directory must be mounted to your virtual machine using OPENVM MOUNT.  Assuming it is mounted at /home/maint/mnt, the command is:

PIPEDDR DUMP MAINT 19E (BFSPATH /home/maint/mnt

The CKSUM and DIGEST options were added so that a disk image file that is stored on a Posix compatible platform, such as Linux, could be verified on that platform.  A program to perform that verification is not supplied, but these examples show the format of the file.

The file is written with a 1024 byte header and a 1024 byte trailer before and after the disk image.  The trailer contains the cksum data in the first 12 bytes, or the digest data.  The header has the input disk information in both EBCDIC and ASCII, with x'0A' characters in between.  If a digest hash is included, the length of the digest is the last word of the header.  A script, such as this, will show the ASCII header:

#!/bin/bash
echo "Show header (ascii version)"
# It is in the 1024 byte header, the 2nd "line" found in there
head -c +1024 $1 | cat | head -n 2 | tail -n 1

The cksum of the disk image can be calculated with this script:

#!/bin/bash
echo "Show cksum of disk contents (decimal cksum and byte count)"
# Take the 1024 byte header and trailer off and input to cksum
tail -c +1025 $1 | head -c -1024 | cksum

The cksum in the tailer record of the file can be shown with this bash script and the following helper Rexx script:

#!/bin/bash
echo "Show cksum values in last record (decimal cksum and byte count)"
# Take the first 12 bytes of the 1024 byte trailer record
tail -c 1024 $1 | head -c 12 | ./cksumshow.rex

Here is the Rexx script

#!/usr/bin/rexx
numeric digits 15
/* Read standard input */
input=charin(,,12) /* cksum data from pipelines is 12 bytes */
cksum = left(input,4)
length = right(input,8)
/* Show cksum and length like the cksum command */
say c2d(cksum) c2d(length)

Creating a single script to do all of these functions is left as an exercise for the reader!  These examples should help you create a program to do that.

Change log, latest changes first

Version    Change Description
-------    ------------------
V1.6.7     Add STABLE option, for stable read only link
V1.6.6     Support flashcopy for copy
V1.6.5     Error dropping short records causing DDR restore to fail
V1.6.4     For dumps with a digest value, also verify original disk data
V1.6.3     Support KM CSL routines instead of Pipe cipher stage
V1.6.2     Support ZLIB compression
V1.6.1     Make it fullscreen friendly and fix other bugs
V1.6.0     Make it release 6
V1.5.29    Add SET options for digest and cipher options
V1.5.28    Support digest options
V1.5.27    Add timeout option for maximum listen time.
V1.5.26    Support cipher stage options CBC and IV
V1.5.25    Return reason for remote rejection in response
V1.5.24    Enhanced TCPIP error messages and errors from Pipes
V1.5.23    Capture TCPIP errors in pipeline and output via emsg
V1.5.22    Output most messages via EMSG, exit RC is message number
V1.5.21    Prevent novalue, handle unknown sizes in disk check
V1.5.20    Add enhanced debug
V1.5.19    Add experimental AWSTAPE option to deblock awstape format
V1.5.18    Add RAW and NOSIZECHECK options
V1.5.17    Add support for CKSUM CRC calculations, for Linux verification
V1.5.16    Use of the CIPHER option requires the uplevel Pipelines
           On z/VM 6.2 and later, CPFMTXA EXEC is found on PMAINT 551
V1.5.15    Trackwrite bug fixed in official Pipelines release - update
V1.5.14    Check pipe level and disk size to avoid trackwrite bug
           Add railroad track syntax diagram to help
V1.5.13    Use GLOBALV to save format and compression preferences
           Add DEBUG option, DRPC and DDR are part of z/VM 6.2
V1.5.12    For dump over TCPIP, send computed crc back to sender for verification
V1.5.11    Add PORT option, to use instead of positional parameter
           Display remote disk user and vaddr for IP restores
           Add MATCHDISK option to check for source and target disk match
           Do more checking for valid options
V1.5.10    Support dump/restore to BFS paths (for NFS support)
V1.5.9     For restore, fix novalue or syntax error with progress messages
V1.5.8     If no VMLINK nickname "TCPIP", try TCPMAINT 592 instead
V1.5.7     Display progress messages while processing
V1.5.6     Allow nopack with DDR compact options
V1.5.5     Add a disk to disk copy option
V1.5.4     Wait time for IP transfers default 2 secs, add SLEEP option
V1.5.3     Add option to not read input file or device twice on restore
V1.5.2     Support CMS DDR format using the DRPC package
V1.5.1     Support FBA disks using FBAREAD/FBAWRITE, add CRC option,
           make PACK the default, add NOPACK option
V1.4.10    Attempt to use a minidisk defined with DEVNO failed
V1.4.9     Fix PIPE stage sep and end chars for ftp/http stages
V1.4.8     Add cipher flag and packing type checks for TCPIP transfers
V1.4.7     Fix checking when new disk has no label
           Fix end of data marker for TCP/IP transfers
V1.4.6     Add Filedef input/output support
V1.4.5     Add single volume Tape input/output support
V1.4.4     Add CIPHER option, usable only of the pipe stage is available
V1.4.3     Add HTTP option and the ability to specify a URL for it and FTP
V1.4.2     Allow use of a NETRC DATA file for FTP parameters
V1.4.1     For FTP option, use DRPC MODULE first if found, then look
           for INSTPIPE MODULE pipe filter on MAINT 193
V1.4.0     Overall clean up
V1.3.11    Add FTP option using DRPC MODULE
V1.3.10    Use CPFMTXA to put label on disk before restore
V1.3.9     Fix math error in report of megabytes transferred
V1.3.8     Fix disk label reading code
V1.3.7     Fix false "success" message