VM Parallel Access Volumes Support

On April 27, 2006, IBM announced the plan to support Parallel Access Volumes (PAVs) as minidisks. With PTF for APAR VM63952, z/VM V5.2 supports Parallel Access Volumes (PAVs) as minidisks for guest operating systems such as z/OS that exploit the PAV architecture. In addition, the APAR provides the potential benefit of PAVs for I/O issued to minidisks owned or shared by guests that do not support native exploitation of PAVs, such as CMS.

Refer to these pages for:

This page contains
z/VM PAV Support Overview

IBM DASD PAV volumes must be defined to z/VM as a 3390 Model 2, 3, or 9 (including the mod-27 and mod-54 variety) DASD on a 3990 Model 3 or 6, 2105, 2107, or 1750 Storage Controller. 3380 track-compatibility mode for the 3390 Model 2 or 3 DASD is also supported.

The PAV hardware feature allows you to configure one or more logical DASD volumes, each with a base and one or more alias subchannels. Figure 1 illustrates the relationship between base and alias subchannels for a volume. The base subchannel represents the real (physical) DASD volume space. Alias subchannels (X and Y in this example) map to the same physical volume space as accessed by the base. The alias subchannels do not have their own data space; they are "shadows" of the base. This architecture allows concurrent accesses to a volume through its base and associated alias subchannels.

Figure 1: 3 DASD Volumes
Figure 1.

Base subchannels are defined in IOCP as UNIT=3990, 2105, 2107, or 1750 on the CNTLUNIT statement and UNIT=3390B (or 3380B) on the IODEVICE statement. Alias subchannels are defined as UNIT=3990, 2105, 2107, or 1750 on the CNTLUNIT statement and UNIT=3390A (or 3380A) on the IODEVICE statement. Each base or alias subchannel can be assigned any available VM real device number. Use the IBM DASD subsystem configuration console to initially define which subchannels are base subchannels, which subchannels are alias subchannels, and which alias subchannels are associated with each base volume. Use the CP QUERY PAV command to view the current allocation of base and alias subchannels.

Base and alias subchannels provide nearly identical functions for the volume. One exception is that "volume-wide" commands such as the Reserve and Release channel commands may only be issued to a base subchannel, but the resulting status applies to all of the alias subchannels as well.

Certain virtual PAV operations require the consistent use of the same real base or alias subchannel. To facilitate this, each virtual PAV base and alias has an "assigned" real device subchannel that may be displayed with the "QUERY VIRTUAL vdev DETAILS" and "QUERY VIRTUAL PAV" commands. The assignment is automatic and may not be changed. One example of this would be the execution of the Read Configuration Data command. The scheduling of I/O to an assigned device is automatically handled by z/VM during its analysis of the virtual channel program. Because each virtual PAV base or alias must have a uniquely assigned real PAV base or alias subchannel, you may not have more virtual aliases than real aliases for a volume.

The Define Extent channel command specifies if a channel program may read and/or write data from or to a volume. For each volume, read operations are permitted concurrently over multiple base or alias subchannels. However, write operations are serialized on the volume when the cylinder ranges specified in the Define Extent channel command overlap with another active CCW chain in any other subchannel for the volume.

A dedicated PAV volume may not have its alias subchannels spread among multiple guests. The use of shared minidisks provides that functionality.

Using PAV Dedicated DASD

To use PAV dedicated devices, the guest must contain support for managing and serializing the volume's data across the subchannels - an "exploiting" operating system. z/VM acts only as the "pipe" between the guest and the hardware. Once the necessary base and associated alias subchannels are attached to the guest, the guest must manage their use.

In a dedicated environment, the performance benefits of PAV are entirely up to the operating system running in the virtual machine. z/VM will not make any attempt to optimize or alter the I/O flowing through the base and alias subchannels. Note that real volumes cannot be dedicated to multiple guests.

Figure 2: Example: PAV Dedicated Configuration, is a typical example of a guest virtual machine using two dedicated PAV volumes with one base and two alias subchannels for each:

Figure 2: Example: PAV Dedicated Configuration
Figure 2: Example: PAV Dedicated Configuration.
 

Using PAV Minidisks

In the context of PAV, the support for both full- and non-full-pack minidisks behaves in the same manner.

A guest virtual machine may define one or more minidisk volumes that exist on a real PAV volume. The real PAV volume has a real base and one or more real PAV alias subchannels. Each minidisk volume has one virtual PAV base and zero or more virtual PAV alias subchannels. These may be displayed with the new QUERY VIRTUAL PAV command. For each minidisk volume, the number of virtual PAV aliases for a guest may not exceed the number of real PAV aliases defined in the hardware for the underlying real volume.

All I/O operations that are directed to a minidisk volume through either its virtual base or alias subchannels are optimized by z/VM's automatic selection of an appropriate real PAV base or alias subchannel for the underlying real volume. In other words, the scheduling of I/O to a virtual PAV base or alias subchannel will be dynamically scheduled and multiplexed on any real PAV base or alias subchannel that is defined in the hardware. This gives z/VM the flexibility to choose a real PAV base or alias subchannel that is not in use at the time. For example, if users GUEST1 and GUEST2 simultaneously issue an I/O request to two different minidisk volumes that are defined on the same real underlying volume, via their respective virtual base subchannels, the result would be that one I/O would be executed on the real base subchannel and the other, simultaneously, would be executed on a real alias subchannel.

The Minidisk Cache and the SET MDCACHE command are supported for PAV minidisk volumes.
 

Using PAV Minidisks with Exploiting Operating Systems

An exploiting operating system is one that is capable of controlling the PAV architecture and is configured to control the features of PAV. Such an operating system understands how to control and utilize virtual PAV aliases. Examples might be z/VM, zOS, or Linux.

To define a full-pack minidisk for an exploiting operating system at virtual E100 with virtual aliases at E101, E102, and E103, you might code the following statements in the user directory: 

MDISK E100 3390 0 END PAK001
DASDOPT PAVALIAS E101-E103
    or
LINK GUEST1 E100 E100 MW
DASDOPT PAVALIAS E101-3103

To define a non-full-pack minidisk for an exploiting operating system at virtual E100 with virtual aliases at E101, E102, and E103, you might code the following statements in the user directory: 

MDISK E100 3390 100 200 PAK002
MINIOPT PAVALIAS E101-E103
          or
LINK GUEST1 E100 E100 MW
MINIOPT PAVALIAS E101-3103

Figure 3: PAV Minidisk Configuration for Exploiting Guests, is a typical example of several guest virtual machines that exploit PAV volumes. One volume is a full-pack minidisk that is shared among the five guests (E100, PAK001), and there are four non-full-pack minidisk volumes (E200s and E300) that share the same underlying real PAV volume (PAK002). Note that there are more PAV minidisk volumes (5 MDISKs) than real volumes (2). z/VM will multiplex I/O operations on the real base and alias subchannels for each:

Figure 3: : PAV Minidisk Configuration for Exploiting Guests
Figure 3: : PAV Minidisk Configuration for Exploiting Guests
 

Using PAV Minidisks with Non-Exploiting Operating Systems

A non-exploiting operating system is one that is not configured to control the features of PAV or has no knowledge of the PAV architecture. Although the guest operating system won't use its minidisk volumes in PAV mode, z/VM will still provide PAV performance optimization across multiple non-exploiting guests. Performance gains may be realized only when full-pack minidisks are shared among guests with multiple LINK statements or when multiple non-full-pack minidisk volumes reside on a real PAV volume. Performance gains are achieved by transparently multiplexing the I/O operations requested on each guest minidisk volume over the appropriate real PAV base and alias subchannels.

z/VM V4.4, V5.1, V5.2 without PAV support, VSE, TPF, and CMS are examples of non-exploiting operating systems. z/OS and Linux can also be considered non-exploiting, depending on how they are configured.

To define a full-pack minidisk for a non-exploiting operating system at virtual E100, you might code the following statements in the user directory: 

MDISK E100 3390 0 END PAK001
         or
LINK GUEST1 E100 E100 MW
To define a non-full-pack minidisk for a non-exploiting operating system at virtual E100, you might code the following statements in the user directory: 
MDISK E100 3390 100 200 PAK002
         or
LINK GUEST1 E100 E100 MW

Figure 4. PAV Minidisk Configuration for Non-Exploiting Guests, is a typical example of several non-exploiting guest virtual machines using PAV for enhanced performance:


Figure 4. PAV Minidisk Configuration for Non-Exploiting Guests
Figure 4. PAV Minidisk Configuration for Non-Exploiting Guests

In the configuration above (Figure 4), there are five links to real volume PAK001. For these five virtual PAV base subchannels, there is one real PAV base and three real PAV alias subchannels that will be used to perform the I/O. z/VM will concurrently multiplex the I/O from the GUEST1-GUEST5 E100 virtual bases onto the real 4580, 4581, 4582, and 4583 subchannels as they are available. Using this strategy, it is possible to have many guest virtual machines sharing a real DASD volume with z/VM dynamically handling the selection of real PAV base and alias subchannels.

I/O operations to the minidisks defined on PAK002 will likewise be optimized by z/VM's dynamic selection of real PAV base and alias subchannels 4584, 4585, 4586, and 4587.

  Note: CMS is a non-exploiting operating system, and therefore the use the DASDOPT PAVALIAS and MINIOPT PAVALIAS user directory statements is not recommended. Additionally, the use of the Class G DEFINE PAVALIAS command by CMS users should be discouraged. It is possible to write a CMS application that may take advantage of PAV volumes, but CMS itself is not PAV-aware. When multiple CMS volumes are defined on a real PAV volume, I/O operations by CMS may be concurrently scheduled on any real PAV base or alias subchannel by z/VM. The CMS user does not need to take any action for this to occur.

z/VM and Dynamic PAV

Dynamic PAV is a hardware capability that allows an Alias device to be removed from its defined Base device and re-associated with a different Base. This Alias movement is restricted to within the same hardware logical controller (control unit). z/VM does not use the Dynamic PAV capability for its own use. z/VM does, however, attempt to tolerate Dynamic PAV on behalf of guest operating systems that support it. Currently, z/OS is the only System z operating system that supports Dynamic PAV. The z/OS Workload Manager uses Dynamic PAV to dynamically move Alias devices to Base devices subjected to the most I/O queuing.

A Dynamic PAV Alias movement can be initiated in the following ways:

  1. Outboard at the hardware HMC
  2. By a z/OS LPAR sharing DASD with z/VM
  3. By a z/OS guest on another z/VM LPAR sharing DASD with z/VM
  4. By a z/OS guest on same z/VM LPAR with access to a Dedicated (ATTACHed) Alias
  5. By a z/OS guest on same z/VM LPAR with access to a Full-pack Minidisk Alias

Each method for initiating a Dynamic PAV movement has specific impacts on z/VM and guests accessing the related PAV devices.

Sharing DASD with another LPAR: The most common impact is a Dynamic PAV movement from a z/OS LPAR sharing DASD with z/VM. When this occurs z/VM will receive an interrupt from the hardware indicating that the change has occurred. In general, z/VM will react to this interrupt by moving the Alias from the old Base to the new Base in VM's internal structures. This change will be reflected in the output of the CP QUERY PAV command. If a VM guest has dedicated access to the Alias device, then the hardware interrupt will also be reflected to the guest so the guest can react to the change.

In some cases, this configuration can have unexpected results if the customer was not aware that the devices were shared with z/VM. That is, all DASD are defined to all LPARs and z/VM dynamic device sensing is enabled by default. If the intent is really not to share the DASD, then the customer could configure the devices as NOTACCEPTed or OFFLine_at_IPL on the DEVICES statement in the VM SYSTEM CONFIG file. The recommended approach is to configure the storage subsystem to place all z/OS Workload Managed PAV volumes into their own logical controller and also place them as "not accepted" or "offline at IPL" via the VM CONFIG file.

z/OS can move an Alias to any Base device in the logical controller; even if the Base device was never configured with an Alias device. Note that from a storage subsystem perspective, any device in the logical controller with real data behind it is considered a Base device and can have Alias volumes moved to it (or configured to it) at a later time. z/VM, however, will not recognize a device as a PAV Base on the QUERY PAV command until an associated Alias is varied on-line with the VARY ONLINE command. In addition, VM will not vary on-line an Alias volume if the Base device is inaccessible to VM or is not currently varied on-line. Dynamic PAV can run into this condition if some of the Base devices are not accessible or on-line to VM when another z/OS LPAR moves an Alias from one Base to a Base that is inaccessible or off-line.

An attempt to vary on-line an Alias without an on-line Base will result in the following VM message:

  • HCP6861I - Device rdev cannot be varied online because it is an alias Parallel Access Volume for which the base could not be located.

A Dynamic PAV movement triggered by the storage subsystem HMC or a z/OS guest on another VM LPAR with access to dedicated (CP ATTACHed) PAVs shared with a z/VM will behave in the same manner as described when sharing PAVs with a native z/OS LPAR.

Following is a list of tips when sharing real PAV devices with other LPARs:

  • An Alias in a logical controller can be dynamically moved to any other Base in the controller
  • If a device is marked by VM as a Base and it loses its last associated Alias, the device will still be marked as a Base (on VM).
  • If a Base has no Alias devices at VM IPL (i.e., not marked as a Base by VM) and later gains an Alias, it will be marked as a PAV Base if 520 & 530 APAR VM64379 is applied.

Dedicated DASD: Dynamic PAV is supported for dedicated DASD as long as all Base and Alias volumes associated with a guest initiated Dynamic PAV operation are dedicated to that guest. VM will reject (with an I/O error) a Dynamic PAV operation if it attempts to move a dedicated Alias from one Base to a Base that is not within the same guest's virtual configuration.

If a Dynamic PAV operation is initiated outside of the VM LPAR, for example by another LPAR sharing the same DASD, a guest supporting Dynamic PAV will be made aware of the change as long as the Alias and the old and new Base devices are all dedicated to that guest. This awareness is done by hand-shaking built into the I/O hardware architecture. If one or more of the associated devices are not dedicated to the guest, the guest will either not notice the change or will orphan the Alias device (i.e., it will no longer be bound to a guest Base device). In either case, subsequent I/O errors will probably occur.

Fullpack Minidisks: Guest Dynamic PAV operations to fullpack minidisks are supported and can work well with careful planning. Fullpack minidisks are defined in the VM user directory with the MDISK statement with either the DEVNO operand or cylinder specifications of '0 END'. For Dynamic PAV, the DASDOPT PAVALIAS definition specifies the number of virtual Alias devices the fullpack minidisk Base will have at guest log on. The MDISK statement represents the Base.

It is highly recommended that Dynamic PAV not be used from a VM guest against non-Fullpack Minidisks. Due to complications with managing this environment, this capability (for non-Fullpack Minidisks) has been fenced in z/VM 5.4.0.

Careful planning is necessary because guest Dynamic PAV operations to a fullpack minidisk do not change the real hardware configuration. That is, these operations are fully simulated by VM and do not change which real Alias devices are associated with the minidisk's real Base device. This simulation prevents guest's from creating a "ping pong" effect of Dynamic PAV transitions when fullpack minidisks are shared among the guests. Specifically, one guest's Dynamic PAV operations have no effect on another guest's virtual Alias to Base associations. The real hardware stays static and each guest's virtual associations are completely independent of each other. When each guest log's on, their virtual Base to Alias associations revert back to that specified in the user directory. This is different than what occurs with dedicated DASD.

Fullpack minidisk planning specifics: Each virtual Alias device must be backed by a real Alias device which is bound to the correct real Base. In addition, each virtual Alias must be backed by a real Alias device that is unique for the guest. VM automatically determines which virtual Alias devices are built on which real Alias devices at guest log in, but there must be enough unique real Aliases bound to a given real Base for the guest's virtual minidisk Base to Alias definitions to be built upon. This is important to understand for Dynamic PAV operations for fullpack minidisks. That is, there must be enough extra real Alias devices for a given real Base (attached to the VM system) to accommodate the maximum number of virtual Aliases that the guest's workload might attempt to move to any given Base minidisk. In other words, the real Base to Alias configuration must contain some "slack" Alias devices to accommodate Dynamic PAV operations when compared to a guest's PAV directory definitions. It is also possible to define a fullpack minidisk without any virtual Alias devices; that is, no DASDOPT statement. As long as there are real Alias devices associated with a minidisk's corresponding real device, VM will recognize the minidisk as a virtual Base. This will allow a virtual Alias from a less utilized virtual Base to be moved to it when necessary.

Fullpack minidisk example: If a guest workload normally uses 4 Aliases for a given Base, but sometimes dynamically expands to using a maximum of 8; define 4 virtual Alias devices with the DASDOPT PAVALIAS statement for the Base minidisk. However, make sure that there are 8 real Aliases configured to the real Base and attached to the system to accommodate the guest changes. Keep in mind, that the expansion to 8 virtual Alias devices can only happen if 4 virtual Aliases are taken from another under utilized virtual Base. Therefore, some virtual Bases will have to be configured with a few more Aliases than usually required.

Tips: z/OS RMF Device Activity report can be used to determine the normal and maximum Aliases used for a given Base device. The goal is to add enough Aliases to reduce I/O Queuing to zero. Also, z/OS running in an LPAR only moves Aliases to Base devices in the same LSS. This rule still applies when z/OS is running under VM.

z/VM Restrictions on Using PAV

  1. A virtual alias subchannel cannot be IPLed.
  2. You should not use PAV alias volumes as z/VM installation volumes (for example, do not use for the 520RES volume).
  3. z/VM Paging and SPOOLing operations do not take advantage of PAV. It is recommended that PAGE and SPOOL areas be placed on DASD devices dedicated to this purpose.
  4. A real alias subchannel may be attached to a guest or SYSTEM only after its associated real base subchannel has been attached to the same guest or SYSTEM.
  5. A real base subchannel may be detached from a guest or SYSTEM only if all of its associated alias subchannels are already free.
  6. A real alias subchannel will not come online to z/VM without an associated real base subchannel. Also, a real base subchannel must have at least one associated real alias subchannel for z/VM (for example, the QUERY PAV command) to recognize the device as a Parallel Access Subchannel.
  7. A real base subchannel cannot be changed or deleted with the SET RDEVICE, DELETE RDEVICE, DELETE DEVICE, or MODIFY DEVICE command unless all associated real alias subchannels have been deleted with the DELETE RDEVICE command..
  8. CMS does not support virtual alias subchannels. Under CMS, you may for example issue a Class G "DEFINE PAVALIAS 291 FOR BASE 191" command followed by an "ACCESS 291 B". If you use "FILELIST * * B", you will see the contents of the 191 disk, but CMS does not understand that 191-A and 291-B are actually the same volume and corruption will occur if changes are made to either 191 or 291. This is similar to the damage that can be caused by issuing "LINK * 191 291 MW".

Reference documents
  • Creating and Updating a User Directory chapter in z/VM CP Planning and Administration: provides for information about the user directory statements referenced on this page

  • DASD Sharing chapter in z/VM CP Planning and Administration: is the source of this web page

  • CP Commands chapter of the z/VM CP Commands and Utilities Reference is where the CP commands referenced on this web page are documented

Return to the top of the page