Preferred Guest Migration Considerations

Two changes in processor and software technology require a z/VM capacity planner to look carefully at processor requirements as he plans a migration to z990, z890, z9, or to z/VM Version 5. These technology changes are:

  • The z890, z990, z9, z10, and z196 are LPAR-only machines.
  • V=R/F and I/O Assist are not present in z/VM Versions 5 or 6.

In this paper we look at what changed, why it changed, how to size the impact, and how to tune in this new environment.

What Changed

z990 and z890 are LPAR-only machines. When z/VM runs in an LPAR, these performance considerations apply:

  • z/VM does not support V=F guests.
  • I/O Assist is not present.
  • Because the LPAR hypervisor is using one level of SIE to dispatch z/VM, z/VM has only one SIE level remaining with which to dispatch its guests. (zSeries hardware supports two levels of SIE.)

z/VM Version 5 does not offer V=R/F guests, nor does it offer I/O Assist.

More Detail

V=R gives several benefits:

  • Entirely resident storage: the guest's storage is entirely resident in contiguous memory all the time.
  • No virtual address translation: the guest's storage is entirely resident in low memory, where guest real addresses equal host real addresses.
  • Automatically dedicated CPUs: if the z/VM system can make processors available, it will permanently assign, or dedicate, real processors to the virtual processors of the guest. A dedicated processor has a 500 msec minor time slice and gets the SIE wait state assist.
  • I/O Assist: eligible for I/O Assist (see below).
  • Channel program translation bypass: if I/O Assist happens not to apply, for I/O to a dedicated device, it is unnecessary for z/VM to translate channel programs.
  • Guest recovery: if the z/VM system bounces, it will recover the guest, so the guest is unable to discern there was a z/VM outage, except for noticing the passage of time.

V=F gives a similar but smaller list of benefits:

  • Entirely resident storage: the guest's storage is entirely resident in contiguous memory all the time.
  • Expedited virtual address translation: translation from guest real addresses to host real addresses requires only addition of a constant value.
  • I/O Assist: eligible for I/O Assist (see below).
  • Expedited channel program translation: if I/O Assist happens not to apply, for I/O to a dedicated device, z/VM's translation of guest channel programs is dramatically simplified.

I/O Assist, present only if z/VM runs in basic mode, offers the following I/O benefits to V=R and V=F guests:

  • Channel programs targeted to dedicated devices do not cause a SIE intercept.
  • Completion interrupts from a dedicated device for a running guest are delivered to the guest without a SIE break.
  • Completion interrupts from a dedicated device for a non-running guest are queued for the guest without a SIE break.
In other words, when I/O Assist applies, z/VM is unaware the guest is doing I/O.

There are various requirements for the guest and device to be eligible for I/O Assist. Refer to I/O Interpretation Facilities in Chapter 3 of the z/VM 4.4.0 edition of z/VM Performance for more information.

Finally, zSeries hardware supports only up to two levels of SIE. When z/VM runs in basic mode (no LPARs), the first-level z/VM uses one level of SIE to dispatch its guests, leaving one level of SIE available for a second-level V=R or V=F z/VM to dispatch its guests. Thus VSE-on-VM-on-VM runs well (no performance impact) in basic mode when preferred guest support is used. This can only be done on a system prior to z/VM 5.1.0. However, when z/VM runs in an LPAR, the LPAR hypervisor uses one level of SIE to dispatch z/VM, and z/VM uses the second level of SIE to dispatch its guest. Thus VSE-on-VM-on-VM in an LPAR is a poor performer, because hardware support of SIE runs out after two levels. z/VM has to simulate the third level of SIE.

Why Things Changed

Several technology and market changes have contributed the necessity of these changes:

  • The implementation of multiple channel sets was easier and more cost effective with the existence of LPAR on the machines. This, and other factors led to LPAR-only (no basic mode) machines starting with the z990 and z890 machines. Future machines (e.g. z9) continue this restriction.
  • Fully supporting V=R/F in the presence of LPAR was expensive in development and testing costs.
  • The market has shifted toward Linux being a more important guest. There are several considerations here:
    • When Linux is present on VM, there aren't just a few guests, there are many, sometimes hundreds. V=R/F offers performance boosts to only a handful of guests, whose storage size together sums to less than 2 GB.
    • For cost reasons, Linux is often run on IFL engines, which require LPAR.
    • We forecast that over time, Linux guests will become more dependent on QDIO and FCP I/O and less dependent on traditional subchannel I/O. Therefore, it is desirable to invest in assists that benefit V=V guests using QDIO and FCP I/O, rather than traditional preferred guests using traditional I/O.
  • Investsments and improvements in other hardware assists can benefit guests besides Linux that also exploit that technology.

How to Size the Impact

As a z/VM capacity planner, you need to size the performance impact of these changes on your particular workload. The following documents or techniques will be of assistance.

Loss of I/O Assist

Check the I/O Interpretation Facility discussion in the z/VM 4.4.0 edition of z/VM Performance for information about the I/Os that are eligible for I/O Assist.

Our VSE Performance web page offers some guidance on the effect of I/O assist on certain VSE workloads.

Run your workload with CP SET IOASSIST OFF and measure the increase in processor time this causes (probably only host time).

Loss of V=R/F

Run your workload V=V. Use the CP Monitor to watch for increased CPU consumption due to channel program translation or paging.

Run your workload V=V. Use the CP Monitor to watch for increased load on the z/VM paging system, such as increased channel utilization on paging chpids, queues forming at paging volumes, increased utilization (percent full) of paging extents, or formation of an eligible list in the z/VM scheduler.

The LPAR Hypervisor's Use of SIE

There is little you can do to assess the effect of this. Rather, get rid of the intermediate z/VM before you move away from basic mode.

How to Tune

To get back some of the lost performance, consider one or more of the following techniques.

  • To compensate a guest for loss of its dedicated processors, use CP SET SHARE ABSOLUTE.

  • To compensate a guest for its loss of dedicated memory, use CP SET RESERVED to guarantee a guest a certain number of real page frames. Choose a reserved value that is about 10% above the guest's apparent working set size as displayed by CP INDICATE USER. Be careful to watch for and to guard against creep in the working set size. Note that we do not recommend using CP LOCK, because it locks the guest pages below the real 2 GB bar and because it is very difficult to determine which guest pages are the correct ones to lock.

  • To compensate a guest for its loss of dedicated memory, you might have to use CP SET SRM STORBUF to increase the z/VM system's tolerance for overcommitting dynamic storage.

  • To compensate a guest for the time it spends waiting for page faults to be resolved, consider enabling asynchronous page fault handing (PAGEX or PFAULT) if your guest supports it.

  • If the paging load on your z/VM system increases as a consequence of the conversion, be prepared to increase the capacity of your paging configuration. Increase the amount of paging space available, spread the paging extents across controllers and chpids, place paging space only on volumes used entirely for paging, allocate some expanded storage (XSTORE) to z/VM, and perhaps issue CP SET SRM LDUBUF to increase the z/VM system's tolerance for guests that induce paging.

  • To compensate for loss of I/O Assist, consider switching from dedicated volumes to minidisks and turning on Minidisk Cache (MDC). There will be some storage and processor cost but I/O response time will be improved.

  • To compensate for the loss of one level of SIE, split your environment into multiple LPARs, or get rid of the second-level z/VM system, that is, move all production guests to the first-level z/VM system.

  • To compensate for loss of V=R guest recovery, make sure your z/VM system and your zSeries processor are up-to-date on service.

Writer(s) Date Remarks
Bitner 2010-11-04 Added additional processor families.
Bitner 2007-03-26 Corrected discussion about the logic behind making the changes
Bitner 2006-05-19 Corrected discussion about two levels of interpreted SIE
Bitner 2006-02-10 Added System z9
Wade 2005-07-28 Clarified references to z/VM Performance book
Wade 2004-11-17 Changed PR/SM to LPAR hypervisor
Wade 2004-09-20 Addressed comments from Wes Ernsberger
Bitner, Wade 2004-09-13 Original edition