z/VM Paging Improvements
Abstract
z/VM 6.4 provides several paging enhancements. The Control Program (CP) was improved to increase both I/O payload sizes and the efficiency of page blocking. Also, CP can use the HyperPAV feature of the IBM System Storage DS8000 line of storage controllers for paging I/O. Further, CP can also use High Performance FICON (zHPF) transport-mode I/O for paging I/O. For amenable z/VM paging workloads, these enhancements can result in increased throughput or the equivalent throughput with fewer physical volumes.
IBM experiments using the command-mode paging driver resulted in a 42% transaction rate improvement. Adding HyperPAV aliases to a paging workload with I/Os queueing on the paging devices resulted in a 42% transaction rate improvement. Using transport-mode I/O resulted in a 98% transaction rate improvement. Using HyperPAV aliases and transport-mode I/Os resulted in a 234% transaction rate improvement.
Introduction
In z/VM 5.3 IBM introduced HyperPAV support for DASD volumes containing guest minidisks. z/VM 6.4 exploits HyperPAV for paging extents. Readers not familiar with HyperPAV or not familiar with z/VM's HyperPAV support should read IBM's HyperPAV technology description before continuing here.
In z/VM 6.2 IBM introduced zHPF support for guest operating system use. z/VM 6.4 exploits zHPF for paging I/O. Readers not familiar with zHPF or not familiar with z/VM's zHPF support for guests should read IBM's High Performance FICON description before continuing here. To use zHPF for paging I/Os, FICON Express8S or newer is required.
z/VM 6.4 also features enhancements to the paging subsystem improving logical page blocking and increasing I/O payload.
The command-mode I/O driver is the default paging I/O driver for z/VM 6.4. zHPF and HyperPAV aliases are optional and need to be enabled for use. Improvements were also made in the paging subsystem to increase the number of contiguous slots written on a single volume by one channel program, resulting in larger I/O payloads.
Method
Paging Evaluations
The new paging I/O options were evaluated with a Virtual Storage Exerciser (VIRSTOR) workload. The particulars of the workload held constant across all paging measurements were:
- z13, dedicated LPAR, eight IFL cores, non-SMT, 128 GB central storage
- Paging volumes on a single LCU of a DS8800
- Eight 18 GB virtual uniprocessor users running VIRSTOEX, a memory thrasher, with VIRSTOEX configured to cause heavy paging by looping through a 16 GB loop touching every page.
- VIRSTOR parameters: loops=1,cpus=01,i=4,v=4,w=50
Command-Mode Paging Driver Measurement
This measurement compares the command-mode paging driver to the z/VM 6.3 paging driver. Both runs use the same four paging extents.
Transport-Mode Paging Driver Measurement
This measurement compares the transport-mode paging driver to the command-mode paging driver. Both runs use the same four paging extents.
HyperPAV Alias Paging Measurement
This measurement demonstrates the effect of adding HyperPAV aliases to a paging workload capable of using them. Both runs use the same four paging extents. The comparison run has four HyperPAV aliases enabled for use with the paging volumes.
HyperPAV Alias Paging DASD Reduction Measurement
This measurement demonstrates the effect of using HyperPAV aliases to replace paging volumes to achieve the same parallelism. The base run uses eight paging extents. The comparison run uses four paging extents and four HyperPAV aliases enabled for use with the paging volumes.
Transport-Mode Paging Driver and HyperPAV Alias Paging Measurement
This measurement demonstrates the effect of using z/VM 6.4 with HyperPAV aliases and transport-mode both enabled for use by four paging volumes.
HyperPAV Alias Sharing Measurement
In z/VM 6.4 the CP SET CU command was enhanced to let an administrator specify a sharing policy the Control Program should observe when minidisk I/O and paging I/O are competing for the HyperPAV aliases of an LCU. The sharing policy implements a relative-share model matching the notion of relative CPU share as implemented by CP SET SHARE. In the present case of the sharing of HyperPAV aliases, minidisk I/O and paging I/O are the "users" and the HyperPAV aliases are the contended-for resource to be shared. When there is more demand for HyperPAV aliases than there are HyperPAV aliases available, the Control Program hands out the aliases to minidisk I/O and to paging I/O in accordance with their respective CU alias share settings.
To measure the effect of the CP SET CU command, IBM set up a hybrid, memory-constrained workload consisting of two groups of users. The first group had a high demand for minidisk I/O. Further, via CP SET RESERVED this first group was kept protected from central storage contention. The second group, memory thrashers, had a high demand for paging I/O. The real DASD volumes holding the minidisks and the real DASD volumes holding the paging space resided together in a single LCU along with some HyperPAV aliases. By using the CP SET CU command to vary the alias-share settings for minidisk I/O and paging I/O, IBM was able to observe whether the Control Program enforced the CU share settings correctly. IBM was also able to see the effect the CU share settings had on the performance of the two groups of users.
The particulars of the workload were:
- z13, dedicated LPAR, four cores, non-SMT, 8 GB central.
- A single LCU of a DS8800, said LCU equipped with four DASD volumes holding minidisks, two DASD volumes holding paging space, and sixteen HyperPAV aliases attached to SYSTEM.
- Thirty-two virtual uniprocessor guests running the IO3390 DASD I/O exerciser, each such guest protected from memory contention via CP SET RESERVED. In this group of guests, a transaction is a minidisk I/O. Notice IBM used more IO3390 guests than there were SYSTEM-attached HyperPAV aliases. This was done so MDISK alias demand would always be high enough to use all the aliases.
- Twelve virtual uniprocessor users running VIRSTOCX, a memory thrasher, with VIRSTOCX configured to cause heavy paging. In this group of guests, a transaction is 1000 references to pages. A reference might or might not cause a page fault. Notice IBM used fewer VIRSTOCX guests than the number of SYSTEM-attached HyperPAV aliases. This was done so PAGING would not have enough alias demand to use all of its alias entitlement when its alias entitlement was very high.
IBM did fifteen runs of this workload. Each run used a different pair of relative-share settings for MDISK and PAGING. The pairs of settings were chosen to cover the spectrum from heavy favor for MDISK, to equal weight, to heavy favor for PAGING. IBM recorded the ETR of the minidisk users and also the ETR of the paging users. IBM also used the D6 R28 MRIODHPP monitor records to observe whether the Control Program were respecting the CU share settings.
Results and Discussion
Command-Mode Paging Driver Measurement
Table 1 reports the result of the command-mode measurement.
| Table 1. Effect of command-mode paging driver. | ||||
| Run ID | V4DVX001 | V4DVY000 | Delta | Pct |
| CP level | 6.3.0 | 6.4.0 | ||
| Type and model | 2964-NC9 | 2964-NC9 | ||
| Processors | 8 | 8 | 0 | 0.0 |
| SYSGEN storage | 131072 | 131072 | 0 | 0.0 |
| Paging devices | 4 | 4 | 0 | 0.0 |
| ETR | 0.0310 | 0.0442 | 0.0132 | 42.6 |
| Total busy | 12.0 | 15.4 | 3.4 | 28.3 |
| Guest busy | 2.5 | 3.7 | 1.2 | 48.0 |
| Chargeable CP busy | 1.1 | 1.2 | 0.1 | 9.1 |
| Nonchargeable CP busy | 8.4 | 10.5 | 2.1 | 25.0 |
| Chargeable CP busy, /tx | 0.0355 | 0.0271 | -0.0084 | -23.7 |
| Nonchargeable CP busy, /tx | 0.2710 | 0.2376 | -0.0334 | -12.3 |
| Pages written to DASD, /sec | 31985 | 44459 | 12474 | 39.0 |
| Pages read from DASD, /sec | 32027 | 44466 | 12439 | 38.8 |
| Paging SSCH insts, /sec | 1487 | 1578 | 91 | 6.1 |
| Pages per SSCH | 43.0 | 56.4 | 13.4 | 31.1 |
| Page read blocking | 28 | 38 | 10 | 35.7 |
| Page write blocking | 27 | 32 | 5 | 18.5 |
| Page slot utilization, % | 11 | 15 | 4 | 36.4 |
| DASD connect time, /IO, ms | 2.250 | 1.970 | -0.280 | -12.4 |
| DASD disconnect time, /IO, ms | 0.019 | 0.006 | -0.013 | -68.4 |
| DASD pending time, /IO, ms | 0.135 | 0.139 | 0.004 | 3.0 |
| DASD service time, /IO, ms | 2.400 | 2.110 | -0.290 | -12.1 |
| Notes: z13, dedicated 8-core, non-SMT, 128 GB central. DS8800 with single LCU containing four DASD volumes holding paging extents, eight VIRSTOEX users, virtual uniprocessor, configured to page heavily. | ||||
The z/VM 6.4 command-mode paging I/O driver and paging subsystem improvements improved ETR 42.6% over z/VM 6.3. Improvements in the paging subsystem increased the pages per SSCH 31.1%, the read blocking 35.7%, and the write blocking 18.5%. Service time improved 12.1% led by a 68.4% decrease in disconnect time. These measurements show the low CPU cost of paging. The z/VM 6.4 command-mode measurement was able to page at a rate of 88,925 pages per second using only 11.7% of an IFL core in chargable-CP and non-chargeable-CP busy.
Transport-Mode Paging Driver Measurement
Table 2 reports the result of the transport-mode measurement.
| Table 2. Effect of transport-mode paging driver. | ||||
| Run ID | V4DVY000 | V4DVYH04 | Delta | Pct |
| CP level | 6.4.0 | 6.4.0 | ||
| Type and model | 2964-NC9 | 2964-NC9 | ||
| Processors | 8 | 8 | 0 | 0.0 |
| SYSGEN storage | 131072 | 131072 | 0 | 0.0 |
| Paging devices | 4 | 4 | 0 | 0.0 |
| ETR | 0.0442 | 0.0877 | 0.0435 | 98.4 |
| Total busy | 15.4 | 29.4 | 14.0 | 90.9 |
| Guest busy | 3.7 | 7.1 | 3.4 | 91.9 |
| Chargeable CP busy | 1.2 | 2.3 | 1.1 | 91.7 |
| Nonchargeable CP busy | 10.5 | 20.0 | 9.5 | 90.5 |
| Chargeable CP busy, /tx | 0.0271 | 0.0262 | -0.0009 | -3.3 |
| Nonchargeable CP busy, /tx | 0.2376 | 0.2281 | -0.0095 | -4.0 |
| Pages written to DASD, /sec | 44459 | 87857 | 43398 | 97.6 |
| Pages read from DASD, /sec | 44466 | 87896 | 43430 | 97.7 |
| Paging SSCH insts, /sec | 1578 | 3172 | 1594 | 101.0 |
| Page read blocking | 38 | 39 | 1 | 2.6 |
| Page write blocking | 32 | 32 | 0 | 0.0 |
| Page slot utilization, % | 15 | 29 | 14 | 93.3 |
| DASD connect time, /IO, ms | 1.970 | 0.965 | -1.005 | -51.0 |
| DASD disconnect time, /IO, ms | 0.006 | 0.007 | 0.001 | 16.7 |
| DASD pending time, /IO, ms | 0.139 | 0.138 | -0.001 | -0.7 |
| DASD service time, /IO, ms | 2.110 | 1.110 | -1.000 | -47.4 |
| Notes: z13, dedicated 8-core, non-SMT, 128 GB central. DS8800 with single LCU containing four DASD volumes holding paging extents, eight VIRSTOEX users, virtual uniprocessor, configured to page heavily. | ||||
The transport-mode I/O driver showed a 98.4% ETR increase over the command-mode I/O driver. The DASD service time decreased 47.4% allowing the paging subsystem to do twice as many I/Os per second.
HyperPAV Alias Paging Measurement
Table 3 reports the result of the HyperPAV alias measurement.
| Table 3. Effect of using HyperPAV aliases for paging. | ||||
| Run ID | V4DVY000 | V4DVYA00 | Delta | Pct |
| CP level | 6.4.0 | 6.4.0 | ||
| Type and model | 2964-NC9 | 2964-NC9 | ||
| Processors | 8 | 8 | 0 | 0.0 |
| SYSGEN storage | 131072 | 131072 | 0 | 0.0 |
| Paging devices | 4 | 4 | 0 | 0.0 |
| Paging aliases | 0 | 8 | 8 | - |
| ETR | 0.0442 | 0.0628 | 0.0186 | 42.1 |
| Total busy | 15.4 | 29.5 | 14.1 | 91.6 |
| Guest busy | 3.7 | 7.1 | 3.4 | 91.9 |
| Chargeable CP busy | 1.2 | 2.3 | 1.1 | 91.7 |
| Nonchargeable CP busy | 10.5 | 20.1 | 9.6 | 91.4 |
| Chargeable CP busy, /tx | 0.0271 | 0.0263 | -0.0008 | -3.0 |
| Nonchargeable CP busy, /tx | 0.2376 | 0.2297 | -0.0079 | -3.3 |
| Pages written to DASD, /sec | 44459 | 62640 | 18181 | 40.9 |
| Pages read from DASD, /sec | 44466 | 62674 | 18208 | 40.9 |
| Page read blocking | 38 | 39 | 1 | 2.6 |
| Page write blocking | 32 | 32 | 0 | 0.0 |
| Page slot utilization, % | 15 | 21 | 6 | 40.0 |
| DASD connect time, /IO, ms | 1.970 | 4.580 | 2.610 | 132.5 |
| DASD disconnect time, /IO, ms | 0.006 | 0.001 | -0.005 | -83.3 |
| DASD pending time, /IO, ms | 0.139 | 0.203 | 0.064 | 46.0 |
| DASD service time, /IO, ms | 2.110 | 4.780 | 2.670 | 126.5 |
| Notes: z13, dedicated 8-core, non-SMT, 128 GB central. DS8800 with single LCU containing four DASD volumes holding paging extents, eight VIRSTOEX users, virtual uniprocessor, configured to page heavily. | ||||
Adding eight aliases and enabling them for use by the paging subsystem improved ETR 42.1%.
The data below shows the experience for a single paging volume in each measurement. Adding aliases resulted in the I/O rate to the volume (T_IOR) increasing and volume percent busy (T_PBSY) increasing. Alias contribution can cause a volume's percent busy to exceed 100%. The queue on the base device (B_QD) did not go to zero because the workload has latent demand.
V4DVY000:
RDEV ___Interval_End____ ___B_QD___ __T_IOR___ __T_PBSY__ BE00 2016-09-14_16:12:10 3.267 408.5 85.3 BE00 2016-09-14_16:12:40 3.933 392.1 84.9 BE00 2016-09-14_16:13:10 3.600 393.3 85.1 BE00 2016-09-14_16:13:40 3.067 395.1 84.9 BE00 2016-09-14_16:14:10 2.200 393.7 85.1 BE00 2016-09-14_16:14:40 3.600 391.6 85.1 BE00 2016-09-14_16:15:10 1.733 394.0 84.5
V4DVYA00:
RDEV ___Interval_End____ ___B_QD___ __T_IOR___ __T_PBSY__ BE00 2016-09-14_19:06:44 2.067 555.6 267.2 BE00 2016-09-14_19:07:14 2.000 556.3 270.4 BE00 2016-09-14_19:07:44 2.733 556.1 273.3 BE00 2016-09-14_19:08:14 3.133 552.9 269.2 BE00 2016-09-14_19:08:44 2.267 557.9 272.4 BE00 2016-09-14_19:09:14 2.000 552.0 271.6 BE00 2016-09-14_19:09:44 1.667 557.4 267.7
HyperPAV Alias Paging DASD Reduction Measurement
Table 4 reports the result of the HyperPAV alias DASD reduction measurement.
| Table 4. Effect of using aliases to reduce paging volume count. | ||||
| Run ID | V8DVY000 | V4DVYA40 | Delta | Pct |
| CP level | 6.4.0 | 6.4.0 | ||
| Type and model | 2964-NC9 | 2964-NC9 | ||
| Processors | 8 | 8 | 0 | 0.0 |
| SYSGEN storage | 131072 | 131072 | 0 | 0.0 |
| Paging devices | 8 | 4 | -4 | -50.0 |
| ETR | 0.0578 | 0.0591 | 0.0013 | 2.2 |
| Total busy | 19.6 | 20.3 | 0.7 | 3.6 |
| Guest busy | 4.7 | 4.8 | 0.1 | 2.1 |
| Chargeable CP busy | 1.5 | 1.6 | 0.1 | 6.7 |
| Nonchargeable CP busy | 13.4 | 13.9 | 0.5 | 3.7 |
| Chargeable CP busy, /tx | 0.0260 | 0.0271 | 0.0011 | 4.2 |
| Nonchargeable CP busy, /tx | 0.2318 | 0.2352 | 0.0034 | 1.5 |
| Pages written to DASD, /sec | 57921 | 59243 | 1322 | 2.3 |
| Pages read from DASD, /sec | 57957 | 59268 | 1311 | 2.3 |
| Page slot utilization, % | 10 | 21 | 11 | 110.0 |
| Notes: z13, dedicated 8-core, non-SMT, 128 GB central. DS8800 with single LCU containing four DASD volumes holding paging extents, eight VIRSTOEX users, virtual uniprocessor, configured to page heavily. | ||||
Reducing the paging devices by four and replacing them with four HyperPAV aliases yielded an ETR within run variation. Paging slot utilization increased 110% because there are fewer physical volumes and the same amount of pages on them.
Transport-Mode Paging Driver and HyperPAV Alias Paging Measurement
Table 5 reports the result of the HyperPAV alias and transport-mode measurement.
| Table 5. Effect of transport-mode and HyperPAV aliases for paging. | ||||
| Run ID | V4DVX001 | V4DYA001 | Delta | Pct |
| CP level | 6.3.0 | 6.4.0 | ||
| Type and model | 2964-NC9 | 2964-NC9 | ||
| Processors | 8 | 8 | 0 | 0.0 |
| SYSGEN storage | 131072 | 131072 | 0 | 0.0 |
| Paging devices | 4 | 4 | 0 | 0.0 |
| ETR | 0.0310 | 0.1037 | 0.0727 | 234.5 |
| Total busy | 12.0 | 35.3 | 23.3 | 194.2 |
| Guest busy | 2.5 | 8.4 | 5.9 | 236.0 |
| Chargeable CP busy | 1.1 | 2.7 | 1.6 | 145.5 |
| Nonchargeable CP busy | 8.4 | 24.2 | 15.8 | 188.1 |
| Chargeable CP busy, /tx | 0.0355 | 0.0260 | -0.0095 | -26.8 |
| Nonchargeable CP busy, /tx | 0.2710 | 0.2334 | -0.0376 | -13.9 |
| Pages written to DASD, /sec | 31985 | 103704 | 71719 | 224.2 |
| Pages read from DASD, /sec | 32027 | 103740 | 71713 | 223.9 |
| Paging SSCH insts, /sec | 1487 | 1752 | 265 | 17.8 |
| Page read blocking | 28 | 39 | 11 | 39.3 |
| Page write blocking | 27 | 32 | 5 | 18.5 |
| Page slot utilization, % | 11 | 34 | 23 | 209.1 |
| DASD connect time, /IO, ms | 2.250 | 1.830 | -0.420 | -18.7 |
| DASD disconnect time, /IO, ms | 0.019 | 0.006 | -0.013 | -68.4 |
| DASD pending time, /IO, ms | 0.135 | 0.157 | 0.022 | 16.3 |
| DASD service time, /IO, ms | 2.400 | 2.000 | -0.400 | -16.7 |
| Notes: z13, dedicated 8-core, non-SMT, 128 GB central. DS8800 with single LCU containing four DASD volumes holding paging extents, eight VIRSTOEX users, virtual uniprocessor, configured to page heavily. | ||||
The transport-mode I/O driver with aliases enabled for use by the paging subsystem yielded the best result in the study. ETR increased 234.5% when compared to z/VM 6.3.
HyperPAV Alias Sharing Measurement
Table 6 reports the result of the set of alias sharing measurements.
| Table 6. Effect of SET CU Command on Hybrid Workload. | |||||
| Run Name | Alias share ratio MDISK:PAGING | Mdisk ETR | Paging ETR | MDISK Avg Aliases In Use | Paging Avg Aliases In Use |
| QRBKW048 | 15:1 | 35904 | 31667 | 14.98 | 1.01 |
| QRBKW049 | 14:2 | 32992 | 43333 | 13.98 | 2.01 |
| QRBKW050 | 13:3 | 30592 | 50943 | 12.98 | 3.01 |
| QRBKW051 | 12:4 | 28640 | 54446 | 11.98 | 4.01 |
| QRBKW052 | 11:5 | 26272 | 62229 | 10.99 | 5.01 |
| QRBKW053 | 10:6 | 24096 | 66212 | 9.99 | 6.00 |
| QRBKW054 | 9:7 | 22400 | 67209 | 8.99 | 7.00 |
| QRBKW055 | 8:8 | 20192 | 71358 | 8.00 | 7.99 |
| QRBKW056 | 7:9 | 18144 | 75481 | 7.01 | 8.98 |
| QRBKW057 | 6:10 | 16288 | 79300 | 6.03 | 9.96 |
| QRBKW058 | 5:11 | 14400 | 83013 | 5.07 | 10.93 |
| QRBKW059 | 4:12 | 12768 | 86452 | 4.12 | 11.88 |
| QRBKW060 | 3:13 | 11136 | 90317 | 3.19 | 12.81 |
| QRBKW061 | 2:14 | 9760 | 93130 | 2.29 | 13.71 |
| QRBKW062 | 1:15 | 8448 | 98224 | 1.47 | 14.53 |
| Notes: z13, dedicated 4-core LPAR, non-SMT, 8 GB central. z/VM 6.4 plus VM65886. DS8800 with single LCU containing four DASD volumes holding minidisks, two DASD volumes holding paging extents, and sixteen HyperPAV aliases. Thirty-two IO3390 users, virtual uniprocessor, protected by SET RESERVED. Twelve VIRSTOCX users, virtual uniprocessor, configured to page heavily. | |||||
The results above show the following:
-
As alias entitlement moved
away from minidisk I/O and
toward paging I/O,
two things happened:
- For the minidisk workload, its use of aliases decreased and its ETR decreased;
- For the paging workoad, its use of aliases increased and its ETR increased.
- In the first few runs, where both kinds of I/O had alias demands exceeding their entitlements, CP distributed the alias use according to the CU share settings. Each of the two types of I/O was permitted to use only the number of aliases prescribed by its entitlement.
- In the last few runs, where paging I/O wanted aliases but did not quite want its entitlement thereof and minidisk I/O still wanted all the alias power the alias scheduler would let it use, CP let minidisk I/O use the aliases paging I/O did not want.
Figure 1 plots the scaled ETRs of the minidisk I/O and paging I/O portions of the workload. IBM scaled each portion's ETR according to the ETR the portion achieved when it had entitlement for only one alias. By scaling the ETRs IBM has made it easy for a single graph to demonstrate that as aliases moved from paging to minidisk, the minidisk ETR increased and the paging ETR decreased.
| Figure 1. z13, HyperPAV alias hybrid workload. Relative ETR as f(MDISK-alias-share). |
|
Summary and Conclusions
For amenable z/VM paging workloads, paging subsystem enhancements provided with z/VM 6.4 can result in increased throughput and/or the equivalent throughput with fewer physical volumes.
For the alias-sharing support, when demand for aliases exceeds availability, CP shares the aliases correctly between minidisk I/O and paging I/O. To move the power of the aliases between a minidisk-intensive workload and a paging-intensive workload, the administrator can issue the CP SET CU command to change the alias share settings.