Concurrent I/O Support for XIV
Abstract
z/VM 6.4 with PTF UM35080 for APAR VM65929 introduces changes to the z/VM SCSI container to allow for concurrent I/O to EDEVICEs defined on XIV hardware. Previously, XIV EDEVICEs were limited to serialized I/O. An experiment with a heavy paging workload using XIV EDEVICEs for the paging devices showed a 96% improvement in transaction rate with concurrent I/O compared to serialized I/O.
Introduction
IBM initially added XIV direct attach support in z/VM 5.4 with the PTF for APAR VM64708. In that PTF IBM decided to limit the concurrent I/O queue depth to one for XIV EDEVICEs. This allowed IBM to deliver support for XIV EDEVICEs quickly. However, serialized I/O has a cost of reduced performance for workloads capable of exploiting concurrent I/O. As a consequence, using XIV EDEVICEs for paging was not recommended for environments with significant storage over-commitment. z/VM 6.4 PTF UM35080 removes this queue depth restriction for XIV EDEVICEs. z/VM 6.4 EDEVICEs with the XIV attribute in the definition will now take advantage of concurrent I/O. In workloads capable of exploiting concurrent I/O this change can result in improved performance.
Method
Concurrent I/O for XIV EDEVICEs was evaluated with the PFAULT workload. Workload details:
- Processor type and model: 2964-NC9
- Dedicated LPAR, sixty-four IFL cores, non-SMT mode.
- No other LPARs activated during the measurement.
- Twenty SLES 12 SP 1 Linux guests running PFAULT.
- Each guest had one virtual processor and 88 GB of virtual storage.
- 1 TB central storage
- 1.2:1 (instantiated:real) storage overcommit
- Twenty 50 GB XIV EDEVICEs with the XIV attribute
PFAULT Internals
The PFAULT program used for this experiment is slightly different than the one described in the link above. Rather than touching random pages, the PFAULT used in this experiment touches pages sequentially. Touching pages sequentially drives a higher paging rate and produces more consistent results. In addition, PFAULT was modified to synchronize the Linux guests such that they begin faulting at the exact same time. This improves consistency between runs.
(88 GB virtual storage) x (20 Linux users) x (70% instantiated) = 1.232 TB total instantiated storage
PFAULT was set to output the number of page touches every thirty seconds. This experiment ran for ten minutes so there were twenty samples' worth of data for each user. The number of page touches for all users and all samples was added up. This total was then divided by the elapsed time in seconds. The transaction rate is therefore the number of page touches per second for all twenty users.
Results and Discussion
| Table 1. Serialized vs. Concurrent I/O. | ||||
| Serialized | Concurrent | Delta | Pct | |
| run ID | 1TX8155 | 1TX4877 | - | - |
| Tx/sec | 669552.07 | 1313779.82 | 644227.75 | 96.2 |
| DASD Page Rate | 123000.0 | 225000.0 | 102000.0 | 82.9 |
| Tx per CPU-sec | 800899.61 | 584421.63 | -216477.98 | -27.0 |
| ITR | 51504005.2 | 37536566.3 | -13967438.9 | -27.1 |
| total busy | 83.6 | 224.8 | 141.2 | 168.9 |
| guest busy | 38.5 | 53.8 | 15.3 | 39.7 |
| c-CP busy | 6.1 | 9.4 | 3.3 | 54.1 |
| nc-CP busy | 39.0 | 161.6 | 122.6 | 314.4 |
| avg busy | 1.3 | 3.5 | 2.2 | 169.2 |
| T/V ratio | 2.17 | 4.18 | 2.01 | 92.6 |
| Notes: See method section for workload details. Serialized run is GA version of z/VM 6.4. Concurrent run is z/VM 6.4 with XIV concurrent I/O PTF. c-CP Busy is CP overhead that is chargable to users. nc-CP is CP overhead that is charged to the system instead of to a user. | ||||
- Transaction rate improved 96.2%
- Page rate (reads+writes) increased by 82.9%
- With concurrent I/O, the CPU utilization increased significantly because of the increased number of page touches. As a result, the ITR actually decreased in the run.
In addition to the number of transactions, the FCX249 SCSI Perfkit report shows a significant improvement with concurrent I/O. In the concurrent I/O run, the number of transfers per second for a typical EDEV was almost twice as high as with serialized I/O. The amount of data moved was also about twice as large in the concurrent I/O measurement.
| Table 2. Typical EDEV Behavior, Concurrent I/O vs. Serialized I/O. From FCX249 SCSI report. | |||||
| Run ID | Device Number | Bytes per Block | KByte/sec | Transfers/sec | |
| Serialized | 1TX8155 | 5070 | 512 | 24635 | 283.0 |
| Concurrent | 1TX4877 | 5070 | 512 | 45001 | 571.9 |
Summary
XIV EDEVICEs now support concurrent I/O. Workloads capable of performing multiple I/Os simultaneously to XIV EDEVICEs might experience a performance improvement. A z/VM paging workload is one such workload. Another such workload might be one in which there are multiple minidisks on the same XIV EDEVICE, with each minidisk being used by a different guest. Using XIV EDEVICEs for paging in storage over-committed environments is now more practical.