Contents | Previous | Next

Concurrent I/O Support for XIV


z/VM 6.4 with PTF UM35080 for APAR VM65929 introduces changes to the z/VM SCSI container to allow for concurrent I/O to EDEVICEs defined on XIV hardware. Previously, XIV EDEVICEs were limited to serialized I/O. An experiment with a heavy paging workload using XIV EDEVICEs for the paging devices showed a 96% improvement in transaction rate with concurrent I/O compared to serialized I/O.


IBM initially added XIV direct attach support in z/VM 5.4 with the PTF for APAR VM64708. In that PTF IBM decided to limit the concurrent I/O queue depth to one for XIV EDEVICEs. This allowed IBM to deliver support for XIV EDEVICEs quickly. However, serialized I/O has a cost of reduced performance for workloads capable of exploiting concurrent I/O. As a consequence, using XIV EDEVICEs for paging was not recommended for environments with significant storage over-commitment. z/VM 6.4 PTF UM35080 removes this queue depth restriction for XIV EDEVICEs. z/VM 6.4 EDEVICEs with the XIV attribute in the definition will now take advantage of concurrent I/O. In workloads capable of exploiting concurrent I/O this change can result in improved performance.


Concurrent I/O for XIV EDEVICEs was evaluated with the PFAULT workload. Workload details:

  • Processor type and model: 2964-NC9
  • Dedicated LPAR, sixty-four IFL cores, non-SMT mode.
  • No other LPARs activated during the measurement.
  • Twenty SLES 12 SP 1 Linux guests running PFAULT.
  • Each guest had one virtual processor and 88 GB of virtual storage.
  • 1 TB central storage
  • 1.2:1 (instantiated:real) storage overcommit
  • Twenty 50 GB XIV EDEVICEs with the XIV attribute

PFAULT Internals

The PFAULT program used for this experiment is slightly different than the one described in the link above. Rather than touching random pages, the PFAULT used in this experiment touches pages sequentially. Touching pages sequentially drives a higher paging rate and produces more consistent results. In addition, PFAULT was modified to synchronize the Linux guests such that they begin faulting at the exact same time. This improves consistency between runs.

(88 GB virtual storage) x (20 Linux users) x (70% instantiated) = 1.232 TB total instantiated storage

PFAULT was set to output the number of page touches every thirty seconds. This experiment ran for ten minutes so there were twenty samples' worth of data for each user. The number of page touches for all users and all samples was added up. This total was then divided by the elapsed time in seconds. The transaction rate is therefore the number of page touches per second for all twenty users.

Results and Discussion

Table 1. Serialized vs. Concurrent I/O.
  Serialized Concurrent Delta Pct
run ID 1TX8155 1TX4877 - -
Tx/sec 669552.07 1313779.82 644227.75 96.2
DASD Page Rate 123000.0 225000.0 102000.0 82.9
Tx per CPU-sec 800899.61 584421.63 -216477.98 -27.0
ITR 51504005.2 37536566.3 -13967438.9 -27.1
total busy 83.6 224.8 141.2 168.9
guest busy 38.5 53.8 15.3 39.7
c-CP busy 6.1 9.4 3.3 54.1
nc-CP busy 39.0 161.6 122.6 314.4
avg busy 1.3 3.5 2.2 169.2
T/V ratio 2.17 4.18 2.01 92.6
Notes: See method section for workload details. Serialized run is GA version of z/VM 6.4. Concurrent run is z/VM 6.4 with XIV concurrent I/O PTF. c-CP Busy is CP overhead that is chargable to users. nc-CP is CP overhead that is charged to the system instead of to a user.

  • Transaction rate improved 96.2%
  • Page rate (reads+writes) increased by 82.9%
  • With concurrent I/O, the CPU utilization increased significantly because of the increased number of page touches. As a result, the ITR actually decreased in the run.

In addition to the number of transactions, the FCX249 SCSI Perfkit report shows a significant improvement with concurrent I/O. In the concurrent I/O run, the number of transfers per second for a typical EDEV was almost twice as high as with serialized I/O. The amount of data moved was also about twice as large in the concurrent I/O measurement.

Table 2. Typical EDEV Behavior, Concurrent I/O vs. Serialized I/O. From FCX249 SCSI report.
  Run ID Device Number Bytes per Block KByte/sec Transfers/sec
Serialized 1TX8155 5070 512 24635 283.0
Concurrent 1TX4877 5070 512 45001 571.9


XIV EDEVICEs now support concurrent I/O. Workloads capable of performing multiple I/Os simultaneously to XIV EDEVICEs might experience a performance improvement. A z/VM paging workload is one such workload. Another such workload might be one in which there are multiple minidisks on the same XIV EDEVICE, with each minidisk being used by a different guest. Using XIV EDEVICEs for paging in storage over-committed environments is now more practical.

Contents | Previous | Next