VM for VSE Guest Performance

VM/ESA and VSE/ESA Technical Conference - May 1998

Bill Bitner
VM Performance Evaluation
IBM Corp.
1701 North St.
Endicott, NY 13760
(607) 752-6022
Internet: bitner@vnet.ibm.com
Expedite: USIB1E29 at IBMMAIL
Home Page: http://www.vm.ibm.com/devpages/bitner/

Foil - Disclaimer

The information contained in this document has not been submitted to any formal IBM test and is distributed on an "As is" basis without any warranty either express or implied. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer's ability to evaluate and integrate them into the operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environment do so at their own risk.

In this document, any references made to an IBM licensed program are not intended to state or imply that only IBM's licensed program may be used; any functionally equivalent program may be used instead.

Any performance data contained in this document was determined in a controlled environment and, therefore, the results which may be obtained in other operating environments may vary significantly.

Users of this document should verify the applicable data for their specific environment.

It is possible that this material may contain reference to, or information about, IBM products (machines and programs), programming, or services that are not announced in your country or not yet announced by IBM. Such references or information must not be construed to mean that IBM intends to announce such IBM products, programming, or services.

Should the speaker start getting too silly, IBM will deny any knowledge of his association with the corporation.

Foil - Trademarks

The following are trademarks of the IBM corporation:
- CICS
- ES/9000
- IBM
- SQL/DS
- VM/ESA
- VSE/ESA

Foil - Overview

Is VM Good for VSE Performance?

It depends.

What does it depend on?

On what you mean by "performance"
Using V=R/F
Running VM on Native or LPAR
Virtual disk in storage
Enhanced minidisk cache
Hardware

Speaker Notes

Abstract

Is VM good for VSE Performance? Bill Bitner of IBM's VM Performance Evaluation department answers that question with "It depends!". This presentation will look at what it depends on, what is meant by performance, and factors that include V=R/F/V guests, enhanced minidisk cache, virtual disk in storage, CCW translation, and hardware assists.

Credits

I'd also like to acknowledge some of the people behind this presentation: Dr. Wolfgang Kraemer, Greg Kudamik, Wes Ernsberger, Frank Brice, Steve Wilkins, Bill Stephens, Bill Guzior, my manager Doug Morrison, and countless others. In particular, a lot of this material is based on Dr. Kraemer's VSE/ESA Performance Considerations package. To get copies of the VSE/ESA 1.3 or 2.1 Performance Papers, IBMers can request the VE13PERF and VE21PERF packages from IBMVSE tools disk. This papers are also available through the VSE Home Page.

Overview

The overhead of running VSE on VM has been a topic has been discussed for years. While it is tempting to give this presentation a marketing spin, I will try to resist that temptation. I believe each customer needs to answer the question of running VSE with VM for themselves. In the process they will have to answer other questions, such as "What is performance?". In this presentation, we will attempt to cover the various trade offs that need to be made in coming to a conclusion to this question. Also, remember there are non-performance factors in answering this question.

Foil - The Short Answer is Yes

Performance Value-Add of VM

Extend capacity of single VSE, by running multiple VSEs.
VM/ESA extensions to scheduling such as limit shares.
Resource sharing
- Real storage is shared for V=V guests
- Channels are shared w/o EMIF
- DASD devices can be split up into minidisks
VM/ESA paging benefits - demand and block paging, use of expanded storage.
HW exploitation - greater N-way, expanded storage
VM/ESA features
- Virtual disk in storage for lockfile
- Enhanced minidisk cache
- SQL/DS Guest Sharing
Non-performance reasons
- CMS features
- Isolate production and test
- Migration vehicle
- Resource management
- Other PPs - OV/VM, IXFP

Speaker Notes

The Short Answer is Yes

The short answer to "Is VM good for VSE performance?" is YES. Listed are some areas where VM/ESA adds value to VSE systems from a performance perspective. VM/ESA has traditionally extended the capacity of a single VSE but providing the ability to run multiple VSE images. While the new Turbo Dispatcher provides multi-processing support, many customers will still need VM/ESA to fully exploit the large N-ways in the IBM processor line. VM/ESA's MP support in conjunction with the extensive scheduling features make it very powerful. VM/ESA allows for efficient sharing of storage for V=V guests and the virtualization of many resources. VM/ESA paging provides the best of both worlds. High-speed demand paging with expanded storage and block paging to the slower DASD.
VM/ESA also allows for several performance features. These include virtual disk in storage, enhanced minidisk cache, and VM data spaces. The latter is exploited by SQL/DS for great performance improvements.

Besides performance, there are other reasons to use VM/ESA with VSE. CMS is a great application development and test platform. The virtual machine model of VM allows for easy migration, test, and isolation. The two systems compliment each other very well with their products and applications. The value of OV/VM just to mention one. IXFP on VM allows VSE guests to get greater benefit from RAMAC Virtual Array DASD.

Foil - What do you mean by "Performance"?

Critical to answering the original question.
Typically one of the following:
- ITR = Internal Throughput Rate = a measure of work per CPU second.
- ETR = External Throughput Rate = a measure of work per wallclock second.
- CPU Utilization = how busy processor is; tied to ITR.
- Response Time (Elapsed Time) = how long jobs take; tied to ETR.
- Interactive Users vs. Batch Work
- How many phone calls you get from people unhappy with performance.

Speaker Notes

While it is important to be clear with the meaning of technology terms, no definition is more critical than that of "Performance". Whenever you get into a performance discussion, make sure you level set on this term first. When I hear people criticize the performance of VM/VSE, I am amazed that they consider CPU utilization to be the only performance indicator.
In general there are two schools of thought, one that looks at CPU utilization and one that looks at response time. The internal throughput rate, or ITR, is a measure of commands per CPU second. Another way of thinking of this is how many commands could be completed if the processor was running at 100%. Many people use ITRs to compare processor performance. When done properly, there should be an implied response time limit as well. External transaction rate is a measure of commands complete per wallclock time. When looking at performance comparisons, it is a good idea to be able to determine both the ETR and the ITR.

You will probably also need to determine the priority of batch versus interactive users.

Foil - CPU Usage by VM/ESA

Base costs and background work
- Scheduling and dispatching
- Accounting
- Monitor
Increases with amount of processing required by VM/ESA
SIE
- Used by VM/ESA to run VSE guest
- Exits from SIE indicate work for VM
- Hardware assists
Most common reasons for exiting SIE
- I/O processing
- Page fault resolution
- Instruction Simulation
- Minor time slice expires
I/O Assist can avoid exiting to SIE to handle the I/O interrupt processing.
CCW translation bypass avoids need for CP to translate I/O address from virtual to real addresses.
Minor time slice impacted by SET SRM DSPSLICE.
Avoiding Paging
- V=R/F
- Reserved pages for V=V
- Sufficient storage

Speaker Notes

One mistake people make is by trying to determine the overhead of running VSE with VM with a trivial test case. There are some base costs to running VM/ESA such as infrastructure, scheduling, dispatching, accounting, and monitor. Many of these are a constant cost. That is the CPU they require stays the same no matter how busy the system. So if you run a trivial VSE workload as a test, you'll see VM/ESA being a larger percentage of the total CPU usage.

VM/ESA uses the SIE (start interpreted execution) instruction to run virtual processors. The overhead and function processing costs in the VM CP are tied to how often we exit from SIE (via intercept or interrupt). This is a case where the hardware assists play a significant role. Listed are the four most common reasons for exiting SIE. I/O Processing tends to be the most significant. VM/ESA gets involved with all V=V I/O and some V=R/F I/O. Minimizing I/O in the guest, by using larger buffers or data-in-memory techniques, can lower VM/ESA overhead. VM/ESA also gets involved for page fault processing for V=V guests. This can be minimized by adding storage or reserving pages as appropriate. SIE is also exited for certain instruction simulation such as unassisted SIGPs and IUCV. VM/ESA will also get control when the minor time slice expires. This can be adjusted with the SET SRM DSPSLICE command. However, caution should be used when adjusting the minor time slice. While increasing it may lower the VM/ESA overhead, it also lowers the ability of VM/ESA scheduling to adjust to system changes. We have seen scenarios where the ITR improves, but ETR gets worse when increasing the minor time slice. Note also that dedicated virtual machines get a 500 millisecond dispatch time slice.

Foil - VM I/O Processing

I/O Assist
- Dedicated Devices
- V=R/F Guest
- IOASSIST ON
- See VM/ESA Performance manual Chapter 3 "I/O Interpretation Facilities" for more details
CCW Translation Bypass
- Dedicated devices and full pack minidisks in some cases
- SET NOTRANS ON (SET CCWTRAN OFF) for V=R guest
Fast CCW Translation
- Software enhancement added via APAR to VM/ESA 1.1.1
- Applies only to select DASD I/O

Speaker Notes

I/O Assist (or I/O Passthru or SIE Assist) is the ability that under certain conditions, I/O interrupts for guests do not cause CP interrupts and thus avoid VM overhead in such cases. On ES/9000 processors this is enabled by the microcode. Only dedicated devices (DASDs) of V=R or V=F guests are eligible for I/O passthru. For additional info, not covered here, refer e.g. to "VM/ESA Performance, SC24-5642-01" Chapter 3, under "I/O Interpretation Facilities". Contains requirements for I/O Assist, reasons for dropping out of I/O Assist, and information on available data.
"CCW translation bypass" can be done for V=R guests for dedicated devices and, in limited cases, full-pack minidisks. It can be controlled by the SET CCWTRAN command. For a V=R machine the default for CCWTRAN is set OFF. CCWTRAN can not be set OFF for a V=F machine. If you SET CCWTRAN ON for a V=R machine, then the benefit of I/O Assist is lost. In the past, CCW translation bypass was often referred to as I/O fast path.

"Fast CCW translation" is related to a CP feature where CP will use a more efficient path in the translation of virtual to real addresses associated with CCWs and untranslation (real to virtual)., It is only available for a subset of DASD I/O. It should not be confused with bypassing of CCW translation.

Foil - I/O Considerations

I/O Assist gives best CPU performance
Dedicated I/O is not eligible for minidisk cache
For V=R CCWTRANS OFF makes guest I/O ineligible for fast CCW translation
For VSE Guests, VSE virtual disk in storage is cheaper than VM virtual disk in storage.
Both VM virtual disk in storage and enhanced minidisk cache require real or expanded storage to provide good performance.
Enhanced minidisk cache read performance is as good as VM virtual disk in storage performance.
DASD Type
- Dedicated Devices
  - Required for full I/O Assist
  - Not eligible for minidisk cache
- Full Pack Minidisks
  - Can be shared between guests
  - Some I/O Assist
  - Define via VOLSER or DEVNO (DEVNO not eligible for MDC)
- Partial Pack Minidisks
  - Can be shared
FBA Volumes
- For full MDC benefit should start on boundary of 64 512 byte blocks and be increments of 64 blocks.
- If above is not possible, then at least use 8 block boundaries.

Speaker Notes

From an ITR view, I/O Assist gives the best performance since it avoids VM processing. However, devices eligible for I/O Assist are not eligible for minidisk cache which can hurt ETR.
Remember that the VM overhead results from exiting SIE for processing. Therefore, features in VSE that can be used to avoid I/O that VM sees can be very helpful. VSE virtual disk in storage is a good example.
Improving I/O performance often comes at a cost in some other resource. Both virtual disk in storage and minidisk cache require sufficient storage to provide good performance. With VM/ESA you can exploit expanded storage if available. If you are only looking to improve read I/O, then minidisk cache is generally the better feature.

The various types of DASD can influence performance. To get the most from I/O Assist, dedicated devices are best, followed by full pack minidisks. Partial pack minidisks are very flexible, but ineligible for I/O Assist. Remember that dedicated devices are not eligible for minidisk cache. There are also a couple of special guidelines for using minidisk cache with FBA devices.

Foil - Paging Considerations

For V=V guests the potential exists for 'Double Paging'
No VM paging for V=R/F
The closer the VSE VSIZE is to the defined storage for the virtual machine, the lower the VSE paging.
Use PAGEX ON where applicable.
VM/ESA can use expanded storage for high speed paging device.

Speaker Notes

Most people recognize that VM/ESA paging is more advanced than VSE/ESA, but there is still room for discussion. One should try to avoid scenarios of double paging. This would happen if a VSE page is not in the VSE space and is paged in to a VSE real address that VM/ESA in turn needs to page in. VM paging is avoided completely for V=R or V=F machines. The closer the VSE VSIZE to the size of the VSE virtual machine, the less VSE system should have to page. If there is no need for paging in VSE, consider the NOPDS option. Use the PAGEX ON option where appropriate.

A trade off can arise between ITR and ETR for paging. If you are not using PAGEX ON, a page fault handled by VM/ESA will serialize the virtual machine (VSE). If the paging configuration is very poor, it might be appropriate to move the paging up to the VSE level. While a page fault here would take more processor resources, it will not serialize the virtual machine. If the configuration includes expanded storage for paging, then let VM/ESA do the paging to here.

Foil - V=R/F/V Considerations

V=R/F potential I/O assist benefit (saves CPU)
V=F avoids overhead of recoverability in V=R.
1 V=R + 5 V=F or 6 V=F
V=V avoids dedicating storage
V=R defaults to dedicating processors
Running VM/ESA in an LPAR -
- If on older processor, performance will be very poor due to lack of Interpreted SIE.
- If on 3090J or newer, will have interpreted SIE assist.
- If on 3090J or newer, No V=F (only V=R) and No I/O Assist.
- Often better off just using V=V and reserving pages.

Speaker Notes

V=R performance can be lower than V=F performance. There is often extra processing required for the recoverability part of V=R support. Preferred guests (V=R/F) on a native VM/ESA avoid VM paging and provide savings with hardware assisted I/O. The total number of preferred guests is still six, even though LPAR can provide more partitions on some processors.

Speaking of LPAR. If you are running VM/ESA in an LPAR, you need to realize that it changes the characteristics. Both LPAR and VM/ESA use SIE. On older processors (3090E and older), the assists were not available to run SIE on top of SIE. Running this configuration would be very costly. All the current processors (3090S with RPQ, 3090J, ES/9000, and 9672s) have the required interpreted SIE assist for running VM/ESA on LPAR. However, with VM/ESA on an LPAR, only the V=R machine is possible and there is no I/O Assist. In this scenario, you are often better off just running the guest as a V=V machine with CP reserved pages.

Foil - Virtual MP Support

May define additional processors dynamically
- In user directory include - MACHINE ESA 2
- CP DEFINE CPU vcpu_addr
Or put everything in the directory
- CPU 00 CPUID 012345 NODEDICATE
- CPU 01 CPUID 112345 NODEDICATE
- CPU 02 CPUID 212345 NODEDICATE
Detaching vCPU resets virtual machine
Can have more virtual processors than real for testing
CP commands of interest -
- QUERY VIRTUAL CPUS
- CPU cpu_addr cmd_line
- DEDICATE and UNDEDICATE
Tuning
- Share setting is for virtual machine, therefore divided amongst all virtual processors
- Processors can be dedicated
- Mixing dedicated and shared virtual processors is not recommended
- Have a defined but inactive vCPU (in stop state) makes the virtual machine ineligible for I/O Assist.
Monitoring
- Monitor, Indicate, and RTM provide statistics for all virtual processors.
- Storage statistics are associated with the base virtual processor
- Watch for cases where normally the max is 100% but with virtual n-way max is now N*100%.
- A dedicated virtual processor looks like it is 100% busy to VM.

Speaker Notes

There are two approaches to creating a virtual MP machine. You can define the virtual processors in the directory so they are available when the virtual machine logs on. Or you can set up the directory so that you can use the DEFINE CPU command to add virtual processors dynamically. Note that detaching a virtual processor resets the virtual machine. Do not define extra virtual processors unless you are going to use them. Defined, but unused, virtual processors can cause performance problems.

From a tuning perspective, it is important to note that the share value is distributed across the virtual machine. For example, if you have a virtual 4-way and a default share of relative 100, then each virtual processor would be scheduled as if it had a relative 25 share value. A virtual processor in CP stopped state makes it ineligible for I/O Assist. Virtual processors can be dedicated to real processors. I do not recommend mixed environments where a single virtual machine has both dedicated and undedicated virtual processors. This can result in performance anomalies that will be difficult to detect and explain.

Foil - VM/ESA Data in Memory Techniques

VM Data Spaces
- Exploited by SQL/DS DSS feature
VM virtual disk in storage
- Introduced in VM/ESA 1.2.1
- Volatile FBA minidisk
- Private or Shareable
- Lock File
Minidisk Cache
- Enhanced in VM/ESA 1.2.2 for VSE
- Supports undedicated 3380, 3390, 9345, and RAMAC
- Supports SSCH, SIO, SIOF
- General system wide psuedo track cache
- Read-once data generally does not benefit.
- New (2.3.0) Record Level MDC does not apply to VSE.
- Do not use MDC for VSE lock-file

Speaker Notes

The three main data-in-memory techniques used by VM/ESA are VM data spaces, virtual disks in storage, and minidisk cache. Many products exploit VM data space, in particular interest to VSE customers is SQL/DS with the DSS feature. The VM virtual disk in storage feature allows for volatile FBA minidisks that can be defined as shareable or private. This are backed by a VM system utility space. It is ideal for the VSE lock file. Minidisk cache was enhanced in VM/ESA 1.2.2 to be more flexible, allow more types of data, and more types of I/O. The enhancements included a series of CP commands to enable/disable the cache, flush the cache; the ability to use real storage as the cache; the eligibility of almost any type of data; and eligibility of SSCH, SIO, and SIOF I/Os. The minidisk cache is track oriented. One should not suppose that MDC will benefit read-once data, particularly if the reading application has been highly tuned.

Foil - Exploitation Examples

Speaker Notes

The above two graphs show exploitation of the new minidisk cache by a VSE guest with two different workloads. They bring to light two important facts. First, it is possible to get better throughput with a guest running under VM/ESA than running native, by trading off processor and storage resources. Second, it is workload dependent based on I/O characteristics.
For the I/O intensive PACEX workload, the new MDC can be very beneficial, especially when combined with virtual disk in storage. Note that when VM/ESA gets more involved, processor utilization can increase. This is a trade-off for better elapsed time.
The VSE CICS workload did not show as significant an improvement as PACEX. The CICS workload is less I/O intensive and has a lower read to write ratio. While there was a significant reduction in the number of DASD I/Os, average response time reduction was not as large. Also, note the diminishing returns as more storage is provided for minidisk cache.

From the CICS graph, you can also see the law of diminishing returns. As more and more storage is given to MDC, the rate of improvement lessens.

Foil - Summary

VM can be very good for VSE performance
Many features to be exploited
Optimum configuration will depend on
- What you mean by term performance
- What resources you have available
Perf tips page = http://www.vm.ibm.com/perf/tips/
- Common problems/solutions
- CCW translation matrix
- VSE Guest Performance
- Performance related APARs
- MDC Guidelines
- N-way and CMOS thoughts
VM and VSE continue to go together well

Speaker Notes

I hope this presentation helped generate questions as to how and where VM/ESA can be used to help with your VSE guest performance. I'm sure not all your questions were answered here. There is a great deal of information available in manuals, listservers, IBMLINK, and the VM or VSE home pages. Check it out.

I welcome your comments and suggestions on this presentation.

Foil - References

VM/ESA Performance. (SC24-5782)

VM/ESA V2 R1.0 Performance Report (GC24-5801-00)

VM/ESA Release 2.2 Performance Report (GC24-5673-01)

VM/ESA Release 2.1 Performance Report (GC24-5673-00)

VM/ESA Release 2.0 Performance Report (GG66-3245)

VM/ESA Release 1.1 Performance Report (GG66-3236)

VM/ESA Planning and Administration (SC24-5750)

VM/ESA CP Command and Utility Reference (SC24-5773)

VM/ESA Running Guest Operating Systems (SC24-5755)

VM/VSE Performance Hints and Tips (GG24-4260)

VM/ESA Home Page

VSE/ESA Home Page

Foil - Acronyms...

VM/ESA: Virtual Machine / Enterprise Systems Architecture
VM/XA: Virtual Machine / Extended Architecture
RTM: Realtime Monitor
VMPRF: VM Performance Reporting Facility
CSL: Callable Services Library