VM/ESA Greater N-way ThoughtsVM/ESA and VSE/ESA Technical Conference - October 1996 (C) Copyright IBM Corporation 1996 - All Rights Reserved
Table of Contents
DISCLAIMER DISCLAIMERThe information contained in this document has not been submitted to any formal IBM test and is distributed on an "As is" basis without any warranty either express or implied. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer's ability to evaluate and integrate them into the operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environment do so at their own risk. In this document, any references made to an IBM licensed program are not intended to state or imply that only IBM's licensed program may be used; any functionally equivalent program may be used instead. Any performance data contained in this document was determined in a controlled environment and, therefore, the results which may be obtained in other operating environments may vary significantly. Users of this document should verify the applicable data for their specific environment. It is possible that this material may contain reference to, or information about, IBM products (machines and programs), programming, or services that are not announced in your country or not yet announced by IBM. Such references or information must not be construed to mean that IBM intends to announce such IBM products, programming, or services. Should the speaker start getting too silly, IBM will deny any knowledge of his association with the corporation. Trademarks
Introduction
References
Several key significant changes have people looking at environments that are new. Well, at least environments that are new to them. These technology improvements include hardware and software changes. IBM continues to exploit CMOS technology to provide better price performance processors. Also, on the software side, VSE/ESA shipped multiprocessor support in the Fall of 1995. This support, known as the Turbo Dispatcher, allows for greater efficiencies with n-way processors. These various changes have people asking where VM/ESA fits in. VM/ESA still has a major role to play. VM/ESA runs well on the new CMOS boxes and the virtual MP support of VM/ESA allows continued added benefit for VSE/ESA customers. This presentation looks at the VM/ESA support of n-way processors and virtual multiprocessor configurations. With the various changes we mentioned, there are scenarios that can get customers into problem situations. My goal is to describe the key considerations to avoid critical situations. Those that know me sometimes get tired of me answering performance questions with it depends. They've started following up the questions with "On what does it depend?". Typically I list references at the end of my presentations. However in this case, I believe it is very important that you do not end your research of this area with this presentation. While I will try to be thorough, there are a number of areas I just can not cover to the degree you may want. A number of the references are being periodically updated and I would recommend making sure you have the most recent. Let me also take a second to thank some people: Wolfgang Kraemer, Wes Ernsberger, and Greg Kudamik (all of IBM) for their expertise in VSE and VM performance; Barton Robinson (Velocity Software) for making me think about some of these issues; Dean Stone (IBM) for his editorial comments on the presentation; and Romney White (Mirasoft) for his help in research. VM/ESA Multiprocessor Support
VM/ESA has always had N-way support. There were some changes back in the VM/ESA 1.1.1 time frame where we added support for greater than 6-way support. We should be good up to a 14-way. A four-bit counter is used for the number of processors whose last-translated address corresponds to a particular frame. This allows values from zero through 15, but 15 would also result if a value of zero was decreased by one (in error), so only values 0 through 14 are legal. Hence, the limit is 14. When looking at n-way configurations, in particular moving to a large n-way from a smaller "N" of faster processors, there are a few items that should be considered. These are: the MP factor, the speed of a single processor in the complex, the requirements of single non-MP virtual machine applications, and master processor usage. We will look at each of these in greater detail in this presentation. MP Factor
If we built operating systems perfectly, an N-way would provide the capacity of single processor times N (or more). Unfortunately, we are not perfect. There is some overhead involved in managing the multiple processors. One consideration of running on N-ways is the MP factor, which is a measure of how much potential capacity is lost when we add processors. For example, assume a workload run on a 1-way can support 100 transactions per second, but when run on a 2-way supports 180 transactions per second. The MP factor is 180/(2*100) or 0.90. You can look at this as a 2-way provides 90% of the throughput of 2 one ways. The MP factor often depends on the workload and on N. For some systems, their design may lend themselves to running well on smaller N, but then suffer large system effects as N increases. Some systems may be designed such that the step from 1 to 2-way is significant but that N > 2 does not have much of an impact. The graph above shows some examples of the MP effect for two workloads for both VM/ESA and MVS/ESA. It is interesting to note that even on the same operating system the MP factor differs depending on the workload. The workloads shown here are just examples with throughput normalized to 1 for a 1-way on all workloads. While I am biased towards VM/ESA and was selective about which workloads to plot, VM/ESA does tend to have a better MP factor than MVS. The data shown is taken from LSPR measurements. There was one MVS workload that did better than the 2 VM workloads; that was the floating point calculation workload which is really a different kind of beast. You might be wondering why the VM/ESA HT workload stops at the 8-way. This is because at this point, the HT workload becomes constrained by the master processor limitation and not the general MP factor. We will address the master processor later. Single Processor Speed
It is interesting to hear people react to ideas such as moving from a 60 mip 2-way to a 60 mip 4-way. This would involve moving from 30 mip engines to 15 mip engines. Some people's immediate reaction is response time will double. Well this probably incorrect. Response time is typically made up of more than just processor time (execution and wait). Often the I/O component can be the largest piece of response time. So if it takes 50% longer for the processor component, that might not mean response time increases 50%. On the other hand, I have also seen people ignore the fact that despite having equivalent capacity, it will take longer to execute the instructions of a single threaded application. I once explained this to a marketing type who did not like the idea of me saying that. He said, "But Bill, that's bad". "No", I replied, "That's the truth". I could also describe scenarios where being able to run multiple virtual machines in parallel can help throughput. For example, there are scenarios where starting I/Os on multiple processors in parallel could benefit I/O intensive applications. Another element to recognize is side effects. For example, if a transaction takes slightly longer because of a slower processor, the pages associated with the transaction will be required for a longer period of time. So there are scenarios where slower processors increase storage and paging requirements. Single non-MP Virtual Machine
A non-MP virtual machine is one that is not structured to exploit multiple virtual processors. In my introduction to performance presentations, I like to describe various system resources by looking at certain common attributes. One of these attributes is granularity. This applies to processor resources in that a single non-MP virtual machine is limited to 1/Nth of the system processor resources where N is the number of processors. This makes sense in that a virtual processor can only run on one real processor at a time. Therefore, you want to make sure key applications are limited by the amount of processor resource that can be delivered by a single processor. Examine the top processor using server virtual machines and applications. Remember to look for peaks. A server may average only 9% of the system, but have peaks of 35%. If you only looked at the average, you might assume it could run on a 10-way without any problems. That would lead to an unpleasant surprise. Master Processor Usage
For example 4-way with current data: CPU Total User Syst Emul 1 73 65 8 40 this is master 2 70 65 5 50 3 68 64 4 48 4 71 66 5 51
Tot_CPU_Master - Emulation_Master (73 - 40) ----------------------------------- = -------------------- = .117 All CPU on All Processors (73 + 70 + 68 + 71)
Tot_CPU_Master - Emulation_Master - AVG(Non_emulation_alternates)
------------------------------------------------------------------- =
All CPU on All Processors
73 - 40 - (20 + 20 + 20)/3 33 - 20
----------------------------- = ---------- = 0.046
73 + 70 + 68 + 71 282
VM/ESA uses a variety of methods to serialize work and protect storage in a multiple processor environment. The master processor is just one of the methods. There is one and only one master processor on a VM/ESA system, with the other processors being referred to as alternate processors. The master is typically the processor on which you IPL the system unless that processor has a unique feature (e.g. vector or crypto). CP was designed such that certain CP functions can only run on the master processor. This includes parts of spool functions and CP command processing and other processing. I use the method shown in these foils to bound the amount of master-only processor work in a system. The worse case assumes that all non-emulation processing on the master processor is master-only. The best case assumes that CP processing above the average CP processing for the alternate processors is master-only. The example shows a 4-way system with the master on processor number 1. Master-only time makes up between 4.6 and 11.7 percent of the workload. This implies that master-only capacity constraints would start to be a factor some time after an 8-way system. Another View of Master Processor
Another place to get information on master processor usage is from processor dispatching statistics kept in the monitor. In particular the high frequency state sampling done of the Processor Local Dispatch Vector (PLDV). The PLDV is a short list of users ready to run. It was created for performance reasons. There is a separate PLDV for each processor plus a special PLDV for master-only work. That is, the master processor has two PLDVs: one for master-only work and the regular PLDV. Monitor state samples these PLDVs and records both the number of times the PLDV was found empty and the count of items on the list when non-empty. Note that for the master processor the numbers recorded are for the combined items on both PLDVs. There is also a value reported in monitor called "moves to master", which some people like to track. The name is a bit misleading. It is actually a count of times we dispatch master-only work on the master processor. There is not a one-to-one correspondence to VM functions. Some master-only tasks are broken up to avoid hogging the master and therefore increase the moved to master count. Virtual MP Support
There are two approaches to creating a virtual MP machine. You can define the virtual processors in the directory so they are available when the virtual machine logs on. Or you can set up the directory so that you can use the DEFINE CPU command to add virtual processors dynamically. Note that detaching a virtual processor resets the virtual machine. Do not define extra virtual processors unless you are going to use them. Defined, but unused, virtual processors can cause performance problems. From a tuning perspective, it is important to note that the share value is distributed across the virtual machine. For example, if you have a virtual 4-way and a default share of relative 100, then each virtual processor would be scheduled as if it had a relative 25 share value. Virtual processors can be dedicated to real processors. I do not recommend mixed environments where a single virtual machine has both dedicated and undedicated virtual processors. This can result in performance anomalies that will be difficult to detect and explain. VSE/ESA Turbo Dispatcher
Some VM/VSE Turbo Thoughts
VSE/ESA MP support first went out as service in the Fall of 1995. Since then, additional enhancements have been made in the service stream. I purposely did not list APAR or PTF numbers here for fear of them becoming obsolete. Please use the normal service process to get the most current level. The VSE/ESA MP support is known as the Turbo Dispatcher. It is easy to draw some comparisons between VSE/ESA and VM/ESA, but we must remember that they are different systems. VSE/ESA dispatching is at the partition level, which you can think of as mapping back to the virtual processor level in VM/ESA. So if you have a single partition that makes up 75% of the workload, then it can be constrained on a 2-way system where it can only get 50% of the resources. With the addition of the Turbo Dispatcher, processor time is broken down into three categories: Parallel, Non-Parallel, and Spin time. Parallel is code that can run on multiple processors at the same time. Non-Parallel, also called serial or privileged code, can only run on a single processor at any given time. This is in ways analogous to VM/ESA's master processor time, except that any processor can run the VSE/ESA non-parallel code (just not at the same time). Therefore, the amount of non-parallel code can be a limiting factor. The TD (Turbo Dispatcher) can be enabled even on a uni-processor. This allows you to collect information on how much non-parallel time is in your workload. This is key information for sizing. When making the move to MP, take it slow and document things thoroughly. Having before and after data can be critical to explaining unexpected results. Documentation should include data from both VM and VSE (include CICS statistics where reasonable). 9672 Sizings
Summary
This last two foils summarize some do's and do not's for sizing of the 9672 processors and migrations to greater N-way configurations. I have seen too many critical customer situations as a result of people sizing boxes on a cheat sheet that lists a single MIPs number. Consider the service offerings from Washington System Center and elsewhere for capacity sizing and modeling, especially for the high-end water cooled to new CMOS processors. Use LSPR data. We know it's light in the VSE guest area. The LSPR VM workloads are good for CMS interactive workloads. However, for guest and database applications look at the MVS workloads as well to get a lower bound on performance. We've recently also found this to be true for some OV/VM workloads as well. When writing up performance commitments, please take time to thoroughly define "performance" (response time, throughput, processor utilization?). Recognize that their are limitations to N-way exploitation, and when those limits will be reached for the given workload. Always gather before and after data, as you make one change at a time. |
