Contents | Previous | Next

Eighty Logical Processors

Abstract

New function Eighty Logical Processors increases the processor support limit on the z14 from 64 logical processors to 80 logical processors. To check scalability we ran sets of runs using a memory-rich Linux AWM-Apache workload and sets of runs using a memory-constrained Linux AWM-Apache workload. We found most measurements scaled correctly. Customers aspiring to run large partitions with a substantial number of guests all coupled to a single layer-3 vswitch will want to proceed with caution. We did not find any other scaling problems up through 80 processors.

PTF UM35474 for APAR VM66265 implements the new function.

Customer results will vary according to system configuration and workload.

Background

New function Eighty Logical Processors increases the processor support limit from 64 logical processors to 80 logical processors. The increase is for only the z14. The increased limit recognizes some customers' desire to run partitions larger than the previous support limit.

More specifically, the support limit works like this: the largest processor number (sometimes called "processor address") supported in a z/VM partition is now x'4F', aka decimal 79. Because the selections made for partition mode, processor types, and SMT setting affect how processors are numbered, the actual number of processors present in a partition whose largest processor number is x'4F' will vary. Table 1 illustrates how the limit plays out, according to the partition mode, and the types of cores present in the partition, and the SMT setting.

Claiming z/VM supports a certain number of logical processors requires far more than just setting array sizes, bit mask lengths, and other constants so all those processors can be handled in a functionally correct way. Rather, to decide whether we can claim to support n logical processors, we run workloads to check whether the system's performance exhibits what we call correct scaling. By this we mean for selected workloads the partition gains capacity in correct proportion to the number of processors added. Now, we cannot check scaling on every possible workload, nor is it necessary for z/VM to exhibit correct scaling on every conceivable workload. Rather, we check scaling on workloads we feel would show us whether some kind of pervasive software scaling problem exists in the z/VM Control Program.

By check scaling we mean we run a given workload in larger and larger configurations in larger and larger partitions and look at how the system performs as we scale up. For example, we might make up a workload that can be run in a variety of sizes by varying the number of participating guests. We might then run the 16-guest edition in a 4-processor partition, and the 32-guest edition in an 8-processor partition, and so on. We would record ITR for each run and then graph ITR as a function of partition size. This procedure gives us a scaling curve that shows how this workload scales as we add both burden and resource. Figure 1 shows the scaling behavior of a hypothetical workload.

Figure 1. Scaling of hypothetical workload.

Notes: This hypothetical data illustrates the general shape of a scaling curve.

The Z Systems CPC itself gives us information that helps us decide whether we are observing correct scaling. As part of developing each Z Systems processor family, IBM Z Systems Processor Development runs a suite of reference workloads built in such a way that the workloads themselves contain no barriers to scaling. Those workloads are run in increasing sizes in partitions of increasing numbers of processors. All runs are memory-rich and non-SMT. Each run is done in such a way that the partition's average logical processor is about 90% busy (FCX225 SYSSUMLG). The results are tallied and then converted into an array of scalars called Capability Adjustment Factors or CAF values. For a partition of n processors we derive a corresponding CAF value c_n, 0<c_n<=1. The value c_n represents how much effective capability each of the n processors brings to the partition. Of course, for a partition of one processor, c₁=1. On the z14, for a partition of two processors, c₂=0.941; this means the two-processor partition has an effective capability of (2 x 0.941) or 1.882 units of capability. The instruction Store System Information (STSI) returns the CAF values. Figure 2 shows a plot of z14 CAF values as a function of number of processors. Remember these CAF values were derived from the behavior of IBM's reference workloads.

Figure 2. z14 CAF values.

Notes: 3906-M05 s/n E46E7, 2019-03-30. Physical cores: 4 CPs, 4 ICFs, 4 zIIPs, 128 IFLs.

As intimated above, the capability of a partition of n processors is (n x c_n). Figure 3 plots the effective partition capability for a z14 as assessed by IBM's reference workloads. For illumination, on the same graph we plotted linear scaling.

Figure 3. z14 effective partition capability.

Notes: Effective capability EC(n) = n x cn.

To compute how the capability of an m-processor partition compares to the capability of an n-processor partition, we just calculate the ratio (m x c_m) / (n x c_n). Using this technique we can draw an expectation curve for how, for example, the capabilities of 32-processor, 48-processor, and 64-processor partitions compare to the capability of a 16-processor partition. Table 2 illustrates how this is done. We'll use such curves later, in the results section of this chapter.

The most important point to keep in mind when using CAF values is that every workload scales differently. The CAF values returned by STSI are indicative of the scaling behavior of only IBM's reference workloads. Different workloads will scale differently; in fact, z/Architecture Principles of Operation recognizes this with a programming note found in the description of STSI: "The applicable MP adjustment factor is an approximation that is sensitive to the particular workload." In other words, we cannot expect just any workload to scale the way the CAF values predict. Different workloads scale differently.

Of what use is the effective capability curve, then? The curve shows us the approximate shape of a successful scaling set. We are looking for gradual rolloff with no sharp inflection points. Figure 4 shows us hypothetical successful and unsuccessful runs.

Figure 4. Hypothetical successful and unsuccessful scaling.

Notes: This hypothetical data shows the difference between a workload that scaled well and a workload that did not scale well.

When a workload does not scale, we look to see what failed. To find the failure, we use MONWRITE data, z/VM Performance Toolkit reports, the CPU MF host counters, the lab-mode sampler, and anything else that might help. Common kinds of failures include use of algorithms that are too computationally complex, or excessive use of serialization, or inadvertent misuse of memory as relates to cache lines. When we find such failures, we make repairs as needed to meet the scaling goal.

The CAF curve and the effective capability curve tell us more than just what acceptable scaling looks like. At higher values of n, c_n decreases to a point where it is clear there is a tradeoff to be made between the administrative convenience of having one large partition and the cost per unit of effective capability. For example, consider Table 3, which shows the capabilities of a number of different z14 deployments that exploit 80 processors in total.

A single large partition might be easier to maintain and administer, but the capability is less than the total capability of a number of smaller partitions. Customers considering large partitions will want to balance their need for low administrative cost against their need for efficient hardware exploitation. Further, though we don't like to think about them, the consequences of an outage of a single large partition as compared to those arising from the failure of only one of a number of smaller partitions also need to be considered.

Method

The studies presented here were all done with the Apache workload. A number of Linux guests running Application Workload Modeler (AWM) were configured to send HTTP transactions to a number of Linux guests running the Apache web server. All clients and all servers resided together in one single partition.

Two different arrangements of clients and servers were used.

• The first was a monolithic arrangement where every client guest directed HTTP transactions to every server guest. All client guests and all server guests were coupled to a single vswitch. To ramp up the workload we added client guests, server guests, processors, and central memory. Figure 5 illustrates the configuration.

  Figure 5. Monolithic Apache workload.

• The next was an arrangement consisting of a number of small groups of client guests and server guests, the members of each group connected to one another through a group-specific vswitch. We called such an arrangement a tiled arrangement. Each tile consisted of some clients, some servers, and a vswitch. All vswitches' uplink ports were connected to the same OSA chpid. Each client guest sent HTTP transactions to all the server guests on its tile. To ramp up the workload we added tiles, processors, and central memory. There was no inter-vswitch traffic during the steady-state portion of the measurement. Figure 6 illustrates the configuration.

  Figure 6. Tiled Apache workload.

All measurements were collected on a 3906-M05. The partition was always dedicated, with all IFL processors. No other partitions were ever activated. The number of processors was varied according to the needs of the particular workload being run.

The z/VM CP load module used was a z/VM 7.1 module containing a pre-GA edition of the PTF. The pre-GA edition let us run beyond 80 logical processors. z/VM was always configured for non-SMT operation, and for vertical mode, and for UNPARKING LARGE.

All clients and all servers ran SLES 12 SP 3. In all cases the Linux guests were configured so they would not need to swap. Each guest had a virtual NIC coupled to a vswitch appropriate for the measurement.

Runs were done in both memory-rich and memory-constrained configurations.

• For the memory-rich runs,
  • All guests were virtual 1-ways with 1024 MB of guest real.
  • The HTTP transactions sent were configured for what we call two small files. What this means is that every HTTP transaction sent from an AWM client to an Apache server was an HTTP GET of either one or the other of two small ballast files held in the Linux server's file system. In this way each server guest will retain the two ballast files in its Linux file cache.
  • The numbers of clients, servers, and processors were chosen so that at the low end of the scaling set, FCX225 SYSSUMLG showed the average logical processor to be in the neighborhood of 90% busy. As we ramped up the configuration it was inevitable that the average logical processor would become more than 90% busy.
  • Central memory was always set far above the needs of the workload.
• For the memory-constrained runs,
  • The client guests were all virtual 4-ways with 4096 MB of guest real. The server guests were all virtual 1-ways with 5 GB of guest real.
  • HTTP transactions sent were configured for what we call 5,000 1-MB files. What this means is that every HTTP transaction sent from an AWM client to an Apache server was an HTTP GET of one out of 5,000 possible files, each file 1 MB in size. In this way, we drove paging I/O and also virtual I/O.
  • FCX225 SYSSUMLG would not approach 90% busy because of limits in our paging subsystem.
  • Central memory was chosen to create a paging rate consistent with the highest paging rates we could remember seeing in recent customer data.
  • All runs were primed in the sense that the run was done twice back-to-back and the second run was the one from which measurements were collected.

The above configurations check scalability of basic z/VM CP functions such as scheduling, dispatching, interrupt handling, and handling of privileged operations. They also check scalability of intra-vswitch traffic. The memory-constrained configurations check scalability of virtual I/O, of memory management, and of paging I/O.

Results and Discussion

This section presents results for the memory-rich and memory-constrained evaluations.

Memory-Rich Workloads

Our first set of memory-rich runs was of monolithic Apache. For the base run we chose 9 clients, 36 servers, and 19 processors because this combination put us close to the target of running the average logical processor 90% busy. We used a layer-3 vswitch to connect the guests together. From that base run we ramped up, adding clients, servers, and processors, all in proportion. We ran this up to a 95-processor configuration. Table 4 shows the findings.

The ITRR (ITR ratio) values show that up to 57 processors, results exceeded the CAF prediction. But between 57 processors and 76 processors, ITR rolled off. We investigated this and found there was a code problem in the implementation of layer-3 vswitches. The code problem involved several aspects. First, associated with the layer-3 vswitch there was a list of control blocks that had to be searched repeatedly. The length of the list was equal to the number of VNICs coupled to the vswitch. Second, the control blocks on that list were being updated right after being found. Third, the control block's list pointers were in the same cache line as the control block field being updated. These factors conspired to slow down the run in the higher N-way configurations.

In studying the vswitch implementation we concluded we could work around the layer-3 problem if we were to move to a tiled arrangement, so that the number of VNICs coupled to any given layer-3 vswitch would be somewhat smaller. This led us to our second set of runs, illustrated in Table 5.

Seeing the 76-processor tiled run did not show the problem, we were confident our tiled approach was a successful workaround. Thus we did not bother to repeat the 95-processor run as a tiled one.

During our study of the layer-3 vswitch problem we learned some other important things. First, we found that layer-2 vswitches are implemented differently than layer-3 vswitches are. In particular, there are no linear searches in the layer-2 implementation. Second, we found that IBM recommends vswitch deployments that use a number of layer-2 vswitches, for reasons such as failover. Third, we found that surveys of customers suggest most customers are using layer-2. For these reasons we saw no need to repeat the monolithic set with a layer-2 vswitch. We proceeded to try a set of tiled layer-2 runs. This set is illustrated in Table 6.

Comparing the tiled layer-2 ITR values to the tiled layer-3 values showed that layer-2 vswitches are a better performer in this workload. Also, the layer-2 ITRR values exceeded the layer-3 ones. So not only is layer-2 a better performer in our workload, it also scales better.

Next we noticed the logical processor percent-busy values in the layer-2 set were higher than we would have liked. We remembered that IBM's reference workloads for processor scaling are run in a fashion that tries to hold percent-busy to about 90%. We tried adjusting the shape of the tile so as to bring percent-busy down a little. We changed the shape of the tile so that in the largest run we would be about 90% busy. This set is illustrated in Table 7.

In the above set we found ITRR did not do as well as the effective capability curve predicted. We feel this is because the ITR in the first run is optimistically high because percent-busy is too low. In other words, because we know the processor is more efficient at lower percents-busy, we also know a low percent-busy in the first run of the set will set the ITRR denominator too high for the subsequent larger runs and thereby depress the larger runs' ITRR values. Even so, we found the shape of the ITRR curve to be OK.

Figure 7 plots the effective capability curve and the ITRR curves for all the memory-rich runs. Up to the vicinity of 80 processors, all runs tracked at or near the effective capability curve.

Figure 7. Scaling of various Apache memory-rich workloads.

Notes: Data from Table 4, Table 5, Table 6, and Table 7.

Memory-Constrained Workload

Next we devised a tile and a partition configuration that together would produce the paging rate we wanted. This tile consisted of 5 clients, 60 servers, and one layer-2 vswitch, run in 16 processors and 256 GB of central memory. We then ramped this up by tiling, with the last size being 80 processors. Table 8 shows the results.

The above table shows a very interesting set of runs. First up is to look at why ETR did not scale. The reason is that the paging activity was too much for our paging DASD to handle. We used the z/VM Performance Toolkit DEVGROUP facility to look at the behavior of our paging volumes -- 224 3390-54 volumes spread over 16 LCUs, all in one storage server -- as we scaled up. From FCX108 DEVICE we found paging volume performance dropped rapidly as we scaled up. The rise in disconnect time suggests trouble in the storage server read cache. FCX176 CTLUNIT, not shown, corroborated this; in the smallest run we had a 96% read hit rate in the storage server, but in the largest run the read hit rate dropped to 58%.

 
(From FCX108 DEVICE)
 
          <-- Device Descr. -->  Mdisk Pa- <-Rate/s-> <------- Time (msec) -------> Req. <Percent>
Run       Addr Type   Label/ID   Links ths  I/O Avoid Pend Disc Conn Serv Resp CUWt Qued Busy READ
 
AP1Z604G  PAGING                  ....  ..  180    .0 .147 .479 .229 .855 1.50 .000  .12   15   97
AP2Z604G  PAGING                  ....  .. 89.4    .0 .156 2.77 .271 3.19 6.46 .000  .29   29   93
AP3Z604G  PAGING                  ....  .. 48.0    .0 .208 7.39 .418 8.02 18.0 .000  .47   38   86
AP4Z604D  PAGING                  ....  .. 43.4    .0 .240 9.58 .483 10.3 25.2 .000  .66   45   84
AP5Z604G  PAGING                  ....  .. 45.5    .0 .241 10.5 .480 11.2 30.8 .000  .89   51   84
 

Next up is to look at how the runs' ITRR curve tracks the effective capability curve. Figure 8 shows the two curves.

Figure 8. Scaling of Apache DASD Paging (ADP) memory-constrained workload.

Notes: Data from Table 8.

The runs' ITRR curve tracks not too far below the effective capability curve. However, in looking at Table 8 we noticed logical processor average busy decreased as partition size increased. This relationship means the ITR values in large partitions might be optimistic compared to the ITR produced in the base run, and so the scaling picture presented in the graph might be optimistic.

We also noticed that percent-busy isn't all that high in these workloads, and further, system T/V is climbing as the partition gets larger. When we see these two situations together, our thought is usually that CP is struggling to deal with the situation that there isn't all that much CP work to do compared to the number of processors trying to do it. To check this out we calculated and plotted guest CPU-busy per transaction and CP CPU-busy per transaction as a function of run size. Figure 9 shows the two curves.

Figure 9. Tiled Apache DASD Paging, guest busy/tx and CP busy/tx.

Notes: Data from Table 8.

The graph shows that guest/tx and CP/tx track pretty well to each other. The largest run does show an uptick in CP/tx. This suggests CP is feeling the effect of the wide partition.

At the high end we noticed we were running the average logical processor around 20% busy and all 80 logical processors were engaged. Knowing PR/SM was having to dispatch all those logical processors on physical processors, and seeing that the partition wasn't terribly busy, we decided to try UNPARKING SMALL to see whether we could get comparable results while disposing of apparently excess logical processors. Table 9 shows the result.

Figure 10 shows ETR and ITR of the UNPARKING LARGE set and of the UNPARKING SMALL set.

Figure 10. Tiled Apache DASD Paging, effect of UNPARKING SMALL on ETR and ITR.

Notes: Data from Table 8 and Table 9.

Aside from one deviation, the performance experience with UNPARKING SMALL was not much different from the experience with UNPARKING LARGE. The big difference is in the dispatch burden the two sets exerted upon PR/SM. In the UNPARKING LARGE cases, PR/SM had to juggle all of the partition's logical processors. In the UNPARKING SMALL cases, PR/SM had to dispatch only the unparked logical processors. In other words, parking unneeded logical processors freed up physical cores for PR/SM to use for other purposes, such as to run other partitions. In the largest UNPARKING SMALL run, z/VM gave about 60 physical cores back to PR/SM with no appreciable performance impact. Figure 11 shows the difference.

Figure 11. Tiled Apache DASD Paging, number of unparked processors.

Notes: Data from Table 8 and Table 9.

There are some other effects of UNPARKING SMALL worth exploring. One effect worth looking at is average logical processor percent busy on the unparked processors. Table 9 is a little misleading in that it suggests the unparked logical processors were in the neighborhood of 25% busy. This isn't true. Table 10 shows average busy per average unparked processor. UNPARKING SMALL squeezes the work into the smallest number of logical processors it seems would suffice to run the workload.

Another effect worth looking at is the change in CPU/tx. Figure 12 shows how CPU/tx in guest and CP changed with unparking model. In UNPARKING SMALL, CP had to deal with far less parallelism and so we could expect its CPU/tx to decrease, because it would spend less time spinning on locks. FCX326 LOCKACT, not shown, corroborated this. At the same time, the guests, squeezed into fewer, much busier logical processors, would experience far more cramped quarters and so we could expect their CPU/tx to rise from cache effects. In our particular case, the increase in guest CPU/tx was just about balanced out by decrease in CP CPU/tx. This is why the ITR curves of the LARGE and SMALL sets are nearly the same.

Figure 12. Tiled Apache DASD Paging, effect of unparking model on CPU/tx.

Notes: Data from Table 8 and Table 9.

Another effect of UNPARKING SMALL is change in dispatch latency. From queueing theory we know that when the service points become more busy, those waiting in line wait longer to be served. z/VM tracks the wait time dispatchable virtual CPUs spend waiting to be dispatched. The data comes out in D4 R3 MRUSEACT records. z/VM Performance Toolkit does not tabulate the data but IBM has a tool that does so. Table 11 illustrates what the tool found for the UNPARKING LARGE set and for the UNPARKING SMALL set. We can see that in the UNPARKING SMALL set, dispatchable virtual CPUs spent more time waiting for dispatch. However, transaction response time was not affected. This means transaction response time was dominated by some other factor, likely the intense paging.

Summary

Eighty Logical Processors provides support, on the z14, for up to 80 logical processors in a partition. To check scalability, we ran memory-rich and memory-constrained workloads. In the workloads we ran, the system usually scaled correctly. The notable exception was when we tried to put all clients and all servers in a large partition onto a single layer-3 vswitch. The workaround there was to change to a tiled configuration.

Our memory-constrained workload had modest CPU utilization in a very large partition and so it gave us an opportunity to exercise UNPARKING SMALL. Using UNPARKING SMALL caused z/VM to release physical cores to PR/SM, thereby giving PR/SM the opportunity to use those cores for other purposes. However, because CPU utilization on the unparked processors was so much larger than it was in the UNPARKING LARGE case, virtual CPU dispatch wait time increased. For our workload the increase was of no consequence, but for other workloads the increase might have consequences, such as affecting transaction response time. Customers pondering UNPARKING SMALL will want to collect measures of application success, such as ETR, transaction response time, and conformance to SLAs, before and after, so they can find the correct balance for their environments.

Contents | Previous | Next