Controlling Vertical-Low Logical Cores

(Last revised: 2017-12-18, BKW)

Abstract

To help reduce PR/SM dispatch contention, the PTF for APAR VM66063 introduces two new unparking heuristics. Both these new heuristics can help decrease PR/SM dispatch contention because they decrease the tendency of z/VM to unpark vertical-low logical cores. This article describes the new support and how to use it.

Description of the Support

Additional Unparking Heuristics

By unparking heuristic we mean the mathematical model z/VM HiperDispatch uses to decide how many logical cores to unpark.

Absent the new support described in this article, z/VM HiperDispatch unparks logical cores according to the following mathematical models:

  • When Global Performance Data (GPD) is on, z/VM decides the unparked configuration based on projected capacity. z/VM projects the minimum core capacity PR/SM is likely to be able to deliver to the z/VM LPAR if z/VM wants to consume it. z/VM then unparks the minimum number of logical cores needed to consume that minimum capacity. For example, if z/VM predicts that if the need arose PR/SM would let z/VM run as much as 2358% core-busy in the next two seconds, z/VM unparks 24 logical cores and runs that way until the next projection. The actual core consumption of the workload is irrelevant.

  • When GPD is off, z/VM decides the unparked configuration based on projected consumption. z/VM projects its workload's likely near-future core consumption maximum and then unparks enough logical cores to be able to contain the prediction plus an administrator-specified padding value. For example, if z/VM predicts the workload will be no more than 1325% core-busy over the next two seconds, and if the administrator specified a padding value of 200%, z/VM unparks 16 logical cores and runs that way until the next projection.

The problem with the GPD-on behavior is this. In cases where the CPC has plenty of spare power and each LPAR is configured with several vertical-low logical cores, the z/VM images in those LPARs unpark all their vertical-low logical cores and then run a little bit of work on each one. This en masse unparking of vertical-low logical cores can tend to promote PR/SM dispatch contention. Increased PR/SM dispatch contention can be seen in increased FCX302 PHYSLOG %LPmgt, increased FCX304 PRCLOG %Susp, increased FCX304 PRCLOG Syst, increased FCX239 PROCSUM Diag9C/sec, increased FCX104 PRIVOP Diagnose X'9C', and increased finite CPI. For more information, see our article about PR/SM overhead.

To offer relief IBM has implemented two additional unparking heuristics for the case where GPD is on. The pre-APAR heuristic is still available, too. New support in the command CP SET SRM selects the unparking heuristic to be used. The three GPD-on heuristics are now these:

  • LARGE: this is the pre-APAR behavior. z/VM unparks logical cores according to the capacity floor projection. In other words, z/VM always unparks all the vertical-high and vertical-medium logical cores; further, it also unparks all the vertical-low logical cores it predicts PR/SM will power. Mathematically stated, the unparking decision U is made based on this relation: U = capacity floor projection

  • MEDIUM: with this new heuristic, z/VM always unparks all the vertical-high and vertical-medium logical cores; further, it also unparks vertical-low logical cores, but only according to the relation min(vertical-lows-needed, vertical-lows-powered). In other words, z/VM unparks only enough vertical-lows to contain what it appears the workload will need, and further, only if it appears PR/SM will power those vertical-lows. Mathematically stated, the unparking decision U is made based on this relation: U = max ( entitlement , min ( capacity floor projection , consumption ceiling projection plus CPUPAD ) )

  • SMALL: with this new heuristic, z/VM unparks no more logical cores than consumption warrants: just enough to cover the consumption ceiling prediction plus CPUPAD, but never more than PR/SM will power. Mathematically stated, the unparking decision U is made based on this relation: U = min ( capacity floor projection , consumption ceiling projection plus CPUPAD )

The names Large, Medium, and Small evoke what happens when the respective heuristics are employed in a CPC and z/VM LPAR for which:

  • There is unentitled power available on the CPC, and
  • The LPAR's configuration includes vertical-low logical cores, and
  • The utilization in the LPAR is small enough that the consumption ceiling projection plus CPUPAD would fit into some subset of the LPAR's entitled logical cores.

Under those conditions,

  • LARGE unparks all the logical cores that are powered, in other words, all the entitled cores and some number of the vertical-lows,
  • MEDIUM unparks all the entitled cores and none of the vertical-lows,
  • SMALL unparks only the the subset of entitled logical cores needed to cover the workload plus the CPUPAD setting.

Another way to think of the names is that they express the heuristics' potential for shrinking the LPAR. For given values of consumption ceiling projection plus CPUPAD, entitlement, and capacity floor projection, the number of logical cores N that would be unparked by each of the models follows this mathematical relation: N(small) <= N(medium) <= N(large).

The PTF does not change the behavior of GPD-off unparking.

Shutting Off Vertical-Lows

To offer further relief for customers for whom vertical-lows can occasionally be troublesome, IBM has also implemented support that lets the administrator configure z/VM always to park all vertical-lows when GPD is on, no matter whether PR/SM might have capacity to power them. New support in the command CP SET SRM EXCESSUSE, a new keyword NONE, lets the system administrator turn off use of vertical-lows when GPD is on.

One envisioned use scenario for EXCESSUSE NONE is the one where PR/SM dispatch contention seems still to be a problem even though the MEDIUM or SMALL unparking heuristic is already being employed. The administrator can forcibly reduce the dispatch contention by using EXCESSUSE NONE in selected LPARs to force those LPARs to run on only their entitled logical cores. After the problem condition passes, the system administrator can use existing support in CP SET SRM EXCESSUSE to let those selected LPARs resume unparking vertical-lows.

Another envisioned use scenario for EXCESSUSE NONE involves using LPAR weights to steer CPC power to a number of competing LPARs. Suppose we have a number of LPARs with each one having far more logical cores defined than its entitlement can cover, that is, each LPAR has a large number of vertical-lows. If all such LPARs are running with UNPARKING MEDIUM and EXCESSUSE NONE, each LPAR will have all of its entitled logical cores unparked and all of its vertical-low logical cores parked. Changing the LPARs' weights will change the LPARs' total entitlements. This in turn will change the entitlements of the LPARs' individual logical cores. The z/VM images will react to the logical cores' new entitlements by adjusting the unparking so that once again all the entitled logical cores are unparked and all the vertical-low logical cores are parked. Thus by changing only the LPARs' weights we have steered power to the LPARs, with the LPARs' excessive logical core counts being of no consequence.

When EXCESSUSE NONE is in force, the LARGE and MEDIUM unparking heuristics will have the same effect, because vertical-lows will never be unparked.

Changes to z/VM Monitor

This PTF updates monitor record D5 R16 MRPRCPUP to include new field PRCPUP_SYSPRKFG that specifies which unparking heuristic is in effect. The PTF also lets new value 100 appear in field PRCPUP_CALFCONF.

This PTF updates monitor record D2 R7 MRSCLSRM to include new field SCLSRM_SYSPRKFG that specifies which unparking heuristic is in effect. The PTF also lets new value 2 appear in array SCLSRM_SRMEXUSE.

Default Behavior

After the PTF is applied, the system's default behavior is exactly as it was prior to applying the PTF, namely, UNPARKING LARGE and EXCESSUSE MEDIUM. To activate the new behavior, the administrator must opt-in, either by issuing the CP commands or by changing the z/VM system configuration file.

On z/VM 7.2 and later, the default unparking model is UNPARKING MEDIUM.

Changed CP Commands

CP SET SRM UNPARKING

CP SET SRM is enhanced to add a new option, UNPARKing, with syntax like this:

>-- CP SET SRM UNPARKing ---+-- Large ----+--------->< +-- Medium ---+ +-- Small ---+

Usage notes:

  1. If the command succeeds, the system prints a line of output: UNPARKING: { LARGE | MEDIUM | SMALL }

  2. The unparking heuristic can be set at any time, but the heuristic is in effect only when the LPAR is vertical and GPD is on.

  3. It might take up to two seconds for the selected heuristic to take effect, because z/VM makes unparking decisions only every two seconds.

  4. Specifying something other than Large, Medium, or Small produces message HCP002E.

CP QUERY SRM UNPARKING

CP QUERY SRM UNPARKING displays the setting of the unparking heuristic, with syntax like this:

>-- CP QUERY SRM UNPARKing -------------><

Usage notes:

  1. The system prints a line of output: UNPARKING: { LARGE | MEDIUM | SMALL }

  2. The output of CP QUERY SRM includes the above line of output.

CP SET SRM EXCESSUSE

The two existing variants of CP SET SRM EXCESSUSE are both enhanced to let the system administrator specify that vertical-low logical cores should always be parked when GPD is on. A new option, NONE, selects this behavior:

(Variant 1) >-- CP SET SRM EXCESSuse --+- High ------+------->< +- Medium ----+ +- Low -------+ +- None ------+ (Variant 2) >-- CP SET SRM EXCESSuse Type -- type --+- High ------+------->< +- Medium ----+ +- Low -------+ +- None ------+

Usage notes:

  1. The options High, Medium, and Low are existing support. Option None is the new support.

  2. The EXCESSUSE setting continues to affect only GPD-on unparking.

  3. Variant 1 selects the vertical-low policy for all core types.

  4. Variant 2 selects the vertical-low policy for the specified core type: ALL, CP, IFL, etc.

  5. When EXCESSUSE NONE is in effect for a core-type, if GPD is on, z/VM always parks all vertical-low logical cores of that type.

  6. The system responds with a line of output specifying the policy for each type. Previously the line of output would specify for each type whether the policy were HIGH, MEDIUM, or LOW. A fourth choice, NONE, can now appear: EXCESSUSE: CP-NONE ZAAP-NONE IFL-NONE ICF-NONE ZIIP-NONE

  7. Invalid syntax continues to produce either message HCP026E or message HCP002E as appropriate.

CP QUERY SRM EXCESSUSE

The only change here is that the choice NONE can now appear as the EXCESSUSE setting for zero or more core types. For example,

EXCESSUSE: CP-NONE ZAAP-NONE IFL-NONE ICF-NONE ZIIP-NONE

This line of output appears for both CP QUERY SRM EXCESSUSE and CP QUERY SRM.

CP SET SRM CPUPAD

The PTF changes a usage note. The CPUPAD setting is now relevant when GPD is on if the MEDIUM or SMALL unparking heuristic is selected. This is because the MEDIUM and SMALL heuristics take the system's consumption ceiling projection plus CPUPAD into account in deciding how many logical cores to unpark.

CP QUERY SRM CPUPAD

The PTF changes a usage note. The CPUPAD setting is now relevant when GPD is on if the MEDIUM or SMALL unparking heuristic is selected. This is because the MEDIUM and SMALL heuristics take the system's consumption ceiling projection plus CPUPAD into account in deciding how many logical cores to unpark.

Changed z/VM System Configuration File Statements

SRM UNPARKING

The SRM statement of the z/VM system configuration file can now specify the unparking heuristic to use. The change parallels the CP command:

>-- SRM UNPARKing ---+-- Large ----+--------->< +-- Medium ---+ +-- Small ---+

Usage notes:

  1. Absent any such statement, the system comes up with UNPARKING LARGE in effect.

  2. Specifying something other than Large, Medium, or Small produces message HCP002E.

SRM EXCESSUSE

The SRM EXCESSUSE clause of the z/VM system configuration file can now specify NONE as the EXCESSUSE value. The change parallels the CP command:

(Variant 1) >-- SRM EXCESSuse --+- High ------+------->< +- Medium ----+ +- Low -------+ +- None ------+ (Variant 2) >-- SRM EXCESSuse Type -- type --+- High ------+------->< +- Medium ----+ +- Low -------+ +- None ------+

Usage notes:

  1. Options High, Medium, and Low are existing support. Option None is the new support.

  2. Specifying something other than High, Medium, Low, or None produces message HCP002E.

Changed Monitor Records

D5 R16 MRPRCPUP is changed. The changes are:

  1. PRCPUP_MRHDRLEN is increased by four for equal configurations.

  2. A field is added at the end of the header section: Offsets Dec Hex Type Len Name (Dim) Description 60 3C Character 1 PRCPUP_SYSPRKFG Flag for how z/VM unparks when GPDC is on: 0 = unpark large 1 = unpark medium 2 = unpark small

  3. When EXCESSUSE NONE is in effect for a type,
    1. PRCPUP_CALFCONF for that type will contain decimal 100, and
    2. PRCPUP_WHIOPCX for that type will contain zero.

D2 R7 MRSCLSRM is changed. The changes are:

  1. A field is added at the end of the fixed section: Offsets Dec Hex Type Len Name (Dim) Description 128 80 Fixed 1 SCLSRM_SYSPRKFG SRM UNPARKing setting 0: large 1: medium 2: small 129 81 3 * Reserved for IBM use

  2. When EXCESSUSE NONE is in effect for a type, the SCLSRM_SRMEXUSE array entry for that type will contain a value of 2.

Impact to Performance Toolkit

Impact is confined to FCX299 PUCFGLOG:

  1. Performance Toolkit does not display PRCPUP_SYSPRKFG.

  2. If EXCESSUSE NONE is selected for a type, the EX column in the row for that type will display ** because the field is too narrow to display 100.

Support Use Information

Who Should Try This?

A customer observing unpalatably high FCX302 PHYSLOG %LPmgt, FCX304 PRCLOG %Susp, FCX304 PRCLOG Syst, FCX239 PROCSUM Diag9C/sec, FCX104 PRIVOP Diagnose X'9C', or finite CPI will want to try using the MEDIUM or SMALL heuristic, especially if he knows his LPAR configurations include more than just one or two vertical-low logical cores per LPAR. The customer should choose MEDIUM if he wants all his entitled logical cores to participate in the running of the workload, no matter how small the workload gets. Choose SMALL if it is desired for z/VM to park entitled cores when workload is light.

How To Migrate

To ease into using a different unparking heuristic, begin by setting CPUPAD to a very large value, such as 6400%, and then selecting either the MEDIUM or SMALL heuristic. When CPUPAD is very large, the MEDIUM and SMALL heuristics will both produce the same unparking result as the LARGE heuristic produces. In this way there will be no immediate change in the system's unparking behavior. Then, gradually decrease the value of CPUPAD, noting the change in the number of unparked logical cores, the change in the amount of PR/SM dispatch contention, the change in CPI, and the change in the behavior of the workload. Keep decreasing CPUPAD until a suitable result is reached. By "suitable result" we mean the balance between system overhead and workload performance seems correct.

What About EXCESSUSE NONE?

In extreme cases it might be necessary to stop z/VM from using vertical-low logical cores even though GPD is on and z/VM has detected the CPC has power available to run the vertical-lows. In such cases the customer can use CP SET SRM EXCESSUSE NONE to shut off the vertical-lows.

Another reason to try EXCESSUSE NONE is if the customer wants to implement the power-steering technique described above.

Tradeoffs

As the number of active logical cores is decreased, the tradeoff being made is among increased dispatch latency for virtual CPUs, changed CPI, and decreased PR/SM dispatch contention. As the number of active logical cores decreases,

  1. If the virtual CPUs' behavior is near to unchanged, the z/VM dispatch queues will become deeper and so it will take longer for a dispatchable virtual CPU to be assigned to a logical CPU;

  2. The decreased number of active vertical-low logical cores will decrease dispatch complexity for PR/SM and therefore decrease PR/SM dispatch contention;

  3. The decreased number of active vertical-low logical cores can help keep the LPARs of the CPC out of one another's way and thereby improve cache performance. This can be especially true if the SMALL heuristic is used.

The increase in virtual CPU dispatch latency might be only in theory for some configurations. Although in a configuration of many logical CPUs z/VM might be able to dispatch the virtual CPU on a logical CPU fairly quickly, owing to dispatch contention in PR/SM the logical CPU might experience delay getting dispatched on a physical CPU. So, depending upon the workload, there might be little to no change in overall virtual CPU dispatch latency as the count of active logical cores decreases, especially if the decrease in active logical cores is not drastic.

Summary

The PTF introduces new support in CP SET SRM and in the z/VM system configuration file. The new support lets the system administrator select an unparking heuristic that is less likely to unpark vertical-low logical cores. The new support also lets the system administrator suppress unparking of vertical-low logical cores. The system administrator can use this new support to cause the system to produce unparked configurations less likely to induce PR/SM dispatch contention.