IBM Systems  >   System z  >   z/VM  >

Understanding z/VM CPU Utilization

(Last revised: 2011-03-23, BKW)

In its most general sense, the phrase "CPU utilization" refers to the in-use fraction of a computing system's capacity to run instructions.

In a stacked environment where virtual servers run on z/VM, which in turn runs in a logical partition, which in turn runs on a physical System z computer, the phrase "CPU utilization" can often lead to more confusion than illumination.

In this article we want to discuss the several places where the z/VM stacked environment reports CPU utilization, and what exactly each one of those reported numbers means, and how to interpret the various reports. We will begin with how to discuss the utilization of an entire System z CEC. Following, we will refine to how to talk about the utilization of a single partition, and then about the guests running within a partition. We'll conclude with a brief summary and then offer a reference table that captures all of what we've explained in this article.

Conventions

A System z computer is outfitted with a number of physical CPUs, organized into types: standard CPs, IFLs, zAAPs, zIIPs, and ICFs. The typical System z machine comes with one or more engines of each type.

When we discuss the utilization of the physical machine, we usually talk about the utilizations of the engine types separately. For example, we discuss the utilization of the standard CPs, and of the IFLs, and so on.

Also when we discuss these utilizations, we usually use percentage expressions. Such percentages always use the standard that "100%" means "one entirely consumed physical engine". So, for example, if we say the IFLs on a machine are "685% utilized", it means that 6.85 physical engines' worth of IFL capacity are being consumed.

This standard about the meaning of "100%" holds true for almost all of the numbers we will describe in this article. We will clearly caution the reader otherwise when we are discussing an exception case.

Logical PUs as Consumers

The System z PR/SM hypervisor makes possible the dividing of a physical System z computer into disjoint computing zones called logical partitions. These partitions are equipped with logical PUs, which PR/SM dispatches on physical PUs so as to run the partitions' workloads.

Recognizing this partitioning scheme, and realizing that PR/SM itself will also consume some CPU for its own ends, we can break down the consumption of System z physical CPU time into three very specific buckets:

1. Cycles consumed by partitions' logical PUs, running their own instructions.

2. Cycles consumed by the PR/SM hypervisor, running its own instructions, but running them in direct support of the deliberate action of some specific logical PU, and consequently accounting the consumed cycles to said logical PU as overhead the logical PU caused or induced.

3. Cycles consumed by the PR/SM hypervisor, running its own instructions, but doing work not directly caused by, and therefore not chargeable to, any given logical PU.

The PR/SM hypervisor keeps counters that measure these three kinds of CPU time. It accounts the first two kinds of time on a per-logical-PU basis. It accounts the third on a per-physical-PU basis.

Periodically z/VM asks PR/SM for the values of all of these counters. z/VM then dumps those counters into a binary data stream called "CP Monitor data". The IBM-supplied utility program called MONWRITE can journal this binary data stream to disk or tape for analysis later; we call such recorded data "MONWRITE data". A reduction and reporting program such as z/VM Performance Toolkit — known herein as "Perfkit" — can report on what the MONWRITE data reveals.

For those of you interested in details, PR/SM's CPU time accumulators appear in these CP Monitor records:

1. D0 R16 MRSYTCUP reports PR/SM's CPU accumulators for exactly one partition. The record first contains a small prologue that describes the partition to which the record applies. The record continues with a sequence of clauses, one clause per logical PU defined for the partition. Each clause identifies the logical PU and conceptually contains two accumulators: one that tells how much CPU time the logical PU has used for itself, and a second that tells how much CPU time the logical PU has induced in PR/SM. A typical CP Monitor data stream would contain a herd of p D0 R16 records every monitor interval, one such record for each of the p partitions defined on the physical machine.

2. D0 R17 MRSYTCUM reports the nonchargeable PR/SM CPU time for each physical PU on the machine. The record first contains a small prologue describing the physical machine. The record continues with a sequence of clauses, one clause per physical PU. Each clause identifies the physical PU and contains one accumulator that tells how much CPU time PR/SM itself used on the physical PU for nonchargeable overhead work. A typical CP Monitor data stream would contain exactly one D0 R17 record per monitor interval.

For those of you with z/OS backgrounds, CP Monitor records are analogous to what z/OS calls "SMF records". The records contain binary data meant for consumption by a reduction program of some kind. On z/OS, that reduction program is called RMF. On z/VM, the reduction program is called z/VM Performance Toolkit, or herein, "Perfkit".

Dedicated Partitions

A dedicated partition is one for which the PR/SM hypervisor has set aside specific physical engines for said partition's exclusive use. For example, if a partition were four-way dedicated with standard CPs, PR/SM would set aside four physical standard CPs and run nothing on them except said partition's four logical standard CPs. This dedication function is great for the anointed partition, but physical CPU cycles the partition doesn't consume are lost to the ages.

Because of this exclusivity property, when PR/SM reports logical PUs' consumption of cycles, it reports "100% busy" for every dedicated logical PU. If we want to know how busy a dedicated partition really is, we have to look at CPU utilization data collected by the operating system running in the dedicated partition. As far as PR/SM is concerned, those logical PUs are 100% busy, and PR/SM reports each one as consuming a whole physical PU.

For any given partition, all of its logical PUs are either dedicated or shared. For example, there is no notion of a z/VM-mode partition where the logical standard CPs are dedicated but the logical zIIPs are shared.

Unfortunately, the System z community's conventional use of the adjectives "dedicated" and "shared" is kind of upside-down. In truth, it's the System z's physical engines themselves that are either "dedicated" to a specific partition or "shared" among multiple partitions. (Think about it: when four people celebrate a birthday by together eating a cake, it's the cake that's said to be "shared", not the people.) Even so, System z fans always use these two particular adjectives to describe partitions and logical PUs. A "shared partition" has its logical PUs time-sliced onto physical PUs. A "dedicated partition" gets its own physical PUs. Do not let this upside-down use of terminology sidetrack you.

Perfkit's Reporting of Partitions' Utilization

The PR/SM counters break out consumption by logical PU, so it makes sense that we can tabulate consumption by logical PU, or by partition. A couple different Perfkit reports give us the lenses we need. Here is a description of the reports and some discussion of their strengths and weaknesses.

FCX126 LPAR is the most complete, most detailed, most voluminous, and perhaps most useful of this family of reports. If you like granularity and details, this report is for you. For each logical PU of each partition, the FCX126 LPAR report displays two columns derived from PR/SM's CPU accumulators:

1. The "%Busy" column is the total activity PR/SM charged to the logical PU. In other words, %Busy accounts for both the logical PU's own work and the PR/SM overhead the logical PU induced.

2. The "%Ovhd" column accounts for only the PR/SM overhead the logical PU induced.

(We will talk about the other FCX126 columns, namely "%VMld", "%Logld", and "%Susp", later.)

The %Busy and %Ovhd percentages are both out of 100%, where "100%" again means "one physical engine's worth". A value of 100% in %Busy would mean the logical PU were either (a) part of a dedicated partition, or (b) completely busy either running its own work or inducing overhead in PR/SM.

One of the strengths of the FCX126 LPAR report is that it shows us separate utilization values for every logical PU of every partition on the whole CEC. In fact, the interim version of the report, FCX126 INTERIM LPAR, gives us said breakout on a time-interval by time-interval basis. Because of this granularity, it is very easy to use FCX126 LPAR to see a runaway, overburdened, underused, or stalled partition, or even to see such a logical PU, no matter its partition.

Keep in mind the FCX126 LPAR report's %Busy and %Ovhd values are the average utilization values over the time interval described by the report. The described time interval is annotated in the upper left hand corner of the report. For interval-by-interval studies, use FCX126 INTERIM LPAR.

A weakness of the FCX126 LPAR report is that it can be somewhat voluminous and consequently overwhelming to digest. Also, rarely do we need to see the logical PUs' utilizations individually.

Here is a excerpt of the FCX126 LPAR report, so you'll recognize it in your own Perfkit reports. This excerpt is edited somewhat, for size, relevance, and appearance on an HTML page.

FCX202 LPARLOG is another report that comments on partitions' CPU utilization. This report is a time-indexed, interval-by-interval summary of the CPU time PR/SM charged to partitions, rolled up by partition instead of broken out by logical PU.

An important trait of FCX202 LPARLOG is that for each partition, LPARLOG adds up the utilizations for the partitions' logical PUs and then divides, to compute the utilization for the average logical PU in the partition. This trait can be either a strength or a weakness:

1. If the partition is homogeneous, the displayed average %Busy and %Ovhd values are usually pretty indicative of the behavior of the individual logical PUs in the partition. Another way to say this is that generally speaking, operating systems running in homogeneous partitions tend to spread their work fairly evenly.

2. However, if the partition is heterogeneous (sometimes also called "mixed-engine"), beware that the FCX202 %Busy and %Ovhd columns DO still report averages, but the individual logical PUs' utilizations might vary widely from one PU type to another. For this reason, for mixed-engine partitions, it's probably better to use FCX126 LPAR to study them.

FCX202 LPARLOG reveals on its right-hand side whether the partition is mixed-engine: the "Type" column will say MIX. For a homogeneous partition, the "Type" column will report the engine type: IFL, CP, or whatever.

(Once again, we will discuss "%VMld", "%Logld", and "%Susp" later.)

Here is a excerpt of FCX202 LPARLOG, again, so you'll recognize it in your own Perfkit reports.

That's pretty much it for the Perfkit reports that show us what PR/SM had to say about CPU utilization.

z/VM's Own Accounting of CPU Utilization

As the z/VM Control Program runs, it keeps track of how it spends its time on each of its logical PUs. z/VM accrues each logical PU's time into four buckets:

1. Bucket 1: time the logical PU spends running guests. z/VM fans call this "guest time", "virtual time", or "emulation time". All three terms mean the same thing.

2. Bucket 2: time the logical PU spends running the Control Program, doing overhead work in support of a specific action taken by a specific guest. Some folks call this "CP time".

3. Bucket 3: time the logical PU spends running the Control Program, doing overhead or system management functions not attributable to the direct actions of, and therefore not chargeable to, any guest. Some folks call this "system time".

4. Bucket 4: time the logical PU spends with a wait PSW loaded. Almost everybody calls this "wait time".

D0 R2 MRSYTPRP contains z/VM's four time accumulators. At each CP Monitor interval, z/VM cuts a herd of these records, one record for each logical PU in the partition.

Perfkit's Reporting of z/VM's Own Accounting of CPU Time

Several different Perfkit reports comment on z/VM's own CPU time accountings. Here's a description of the reports, again, with their strengths and weaknesses.

FCX126 LPAR again is the most granular, most detailed member of this family of reports. Notice that for the partition that collected the MONWRITE data, the LPAR report includes, for every logical PU in the partition, columns labelled "%VMld" (say "percent-VM-load") and "%Logld" (say "percent-logical-load"). These two percentage columns are derived from the four z/VM buckets we described above, calculated as follows:

1. %VMld is the percent utilization of a physical engine z/VM believes it achieved on the logical PU. A value of 100% would mean that z/VM was able to use that logical PU to consume one physical engine's worth of power. Another way to think of this is that %VMld is just the sum of the first three z/VM buckets named above: guests, plus induced Control Program, plus nonchargeable Control Program.

2. %Logld is more complicated. %Logld expresses the percent of z/VM-accounted time that is not z/VM-accounted wait time. In other words, the formula is 100 * (b1+b2+b3) / (b1+b2+b3+b4). %Logld is not an expression of consumption of a physical PU. Rather, %Logld merely expresses the fraction of the z/VM-accounted time that is not z/VM-accounted wait time.

Returning to the FCX126 LPAR excerpt above, we notice that %VMld and %Logld are filled in for every logical PU. It's important to note that this particular excerpt is only the little piece of FCX126 LPAR that applies to the partition that collected the MONWRITE data. For the other partitions on the CEC, FCX126 LPAR of course reports no values in these columns.

Also notice in the excerpt that %VMld is very close to the difference (%Busy - %Ovhd). Recalling the definitions of %Busy and %Ovhd, this makes perfect sense. Any discrepancy is explained by the idea that two different entities — z/VM and PR/SM — are accounting the very same phenomenon.

Again, FCX126 LPAR's granularity and detail are both its strength and its weakness. The strength is that for each logical PU, the report shows all of the percentages derivable from either the PR/SM-maintained counters or the z/VM-maintained counters. The corresponding weakness is the bulkiness of the report.

FCX144 PROCLOG also reports on the logical PUs' utilizations, again, using these four z/VM time buckets, but combined a little differently.

1. The "Emul" time is time spent running guests, aka bucket 1 above.
2. The "User" time is the sum of bucket 1 and bucket 2, in other words, time either used directly by guests or directly chargeable to them as Control Program overhead they caused.
3. "Syst" time is bucket 3: nonchargeable Control Program overhead.
4. "Total" time is the sum of "User" and "Syst".

Each of these ranges up to 100%, again, "100%" meaning "one entire physical engine's worth".

A strength of FCX144 PROCLOG is that it reports on only the collecting partition. This makes it somewhat smaller than FCX126 LPAR. A weakness of PROCLOG is that it doesn't present PR/SM's counters alongside z/VM's counters, so some of the utilization picture is missing.

Here is a small excerpt from FCX144 PROCLOG.

FCX239 PROCSUM contains one column, "Pct Busy", that reports on the average CPU-busy across all logical PUs in the partition, time-interval by time-interval, again, using z/VM's first three buckets. Again, "100%" would mean "one entire physical engine's worth". The averaging PROCSUM does can be handy, but like FCX202 LPARLOG, it's misleading in mixed-engine environments.

FCX225 SYSSUMLG reports the same average CPU-busy value as FCX239 PROCSUM, also time-interval by time-interval, and with the same strengths and weaknesses. A nice property of SYSSUMLG is that it also includes, interval by interval, a wide assortment of system performance metrics drawn from CPU consumption, I/O, paging, and other interesting behaviors. The variety present in FCX225 SYSSUMLG makes it a good first stop for examining a system's basic performance properties.

FCX100 CPU is similar to FCX144 PROCLOG in that it reports on each logical PU separately. For each logical PU, we see %EMU (bucket 1), %CP (bucket 2), %SYS (bucket 3), %WT (bucket 4), %CPU (sum of %EMU, %CP, and %SYS), and %LOGLD (same as "%Logld" on FCX126 LPAR). The FCX100 CPU values are averages over the time interval of the report, notated in the report's upper-left corner.

FCX101 REDISP (format 1) presents the four z/VM buckets a little differently. The thing to remember here is that for FCX101 REDISP, the columns are summations over all of the logical PUs in the partition. So, the "CPU" column is the grand total of the first three z/VM buckets, across all logical PUs, over the time interval of the report, expressed as a percent; again, "100" would mean one physical engine's worth. The %EM column matches up to bucket 1, %CP to bucket 2, %SY to bucket 3, and %WT to bucket 4.

The Notion of Suspend Time

By now you have undoubtedly noticed that FCX126 LPAR and FCX202 LPARLOG contain a column called "%Susp". This column, called "suspend time", deserves special explanation, because it is widely misunderstood.

We can begin to understand %Susp by taking another look at those four z/VM buckets described above and reported in D0 R2. Again, those buckets record, for each logical PU, the amount of time the logical PU spends in the four states z/VM itself can see: the three distinct kinds of running, and time spent with a wait PSW loaded.

Now, here's the thing about those four D0 R2 counters. At each interval, the counters might not add up to 100%. In other words, time appears to be missing. Why? Here's why. z/VM measures those four values using a System z facility called the "CPU timer". Here's the thing about a logical PU's CPU timer: it doesn't advance when the logical PU is not dispatched. So, the logical PU's undispatched time isn't recorded in any of those four z/VM buckets.

So, we arrive at the definition of %Susp, or "suspend time". %Susp is just 100% minus what the four z/VM buckets account for. That's it. No CP Monitor counter directly reports %Susp. Perfkit just calculates %Susp, by starting with 100% and subtracting out what the four z/VM buckets account for.

To understand the significance of %Susp, we need to examine the reasons why PR/SM would decline to dispatch a logical PU. What might those reasons be?

1. The logical PU was ready to run, but PR/SM couldn't find a place to run it, because all physical PUs of the matching type were busy. In other words, you might need a bigger CEC.

2. The logical PU did something that induced PR/SM overhead, and PR/SM responded, running its own instructions and charging them to the logical PU. In other words, %Ovhd is accruing.

3. The logical PU invoked some PR/SM function that resulted in the logical PU becoming temporarily undispatchable. For example, the logical PU issued a Diag x'44' to give up its PR/SM time slice. The z/VM spin lock manager does this sometimes. When this is happening in excess, you are going to see large values in FCX265 LOCKLOG.

4. The logical PU is running in a capped partition, and guess what, the partition has used its entitlement. Some time soon the logical PU will get to run, but right now PR/SM has applied the brakes. When this is happening, you will see in FCX126 LPAR that the partition is capped, and via some quick manual arithmetic, you'll see that the partition has used its entitlement.

The misunderstanding of %Susp usually happens because someone forgets there are so many reasons why it can accrue. The biggest misconception is that %Susp should be able to accrue only if the corresponding physical PU pool is entirely utilized. This is patently false. The most common cause of %Susp is that the logical PU itself does something which makes it temporarily undispatchable, such as issuing a Diag x'44' or x'9C'. When a logical PU uses these diagnoses to excess, %Susp will start to show up. Unfortunately, no observer counts a logical PU's use of these diagnose codes. We can infer their presence by looking at FCX265 LOCKLOG, and to a lesser extent, fields such as FCX101 REDISP %SY.

Finally, a historical note. VM performance reporting products of ages past used to call our "suspend time" by another term, "involuntary wait", and used column headings such as "%IW" to denote it. Some readers might find this old term illuminates their understanding of what's occuring. For that reason alone, we mention it here.

The Whole Picture

At this point we've explained everything about all five interesting columns on FCX126 LPAR and FCX202 LPARLOG:

1. %Busy and %Ovhd, which come from PR/SM's timers;
2. %VMld and %Logld, which come from z/VM's timers;
3. %Susp, which is a value synthesized by Perfkit.

We've also explained the PROCLOG, PROCSUM, SYSSUMLG, CPU, and REDISP views of CPU consumption, and how Perfkit calculates them from z/VM's D0 R2 timers.

This pretty much wraps up the discussion of how Perfkit expresses CPU utilization, for partitions and for whole CECs.

Remember, with the exception of %Logld and %LOGLD, all of the columns we've discussed have the standard that "100%" means "an entire physical engine's worth".

The Notion of T/V Ratio

When people discuss the notion of "z/VM overhead", one of the ways they do it is to talk about a computed health metric called "system total-to-virtual ratio", or "system T/V ratio". This number is the ratio of total CPU time used to virtual time used, as z/VM sees things. Another way to think of this is by returning to those four z/VM D0 R2 time buckets we mentioned earlier. System T/V ratio is just the total run cycles z/VM accounts, divided by guest run cycles z/VM accounts, or formulaically, (b1+b2+b3) / b1.

When system T/V is 1.0, it means all of the z/VM-accounted time was spent running guests. This is the perfect world: no cycles spent running the Control Program.

When system T/V is 2.0, it means for every second spent actually running guests, the z/VM Control Program spent one second running itself. Such a high T/V would usually be a sign of distress and would be a call to do some diagnosis and tuning.

System T/V ratio is very dependent on workload characteristics. Typical values for system T/V are in the 1.0 to 1.2 range.

The FCX225 SYSSUMLG and FCX239 PROCSUM reports both have a "T/V" column, calculated exactly as described here.

Guest CPU Utilization

To round out this article, we should talk about where Perfkit reports on guest CPU utilization, and what the reported numbers mean.

FCX112 USER is very useful in understanding guest CPU utilization. Four of its columns are particularly interesting. The columns apply to the time interval of the report, as notated in the report's upper-left corner. The columns are:

1. TCPU: the total CPU-seconds either consumed directly by the guest's virtual PUs or charged to the guest's virtual PUs by the Control Program as billable overhead;

2. VCPU: the total CPU-seconds consumed directly by the guest's virtual PUs;

3. %CPU: TCPU divided by the time interval of the report. Here again, "100%" means "one physical engine's worth";

4. T/V: user T/V ratio, aka TCPU divided by VCPU.

Here is a small excerpt from FCX112 USER.

One nice property of the FCX112 USER %CPU column is that one can use it in conjunction with the FCX226 UCONF user configuration report to determine whether a guest has too many virtual CPUs. If a guest's FCX112 USER %CPU value is, say, 125% but FCX226 UCONF reports the guest is a virtual six-way, one might want to reconsider whether the guest might operate better as a virtual two-way. If the guest is Linux, another option might be to consider activating the cpuplugd daemon. This daemon, which runs as a service process in Linux, turns off some of the guest's virtual CPUs when guest computing load seems too low for the virtual configuration. Spreading a guest's workload across too many virtual CPUs is a common configuration error which usually results in the guest inducing too much Control Program overhead.

FCX162 USERLOG is another place where we can find guest CPU utilization numbers. The first few columns match up to the FCX112 USER report and have the same definitions. But unlike FCX112 USER, FCX162 USERLOG tabulates one user's use of CPU over time. Here's an example.

Be aware that no Perfkit report breaks down guest CPU time by virtual CPU. The fields are there in CP Monitor but Perfkit does not report on them. However, if you are running Perfkit on a live system, one of the USER drill-down screens does report on each virtual PU separately. Perfkit creates this drill-down screen by mining timer values right out of CP control blocks, instead of by looking at CP Monitor records.

The counters that feed FCX112 USER and FCX162 USERLOG come out in various records in CP Monitor domain 4, otherwise known as the User Domain.

CPU Utilizations from Places Other Than Perfkit

So far we have devoted our entire article to discussing CPU utilization in the context of MONWRITE data and Perfkit reports.

The outputs of certain CP commands also offer us some insight about CPU consumption on the running system. These command outputs are useful for informal, ad-hoc assessments of CPU consumption at the moment. For capacity planning, benchmarking, routine health assessments, or system monitoring, always use CP Monitor data and Perfkit or equivalent.

The command CP INDICATE LOAD shows percent utilizations by logical PU. The values are not load-right-now, nor are they average-load-since-IPL. They are smoothed averages computed from datums collected at intervals over the recent past, with very recent samples having more weight than less recent samples. It is based on processor time used and voluntary wait time. Therefore, it is skewed when running in an LPAR when the partition is using shared processors. When there is contention, the value reported can be higher than actual processor time used because involuntary wait is not included. Inflated utilizations here can also occur from capped partitions.

The command CP QUERY TIME shows total CPU time and virtual CPU time used by the issuing guest, since guest logon.

The commands CP INDICATE USER and CP INDICATE USER EXPANDED show total CPU time and virtual CPU time used by the named guest, again, since guest logon.

Summary

In this article we've described that both PR/SM and z/VM keep accumulators that track CPU consumption. All of these accumulators come out in CP Monitor. Perfkit reports on all of them.

PR/SM's accumulators describe time used by logical PUs, time logical PUs induce in PR/SM, and time PR/SM uses for itself. Utilization percentages calculated from these three accumulators come out in the FCX126 LPAR and FCX202 LPARLOG reports.

z/VM's accumulators describe, for each logical PU, time used by guests, time induced in the Control Program, time spent by the Control Program for its own purposes, and time spent waiting. Utilization percentages calculated from these accumulators find their way into many Perfkit reports, notable examples being FCX126 LPAR and FCX225 SYSSUMLG.

Guests' individual CPU utilization numbers come out in a few places, notably FCX112 USER and FCX162 USERLOG. These two reports comment on two phenomena: the individual guest's own consumption, and the consumption the guest's actions induced in the Control Program.

Almost all the time, all of these percentages are of a whole physical processor, and "100%" means "a whole physical engine's worth". The exceptions to this are FCX126 LPAR %Logld and FCX100 CPU %LOGLD, described earlier.

T/V ratio is a useful metric of overhead. FCX225 SYSSUMLG and FCX239 PROCSUM both report system T/V ratio. Individual users' T/V ratios appear on FCX112 USER and FCX162 USERLOG.

Summary Table

This table summarizes the various notions of CPU utilization, on what basis they are accounted, the CP Monitor records that contain the counters, and the Perfkit report fields that relate to them.