Problem: Performance Toolkit (or insert favorite monitor here) is giving me alerts about the C1ETS being too high.
Solution: The C1ETS stands for class 1 Elapsed Time Slice. Each scheduler class has an Elapsed Time Slice (ETS) associated with it. The Class 1 ETS is dynamically adjusted by the scheduler. All the other time slices are multiples of the C1ETS (classes 0/2/3 multiplication factors are 6/8/48 respectively). The scheduler adjusts C1ETS in order to try and keep 85% of the transactions as trivial (that is within the first ETS). On systems where there are guests that never go truly idle, the transactions are very infrequent and therefore can cause the scheduler to increase the C1ETS. This isn't necessarily a problem since the transactions are not real transactions.

Problem: I am looking at my performance monitor and I see transaction rates and transaction response times. However, they make no sense to me. There also appear to be very large spikes in these numbers. My workload is primarily Linux guests.
Solution: You can ignore these transaction related numbers. They are based on the Control Program's (CP) view of a transaction, which is actually an estimation. CP basically defines a transaction ending as whenever a virtual processor goes long term dormant. That is a transaction will span test idle grace periods. So in many Linux environments, the guests never go long term dormant frequently enough to make the transaction counts of value.

Problem: I increased the number of virtual processors for a given virtual machine, but it doesn't seem to be able to use additional processor resources.
Solution: First, did you really need to add virtual processors? Often you can achieve the same result by increasing the Share setting. Second, if you increased the number of virtual processors, you might need to increase the share setting as it is distributed across the virtual processors for a given machine. For example, a 1-way virtual machine with default relative share setting is given a second virtual processor. If the Share setting is left the same, both virtual processors are run as if they were each Share Relative 50 virtual machines. A common rule is to increase the Share setting proportional to the number of virtual processors for the given virtual machine. So in our example, set the share to Relative 200 for the virtual 2-way virtual machine. This is a simple approach and you might find changing share values for other considerations also helpful.

Problem: Very high processor overhead with z/OS guests using virtual CTCs and CFs
Solution: Check if IRD and WLM is attempting to be used on the z/OS virtual machines. If so, disable it.

Problem: MTU sizes don't match MPROUTE CONFIG with TCP/IP 510.
Solution: There are some changes to MTU processing within the TCP/IP server. A new optional MTU operand has been added to all LINK statements. In 510, this is the value that will be shown by IFCONFIG and NETSTAT DEVLINKS (NETSTAT GATE shows MTU if set by gateway statement). If the value is not specified, it results in a default value shown in the "Intelligent Default MTU Values Based on the Device and Link Type" section at the beginning of the "DEVICE and LINK Statements" section in Chapter 24 ("Configuring the TCP/IP Server") of the z/VM TCP/IP Planning and Customization manual. For HiperSockets and OSD devices, if the MTU value is omitted, this defaults to the value returned by the device. The TCP/IP server now checks MTU values for OSD and HiperSockets devices against the maximum buffer capacity returned from the device and does not allow the MTU to exceed this value. Chapter 24 also has information on how MTU specifications from MPROUTE and ROUTED are handled. Refer to "Step 3: Create an Initial Configuration File".

Problem: I'm seeing high processor utilization when running a VM system as a guest of VM when running on LPAR.
Solution: This type of configuration results in three levels of dispatching (SIE - start interpretive execution): LPAR, 1st level VM, and 2nd level VM. The hardware implementation supports 2 layers of SIE. However, when the third level is reached, a layer of SIE needs to be virtualized which is very expensive in terms of processor time. This third level is not supported for production work.

Problem: High channel utilzation that cannot be explained by activity on active logical partitions.
Solution: Check if you have a partition that is sitting in a wait state (such as 1010 for z/VM). If you have a Shark box or other device that is signaling all known partitions for things like state change (flash copies), the state changes are reflected to all active partitions whether they are able to deal with them or not. Deactivate the idle partition to avoid extra traffic.

Problem: z/VM 4.4.0 system with more than 16GB of processor memory, mostly configured as central storage, showing poor performance and heavy contention for pages below 2GB.
Solution: If there is significant page movement to/from expanded storage and movement below 2GB, consider limiting use of Minidisk Cache (MDC) in expanded storage via the CP command SET MDC XSTORE. You can set a hard limit or bias against MDC. Expanded storage is often used in the steal processing for memory below 2GB. Ordinarily MDC favors expanded storage, but in very large memory configurations, you will want to tune this in the above method.

Problem: Linux guest has very slow response when connecting to the network.
Solution: Several things can affect performance when connecting, whether by ftp, telnet or any other method. Here is a list of things to check and watch for:

• Make sure your specification for the name server is correct. Look in /etc/resolv.conf.
• Ensure that the mtu size is the same, if possible, through the entire path taken. Issue traceroute to determine the path. If negotiation needs to take place before the message is delivered it can affect performance.

Problem: On RTM, I see rates for diagnoses that do not exist, such as diagnose x'38'.
Solution: You need to get corrective service for RTM. In particular, APAR GC05473.

Problem: I shutdown a number of my Linux systems running in virtual machines, but I still see them consuming storage in memory or on paging DASD.
Solution: Since VM virtualizes the architecture, it cannot free up backing storage unless that is explicitly requested. In the case where you shutdown the Linux system but keep the virtual machine logged on or disconnected, VM will continue to maintain the memory that Linux last used. To clear this up, either logoff the virtual machine or issue a CP SYSTEM CLEAR command.

Problem: The system is paging to DASD even though there is plenty of central storage available.
Solution: On z/VM 4.2.0 and earlier, you need to define some central storage as expanded storage even if running in 64-bit mode. This is especially true if you have contention for the storage below 2 GB. See Configuring Processor Storage for more details.

Problem: On the TCP/IP FL310 stack machine, I see the following alert:

      Serious problem encountered: 10:30:29 04/12/00
          SCB pool is low

Problem: I get the following error messages on the SFS server:

DMS5GM3001W The file pool server has used up 92 percent of the
DMS5GM3001W available virtual storage

Problem: For my network connection I have 2 devices defined: one for input and one for output. The utilization of the input device appears to be near 100% all the time, while the output line is much lower.
Solution: TCP/IP and other network servers often use seldom-ending channel programs for I/O on the device line that receives data. If you look at the breakdown of service time (disconnect, pending, and connect time) you will see that the near-100% device has most of the service time in the disconnect state. This is really the idle time between actual I/Os in this scenario. The connect time is the time data is actually being moved on the channel subsystem. The connect times for the two devices should be fairly close (assuming the same amount of data is coming in as going out). So the high utilization is only an anomaly, and not something to be concerned about.

On products such as RTM that flag devices with a high utilization, this can be annoying. RTM can be modified to avoid logmsg 22 for specific devices. See RTM Common Problems and Suggestions for help.

Problem: I used the QMDC tool from the VM/ESA download library, but it seems to give strange results for SFS minidisks.
Solution: There is a problem in that the VDEVIOCA counter is not updated in some scenarios, particularly those involving multiple block I/O requests. This results in low MDC rates as reported by QMDC. However, MDC is working. Use VMPRF or similar tool to look at the real I/O avoided due to MDC at the real device level.

Problem: I see large amounts of CP CPU time being charged to my DB2 Server for VSE and VM (SQL/DS).
Solution: Check if you are using minidisks mapped to VM data spaces for SQL, and if so, make sure those minidisks are disabled for MDC. The large amount of CP processor time is most likely a result of overhead in searching for data from that minidisk in MDC that needs to be flushed when SQL uses the SAVE function to force the data in the data space to be written to the mapped minidisks.

Problem: I have a processor with storage that can be configured as either central or expanded storage. Which is better?
Solution: Typically, it is better to have more real storage than expanded storage. However, rule number one applies. "It depends." See Configuring Processor Storage for more details.

Problem: I migrated from VM/ESA 1.2.1 to VM/ESA 2.2.0 and now my full pack minidisks (FPMs) are being minidisk cached. I don't want them to be eligible for MDC.
Solution: Yes, prior to VM/ESA 1.2.2, full pack minidisks were not eligible for MDC. Now FPMs defined via VOLSER are eligible. Unfortunately, the MINIOPT directory statement is not valid for FPMs and the DASDOPT directory statement does not include a NOMDC option. The following are alternatives.

• Use system config file and the RDEV statement to set MDC off for the entire volume. This is most effective when the FPMs are used exclusively for guest systems.
• Use system config file and the RDEV statement to set MDC off by default (DFLTOFF) for the volume and then explicitly set MDC on for minidisks (via MINIOPT) overlaid by the FPM. The downside to this solution is all the work to explicitly turn on MDC for each new minidisk.
• Autolog a userid that links to all the FPMs and then issues SET MDC MDISK commands to disable MDC for those FPMs. This is effective but has a few more moving parts than we typically like.
• Allow MDC to be on by default for the volume, but use SET MDC INSERT OFF in the backup program (or which ever userid you do not want to use MDC). This can be appropriate for backup software and scanning software.
• Just let the system take the defaults. In many systems the fair share algorithms will keep any one user from abusing MDC.

See Minidisk Cache Guidelines for more details.

Problem: The RTM SYSDASD display or other vendor products show paging activity to non-paging devices.
Solution: This is most likely paging activity for VM data spaces that are mapped to minidisks. This allows an application to just reference a storage location in the VM data space and have CP handle making sure the data is moved from the minidisk to virtual storage. CP uses the paging subsystem to accomplish this. See VM Data Spaces for more details.

Problem: INDICATE LOAD, Monitor, and other sources say one of my processors is 100% busy. But it is clear there really is not that much work. Other processors are not near 100%.
Solution: Check if the processor is a dedicated processor. If so, it will look to VM as if it is 100% or near 100% busy. Some background: for dedicated virtual processors, CP marks the virtual processor as being eligible for the wait state assist. This means that when the virtual processor loads a wait state, the hardware will not automatically exit SIE. The virtual processor continues to run under SIE until an interrupt occurs. The virtual machine will exit SIE eventually and CP will update the timers appropriately but while the virtual processor has been in a wait state, VM thinks it has been running the entire time. For dedicated processors, one really needs to look at the 2nd level guest's performance monitor tools to get accurate measures.

Problem: It takes 15 minutes for the spool initialization on my system to complete. There are 90,000 files in the system.
Solution: This is as expected. While VM/ESA spool file initialization has been improved greatly over VM/XA, it still takes a noticeable amount of time for the spool file to initialize with large numbers of spool files. Additional information can be found on the Understanding Spool File Initialization Performance page.

Problem: Disconnect time for certain volumes is huge.
Solution: Validate that processor microcode is current. Disconnect time is recorded by the hardware in the subchannel measurement block in a 4 byte field. Several processor lines have had a problem where they accidentally increment the first byte out of sync with the other 3 bytes. Ouch!

Problem: After migration to VM/ESA 1.2.2, minidisk cache has very high steal rate and low hit ratio.
Solution: Make sure there is some central storage being used for minidisk cache. This is especially important when the virtual I/O buffers are not page aligned. For the not page aligned case, CP can not move data directly from expanded storage to user's buffer. There are stats in RTM and monitor data that reflect this.

Problem: After migration to VM/ESA 1.2.2, minidisk cache seems less efficient, and the MDC inserts rejected due to fair share limit have increased.
Solution: Make sure the directory option NOMDCFS is used for all the appropriate server virtual machines. Also note that the fair share limit minimum value in 1.2.2 may be too low. This has been changed from 8 to 150 in VM/ESA 2.1.0. The APAR to VM/ESA 1.2.2 is VM59590 (PTF UM27424).

It can be modded by changing the values in HCPTCMBK to the following:

   TCMFSLIM DC    F'150'         Fair share insert limit for this
                                 interval
   TCMFSLMM DC    F'150'         Minimum fair share insert limit
                                 per interval

Problem: After migration to VM/ESA 1.2.2, minidisk cache hit ratio went down (got worse).
Solution: This may not be a problem. If the hit ratio is from RTM/ESA, note that prior to VM/ESA 1.2.2, RTM computed hit ratio by only looking at MDC eligible I/Os while in 2.2 and beyond all virtual I/Os are considered. If the hit ratio is from the INDICATE LOAD command (which only looks at eligible I/Os), note that the count of eligible I/Os may be greater in VM/ESA 1.2.2. So while hit ratio went down, total I/Os avoided went up. I/Os avoided is reported on VMPRF DASD_BY_ACTIVITY reports.

Problem: After migration to VM/ESA 1.2.2, system is paging more.
Solution: In VM/ESA 1.2.2, minidisk cache will use real storage as part for the cache by default. Having less real storage may increase paging, but it is the designs hope that overall performance is better. Check on the total real I/O rates to DASD. If all goes as planned, the I/O increase to paging DASD will be easily offset by I/O decrease for virtual DASD I/O. If it is still a concern, set a more appropriate limit for real storage (SET MDC STORAGE 0M ?M). It is strongly recommended you do not set real storage off for MDC entirely. Note that with MDC changes more data is eligible for MDC (1K and 2K formatted minidisks or VSE data for example). Therefore, MDC may require additional storage which would lead to less real DASD I/O for user I/O at the cost of extra paging I/O.

Problem: Guest with multiple virtual processors waits for CPU.
Solution: Increase Share setting. Basically, the current share setting is divided among the virtual processors. For example a virtual 4 way with default Relative 100 Share is treated as 4 relative 25 share processors in making scheduler decisions.

Problem: System performance slows down whenever DIRECTXA is issued.
Solution: Check if DIRECTXA target is full pack minidisk. If so, data in the minidisk cache for any other minidisks overlapped by the full pack are invalidated when the directory writes occur. This should not be as significant after VM/ESA 1.2.2 which does better job of handling overlapping minidisks.

Problem: VSE guests get stuck in E-list (seen with IND QUEUES EXP command).
Solution: Increase SET SRM STORBUF settings. Rough starting point could be multiple by 3 (100 85 75 becomes 300 255 225). Might possibly need to increase LDUBUF values as well, but just try STORBUF to start.

Problem: Recent migration shows higher paging and less available real storage.
Solution: Check if default trace table size being used and adjust accordingly. Can set without regen on VM/ESA 1.2.0 and higher with CP SET TRACEFRAMES command.

Problem: Poor response time with much more paging in system. CMS working sets have increased.
Solution: Check for missing saved segments (VMLIB, VMMTLIB, Help, Pipelines, modules moved from CMS nuc to S-disk (installed as lsegs CMSQRYH and CMSQRYL in VM/ESA 1.2.2). The Q NSS MAP command can be helpful in determining which segments are being used. The CMS NUCXMAP * (SEGINFO and QUERY SEGMENT commands can be helpful to see what is being loaded for CMS.

Problem: Recent migration to or over VM/ESA 1.2.0 shows higher paging especially during morning crunch period with high logon rate. CMS 9 (VM/ESA 1.2.0) references about 25 additional pages during initialization. Therefore high logon rate means high IPL CMS rate. Also associated with shops where users have short sessions (user logons each time they want to check mail and then logs off right away).
Solution: Double check all segments to make sure not making it worse than necessary. Try saving storage elsewhere (exploit segments, SAVEFD, tracetable, etc). May require more storage. Some shops have found relief by not forcing idle users as quickly or by autologging users at 4am so overhead of IPL is over by the time people come in.

Problem: Paging seems less efficient as seen in higher service time for paging I/O.
Solution: Check that you are not running with paging/spooling config out of the box. As delivered, page and spool and user are mixed on same volumes as small areas. This causes bad seek patterns, interferes with Paging subsystem never-ending channel program and small areas are bad for block paging efficiency. Suggest you reconfigure with dedicated page and dedicated spool volumes.

Problem: You just upgraded processor to relieve CPU bottleneck. New processor is 40% faster, but performance didn't improve 40%.
Solution: Check for latent demand and whether a different bottle neck (I/O or storage) has been hit first.

Problem: You just migrated to 9221 and didn't get expected performance.
Solution: Check if ESA or 370. Could be caused by misleading marketing information (i.e. sized an ESA system based on 370 planning Mips). There are some things we can do to soften the blow:

• exploit MDC if possible
• increase VTAM delay factor
• increase DSPSLICE

Problem: DASD on 3990 Cache Controllers do not appear to be caching.
Solution: Do QUERY DASD DETAILS for the volume and make sure CACHE is Yes for both subsystem and device for regular cache. For DASD fast write, NVS must be Y for subsystem and DFW must be Y for device. Use SET CACHE, SET NVS, and SET DASDFW to correct.

Problem: Short sporadic periods of terrible response time with lots of expanded storage (> 1GB) on VM/ESA 1.2.1 or earlier. The sporadic hits map to high system spin time (%SP on RTM SLOG display).
Solution: Use RETAIN XSTORE to fix amount of storage used for MDC. The sporadic hits are caused by holding the MDC lock while re-hashing the hash table for different size cache. Fixing the amount of storage for MDC avoids the size change.

Problem: Devices show 0% on 9221. They are attached via integrated adapters. Device utilization in RTM and Monitor is computed by using the timing values supplied in the subchannel measurement block (Connect, Disconnect, Pending, etc) which is updated by the hardware. Unfortunately, the IA hardware doesn't support the timing values and therefore everything shows up as zero.
Solution: Some information can be determined by looking at queueing on devices. Monitor does hi-frequency sampling on queue value in RDEV and VMPRF reports it in DASD_BY_ACTIVITY report. This value will not be accurate for page, spool, or mapped minidisks since the paging subsystem keeps its own queueing stats. For page, spool, and mapped mdisk see the VMPRF report DASD_SYSTEM_AREAS or the RTM SYSDASD display.

Problem: System seems sluggish with no obvious reason.
Solution: Check to see if IPLed last time and changed the TOD, and if so, whether it was for a large amount of time. If the TOD is changed by more than a few minutes, unpredictable results can occur since timer blocks which were suppose to go off for system feedback algorithms may not go off for a long period of time and that could bad. Note, this does not apply to systems with APAR VM60324 applied or at VM/ESA 2.2.0 or higher.

Problem: Performance Toolkit reports misleading I/O response times for devices that are part of a Parallel Access Volumes (PAV) group.
Solution: There is a known problem with z/VM 5.2.0's Performance Toolkit that causes it to report I/O response times incorrectly in certain PAV situations. Refer to the z/VM 5.2.0 performance management discussion for more information.