IBM Systems  >   System z  >   z/VM  >

Understanding Poor Performance Due to Paging Increases

Basic theme - system performance is worse than it use to be and it seems related to increased paging.

Increased paging can mean any of the following

1. The number of pages moved in/out of real storage per second (or per command) goes up.
2. The number of page I/Os per second (or per command) goes up. Note this is different from number 1 since we can move multiple pages per single I/O.
3. Paging to expanded storage stays the same or decreases, while paging to DASD increases.

Keep in mind that we page because we can't fit everything in real storage at once. Therefore paging increases are caused by

1. Decrease in available storage
2. Increase in storage requirements
3. Or combination of 1 and 2

Decrease in available storage?

QUERY FRAMES is my favorite way of checking storage, but it's only online. For 5.1.0 the AVAIL field, and for 5.2.0 the PAGEABLE field are the ones you care about. If it goes down, check for others going up. In particular, for 5.1.0, FREE, PAGNUC, TRACE, and V=R. For 5.2.0, LogicalFreeStorage plus RealFreeStorage, Nucleus/Prefix and TRACE.

  Example for 5.1.0 =
          SYSGEN  REAL    USABLE  OFFLINE
          260352  260352  260352  000000
          V=R     RESNUC  PAGING  TRACE   RIO370
          000000  000545  259222  000554  000031
          AVAIL   PAGNUC  LOCKRS  LOCKCP  SAVE    FREE    LOCKRIO
          246129  003891  000240  000000  002854  006108  000011
 
  Example for 5.2.0 =
   All Frames:
      Configured=7864319  Real=7864319  Usable=7864319  Offline=0
      Pageable=7784415  NotInitialized=0  GlobalClearedAvail=288
      LocalClearedAvail=288  LocalUnclearedAvail=279
 
   Frames < 2G:
      GlobalUnclearedAvail=357  Pageable=515579  LogicalFreeStorage=623
      RealFreeStorage=659  LockedRS=1485  LockedCmd=0
      MinidiskCache=0  Nucleus/Prefix=3062  Trace=354  Other=2526
 
   Frames > 2G:
      GlobalUnclearedAvail=1145  Pageable=7268836  LogicalFreeStorage=1357
      RealFreeStorage=0  LockedRS=1212  LockedCmd=0
      MinidiskCache=13  Other=68627
 
   Other sources of these numbers are:
   RTM = Field PPAG from DISPLAY SYSTEM - pageable pages
         Field FPGS from DISPLAY SYSTEM - free storage pages
   VMPRF=  See PRF072 SYSTEM_CONFIGURATION report section on Initial Main
               Storage Allocation
           See also PRF004 PAGING_BY_TIME report fields:
                 DPA Pagable Pages
                 Non Pagable Pages
   Performance Toolkit = STORAGE or STORLOG command
   Note: Do not try to compare fields from different sources, they do
         not all use the same method to calculate the fields.

If there are fewer available pages due to increase in free storage, it could be due to larger control blocks in a new release, storage cancer, or more control blocks (more spool files, etc.).

Note that use of storage for data-in-memory approaches can decrease storage available for paging. This includes minidisk cache and virtual disk in storage. Pages associated with these two features are mostly covered in the "PAGNUC" field of the QUERY FRAMES output (for 5.1.0) or in the "Nucleus/Prefix" field of the QUERY FRAMES output (for 5.2.0).

Increase in storage requirements?

This normally takes the form of greater virtual storage requirements for CMS, Linux, applications, etc., but could also show up as data space usage by SQL DSS or other forms. Basically, if applications touch more pages, we require more storage to back this virtual storage.

You can check the Performance Toolkit report FCX113 (upage). The total virtual storage requirements are approximately equal to the sum of pages in Resident (<2GB and >2GB), XSTORE pages and DASD slots. These three fields are in the abridged report below.

FCX113      Data for 2006/06/27        Initial  13:30:24         Monitor
______        .      .     .      .      .    .    .    .      .      .      .      .     .      .      .      .
           Data  <--------- Paging Activity/s ---------->  <----------------- Number of Pages ----------------->
         Spaces     Page  <--Page Migration-->                <-Resident-> <--Locked-->
Userid    Owned  Reads Write Steals  >2GB> X>MS MS>X X>DS    WSS Resrvd  R<2GB  R>2GB L<2GB  L>2GB  XSTOR   DASD
>System<     .0    ...   ...    ...    ...  ...  ...  ...   9679      0     62  11096    39     31      0      0
BARNESC       0     ..    ..     ..     ..   ..   ..   ..   2357      0      0   2386     0      0      0      0
EREP          0     ..    ..     ..     ..   ..   ..   ..   1226      0    159   1088     0      0      0      0
FTPSERVE      0     ..    ..     ..     ..   ..   ..   ..   1367      0      0   1368     0      1      0      0
GCSXA         0     ..    ..     ..     ..   ..   ..   ..     51      0      0     52     0      1      0      0
LNXIMG1       0     ..    ..     ..     ..   ..   ..   ..  23125      0      0  27115     0     13      0      1

From comparing the above report to before and after data for each user, you should be able to determine which group of users increased the most. Then you can dig further. You could also get the above information in snapshot form from INDICATE USER or INDICATE USER EXPanded, but it's only for snap shot and a single user

Next Steps