VM Data Spaces

The following article was written by Kris Buelens and Guy De Ceulaer for their VM Newsletter. Their customers are fortunate to have such talented and dedicated support personnel. So in order to allow the rest of the world to share in the benefits, the article has been incorporated here into the VM Home Page.

Table of Contents

VM Data In Memory Techniques

The term Data In Memory is often abbreviated to DIM.

VM provides some "Data In Memory Techniques", (such as Minidisk Cache, and Virtual Disks in Storage), and exploits some other (such as Dataspaces and DASD controller caching).

Problems can be :

What DIM technique to use ?

You can find information to help with this decision in VM/ESA Data in Memory Techniques and one of Bill Bitner's presentations VM/ESA Data in Memory Techniques

How to interpret "page rate" when using VM dataspaces?

Simply looking at the page rate returned by CP INDICATE LOAD is completely misleading. This topic explains why.

To understand why pagerates can be misleading, we must cover first :

• What are VM Dataspaces ?

• Primary Address Space

• Data Spaces

• IO Performed to Read CMS Files
  • Reading CMS Files Stored on Minidisks
  • Reading a Normal SFS File
  • Reading an SFS File Mapped in a Dataspace

• DataSpace Usage by DB2/VM(1)
  • Page Rate Reported for DB2/VM

• Performance monitors

What are VM Dataspaces ?

A dataspace is similar to the primary address space of any VM user: it is virtual storage that can be directly addressed by assembler programs. As any virtual storage, CP can page it out to its page datasets on DASD. Whereas a primary address space can contain executable programs and data, a dataspace only contains data.

With CP Q SPACES PERMITTED <userid> one can query which address spaces one has access to.

Primary Address Space

When a user logs on, CP creates its primary address space, and it even assigns it a data space name, namely userid:BASE. The size of the primary address space is defined in the CP directory and can be

• questioned with Query Virtual STORage
  or with INDicate USER

• modified with DEFine STORage xxxM

Dataspaces

Authorized users can create dataspaces, these dataspaces can be shared or private and mapped or unmapped. Dataspaces get also a name, for example VMSYSU:0000000016SFSUSR

Shared

A shared dataspace can be directly addressed by users that have permission. Users can even tell CP that they want to make their primary address space shareable. So there is no need to use APPC (or any other protocol) to access the data that a server maintains in a shared dataspace.

VM's Shared Files System (SFS) uses shared dataspaces for directories indicated by the SFS administrator. This gives a good performance advantage for read-only data.

Private

A Private dataspace is simply extra virtual storage for the user that creates it.

DB2/VM (formerly named SQL/DS) uses VM dataspaces in this way. SQL end-users cannot directly access SQL data, they still use APPC to send their request to the SQL server. (It is impossible for an SQL system to work with shared dataspaces the way SFS does as is explained below).

Mapped

Each virtual storage page of a mapped dataspace is mapped to a 4K block on a minidisk of the dataspace owner. When a pagefault occurs, CP will read the data from the appropriate minidisk. Mapped dataspaces are used by SFS and DB2/VM.

See the detailed SFS example below.

Unmapped

This is more like primary address spaces: when a pagefault occurs, CP will get the data from its paging datasets. Pageouts go to CP's paging datasets as well. Unmapped dataspaces may be used by DB2/VM for workspace (the so called "Internal DBspaces").

I/O Performed to Read CMS Files

In this section we want to explain where the I/O instructions are counted when CMS files are read (Access to SQL tables is mentioned later). The explanation is however simplified a bit.

To manage its file system, CMS uses two important control blocks:

1. Active Disk Table or ADT
   This table has an entry for each letter of the alphabet. It contains more or less the information displayed by Q DISK and Q ACCESSED. In other words, it allows CMS to know that disk D, for example, corresponds to minidisk 192, and disk Z with SFS directory VMSYSU:HTTPD.WEBSHARE.KRIS.

2. Files Status Table or FST
   This table has an entry for each file of a minidisk/directory. It contains more or less the information displayed by FILELIST, and also a pointer to where the file starts on the minidisk (in practice, the pointer may point to an index of the file's records, but we will ignore this here).

Reading CMS Files Stored on Mindisks

When a CMS user starts a program that reads a file (for example XEDIT MYFILE ONE D), CMS will consult its ADT to find on which minidisk the file is stored and then the FST (File Status Table) to find where the file lives on the minidisk. Then CMS asks CP to read the appropriate file blocks from that minidisk. CP places the data of the file in buffers provided by CMS or by the program (XEDIT in our example).

Counters: The I/O's involved are issued by the end-user's virtual machine and simply counted as I/O instructions in all performance data collectors; for example in CP IND USER or PerfKit's UPAGE subcommand.

Reading a normal SFS file

When CMS finds in the users' ADT that the file is a "normal" SFS file, it will use APPC to send a ReadFile request to the SFS server. The SFS server consults its catalog to find where the file resides, and performs one or more I/O requests (using *BLOCKIO) to read the file data from its minidisk(s) (a single SFS file can span minidisks). It then uses APPC to send the data back to the end-user, where CMS receives the data in the buffers provided by the the program (XEDIT in our example).

Counters: Now it is no longer the end-user that performs the I/O, but the SFS service machine. Hence the I/O counts for the end-user are not incremented. CP counts the I/O's with the SFS server (CP cannot know which end-user request resulted in the I/O instructions issued by the SFS server).

Reading an SFS File Mapped in a Dataspace

Reading an SFS file residing in a dataspace is a bit more complex, but faster. First we must explain how a dataspace is created for the directory.

Mapping

When some user ACCESSes a directory, the SFS server checks if it is eligible for a dataspace. When the user is the first one to access that directory, SFS will define a new dataspace (for example VMSERRUN:0000000016SFSRUN) and tell CP how it must be mapped. In human words this means that, for each file in the directory, the SFS server will assign a place in the dataspace, and tell CP where the data of the files reside on its mindisks.

A basic, simplified, example:

• suppose the directory contains only two files. File 1 (MYFILE ONE) has two datablocks, datablock 1 resides on block 102 of minidisk 423 and datablock 2 on block 326 of minidisk 525. File 2 (MYFILE TWO) has one datablock, residing on block 103 of minidisk 423.

• SFS could assign dataspace pages 1 and 2 to the datablocks of MYFILE ONE, and page 3 to MYFILE TWO. So it will tell CP that the contents for page 1 of the dataspace are located on block 102 of minidisk 423, page 2 is on block 326 of minidisk 252 and page 3 is on block 103 of minidisk 423.

• SFS will also create the FST's corresponding to the 2 files in the dataspace. The FST entry for MYFILE ONE will point to pages 1 and 2 of the dataspace, the entry for MYFILE TWO will point to page 3.

When all this is built, the SFS server tells the end-user accessing this directory in which dataspace it can find everything. Obviously, when a second user accesses the same directory, the mapping is already done and SFS can simply tell the user which dataspace it can use.

Here ends the processing for the ACCESS command of a "dataspaced" directory.

Note: At this stage, the data for the files are not yet read in central storage, they remain on the SFS minidisks until some user needs them.

Note: This also means that if you keep at least one user ACCESSed to the directory, it may improve the performance. You could for example keep the access in some service machines like VMUTIL.

Using the files

We saw that the SFS server constructed a dataspace to be used by the accessors of the directory. Reading a file becomes simple.

Suppose again the user issues XEDIT MYFILE ONE ...

1. XEDIT allocates buffers to work with the file, and asks CMS to read the file.

2. CMS finds out that MYFILE TWO is located in pages 1 and 2 of the dataspace and simply executes a MOVE instruction to copy pages 1 and 2 in the buffer provided by XEDIT.

3. If this is the first user to refer to file MYFILE ONE, a pagefault for the dataspace pages will be the result when CMS starts to copy page 1 (we mentioned above that when the directory is mapped, the data for the files are not yet read into storage). To resolve this page fault, CP's paging routines will see that page 1 is mapped to minidisk 423 of the SFS server and so CP will pagein from the minidisk instead of from its page datasets.

   As CP uses a Block Paging algorithm, CP will probably not pagein just a single page. If plenty of real storage is available, CP might pagein many pages with one operation. The pages may or may not belong to the same file. As the dataspace is shared amongst all users of the directory, other users referring to the same file may profit from the pagein caused by the first user.

Counters: It should be clear that neither the SFS server, nor the end-user issues an I/O instruction. But, CP's paging routines are called for the user that encountered the page-fault. Hence here the end-user's pageread numbers are incremented (e.g. CP IND USER) as well as the system-wide pagerate.

Note: To build the data space, the SFS server performs also some I/O to read its catalog.

DataSpace Usage by DB2/VM

Above we mentioned that DB2/VM uses only private dataspaces. Indeed requests to SFS files are simple : "give me one or more records of a file". SQL requests are complex, and SQL has finer security (some users cannot see all columns of a table for example). This makes the use of shared dataspaces impossible or unacceptable.

• Mapped : DB2/VM uses mapped dataspaces to access its minidisks. DB2/VM still uses its internal buffers (as before), but instead of executing I/O instructions to read data from its minidisks, it simply copies data from its dataspaces into the buffers (resulting in CP page reads if the data is not yet in real storage).

• Unmapped : DB2/VM can also use unmapped dataspaces ; DB2/VM may need much working storage (such as to SORT tables). Without dataspaces, DB2/VM stores this working storage on minidisks of its own.

  With dataspaces, one can instruct DB2/VM to use dataspaces as working storage. These dataspaces are then not mapped to minidisks, but are paged to CP's page datasets.

We won't explain the mechanisms DB2/VM uses to optimize its performance (such as page fault handshaking with CP). But, after having read the section on SFS, it should be clear that when DB2/VM uses dataspaces, the I/O counts for the DB2/VM server will be much lower, and the reported page rate much higher.

Page Rate Reported for DB2/VM

A high pagerate now means either that the server works a lot with its database, or that the system is too short on storage.

Performance monitors

The performance monitors cannot differentiate between page rate to primary address spaces and to dataspaces. We have to repeat over and over again that interpreting the systems' "pagerate", as reported by CP INDICATE or PerfKit must be done with care, as it is no longer "true" paging, but it includes database I/O and some SFS I/O.

The VM lab is aware of this problem and may provide more detailed reporting tools in the future.

Performance Toolkit (known as PerfKit) gives a report (STORAGE) that helps a bit. On the other hand, CP IND SPACES USER xxx can be issued to get actual counters (totals for page reads and writes of each address space a user has, but that doesn't give a rate/second).

PerfKit's CP Owned Device Report

FCX109      Data for YYYY/MM/DD  Interval HH:MM:SS - HH:MM:SS    Monitor
 
Page / SPOOL Allocation Summary
PAGE slots available      1220016          SPOOL slots available      1220016
PAGE slot utilization           0%         SPOOL slot utilization          47%
T-Disk cylinders avail.   .......          DUMP slots available             0
T-Disk space utilization      ...%         DUMP slot utilization           ..%
 
____ .             .                          .     .     .     .     .      .
< Device Descr. ->                        <------------- Rate/s ------------->
            Volume Area   Area      Used  <--Page---> <--Spool-->         SSCH
Addr Devtyp Serial Type   Extent       %  P-Rds P-Wrt S-Rds S-Wrt Total  +RSCH
3000 3390-3 PERF1  PAGE    549- 648    0     .0    .0   ...   ...   ...    ...
                   SPOOL   649- 748   91    ...   ...    .0    .0    .0     .1
3008 3390-3 PG401  PAGE      0-3338    0     .0    .0   ...   ...    .0     .1
3009 3390-3 PG402  PAGE      0-3338    0     .0    .0   ...   ...    .0     .1
300B 3390-3 SP401  SPOOL     0-3338   80     .0    .0    .0    .1    .1     .1
300C 3390-3 SP402  SPOOL     0-3338   12     .0    .0    .0    .0    .0     .1
 

PerfKit's Shared Data Spaces Report

FCX134      Data for YYYY/MM/DD  Interval HH:MM:SS - HH:MM:SS    Monitor
______                                 .     .     .     .     .     .
                                         <--------- Rate per Sec. -------
Owning                             Users
Userid    Data Space Name          Permt Pgstl Pgrds Pgwrt X-rds X-wrt X-
>System<  --------                     0  .000  .000  .000  .000  .000  .
SYSTEM    FULL$TRACK$CACHE$1           0  .000  .000  .000  .000  .000  .
SYSTEM    ISFCDATASPACE                0  .000  .000  .000  .000  .000  .
SYSTEM    PTRM0000                     0  .000  .000  .000  .000  .000  .
SYSTEM    REAL                         0  .000  .000  .000  .000  .000  .
SYSTEM    SYSTEM                       0  .000  .000  .000  .000  .000  .
SYSTEM    VIRTUAL$FREE$STORAGE         0  .000  .000  .000  .000  .000  .

This report does not show private dataspaces.

INDICATE SPACES

One can indeed write an EXEC that issues an INDICATE SPACE USER xyz every so often and calculates a rate per second. The QDBPAG EXEC listed here does this for you.

/* This exec is a sample to tailor to your needs.
       (look for /*>>TAILOR<<*/ flags in the code).
 
   It collects data to distinguish dataspace paging from other paging.
   Very important to check on SQL/DS when it uses mostly dataspaces, as
   then the database IO's are reported as paging.
   We analyze INDICATE SPACES to find the real counts, and subtract the
   counts of two executions of IND SPACED.
 
   Note that we add Page Write and Page Migrations (the first is from
   an address space to DASD, the second is from Xstore to DASD)
 
   And, as we are counting these pagings, we also collect IO and %CPU
    +-----------------------------------------------------------+
    | format:  | QDBPAG <SEND>                                  |
    +-----------------------------------------------------------+
 
Notes:
1. It is supposed that this exec runs for example in VMUTIL, where
   it could be started every 5 minutes for example.  You can freely
   choose any interval you like.
   The intermediate paging counters are stored in GLOBALV.
2. It is no problem for the exec that an SQL database would be restarted
   from time to time.  The exec will detect this and ignore the interval
   in which a restart occurs.  Similar, the very first time this exec
   runs, it cannot calculate any page rate per second as it misses data
   to compare with.  So, the first time the exec encounters a given SQL
   database, it simply stores the paging counters in GLOBALV.
3. Also, every day the collected file should be sent to some other user
   and VMUTIL would start a new file.  This is achieved with the SEND
   parameter.
The two lines for VMUTIL's WAKEUP PARMS file could be:
M-F      +5       10:00:16 CMS EXEC QDBPAG
M-F      23:55:00 09/02/97 CMS EXEC QDBPAG SEND
 
Written by: Kris Buelens IBM Belgium;  31 Jul 1996*/
 
parse upper source . . myname mytype . syn addr .
if addr=? then signal SubPipe
address command
parse upper arg send .
If send='SEND' then signal send
signal on syntax
/*>>TAILOR<<*/
/* Find what userids run an SQL database */
'PIPE CP Q RESOURCE',
  '| LOCATE /SQL/',
  '| SPEC W7 1',
  '| STEM USER.'
/*>>end TAILOR<<*/
sayit=(linesize()<>0)  /* If running connected: show on console */
If Sayit then
   Say left('',8) 'Secs   PagRd   DatRd   PagWr   DatWr      IO  CPU'
/*user.0=1 ;user.1='SQLIBS'*/
do i=1 to user.0
   user=user.i
 
   'PIPE (end ?)',
      '  CP IND SPACES USER' user|| '15'x || 'IND USER' user 'EXP',
      '|T: Tolabel Userid='||,            /* Send INDICATE data down */
      '| BETWEEN /Spaceid=/ 5',
      '| XLATE = 40',
      '| SPECS W2 1',                     /* Dataspace id */
         'read read read w7 Nw',          /* Xstore migrates */
         'read w3 Nw W5 Nw',              /* Dasd Read & Writes */
      '|O: Fanout',                       /* Copies to allow SUMming */
      '|   SPECS W1 1|Not Chop' length(user)+1,
      '|   JOIN * $/$',                   /* Keep list of space ids .. */
      '|   CRC crc16i|SPEC 1-2 C2X 1',    /* replace by a CRC as list  */
      '|   BUFFER',                       /* can be too big for GLOBALV*/
      '|S: SPECS /"' time('S') '/ 1 W1 Next /"/ Next W2 Nw',
      '      Select 1 W1-* NextWord',       /* The reads */
      '      Select 2 W1-* NextWord',       /* The Writes */
      '      Select 3 W1-* NextWord',       /* The Migrates */
      '    /0 0 0/         NextWord',       /* For user without DataSp*/
      '|REXX('myname mytype')',             /* Sum them for DataSpaces*/
      '|SPECS W1-8 1',                   /* Remove 0 0 0 if not needed */
      '|F: Fanin|Join 1 / /',               /* Add IO and CPU */
      '|   Var PagIO.'user,
      '?O:|SPECS W3|PAD 10|JOIN * $+$|XLATE 11 + 40|S:', /* Keep Reads */
      '?O:|SPECS W4|PAD 10|JOIN * $+$|XLATE 11 + 40|S:', /* Keep Writes*/
      '?O:|SPECS W2|PAD 10|JOIN * $+$|XLATE 11 + 40|S:', /* Keep Migr. */
      '?T:',
         '| FROMLABEL CPU '||,             /* Anal INDICATE data */
         '| XLATE : 40 = 40',
         '| SPECS W14  Nw /*24*3600 + / Next',/* TotCpu days     */
                 'W15  N  /*3600 + /    Next',/* TotCpu Hours    */
                 'W16  N  /*60 + /      Next',/* TotCpu Minutes  */
                 'W17  N ',                   /* TotCpu Seconds  */
                 'Read W8 Nw',                /* IO count */
         '|REXX('myname mytype')',            /* Calc CPU seconds*/
         '|F:'                         /* Pass to main stream*/
 
   parse value value('PagIO.'user,PagIO.user,'GLOBAL QIO'),
         with  otime oids oPagR oDatR oPagW oDatW oPagM oDatM oCPU oIO .
   parse var pagio.user ,
               time   ids  PagR  DatR  PagW  DatW  PagM  DatM  CPU  IO .
 
   if ids<>oids  |,           /* Dataspaces changed: ignore */
      time<otime |,           /* Date changed: ignore */
      cpu <ocpu  then iterate /* User must have been logged of */
   secs=time-otime
   /* We add the Xstore Migrates to the Writes */
   Parse value PagW+PagM  oPagW+oPagM DatW+DatM oDatW+oDatM,
         with  PagW       oPagW       DatW       oDatW
 
   data= format((PagR -oPagR ) /secs,5,1),
         format((DatR -oDatR ) /secs,5,1),
         format((PagW -oPagW ) /secs,5,1),
         format((DatW -oDatW ) /secs,5,1),
         format((IO   -oIO   ) /secs,5,1),
         format((CPU  -oCPU  ) /secs*100,3,0)
 
   If Sayit then say left(user,8) right(secs,4) data
   'EXECIO 1 DISKW' user 'PERFLOG A (FINIS STRING' left(time(),5) data
end i
exit
SEND:
/*>>TAILOR<<*/
 'PIPE COMMAND LISTFILE SQL* PERFLOG A|STEM FILE.',
     '|SPEC /ERASE/ 1 W1 NW /PERFLOG- A/ NW|COMMAND'
 do i=1 to file.0
    'EXEC SENDFILE' file.i 'TO MAINT(NOLOG'
    if rc=0 then 'RENAME' file.i '= PERFLOG- ='
    'RENAME' file.i '= PERFLOG- A'
    /* Start a new file */
    'EXECIO 1 DISKW' file.i '(FINIS STRING',
               'Time    PagRd   DatRd   PagWr   DatWr      IO  CPU'
 end i
/*>>end TAILOR<<*/
exit rc
/*******************************************************************/
SYNTAX: /*  we come here when SIGNAL ON SYNTAX traps an error      */
/*******************************************************************/
 parse upper source . how myname mytype . syn .
 call errexit rc,'REXX problem in' myname mytype 'line' sigl':' ,
      'ERRORTEXT'(rc), sigl':'||'SOURCELINE'(sigl)
/*******************************************************************/
ERREXIT: /*  exit with retcode & errormsg                          */
/*******************************************************************/
 do i=2 to 'ARG'()
    say 'ARG'(i)
 end
 'EXEC REXXVARS'               /* show value of all REXX variables */
 exit arg(1)
/*******************************************************************/
SubPipe:
/*******************************************************************/
 Signal on Error
 do forever
    'PEEKTO Data'
    interpret "'OUTPUT'" data
    'READTO'
 end
Error:
exit rc*(ec<>12)

(1) As you could read in the first article, SQL/DS was renamed to IBM DB2 Server for VSE & VM. We use the short notation DB2/VM here.