VM/ESA Data in Memory Techniques
Note: This discussion is adapted from a paper by Kris Buelens and Guy De Ceulaer -- IBM Belgium.
VM/ESA provides so many different techniques to put data in memory and/or to share storage that some people get a bit lost: for example: one may think that a VM dataspace can replace a saved segment. All data in memory techniques are meant to boost the system's performance by reducing or eliminating I/Os or reducing real storage consumption via sharing.
We, the authors, always want to explain the basics, so that you can understand the reasons why some technique is better in some circonstances. So, here we'll start with basics too.
Terminology
Before we can discuss the pros and cons of the different data in memory techniques, we need to review some VM terminology.- Module
- A module is a CMS file containing an executable program. A module resides on a minidisk or SFS directory and has to be loaded into storage before execution.
- Nucleus Resident
- Most CMS commands are not modules, but are nucleus resident programs. That is, their coding is included in the CMS nucleus and, as the nucleus resides in a saved segment, their code does not need to be read from disk at each invocation.
- Nucleus Extension
- A module can be made resident in storage via the NUCXLOAD command. This means that it is loaded from disk to storage only once and can then repeatedly be invoked from there.
- Linkedit
- Is the process that "glues" together different parts of a program (such as subroutines). In CMS, those parts have a filetype of TEXT or are members of a TXTLIB. The process replaces the names of called subroutines by the address at which they are loaded. LOAD is the native CMS linkage editor and it creates MODULES. LKED is a CMS command that calls the MVS linkage editor and creates executables in a LOADLIB.
- Relocatable
- A program is said to be relocatable if it can be executed at another address than the one at which it has been linkedited. The linkage editor can save the list of subroutines in the module, permitting the loader to adapt their addresses to the storage locations where they are loaded, just before execution. This is required for a program to be NUCXLOADed. A CMS program can be made relocatable by using the RLDSAVE option on the LOAD command.
- Reusable
- A program is reusable when it can be re-executed without reload from disk. In practice, this means the program may have to re-initialize any area it might have changed during a previous execution. It is obvious that the programmer has to take care of this. This is also required for a program to be NUCXLOADed.
- Re-entrant
- For a program to be re-entrant it must not modify itself, including data areas that are imbedded within the program. To get a re-entrant program, the programmer has to use specific techniques. As saved segments are R/O, re-entrancy is, of course, a requirement for a program to be included in a saved segment (otherwise modifications applied by one virtual machine would also influence the program's behaviour in other virtual machines using the same segment).
- Address Space
- Is the addressable storage area where programs are executed and data resides. When a virtual machine is created, CP immediately creates this space. The size of this space is equal to the so-called VM size, defined in the CP directory, and can be altered by a CP DEFine STORage command. An address space is divided into segments of 1 Mbyte each (in the XA architecture; was 64K in 370 architecture).
- Segment Tables, Page Tables and Page Management Blocks (PMB)
- Segment and pages tables are built by CP and used by the hardware to describe the virtual storage virtual machines. A segment table entry points to a page table. A page table has 256 elements, each entry describes the state of a virtual storage page. With the page table entries, the HW can find if a page is in real storage, and where it is located. In VM/ESA, a page table is placed at the start of a Page Management Block (PMB). The PMB also includes information that allows finding pages located on DASD (eg in the paging area's).
- Data spaces
-
Are similar to address spaces, but contain only data. Programs cannot be
executed directly from them. Data spaces can be shared among users and
are defined by the operating system on request of a program. Two types
exist: ESA/370 data spaces and ESA/390 VM data spaces.
ESA/370 data spaces are sharable only by operating systems running in paging mode (i.e. CP, MVS and VSE). They can exist on any ESA-capable processor.ESA/390 VM data spaces are exclusive to VM/ESA and can be shared between operating systems running with Dynamic Address Translation OFF (i.e. CMS). These require an ES/9000 machine. See also VM Data Spaces for futher discussion and guidance on tracking page I/O associated with VM data spaces.
Both address spaces and data spaces are virtual storage and thus are pageable.
- Saved Segment
- A saved segment (often called a shared segment) is an area of an address space that can be shared among different virtual machines. Saved segments can contain programs and/or data. Note: A saved segment itself is non-relocatable in that it will always be loaded at the same virtual address as where it was generated.
- Saved segmenst can be loaded Diagnose 64, or, better, by CMS' SEGMENT command or macro. Whena segment is loaded, CP changes the segment table of the virtual machine to make one or more entries point to the page tables of the saved segment. The pages of the saved segment are not directly paged in, this will only happen when users try to reference pages of the saved segment. When segment table entries of different virtual machines point to the same page tables, storage is shared. The information in the PMB will also guide CP's paging routines to page-in the pages from the spool (where the code of saved segments resides).
- LSEG
-
A logical saved segment (LSEG) is a CMS concept that eases the inclusion
of MODULEs, EXECs,etc. into a saved segment. A classical segment is one
big piece of coding, whereas an LSEG is a kind of library. When an LSEG
gets loaded, all its objects become "known" to the virtual machine. That
is, MODULEs are considered NUCXLOADed, EXECs become EXECLOADed, etc. From
then on, the fact that the elements are in a saved segment is transparent
and the saved segment makes the code of the objects sharable among virtual
machines.
An LSEG resides in a PSEG (Physical Segment). When loading an LSEG, CMS requests CP to get the appropriate PSEG. To CP, a PSEG is an ordinary saved segment. The SYSTEM SEGID file on the S-disk defines the relationship between LSEGs and PSEGs. LSEGs are created with the SEGGEN command (VMFBLD can call SEGGEN too).
- FST
- Each CMS-formatted minidisk has a directory (list of the files and their attributes). When a CMS minidisk (or an SFS directory) is accessed, this directory is copied from disk into the user's address space, where it is called a File Status Table (FST).
- CU Caching
- DASD control units can also keep data in their caching storage. When the data to be read are available from the cache, it is sent to the CPU roughly 10 times faster than when read from DASD. We won't discuss this technique any further as it only speeds up I/O and has not much to do with storage sharing. The Redbook VM/ESA Storage Management with Tuning Guidelines (GG24-3944) contains much useful information in this area and is recommended reading.
- Minidisk Caching
-
Minidisk caching (MDC) is a CP service to avoid disk I/O. When a virtual
machine reads a block from disk, CP saves a copy in real storage. From
then on, any user issuing an I/O for the same block gets it transparently
from CP's in-storage copy. CP has an arbiter to optimize the use of central
and/or expanded storage for MDC and paging.
Up through VM/ESA Release 1.2.1, MDC was limited to 4K-formatted CMS minidisks and caching was done in expanded storage only. Since VM/ESA Release 1.2.2, MDC is enhanced to support any minidisk (guest or CMS), and can use both central and expanded storage.
- SFS Data Spaces
- An SFS directory can be mapped to a VM data space (we'll abbreviate this technique as SFS-DS). For our discussion, SFS directories are similar to minidisks -- you typically ACCESS them before using the files. However, the SFS files are stored on the minidisks owned by the SFS server. If an SFS directory is associated with a VM data space, the SFS server shares the data space with any user referencing the files. This means that the transmission of file information is no longer over APPC/VM path between the SFS server and the CMS client, but is directly available in virtual storage. When the data space is created, the data blocks on the SFS minidisks are mapped to page frames in the data space. The SFS server itself will not read the data into the data space but, when the user references a file, CP will use its high-performing paging routines to get the referenced data blocks from the SFS minidisks into the data space. Other users referencing the same file refer to the same data space pages, so effectively sharing storage.
- Virtual Disks in Storage
- Virtual disks in storage (V-disks) were introduced by VM/ESA Release 1.2.1. A V-disk is a minidisk emulated in CP virtual storage (and thus can be paged out). Virtual disks in storge behave as fast 9336 FBA disks. CP creates them in an ESA/370 data space. When the system goes down, the data is lost. Virtual disks in storage can be shared between virtual machines. They are accessed using any I/O method supported by VM (SIO or SSCH for guests, Diagnose or BLOCKIO for CMS).
Comparing the techniques
We will now cover, one by one, the performance aspects for FSTs, programs, data files, REXX execs, and so forth. In general, performance can be improved by avoiding I/O and/or minimizing paging via storage sharing. But is a virtual disk in storage equivalent to a saved segment? Is a data space a winner? Do saved segments perform better than MDC?We start with programs. REXX execs are covered later because, to computers, REXX execs are ordinary data files that get read and handled by a real program, namely the REXX interpreter. CSP applications have a similar behavior, whereas compiled REXX execs are a special case of modules.
What's the best technique for programs?
One aspect you have to keep in mind is that, when a program executes, not necessarily all its subroutines will be executed. Exception or error routines are examples of that. Loading them in storage is thus a form of overhead that can only be avoided with some of the described techniques.For program products designed to use saved segments, the choice is clear -- you have to use them when you care for performance. Even if there is only one user, there is a gain as only referenced parts of the program will be paged in.
For MODULEs, you have the choice between:
- leaving them on a minidisk (and hope for MDC benefits)
- storing them in an SFS-DS
- copying them to a virtual disk in storage shared among all users
- NUCXLOADing them (if reusable)
- placing them in an LSEG (if re-entrant)
- In case 1, performance is improved when the MDC has a high hit ratio. When CMS reads the program, the I/O will be avoided if CP still has a copy in the MDC. But ..
- The running program itself is not shared, so each user of the program has a separate copy in private storage.
- The whole program must be read, including exception routines (it's likely that these will be selected for paged out later due to lack of reference).
- CP will not keep the program in MDC if it is not started frequently. CP tends to cache what is read frequently, and a program is only read when started.
- With an SFS-DS, the CMS file containing the program may be in real storage if it is often read. But since the program resides in a data space and programs can't be executed from data spaces, it must be moved to the user's private address space. So the same remarks for case 1 apply here as well.
- Storing on a virtual disk in storage is again very similar to the SFS-DS solution, but a virtual disk in storage is less practical for this purpose because after each IPL you'd have to copy the programs from a real disk to the virtual disk in storage. Virtual disks in storage are useful for items that don't support SFS. (such as VSAM-formatted minidisks or VSE guest minidisks).
- By NUCXLOADing, you only read the program once and keep it in private storage (from where it can be paged out to expanded storage or to DASD). However, the program isn't shared at all. The NUCXLOAD technique is suitable if the program is exclusive to one or a few users. For example, the action routines of a PROP should be NUCXLOADed (or EXECLOADed for execs) before starting the program.
- Yes, only saved segments allow effective sharing of programs among users and ,as an extra benefit, only those parts that get executed will be paged in.
- The best choice is to use saved segments whenever possible. However, they require more planning and maintenance from the systems programmer, while MDC and SFS-DS are more self-regulating processes.
- MDC, SFS-DS, and virtual disks in storage can speed up reading the program from disk but only if the file is read frequently.
What's the best technique for data files?
Data files can, of course, get the same benefits from MDC and virtual disks in storage as is the case for programs. Using MDC requires minimal effort from the systems programmer.However, data can be stored in saved segments too, giving the great advantage of sharing. The former 16 Mbyte limit explains why the technique was not used frequently in the past.
Normally, programs that want to read data from shared segments instead of disk have to be specifically designed to do so. However, with CMS Pipelines, reading an EXECLOADed file or reading from disk becomes transparent| Yes, loading a data file in storage with EXECLOAD is fooling CMS, but is works and is supported by CMS Pipelines. Try this, for example:
PIPE LITERAL Card 2 ? LITERAL Card 1 ? > TEST FILE A EXECLOAD TEST FILE A MYTEST DATA PIPE < MYTEST DATA ? CONSOLEBecause placing data in saved segments is so easy, it is worth considering for highly used data that is not frequently modified.
The very best would be the direct use of VM data spaces but that requires the program to be adapted to use the data spaces.
You can, however, indirectly benefit from VM data spaces via the SFS-DS technique, in which case your data processing program doesn't require changes. But realize that, compared to direct use of VM data spaces, you then share in "move mode". That is, when your program does a read to get data, CMS has to move it from the data space to the program's buffer. Note that with MDC, one also shares storage in "move mode".
Improving your execs
Note first that in this discussion, Xedit macros and Pipeline stages are also "execs" -- only the filetype differs. So do compiled REXX procedures, unless they are compiled into a TEXT object and link-edited into a module.For sharing and performance aspects, execs compare very well to what we explained for programs. So the list of possibilities is very similar:
- leave them on a minidisk (and hope for MDC benefits)
- store them in an SFS-DS
- copy them to a virtual disk in storage shared by all users
- EXECLOAD them
- place them in an LSEG or in the CMSINST segment
Conclusion: Place your highly used REXX execs in saved segments. Starting with VM/ESA 1.1.0, REXX execs can be placed above the 16M line, relieving the former space constraint. EXEC2 execs can also be put into saved segments, but only below 16M. Although it's clear that compiled execs run much faster, they result in about 4 times larger files. This means that sharing the coding and avoiding the I/O to load them is even more important once execs are compiled.
What about FSTs?
FSTs can consume a lot of storage. An FST entry for a minidisk file needs 64 bytes (an entry for an SFS file is a little bigger). Thus, when you ACCESS a minidisk with 5000 files, the cost in your address space is over 300K (75 pages), which all have to be read from disk too. And, even though CMS is clever enough to use hashing techniques to drastically minimize the number of pages to be scanned (2 pages/filemode), searching for files is a job CMS has to perform very frequently.How can we gain performance here?
- minidisk caching?
- saved segments?
Although MDC may speed up obtaining the FSTs during ACCESS, it will not help the process of scanning the FSTs. CMS keeps the FSTs in virtual storage so, to find a file, no I/O is required. Since MDC works by eliminating I/Os, it will not help here.
Yes, FSTs fit well into saved segments. The pages will effectively be
shared (no moves required), but they still must remain below 16M. SAVEFD
can be used to place FSTs in a "normal" physical segment, while SEGGEN
is the command to place them in an LSEG. The drawback is that each time
something changes on the minidisk, the segment must be resaved (use ACCESS
(SAVEONLY to verify if a segment is still valid). Remember also that the
FSTs for the S- and Y-disk are saved together with the CMS saved system,
so the 19E is a good candidate to receive frequently used files.
data spaces?
For an SFS-DS, the FSTs do reside in the shared data space and they don't have to be moved to your address space. The extra advantage of an SFS-DS over a CMS minidisk is that not only are the FSTs shared, but also the files themselves. Furthermore, the FSTs do not consume precious address space below 16M and they don't have to be manually resaved after files have been updated.
You can use the CP Q NSS USERS segname command to find out how many copies exist and who's using which copy. By restarting users of obsolete class P segments storage sharing is improved (the CPQUERY EXEC -available on the VM download library- can help you with this task).
Highest address
To complete this discussion, a short remark about the "highest" address one should try to respect with saved segments and virtual machine sizes.In order to describe your address space, CP has to build a so-called "segment table". Initially, your segment table has just enough entries to describe your virtual machine size. When you activate a saved segment that was generated at a higher address, CP needs to enlarge your segment table. When you later detach the segment, CP will not downsize your segment table because it figures that you may use the same segment again later on. Note that since CMS Release 6, saved segments can be loaded inside the virtual machine size (but adding SEGMENT RESERVE commands in the PROFILE or SYSPROF EXEC may be required). And, obviously, the storage occupied by the saved segments cannot be used as private R/W storage.
Knowing this, you should remember that there are three important limits:
- A segment table describing 32M fits into the base VMDBK (virtual machine description block) and so has no additional storage cost relative to smaller virtual address sizes.
- From 33M to 1024M, the segment table needs one extra 4K page/user.
- Above 1024M, CP needs yet one more page, and it must be contiguous with the other.
- Up to VM/ESA 2.2.0: you can place them very high, but not above 1024M: the extra real storage cost for a segment at 33M or at 1024M is exactly the same: 1 page per user.
- In VM/ESA 2.3.0 CP became a bit more clever: the un-used upper part of segment table pages is reclaimed by CP and used as system "free" storage.
- Before VM/ESA 2.3.0 that half page is indeed wasted; from 2.3.0 on,
CP can use it for free storage.
Concluding guidelines
Data in memory techniques can greatly enhance the performance of your system. These techniques eliminate I/Os and, when sharing among users is possible, they reduce real storage consumption.- saved segments
- Even with MDC and VM data spaces, saved segments are invaluable to share programs, execs and minidisk FSTs.
- VM data spaces
- An SFS directory in a VM data space performs as well as minidisks with MDC and shared FSTs. In addition, you get better disk management and you can share more data. Note, however, that SFS file control directories (with full support of aliases, sharing, etc.) can't be placed in a VM data space -- only directory control directories can.
- MDC
- MDC is a good performance booster for minidisks. Note that the CMS minidisks used by SQL/DS (now known as "DB2 for VM and VSE") and SFS servers are also eligible for MDC, resulting in an effect of "bigger buffers" for SFS and SQL/DS.
- Note though that since VM/ESA 1.2.2, the MDC uses by default full-track reads, what is good for sequential access (i.e. good for most CMS files), but not for random access, such as SQL databases. For SQL/DS, it is best to use the "SQL Dataspace Feature", or, if using VM/ESA 2.3.0 or 2.2.0+PTF UM28392, use the new "Record MDC". Probably, the minidisks used by the SFS catalog (storage pool 1) perform also better with "Record MDC".
- virtual disks in storage
- are primarily meant to be shared by VSE guests, used by old CMS applications, or used as a replacement for TDISKs. By old CMS applications, we mean applications that cannot profit from such things as large virtual storage, VM data spaces, or files in an SFS data space. Note, however, that virtual disks in storage are not for free. With the current design, CP considers the pages in use for a virtual disk in storage as shared storage, making them less eligible to be paged out. Hence, a single user with a big and very active virtual disk in storage, can take over a big part of central storage.
- NUCXLOAD
- (or EXECLOAD for procedures) is easy to implement and is especially useful when sharing is not important.
Creating Logical Segments
We have been telling that saved segments are still the best option for programs, execs and FSTs. Creating Logical segments is easy, but we guess that many readers are not familiar with it. Therefore it may be appropriate to mention how LSEGs can be created. For more details, refer to the VM/ESA Planning and Administration manual, or, have a look in the VM/ESA Performance manual.- Find out what objects you want to place in an LSEG: FSTs, MODULEs, EXECs, XEDIT macros, ... Except for FSTs, objects of different types can be placed in an LSEG. A PSEG hosts one or more LSEGs. So, you also ha ve to decide how many LSEGs and PSEGs you'll make. Here we'll suppose you will place your tools (REXX execs, XEDIT macro's and some MODULEs) in one LSEG
- Find a place in real storage to place the segment (remember that LSEGs with FSTs or EXEC2 execs must be located below 16M). To map the storage used by segments various tools exist:
- VMFSGMAP the official VM solution: issue EXEC VMFSGMAP SEGBLD ESASEGS SEGBLIST
- CPQUERY from the donwload library.
- The "quick&dirty" solution: QNSSMAP EXEC, which is appended below.
- Define the segment skeleton. Suppose you found room at 25Meg and 1 Meg is enough: issue
- Define your storage at least 1 Meg higher than the address of the segment.
- Use XEDIT to create the "mylseg LSEG" file. Insert lines to describe each object:
- Without INSTSEG, the exec is considered EXECLOADed, and it will be used even the user has a copy on his A-disk for example.
- With INSTSEG, when the exec will be found depends on the setting of INSTSEG. By default, INSTSEG is "ON S", which means that this exec will be found just before the search of the S-disk starts. So, if for example a STARTXED EXEC is found on the R-disk, the copy in the segment will not be executed, the disk resident version is take instead.
- Use XEDIT to create the "mypseg PSEG" file. In our case only 1 line is required:
- Access the minidisks containing your objects, and access CMS resident (MAINT 190) in R/W as Z (this way SEGGEN can update the SYSTEM SEGID file immediately)
- Save the PSEG and LSEG(s) by issuing:
- Now that the LSEG is created, you still must make your users use it. For FSTs, the ACCESS command will try to use the segment autmatically. For other objects, a "SEGMENT LOAD mylseg (SYSTEM" must be executed. So, you have to include a SEGMENT LOAD in the SYSPROF EXEC, or in another exec that your users surely execute before they use the code you so carefully placed in the LSEG.
(as we can't remember that command, we created a SEGMAP EXEC that simply issues the above command)
CP DEFSEG mypseg 1900-19FF SR
EXEC STARTXED EXEC * (INSTSEG EXEC SUBMIT EXEC * EXEC OURPROFL XEDIT * (INSTSEG EXEC OURFILEL EXEC * FILELIST EXEC MODULE MYBROWSE MODULE * EXEC SOMEPIPE REXX * (INSTSEGYou should remark the option INSTSEG. It means that this exec is considered to be part of the "CMS Installation Segment" (CMSINST by default). It influences when the exec will be found in the search order:
LSEG mylseg LSEG
SEGGEN mypseg PSEG A SYSTEM SEGID Z
The QNSSMAP EXEC
Here is the quick&dirty but fast QNSSMAP EXEC./* This exec creates a simple NSS MAP +-----------------------------------------------------------+ | format: | QNNSMAP <ALL> | +-----------------------------------------------------------+ - without option ALL: segment spaces, CMS and GCS segments are not listed - with option ALL: everything is listed Written by: ; Kris Buelens, IBM Belgium 10 Aug 1992*/ /* 13 Feb 1997: Don't XEDIT file if disconnected user */ parse upper source . . myname mytype . syn . address command parse upper arg all . if all='ALL' then /* All stuff wanted, also CMS NSS-es */ 'PIPE (end ?) CP Q NSS MAP', /* .. so must fill in cols 1-32 and */ '|D: DROP 1', /* .. 52-61 of all records */ '|F: FANOUT', '| SPEC 1-32 1 52.10 33|NFIND _'||, '|J: JUXTAPOSE', '| SPEC 1-32 1 43-* 33 33.10 52 |NFIND _'||, '| XLATE 33-37 A-F FA-FF', '| XLATE 20-20 S L', /* SORT S(pace) Before M(ember)*/ '| SORT 33-37 20', '| XLATE 33-37 FA-FF A-F', '| XLATE 20-20 L S', '|T: FANIN 1 0', '| > NSS MAP A', '?F:| SPEC 33-* |J:?D:|T:' else 'PIPE (end ?) CP Q NSS MAP', '|D: DROP 1', '|NFIND ___'||, '|NLOCATE 15.3 /NSS/', '|NLOCATE 15.6 /DCSS-S/', '|NLOCATE 15.6 /CPDCSS/', '|XLATE 33-37 A-F FA-FF', '|SORT 33-37', '|XLATE 33-37 FA-FF A-F', '|T: FANIN 1 0', '|> NSS MAP A', '?D:|T:' if rc=0 then do if linesize()>0 then 'EXEC REXEDIT NSS MAP A NORC' else say 'NSS map stored in file NSS MAP A' end exit rc