Note: This discussion is adapted from a paper by Kris Buelens and Guy De Ceulaer -- IBM Belgium.

VM/ESA provides so many different techniques to put data in memory and/or to share storage that some people get a bit lost: for example: one may think that a VM dataspace can replace a saved segment.  All data in memory techniques are meant to boost the system's performance by reducing or eliminating I/Os or reducing real storage consumption via sharing.

We, the authors, always want to explain the basics, so that you can understand the reasons why some technique is better in some circonstances.  So, here we'll start with basics too.

• Terminology

Terminology

Module

A module is a CMS file containing an executable program. A module resides on a minidisk or SFS directory and has to be loaded into storage before execution.

Nucleus Resident

Most CMS commands are not modules, but are nucleus resident programs. That is, their coding is included in the CMS nucleus and, as the nucleus resides in a saved segment, their code does not need to be read from disk at each invocation.

Nucleus Extension

A module can be made resident in storage via the NUCXLOAD command. This means that it is loaded from disk to storage only once and can then repeatedly be invoked from there.

Linkedit

Is the process that "glues" together different parts of a program (such as subroutines). In CMS, those parts have a filetype of TEXT or are members of a TXTLIB. The process replaces the names of called subroutines by the address at which they are loaded. LOAD is the native CMS linkage editor and it creates MODULES. LKED is a CMS command that calls the MVS linkage editor and creates executables in a LOADLIB.

Relocatable

A program is said to be relocatable if it can be executed at another address than the one at which it has been linkedited. The linkage editor can save the list of subroutines in the module, permitting the loader to adapt their addresses to the storage locations where they are loaded, just before execution. This is required for a program to be NUCXLOADed. A CMS program can be made relocatable by using the RLDSAVE option on the LOAD command.

Reusable

A program is reusable when it can be re-executed without reload from disk. In practice, this means the program may have to re-initialize any area it might have changed during a previous execution. It is obvious that the programmer has to take care of this. This is also required for a program to be NUCXLOADed.

Re-entrant

For a program to be re-entrant it must not modify itself, including data areas that are imbedded within the program.  To get a re-entrant program, the programmer has to use specific techniques.  As saved segments are R/O, re-entrancy is, of course, a requirement for a program to be included in a saved segment (otherwise modifications applied by one virtual machine would also influence the program's behaviour in other virtual machines using the same segment).

Address Space

Is the addressable storage area where programs are executed and data resides. When a virtual machine is created, CP immediately creates this space. The size of this space is equal to the so-called VM size, defined in the CP directory, and can be altered by a CP DEFine STORage command. An address space is divided into segments of 1 Mbyte each (in the XA architecture; was 64K in 370 architecture).

Segment Tables, Page Tables and Page Management Blocks (PMB)

Segment and pages tables are built by CP and used by the hardware to describe the virtual storage virtual machines.  A segment table entry points to a page table.  A page table has 256 elements, each entry describes the state of a virtual storage page.   With the page table entries, the HW can find if a page is in real storage, and where it is located.  In VM/ESA, a page table is placed at the start of a Page Management Block (PMB).  The PMB also includes information that allows finding pages located on DASD (eg in the paging area's).

Data spaces

Are similar to address spaces, but contain only data. Programs cannot be executed directly from them. Data spaces can be shared among users and are defined by the operating system on request of a program. Two types exist: ESA/370 data spaces and ESA/390 VM data spaces.

 

ESA/370 data spaces are sharable only by operating systems running in paging mode (i.e. CP, MVS and VSE). They can exist on any ESA-capable processor.

 

ESA/390 VM data spaces are exclusive to VM/ESA and can be shared between operating systems running with Dynamic Address Translation OFF (i.e. CMS). These require an ES/9000 machine. See also VM Data Spaces for futher discussion and guidance on tracking page I/O associated with VM data spaces.

 

Both address spaces and data spaces are virtual storage and thus are pageable.

Saved Segment

A saved segment (often called a shared segment) is an area of an address space that can be shared among different virtual machines. Saved segments can contain programs and/or data. Note: A saved segment itself is non-relocatable in that it will always be loaded at the same virtual address as where it was generated.

Saved segmenst can be loaded Diagnose 64, or, better, by CMS' SEGMENT command or macro.  Whena  segment is loaded, CP changes the segment table of the virtual machine to make one or more entries point to the page tables of the saved segment.  The pages of the saved segment are not directly paged in, this will only happen when users try to reference pages of the saved segment.  When segment table entries of different virtual machines point to the same page tables, storage is shared.   The information in the PMB will also guide CP's paging routines to page-in the pages from the spool (where the code of saved segments resides).

LSEG

A logical saved segment (LSEG) is a CMS concept that eases the inclusion of MODULEs, EXECs,etc. into a saved segment. A classical segment is one big piece of coding, whereas an LSEG is a kind of library. When an LSEG gets loaded, all its objects become "known" to the virtual machine. That is, MODULEs are considered NUCXLOADed, EXECs become EXECLOADed, etc. From then on, the fact that the elements are in a saved segment is transparent and the saved segment makes the code of the objects sharable among virtual machines.

 

An LSEG resides in a PSEG (Physical Segment). When loading an LSEG, CMS requests CP to get the appropriate PSEG. To CP, a PSEG is an ordinary saved segment. The SYSTEM SEGID file on the S-disk defines the relationship between LSEGs and PSEGs. LSEGs are created with the SEGGEN command (VMFBLD can call SEGGEN too).

FST

Each CMS-formatted minidisk has a directory (list of the files and their attributes). When a CMS minidisk (or an SFS directory) is accessed, this directory is copied from disk into the user's address space, where it is called a File Status Table (FST).

CU Caching

DASD control units can also keep data in their caching storage. When the data to be read are available from the cache, it is sent to the CPU roughly 10 times faster than when read from DASD. We won't discuss this technique any further as it only speeds up I/O and has not much to do with storage sharing. The Redbook VM/ESA Storage Management with Tuning Guidelines (GG24-3944) contains much useful information in this area and is recommended reading.

Minidisk Caching

Minidisk caching (MDC) is a CP service to avoid disk I/O. When a virtual machine reads a block from disk, CP saves a copy in real storage. From then on, any user issuing an I/O for the same block gets it transparently from CP's in-storage copy. CP has an arbiter to optimize the use of central and/or expanded storage for MDC and paging.

 

Up through VM/ESA Release 1.2.1, MDC was limited to 4K-formatted CMS minidisks and caching was done in expanded storage only. Since VM/ESA Release 1.2.2, MDC is enhanced to support any minidisk (guest or CMS), and can use both central and expanded storage.

SFS Data Spaces

An SFS directory can be mapped to a VM data space (we'll abbreviate this technique as SFS-DS). For our discussion, SFS directories are similar to minidisks -- you typically ACCESS them before using the files. However, the SFS files are stored on the minidisks owned by the SFS server. If an SFS directory is associated with a VM data space, the SFS server shares the data space with any user referencing the files. This means that the transmission of file information is no longer over APPC/VM path between the SFS server and the CMS client, but is directly available in virtual storage. When the data space is created, the data blocks on the SFS minidisks are mapped to page frames in the data space. The SFS server itself will not read the data into the data space but, when the user references a file, CP will use its high-performing paging routines to get the referenced data blocks from the SFS minidisks into the data space. Other users referencing the same file refer to the same data space pages, so effectively sharing storage.

Virtual Disks in Storage

Virtual disks in storage (V-disks) were introduced by VM/ESA Release 1.2.1. A V-disk is a minidisk emulated in CP virtual storage (and thus can be paged out). Virtual disks in storge behave as fast 9336 FBA disks. CP creates them in an ESA/370 data space. When the system goes down, the data is lost. Virtual disks in storage can be shared between virtual machines. They are accessed using any I/O method supported by VM (SIO or SSCH for guests, Diagnose or BLOCKIO for CMS).

Comparing the techniques

We start with programs. REXX execs are covered later because, to computers, REXX execs are ordinary data files that get read and handled by a real program, namely the REXX interpreter. CSP applications have a similar behavior, whereas compiled REXX execs are a special case of modules.

What's the best technique for programs?

For program products designed to use saved segments, the choice is clear -- you have to use them when you care for performance. Even if there is only one user, there is a gain as only referenced parts of the program will be paged in.

For MODULEs, you have the choice between:

1. leaving them on a minidisk (and hope for MDC benefits)
2. storing them in an SFS-DS
3. copying them to a virtual disk in storage shared among all users
4. NUCXLOADing them (if reusable)
5. placing them in an LSEG (if re-entrant)

1. In case 1, performance is improved when the MDC has a high hit ratio. When CMS reads the program, the I/O will be avoided if CP still has a copy in the MDC. But ..
• The running program itself is not shared, so each user of the program has a separate copy in private storage.
• The whole program must be read, including exception routines (it's likely that these will be selected for paged out later due to lack of reference).
• CP will not keep the program in MDC if it is not started frequently. CP tends to cache what is read frequently, and a program is only read when started.
2. With an SFS-DS, the CMS file containing the program may be in real storage if it is often read. But since the program resides in a data space and programs can't be executed from data spaces, it must be moved to the user's private address space. So the same remarks for case 1 apply here as well.
3. Storing on a virtual disk in storage is again very similar to the SFS-DS solution, but a virtual disk in storage is less practical for this purpose because after each IPL you'd have to copy the programs from a real disk to the virtual disk in storage. Virtual disks in storage are useful for items that don't support SFS. (such as VSAM-formatted minidisks or VSE guest minidisks).
4. By NUCXLOADing, you only read the program once and keep it in private storage (from where it can be paged out to expanded storage or to DASD). However, the program isn't shared at all. The NUCXLOAD technique is suitable if the program is exclusive to one or a few users. For example, the action routines of a PROP should be NUCXLOADed (or EXECLOADed for execs) before starting the program.
5. Yes, only saved segments allow effective sharing of programs among users and ,as an extra benefit, only those parts that get executed will be paged in.

• The best choice is to use saved segments whenever possible. However, they require more planning and maintenance from the systems programmer, while MDC and SFS-DS are more self-regulating processes.
• MDC, SFS-DS, and virtual disks in storage can speed up reading the program from disk but only if the file is read frequently.

What's the best technique for data files?

However, data can be stored in saved segments too, giving the great advantage of sharing. The former 16 Mbyte limit explains why the technique was not used frequently in the past.

Normally, programs that want to read data from shared segments instead of disk have to be specifically designed to do so. However, with CMS Pipelines, reading an EXECLOADed file or reading from disk becomes transparent| Yes, loading a data file in storage with EXECLOAD is fooling CMS, but is works and is supported by CMS Pipelines. Try this, for example:

      PIPE LITERAL Card 2 ? LITERAL Card 1 ? > TEST FILE A
 
      EXECLOAD TEST FILE A MYTEST DATA
 
      PIPE < MYTEST DATA ? CONSOLE

The very best would be the direct use of VM data spaces but that requires the program to be adapted to use the data spaces.

You can, however, indirectly benefit from VM data spaces via the SFS-DS technique, in which case your data processing program doesn't require changes. But realize that, compared to direct use of VM data spaces, you then share in "move mode".  That is, when your program does a read to get data, CMS has to move it from the data space to the program's buffer.  Note that with MDC, one also shares storage in "move mode".

Improving your execs

For sharing and performance aspects, execs compare very well to what we explained for programs. So the list of possibilities is very similar:

1. leave them on a minidisk (and hope for MDC benefits)
2. store them in an SFS-DS
3. copy them to a virtual disk in storage shared by all users
4. EXECLOAD them
5. place them in an LSEG or in the CMSINST segment

Conclusion: Place your highly used REXX execs in saved segments. Starting with VM/ESA 1.1.0, REXX execs can be placed above the 16M line, relieving the former space constraint. EXEC2 execs can also be put into saved segments, but only below 16M. Although it's clear that compiled execs run much faster, they result in about 4 times larger files. This means that sharing the coding and avoiding the I/O to load them is even more important once execs are compiled.

What about FSTs?

How can we gain performance here?

1. minidisk caching?

Although MDC may speed up obtaining the FSTs during ACCESS, it will not help the process of scanning the FSTs. CMS keeps the FSTs in virtual storage so, to find a file, no I/O is required. Since MDC works by eliminating I/Os, it will not help here.

2. saved segments?

Yes, FSTs fit well into saved segments. The pages will effectively be shared (no moves required), but they still must remain below 16M. SAVEFD can be used to place FSTs in a "normal" physical segment, while SEGGEN is the command to place them in an LSEG. The drawback is that each time something changes on the minidisk, the segment must be resaved (use ACCESS (SAVEONLY to verify if a segment is still valid). Remember also that the FSTs for the S- and Y-disk are saved together with the CMS saved system, so the 19E is a good candidate to receive frequently used files.
 
 data spaces?

For an SFS-DS, the FSTs do reside in the shared data space and they don't have to be moved to your address space. The extra advantage of an SFS-DS over a CMS minidisk is that not only are the FSTs shared, but also the files themselves. Furthermore, the FSTs do not consume precious address space below 16M and they don't have to be manually resaved after files have been updated.

Highest address

In order to describe your address space, CP has to build a so-called "segment table". Initially, your segment table has just enough entries to describe your virtual machine size. When you activate a saved segment that was generated at a higher address, CP needs to enlarge your segment table. When you later detach the segment, CP will not downsize your segment table because it figures that you may use the same segment again later on. Note that since CMS Release 6, saved segments can be loaded inside the virtual machine size (but adding SEGMENT RESERVE commands in the PROFILE or SYSPROF EXEC may be required).  And, obviously, the storage occupied by the saved segments cannot be used as private R/W storage.

Knowing this, you should remember that there are three important limits:

1. A segment table describing 32M fits into the base VMDBK (virtual machine description block) and so has no additional storage cost relative to smaller virtual address sizes.
2. From 33M to 1024M, the segment table needs one extra 4K page/user.
3. Above 1024M, CP needs yet one more page, and it must be contiguous with the other.

• Up to VM/ESA 2.2.0: you can place them very high, but not above 1024M: the extra real storage cost for a segment at 33M or at 1024M is exactly the same: 1 page per user.
• In VM/ESA 2.3.0 CP became a bit more clever: the un-used upper part of segment table pages is reclaimed by CP and used as system "free" storage.

Before VM/ESA 2.3.0 that half page is indeed wasted; from 2.3.0 on, CP can use it for free storage.

Concluding guidelines

saved segments

Even with MDC and VM data spaces, saved segments are invaluable to share programs, execs and minidisk FSTs.

VM data spaces

An SFS directory in a VM data space performs as well as minidisks with MDC and shared FSTs. In addition, you get better disk management and you can share more data. Note, however, that SFS file control directories (with full support of aliases, sharing, etc.) can't be placed in a VM data space -- only directory control directories can.

MDC

MDC is a good performance booster for minidisks. Note that the CMS minidisks used by SQL/DS (now known as "DB2 for VM and VSE") and SFS servers are also eligible for MDC, resulting in an effect of "bigger buffers" for SFS and SQL/DS.

Note though that since VM/ESA 1.2.2, the MDC uses by default full-track reads, what is good for sequential access (i.e. good for most CMS files), but not for random access, such as SQL databases.  For SQL/DS, it is best to use the "SQL Dataspace Feature", or, if using VM/ESA 2.3.0 or 2.2.0+PTF UM28392, use the new "Record MDC".  Probably, the minidisks used by the SFS catalog (storage pool 1) perform also better with "Record MDC".

virtual disks in storage

are primarily meant to be shared by VSE guests, used by old CMS applications, or used as a replacement for TDISKs. By old CMS applications, we mean applications that cannot profit from such things as large virtual storage, VM data spaces, or files in an SFS data space. Note, however, that virtual disks in storage are not for free. With the current design, CP considers the pages in use for a virtual disk in storage as shared storage, making them less eligible to be paged out. Hence, a single user with a big and very active virtual disk in storage, can take over a big part of central storage.

NUCXLOAD

(or EXECLOAD for procedures) is easy to implement and is especially useful when sharing is not important.

Creating Logical Segments

1. Find out what objects you want to place in an LSEG: FSTs, MODULEs, EXECs, XEDIT macros, ...   Except for FSTs, objects of different types can be placed in an LSEG.  A PSEG hosts one or more LSEGs.  So, you also ha    ve to decide how many LSEGs and PSEGs you'll make.  Here we'll suppose you will place your tools (REXX execs, XEDIT macro's and some MODULEs) in one LSEG
2. Find a place in real storage to place the segment (remember that LSEGs with FSTs or EXEC2 execs must be located below 16M).  To map the storage used by segments various tools exist:
1. VMFSGMAP the official VM solution: issue EXEC VMFSGMAP SEGBLD ESASEGS SEGBLIST

(as we can't remember that command, we created a SEGMAP EXEC that simply issues the above command)
2. CPQUERY from the donwload library.
3. The "quick&dirty" solution: QNSSMAP EXEC, which is appended below.
3. Define the segment skeleton.  Suppose you found room at 25Meg and 1 Meg is enough: issue

CP DEFSEG mypseg 1900-19FF SR
4. Define your storage at least 1 Meg higher than the address of the segment.
5. Use XEDIT to create the "mylseg LSEG" file.  Insert lines to describe each object:

  EXEC  STARTXED EXEC    *               (INSTSEG     
  EXEC  SUBMIT   EXEC    *               
  EXEC  OURPROFL XEDIT   *               (INSTSEG     
  EXEC  OURFILEL EXEC    * FILELIST EXEC 
  MODULE MYBROWSE MODULE *              
  EXEC  SOMEPIPE REXX    *               (INSTSEG

You should remark the option INSTSEG.  It means that this exec is considered to be part of the "CMS Installation Segment" (CMSINST by default).  It influences when the exec will be found in the search order:
• Without INSTSEG, the exec is considered EXECLOADed, and it will be used even the user has a copy on his A-disk for example.
• With INSTSEG, when the exec will be found depends on the setting of INSTSEG.  By default, INSTSEG is "ON S", which means that this exec will be found just before the search of the S-disk starts.  So, if for example a STARTXED EXEC is found on the R-disk, the copy in the segment will not be executed, the disk resident version is take instead.
So, you must carefully think about the INSTSEG option.  EXECMAP can be used to see how often execs are executed.
6. Use XEDIT to create the "mypseg PSEG" file.  In our case only 1 line is required:

  LSEG mylseg LSEG

7. Access the minidisks containing your objects, and access CMS resident (MAINT 190) in R/W as Z (this way SEGGEN can update the SYSTEM SEGID file immediately)
8. Save the PSEG and LSEG(s) by issuing:

  SEGGEN mypseg PSEG A SYSTEM SEGID Z

9. Now that the LSEG is created, you still must make your users use it.  For FSTs, the ACCESS command will try to use the segment autmatically.  For other objects, a "SEGMENT LOAD mylseg (SYSTEM" must be executed.  So, you have to include a SEGMENT LOAD in the SYSPROF EXEC, or in another exec that your users surely execute before they use the code you so carefully placed in the LSEG.

The QNSSMAP EXEC

/* This exec creates a simple NSS MAP
    +-----------------------------------------------------------+
    | format:  | QNNSMAP  <ALL>                                 |
    +-----------------------------------------------------------+
   - without option ALL: segment spaces, CMS and GCS segments are not listed
   - with option ALL: everything is listed 
  Written by: ; Kris Buelens, IBM Belgium  10 Aug 1992*/
/* 13 Feb 1997: Don't XEDIT file if disconnected user              */
parse upper source . . myname mytype . syn .
address command
parse upper arg all .
if all='ALL' then               /* All stuff wanted, also CMS NSS-es */
   'PIPE (end ?) CP Q NSS MAP', /* .. so must fill in cols 1-32 and  */
           '|D:  DROP 1',       /* .. 52-61 of all records */
           '|F:  FANOUT',
           '|    SPEC 1-32 1 52.10 33|NFIND _'||,
           '|J:  JUXTAPOSE',
           '|    SPEC 1-32 1 43-* 33 33.10 52 |NFIND _'||,
           '|    XLATE 33-37 A-F FA-FF',
           '|    XLATE 20-20 S L',      /* SORT S(pace) Before M(ember)*/
           '|    SORT 33-37 20',
           '|    XLATE 33-37 FA-FF A-F',
           '|    XLATE 20-20 L S',
           '|T:  FANIN 1 0',
           '|    > NSS MAP A',
           '?F:| SPEC 33-* |J:?D:|T:'
else
   'PIPE (end ?) CP Q NSS MAP',
         '|D: DROP 1',
         '|NFIND ___'||,
         '|NLOCATE 15.3 /NSS/',
         '|NLOCATE 15.6 /DCSS-S/',
         '|NLOCATE 15.6 /CPDCSS/',
         '|XLATE 33-37 A-F FA-FF',
         '|SORT 33-37',
         '|XLATE 33-37 FA-FF A-F',
         '|T: FANIN 1 0',
         '|> NSS MAP A',
         '?D:|T:'
if rc=0 then do
   if linesize()>0 then 'EXEC REXEDIT NSS MAP A NORC'
   else say 'NSS map stored in file NSS MAP A'
end
exit rc