VM/ESA Data in Memory Techniques
Note: This discussion is adapted from a paper by Kris Buelens and
Guy De Ceulaer -- IBM Belgium.
VM/ESA provides so many different techniques to put data in memory and/or
to share storage that some people get a bit lost: for example: one may
think that a VM dataspace can replace a saved segment. All data in
memory techniques are meant to boost the system's performance by reducing
or eliminating I/Os or reducing real storage consumption via sharing.
We, the authors, always want to explain the basics, so that you can
understand the reasons why some technique is better in some circonstances.
So, here we'll start with basics too.
Before we can discuss the pros and cons of the different data in memory
techniques, we need to review some VM terminology.
Note: the pages in real storage being used by saved segment, shared dataspaces
and V-disks is considered as "shared storage", what means that these pages
are selected for paged out later than other pages, regardless of how many
virtual machines actually use them.
A module is a CMS file containing an executable program. A module resides
on a minidisk or SFS directory and has to be loaded into storage before
Most CMS commands are not modules, but are nucleus resident programs.
That is, their coding is included in the CMS nucleus and, as the nucleus
resides in a saved segment, their code does not need to be read from disk
at each invocation.
A module can be made resident in storage via the NUCXLOAD command. This
means that it is loaded from disk to storage only once and can then repeatedly
be invoked from there.
Is the process that "glues" together different parts of a program (such
as subroutines). In CMS, those parts have a filetype of TEXT or are members
of a TXTLIB. The process replaces the names of called subroutines by the
address at which they are loaded. LOAD is the native CMS linkage editor
and it creates MODULES. LKED is a CMS command that calls the MVS linkage
editor and creates executables in a LOADLIB.
A program is said to be relocatable if it can be executed at another
address than the one at which it has been linkedited. The linkage editor
can save the list of subroutines in the module, permitting the loader to
adapt their addresses to the storage locations where they are loaded, just
before execution. This is required for a program to be NUCXLOADed. A CMS
program can be made relocatable by using the RLDSAVE option on the LOAD
A program is reusable when it can be re-executed without reload
from disk. In practice, this means the program may have to re-initialize
any area it might have changed during a previous execution. It is obvious
that the programmer has to take care of this. This is also required for
a program to be NUCXLOADed.
For a program to be re-entrant it must not modify itself, including
data areas that are imbedded within the program. To get a re-entrant
program, the programmer has to use specific techniques. As saved
segments are R/O, re-entrancy is, of course, a requirement for a program
to be included in a saved segment (otherwise modifications applied by one
virtual machine would also influence the program's behaviour in other virtual
machines using the same segment).
Is the addressable storage area where programs are executed and data resides.
When a virtual machine is created, CP immediately creates this space. The
size of this space is equal to the so-called VM size, defined in the CP
directory, and can be altered by a CP DEFine STORage command. An address
space is divided into segments of 1 Mbyte each (in the XA architecture;
was 64K in 370 architecture).
Segment Tables, Page Tables and Page Management Blocks (PMB)
Segment and pages tables are built by CP and used by the hardware to describe
the virtual storage virtual machines. A segment table entry points
to a page table. A page table has 256 elements, each entry describes
the state of a virtual storage page. With the page table entries,
the HW can find if a page is in real storage, and where it is located.
In VM/ESA, a page table is placed at the start of a Page Management Block
(PMB). The PMB also includes information that allows finding pages
located on DASD (eg in the paging area's).
Are similar to address spaces, but contain only data. Programs cannot be
executed directly from them. Data spaces can be shared among users and
are defined by the operating system on request of a program. Two types
exist: ESA/370 data spaces and ESA/390 VM data spaces.
ESA/370 data spaces are sharable only by operating systems running
in paging mode (i.e. CP, MVS and VSE). They can exist on any ESA-capable
ESA/390 VM data spaces are exclusive to VM/ESA and can be shared between
operating systems running with Dynamic Address Translation OFF (i.e. CMS).
These require an ES/9000 machine. See also VM
Data Spaces for futher discussion and guidance on tracking page I/O
associated with VM data spaces.
Both address spaces and data spaces are virtual storage and thus are
A saved segment (often called a shared segment) is an area of an address
space that can be shared among different virtual machines. Saved segments
can contain programs and/or data. Note: A saved segment itself is non-relocatable
in that it will always be loaded at the same virtual address as where it
Saved segmenst can be loaded Diagnose 64, or, better, by CMS' SEGMENT command
or macro. Whena segment is loaded, CP changes the segment table
of the virtual machine to make one or more entries point to the page tables
of the saved segment. The pages of the saved segment are not directly
paged in, this will only happen when users try to reference pages of the
saved segment. When segment table entries of different virtual machines
point to the same page tables, storage is shared. The information
in the PMB will also guide CP's paging routines to page-in the pages from
the spool (where the code of saved segments resides).
A logical saved segment (LSEG) is a CMS concept that eases the inclusion
of MODULEs, EXECs,etc. into a saved segment. A classical segment is one
big piece of coding, whereas an LSEG is a kind of library. When an LSEG
gets loaded, all its objects become "known" to the virtual machine. That
is, MODULEs are considered NUCXLOADed, EXECs become EXECLOADed, etc. From
then on, the fact that the elements are in a saved segment is transparent
and the saved segment makes the code of the objects sharable among virtual
An LSEG resides in a PSEG (Physical Segment). When loading an LSEG,
CMS requests CP to get the appropriate PSEG. To CP, a PSEG is an ordinary
saved segment. The SYSTEM SEGID file on the S-disk defines the relationship
between LSEGs and PSEGs. LSEGs are created with the SEGGEN command (VMFBLD
can call SEGGEN too).
Each CMS-formatted minidisk has a directory (list of the files and their
attributes). When a CMS minidisk (or an SFS directory) is accessed, this
directory is copied from disk into the user's address space, where it is
called a File Status Table (FST).
DASD control units can also keep data in their caching storage. When the
data to be read are available from the cache, it is sent to the CPU roughly
10 times faster than when read from DASD. We won't discuss this technique
any further as it only speeds up I/O and has not much to do with storage
sharing. The Redbook VM/ESA Storage Management with Tuning Guidelines
(GG24-3944) contains much useful information in this area and is recommended
Minidisk caching (MDC) is a CP service to avoid disk I/O. When a virtual
machine reads a block from disk, CP saves a copy in real storage. From
then on, any user issuing an I/O for the same block gets it transparently
from CP's in-storage copy. CP has an arbiter to optimize the use of central
and/or expanded storage for MDC and paging.
Up through VM/ESA Release 1.2.1, MDC was limited to 4K-formatted CMS
minidisks and caching was done in expanded storage only. Since VM/ESA Release
1.2.2, MDC is enhanced to support any minidisk (guest or CMS), and can
use both central and expanded storage.
SFS Data Spaces
An SFS directory can be mapped to a VM data space (we'll abbreviate this
technique as SFS-DS). For our discussion, SFS directories are similar to
minidisks -- you typically ACCESS them before using the files. However,
the SFS files are stored on the minidisks owned by the SFS server. If an
SFS directory is associated with a VM data space, the SFS server shares
the data space with any user referencing the files. This means that the
transmission of file information is no longer over APPC/VM path between
the SFS server and the CMS client, but is directly available in virtual
storage. When the data space is created, the data blocks on the SFS minidisks
are mapped to page frames in the data space. The SFS server itself will
not read the data into the data space but, when the user references a file,
CP will use its high-performing paging routines to get the referenced data
blocks from the SFS minidisks into the data space. Other users referencing
the same file refer to the same data space pages, so effectively sharing
Virtual Disks in Storage
Virtual disks in storage (V-disks) were introduced by VM/ESA Release 1.2.1.
A V-disk is a minidisk emulated in CP virtual storage (and thus can be
paged out). Virtual disks in storge behave as fast 9336 FBA disks. CP creates
them in an ESA/370 data space. When the system goes down, the data is lost.
Virtual disks in storage can be shared between virtual machines. They are
accessed using any I/O method supported by VM (SIO or SSCH for guests,
Diagnose or BLOCKIO for CMS).
We will now cover, one by one, the performance aspects for FSTs, programs,
data files, REXX execs, and so forth. In general, performance can be improved
by avoiding I/O and/or minimizing paging via storage sharing. But is a
virtual disk in storage equivalent to a saved segment? Is a data space
a winner? Do saved segments perform better than MDC?
We start with programs. REXX execs are covered later because, to computers,
REXX execs are ordinary data files that get read and handled by a real
program, namely the REXX interpreter. CSP applications have a similar behavior,
whereas compiled REXX execs are a special case of modules.
One aspect you have to keep in mind is that, when a program executes, not
necessarily all its subroutines will be executed. Exception or error routines
are examples of that. Loading them in storage is thus a form of overhead
that can only be avoided with some of the described techniques.
For program products designed to use saved segments, the choice is clear
-- you have to use them when you care for performance. Even if there is
only one user, there is a gain as only referenced parts of the program
will be paged in.
For MODULEs, you have the choice between:
What are the pros and cons of each alternative?
leaving them on a minidisk (and hope for MDC benefits)
storing them in an SFS-DS
copying them to a virtual disk in storage shared among all users
NUCXLOADing them (if reusable)
placing them in an LSEG (if re-entrant)
In case 1, performance is improved when the MDC has a high hit ratio. When
CMS reads the program, the I/O will be avoided if CP still has a copy in
the MDC. But ..
With an SFS-DS, the CMS file containing the program may be in real storage
if it is often read. But since the program resides in a data space and
programs can't be executed from data spaces, it must be moved to the user's
private address space. So the same remarks for case 1 apply here as well.
Storing on a virtual disk in storage is again very similar to the SFS-DS
solution, but a virtual disk in storage is less practical for this purpose
because after each IPL you'd have to copy the programs from a real disk
to the virtual disk in storage. Virtual disks in storage are useful for
items that don't support SFS. (such as VSAM-formatted minidisks or VSE
By NUCXLOADing, you only read the program once and keep it in private storage
(from where it can be paged out to expanded storage or to DASD). However,
the program isn't shared at all. The NUCXLOAD technique is suitable if
the program is exclusive to one or a few users. For example, the action
routines of a PROP should be NUCXLOADed (or EXECLOADed for execs) before
starting the program.
Yes, only saved segments allow effective sharing of programs among users
and ,as an extra benefit, only those parts that get executed will be paged
The running program itself is not shared, so each user of the program has
a separate copy in private storage.
The whole program must be read, including exception routines (it's likely
that these will be selected for paged out later due to lack of reference).
CP will not keep the program in MDC if it is not started frequently. CP
tends to cache what is read frequently, and a program is only read when
Data files can, of course, get the same benefits from MDC and virtual disks
in storage as is the case for programs. Using MDC requires minimal effort
from the systems programmer.
The best choice is to use saved segments whenever possible. However, they
require more planning and maintenance from the systems programmer, while
MDC and SFS-DS are more self-regulating processes.
MDC, SFS-DS, and virtual disks in storage can speed up reading the program
from disk but only if the file is read frequently.
However, data can be stored in saved segments too, giving the great
advantage of sharing. The former 16 Mbyte limit explains why the technique
was not used frequently in the past.
Normally, programs that want to read data from shared segments instead
of disk have to be specifically designed to do so. However, with CMS Pipelines,
reading an EXECLOADed file or reading from disk becomes transparent| Yes,
loading a data file in storage with EXECLOAD is fooling CMS, but is works
and is supported by CMS Pipelines. Try this, for example:
PIPE LITERAL Card 2 ? LITERAL Card 1 ? > TEST FILE A
EXECLOAD TEST FILE A MYTEST DATA
PIPE < MYTEST DATA ? CONSOLE
Because placing data in saved segments is so easy, it is worth considering
for highly used data that is not frequently modified.
The very best would be the direct use of VM data spaces but that requires
the program to be adapted to use the data spaces.
You can, however, indirectly benefit from VM data spaces via the SFS-DS
technique, in which case your data processing program doesn't require changes.
But realize that, compared to direct use of VM data spaces, you then share
in "move mode". That is, when your program does a read to get data,
CMS has to move it from the data space to the program's buffer. Note
that with MDC, one also shares storage in "move mode".
Note first that in this discussion, Xedit macros and Pipeline stages are
also "execs" -- only the filetype differs. So do compiled REXX procedures,
unless they are compiled into a TEXT object and link-edited into a module.
For sharing and performance aspects, execs compare very well to what
we explained for programs. So the list of possibilities is very similar:
Attentive readers could remark that non compiled execs are data to computers,
so it must be possible to interpret them directly from a VM data space.
Theoretically they're right, but we've seen that accessing data directly
in a data space is not transparent and, for the moment, the CMS REXX interpreter
isn't adapted to it. A REXX procedure can, of course, be stored in an SFS-DS
but before execution it will be copied into your address space. For frequently
started procedures, chances are then great that no page-in is required.
leave them on a minidisk (and hope for MDC benefits)
store them in an SFS-DS
copy them to a virtual disk in storage shared by all users
place them in an LSEG or in the CMSINST segment
Conclusion: Place your highly used REXX execs in saved segments.
Starting with VM/ESA 1.1.0, REXX execs can be placed above the 16M line,
relieving the former space constraint. EXEC2 execs can also be put into
saved segments, but only below 16M. Although it's clear that compiled execs
run much faster, they result in about 4 times larger files. This means
that sharing the coding and avoiding the I/O to load them is even more
important once execs are compiled.
FSTs can consume a lot of storage. An FST entry for a minidisk file needs
64 bytes (an entry for an SFS file is a little bigger). Thus, when you
ACCESS a minidisk with 5000 files, the cost in your address space is over
300K (75 pages), which all have to be read from disk too. And, even though
CMS is clever enough to use hashing techniques to drastically minimize
the number of pages to be scanned (2 pages/filemode), searching for files
is a job CMS has to perform very frequently.
How can we gain performance here?
So, also for FSTs, saved segments are recommended. Beware however:
if you often update the disk and resave the segment, you will end with
many copies of the saved segment, thereby reducing the storage effectively
shared. In an extreme case, each virtual machine could have its own
copy of the saved segment, and nothing is shared anymore. How many
copies of a segment are acceptable then ? It depends: for the CMS
case, a little calculation seems to indicate that on a system where daily
some 300 users log on and back off, it is still worthwhile to resave CMS
when there are already 10 copies of the CMS segment.
Although MDC may speed up obtaining the FSTs during ACCESS, it will
not help the process of scanning the FSTs. CMS keeps the FSTs in virtual
storage so, to find a file, no I/O is required. Since MDC works by eliminating
I/Os, it will not help here.
Yes, FSTs fit well into saved segments. The pages will effectively be
shared (no moves required), but they still must remain below 16M. SAVEFD
can be used to place FSTs in a "normal" physical segment, while SEGGEN
is the command to place them in an LSEG. The drawback is that each time
something changes on the minidisk, the segment must be resaved (use ACCESS
(SAVEONLY to verify if a segment is still valid). Remember also that the
FSTs for the S- and Y-disk are saved together with the CMS saved system,
so the 19E is a good candidate to receive frequently used files.
For an SFS-DS, the FSTs do reside in the shared data space and they
don't have to be moved to your address space. The extra advantage of an
SFS-DS over a CMS minidisk is that not only are the FSTs shared, but also
the files themselves. Furthermore, the FSTs do not consume precious address
space below 16M and they don't have to be manually resaved after files
have been updated.
You can use the CP Q NSS USERS segname command to find out
how many copies exist and who's using which copy. By restarting users
of obsolete class P segments storage sharing is improved (the CPQUERY EXEC
-available on the VM download library- can help you with this task).
To complete this discussion, a short remark about the "highest" address
one should try to respect with saved segments and virtual machine sizes.
In order to describe your address space, CP has to build a so-called
"segment table". Initially, your segment table has just enough entries
to describe your virtual machine size. When you activate a saved segment
that was generated at a higher address, CP needs to enlarge your segment
table. When you later detach the segment, CP will not downsize your segment
table because it figures that you may use the same segment again later
on. Note that since CMS Release 6, saved segments can be loaded inside
the virtual machine size (but adding SEGMENT RESERVE commands in the PROFILE
or SYSPROF EXEC may be required). And, obviously, the storage occupied
by the saved segments cannot be used as private R/W storage.
Knowing this, you should remember that there are three important limits:
So, if possible, keep virtual machine sizes, and the most commonly used
saved segments below 32M. If that isn't possible, what then ?
A segment table describing 32M fits into the base VMDBK (virtual machine
description block) and so has no additional storage cost relative to smaller
virtual address sizes.
From 33M to 1024M, the segment table needs one extra 4K page/user.
Above 1024M, CP needs yet one more page, and it must be contiguous with
An example may better illustrate the difference. Suppose a user with
a DEF STOR or 32M, loads a 1M segment located at 511M: CP fetches a free
page, moves the user's segment table inside it and updates the user's control
register 7 to reflect the new segment table location and size. It
should be clear that half of the segment table page is not used.
Up to VM/ESA 2.2.0: you can place them very high, but not above 1024M:
the extra real storage cost for a segment at 33M or at 1024M is exactly
the same: 1 page per user.
In VM/ESA 2.3.0 CP became a bit more clever: the un-used upper part of
segment table pages is reclaimed by CP and used as system "free" storage.
Before VM/ESA 2.3.0 that half page is indeed wasted; from 2.3.0 on,
CP can use it for free storage.
So, from 2.3.0 on for segments that must be placed above 32M, you gain
some space by placing them as low as possible. Don't be overly conservative:
if 150 users use a segment at 200M instead of at 64M, the extra storage
cost is only 32*(200-64)*150 bytes or 640K.
Data in memory techniques can greatly enhance the performance of your system.
These techniques eliminate I/Os and, when sharing among users is possible,
they reduce real storage consumption.
Even with MDC and VM data spaces, saved segments are invaluable to share
programs, execs and minidisk FSTs.
VM data spaces
An SFS directory in a VM data space performs as well as minidisks with
MDC and shared FSTs. In addition, you get better disk management and you
can share more data. Note, however, that SFS file control directories
(with full support of aliases, sharing, etc.) can't be placed in a VM data
space -- only directory control directories can.
MDC is a good performance booster for minidisks. Note that the CMS minidisks
used by SQL/DS (now known as "DB2 for VM and VSE") and SFS servers are
also eligible for MDC, resulting in an effect of "bigger buffers" for SFS
Note though that since VM/ESA 1.2.2, the MDC uses by default full-track
reads, what is good for sequential access (i.e. good for most CMS files),
but not for random access, such as SQL databases. For SQL/DS, it
is best to use the "SQL Dataspace Feature", or, if using VM/ESA 2.3.0 or
2.2.0+PTF UM28392, use the new "Record MDC". Probably, the minidisks
used by the SFS catalog (storage pool 1) perform also better with "Record
virtual disks in storage
are primarily meant to be shared by VSE guests, used by old CMS
applications, or used as a replacement for TDISKs. By old CMS applications,
we mean applications that cannot profit from such things as large virtual
storage, VM data spaces, or files in an SFS data space. Note, however,
that virtual disks in storage are not for free. With the current design,
CP considers the pages in use for a virtual disk in storage as shared
storage, making them less eligible to be paged out. Hence, a single user
with a big and very active virtual disk in storage, can take over a big
part of central storage.
(or EXECLOAD for procedures) is easy to implement and is especially useful
when sharing is not important.
Creating Logical Segments
We have been telling that saved segments are still the best option for
programs, execs and FSTs. Creating Logical segments is easy, but
we guess that many readers are not familiar with it. Therefore it
may be appropriate to mention how LSEGs can be created. For more
details, refer to the VM/ESA Planning and Administration manual, or, have
a look in the VM/ESA Performance manual.
Find out what objects you want to place in an LSEG: FSTs, MODULEs, EXECs,
XEDIT macros, ... Except for FSTs, objects of different types
can be placed in an LSEG. A PSEG hosts one or more LSEGs. So,
you also ha ve to decide how many LSEGs and PSEGs you'll
make. Here we'll suppose you will place your tools (REXX execs, XEDIT
macro's and some MODULEs) in one LSEG
Find a place in real storage to place the segment (remember that LSEGs
with FSTs or EXEC2 execs must be located below 16M). To map the storage
used by segments various tools exist:
VMFSGMAP the official VM solution: issue EXEC VMFSGMAP SEGBLD ESASEGS SEGBLIST
(as we can't remember that command, we created a SEGMAP
EXEC that simply issues the above command)
CPQUERY from the donwload library.
The "quick&dirty" solution: QNSSMAP EXEC, which is appended below.
Define the segment skeleton. Suppose you found room at 25Meg and
1 Meg is enough: issue
CP DEFSEG mypseg 1900-19FF SR
Define your storage at least 1 Meg higher than the address of the segment.
Use XEDIT to create the "mylseg LSEG" file. Insert lines to describe
EXEC STARTXED EXEC * (INSTSEG
EXEC SUBMIT EXEC *
EXEC OURPROFL XEDIT * (INSTSEG
EXEC OURFILEL EXEC * FILELIST EXEC
MODULE MYBROWSE MODULE *
EXEC SOMEPIPE REXX * (INSTSEG
You should remark the option INSTSEG. It means that this exec is
considered to be part of the "CMS Installation Segment" (CMSINST by default).
It influences when the exec will be found in the search order:
So, you must carefully think about the INSTSEG option. EXECMAP can
be used to see how often execs are executed.
Use XEDIT to create the "mypseg PSEG" file. In our case only 1 line
Without INSTSEG, the exec is considered EXECLOADed, and it will be used
even the user has a copy on his A-disk for example.
With INSTSEG, when the exec will be found depends on the setting of INSTSEG.
By default, INSTSEG is "ON S", which means that this exec will be found
just before the search of the S-disk starts. So, if for example a
STARTXED EXEC is found on the R-disk, the copy in the segment will not
be executed, the disk resident version is take instead.
LSEG mylseg LSEG
Access the minidisks containing your objects, and access CMS resident (MAINT
190) in R/W as Z (this way SEGGEN can update the SYSTEM SEGID file immediately)
Save the PSEG and LSEG(s) by issuing:
SEGGEN mypseg PSEG A SYSTEM SEGID Z
Now that the LSEG is created, you still must make your users use it.
For FSTs, the ACCESS command will try to use the segment autmatically.
For other objects, a "SEGMENT LOAD mylseg (SYSTEM" must be executed.
So, you have to include a SEGMENT LOAD in the SYSPROF EXEC, or in another
exec that your users surely execute before they use the code you so carefully
placed in the LSEG.
The QNSSMAP EXEC
Here is the quick&dirty but fast QNSSMAP EXEC.
/* This exec creates a simple NSS MAP
| format: | QNNSMAP <ALL> |
- without option ALL: segment spaces, CMS and GCS segments are not listed
- with option ALL: everything is listed
Written by: ; Kris Buelens, IBM Belgium 10 Aug 1992*/
/* 13 Feb 1997: Don't XEDIT file if disconnected user */
parse upper source . . myname mytype . syn .
parse upper arg all .
if all='ALL' then /* All stuff wanted, also CMS NSS-es */
'PIPE (end ?) CP Q NSS MAP', /* .. so must fill in cols 1-32 and */
'|D: DROP 1', /* .. 52-61 of all records */
'| SPEC 1-32 1 52.10 33|NFIND _'||,
'| SPEC 1-32 1 43-* 33 33.10 52 |NFIND _'||,
'| XLATE 33-37 A-F FA-FF',
'| XLATE 20-20 S L', /* SORT S(pace) Before M(ember)*/
'| SORT 33-37 20',
'| XLATE 33-37 FA-FF A-F',
'| XLATE 20-20 L S',
'|T: FANIN 1 0',
'| > NSS MAP A',
'?F:| SPEC 33-* |J:?D:|T:'
'PIPE (end ?) CP Q NSS MAP',
'|D: DROP 1',
'|NLOCATE 15.3 /NSS/',
'|NLOCATE 15.6 /DCSS-S/',
'|NLOCATE 15.6 /CPDCSS/',
'|XLATE 33-37 A-F FA-FF',
'|XLATE 33-37 FA-FF A-F',
'|T: FANIN 1 0',
'|> NSS MAP A',
if rc=0 then do
if linesize()>0 then 'EXEC REXEDIT NSS MAP A NORC'
else say 'NSS map stored in file NSS MAP A'