VM Resource Manager (VMRM) Cooperative Memory Management

Updated: 15 July 2008

VMRM Cooperative Memory Management (VMRM-CMM) between a z/VM system and Linux guests assists in managing memory constraint in the system. Based on several variables obtained from the system and storage domain CP monitor data, the Virtual Machine Resource Manager (VMRM) detects when there is such constraint, and notifies specific Linux virtual guests when this occurs. The guests can then take the appropriate action to adjust their memory utilization in order to relieve this constraint on the system

A user-supplied VMRM configuration file is used to identify which guests should be notified. These guests must be prepared to handle the data being sent in the notification and take appropriate actions, such as issuing a CP Diagnose instruction to release pages of storage.

In addition to the workload management functions for CPU and DASD I/O provided by VMRM, the following enhancements have been made to the Virtual Machine Resource Manager for Linux guests:

• A new NOTIFY statement with a MEMORY keyword is supported in the Virtual Machine Resource Manager configuration file. Following the keyword is a user ID or a list of user IDs to be notified when virtual memory becomes constrained. (For the format of this statement, see "External overview: configuration file NOTIFY statement.")
• System and storage domains are monitored for data to be used for calculating memory constraint, as well as how much memory to request the guest machine to release
• The Virtual Machine Resource Manager issues a CP SMSG to notify the specified guests when memory is constrained and the amount required to release in order to relieve the constraint. (For the format of the SMSG buffer, see "Format of the CP SMSG command buffer.")
• A message is logged in the VMRM logfile, indicating which users were sent an SMSG, and the text of the SMSG buffer. Also, if MSGUSER is specified on the VMRM ADMIN statement, the same message written to the log is written to the MSGUSER user ID's console as well.

The NOTIFY statement with a MEMORY keyword is used to list the Linux guest image names to be notified when memory constraint is detected by the resource manager.

The new statement has the following format:

Where:

Note: The NOTIFY statement can be the only statement in a configuration file, or it may be specified in addition to other VMRM configuration statements, such as WORKLOAD, GOAL, MANAGE, and ADMIN statements.

For a detailed description of these statements, see Chapter 17 of the z/VM: Performance publication.

To start the VMRM server, specify the exec name, IRMSERV, with a configuration file name. For example: IRMSERV MYCONFIG FILE A. To stop the server, log onto the VMRMSVM user ID and issue the HMONITOR command. This allows VMRM termination and end-of-command procedures to occur.

In addition to the monitor records being used for CPU and DASD velocity goals, VMRM enables and monitors some system domain records. There are two types of problems to solve using this data: 1) determining when there is memory constraint, and 2) when memory is constrained, calculating how much memory to notify the guest to release.

Once system memory constraint has been detected, VMRM calculates how much memory each Linux guest should release to relieve the constraint. Using the SHRINK keyword on the SMSG command, a message indicating the amount of storage to release is sent to each logged on Linux guest in the notify list.

When system memory is no longer constrained, another SHRINK message with a smaller absolute value is issued. A smaller SHRINK request than the previous one effectively instructs the guest to reclaim some of the storage previously released.

The CP SMSG buffer being sent to the guest has the following general format:

Where:

The use of VMRM Cooperative Memory Management can significantly improve overall system performance in cases where the overall z/VM system is constrained for real storage and much of that storage is being held by one or more Linux guests. However, use of VMRM-CMM can sometimes reduce the performance of one or more of the participating Linux guests.

Accordingly, we recommend that the performance of the Linux guests be monitored before and after the use of VMRM-CMM is introduced, allowing you to determine whether any performance-critical guests are being adversely affected and, if so, to remove them from the NOTIFY list. For a Linux guest excluded from VMRM-CMM, the best way to constrain the storage usage is to reduce its virtual storage size as much as is practical.

z/VM V5.2 has the base VMRM CMM support. An additional CMS APAR is required, VM64085, for full functionality. This APAR is available as of August 31, 2006.

The relevant source code with Linux on System z support, also referred to as CMM1, for Cooperative Memory Management is available from developerworks as:

• Part of the April 2004 stream (since kernel patch 34)


• And also as part of the October 2005 stream" (from the start)

Linux on System z support for CMM1 is available in:

• Novell SUSE Linux Enterprise Server 9 (SLES9) SP3 since kernel level: kernel-s390(x)-2.6.5-7.257 dated 2006-05-16
• Novell SUSE Linux Enterprise Server 10 (SLES10) since GA
• Red Hat RHEL4 U7 2.6.9-73 (includes Out of Memory Notifier)
• Red Hat RHEL5.1 2.6.18-53 (includes Out of Memory Notifier)

IBM is working with its Linux distribution partners that this function will be provided in future Linux on System z distribution releases or service updates.

For relevant documentation on Linux on System z refer to the chapter "Cooperative memory management" in the latest Device Drivers, Features, and Commands manual

• April 2004 stream" documentation page
  (SC33-8281) when you use "April 2004 stream" code or SLES9
• October 2005 stream" documentation page
  (SC33-8289) when you use "October 2005 stream" code or SLES10

Recommended Patches
Out of Memory Notifier (for Levels prior to SLES10 SP1)