Last Updated: 6 January 2022

Dynamic Memory Downgrade information



Background

Dynamic memory downgrade (DMD), also known as memory reclamation or storage reclamation, extends the real storage dynamic management characteristics of z/VM to include removing real storage from a running z/VM system. z/VM memory reconfiguration gives a system administrator the ability to add or remove memory dynamically from a z/VM partition, making it available to other partitions in the central processor complex (CPC), without the need to re-IPL the system. This flexibility allows installations to react to changing workloads.


Required Software Levels

  • PTF UM35561 for APAR VM66271 enables DMD for z/VM 7.2
  • PTF UM35783 for APAR VM66320 recommended to avoid a possible downgrade stall scenario


Required Hardware Levels

Memory reclamation by a first-level z/VM system can be performed on the IBM z14 and (or later) family of servers (or equivalent LinuxONE models). The following hardware service levels (or later) are necessary to prevent possible memory reclamation stalls for customers with guests that have dedicated FCP devices in the virtual I/O configuration:

  • IBM z14 or LinuxONE Rockhopper II or LinuxONE Emperor II - Bundle 65
  • IBM z15 or LinuxONE III - Bundle 46

The STSIUSE EXEC can be used to verify that the minimum required hardware service levels are installed to support dynamic memory downgrade. The output of STSIUSE includes sections for various functions of the STSI hardware instruction. In the section for STSI 1.1.1 are included lines for "Firmware Identifier" and "Type". Type identifies the model of the server, 8561 or 8562 for IBM z15 or LinuxONE III and 3906 or 3907 for IBM z14 or LinuxONE II. The first 4 hex digits of the "Firmware Identifier" must be greater than or equal to 304E for IBM z15 or LinuxONE III and 6019 for IBM z14 or LinuxONE II to ensure that the appropriate hardware service level has been applied.

If these hardware service levels are not installed, then a DMD might stall due to some virtual/guest memory related to dedicated FCP devices being pinned/locked in reconfigurable memory. Techniques for responding to a stalled DMD are provided below in the "What To Do If a DMD Does Not Complete" section.

Note that the recommended minimum hardware service levels/bundles listed above are specific to DMD functionality. Customers should review all available service to determine the appropriate level to apply in their environment.


Using Dynamic Memory Downgrade (DMD)

A z/VM partition's real memory/storage can be classified as either permanent or reconfigurable, with all permanent being the default unless otherwise specified. Reconfigurable memory is only permitted when the z/VM partition has at least 4GB of permanent memory, and is limited to no more than 50% of total real memory in the partition. Some amount of real memory can be classified as reconfigurable either by the STORAGE statement in the system configuration file, or when memory is dynamically added to the z/VM partition with the CP SET STORAGE command. Note that the STORAGE statement in the system configuration file also controls how memory is classified as permanent and reconfigurable following different types of system restarts. Only the portion of memory that has been classified as reconfigurable can be removed by a DMD. Therefore, planning is recommended to determine an appropriate amount of memory to define as reconfigurable. Additional discussion on this topic is available in z/VM CP Planning and Administration, Chapter 21. Host Storage Planning and Administration.

A typical use case for dynamic memory management is to temporarily increase real memory in a z/VM partition to respond to an increase in demand. Later, a system administrator might want to dynamically (without a re-IPL of the z/VM system) remove that memory from the z/VM partition to make it available for workload in other partitions on the CPC. This can be accomplished by issuing the CP SET STORAGE command and specifying a smaller amount of reconfigurable memory. This will initiate an asynchronous task to perform the DMD or memory reclamation in the z/VM partition. The reclamation task involves moving guest pages out of the reconfigurable memory that is being removed. These pages might end up being moved to permanent memory, reconfigurable memory that is not being removed, or written to paging space. The system administrator can observe the progress of a DMD using successive invocations the CP QUERY STORAGE RECONFIGURATION command and inspecting the "Remainder to remove" line of the command output as well as how long the DMD has been active from the "Elapsed time" portion of the command output. Once the reclamation has completed, the memory is available on the CPC to be added dynamically to other partitions.


What To Do If a DMD Does Not Complete

The amount of time it takes for a memory downgrade to complete depends on several factors, including the amount of memory being removed and memory over-commit level. Typically, downgrades might be expected to take on the order of several minutes up to an hour or two. Progress of a DMD can be monitored by comparing the output of successive CP QUERY STORAGE RECONFIGURATION commands. If after some time it appears that no additional progress is being made, there are several options that can be taken.

  • Use STSIUSE (as described above in the Required Hardware Levels section) to verify that the correct hardware service level is installed on your IBM Z server. If not, apply the hardware service and re-IPL the z/VM partition.
  • Use DMDQPIN EXEC to identify users which have virtual/guest memory pinned/locked in reconfigurable memory. Logging off, re-IPLing, or relocating these users should free up the virtual/guest memory and allow a DMD to complete. This tool also provides a finer-grained view of reclamation progress, so successive invocations might reveal whether a DMD is truly stalled or just progressing slower than expected.
  • The CP SET STORAGE HALT command can be used to terminate storage reconfiguration. All memory that was already able to be vacated and reclaimed will remain in that state and available for use by other partitions on the CPC. The amount which was unable to be reclaimed will remain available for use in the current z/VM partition.


Helpful Commands and Tools

  • CP QUERY STORAGE RECONFIGURATION - CP command that provides information about the status of an in-progress storage reconfiguration
  • VIR2REAL - a tool from the VM Download page which provides information about current real memory configuration and calculates the ratio of total virtual to real memory
  • CHKRECLM - a tool from the VM Download page which functions similarly to VIR2REAL, but calculations are done assuming a user-specified amount of real memory has been reclaimed
  • STSIUSE - a sample exec which parses the data returned by various calls to the CMS REXX STSI function. It can be found on the MAINT 193 disk as STSIUSE SAMPEXEC. It will need to be renamed to STSIUSE EXEC to be used. STSIUSE has been available for several releases and is not new with this support.
  • DMDQPIN - a tool from the VM Download page to identify users who have virtual memory that is delaying the completion of a dynamic memory downgrade (DMD).

Frequently Asked Questions

Q: When I look at the amount of memory for "Reserved" on the logical partition profile on the HMC, it is not the same as the amount I see in QUERY STORAGE. What's wrong?
A: Nothing is wrong. "Reserved" has different meanings in the two contexts. While "Reserved" in the HMC context means how much memory could be added to the logical partition if it is available, the QUERY command breaks that down to STANDYBY and RESERVED, where STANDBY is the memory that is not assigned to any other partition on the CPC and therefore could be added, and RESERVED is how much beyond that it could go except presently there is not available memory.

Q: Is there any message or other indication that Dynamic Memory Downgrade (DMD) is not available when running first-level on a machine that is not an IBM z14 and (or later) family of servers (or equivalent LinuxONE models)?
A: In order to perform a DMD, you must first define some portion of memory as reconfigurable. Attempts to define reconfigurable memory in such an environment, either by system configuration statement or CP SET STORAGE command, will results in the following error message: HCP2588E RECONFIGURABLE storage is not supported on this system.


Additional Sources of Information and Documentation

  • SC24-6268-08, z/VM: CP Commands and Utilities Reference
  • GC24-6270-07, z/VM: CP Messages and Codes
  • SC24-6271-07, z/VM: CP Planning and Administration
  • GC24-6286-09, z/VM: General Information
  • GC24-6294-08, z/VM: Migration Guide
  • z/VM Performance Report

 

z/VM Memory Management information