MULTIPLE CONNECTIONS CAUSE THE SSL SERVER TO ABEND
APAR Identifier ...... PK51954 Last Changed ........ 09/08/24
MULTIPLE CONNECTIONS CAUSE THE SSL SERVER TO ABEND
Symptom ...... AB ABEND Status ........... CLOSED PER
Severity ................... 4 Date Closed ......... 09/04/24
Component .......... 5735FAL00 Duplicate of ........
Reported Release ......... 520 Fixed Release ............ 999
Component Name TCP/IP V2 FOR V Special Notice
Current Target Date .. Flags
SCP ...................
Platform ............
Status Detail: SHIPMENT - Packaged solution is available for
shipment.
PE PTF List:
PTF List:
Release 520 : UK46022 available 09/04/29 (1000 )
Release 530 : UK46023 available 09/04/29 (0902 )
Release 540 : UK46024 available 09/04/29 (0902 )
Parent APAR:
Child APAR list:
ERROR DESCRIPTION:
When multiple secure connections are open and active, the
SSL server may exhibit degraded performance and response time
when handling connections, administrative requests, or when
processing Linux commands issued at the server console.
Messages indicative of this problem are:
hh:mm:ss __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process vmssl
Eventually, the server may abend, or become unresponsive and
then be restarted by the z/VM TCP/IP stack server, at which
point the message that follows will be issued on the SSL server
console:
Restarting you because KillClient called for reason: Fatal
inter-VM communication error
LOCAL FIX:
None. However, increasing the virtual storage size of the
of the SSL server virtual machine may provide some relief
for instances where connection activity is low or
intermittent.
PROBLEM SUMMARY:
****************************************************************
* USERS AFFECTED: All customers running the z/VM SSL server. *
****************************************************************
* PROBLEM DESCRIPTION: *
****************************************************************
* RECOMMENDATION: APPLY PTF *
****************************************************************
The reported problem was caused by incorrect logic in the vmssl
daemon cleanCertblock() routine, which erroneously allocated a
new certificate data structure (and associated storage) as part
of its processing, and which incorrectly processed the chain of
pertinent elements for the removal of unused elements (and the
freeing of their associated storage).
Logic within the cleanCertblock() routine has been updated to
rectify these errors.
With this APAR, a number of Reliability, Availability and
Serviceability (RAS) enhancements have been implemented within
the level 520 SSL daemon to provide improved diagnostic
capabilities and certain aspects of internal server operations.
Most of these changes already have been incorporated, as
applicable, within the 530 level SSL server (via APARS PK52298
and PK53928) and the 540 level SSL server (via APAR PK65850).
However, to achieve greater consistency across all levels, minor
RAS updates have been implemented via this APAR for the 530 and
540 level SSL servers as well. For the most part, these changes
are internal to the server, with some changed messages.
With this APAR, the following VMSSL command operands are
introduced for (only) level 520:
* LOGMODE
- allows one to direct the SSL server to log message and
trace information to either the log file or to the server
console
* GSKTRACE Trace operand
- activates GSKit tracing of SSL connection processing
* DEBUG Trace operand
- activates detailed diagnostic tracing
This APAR also provides changes to the level 520 administrative
interface (SSLADMIN) to resolve problems with use of an
(undocumented) '=' command diagnostic operator.
An attempt to use this operator to forward unparsed command
input to the SSL server currently fails, as shown in this
example:
ssladmin = TRACE DEBUG
Sending unparsed input: TRACE DEBUG
DTCSSL2423E An unexpected error occurred while processing
DTCSSL2423E the SSLADMIN command.
Command Complete. rc: -1
Ready;
Updates to the simpleSend() routine resolve this problem
and allow the '=' operator to be used as intended. For
reference, the diagnostic operators available for use with
the SSLADMIN command are:
* Causes operational diagnostic messages to be produced for
debugging purposes.
+ Causes all command results to be placed in the file:
SSLADMIN $RESULT$.
= Causes the supplied subcommand and operands to be
forwarded to the SSL server as-is, with results displayed
at the console. No validation or adjustment of operands
is attempted.
Note: The previously listed diagnostic operators are intended
for use in diagnosing SSL server operational problems,
in consultation with the IBM support center.
PROBLEM CONCLUSION:
The revised information that follows will be included in any
future updates to the following publication(s):
================================================================
SC24-6125-02 -- z/VM: TCP/IP Level 520 Planning and
Customization
Chapter 22. "Configuring the SSL Server"
Section: "VMSSL Command" ; Page(s): 560-562
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Descriptions for the VMSSL command operands that follow are
added.
---------------------------------------------------------------
The VMSSL command LOGMODE operand is introduced:
.--LOGMOde--FILE-----------.
>>--VMSSL--...--+--------------------------+--....----------><
'--LOGMOde--+--FILE-----+--'
'--CONSOLE--'
LOGMOde location
specifies the location to which server message and trace
information should be directed.
FILE
Specifies that messages and server trace information should
be directed to and maintained using a file. FILE is the
default.
CONSole
specifies that messages and server trace information should
be directed to the server console.
Usage Notes:
1. CONSOLE is the preferred LOGMODE setting for normal server
operations, and especially when server tracing is performed
during problem diagnosis. This mode allows for simplified
management and retention of all server messages. Console
information can readily be captured by using the CP SEND
command; for example:
CP SEND CP SSLSERV SPOOL CONSOLE CLOSE
---------------------------------------------------------------
The VMSSL command TRACE GSKTRACE and DEBUG operands are
introduced:
.-NORMal--------------. .-ALL-----------------.
>-.-TRACE-+---------------------+-+-------------------+-.-.-><
| | .-...-. | |-ip_address--------| | |
| |-CONNections-+-----+-| |-:--port-----------| | |
| | +-...-' | |-ip_address-:-port-| | |
| |-FLOW----------------' '-conn_number-------' | |
| |-GSKtrace------------------------------------| |
| '-DEBug---------------------------------------' |
'-NOTRACE-----------------------------------------------'
GSKtrace
specifies that GSKit tracing of SSL connection processing is
to be initiated.
Note: When GSKit tracing is active, data for all SSL server
connections is acquired and stored in a file that is
separate from other SSL server trace data. The
GSKTRACE operand is intended for use in consultation
with the IBM Support Center.
DEBug
Specifies that detailed diagnostic tracing is to be
initiated. The DEBUG operand is intended for use in
consultation with the IBM Support Center.
================================================================
Documentation for these new messages will be included in any
future updates to the following publication:
GC24-6124-01 -- z/VM: TCP/IP Level 520 Messages and Codes
Chapter 19. SSL Messages ; Page(s): 305-313
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
New Message and Text:
---------------------
DTCSSL002I <text>
Explanation: This message provides informative text about the
state of select configuration operands or other
characteristics that pertain to overall server operations.
System Action: SSL processing continues.
System Programmer Response: None.
Changed Message and Text:
-------------------------
DTCSSL005E Pthread_create() error creating mainSSL Thread;
errno: <errno>
DTCSSL005E Pthread_create() error; errno: <errno>; At most
<nnn> sessions can be accommodated
Explanation: A Pthread_create() call failed. For the meaning
of <errno> values, see Table 4, "SSL Server Reason Codes".
Note that either format of this message might result because
the SSL server has been defined with insufficient virtual
storage to allow for its operation. For the second message
format, the defined virtual storage is insufficient to
accommodate the number of connections specified by the VMSSL
MAXUSERS operand, or the value specified exceeds the
implementation maximum ( the 520 SSL server implementation
can accommodate a maximum of 128 concurrent connections).
System Action: SSL processing continues.
System Programmer Response: Set the MAXUSERS value to less
than <nnn> or increase the virtual storage size of the SSL
server virtual machine, and restart the SSL server. If the
error persists, contact the IBM Support Center for
assistance.
Changed Message and Text:
-------------------------
DTCSSL073E Broken pipe condition encountered
The message text has been augmented for clarity outside of
a debugging context.
(The 'Explanation', 'System Action' and 'System Programmer
Response' for this message remain unchanged.)
New Message and Text:
---------------------
DTCSSL095I errno detail: <text>
Explanation: This message provides textual information defined
by the system for an error number (errno) that has been
reported as part of a previously encountered error.
System Action: SSL processing continues.
System Programmer Response: None.
New Message and Text:
---------------------
DTCSSL097I Commencing RESET of server to again accept
connections
Explanation: This message acknowledges receipt of an SSLADMIN
RESET command that instructs the server to resume acceptance
and processing of new connection requests.
System Action: SSL processing continues.
System Programmer Response: None.
New Message and Text:
---------------------
DTCSSL107E Internal Error: ({S|U}) Send incomplete rc: <rc>
len: <length_to_send>
Explanation: An error occurred while attempting to transmit
data for a secured (S) or a clear/unsecured (U) connection.
The total length of the data that was to be transmitted is
indicated in the message. For the meaning of <rc> values
see Table 3, "SSL Server Return Codes".
System Action: The connection for which this error
occurred is stopped; other connections continue to be
processed.
System Programmer Response: Contact the IBM Support Center for
assistance.
New Message and Text:
---------------------
DTCSSL115W LOG command declined; LOGMODE CONSOLE is in
effect
Explanation: An SSLADMIN LOG command has been received, but
the given command cannot be processed because all message
output is currently being directed to the server console.
For an SSLADMIN LOG command to be processed, file logging
(LOGMODE FILE) must be in effect.
System Action: SSL processing continues.
System Programmer Response: To obtain the current server
console log information while LOGMODE CONSOLE is in effect,
use the CP SEND command, issued from a user ID that has
appropriate CP command privilege to use this command. For
example:
CP SEND CP SSLSERV SPOOL CONSOLE CLOSE
New Message and Text:
---------------------
DTCSSL120W Acceptance of new connections has been suspended
Explanation: The server has identified a problem with the
socket connection used to handle new secure connection
requests. Because this problem exists, new secure
connections cannot be established.
System Action: SSL processing continues. Existing secure
connections continue to be processed.
System Programmer Response: Examine the TCP/IP and SSL server
log information for messages indicative of a problem
associated with the sockets maintained between these two
servers. If the problem appears to be transient, instruct
the SSL server to resume the acceptance of connections via
this command: SSLADMIN = RESET
For a non-transient problem, ensure that any existing secure
connections have been closed, then shutdown and restart the
SSL server.
New Message and Text:
---------------------
DTCSSL127E ckEPIPE() invoking shutdownNow() -- <reason>
Explanation: An error has been encountered that justifies a
shutdown of the server. As indicated in the message, The
ckEPIPE() function has either detected an EPIPE error or has
been directed by a calling function to initiate a shutdown.
System Action: SSL processing stops.
System Programmer Response: Examine the TCP/IP and SSL server
log information for messages indicative of a socket error
that corresponds to the handling of a secure connection or
other server operation. Review accompanying messages for
more information. If the reason for this error is not
apparent and the error persists contact the IBM Support
Center for assistance.
Changed Message and Text:
-------------------------
DTCSSL202E Internal error <location>; VMSSLGSK error <code>
Explanation: An error occurred at the source code location
cited in the message while using functions that handle
specific aspects of the SSL protocol. For the meaning of
the error code, see Table 3, "SSL Server Return Codes".
(The 'System Action' and 'System Programmer Response' for this
message remain unchanged.)
Changed Message and Text:
-------------------------
DTCSSL206E SSLADMIN peer closed connection
'SSLADMIN' has been added to the message text for clarity
outside of a debugging context.
(The 'Explanation', 'System Action' and 'System Programmer
Response' for this message remain unchanged.)
New Message and Text:
---------------------
DTCSSL424E The label already exists in the request or
certificate database.
Explanation: An SSLADMIN REQUEST command was issued for which
the specified label is already in use. The given label is
associated with a certificate or certificate request that
already has been stored in the SSL server certificate
database.
System Action: SSL processing continues.
System Programmer Response: If appropriate, delete the
existing certificate request from the SSL server certificate
database and then reissue the subject SSLADMIN REQUEST
command. Alternately, rename the X509INFO file with which
the subject label is associated, then reissue the SSLADMIN
REQUEST command, with the new file name specified as the
label operand.
New Message and Text:
---------------------
DTCSSL425E TCP/IP service is not available; confirm that the
TCP/IP server (<userid>) is running
Explanation: The TCP/IP stack server, with which the SSL
server is associated, is either not logged on or is not
running.
System Action: SSL processing stops.
System Programmer Response: Start the indicated virtual
machine or ensure it is in a working state (for example, not
in a CP READ).
Changed Message and Text:
-------------------------
DTCSSL501E No SERVER certificate was found with this label
<label>
Explanation: A server certificate with the label cited in the
message cannot be located within the SSL key database. The
label in question has been specified to secure a specific
port or port range.
System Action: The attempted connection is terminated.
System Programmer Response: Review the TCP/IP server
configuration file and verify that all server certificate
labels that have been specified as part of the PORT
statement are correct. If necessary, use the SSLADMIN QUERY
CERT * command to review the labels associated with the
server certificates that have been stored in the SSL key
database. Correct any problems, then retry the connection.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The revised information that follows will be included in any
future updates to the following publication(s):
================================================================
Documentation for these new messages will be included in any
future updates to the following publications:
GC24-6124-02 -- z/VM: TCP/IP Level 530 Messages and Codes
Chapter 19. SSL Messages
Page(s): 375-385 (level 530)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
New Message and Text:
---------------------
DTCSSL002I <text>
Explanation: This message provides informative text about the
state of select configuration operands or other
characteristics that pertain to overall server operations.
System Action: SSL processing continues.
System Programmer Response: None.
Changed Message and Text:
--------------------------
DTCSSL005E Pthread_create() error creating <infoThread |
mainSSL Thread | cloneThread (S|U)> errno: <errno>
DTCSSL005E Pthread_create() error; errno: <errno>; At most
<nnn> sessions can be accommodated
Explanation: A Pthread_create() call failed. For the meaning
of <errno> values, see Table 4, "SSL Server Reason Codes".
Note that either format of this message might result because
the SSL server has been defined with insufficient virtual
storage to allow for its operation. For the second message
format, the defined virtual storage is insufficient to
accommodate the number of connections specified by the VMSSL
MAXUSERS operand.
System Action: SSL processing continues.
System Programmer Response: Set the MAXUSERS value to less
than <nnn> or increase the virtual storage size of the SSL
server virtual machine, and restart the SSL server. If the
error persists, contact the IBM Support Center for
assistance.
New Message and Text:
---------------------
DTCSSL127E ckEPIPE() invoking shutdownNow() -- <reason>
Explanation: An error has been encountered that justifies a
shutdown of the server. As indicated in the message, The
ckEPIPE() function has either detected an EPIPE error or has
been directed by a calling function to initiate a shutdown.
System Action: SSL processing stops.
System Programmer Response: Examine the TCP/IP and SSL server
log information for messages indicative of a socket error
that corresponds to the handling of a secure connection or
other server operation. Review accompanying messages for
more information. If the reason for this error is not
apparent and the error persists contact the IBM Support
Center for assistance.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The revised information that follows will be included in any
future updates to the following publication(s):
================================================================
Documentation for these new messages will be included in any
future updates to the following publications:
GC24-6124-04 -- z/VM: TCP/IP Level 540 Messages and Codes
Chapter 18. SSL Messages
Page(s): 377-383 (level 540)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
New Message and Text:
---------------------
DTCSSL002I <text>
Explanation: This message provides informative text about the
state of select configuration operands or other
characteristics that pertain to overall server operations.
System Action: SSL processing continues.
System Programmer Response: None.
Changed Message and Text:
--------------------------
DTCSSL005E Pthread_create() error creating <infoThread |
mainSSL Thread | cloneThread (S|U)> errno: <errno>
DTCSSL005E Pthread_create() error; errno: <errno>; At most
<nnn> sessions can be accommodated
Explanation: A Pthread_create() call failed. For the meaning
of <errno> values, see Table 4, "SSL Server Reason Codes".
Note that either format of this message might result because
the SSL server has been defined with insufficient virtual
storage to allow for its operation. For the second message
format, the defined virtual storage is insufficient to
accommodate the number of connections specified by the VMSSL
MAXSESSIONS operand.
System Action: SSL processing continues.
System Programmer Response: Set the MAXSESSIONS value to less
than <nnn> or increase the virtual storage size of the SSL
server virtual machine, and restart the SSL server. If the
error persists, contact the IBM Support Center for
assistance.
New Message and Text:
---------------------
DTCSSL121E ckEPIPE() invoking shutdownNow() -- <reason>
Explanation: An error has been encountered that justifies a
shutdown of the server. As indicated in the message, The
ckEPIPE() function has either detected an EPIPE error or has
been directed by a calling function to initiate a shutdown.
System Action: SSL processing stops.
System Programmer Response: Examine the TCP/IP and SSL server
log information for messages indicative of a socket error
that corresponds to the handling of a secure connection or
other server operation. Review accompanying messages for
more information. If the reason for this error is not
apparent and the error persists contact the IBM Support
Center for assistance.
TEMPORARY FIX:
COMMENTS:
MODULES/MACROS: SSLADMIN SSLSERV SSLVMADM SSLVMAIN SSLVMCOM
SSLVMDB SSLVMGSK VMSR3 VMSR3S VMSR3X VMSR3XS VMSR4
VMSR4S VMSR4X VMSR4XS VMSS8 VMSS8S VMSS9 VMSS9S
VMSS9X VMSS9XS
SRLS: SC24612502 GC24612401 GC24612404 GC24612402
RTN CODES:
CIRCUMVENTION:
MESSAGE TO SUBMITTER: