Contents | Previous | Next

z/VM Virtual Switch

Discussion:  

z/VM 4.4.0 has added a special type of Guest LAN, called a virtual switch, which is capable of bridging a z/VM Guest LAN (type QDIO) to an associated real LAN connected by an OSA-Express adapter. The virtual switch is designed to help eliminate the need for virtual machines acting as routers. Virtual routers consume valuable processor cycles to process incoming and outgoing packets, requiring additional copying of the data being transported. The virtual switch helps alleviate this problem by moving the data directly between the real network adapter and the target or originating guest data buffers.

This section summarizes measurement results that assess the performance of the virtual switch comparing it with a router stack. These measurements were done for Linux as well as for the VM TCP/IP stack.

Methodology:   An internal tool was used to drive request-response (RR), connect-request-response (CRR) and streaming (S) workloads. The request-response workload consisted of the client sending 200 bytes to the server and the server responding with 1000 bytes. This interaction lasted for 200 seconds. The connect-request-response workload had the client connecting, sending 64 bytes to the server, the server responding with 8K and the client then disconnecting. This same sequence was repeated for 200 seconds. The streaming workload consisted of the client sending 20 bytes to the server and the server responding with 20MB. This sequence was repeated for 400 seconds.

A complete set of runs, consisting of 3 trials for each case, was done with the maximum transmission unit (MTU) set to 1492 and 8992.

The measurements were done on a 2064-109 with 3 dedicated processors in each LPAR used. Each LPAR had 1GB of central storage and 2GB expanded storage. CP monitor data was captured for one LPAR during the measurement and reduced using VMPRF.

Figure 1. Environment

Figure 2. Environment

Figure 1 shows the measurement environment for the traditional case where a TCP/IP stack (either VM or Linux) goes to a router (either VM or Linux) that has the OSA card attached. Communication between the stack and the router is done over a guest LAN (type QDIO). Figure 2 shows the measurement environment for the new environment where the virtual switch replaces the router. The dotted arrow shows the communication done by the controller stack at initialization. After initialization, all communication with the card is done by CP as shown by the black arrows. Both figures show more than one TCP/IP stack for illustration purposes. The measurement environment, however, had only one TCP/IP stack for both the router case and the virtual switch case. Five environments were measured:

1. a VM stack communicating with a VM router which was attached to an OSA, the requests then went through another OSA card which was attached to a VM router in another LPAR and it communicated with a VM stack in that LPAR

2. the same was done for a Linux stack communicating with a Linux router

3. the VM router in environment 1 was replaced with a virtual switch

4. the Linux router in environment 2 was replaced with a virtual switch

5. a VM router was substituted for the Linux router (on both sides)

Results:   The following tables compare results when running Linux TCP/IP communicating through a Linux router, a VM router and a virtual switch. Absolute values are given for throughput and CPU time and then ratios for comparison of a Linux router replaced with a virtual switch and also for a VM router replaced with virtual switch. All samples are from 50 client-server pairs. In almost all cases the throughput increased and CPU time decreased when a virtual switch was used. Data is not included here for the VM stack to VM router and VM stack to virtual switch measurements, but the results were similar. CPU time went down and throughput went up for most measurements.

Table 1. Comparison - Linux Router/VM Router with VSwitch - Streaming

MTU Size 1492	Linux to Linux Router	Linux to VM Router	Linux to VSwitch
MB/sec CPU msec/MB	28.09 54.68	53.98 20.06	53.24 16.23
ratio MB/sec ratio CPU msec/MB	1.00 1.00		1.90 0.30
ratio MB/sec ratio CPU msec/MB		1.00 1.00	0.99 0.81
Note: 2064-109; z/VM 4.4.0 with 64-bit CP; TCP/IP 440; Linux 2.4.19 31-bit

Table 2. Comparison - Linux Router/VM Router with VSwitch - Streaming

MTU Size 8992	Linux to Linux Router	Linux to VM Router	Linux to VSwitch
MB/sec CPU msec/MB	78.34 18.88	71.94 12.93	108.59 10.22
ratio MB/sec ratio CPU msec/MB	1.00 1.00		1.39 0.54
ratio MB/sec ratio CPU msec/MB		1.00 1.00	1.51 0.79
Note: 2064-109; z/VM 4.4.0 with 64-bit CP; TCP/IP 440; Linux 2.4.19 31-bit

Table 3. Comparison - Linux Router/VM Router with VSwitch - CRR

MTU Size 1492	Linux to Linux Router	Linux to VM Router	Linux to VSwitch
trans/sec CPU msec/trans	1426.63 1.09	2778.33 0.60	2966.96 0.40
ratio trans/sec ratio CPU msec/trans	1.00 1.00		2.08 0.37
ratio trans/sec ratio CPU msec/trans		1.00 1.00	1.07 0.67
Note: 2064-109; z/VM 4.4.0 with 64-bit CP; TCP/IP 440; Linux 2.4.19 31-bit

Table 4. Comparison - Linux Router/VM Router with VSwitch - CRR

MTU Size 8992	Linux to Linux Router	Linux to VM Router	Linux to VSwitch
trans/sec CPU msec/trans	2220.97 0.73	4146.52 0.41	4187.93 0.29
ratio trans/sec ratio CPU msec/trans	1.00 1.00		1.89 0.40
ratio trans/sec ratio CPU msec/trans		1.00 1.00	1.01 0.71
Note: 2064-109; z/VM 4.4.0 with 64-bit CP; TCP/IP 440; Linux 2.4.19 31-bit

Table 5. Comparison - Linux Router/VM Router with VSwitch - RR

MTU Size 1492	Linux to Linux Router	Linux to VM Router	Linux to VSwitch
trans/sec CPU msec/trans	9915.42 0.15	13078.97 0.09	18064.12 0.06
ratio trans/sec ratio CPU msec/trans	1.00 1.00		1.82 0.40
ratio trans/sec ratio CPU msec/trans		1.00 1.00	1.38 0.67
Note: 2064-109; z/VM 4.4.0 with 64-bit CP; TCP/IP 440; Linux 2.4.19 31-bit

Table 6. Comparison - Linux Router/VM Router with VSwitch - RR

MTU Size 8992	Linux to Linux Router	Linux to VM Router	Linux to VSwitch
trans/sec CPU msec/trans	9942.71 0.15	12985.37 0.09	17970.93 0.07
ratio trans/sec ratio CPU msec/trans	1.00 1.00		1.81 0.47
ratio trans/sec ratio CPU msec/trans		1.00 1.00	1.38 0.78
Note: 2064-109; z/VM 4.4.0 with 64-bit CP; TCP/IP 440; Linux 2.4.19 31-bit

The following charts compare the same environments but are focused on CPU time spent. The tables show virtual and CP msec/MB (for streaming) or msec/trans (for CRR and RR) for both the client and router stacks. For the virtual switch case, the router stack is the virtual switch controller. Absolute values are shown. Notice that there is no virtual time for the router in the virtual switch case.

Figure 3. CPU time Streaming

When using a virtual switch, CP time is charged to the VM TCP/IP controller while handling interrupts (ie processing QDIO buffers). Once the datagrams have been queued to the receiving stack, and receive processing starts, CP time is charged to the receiving stack. The reverse is true for the send case. While CP is extracting segments from the stack's buffers, the time is charged to the sending stack. At this point the buffers are given to the OSA-Express card and no further time is charged.

Figure 4. CPU time CRR and RR

The results for MTU size of 8992 were similar to those shown for MTU of 1492 and are therefore not shown.

Summary:   Because the amount of processing needed to handle the data being presented by/to the OSA-Express has been compressed, fewer processor cycles are needed to handle the workload. This means there is the potential for improved throughput in cases that were CPU-constrained.

There are no known cases where a virtual switch would not improve either CPU consumption, throughput or both. So, if you now have stacks communicating to an OSA-Express through a stack router, then you will benefit by converting that router to a virtual switch.

Contents | Previous | Next