System iNetwork

February 2008

February 1, 2008 4:46 PM

Switch Port Broadcast Flooding in ISA Server Load Balancing

Dear Doctor,
Our network border is controlled by a pair of Windows servers running Microsoft's Internet Security and Acceleration (ISA) Server 2004 in a load-balancing configuration. We process a high volume of SSL transactions, and the ISA server does a good job of distributing the encryption workload, while also providing for emergency failover should one ISA server die. We've noticed a problem, however, on both the LAN and WAN switch ports to which the ISA server connects. These ports have high traffic volumes, even when the level of SSL connections is low. After connecting a sniffer, we see packets unrelated to the ISA servers — packets that should be going to other devices on our LAN! How can this be? I thought switches ensured that such traffic mixing never occurred.

Gentle User,
Switches normally do isolate traffic so that each port carries only packets destined for the devices connected to that port. However, there are two instances when this rule is broken: when the link state of a port changes (e.g., from down to up), and when the switch sees a MAC address arrive on a port that isn't the MAC address table entry for that port. In some switches when these events occur, the switch invalidates its MAC address table and reverts to broadcast mode — sending every packet to every port — until it relearns the port locations for each destination MAC address.

Normally, both of these events are rare, so broadcast traffic is infrequent. A failing Ethernet port, continually fluctuating between up and down, can cause a broadcast storm by constantly invalidating the MAC address table. So can a pair of devices, such as the ISA server, that uses MAC address spoofing to balance incoming traffic. Some Ethernet switches (particularly Cisco) do not behave well when confronted with the same MAC address on two different ports, resulting in the broadcast storm that you observed.

The best fix is to swap in Ethernet switches that are known to be compatible with ISA Server load balancing. If that's not practical, you may want to contact your current switch vendor to see whether it has a programmatic workaround.

Posted by mbeckman on February 1, 2008 at 4:46 PM | Comments (0)

Jumbo Frames on Virtual Ethernet Interfaces

Dear Doctor,
Our System i includes a couple of Linux LPAR partitions that we use to collect data from other systems; the data is then manipulated nightly in a very I/O intensive process. I discovered that I could speed the nightly process by a factor of four by simply changing the default Ethernet frame size of the virtual LAN interfaces from 1,500 to 9,000 bytes (so-called "jumbo" frames). Even though our System i has 100BaseT physical interfaces, I figured inter-Linux traffic won't use those, so I went for the maximum frame size. All went well the first night, but the next day users complained that the Linux applications had very slow network performance. I discovered that switching the frame size back to 1,500 bytes restored daytime performance, but now nighttime processing is slow again. Can't I have my cake and eat it too?

Gentle User,
You can have your cake and eat it too, even the jumbo-sized cake you want to consume at night. The trick is in adapting your virtual network to the physical network that you users traverse when using Linux applications during the day.

As you've seen, bumping the frame size can really speed network performance on the virtual network. The reason you encountered problems the next day is that your physical Ethernet, running at only 100 Mbps, isn't able to process jumbo frames. So it fragments them into convenient 1,500-byte chunks. Alas, all this fragmenting activity takes time and memory, both in i5/OS and in the destination host systems, which is why your users saw degraded performance.

You can circumvent this problem by creating two virtual networks, or VLANs: one for inter-LPAR traffic and one for communication with the outside world. Dr. I Doctor recommends restoring the 1,500-byte frame size for your existing VLAN and then creating a second VLAN for jumbo packet traffic, configuring the Linux VLAN interfaces with 9,000-byte frames. Take care to use a completely separate IP subnet for VLAN2. Your nightly Linux applications will now use this subnet for inter-LPAR communications. Devices on your physical LAN won't know how to reach this subnet, and thus will never traverse the jumbo-frame interfaces. Performance will be optimized for each VLAN.

Posted by mbeckman on February 1, 2008 at 4:44 PM | Comments (0)