System iNetwork

January 2008

January 1, 2008 4:51 PM

Achieving Gigabit Ethernet Benefits in i5/OS

Dear Doctor,
We recently upgraded our entire backbone infrastructure to Gigabit Ethernet. As part of the upgrade, we also added Gigabit Ethernet NICs to our Windows servers (our System i already had Gigabit NICs). As expected, network performance on the Windows servers improved dramatically; for example, our nightly backups complete in one-fifth of the time it used to take! Alas, our System i is seeing no speed improvements at all, even in large file transfers to the Gigabit-enabled Windows servers. We ran an SNMP monitor and are seeing no packet errors on the servers, System i, or intervening switches. What could be wrong?

Gentle User,
Dr. I Doctor believes you may be experiencing the ill effects of obsolete default setting in i (nee i5/OS) TCP/IP. The problem is that many of these defaults were chosen back when 128K bps was considered "high speed" Internet access. The particular setting you should check is the global TCP send and receive buffer sizes. The default for this is a measly 8,192 bytes, but for Gigabit performance you should increase this to at least 65,536 bytes.

Use the Change TCP/IP Attributes (CHGTCPA) command to display the current buffer sizes and make the necessary changes. You may have to experiment with size to achieve optimal results. Keep in mind that these values will be used as the buffer allocation for every TCP/IP socket opened. Thus changing from 8,192 to 65,536 bytes means that every socket will now consume 128K of memory rather than the previous 16K. If you run many simultaneous open sockets, this could affect your overall memory utilization and require a memory upgrade.

Posted by mbeckman on January 1, 2008 at 4:51 PM | Comments (0)

High CPU Load on DHCP Servers

Dear Doctor,
We recently extended our campus LAN to several new buildings with hundreds of new users. To limit the range of network broadcasts and potential broadcast storms, we've followed the recommended practice of segmenting the network into multiple IP subnets using layer-3 switches. However, we have only two DHCP servers for the entire network, a primary and a backup, to avoid replicating the DHCP hardware on every subnet. We use DHCP Helper addresses in the switch VLAN configurations to forward DHCP requests from each subnet to the DHCP servers. As we add new subnets, we've notice the CPU load on the DHCP servers is steadily climbing. It's alarmingly at about 80 percent, and we've added only half of the users. A sniffer analysis shows a great deal of UDP traffic redirected by the switches. This seems surprising, as I never considered DHCP a very high-traffic protocol. What can we do to ameliorate this?

Gentle User,
You have admirably followed best practices, yet you've still encountered problems. As practices go, what you've done so far is indeed pretty good. Alas, as you've seen, it's not quite good enough. Fortunately Dr. I Doctor can make it all good.

The problem you're seeing is due to an artifact of the DHCP Helper mechanism implemented in routers and layer-3 switches. For the purpose of this discussion, routers and layer-3 switches are identical devices. The default behavior for most routers is to forward all UDP broadcast traffic from a subnet to the designated DHCP servers. It turns out that this can be quite a lot of traffic, as broadcast DHCP is employed for services such as device bootstrap, netbios, TAC authentication, and network time protocol (NTP), to name just a few. Your DHCP servers are receiving all this traffic, and the need to examine every packet to filter out just the DHCP requests is a processor-intensive task, resulting in the high CPU loads you've observed.

The fix is to restrict DHCP Helper forwarding to just DHCP UDP packets. Not all routers and switches have this capability, but most higher-end products do. The procedure should be documented in the device configuration guide. For Cisco routers, you enter the following commands in global configuration mode:

no ip forward-protocol udp boot-pc
no ip forward-protocol udp boot-ps
no ip forward-protocol udp discard
no ip forward-protocol udp domain
no ip forward-protocol udp echo
no ip forward-protocol udp isakmp
no ip forward-protocol udp nameserver
no ip forward-protocol udp netbios-dgm
no ip forward-protocol udp netbios-ns
no ip forward-protocol udp netbios-ss
no ip forward-protocol udp tacacs
no ip forward-protocol udp time
no ip forward-protocol udp tftp
no ip forward-protocol udp xdmcp

Not all of these protocols are common bandwidth hogs, but it's a good idea to block forwarding of them unless you specifically use them in your network. In particular, echo, time, and xdmcp are exploited by hackers as network discovery probes.

Posted by mbeckman on January 1, 2008 at 4:48 PM | Comments (0)