Dr. I Doctor
Dr. I Doctor's Informational Juggernaut
January 1, 2008
High CPU Load on DHCP Servers
Dear Doctor,
We recently extended our campus LAN to several new buildings with hundreds of new users. To limit the range of network broadcasts and potential broadcast storms, we've followed the recommended practice of segmenting the network into multiple IP subnets using layer-3 switches. However, we have only two DHCP servers for the entire network, a primary and a backup, to avoid replicating the DHCP hardware on every subnet. We use DHCP Helper addresses in the switch VLAN configurations to forward DHCP requests from each subnet to the DHCP servers. As we add new subnets, we've notice the CPU load on the DHCP servers is steadily climbing. It's alarmingly at about 80 percent, and we've added only half of the users. A sniffer analysis shows a great deal of UDP traffic redirected by the switches. This seems surprising, as I never considered DHCP a very high-traffic protocol. What can we do to ameliorate this?
Gentle User,
You have admirably followed best practices, yet you've still encountered problems. As practices go, what you've done so far is indeed pretty good. Alas, as you've seen, it's not quite good enough. Fortunately Dr. I Doctor can make it all good.
The problem you're seeing is due to an artifact of the DHCP Helper mechanism implemented in routers and layer-3 switches. For the purpose of this discussion, routers and layer-3 switches are identical devices. The default behavior for most routers is to forward all UDP broadcast traffic from a subnet to the designated DHCP servers. It turns out that this can be quite a lot of traffic, as broadcast DHCP is employed for services such as device bootstrap, netbios, TAC authentication, and network time protocol (NTP), to name just a few. Your DHCP servers are receiving all this traffic, and the need to examine every packet to filter out just the DHCP requests is a processor-intensive task, resulting in the high CPU loads you've observed.
The fix is to restrict DHCP Helper forwarding to just DHCP UDP packets. Not all routers and switches have this capability, but most higher-end products do. The procedure should be documented in the device configuration guide. For Cisco routers, you enter the following commands in global configuration mode:
no ip forward-protocol udp boot-pc
no ip forward-protocol udp boot-ps
no ip forward-protocol udp discard
no ip forward-protocol udp domain
no ip forward-protocol udp echo
no ip forward-protocol udp isakmp
no ip forward-protocol udp nameserver
no ip forward-protocol udp netbios-dgm
no ip forward-protocol udp netbios-ns
no ip forward-protocol udp netbios-ss
no ip forward-protocol udp tacacs
no ip forward-protocol udp time
no ip forward-protocol udp tftp
no ip forward-protocol udp xdmcp
Not all of these protocols are common bandwidth hogs, but it's a good idea to block forwarding of them unless you specifically use them in your network. In particular, echo, time, and xdmcp are exploited by hackers as network discovery probes.
Posted by mbeckman at January 1, 2008 4:48 PM
| Sun | Mon | Tue | Wed | Thu | Fri | Sat |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| 9 | 10 | 11 | 12 | 13 | 14 | 15 |
| 16 | 17 | 18 | 19 | 20 | 21 | 22 |
| 23 | 24 | 25 | 26 | 27 | 28 | 29 |
| 30 | 31 |
Blog Policy
We welcome your comments and opinions and encourage lively debate on the issues. However, Penton Media reserves the right to delete or move any content that it may determine, in its sole discretion, violates or may violate its Terms of Use or is otherwise unacceptable. For more information, see Penton Media's Terms of Use.