System iNetwork

Ask Dr. I. Doctor

September 1, 2008

Sharing Internet for Redundancy

Dear Doctor,
A neighboring company with a high-speed Internet connection similar to ours, but from a different provider, has proposed that we pool our resources so that one company's connection can support the other company should it lose Internet access. A Cat-5 cable already interconnects our two data centers. I've found a number of firewalls that support dual-WAN configuration, so we bought a pair and paid for a fiber optic cable between our buildings. Alas, I can't figure out how to interconnect the two firewalls so that they fail over correctly. Each has its own WAN IP address, and connecting the four WAN ports in a common switch (one at each location) results in "IP address spoof" error messages, and failover doesn't work. How can we disable these errors in our firewalls and get them to fail over correctly?

Gentle User,
Don't touch those firewalls! They're only doing their job. Disabling "spoof" messages without addressing the cause of the message is like disabling a beeping smoke detector without looking for actual smoke. The firewalls are telling you that there is a problem, and in this case it's a simple one: You can't securely mix traffic from two different IP subnets on the same Layer-2 Ethernet domain.

The fix is easy: Replace the low-end switches you installed at each location with more modern, full-managed Ethernet switches that have Virtual LAN (VLAN) capabilities. Then create two VLANs, one for each firewall WAN IP subnet, and trunk the VLANs between buildings. You can now plug the two firewall WAN ports (primary and failover) on each site in to the two VLANs. Spoofing errors will cease, and failover will work like a charm.

Posted by mbeckman on September 1, 2008 at 4:06 PM

Missing Incoming Email

Dear Doctor,
We've recently seen a sudden increase in reports of lost email — email our users send that recipients never receive. Last month we moved our System i email server to a collocation facility, which entailed changing the server's IP address. Everything seemed to be working fine, but now this problem has cropped up. The colo facility manager sees no connectivity or traffic problems, and we're receiving no bounce or other error messages. What would cause only certain messages to be dropped?

Gentle User,
Dr. I Doctor wishes you had provided more detailed information about the kind of messages being dropped. Are all messages to particular destinations dropped? Or only those mailed from certain originators? Is there any kind of pattern? Alas, you are not here to answer these queries, so Dr. I Doctor will have to use the shotgun approach.

But first the answer to your last question: What would cause only certain messages to be dropped? The unhappy answer is that not all mail servers apply the same rules when accepting inbound mail. Some will take mail from anybody, others conduct various tests to try to reject potential spam and malicious emails before they gain a foothold on user computers. Dr. I Doctor suspects your mail server is being misidentified as harmful. Common practice dictates that harmful messages be dropped with no warnings, which might give interlopers insight into a destination mail server's filter process.

The first check to conduct is an easy one: see whether your mail server is listed on any email blacklists. You can do this easily at senderbase.org, a site that will match your mail server's IP address and domain name against dozens of potential blacklists. If you're on a blacklist, SenderBase will direct you to instructions for remediation. Addressing whatever caused your server's blacklisting in the first place will likely solve your problems.

If you're not blacklisted, the second most common problem is incorrect DNS information for your mail server's IP address, a plausible concern given you just changed that address. A reverse-DNS lookup on the IP address must return a name, and a DNS lookup on that name, in turn, must return the original IP address. A failure of either test means your mail server looks a lot like a spoofing source, one from which many reputable mail servers will refuse to accept mail.

The third most common problem is server misconfiguration. Often you simply have a spelling error in the server's official name in the mail configuration file. For example, if your server is named "postmaster.acme.com" and you inadvertently have "toastmaster.acme.com," the greeting message your server gives to other servers might not be universally accepted, causing outbound mail dropping.

A fourth issue is an incorrect Sender Provider Framework (SPF) DNS record value. Although not required, when an SPF record exists, it must accurately list the mail servers authorized to transmit mail for your domain. Given the recent move of your mail server, if you have an SPF (always a good idea) and forgot to update it with your mail server's new IP address, recipients that check the SPF record would drop your transmissions.

Posted by mbeckman on September 1, 2008 at 4:04 PM

August 1, 2008

Choosing Between FC and iSCSI SAN

Dear Doctor,
We're moving to server virtualization and are planning for a Storage Area Network (SAN) array to provide disk storage for all our virtual machines. My problem is choosing between the two SAN technologies, Fibre Channel (FC) and iSCSI. I know iSCSI is newer, but FC is 4 Gbps versus iSCSI's 1 Gbps. FC costs more, but I don't want to saddle our virtualization effort with a slow storage solution unnecessarily. We will be running mission-critical database applications on the SAN, and performance must equal what we see with native hardware. Is there any clear right answer to this question?

Gentle User,
The essence of your question is performance, so the answer depends on the severity of the disk transaction load you want to virtualize. It's true that FC is faster today, but iSCSI's next speed bump is from 1 Gbps to 10 Gbps, more than double the performance of FC. If possible, try testing your application on an iSCSI SAN before deployment. Some iSCSI vendors also have modeling tools that can help you predict performance.

The most expensive aspect of FC is the infrastructure: You must purchase expensive FC Host Bus Adapter (HBA) cards for every virtual host server and interconnect them with pricey, proprietary FC switches. This can double or triple the overall cost of virtualization deployment. iSCSI uses standard off-the-shelf Ethernet equipment that is much less costly. Where necessary, you can augment standard hardware with high-performance iSCSI TCP Offload Engine (TOE) or HBAs to address performance needs on individual servers. If money is no object, FC is the safest solution today.

Posted by mbeckman on August 1, 2008 at 1:01 AM

Backscatter Spam

Dear Doctor,
Over the last few weeks, our users have seen an alarming increase in e-mail bounce messages that claim we are sending spam. The messages have subjects such as "Message rejected as spam" and "Message could not be delivered: policy reject." The thing is, they're definitely from legitimate companies, not spoofed. I immediately suspected some kind of spam-spewing virus on our network, but after a careful check of our intrusion detection system (which is great about flagging illegitimate messages originating from within our network), I find no evidence of a problem. Moreover, the only mail server permitted by our firewall to transmit SMTP (port 25) packets has a detailed log of every message sent, and none of the bounce messages' mail addresses are in the log. Meanwhile, the problem continues, and I'm concerned that our mail server will get blacklisted.

Gentle User,
Dr. I Doctor admires your diligence at both instituting quality network protections and searching for the cause of this problem. In this instance, however, the problem is not with your network, as your network never originated the messages in question. No computers have been infected, no servers hijacked, and no borders penetrated. What you're seeing is the fallout from the latest spammer attack strategy called "backscatter spam."

You were close to the cause of the problem when you suspected spoofing. It's not the source of the bounce-back message being spoofed, however; it's the source of a message the spammer meant for you to receive. The spammer simply put one of your users' e-mail addresses on the "from" line, then sent the message to hundreds of perfectly legitimate companies, which then promptly identified it as spam and bounced it back to what they though was the source but, in reality, was your spoofed e-mail address.

How did the spammer get your e-mail addresses? Let Dr. I Doctor count the ways: viruses in the computers of your correspondents, past viruses on your old network, Internet posts, compromised membership mailing lists, drive-by Trojan websites, backdoor sales of mailing lists, your own corporate website, even simply guessing. Getting addresses isn't a problem for spammers; their problem is getting the message into your mailbox and making you read it. Bounce messages are the perfect vehicle for that, at least for now.

Undoubtedly you want to know how to block these messages. It's not simple, since you would like to get bounce-back e-mail to alert you when a destination address is unreachable or nonexistent. Although one of the most powerful tools against spam, Sender Provider Framework (SPF) fails to block backscatter because the messages originate from a legitimate server and are addressed to a correct destination.

Currently there's no foolproof fix. You can block all bounce messages — most mail servers have provisions for this, and bounce messages have a "bounce" flag in their headers, but then you'll miss important feedback about your messages in flight. The phenomenon is new, and soon spam filter vendors should have the ability to detect these messages based on content. But perhaps the most useful measure is for your own mail server to never bounce messages that have invalid destinations — simply drop them. That way, at least you're not contributing to the problem.

Posted by mbeckman on August 1, 2008 at 1:01 AM

July 1, 2008

Fiber Optic Transceiver ARP Caching

Dear Doctor,
One interface on our System i box connects to an Ethernet switch that connects through a fiber-optic transceiver (FOT) and cable that runs 3,000 feet to our training center, where another FOT turns the signal back into Cat-5e that plugs in to another Ethernet switch. We sometimes need to unplug PCs from our main building and take them to the training center for special classes. Often when we do this, the PCs won't communicate to our System i server over the fiber, yet they can connect to other Ethernet devices in the training center. I've tried rebooting the switches at either end (and even rebooting i5/OS) with no change. Oddly enough, computers left in the training center overnight always work fine the next day. What gives?

Gentle User,
Your timing with this question is perfect, as many organizations are expanding their enterprise LANs with fiber cabling and encountering similar problems. The problem itself is one of ARP cache timing — as Dr. I Doctor will explain in a moment. But first, here is an immediate solution to your problem: power cycle the FOTs at each end of the fiber link. This may seem like voodoo, but trust your Doctor, it will work.

FOTs are often treated as passive connectors, especially in legacy networks where they're used to interconnect copper-based Ethernet switches. You naturally focus on the switches as the active components and think of the FOTs as just another kind of adapter. But a FOT is, in fact, a full-fledged Ethernet bridge. Like any bridge, it switches packets between interfaces based on Ethernet hardware (MAC) addresses it learns from the devices plugged in at each end. Two FOTs also exchange bridge information between themselves. It's a complicated process, involving the spanning tree protocol and, if you're not careful, can cause the symptoms you're seeing.

The FOT in your main office learns the MAC addresses for all the PCs connected to your main office switch, retaining them for some preconfigured timeout interval, which could be from five minutes to five hours. In your case, the timeout appears to be on the long side. If a PC moves to the other end of the fiber link, the FOT at that location may refuse to learn the PCs MAC address because it is already in its table (obtained from the main-office FOT). The FOT does this to avoid a potential bridge loop.

Power cycling the FOTs clears their MAC address tables, letting each side learn the current location of all PCs, thus solving the problem. Some sophisticated FOTs are programmable, letting you configure the MAC address timeout. If yours isn't, your only alternative is to power cycle the FOTs or replace your existing switches with fiber-capable switches, eliminating the FOTs.

Posted by mbeckman on July 1, 2008 at 1:01 AM

June 1, 2008

Duplicated Incoming Email Messages

Dear Doctor,
We're a total i5/OS shop and run several different kinds of mail servers: Lotus Notes, CommuniGate Pro, and the native i5/OS SMTP server. Each server provides e-mail for a specific suborganization of our enterprise. Recently, we've had a strange problem that affects all three servers: Incoming mail messages occasionally get duplicated, or worse. I've seen up to six copies of the same message. I see no pattern in where the duped messages originate, and with three completely different mail servers, it's hard to believe this is our problem.

Gentle User,
The problem you see is a common artifact of the Simple Mail Transport Protocol (SMTP) used to deliver messages to your servers. They don't call it "simple" for nothing.

SMTP is generally reliable, but it has serious problems when faced with network congestion or high server workloads that slow the interaction between sending and receiving server. When a sending server establishes an SMTP session with one of your servers and transmits an e-mail, the sender expects a "250" reply confirming reception. If your mail server is too busy to respond quickly, or severe network congestion requires one or more retransmissions of the "250" reply, the SMTP session could time out, leaving the sender with the impression that the message wasn't delivered. So it sends it again. And again.

Since multiple originating mail servers exhibit this behavior, the problem is likely with your network. Either the network itself is too congested at the Internet border, or your mail servers are all very busy. One cause for pointless busyness among related mail servers is mail looping from one server to another — a problem you can detect by monitoring the mail queues on each server.

Posted by mbeckman on June 1, 2008 at 4:00 PM

VoIP echo on POTS lines

Dear Doctor,
Our VoIP system has two kinds of trunk connections to the local exchange carrier (LEC): an ISDN PRI T1 circuit with six channels for long distance and another four plain old telephone service (POTS) analog voice lines used for local calls. We're charged by the minute for both local and long distance on the ISDN PRI, while there are no per-minute charges on the POTS lines for local calls. The problem is that calls through the POTS lines suddenly have a terrible echo on them. The phone company tested and says that it's our gear; the VoIP vendor blames the phone company. Whose fault is it, really? And how do we fix it?

Gentle User,
Dr. I Doctor will have to play Solomon in reverse to solve your problem. Fortunately, it is one that has become common enough lately that the cause is easily determined. Alas, Dr. I Doctor cannot promise as much for the solution.

The cause of the problem is your LEC, which has almost certainly made a change to its system to which it's unwilling to admit. Small and large businesses aren't the only entities that recognize the economics of VoIP. Many LECs thirst for the savings VoIP promises in reduced bandwidth and cheaper infrastructure; your LEC is likely one of them. All over the country, LECs are switching their internal voice transport from the traditionally reliable, but fairly expensive, circuit-switched (CS) network to much cheaper Internet Protocol (IP) networks.

Your LEC is not routing your phone calls through the Internet. That would be truly awful, given the vagaries of Internet routing and congestion. However, the LEC is routing your calls through its own internal private IP network, which is much less costly to maintain than the old CS network. Theoretically, this shouldn't be a problem — the LEC still accepts analog calls and digitizes them for transport over IP, just as it did with its CS network. Calls originating on ISDN or T1 digital trunks are digitized by the customer before they get to the phone company, so they are not affected by this change.

But with your VoIP system, calls are digitized in your phones, then sent to the POTS gateway where they are converted to analog signaling, then routed to the LEC, which redigitizes the signal for IP transport. It's the intermediate conversion to analog that's causing difficulties; this conversion often frustrates the echo-cancelling technology normally built into your phone system. The LEC's intermediate digitization step causes enough signal degradation to prevent echo recognition, so echoed voices that would normally be canceled get through to the end user.

The unfortunate news is that you likely can't fix this and still use POTS lines with your VoIP PBX. You might have the VoIP vendor try adjusting your PBX echo cancellation settings, but that usually won't remedy the problem for all calls. Your best bet is to bite the cost bullet and simply make local calls through your ISDN PRI lines, which are immune to this particular malady.

Posted by mbeckman on June 1, 2008 at 3:54 PM

May 1, 2008

Clipped Speech During VoIP Calls

Dear Doctor,
I installed a Voice over IP (VoIP) phone system for our office, initially set up to use Internet VoIP trunks. The reliability and sound quality of Internet phone calls was too poor for our management's taste, so I switched to Plain Old Telephone Service (POTS) analog lines. Alas, to my surprise, we still have significant sound-quality problems. The worst is that the beginning and ending of every spoken sentence seems to be clipped off. But we also hear a distinctive hiss that is both annoying and not audible when I place a call with an analog phone on the same lines. Management is ready to throw out the VoIP system. Help!

Gentle User,
When directly connected to analog POTS lines, a VoIP system should have sound quality very close to that of a traditional PBX. The clipping and hiss symptoms indicate that you may have inadvertently left some sound compression features enabled — features that aren't needed with direct POTS phone trunks.

The most common cause of word clipping is a bandwidth-saving feature called Silence Suppression, which attempts to detect gaps in spoken conversation to reduce the amount of encoded data being sent over the Internet. For analog lines, you should disable this feature.

Hiss is a characteristic of some voice compressor/decompressor (codec) algorithms, which compress encoded audio to further reduce bandwidth requirements. You generally can configure the codec used for specific kinds of calls; for analog calls, the codec should be G.711, resulting in a 64 Kbps uncompressed audio stream (which takes about 80 Kbps in network traffic to transport). On a LAN this bandwidth is negligible. Dr. I Doctor suspects you have inadvertently configured one of the high-compression codecs, such as G.729. Reconfiguring to G.711 should remove all hiss.

Posted by mbeckman on May 1, 2008 at 4:28 PM

Windows Remote File Copy Timeout Errors

Dear Doctor,
Our remote offices' Windows servers nightly perform file copy operations from our central System i server for database synchronization. Recently, however, the file copy operations have begun randomly failing. Some nights they all work, but other nights one or more copies fail, with Windows giving the inexplicable error "timeout occurred." We have plenty of bandwidth, and other non-Windows copies running in parallel prove that network outages aren't occurring. Packet captures at each end reveal TCP RST (reset) packets being received that were not sent by the other end, which correlates with the timeout errors. What could be causing this?

Gentle User,
The mystery of random TCP resets is no mystery to those familiar with intrusion detection and prevention services. This behavior is the classic signature of a hacker filter attempting to prohibit what it sees as illicit traffic. That hacker filter is likely on a firewall at one end or the other of your communication link, but it could also be one employed by the ISP at either end as well.

Intrusion detection systems (IDS) and intrusion prevention systems (IPS) are mechanisms for detecting and blocking, respectively, hacker attempts to break into a network. The IDS process monitors all traffic, looking for patterns defined in an attack signature database. When the detection process gets a match, it may notify the IPS component to put a stop to the traffic. One method of forcing a stop is to send a TCP Reset to each side, shutting down the connection.

This process appears to be working perfectly in your case. The IDS identifies your legitimate file transfers as intrusion attempts, and the IPS component promptly shuts them down. Windows reports this as a timeout (although it should probably report it as unexpected termination).

The problem, of course, is that your file transfers are not intrusion attempts. Alas, one of the weaknesses of IDS is false positives — normal traffic erroneously identified as malicious. Fortunately, you can usually adjust IDS sensitivity to circumvent this problem. Check the logs of any intervening firewalls to see whether any IDS alarms are raised that identify the IP addresses involved in your file transfers. Then either white-list those IP addresses or reduce the sensitivity of the particular attack signatures being triggered.

If you don't have IDS/IPS enabled in any of your own firewalls, then look to the ISPs at either end. More and more ISPs are employing IDS on their edge networks in an effort to limit the affect of virus traffic. One ISP or the other may have its IDS screwed down a little too tight. If you can't get the ISPs to address the problem, then set up VPN tunnels between the locations. IDS by design will recognize and let pass IPSec VPN traffic.

Posted by mbeckman on May 1, 2008 at 4:27 PM

April 1, 2008

Slow File Transfers in Windows Vista

Dear Doctor,
Our branch office network recently upgraded many workstations to Windows Vista. We've noticed that file transfers to and from these machines and the Internet is much slower — about 10 times slower — than our XP and Mac machines. Yet the same machines can transfer files fine over our VPN to the home office. We use a Cisco PIX firewall for our Internet connection and a Sonicwall VPN appliance for our corporate VPN. I'm perplexed!

Gentle User,
You are not alone in your perplexity. Many Vista users have reported this problem, and Microsoft has explained the behavior and published a workaround.

Windows Vista enables a little-used TCP/IP option called Window Scaling, which allows for more than 65K in a TCP receive window. While useful for high-speed LANs, many edge devices, such as routers and firewalls, don't support this option, causing the performance problem you've observed.

The fix is to simply disable Window Scaling in Vista. You must run the following DOS command as Administrator (right-click the Command Prompt icon and choose "Run as Administrator"):

netsh interface tcp set global
autotuninglevel=disabled

Posted by mbeckman on April 1, 2008 at 4:32 PM

Deploying Domain Key Identified Mail (DKIM)

Dear Doctor,
In an effort to reduce the amount of phishing e-mail going to our users, I've been asked to set up processing for something called Domain Key Identified Mail (DKIM). I gather DKIM is some kind of encryption system, but I don't have the foggiest idea where to begin to implement it. We currently use CommuniGate Pro mail server running under i5/OS, but the vendor doesn't seem to have much info on the subject.

Gentle User,
Your managers are thinking well to implement Domain Key Identified Mail, as it's one of the best antiphishing tools to come down the pike in a long time. DKIM does use encryption technology, as you suggest, although it does not actually encrypt message content. Phishing target organizations, such as financial institutions, e-commerce sites, and online auctions, increasingly employ DKIM to let users verify that mail purporting to be from one of the target organizations actually is from that organization.

The sending organization digitally signs all of its outgoing mail with a private key and then publishes its public key via DNS. A receiving mail server (called a mail transfer agent, or MTA, in the e-mail biz) uses DNS to check for a Sender Signing Policy (SSP) for the domain of every incoming message. If an SSP exists, the receiving MTA retrieves the public key using another DNS query, then uses that key to verify the digital signature in the incoming message. If the signature is invalid, or nonexistent, the message is fraudulent and gets discarded.

Getting this to work in legacy MTAs, such as CommuniGate Pro, can be challenging. In your case, CGPro's vendor, Stalker Software, has not yet built DKIM into its product. A third-party CGPro plug-in, DomainKeys Helper from Niversoft.com, adds this support to CGPro, but only for Intel-architecture servers, excluding i5/OS (which employs PowerPC architecture). In this situation, the easiest solution is to interpose a DKIM-capable MTA between your MTA and the Internet. You can do this using a separate hardware server or appliance, or on your System i box via Linux under LPAR. With either method, you would configure the front-end MTA to apply the DKIM process to all incoming messages, and pass any not dropped to your i5/OS MTA.

If you're handy with Linux, you could roll your own DKIM MTA using Sendmail (http://www.sendmail.org), an open-source mail server that includes MTA support. Some other open-source MTA's also have DKIM support, as do some e-mail filtering services, for which you pay a per-month, per-mailbox support fee.

Whatever method you choose, a critical prerequisite for DKIM is that your DNS be secure and reliable. If an interloper can compromise DNS, she can bypass DKIM protection. You should ensure that your DNS server code is up to date and fully patched, and that you employ separate DNS servers for inside and outside name resolution. Your inside DNS server must not be open to outside queries, and your external DNS server should not open to recursive queries (called an "open DNS").

Posted by mbeckman on April 1, 2008 at 4:30 PM | Comments (0)

March 1, 2008

T1/T3 Circuit Troubleshooting Using Loopback

Dear Doctor,
We have several T1 and T3 circuits between our main office and branch locations. These all connect to a central router, which relays traffic between the branches and to our centralized server farm. Lately we've experienced a series of circuit outages that we can't diagnose. We suspect the telco provider, but it always runs a "loop-back" test to our equipment at each location and reports that the failed circuit is operating normally, indicating a hardware failure with our gear. But after the provider conducts its tests, the circuit mysteriously comes back online. We ask for support from our hardware vendor, and the vendor says it's the telco's fault. How can we stop the finger pointing and get to the bottom of this?

Gentle User,
The failure is not just with your routers and circuits, but with the hardware vendor and telco. You are the victim of misdirection from both parties.

In its loop-back test, the telco sends a special code to its premises equipment (called the Network Interface Unit) that creates a temporary connection between the send-side and receive-side of the circuit, so that test data transmitted by the telco "loops back" to the telco where it can be verified. The telco's misdirection is that a successful loop-back test proves a circuit is working normally. It does no such thing, because the telco loop-back test doesn't exercise two critical components in a circuit: the cabling between your router and the NIU, and the "customer" side of the NIU's electronics and connectors.

The second bit of misdirection is from your hardware vendor, which rather than finger pointing should be performing its own end-to-end loop-back tests. Only when both tests are correlated can you identify the failing component in the circuit. Dr. I Doctor recommends that you get the hardware vendor and telco on a three-way call, then insist that they run the full set of tests before unhostering their fingers.

Posted by mbeckman on March 1, 2008 at 4:42 PM | Comments (0)

Proxy ARP Timeouts

Dear Doctor,
Our System i is in a secure collocation facility with redundant upstream Internet feeds connected to redundant firewalls. Between the firewalls and the System i are two Ethernet switches, one connected to each of the two System i Ethernet NICs. I'm trying to use the i5/OS Virtual IP Addressing feature to automatically fail over from one port to the other should any one of the firewalls, switches, or Ethernet NICs fail. To that end, I've configured NIC A on the System i with IP address 10.0.0.1 and NIC B with 10.0.0.2; the VIPA IP address is 10.0.0.3. Alas, when I test this by failing any one of the redundant components, failover doesn't seem to happen.

Gentle User,
Ethernet resilience is one of those technical achievements that engenders many approaches but few agreed-upon standards. The failover problem is simple: getting other devices in a network to send traffic to a different physical interface once the Ethernet hardware (MAC) address for a particular IP address has been resolved using Address Resolution Protocol (ARP). Dr. I Doctor is certain that in your situation the devices haven't agreed upon a solution, hence, the failure to failover.

One common solution to this problem is to generate a fictitious MAC address and assign that to the VIPA IP address. In normal operation, the primary Ethernet NIC responds to the fictitious MAC, but when a failure occurs, the secondary NIC takes on that responsibility. Intervening switches will see the MAC change as a device move and reroute traffic accordingly. Other devices in the network are none the wiser, and the failover is completely transparent to them.

i5/OS uses an older approach called Proxy ARP. Proxy ARP lets a particular NIC respond to ARP requests for an IP address even if that address is not physically assigned to the NIC. In normal operation, the primary NIC performs Proxy ARP, and thus ARP requests resolve to the physical MAC of that interface. If that NIC fails, the secondary NIC takes over Proxy ARP, giving out the secondary NIC's MAC for ARP queries to the VIPA IP address.

The problem at this point is that other devices on the LAN have cached the MAC address of the primary NIC, and they may not re-ARP the address for many minutes, or even hours. In failover, these other devices continue to address packets to the failed primary NIC's MAC. The fix, then, is to somehow force these devices to re-ARP more often.

Usually this is a configurable option in the TCP/IP settings for the device, with an attribute name of "ARP timer" or "ARP timeout." You should set this value to the longest time you're willing to delay failover, keeping in mind that shorter timeouts will increase the amount of broadcast traffic on your LAN. A value of 30 seconds isn't unreasonable and shouldn't greatly increase broadcast traffic. With this setting, failover should never take more than 30 seconds.

Posted by mbeckman on March 1, 2008 at 4:40 PM | Comments (0)

February 1, 2008

Switch Port Broadcast Flooding in ISA Server Load Balancing

Dear Doctor,
Our network border is controlled by a pair of Windows servers running Microsoft's Internet Security and Acceleration (ISA) Server 2004 in a load-balancing configuration. We process a high volume of SSL transactions, and the ISA server does a good job of distributing the encryption workload, while also providing for emergency failover should one ISA server die. We've noticed a problem, however, on both the LAN and WAN switch ports to which the ISA server connects. These ports have high traffic volumes, even when the level of SSL connections is low. After connecting a sniffer, we see packets unrelated to the ISA servers — packets that should be going to other devices on our LAN! How can this be? I thought switches ensured that such traffic mixing never occurred.

Gentle User,
Switches normally do isolate traffic so that each port carries only packets destined for the devices connected to that port. However, there are two instances when this rule is broken: when the link state of a port changes (e.g., from down to up), and when the switch sees a MAC address arrive on a port that isn't the MAC address table entry for that port. In some switches when these events occur, the switch invalidates its MAC address table and reverts to broadcast mode — sending every packet to every port — until it relearns the port locations for each destination MAC address.

Normally, both of these events are rare, so broadcast traffic is infrequent. A failing Ethernet port, continually fluctuating between up and down, can cause a broadcast storm by constantly invalidating the MAC address table. So can a pair of devices, such as the ISA server, that uses MAC address spoofing to balance incoming traffic. Some Ethernet switches (particularly Cisco) do not behave well when confronted with the same MAC address on two different ports, resulting in the broadcast storm that you observed.

The best fix is to swap in Ethernet switches that are known to be compatible with ISA Server load balancing. If that's not practical, you may want to contact your current switch vendor to see whether it has a programmatic workaround.

Posted by mbeckman on February 1, 2008 at 4:46 PM | Comments (0)

Jumbo Frames on Virtual Ethernet Interfaces

Dear Doctor,
Our System i includes a couple of Linux LPAR partitions that we use to collect data from other systems; the data is then manipulated nightly in a very I/O intensive process. I discovered that I could speed the nightly process by a factor of four by simply changing the default Ethernet frame size of the virtual LAN interfaces from 1,500 to 9,000 bytes (so-called "jumbo" frames). Even though our System i has 100BaseT physical interfaces, I figured inter-Linux traffic won't use those, so I went for the maximum frame size. All went well the first night, but the next day users complained that the Linux applications had very slow network performance. I discovered that switching the frame size back to 1,500 bytes restored daytime performance, but now nighttime processing is slow again. Can't I have my cake and eat it too?

Gentle User,
You can have your cake and eat it too, even the jumbo-sized cake you want to consume at night. The trick is in adapting your virtual network to the physical network that you users traverse when using Linux applications during the day.

As you've seen, bumping the frame size can really speed network performance on the virtual network. The reason you encountered problems the next day is that your physical Ethernet, running at only 100 Mbps, isn't able to process jumbo frames. So it fragments them into convenient 1,500-byte chunks. Alas, all this fragmenting activity takes time and memory, both in i5/OS and in the destination host systems, which is why your users saw degraded performance.

You can circumvent this problem by creating two virtual networks, or VLANs: one for inter-LPAR traffic and one for communication with the outside world. Dr. I Doctor recommends restoring the 1,500-byte frame size for your existing VLAN and then creating a second VLAN for jumbo packet traffic, configuring the Linux VLAN interfaces with 9,000-byte frames. Take care to use a completely separate IP subnet for VLAN2. Your nightly Linux applications will now use this subnet for inter-LPAR communications. Devices on your physical LAN won't know how to reach this subnet, and thus will never traverse the jumbo-frame interfaces. Performance will be optimized for each VLAN.

Posted by mbeckman on February 1, 2008 at 4:44 PM | Comments (0)

January 1, 2008

Achieving Gigabit Ethernet Benefits in i5/OS

Dear Doctor,
We recently upgraded our entire backbone infrastructure to Gigabit Ethernet. As part of the upgrade, we also added Gigabit Ethernet NICs to our Windows servers (our System i already had Gigabit NICs). As expected, network performance on the Windows servers improved dramatically; for example, our nightly backups complete in one-fifth of the time it used to take! Alas, our System i is seeing no speed improvements at all, even in large file transfers to the Gigabit-enabled Windows servers. We ran an SNMP monitor and are seeing no packet errors on the servers, System i, or intervening switches. What could be wrong?

Gentle User,
Dr. I Doctor believes you may be experiencing the ill effects of obsolete default setting in i (nee i5/OS) TCP/IP. The problem is that many of these defaults were chosen back when 128K bps was considered "high speed" Internet access. The particular setting you should check is the global TCP send and receive buffer sizes. The default for this is a measly 8,192 bytes, but for Gigabit performance you should increase this to at least 65,536 bytes.

Use the Change TCP/IP Attributes (CHGTCPA) command to display the current buffer sizes and make the necessary changes. You may have to experiment with size to achieve optimal results. Keep in mind that these values will be used as the buffer allocation for every TCP/IP socket opened. Thus changing from 8,192 to 65,536 bytes means that every socket will now consume 128K of memory rather than the previous 16K. If you run many simultaneous open sockets, this could affect your overall memory utilization and require a memory upgrade.

Posted by mbeckman on January 1, 2008 at 4:51 PM | Comments (0)

High CPU Load on DHCP Servers

Dear Doctor,
We recently extended our campus LAN to several new buildings with hundreds of new users. To limit the range of network broadcasts and potential broadcast storms, we've followed the recommended practice of segmenting the network into multiple IP subnets using layer-3 switches. However, we have only two DHCP servers for the entire network, a primary and a backup, to avoid replicating the DHCP hardware on every subnet. We use DHCP Helper addresses in the switch VLAN configurations to forward DHCP requests from each subnet to the DHCP servers. As we add new subnets, we've notice the CPU load on the DHCP servers is steadily climbing. It's alarmingly at about 80 percent, and we've added only half of the users. A sniffer analysis shows a great deal of UDP traffic redirected by the switches. This seems surprising, as I never considered DHCP a very high-traffic protocol. What can we do to ameliorate this?

Gentle User,
You have admirably followed best practices, yet you've still encountered problems. As practices go, what you've done so far is indeed pretty good. Alas, as you've seen, it's not quite good enough. Fortunately Dr. I Doctor can make it all good.

The problem you're seeing is due to an artifact of the DHCP Helper mechanism implemented in routers and layer-3 switches. For the purpose of this discussion, routers and layer-3 switches are identical devices. The default behavior for most routers is to forward all UDP broadcast traffic from a subnet to the designated DHCP servers. It turns out that this can be quite a lot of traffic, as broadcast DHCP is employed for services such as device bootstrap, netbios, TAC authentication, and network time protocol (NTP), to name just a few. Your DHCP servers are receiving all this traffic, and the need to examine every packet to filter out just the DHCP requests is a processor-intensive task, resulting in the high CPU loads you've observed.

The fix is to restrict DHCP Helper forwarding to just DHCP UDP packets. Not all routers and switches have this capability, but most higher-end products do. The procedure should be documented in the device configuration guide. For Cisco routers, you enter the following commands in global configuration mode:

no ip forward-protocol udp boot-pc
no ip forward-protocol udp boot-ps
no ip forward-protocol udp discard
no ip forward-protocol udp domain
no ip forward-protocol udp echo
no ip forward-protocol udp isakmp
no ip forward-protocol udp nameserver
no ip forward-protocol udp netbios-dgm
no ip forward-protocol udp netbios-ns
no ip forward-protocol udp netbios-ss
no ip forward-protocol udp tacacs
no ip forward-protocol udp time
no ip forward-protocol udp tftp
no ip forward-protocol udp xdmcp

Not all of these protocols are common bandwidth hogs, but it's a good idea to block forwarding of them unless you specifically use them in your network. In particular, echo, time, and xdmcp are exploited by hackers as network discovery probes.

Posted by mbeckman on January 1, 2008 at 4:48 PM | Comments (0)