TCP Connection States - A Clue to Network Health
Noah Davids Most network administrators know that the netstat -n command will display a list of TCP connections and their states. Usually, the connection state is ESTABLISHED. Occasionally, the state is FIN_WAIT_2 or CLOSE_WAIT. Very rarely it will be SYN_SENT, SYN_RCVD, or FIN_WAIT_1. Although all these states are normal, connections are typically in these states for only a brief time. Having connections in these states for longer than a few seconds, or having more than a few connections in these states at any one time may indicate a network problem. But in any case, the cause of this condition should be determined. This article explores how to diagnose network problems based on the state of the TCP connection. The TCP States The TCP states are defined in RFC-793. Figure 1 shows the TCP state machine, which defines what state to move to next and what type of packet to send back when a given type of packet is received. In general, TCP connections involve synchronizing the connection (SYN), transmitting the data, and then finishing, or closing the connection (FIN). As requests are made between systems, acknowledgements (ACK) are transmitted back and forth to assure that both sides of the connection agree on the state of the connection. For a more detailed explanation of these states, see the companion Web article on the Sys Admin Web site: samag.com. Diagnoses Table 1 lists most of the likely (and not so likely) causes of state problems. In all cases, host 1 is the host with the problem, and host 2 is the "other" host. Table 2 lists the causes that I see most often for each stuck state. This is not to say that I never see other causes. For example, I once saw a connection stuck in SYN_RCVD state because host 2 crashed after sending the connection request (cause 5). However, I cannot count the number of times that the routing table on host 1 was wrong (cause 2). Diagnosing problems Let's examine each of the probable causes as a means of diagnosing the network problem. Remember that host 1 is the local host (i.e., the host with the state problem), and host 2 is the remote host. Host 1's Network Connection Is Down This is usually simple to determine. If you can observe host 1, you can check its network interface card to see if the link light is on (I'm assuming 10 or 100BaseT connections). The link light indicates that a link pulse is being received from the device at the other end of the wire. The link pulse is received on the same pair of wires on which data is received and must be received before the network interface card will transmit any data. Since the link pulse/data is received on one pair of wires and transmitted on a different pair of wires, it's possible that host 1 is receiving a link pulse from its hub but the hub is not receiving a link pulse from the host. Hence, you also need to check the link light on the hub. And remember, just because both sides see a link pulse does not mean that the wire is capable of transmitting or receiving uncorrupted data. Provided both ends of the wire are showing a link light (or if you have to assume that they are because you cannot check), the next step is to try pinging another host on the local network. (Note that if a host has multiple NICs on the same network or subnet, routing can get tricky. Typically, local traffic will go over 1 card and traffic to other networks and subnets will go over the other card. This means that testing a network connection by pinging other hosts on the local network or subnet may not be testing the card that you are using to get to host 2.) This assumes that there is another host and that you know its IP address. If you don't know of any, check the routing table on host 1 and try pinging one of the routers. If the pings work, you know the network connection is OK, but if the pings time out, you don't know if the problem is with host 1's connection or the other host's. The more hosts that you try and fail, the more likely the problem is with host 1. You can also look at the Ethernet statistics for the network interface card. Even if no one is trying to communicate with host 1, the interface should see broadcasts. The Ethernet statistics should show an increasing number of bytes and packets received. Of course, the rate of increase will depend on the size of the network and what types of hosts are connected to it. On UNIX systems, the netstat -ni command will show the number of packets received on the interface. On Windows NT and 98, netstat -e executed from a command prompt window will show that number. In Figure 2, the UNIX system received 66 packets (458827 - 458761) within a few seconds. Note also the incoming error count. If errors are going up, we may be connected to the network but the data in the frames may be corrupted. This could happen if the cables have been damaged or installed improperly. If you can, try looking at the port statistics on the hub. If host 1 sends a ping, does the hub see an increase in packets and bytes received? No Route to Host 2 Without a route, host 1 will not know where to send the packet for forwarding to host 2. The command netstat -nr will display the routing table. On a UNIX system, there should be a row with either host 2's IP address or host 2's network or subnetwork address or the word default (see Figure 3). If host 2's IP address is shown, there should be an "H" in the flags portion of the row along with the flags "UG". If the "UG" flags do not appear or the "H" is missing from the host route, then the route is not defined correctly. On a Windows NT system, there will either be host 2's IP address or its network or subnetwork address or a destination address of 0.0.0.0, which stands for default. If the host 2's IP address is shown, the netmask will be 255.255.255.255. In Figure 3, the first blue line with the destination 192.168.13.151 represents a host route. It is followed by the default route, a network route to a remote network (10) and a route to the local network (192.168.33) Assuming there is a route, you need to make sure that the defined router (in the gateway column) is reachable. Try pinging it; if you don't get a response, it's likely that the router is down or has some kind of problem. Even if host 1 and host 2 are on the same network or subnet, there should be an entry in the routing table. In this case, the gateway address is the address of host 1. Note that the flags for this entry are just U, for up. Host 1 to Host 2 Link Down This one is tricky; how do you distinguish between a network link between host 1 and 2 being down, host 2 or its network connection being down, a routing problem on host 2, or a link between host 2 and 1 being down? Ideally, you have access to host 2 via some other link or are in communication with someone who has access to host 2. Given that, you should first verify that host 2 is not down and that its network connection is OK and that it has a route to host 1. Once all that is confirmed, you can execute the command netstat -s and look at the ICMP statistics on host 2 before and after you send several pings to it from host 1. (Note that routers and firewalls may be configured to filter ICMP echo packets (pings). If pings are failing and you think this may be a possibility, about all you can do is contact the firewall and router administrators to confirm the filtering and request that it be turned off.) If the counter for incoming echo packets is going up, you know that the pings are reaching host 2 and that there is a problem with the echo reply being sent back. If the echo counter is not going up, you know that the problem is between host 1 and host 2. If you do not have access to host 2, there is not much you can do. It's possible that the ping command on host 1 will report a network or host unreachable error (Figure 5) or a time limit exceeded error. Both of these indicate a network link problem between host 1 and host 2. A netstat -s on the host 1 system may also show these error counters increasing even if the ping command does not report them (Figure 4). Note that on many UNIX systems a statistic for an ICMP message is only displayed if the counter is non-zero. The traceroute command can also be used to determine the route that the packet is taking. This will help identify when a wrong route is being used. Note that not all UNIX systems come with a traceroute command, and on Windows NT it's the tracert command, executed from a command prompt window (Figure 6). Host 2 or Its Network Connection Down The best way to eliminate these two possibilities is to speak to someone who has access to host 2 or to get access via some other link. You then know that host 2 is working, and you can run through the "host 1's network connection is down" tests to make sure that the network link is OK. If you are on the other side of a continent and it's 3 a.m. and there is no alternate link, you've got a problem. If you can get access to the hub to which host 2 is connected, you can query the hub to see if it's getting a link pulse and receiving any packets. Unfortunately, just because the hub is getting host 2's link pulse does not mean that host 2 is getting the hub's link pulse. Also just because you are getting a link pulse does not mean that the OS is up. Some network interface cards will send a link pulse as long as they have power - even if the OS is crashed. The best you can do is try to get a login session on another host on the same network or subnet as host 2. From there, you can try pinging host 2. If the local pings work while the remote pings do not, routing is the most likely suspect. If the local pings fail, then host 2 (or at least its network connection) is down. No Route to Host 1 Again, the best approach is to gain access to host 2 via some other link and check the routing table on host 2. If that's not possible, then as in the previous case you need to gain access to a third host on the same network or subnet as host 2. Host 2 to Host 1 Link Down Without access to host 2, it is very difficult to diagnose this condition from host 1. Getting to a third host on host 2's subnet is not a guarantee that all the links between host 2 and host 1 are okay. Host 2 may be using a different router than host 3, so the network links could be different. If you have access to host 2, you can try the ping tests described in "some network link between host 1 and host 2 is down" and look at the echo statistics. Server Too Busy When host 2 receives a connection request, it will check its backlog queue, and if the queue is not full it will respond with a SYN-ACK packet. Once host 2 gets the ACK from host 1, it notifies the server application that a connection request has come in. The request stays on the queue until the server application accepts it. If there are many requests, the server may not be able to accept them fast enough and the backlog queue fills up. Once that happens, host 2 will ignore any further connection requests until there is space in the queue. The symptoms of this condition are: host 1 in a SYN_SENT state, host 2 has a number of connections in SYN_RCVD or ESTABLISHED states (they move to ESTABLISHED as soon as the ACK is received even if the server application has not accepted them yet), and new connection requests time out. This appears very similar to the next two causes. If you have access to the kernel structures on host 2 via some kind of debugging command, you may be able to check the actual backlog value. If not, you can look at system and server application performance. A heavily loaded system may not give the server application enough CPU cycles. A server application that uses all possible CPU cycles could be overloaded or stuck in a loop. Host 2 Under Attack A denial of service attack attempts to deny service to valid users of that service. These attacks make use of the connection request process described above by filling up the backlog queue with requests that cannot be completed. Basically, a connection request is sent to host 1, which appears to come from host 3. Typically, host 3 does not exist, but host 1 does not know that and will send a SYN-ACK packet to host 3. Since host 1 expects an ACK packet from host 3, it will eventually time out this half-open connection but until then it uses a slot in the backlog queue. Receiving lots of these types of requests at once will drown out the valid users. You can get some clue that this is happening by looking at the remote addresses of the connections in the SYN_RCVD state. Are they valid users of the server application? Even if they are, should they be sending a large number of requests? It's possible that the hacker who set up this attack is using a valid address but flooding host 1 with an unreasonably large number of requests (Figure 7). Network Slow or Noisy A noisy network connection or link corrupts or drops packets and requires retransmissions. The result is that connections stay in states that require acknowledgments longer than necessary. The netstat -s statistics include a count of retransmitted and duplicate received packets. The duplicate packets indicate that host 1's ACK packet to host 2 was lost, so host 2 retransmitted the packet. Keep in mind that the statistics are for all TCP connections, not just the connection you're interested in. If the network connection is causing the problem and the problem is occurring on the receiving end as well as the transmitting end, you should see input errors as well (Figure 2). If you have access to the hub to which the interface is connected, you can also check to see whether it's receiving errors. A slow network link also forces connections to stay in states longer than usual. The only good way to tell if this is happening is to ask the people in charge of the network. You can use the traceroute command (Figure 6) to determine the route and then contact the administrators of the routers. Besides giving you a list of routers, the traceroute command also gives you three times for each router. These times indicate the amount of time that the test packet took to get from host 1 to the router and back again. Three test packets are sent and each is timed separately. Treat these times gently. You'll notice that is quite possible for the round trip time to the 5th router to be faster than the round trip time to the 4th router. Many things affect the round trip times; it's even possible that the path of the test packet to or from router 5 did not pass through router 4. A Firewall or Router Between the Two Hosts is Preventing Traffic From a user's point of view, the difference between a firewall preventing traffic from flowing and a network link failure is the ease of getting it fixed. You can be pretty confident that a network link failure will be fixed. However, a firewall problem becomes an administrative and policy issue. For those familiar with the OSI layer model, this is sometimes called layer 8 - the administrative layer. Diagnosing that a firewall (instead of network link failure) is causing a problem can be difficult. It depends on what the firewall is configured to block. You can sometimes pinpoint the firewall by noting contradictions. For example, UDP traffic works, while TCP traffic does not. Or you can ftp to host 2 and get a connection, but ftp's data transfer connection requests time out. Connection is Flow Controlled This situation is not very likely but easy to diagnose. It occurs when the transmit window for the connection between host 1 and host 2 is closed, and the local application on host 1 closes the connection. Since the transmit window is closed, host 1 cannot send a FIN packet. You can tell this has happened with the netstat display on the host 1. The connection state will be FIN_WAIT_1, but the send queue column will be non-zero and will remain unchanged. The state on host 2 remains in the ESTABLISHED state since the FIN packet is never sent. Some TCP stacks will remain in this state indefinitely, others will eventually time out the connection. How is the transmit window closed? Every TCP packet header includes a window size value. This value represents the number of bytes currently in the sender's receive buffer. When a packet is acknowledged, this window size is set to the current free space in the receive buffer. If host 2 runs out of space, it sets the window size to 0. When a packet is received by host 1 from host 2 with a 0 window size, host 1 closes the transmit window for that connection. Keep in mind that each connection has its own buffers and that what happens on one connection has no bearing on other connections, even when they are between the same hosts. Once the transmit window is closed, no more data can be sent until a packet is received from host 2 with a non-zero window size value. Some TCP stacks will not send any data until the window opens to a significant value (i.e., 512 or 1024 bytes). How could host 2 run out of buffer space? Typically the application on host 2 is either hung or in a tight loop, but whatever it's doing, it's not reading TCP data, so the data sent by host 1 is just accumulating in the host 2's buffers. In this instance, shutting down the application on host 1 will not unstick the connection. You must shut down the application on host 2, or somehow get it to start reading TCP data again so that the flow control can be turned off. The Application on Host 2 Has Not Closed the Connection From host 1's perspective, the connection is sitting in a FIN_WAIT_2 state. Host 1 has closed the connection, sent its FIN packet to host 2 and gotten host 2's ACK packet for the FIN. Now host 1 is waiting for host 2 to send its FIN packet. Without checking host 2, you cannot be sure why host 1 hasn't gotten host 2's FIN packet. Its possible that host 2 has crashed or there has been a network failure, but more than likely host 2 has not closed its side of the connection. If you can check host 2 and the connection state is CLOSE_WAIT, then you can be sure that the application running on host 2 has not closed the connection. Either the application is busy or waiting or it just has a bug and is hung. For whatever reason, it has not responded to the indication sent up from the TCP stack saying that the remote side has closed its connection. Depending on what the application is doing and how it was written, it may be possible to tell it to close the connection. Or, you may just have to terminate the application. Terminating the application on host 2 should cause the TCP stack to shut down the connection, thereby sending a FIN packet back to host 1. (Note that a reset (RST) packet may be sent instead, but either way host 1 will close the connection when it receives the packet.)
The Application on Host 1 Has Not Closed This is the mirror image of the previous problem. Now host 1 is in the CLOSE_WAIT state, and host 2 will have a FIN_WAIT_2 state or no state. As described in the Web companion piece, some TCP stacks will time out a connection in FIN_WAIT_2 and release all its resources. As stated above, the application may be busy or waiting for some resource or just be hung. The hard part, is figuring out which application process is associated with the connection. Many times at least one of the ports associated with the connection is a well-known server port. If the local port is the server port then identification of the application is not a problem: it's the server application, and hopefully there is only one application running. If the remote port is the server port, then you must identify the client that was accessing the server. Good luck. How you do this or distinguish between multiple server applications will depend on your environment and your access and familiarity with kernel structures. As stated above, unless the application has a mechanism to allow you to tell it to close a connection (and it is responding to that mechanism), you will have to kill the application process to clear the CLOSE_WAIT condition. One final warning - beware of rapidly changing connections that appear stuck. Figure 8 shows the four iterations of netstat output. A casual look at the connection between 192.168.13.50 and 192.168.91.9 might lead you to think that the connection is stuck in SYN_RCVD. However, a close look shows that the port number is changing. There is nothing wrong here, just connections that are coming and going quickly. Conclusions Most people never pay any attention to the TCP connection state, until something goes wrong. Then they want to know what 1) what it means, 2) why it's in that state, and 3) what to do about it. This article has explored some of the typical TCP state problems, how to diagnose the causes, and how to resolve the problem. No set of canned answers can be applied in all cases. Armed with a better understanding of TCP states and the commands used to diagnose network problems, however, your network life should be more peaceful.
About the Author
Noah Davids works in the Customer Assistance Center of Stratus Computer Inc. He specializes in the diagnosis and correction of LAN problems. "It's the wire" followed by "it's the router" covers most problems. The trick is finding which wire or router. He can be reached at Noah_Davids@stratus.com.
|