Writing to a closed, local TCP socket not failing

by

This is what the Linux man page says about write and EPIPE:

   EPIPE  fd is connected to a pipe or socket whose reading end is closed.
          When this happens the writing process will also receive  a  SIG-
          PIPE  signal.  (Thus, the write return value is seen only if the
          program catches, blocks or ignores this signal.)

When Linux is using a pipe or a socketpair, it can and will check the reading end of the pair, as these two programs would demonstrate:

void test_socketpair () {
    int pair[2];
    socketpair(PF_LOCAL, SOCK_STREAM, 0, pair);
    close(pair[0]);
    if (send(pair[1], "a", 1, MSG_NOSIGNAL) < 0) perror("send");
}

void test_pipe () {
    int pair[2];
    pipe(pair);
    close(pair[0]);
    signal(SIGPIPE, SIG_IGN);
    if (write(pair[1], "a", 1) < 0) perror("send");
    signal(SIGPIPE, SIG_DFL);
}

Linux is able to do so, because the kernel has innate knowledge about the other end of the pipe or connected pair. However, when using connect, the state about the socket is maintained by the protocol stack. Your test demonstrates this behavior, but below is a program that does it all in a single thread, similar to the two tests above:

int a_sock = socket(PF_INET, SOCK_STREAM, 0);
const int one = 1;
setsockopt(a_sock, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
struct sockaddr_in a_sin = {0};
a_sin.sin_port = htons(4321);
a_sin.sin_family = AF_INET;
a_sin.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
bind(a_sock, (struct sockaddr *)&a_sin, sizeof(a_sin));
listen(a_sock, 1);
int c_sock = socket(PF_INET, SOCK_STREAM, 0);
fcntl(c_sock, F_SETFL, fcntl(c_sock, F_GETFL, 0)|O_NONBLOCK);
connect(c_sock, (struct sockaddr *)&a_sin, sizeof(a_sin));
fcntl(c_sock, F_SETFL, fcntl(c_sock, F_GETFL, 0)&~O_NONBLOCK);
struct sockaddr_in s_sin = {0};
socklen_t s_sinlen = sizeof(s_sin);
int s_sock = accept(a_sock, (struct sockaddr *)&s_sin, &s_sinlen);
struct pollfd c_pfd = { c_sock, POLLOUT, 0 };
if (poll(&c_pfd, 1, -1) != 1) perror("poll");
int erropt = -1;
socklen_t errlen = sizeof(erropt);
getsockopt(c_sock, SOL_SOCKET, SO_ERROR, &erropt, &errlen);
if (erropt != 0) { errno = erropt; perror("connect"); }
puts("P|Recv-Q|Send-Q|Local Address|Foreign Address|State|");
char cmd[256];
snprintf(cmd, sizeof(cmd), "netstat -tn | grep ':%hu ' | sed 's/  */|/g'",
         ntohs(s_sin.sin_port));
puts("before close on client"); system(cmd);
close(c_sock);
puts("after close on client"); system(cmd);
if (send(s_sock, "a", 1, MSG_NOSIGNAL) < 0) perror("send");
puts("after send on server"); system(cmd);
puts("end of test");
sleep(5);

If you run the above program, you will get output similar to this:

P|Recv-Q|Send-Q|Local Address|Foreign Address|State|
before close on client
tcp|0|0|127.0.0.1:35790|127.0.0.1:4321|ESTABLISHED|
tcp|0|0|127.0.0.1:4321|127.0.0.1:35790|ESTABLISHED|
after close on client
tcp|0|0|127.0.0.1:35790|127.0.0.1:4321|FIN_WAIT2|
tcp|1|0|127.0.0.1:4321|127.0.0.1:35790|CLOSE_WAIT|
after send on server
end of test

This shows it took one write for the sockets to transition to the CLOSED states. To find out why this occurred, a TCP dump of the transaction can be useful:

16:45:28 127.0.0.1 > 127.0.0.1
 .809578 IP .35790 > .4321: S 1062313174:1062313174(0) win 32792 <mss 16396,sackOK,timestamp 3915671437 0,nop,wscale 7>
 .809715 IP .4321 > .35790: S 1068622806:1068622806(0) ack 1062313175 win 32768 <mss 16396,sackOK,timestamp 3915671437 3915671437,nop,wscale 7>
 .809583 IP .35790 > .4321: . ack 1 win 257 <nop,nop,timestamp 3915671437 3915671437>
 .840364 IP .35790 > .4321: F 1:1(0) ack 1 win 257 <nop,nop,timestamp 3915671468 3915671437>
 .841170 IP .4321 > .35790: . ack 2 win 256 <nop,nop,timestamp 3915671469 3915671468>
 .865792 IP .4321 > .35790: P 1:2(1) ack 2 win 256 <nop,nop,timestamp 3915671493 3915671468>
 .865809 IP .35790 > .4321: R 1062313176:1062313176(0) win 0

The first three lines represent the 3-way handshake. The fourth line is the FIN packet the client sends to the server, and the fifth line is the ACK from the server, acknowledging receipt. The sixth line is the server trying to send 1 byte of data to the client with the PUSH flag set. The final line is the client RESET packet, which causes the TCP state for the connection to be freed, and is why the third netstat command did not result in any output in the test above.

So, the server doesn’t know the client will reset the connection until after it tries to send some data to it. The reason for the reset is because the client called close, instead of something else.

The server cannot know for certain what system call the client has actually issued, it can only follow the TCP state. For example, we could replace the close call with a call to shutdown instead.

//close(c_sock);
shutdown(c_sock, SHUT_WR);

The difference between shutdown and close is that shutdown only governs the state of the connection, while close also governs the state of the file descriptor that represents the socket. A shutdown will not close a socket.

The output will be different with the shutdown change:

P|Recv-Q|Send-Q|Local Address|Foreign Address|State|
before close on client
tcp|0|0|127.0.0.1:4321|127.0.0.1:56355|ESTABLISHED|
tcp|0|0|127.0.0.1:56355|127.0.0.1:4321|ESTABLISHED|
after close on client
tcp|1|0|127.0.0.1:4321|127.0.0.1:56355|CLOSE_WAIT|
tcp|0|0|127.0.0.1:56355|127.0.0.1:4321|FIN_WAIT2|
after send on server
tcp|1|0|127.0.0.1:4321|127.0.0.1:56355|CLOSE_WAIT|
tcp|1|0|127.0.0.1:56355|127.0.0.1:4321|FIN_WAIT2|
end of test

The TCP dump will show also show something different:

17:09:18 127.0.0.1 > 127.0.0.1
 .722520 IP .56355 > .4321: S 2558095134:2558095134(0) win 32792 <mss 16396,sackOK,timestamp 3917101399 0,nop,wscale 7>
 .722594 IP .4321 > .56355: S 2563862019:2563862019(0) ack 2558095135 win 32768 <mss 16396,sackOK,timestamp 3917101399 3917101399,nop,wscale 7>
 .722615 IP .56355 > .4321: . ack 1 win 257 <nop,nop,timestamp 3917101399 3917101399>
 .748838 IP .56355 > .4321: F 1:1(0) ack 1 win 257 <nop,nop,timestamp 3917101425 3917101399>
 .748956 IP .4321 > .56355: . ack 2 win 256 <nop,nop,timestamp 3917101426 3917101425>
 .764894 IP .4321 > .56355: P 1:2(1) ack 2 win 256 <nop,nop,timestamp 3917101442 3917101425>
 .764903 IP .56355 > .4321: . ack 2 win 257 <nop,nop,timestamp 3917101442 3917101442>
17:09:23
 .786921 IP .56355 > .4321: R 2:2(0) ack 2 win 257 <nop,nop,timestamp 3917106464 3917101442>

Notice the reset at the end comes 5 seconds after the last ACK packet. This reset is due to the program shutting down without properly closing the sockets. It is the ACK packet from the client to the server before the reset that is different than before. This is the indication that the client did not use close. In TCP, the FIN indication is really an indication that there is no more data to be sent. But since a TCP connection is bi-directional, the server that receives the FIN assumes the client can still receive data. In the case above, the client in fact does accept the data.

Whether the client uses close or SHUT_WR to issue a FIN, in either case you can detect the arrival of the FIN by polling on the server socket for a readable event. If after calling read the result is 0, then you know the FIN has arrived, and you can do what you wish with that information.

struct pollfd s_pfd = { s_sock, POLLIN|POLLOUT, 0 };
if (poll(&s_pfd, 1, -1) != 1) perror("poll");
if (s_pfd.revents|POLLIN) {
    char c;
    int r;
    while ((r = recv(s_sock, &c, 1, MSG_DONTWAIT)) == 1) {}
    if (r == 0) { /*...FIN received...*/ }
    else if (errno == EAGAIN) { /*...no more data to read for now...*/ }
    else { /*...some other error...*/ perror("recv"); }
}

Now, it is trivially true that if the server issues SHUT_WR with shutdown before it tries to do a write, it will in fact get the EPIPE error.

shutdown(s_sock, SHUT_WR);
if (send(s_sock, "a", 1, MSG_NOSIGNAL) < 0) perror("send");

If, instead, you want the client to indicate an immediate reset to the server, you can force that to happen on most TCP stacks by enabling the linger option, with a linger timeout of 0 prior to calling close.

struct linger lo = { 1, 0 };
setsockopt(c_sock, SOL_SOCKET, SO_LINGER, &lo, sizeof(lo));
close(c_sock);

With the above change, the output of the program becomes:

P|Recv-Q|Send-Q|Local Address|Foreign Address|State|
before close on client
tcp|0|0|127.0.0.1:35043|127.0.0.1:4321|ESTABLISHED|
tcp|0|0|127.0.0.1:4321|127.0.0.1:35043|ESTABLISHED|
after close on client
send: Connection reset by peer
after send on server
end of test

The send gets an immediate error in this case, but it is not EPIPE, it is ECONNRESET. The TCP dump reflects this as well:

17:44:21 127.0.0.1 > 127.0.0.1
 .662163 IP .35043 > .4321: S 498617888:498617888(0) win 32792 <mss 16396,sackOK,timestamp 3919204411 0,nop,wscale 7>
 .662176 IP .4321 > .35043: S 497680435:497680435(0) ack 498617889 win 32768 <mss 16396,sackOK,timestamp 3919204411 3919204411,nop,wscale 7>
 .662184 IP .35043 > .4321: . ack 1 win 257 <nop,nop,timestamp 3919204411 3919204411>
 .691207 IP .35043 > .4321: R 1:1(0) ack 1 win 257 <nop,nop,timestamp 3919204440 3919204411>

The RESET packet comes right after the 3-way handshake completes. However, using this option has its dangers. If the other end has unread data in the socket buffer when the RESET arrives, that data will be purged, causing the data to be lost. Forcing a RESET to be sent is usually used in request/response style protocols. The sender of the request can know there can be no data lost when it receives the entire response to its request. Then, it is safe for the request sender to force a RESET to be sent on the connection.

More Related Contents:

  1. Connection refused – tcp socket in linux [closed]
  2. Linux: is there a read or recv from socket with timeout?
  3. What’s the correct answer about getservbyname?
  4. How to set socket timeout in C when making multiple connections?
  5. building a .so that is also an executable
  6. listen() ignores the backlog argument?
  7. Get IP address of an interface on Linux
  8. Sending file descriptor by Linux socket
  9. What are the use cases of SO_REUSEADDR?
  10. How to detect a TCP socket disconnection (with C Berkeley socket)
  11. Why do we cast sockaddr_in to sockaddr when calling bind()?
  12. Error: Address already in use while binding socket with address but the port number is shown free by `netstat`
  13. Turn a simple socket into an SSL socket
  14. How to compile my own glibc C standard library from source and use it?
  15. Bind failed: Address already in use
  16. Receiving multiple multicast feeds on the same port – C, Linux
  17. Problems with SO_BINDTODEVICE Linux socket option
  18. Force free() to return malloc memory back to OS
  19. Finding out if a message over tcp was delivered
  20. close() is not closing socket properly
  21. Cannot assign requested address – possible causes?
  22. Is this possible to customize printf?
  23. how to bind raw socket to specific interface
  24. bind socket to network interface
  25. What can cause a “Resource temporarily unavailable” on sock send() command
  26. With C TCP sockets, can ‘send’ return zero?
  27. Sockets On Same Machine For Windows and Linux
  28. passing a struct over TCP (SOCK_STREAM) socket in C
  29. multiple threads able to get flock at the same time
  30. What is the difference between AF_INET and PF_INET in socket programming?

Leave a Comment