SO_KEEPALIVE 在调用 write() 期间不起作用?
我正在开发一个套接字应用程序,它必须能够应对网络故障。
该应用程序有 2 个正在运行的线程,一个等待来自套接字的消息(read() 循环),另一个将消息发送到套接字(write() 循环)。
我目前正在尝试使用 SO_KEEPALIVE 来处理网络故障。 如果我只在 read() 上被阻止,它就可以正常工作。连接丢失(网络电缆被移除)几秒钟后,read() 将失败并显示消息“连接超时”。
但是,如果我在网络断开连接后(且在超时结束之前)尝试 wrte(),则 write() 和 read() 都将永远阻塞,不会出现错误。
这是一个将 stdin/stdout 定向到套接字的精简示例代码。它侦听端口 5656:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <pthread.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
int socket_fd;
void error(const char *msg) {
perror(msg);
exit(1);
}
//Read from stdin and write to socket
void* write_daemon (void* _arg) {
while (1) {
char c;
int ret = scanf("%c", &c);
if (ret <= 0) error("read from stdin");
int ret2 = write(socket_fd, &c, sizeof(c));
if (ret2 <= 0) error("write to socket");
}
return NULL;
}
//Read from socket and write to stdout
void* read_daemon (void* _arg) {
while (1) {
char c;
int ret = read(socket_fd, &c, sizeof(c));
if (ret <= 0) error("read from socket");
int ret2 = printf("%c", c);
if (ret2 <= 0) error("write to stdout");
}
return NULL;
}
//Enable and configure KEEPALIVE - To detect network problems quickly
void config_socket() {
int enable_no_delay = 1;
int enable_keep_alive = 1;
int keepalive_idle =1; //Very short interval. Just for testing
int keepalive_count =1;
int keepalive_interval =1;
int result;
//=> http://tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/#setsockopt
result = setsockopt(socket_fd, SOL_SOCKET, SO_KEEPALIVE, &enable_keep_alive, sizeof(int));
if (result < 0)
error("SO_KEEPALIVE");
result = setsockopt(socket_fd, SOL_TCP, TCP_KEEPIDLE, &keepalive_idle, sizeof(int));
if (result < 0)
error("TCP_KEEPIDLE");
result = setsockopt(socket_fd, SOL_TCP, TCP_KEEPINTVL, &keepalive_interval, sizeof(int));
if (result < 0)
error("TCP_KEEPINTVL");
result = setsockopt(socket_fd, SOL_TCP, TCP_KEEPCNT, &keepalive_count, sizeof(int));
if (result < 0)
error("TCP_KEEPCNT");
}
int main(int argc, char *argv[]) {
//Create Server socket, bound to port 5656
int listen_socket_fd;
int tr=1;
struct sockaddr_in serv_addr, cli_addr;
socklen_t clilen = sizeof(cli_addr);
pthread_t write_thread, read_thread;
listen_socket_fd = socket(AF_INET, SOCK_STREAM, 0);
if (listen_socket_fd < 0)
error("socket()");
if (setsockopt(listen_socket_fd,SOL_SOCKET,SO_REUSEADDR,&tr,sizeof(int)) < 0)
error("SO_REUSEADDR");
bzero((char *) &serv_addr, sizeof(serv_addr));
serv_addr.sin_family = AF_INET;
serv_addr.sin_addr.s_addr = INADDR_ANY;
serv_addr.sin_port = htons(5656);
if (bind(listen_socket_fd, (struct sockaddr *) &serv_addr, sizeof(serv_addr)) < 0)
error("bind()");
//Wait for client socket
listen(listen_socket_fd,5);
socket_fd = accept(listen_socket_fd, (struct sockaddr *) &cli_addr, &clilen);
config_socket();
pthread_create(&write_thread, NULL, write_daemon, NULL);
pthread_create(&read_thread , NULL, read_daemon , NULL);
close(listen_socket_fd);
pthread_exit(NULL);
}
要重现该错误,请使用 telnet 5656。 如果连接丢失后几秒后将退出,除非我尝试在终端中写入内容。在这种情况下,它将永远阻塞。
所以,问题是:出了什么问题?如何修复它?还有其他选择吗?
谢谢!
我尝试使用 Wireshark 来检查网络连接。如果我不调用 write(),我可以看到 TCP keep-alive 包正在发送,并且连接在几秒钟后关闭。
相反,如果我尝试 write(),它会停止发送 Keep-Alive 数据包,并开始发送 TCP 重传(对我来说似乎没问题)。问题是,每次失败后,重传之间的时间变得越来越长,而且似乎永远不会放弃并关闭套接字。
有没有办法设置最大重传次数或类似的东西? 谢谢
I'm developing a socket application, which must must be to be robust to network failures.
The application has 2 running threads, one waiting messages from the socket (a read() loop) and the other send messages to the socket (a write() loop).
I'm currently trying to use SO_KEEPALIVE to handle the network failures.
It works ok if I'm only blocked on read(). A few seconds after the connection is lost (network cable removed), read() will fail with the message 'Connection timed out'.
But, if I try to wrte() after the network is disconnected (and before the timeout ends), both write() and read() will block forever, without error.
This is a stripped sample code which directs stdin/stdout to the socket. It listens on port 5656:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <pthread.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
int socket_fd;
void error(const char *msg) {
perror(msg);
exit(1);
}
//Read from stdin and write to socket
void* write_daemon (void* _arg) {
while (1) {
char c;
int ret = scanf("%c", &c);
if (ret <= 0) error("read from stdin");
int ret2 = write(socket_fd, &c, sizeof(c));
if (ret2 <= 0) error("write to socket");
}
return NULL;
}
//Read from socket and write to stdout
void* read_daemon (void* _arg) {
while (1) {
char c;
int ret = read(socket_fd, &c, sizeof(c));
if (ret <= 0) error("read from socket");
int ret2 = printf("%c", c);
if (ret2 <= 0) error("write to stdout");
}
return NULL;
}
//Enable and configure KEEPALIVE - To detect network problems quickly
void config_socket() {
int enable_no_delay = 1;
int enable_keep_alive = 1;
int keepalive_idle =1; //Very short interval. Just for testing
int keepalive_count =1;
int keepalive_interval =1;
int result;
//=> http://tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/#setsockopt
result = setsockopt(socket_fd, SOL_SOCKET, SO_KEEPALIVE, &enable_keep_alive, sizeof(int));
if (result < 0)
error("SO_KEEPALIVE");
result = setsockopt(socket_fd, SOL_TCP, TCP_KEEPIDLE, &keepalive_idle, sizeof(int));
if (result < 0)
error("TCP_KEEPIDLE");
result = setsockopt(socket_fd, SOL_TCP, TCP_KEEPINTVL, &keepalive_interval, sizeof(int));
if (result < 0)
error("TCP_KEEPINTVL");
result = setsockopt(socket_fd, SOL_TCP, TCP_KEEPCNT, &keepalive_count, sizeof(int));
if (result < 0)
error("TCP_KEEPCNT");
}
int main(int argc, char *argv[]) {
//Create Server socket, bound to port 5656
int listen_socket_fd;
int tr=1;
struct sockaddr_in serv_addr, cli_addr;
socklen_t clilen = sizeof(cli_addr);
pthread_t write_thread, read_thread;
listen_socket_fd = socket(AF_INET, SOCK_STREAM, 0);
if (listen_socket_fd < 0)
error("socket()");
if (setsockopt(listen_socket_fd,SOL_SOCKET,SO_REUSEADDR,&tr,sizeof(int)) < 0)
error("SO_REUSEADDR");
bzero((char *) &serv_addr, sizeof(serv_addr));
serv_addr.sin_family = AF_INET;
serv_addr.sin_addr.s_addr = INADDR_ANY;
serv_addr.sin_port = htons(5656);
if (bind(listen_socket_fd, (struct sockaddr *) &serv_addr, sizeof(serv_addr)) < 0)
error("bind()");
//Wait for client socket
listen(listen_socket_fd,5);
socket_fd = accept(listen_socket_fd, (struct sockaddr *) &cli_addr, &clilen);
config_socket();
pthread_create(&write_thread, NULL, write_daemon, NULL);
pthread_create(&read_thread , NULL, read_daemon , NULL);
close(listen_socket_fd);
pthread_exit(NULL);
}
To reproduce the error, use telnet 5656.
If will exit after a couple os seconds after the connection is lost, unless I try to write something in the terminal. In this case, it will block forever.
So, the questions are: what's wrong? how to fix it? Are there other alternatives?
Thanks!
I've tried using Wireshark to inspect the network connection. If I don't call write(), I can see TCP keep-alive packages being sent and the connection is close after a few seconds.
If, instead, I try to write(), it stops sending the Keep-Alive packets, and starts sending TCP retransmissions instead (It seems OK to me). The problem is, the time between the retransmissions grows bigger and bigger after each failure, and it seems to never give-up and close the socket.
Is there a way to set the maximum number of retransmissions, or anything similar?
Thanks
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(6)
我找到了 TCP_USER_TIMEOUT 套接字选项(rfc5482),如果发送的数据在指定的时间间隔后没有被确认,它将关闭连接。
它对我来说工作得很好=)
但是,我觉得我不应该同时使用SO_KEEP_ALIVE和TCP_USER_TIMEOUT。
也许是某个地方有错误?
I have found the TCP_USER_TIMEOUT socket option (rfc5482), which closes the connection if the sent data is not ACK'ed after the specified interval.
It works fine for me =)
Yet, I feel I shouldn't have to use both SO_KEEP_ALIVE and TCP_USER_TIMEOUT.
Maybe it's bug somewhere?
不确定其他人是否会给你更好的选择,但在我参与的几个项目中,我们遇到了非常相似的情况。
对于我们来说,解决方案就是简单地将控制权掌握在自己手中,而不是依赖底层操作系统/驱动程序来告诉您连接何时终止。如果您同时控制客户端和服务器端,则可以引入自己的 ping 消息,这些消息会在客户端和服务器之间反弹。通过这种方式,您可以 a) 控制自己的连接超时 b) 轻松保存指示连接健康状况的记录。
在最新的应用程序中,我们将这些 ping 作为带内控制消息隐藏在通信库本身内,因此就实际的客户端/服务器应用程序代码而言,连接超时就起作用了。
Not sure if someone else will give you a better alternative, but in several projects I've been involved with, we've run into very similar situations.
For us the solution was to simply take control into your own hands and not rely on underlying OS/drivers to tell you when connection dies. If you control both client and server sides, you could introduce your own ping messages which bounce between the client and the server. This way you can a) control your own connection timeouts and b) easily keep a record indicating the health of the connection.
In the most recent application, we've hid these pings as in-band control messages within the communication library itself so as far as actual client/server application code was concerned, connection timeouts just worked.
TCP 保持活动状态在 RFC1122 中指定。 TCP 的“保持活动”功能不是检测短期网络中断,而是清理可能占用宝贵资源的 TCP 控制块/缓冲区。该 RFC 也是于 1989 年编写的。RFC 明确规定 TCP Keep Alives 不得每两小时发送一次以上,并且只有在没有其他流量的情况下才需要发送。如果较高级别的协议需要检测连接丢失,则较高级别的协议需要自行完成此操作。在 TCP 之上运行的 BGP 路由协议默认每 60 秒发送一次它自己的保持活动消息形式。 BGP 规范规定,如果在过去 3*keep_alive_interval 秒内没有看到新流量,则连接将被视为已失效。 OpenSSH 以 ping 和 pong 的形式实现它自己的保持活动状态。它将重试发送最多 X 个 ping,并期望在 Y 时间内得到响应(pong),否则会终止连接。 TCP 本身在遇到临时网络中断时会尽力传输数据,但它本身对于检测网络中断没有什么用处。
通常,如果您想实现保持活动状态并避免阻塞,则可以切换到非阻塞 I/O 并维护一个计时器,该计时器可与带有超时的 select()/poll() 调用一起使用。另一种选择可能是使用单独的计时器线程,甚至是使用 SIGALARM 的更粗略的方法。我建议使用 O_NONBLOCK 和 fcntl() 将套接字设置为非阻塞 I/O。然后,您可以使用 gettimeofday() 记录接收传入 I/O 的时间,并使用 select() 休眠,直到下一个“保持活动”到期或发生 I/O。
TCP Keep Alive is specified in RFC1122. The Keep Alive feature of TCP is not to detect short-term network outages, but instead to clean up TCP Control Blocks/Buffers that might be using up precious resources. That RFC was also written in 1989. The RFC explicitly states that TCP Keep Alives are not to be sent more than once every two hours, and then, it is only necessary if there was no other traffic. If a higher-level protocol needs to detect a loss of connection, it is the higher-level protocol's job to do it itself. The BGP routing protocol, which operates above TCP, sends it's own form of Keep Alive message once every 60 seconds by default. The BGP Spec says a connection is to be considered dead if there has been no new traffic seen in the last 3*keep_alive_interval seconds. OpenSSH implements it's own keep alive in the form of a ping and pong. It will retry sending up to X pings which it expects a response (pong) to within Y time or it kills the connection. TCP itself tries really hard to deliver data in the face of temporary network outages and isn't useful by itself to detect network outage.
Normally, if you want to implement a keep alive and want to avoid blocking, one would switch to non-blocking I/O and maintain a timer for which can be used with select()/poll() calls with a timeout. Another option could be to use a separate timer thread or even a more crude approach of using SIGALARM. I recommend using the O_NONBLOCK with fcntl() to set the socket to non-blocking I/O. You can then use gettimeofday() to record when incoming I/O is received and sleep with select() until either the next Keep Alive is due or I/O happens.
在断开电缆之前,您是否成功收到来自另一端的字节或 ACK?也许这与 http://lkml.indiana 中描述的行为有关。 edu/hypermail/linux/kernel/0508.2/0757.html :
您的测试用例是有问题的,因为您在已建立状态下甚至没有收到一个 ACK,因此tp->rcv_tstamp 变量无法初始化。您收到的唯一 ACK 是对连接设置 SYN 的响应,并且我们不会为该 ACK 初始化 tp->rcv_stamp。
保活时间检查绝对要求 tp->rcv_tstamp 具有有效值,并且在您处理 ESTABLISHED 状态下的 ACK 之前,它不会。
如果您通过连接成功发送或成功接收至少一个字节,从而在 ESTABLISHED 状态下处理至少一个 ACK,我想您会发现 keepalive 行为正常。
这是一种晦涩的 SO_KEEPALIVE 行为。
Did you received sucesfully a byte or an ACK from the other side before disconnecting the cable? Maybe this is related to the behaviour described in http://lkml.indiana.edu/hypermail/linux/kernel/0508.2/0757.html :
Your test case is questionable, because you do not receive even one ACK in established state, thus the tp->rcv_tstamp variable has no way to get initialized. The only ACK you receive is the one in response to the connection setup SYN, and we don't initialize tp->rcv_stamp for that ACK.
The keepalive time checks absolutely require that tp->rcv_tstamp has a valid value, and until you process an ACK in ESTABLISHED state it does not.
If you send successfully or receive successfully at least one byte over the connection, and thusly process at least one ACK in ESTABLISHED state, I think you'll find that the keepalives behave properly.
It's an obscure SO_KEEPALIVE behaviour.
在
write_daemon()
中,您将write()
的返回值存储到ret2
变量中,然后使用相反,您将永远不会捕获任何 write() 错误。In
write_daemon()
, you are storing the return value ofwrite()
into theret2
variable, but then checking for a socket error using theret
variable instead, so you will never actually catch anywrite()
errors.这是因为tcp堆栈在你无意识的情况下进行了tcp重传。
以下是解决方案。
即使您已经为应用程序套接字设置了 keepalive 选项,但如果您的应用程序不断在套接字上写入,您也无法及时检测到套接字的死连接状态。
这是因为内核 tcp 堆栈进行了 tcp 重传。
tcp_retries1和tcp_retries2是配置tcp重传超时的内核参数。
由于重传超时是通过RTT机制计算的,所以很难预测准确的时间。
你可以在 rfc793 中看到这个计算。 (3.7.数据通信)
https://www.rfc-editor.org/rfc/ rfc793.txt
每个平台都有用于 tcp 重传的内核配置。
http://linux.die.net/man/7/tcp
http://www.hpuxtips.es/?q=node/53
http://www-903.ibm.com/kr/event/ download/200804_324_swma/socket.pdf
如果需要,您应该为 tcp_retries2 设置较低的值(默认 15)尽早检测到死连接,但这不是我已经说过的精确时间。
此外,目前您无法仅为单个套接字设置这些值。这些是全局内核参数。
有一些尝试将 TCP 重传套接字选项应用于单个套接字(http://patchwork.ozlabs.org /patch/55236/),但我不认为它被应用到内核主线中。我在系统头文件中找不到这些选项定义。
作为参考,您可以通过“netstat --timers”监视您的 keepalive 套接字选项,如下所示。
https://stackoverflow.com/questions/34914278
另外,keepalive超时时,根据使用的平台不同,会遇到不同的返回事件,所以一定不要决定死连接状态仅由返回事件决定。
例如,当发生 keepalive 超时时,HP 返回 POLLERR 事件,而 AIX 仅返回 POLLIN 事件。
此时你会在recv()调用中遇到ETIMEDOUT错误。
在最近的内核版本(自 2.6.37 起)中,您可以使用 TCP_USER_TIMEOUT 选项会很好地工作。该选项可用于单个套接字。
That's because of tcp retransmission acted by tcp stack without your consciousness.
Here are solutions.
Even though you already set keepalive option to your application socket, you can't detect in time the dead connection state of the socket, in case of your app keeps writing on the socket.
That's because of tcp retransmission by the kernel tcp stack.
tcp_retries1 and tcp_retries2 are kernel parameters for configuring tcp retransmission timeout.
It's hard to predict precise time of retransmission timeout because it's calculated by RTT mechanism.
You can see this computation in rfc793. (3.7. Data Communication)
https://www.rfc-editor.org/rfc/rfc793.txt
Each platforms have kernel configurations for tcp retransmission.
http://linux.die.net/man/7/tcp
http://www.hpuxtips.es/?q=node/53
http://www-903.ibm.com/kr/event/download/200804_324_swma/socket.pdf
You should set lower value for tcp_retries2 (default 15) if you want to early detect dead connection, but it's not precise time as I already said.
In addition, currently you can't set those values only for single socket. Those are global kernel parameters.
There was some trial to apply tcp retransmission socket option for single socket(http://patchwork.ozlabs.org/patch/55236/), but I don't think it was applied into kernel mainline. I can't find those options definition in system header files.
For reference, you can monitor your keepalive socket option through 'netstat --timers' like below.
https://stackoverflow.com/questions/34914278
In addition, when keepalive timeout ocurrs, you can meet different return events depending on platforms you use, so you must not decide dead connection status only by return events.
For example, HP returns POLLERR event and AIX returns just POLLIN event when keepalive timeout occurs.
You will meet ETIMEDOUT error in recv() call at that time.
In recent kernel version(since 2.6.37), you can use TCP_USER_TIMEOUT option will work well. This option can be used for single socket.