如何释放 mongodb 连接？

发布于 2024-11-09 22:13:53 字数 7132 浏览 3 评论 0原文

我有一个短时间内读/写较高的mongo服务器。我使用了python和pymongo，当我今天早上醒来时，我发现无法与mongod master建立连接，因为它的连接数达到了19992，

即使我停止了所有程序，它也是一个非常可怕的数字，连接数似乎没有变化

insert  query update delete getmore command flushes mapped  vsize    res faults locked % idx miss %     qr|qw   ar|aw  netIn netOut  conn repl       time

localhost:27417      0      0      0      0       2       1       0   624m  2.87g   287m      0        0          0       0|0     7|0   162b     1k 19992    M   10:36:16

> db.serverStatus(1)
{
        "host" : "42yl:27417",
        "version" : "1.8.1",
        "process" : "mongod",
        "uptime" : 71732,
        "uptimeEstimate" : 71470,
        "localTime" : ISODate("2011-05-26T03:02:48.301Z"),
        "globalLock" : {
                "totalTime" : 71732232290,
                "lockTime" : 149471421,
                "ratio" : 0.002083741384148133,
                "currentQueue" : {
                        "total" : 0,
                        "readers" : 0,
                        "writers" : 0
                },
                "activeClients" : {
                        "total" : 7,
                        "readers" : 7,
                        "writers" : 0
                }
        },
        "mem" : {
                "bits" : 32,
                "resident" : 258,
                "virtual" : 910,
                "supported" : true,
                "mapped" : 624
        },
        "connections" : {
                "current" : 19792,
                "available" : 208
        },
        "extra_info" : {
                "note" : "fields vary by platform",
                "heap_usage_bytes" : 562688,
                "page_faults" : 965
        },
        "indexCounters" : {
                "btree" : {
                        "accesses" : 12789,
                        "hits" : 12789,
                        "misses" : 0,
                        "resets" : 0,
                        "missRatio" : 0
                }
        },
        "backgroundFlushing" : {
                "flushes" : 1195,
                "total_ms" : 848633,
                "average_ms" : 710.1531380753138,
                "last_ms" : 101,
                "last_finished" : ISODate("2011-05-26T03:02:18.691Z")
        },
        "cursors" : {
                "totalOpen" : 7,
                "clientCursors_size" : 7,
                "timedOut" : 0
        },
        "network" : {
                "bytesIn" : 685742402,
                "bytesOut" : 2742190274,
                "numRequests" : 3800041
        },
        "repl" : {
                "ismaster" : true
        },
        "opcounters" : {
                "insert" : 104225,
                "query" : 9,
                "update" : 925044,
                "delete" : 45734,
                "getmore" : 1642979,
                "command" : 1119290
        },
        "asserts" : {
                "regular" : 0,
                "warning" : 56,
                "msg" : 0,
                "user" : 0,
                "rollovers" : 0
        },
        "writeBacksQueued" : false,
        "ok" : 1
}

我检查了套接字连接

Active Internet connections (servers and established) 
Proto Recv-Q Send-Q Local Address Foreign Address State 
tcp 0 0 0.0.0.0:60000 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27424 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28417 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28418 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28419 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28420 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28421 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28422 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28423 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28424 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:38422 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:58422 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27417 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27418 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27419 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27420 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27421 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27422 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27423 0.0.0.0:* LISTEN 
tcp 0 0 222.73.61.42:27420 222.73.61.43:38249 ESTABLISHED 
tcp 0 0 127.0.0.1:27417 127.0.0.1:56699 ESTABLISHED 
tcp 0 0 127.0.0.1:27417 127.0.0.1:56698 ESTABLISHED 
tcp 0 0 127.0.0.1:27417 127.0.0.1:56697 ESTABLISHED 
tcp 0 0 127.0.0.1:27417 127.0.0.1:56696 ESTABLISHED 
tcp 0 0 127.0.0.1:27417 127.0.0.1:56702 ESTABLISHED 
tcp 0 0 127.0.0.1:27417 127.0.0.1:56701 ESTABLISHED 
tcp 0 0 127.0.0.1:27417 127.0.0.1:56700 ESTABLISHED 
tcp 0 0 222.73.61.42:27422 222.73.61.43:33616 ESTABLISHED 
tcp 0 0 222.73.61.42:27417 222.73.61.43:60218 ESTABLISHED 
tcp 0 0 222.73.61.42:27423 222.73.61.43:33035 ESTABLISHED 
tcp 0 3324 222.73.61.42:58422 119.85.195.88:54295 ESTABLISHED 
tcp 0 0 222.73.61.42:27424 222.73.61.43:55825 ESTABLISHED 
tcp 0 0 222.73.61.42:54279 222.215.136.8:80 ESTABLISHED 
tcp 0 0 222.73.61.42:27418 222.73.61.43:37093 ESTABLISHED 
tcp 0 0 222.73.61.42:27419 222.73.61.43:38346 ESTABLISHED 
tcp 0 0 127.0.0.1:56702 127.0.0.1:27417 ESTABLISHED 
tcp 0 0 127.0.0.1:56701 127.0.0.1:27417 ESTABLISHED 
tcp 0 0 127.0.0.1:56700 127.0.0.1:27417 ESTABLISHED 
tcp 0 0 127.0.0.1:56699 127.0.0.1:27417 ESTABLISHED 
tcp 0 0 127.0.0.1:56698 127.0.0.1:27417 ESTABLISHED 
tcp 0 0 127.0.0.1:56697 127.0.0.1:27417 ESTABLISHED 
tcp 0 0 127.0.0.1:56696 127.0.0.1:27417 ESTABLISHED 
tcp 0 0 222.73.61.42:27421 222.73.61.43:39843 ESTABLISHED 
udp 0 0 0.0.0.0:48514 0.0.0.0:* 
udp 0 0 222.73.61.42:50721 61.128.128.68:53 ESTABLISHED 
udp 0 0 127.0.0.1:52274 127.0.0.1:52274 ESTABLISHED 
Active UNIX domain sockets (servers and established) 
Proto RefCnt Flags Type State I-Node Path 
unix 2 [ ACC ] STREAM LISTENING 9081684 /var/run/nscd/socket 
unix 2 [ ACC ] STREAM LISTENING 18011686 /tmp/mongodb-27417.sock 
unix 2 [ ACC ] STREAM LISTENING 18011689 /tmp/mongodb-27422.sock 
unix 2 [ ACC ] STREAM LISTENING 18011691 /tmp/mongodb-28422.sock 
unix 2 [ ACC ] STREAM LISTENING 18011697 /tmp/mongodb-27420.sock 
unix 2 [ ACC ] STREAM LISTENING 18011702 /tmp/mongodb-28417.sock 
unix 2 [ ACC ] STREAM LISTENING 18011693 /tmp/mongodb-27421.sock 
unix 2 [ ACC ] STREAM LISTENING 18011695 /tmp/mongodb-28421.sock 
unix 2 [ ACC ] STREAM LISTENING 18011699 /tmp/mongodb-28420.sock 
unix 2 [ ACC ] STREAM LISTENING 18011710 /tmp/mongodb-27419.sock 
unix 2 [ ACC ] STREAM LISTENING 18011713 /tmp/mongodb-28419.sock 
unix 2 [ ACC ] STREAM LISTENING 18011716 /tmp/mongodb-27418.sock 
unix 2 [ ACC ] STREAM LISTENING 18011719 /tmp/mongodb-28418.sock 
unix 2 [ ACC ] STREAM LISTENING 18011722 /tmp/mongodb-27424.sock 
unix 2 [ ACC ] STREAM LISTENING 18011725 /tmp/mongodb-28424.sock 
unix 2 [ ACC ] STREAM LISTENING 18011728 /tmp/mongodb-27423.sock 
unix 2 [ ACC ] STREAM LISTENING 18011731 /tmp/mongodb-28423.sock 
unix 2 [ ACC ] STREAM LISTENING 12771288 /tmp/.s.PGSQL.5432 
unix 2 [ ] DGRAM 3651 @/org/kernel/udev/udevd 
unix 5 [ ] DGRAM 16472048 /dev/log 
unix 2 [ ] STREAM CONNECTED 18706425 /var/run/nscd/socket 
unix 2 [ ] DGRAM 16792651 
unix 2 [ ] DGRAM 16472057 
unix 2 [ ] DGRAM 16472052

原文

I have a mongo server with high read/write in a short time. I used python and pymongo, when I wake up this morning I found no connection can make to mongod master cause it's connections reached 19992, its a pretty much scary number

even I stopped all the program, the connection number seems no change

insert  query update delete getmore command flushes mapped  vsize    res faults locked % idx miss %     qr|qw   ar|aw  netIn netOut  conn repl       time

localhost:27417      0      0      0      0       2       1       0   624m  2.87g   287m      0        0          0       0|0     7|0   162b     1k 19992    M   10:36:16

> db.serverStatus(1)
{
        "host" : "42yl:27417",
        "version" : "1.8.1",
        "process" : "mongod",
        "uptime" : 71732,
        "uptimeEstimate" : 71470,
        "localTime" : ISODate("2011-05-26T03:02:48.301Z"),
        "globalLock" : {
                "totalTime" : 71732232290,
                "lockTime" : 149471421,
                "ratio" : 0.002083741384148133,
                "currentQueue" : {
                        "total" : 0,
                        "readers" : 0,
                        "writers" : 0
                },
                "activeClients" : {
                        "total" : 7,
                        "readers" : 7,
                        "writers" : 0
                }
        },
        "mem" : {
                "bits" : 32,
                "resident" : 258,
                "virtual" : 910,
                "supported" : true,
                "mapped" : 624
        },
        "connections" : {
                "current" : 19792,
                "available" : 208
        },
        "extra_info" : {
                "note" : "fields vary by platform",
                "heap_usage_bytes" : 562688,
                "page_faults" : 965
        },
        "indexCounters" : {
                "btree" : {
                        "accesses" : 12789,
                        "hits" : 12789,
                        "misses" : 0,
                        "resets" : 0,
                        "missRatio" : 0
                }
        },
        "backgroundFlushing" : {
                "flushes" : 1195,
                "total_ms" : 848633,
                "average_ms" : 710.1531380753138,
                "last_ms" : 101,
                "last_finished" : ISODate("2011-05-26T03:02:18.691Z")
        },
        "cursors" : {
                "totalOpen" : 7,
                "clientCursors_size" : 7,
                "timedOut" : 0
        },
        "network" : {
                "bytesIn" : 685742402,
                "bytesOut" : 2742190274,
                "numRequests" : 3800041
        },
        "repl" : {
                "ismaster" : true
        },
        "opcounters" : {
                "insert" : 104225,
                "query" : 9,
                "update" : 925044,
                "delete" : 45734,
                "getmore" : 1642979,
                "command" : 1119290
        },
        "asserts" : {
                "regular" : 0,
                "warning" : 56,
                "msg" : 0,
                "user" : 0,
                "rollovers" : 0
        },
        "writeBacksQueued" : false,
        "ok" : 1
}

I checked the socket connections

Active Internet connections (servers and established) 
Proto Recv-Q Send-Q Local Address Foreign Address State 
tcp 0 0 0.0.0.0:60000 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27424 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28417 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28418 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28419 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28420 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28421 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28422 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28423 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:28424 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:38422 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:58422 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27417 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27418 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27419 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27420 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27421 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27422 0.0.0.0:* LISTEN 
tcp 0 0 0.0.0.0:27423 0.0.0.0:* LISTEN 
tcp 0 0 222.73.61.42:27420 222.73.61.43:38249 ESTABLISHED 
tcp 0 0 127.0.0.1:27417 127.0.0.1:56699 ESTABLISHED 
tcp 0 0 127.0.0.1:27417 127.0.0.1:56698 ESTABLISHED 
tcp 0 0 127.0.0.1:27417 127.0.0.1:56697 ESTABLISHED 
tcp 0 0 127.0.0.1:27417 127.0.0.1:56696 ESTABLISHED 
tcp 0 0 127.0.0.1:27417 127.0.0.1:56702 ESTABLISHED 
tcp 0 0 127.0.0.1:27417 127.0.0.1:56701 ESTABLISHED 
tcp 0 0 127.0.0.1:27417 127.0.0.1:56700 ESTABLISHED 
tcp 0 0 222.73.61.42:27422 222.73.61.43:33616 ESTABLISHED 
tcp 0 0 222.73.61.42:27417 222.73.61.43:60218 ESTABLISHED 
tcp 0 0 222.73.61.42:27423 222.73.61.43:33035 ESTABLISHED 
tcp 0 3324 222.73.61.42:58422 119.85.195.88:54295 ESTABLISHED 
tcp 0 0 222.73.61.42:27424 222.73.61.43:55825 ESTABLISHED 
tcp 0 0 222.73.61.42:54279 222.215.136.8:80 ESTABLISHED 
tcp 0 0 222.73.61.42:27418 222.73.61.43:37093 ESTABLISHED 
tcp 0 0 222.73.61.42:27419 222.73.61.43:38346 ESTABLISHED 
tcp 0 0 127.0.0.1:56702 127.0.0.1:27417 ESTABLISHED 
tcp 0 0 127.0.0.1:56701 127.0.0.1:27417 ESTABLISHED 
tcp 0 0 127.0.0.1:56700 127.0.0.1:27417 ESTABLISHED 
tcp 0 0 127.0.0.1:56699 127.0.0.1:27417 ESTABLISHED 
tcp 0 0 127.0.0.1:56698 127.0.0.1:27417 ESTABLISHED 
tcp 0 0 127.0.0.1:56697 127.0.0.1:27417 ESTABLISHED 
tcp 0 0 127.0.0.1:56696 127.0.0.1:27417 ESTABLISHED 
tcp 0 0 222.73.61.42:27421 222.73.61.43:39843 ESTABLISHED 
udp 0 0 0.0.0.0:48514 0.0.0.0:* 
udp 0 0 222.73.61.42:50721 61.128.128.68:53 ESTABLISHED 
udp 0 0 127.0.0.1:52274 127.0.0.1:52274 ESTABLISHED 
Active UNIX domain sockets (servers and established) 
Proto RefCnt Flags Type State I-Node Path 
unix 2 [ ACC ] STREAM LISTENING 9081684 /var/run/nscd/socket 
unix 2 [ ACC ] STREAM LISTENING 18011686 /tmp/mongodb-27417.sock 
unix 2 [ ACC ] STREAM LISTENING 18011689 /tmp/mongodb-27422.sock 
unix 2 [ ACC ] STREAM LISTENING 18011691 /tmp/mongodb-28422.sock 
unix 2 [ ACC ] STREAM LISTENING 18011697 /tmp/mongodb-27420.sock 
unix 2 [ ACC ] STREAM LISTENING 18011702 /tmp/mongodb-28417.sock 
unix 2 [ ACC ] STREAM LISTENING 18011693 /tmp/mongodb-27421.sock 
unix 2 [ ACC ] STREAM LISTENING 18011695 /tmp/mongodb-28421.sock 
unix 2 [ ACC ] STREAM LISTENING 18011699 /tmp/mongodb-28420.sock 
unix 2 [ ACC ] STREAM LISTENING 18011710 /tmp/mongodb-27419.sock 
unix 2 [ ACC ] STREAM LISTENING 18011713 /tmp/mongodb-28419.sock 
unix 2 [ ACC ] STREAM LISTENING 18011716 /tmp/mongodb-27418.sock 
unix 2 [ ACC ] STREAM LISTENING 18011719 /tmp/mongodb-28418.sock 
unix 2 [ ACC ] STREAM LISTENING 18011722 /tmp/mongodb-27424.sock 
unix 2 [ ACC ] STREAM LISTENING 18011725 /tmp/mongodb-28424.sock 
unix 2 [ ACC ] STREAM LISTENING 18011728 /tmp/mongodb-27423.sock 
unix 2 [ ACC ] STREAM LISTENING 18011731 /tmp/mongodb-28423.sock 
unix 2 [ ACC ] STREAM LISTENING 12771288 /tmp/.s.PGSQL.5432 
unix 2 [ ] DGRAM 3651 @/org/kernel/udev/udevd 
unix 5 [ ] DGRAM 16472048 /dev/log 
unix 2 [ ] STREAM CONNECTED 18706425 /var/run/nscd/socket 
unix 2 [ ] DGRAM 16792651 
unix 2 [ ] DGRAM 16472057 
unix 2 [ ] DGRAM 16472052

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

断爱 2024-11-16 22:13:53

我们之前也遇到过同样的问题，我认为 Linux 系统的 TCP_KEEPALIVE_TIME 调整存在问题，它指定了给定 tcp 连接的超时时间。对于您的情况，您在短时间内具有较高的读/写能力，这可以为 tcp 连接设置更低的超时配置。

通过使用以下命令，可能会对您有所帮助：

Checking current config:
[root@monitor-hk-1 ~]# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200

Changing config:
echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time

以下是我公司的一些内部备忘录：

After some researching for 30,Jul's failure of mongo on testbed, ip 118.26, I found something familiar with production:
1.  Total Connections reach 970+ which mongo start to block all the incoming connections.
2.  Check netstat, only nearly 100 something connections are kept listening or waiting.
3.  Check iostat, and cpu, memory, not a high utilization rate, several percent around 10.
Log:
Tue Jul 30 10:19:03.575 [initandlisten] connection accepted from 192.168.118.18:52858 #261673 (974 connections now open)
Tue Jul 30 10:19:03.575 [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
Tue Jul 30 10:19:03.575 [initandlisten] can't create new thread, closing connection
After checking official manual, I found that our tcp keep-alive value may be too high:
[root@monitor-hk-1 ~]# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
I suggest to change the such value to 300 for a short keep-alive for tcp connections.
Can be done with following command:
echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time
Notes: This can only change this value temporarily. If we reboot the system, it would be reset to default value. If we want to make persistent change, please reference at the Linux operation guide.

希望这可以帮助您。更多信息：

Linux TCP I/O 调整：

TCP 调优 MongoDB 官方文档

We have come across same problem before, I think there is a problem with linux system tuning for TCP_KEEPALIVE_TIME, which specify a timeout for a given tcp connection. For your case, you have a high read/write in a short time, which can make a even lower timeout config for tcp connection.

By using following command, it may help you:

Checking current config:
[root@monitor-hk-1 ~]# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200

Changing config:
echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time

Below are some internal memo by my company:

After some researching for 30,Jul's failure of mongo on testbed, ip 118.26, I found something familiar with production:
1.  Total Connections reach 970+ which mongo start to block all the incoming connections.
2.  Check netstat, only nearly 100 something connections are kept listening or waiting.
3.  Check iostat, and cpu, memory, not a high utilization rate, several percent around 10.
Log:
Tue Jul 30 10:19:03.575 [initandlisten] connection accepted from 192.168.118.18:52858 #261673 (974 connections now open)
Tue Jul 30 10:19:03.575 [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
Tue Jul 30 10:19:03.575 [initandlisten] can't create new thread, closing connection
After checking official manual, I found that our tcp keep-alive value may be too high:
[root@monitor-hk-1 ~]# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
I suggest to change the such value to 300 for a short keep-alive for tcp connections.
Can be done with following command:
echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time
Notes: This can only change this value temporarily. If we reboot the system, it would be reset to default value. If we want to make persistent change, please reference at the Linux operation guide.

Hope this can help you. More Info:

Linux TCP I/O Tuning:

MongoDB Official Doc for TCP Tuning

回复收藏 0 原文

羁拥 2024-11-16 22:13:53

我为你搜索了一下，在 mongodb 官方网站上找到了这个： mongodb-docs

尝试使用：

waitQueueTimeoutMS=ms

"The amount of time a thread can wait for a connection to become available before timing out. This only applies if the connection pool has reached the maximum size and all connections are already in use."

或者：

waitQueueMultiple=n

"The drivers impose a limit on how many threads can be waiting for a connection at the same time. This limit is expressed as a multiple of maxPoolSize."

i googled a bit for you and found on mongodb official site this: mongodb-docs

try using:

waitQueueTimeoutMS=ms

"The amount of time a thread can wait for a connection to become available before timing out. This only applies if the connection pool has reached the maximum size and all connections are already in use."

or:

waitQueueMultiple=n

"The drivers impose a limit on how many threads can be waiting for a connection at the same time. This limit is expressed as a multiple of maxPoolSize."

回复收藏 0 原文