Mysql InnoDB 内存泄漏

发布于 2025-01-10 16:49:35 字数 6757 浏览 0 评论 0原文

我在 x86_64 上安装了适用于 Linux 的 MySQL Ver 8.0.26,它分配的内存比我预期的多很多。我想了解它去哪里解决根本原因。

Linux 服务器总共有 370GB RAM,但它不是数据库服务器,只是运行应用程序的地方。我现在的数据库大小约为 5GB,总共不到 25 个表。

在尝试找到内存泄漏后,我偶然发现了 Mysql Open 表,它似乎与内存使用量的增加相关。 (下图是使用网络数据超过 5 天的时间)。

随着时间的推移内存使用情况: 输入图片此处描述

打开的桌子数量随着时间的推移而增长。 输入图片这里的描述

这是我在运行 show engine innodb status; 后在 my.cnf 文件中所做的更改

[mysqld]
max_connections = 512
innodb_log_file_size = 1024M
innodb_buffer_pool_size = 16G
innodb_print_all_deadlocks = 1
wait_timeout = 31536000
interactive_timeout = 31536000
innodb_autoinc_lock_mode=2
max_allowed_packet=512M
default_storage_engine = InnoDB

; 这是我得到的结果。

--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
 ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 9626
877 OS file reads, 109038826 OS file writes, 78122941 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 238.31 writes/s, 181.31 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
 insert 0, delete mark 0, delete 0
discarded operations:
 insert 0, delete mark 0, delete 0
Hash table size 4425293, node heap has 1237 buffer(s)
Hash table size 4425293, node heap has 3907 buffer(s)
Hash table size 4425293, node heap has 18904 buffer(s)
Hash table size 4425293, node heap has 10211 buffer(s)
Hash table size 4425293, node heap has 135 buffer(s)
Hash table size 4425293, node heap has 2833 buffer(s)
Hash table size 4425293, node heap has 1450 buffer(s)
Hash table size 4425293, node heap has 469 buffer(s)
3181.09 hash searches/s, 6282.52 non-hash searches/s
---
LOG
---
Log sequence number          48666558274
Log buffer assigned up to    48666558274
Log buffer completed up to   48666558274
Log written up to            48666558274
Log flushed up to            48666558274
Added dirty pages up to      48666558274
Pages flushed up to          48496882991
Last checkpoint at           48496882991
98154440 log i/o's done, 222.11 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 17541103616
Dictionary memory allocated 702518
Buffer pool size   1048480
Free buffers       482537
Database pages     526797
Old database pages 194298
Modified db pages  24379
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 2328320, not young 21676
0.00 youngs/s, 0.00 non-youngs/s
Pages read 854, created 539313, written 7072261
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 526797, unzip_LRU len: 0
I/O sum[3200]:cur[64], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size   131058
Free buffers       60244
Database pages     65964
Old database pages 24329
Modified db pages  3277
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 957873, not young 10993
0.00 youngs/s, 0.00 non-youngs/s
Pages read 175, created 71085, written 1399422
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 65964, unzip_LRU len: 0
I/O sum[400]:cur[8], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size   131060
Free buffers       60410
Database pages     65754
Old database pages 24252
Modified db pages  2831
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 787777, not young 8988
0.00 youngs/s, 0.00 non-youngs/s
Pages read 130, created 69632, written 1222975
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 65754, unzip_LRU len: 0
I/O sum[400]:cur[8], unzip sum[0]:cur[0]

.. and so forth for the rest of the buffers

我的问题当然是,如何解决根本原因?这是造成这种情况的原因吗?

编辑1:

为了验证它是MySQL还是与内存泄漏有关的某些设置,我正在考虑停止并重新启动MySQL服务,但不停止任何其他应用程序,这应该释放它分配的所有内存,并且如果我看向错误的方向,应该立即显示,对吗?

编辑2:

#  top -o %MEM -n 1 | head -n 15
top - 11:04:29 up 78 days, 20:49,  1 user,  load average: 11.16, 11.51, 12.71
Tasks: 925 total,   4 running, 921 sleeping,   0 stopped,   0 zombie
%Cpu(s): 66.1 us,  1.4 sy,  0.0 ni, 32.0 id,  0.0 wa,  0.4 hi,  0.1 si,  0.0 st
MiB Mem : 386566.5 total, 143783.1 free, 186723.6 used,  56059.8 buff/cache
MiB Swap:   4091.0 total,   4091.0 free,      0.0 used. 196546.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  43237 mysql     20   0   37.0g  11.3g  38336 S  29.4   3.0   1585:24 mysqld
1235657 root      20   0 4392096   3.4g  63936 S   5.9   0.9 366:28.82 node
1395899 root      20   0 2105132   1.1g  62748 S   0.0   0.3 103:38.66 node
1485407 root      20   0 2092800   1.1g  63400 S   0.0   0.3 169:23.37 node
1537596 root      20   0 2012632   1.0g  62328 S   0.0   0.3 134:55.01 node
1950191 root      20   0 1971424   1.0g  59572 S   0.0   0.3  69:58.87 node
1009156 root      20   0 1947612 985.4m  62700 S   0.0   0.3  67:36.20 node
1048076 root      20   0 1897160 991924  63304 S   5.9   0.3  77:52.66 node


# free
              total        used        free      shared  buff/cache   available
Mem:      395844132   191200164   147219260        1552    57424708   201268652
Swap:       4189180           0     4189180

I have an installation of MySQL Ver 8.0.26 for Linux on x86_64 which allocates a lot more memory than I expected. I'd like to understand where it has gone to fix the root cause.

The Linux server has a total of 370GB total RAM but it's not a database server only it's where the applications are also run from. My database size right now is about 5GB with a total of less than 25 tables.

After trying to locate the memory leak I stumbled upon the Mysql Open tables which seems to be correlating with the increased memory usage. (Graph shown below is over 5 days time using netdata).

Memory usage over time:
enter image description here

Open tables number growing over time.
enter image description here

Here are the changes I have done in the my.cnf file

[mysqld]
max_connections = 512
innodb_log_file_size = 1024M
innodb_buffer_pool_size = 16G
innodb_print_all_deadlocks = 1
wait_timeout = 31536000
interactive_timeout = 31536000
innodb_autoinc_lock_mode=2
max_allowed_packet=512M
default_storage_engine = InnoDB

After running the show engine innodb status; here is what I get.

--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
 ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 9626
877 OS file reads, 109038826 OS file writes, 78122941 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 238.31 writes/s, 181.31 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
 insert 0, delete mark 0, delete 0
discarded operations:
 insert 0, delete mark 0, delete 0
Hash table size 4425293, node heap has 1237 buffer(s)
Hash table size 4425293, node heap has 3907 buffer(s)
Hash table size 4425293, node heap has 18904 buffer(s)
Hash table size 4425293, node heap has 10211 buffer(s)
Hash table size 4425293, node heap has 135 buffer(s)
Hash table size 4425293, node heap has 2833 buffer(s)
Hash table size 4425293, node heap has 1450 buffer(s)
Hash table size 4425293, node heap has 469 buffer(s)
3181.09 hash searches/s, 6282.52 non-hash searches/s
---
LOG
---
Log sequence number          48666558274
Log buffer assigned up to    48666558274
Log buffer completed up to   48666558274
Log written up to            48666558274
Log flushed up to            48666558274
Added dirty pages up to      48666558274
Pages flushed up to          48496882991
Last checkpoint at           48496882991
98154440 log i/o's done, 222.11 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 17541103616
Dictionary memory allocated 702518
Buffer pool size   1048480
Free buffers       482537
Database pages     526797
Old database pages 194298
Modified db pages  24379
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 2328320, not young 21676
0.00 youngs/s, 0.00 non-youngs/s
Pages read 854, created 539313, written 7072261
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 526797, unzip_LRU len: 0
I/O sum[3200]:cur[64], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size   131058
Free buffers       60244
Database pages     65964
Old database pages 24329
Modified db pages  3277
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 957873, not young 10993
0.00 youngs/s, 0.00 non-youngs/s
Pages read 175, created 71085, written 1399422
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 65964, unzip_LRU len: 0
I/O sum[400]:cur[8], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size   131060
Free buffers       60410
Database pages     65754
Old database pages 24252
Modified db pages  2831
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 787777, not young 8988
0.00 youngs/s, 0.00 non-youngs/s
Pages read 130, created 69632, written 1222975
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 65754, unzip_LRU len: 0
I/O sum[400]:cur[8], unzip sum[0]:cur[0]

.. and so forth for the rest of the buffers

My question of course is, how can I fix the root cause and is this the cause of it?

Edit 1:

In order to verify whether it's MySQL or a some setting that has to do with the memory leak I was thinking of stopping and re-starting the service of MySQL but without stopping any other application, which should free all the memory it has allocated and should show right away if I am looking in the wrong direction or not, correct?

Edit 2:

#  top -o %MEM -n 1 | head -n 15
top - 11:04:29 up 78 days, 20:49,  1 user,  load average: 11.16, 11.51, 12.71
Tasks: 925 total,   4 running, 921 sleeping,   0 stopped,   0 zombie
%Cpu(s): 66.1 us,  1.4 sy,  0.0 ni, 32.0 id,  0.0 wa,  0.4 hi,  0.1 si,  0.0 st
MiB Mem : 386566.5 total, 143783.1 free, 186723.6 used,  56059.8 buff/cache
MiB Swap:   4091.0 total,   4091.0 free,      0.0 used. 196546.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  43237 mysql     20   0   37.0g  11.3g  38336 S  29.4   3.0   1585:24 mysqld
1235657 root      20   0 4392096   3.4g  63936 S   5.9   0.9 366:28.82 node
1395899 root      20   0 2105132   1.1g  62748 S   0.0   0.3 103:38.66 node
1485407 root      20   0 2092800   1.1g  63400 S   0.0   0.3 169:23.37 node
1537596 root      20   0 2012632   1.0g  62328 S   0.0   0.3 134:55.01 node
1950191 root      20   0 1971424   1.0g  59572 S   0.0   0.3  69:58.87 node
1009156 root      20   0 1947612 985.4m  62700 S   0.0   0.3  67:36.20 node
1048076 root      20   0 1897160 991924  63304 S   5.9   0.3  77:52.66 node


# free
              total        used        free      shared  buff/cache   available
Mem:      395844132   191200164   147219260        1552    57424708   201268652
Swap:       4189180           0     4189180

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文