Docker之Redis 大量抛Can’t save in background: fork: Out of memory错误的解决方案

  • A+
所属分类:Docker  Linux  虚拟化
摘要

容器中Redis抛Can’t save in background: fork: Out of memory错的解决方法

[root@ltapi2 ~]# docker logs redis_server 
Redis Server Auth Password : ****************
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 3.2.5 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 1
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

1:M 30 Oct 12:01:05.420 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 30 Oct 12:01:05.420 # Server started, Redis version 3.2.5
1:M 30 Oct 12:01:05.420 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 30 Oct 12:01:05.420 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 30 Oct 12:01:06.947 * DB loaded from append only file: 1.527 seconds
1:M 30 Oct 12:01:06.947 * The server is now ready to accept connections on port 6379
1:M 30 Oct 12:01:16.074 * 100 changes in 10 seconds. Saving...
1:M 30 Oct 12:01:16.081 * Background saving started by pid 26
26:C 30 Oct 12:01:16.571 * DB saved on disk
26:C 30 Oct 12:01:16.577 * RDB: 8 MB of memory used by copy-on-write
1:M 30 Oct 12:01:16.582 * Background saving terminated with success
1:M 30 Oct 12:01:39.144 * 100 changes in 10 seconds. Saving...
1:M 30 Oct 12:01:39.144 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:01:45.062 * 100 changes in 10 seconds. Saving...
1:M 30 Oct 12:01:45.062 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:01:51.077 * 100 changes in 10 seconds. Saving...
1:M 30 Oct 12:01:51.077 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:01:57.091 * 100 changes in 10 seconds. Saving...
1:M 30 Oct 12:01:57.091 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:02:03.008 * 100 changes in 10 seconds. Saving...
1:M 30 Oct 12:02:03.008 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:02:09.023 * 100 changes in 10 seconds. Saving...
1:M 30 Oct 12:02:09.023 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:02:15.038 * 100 changes in 10 seconds. Saving...
1:M 30 Oct 12:02:15.038 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:02:21.055 * 100 changes in 10 seconds. Saving...
1:M 30 Oct 12:02:21.055 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:02:27.071 * 100 changes in 10 seconds. Saving...
1:M 30 Oct 12:02:27.071 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:02:33.086 * 100 changes in 10 seconds. Saving...
1:M 30 Oct 12:02:33.086 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:02:39.004 * 100 changes in 10 seconds. Saving...
1:M 30 Oct 12:02:39.004 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:02:45.020 * 100 changes in 10 seconds. Saving...
1:M 30 Oct 12:02:45.021 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:02:51.035 * 100 changes in 10 seconds. Saving...
1:M 30 Oct 12:02:51.035 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:02:57.053 * 10 changes in 100 seconds. Saving...
1:M 30 Oct 12:02:57.053 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:03:03.069 * 10 changes in 100 seconds. Saving...
1:M 30 Oct 12:03:03.069 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:03:09.085 * 10 changes in 100 seconds. Saving...
1:M 30 Oct 12:03:09.085 # Can't save in background: fork: Out of memory
1:M 30 Oct 12:03:15.001 * 10 changes in 100 seconds. Saving...
1:M 30 Oct 12:03:15.001 # Can't save in background: fork: Out of memory

由于将线上Redis更换到Docker中跑了,但是今天遇到了一个问题大量抛出Can't save in background: fork: Out of memory错误,GG了下在非容器环境下其实只要该下内核参数就好。
但是我在Docker中测试直接用sysctl 来修改还不行,于是就用了写文件的方法实现

echo 1 > /proc/sys/vm/overcommit_memory

其实上面的命令和sysctl vm.overcommit_memory=1命令是同样的效果,但是后面的方法在容器中不一定可行。

在小内存的进程上做一个fork,不需要太多资源,但当这个进程的内存空间以G为单位时,fork就成为一件很恐怖的操作。

何况在2G内存的主机上fork 1G内存的进程呢?肯定会报内存无法分配的。更可气的是,越是改动频繁的主机上fork也越频繁,fork操作本身的代价恐怕也不会比假死好多少。

找到原因之后,直接修改内核参数 vm.overcommit_memory = 1

Linux内核会根据参数vm.overcommit_memory参数的设置决定是否放行。

vm.overcommit_memory = 1:直接放行

vm.overcommit_memory = 0:则比较 此次请求分配的虚拟内存大小和系统当前空闲的物理内存加上swap,决定是否放行。

vm.overcommit_memory = 2:则会比较 进程所有已分配的虚拟内存加上此次请求分配的虚拟内存和系统当前的空闲物理内存加上swap,决定是否放行。

linux设置vm.overcommit_memory 方法,永久性修改内核参数

在/etc/sysctl.conf文件里面加入或者直接删除也可以,因为它缺省值就是0

vm.overcommit_memory = 0运行使之生效

lookback

发表评论

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: