内存持续增加导致内存告警

EMQX 版本

EMQX 5.0.23

EMQX 安装部署方式

通过 deb 包安装部署

EMQX 集群情况

使用 3 个节点组成集群

服务器(运行 EMQX 的机器)硬件配置

16核 64GB,CPU 型号:Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz

服务器操作系统和平台

CentOS Linux release 7.9.2009 (Core)

服务端参数优化情况

fs.file-max = 1048576
fs.nr_open=2097152
net.core.somaxconn=32768
net.ipv4.tcp_max_syn_backlog=16384
net.core.netdev_max_backlog=16384
net.ipv4.ip_local_port_range=1024 65535
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.optmem_max=16777216
net.ipv4.tcp_rmem=1024 4096 16777216
net.ipv4.tcp_wmem=1024 4096 16777216
net.nf_conntrack_max=1000000
net.netfilter.nf_conntrack_max=1000000
net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
net.ipv4.tcp_max_tw_buckets=1048576
net.ipv4.tcp_fin_timeout=15
vm.swappiness = 0
kernel.sysrq = 1

net.ipv4.neigh.default.gc_stale_time = 120

net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2

net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_slow_start_after_idle = 0

persistent_session_store.enabled=true
node {
name = “emqx@s1.emqx.io
cookie = “emqxsecretcookie”
data_dir = “/var/lib/emqx”
process_limit = 2097152
max_ports = 2097152
}

log {
file_handlers.default {
level = warning
file = “/var/log/emqx/emqx.log”
}
}

cluster {
name = emqxcl
discovery_strategy = static
static {
seeds = [“emqx@s1.emqx.io”, “emqx@s2.emqx.io”, “emqx@s3.emqx.io”]
}
}

listeners.tcp.default {
bind = “0.0.0.0:1883”
max_connections = 1024000
acceptors = 128
}

listeners.ssl.default {
bind = “0.0.0.0:8883”
max_connections = 512000
ssl_options {
keyfile = “/etc/emqx/certs/key.pem”
certfile = “/etc/emqx/certs/cert.pem”
cacertfile = “/etc/emqx/certs/cacert.pem”
}
}

listeners.ws.default {
bind = “0.0.0.0:8083”
max_connections = 1024000
websocket.mqtt_path = “/mqtt”
}

listeners.wss.default {
bind = “0.0.0.0:8084”
max_connections = 512000
websocket.mqtt_path = “/mqtt”
ssl_options {
keyfile = “/etc/emqx/certs/key.pem”
certfile = “/etc/emqx/certs/cert.pem”
cacertfile = “/etc/emqx/certs/cacert.pem”
}
}

dashboard {
listeners.http {
bind = 18083
}
default_username = “*"
default_password = "

}

authorization {
deny_action = ignore
no_match = allow
cache = { enable = true }
sources = [
{
type = file
enable = true
path = “/etc/emqx/acl.conf”
}
]
}

在 EMQX 中启用的功能

启用了 WebHook

场景

具体问题

内存使用率持续增加触发告警,但是平均每台机器连接数只有10w,却使用了59G的内存。

执行命令:spawn(fun() → etop:start([{output, text}, {interval, 5}, {lines, 20}, {sort, memory}]) end).


执行命令:spawn(fun() → etop:start([{output, text}, {interval, 5}, {lines, 20}, {sort, msg_q}]) end).


数据桥接:


日志:


日志中有很多webhook的报错,不知道内存持续增长是否和这个有关

1 个赞

感谢提供详细的说明,这里可以看到 Ets 就占用了 22GB

看看是哪个表占用比较多呢

因为你开启了持久化会话 关闭然后重启下节点就好了

我确实开启了持久化会话,但是没有设置是存储在磁盘还是内存中,看文档不应该是默认存储在磁盘上吗,为什么会占用内存。