EMQX 版本
EMQX 5.0.23
EMQX 安装部署方式
通过 deb 包安装部署
EMQX 集群情况
使用 3 个节点组成集群
服务器(运行 EMQX 的机器)硬件配置
16核 64GB,CPU 型号:Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
服务器操作系统和平台
CentOS Linux release 7.9.2009 (Core)
服务端参数优化情况
fs.file-max = 1048576
fs.nr_open=2097152
net.core.somaxconn=32768
net.ipv4.tcp_max_syn_backlog=16384
net.core.netdev_max_backlog=16384
net.ipv4.ip_local_port_range=1024 65535
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.optmem_max=16777216
net.ipv4.tcp_rmem=1024 4096 16777216
net.ipv4.tcp_wmem=1024 4096 16777216
net.nf_conntrack_max=1000000
net.netfilter.nf_conntrack_max=1000000
net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
net.ipv4.tcp_max_tw_buckets=1048576
net.ipv4.tcp_fin_timeout=15
vm.swappiness = 0
kernel.sysrq = 1
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_slow_start_after_idle = 0
persistent_session_store.enabled=true
node {
name = “emqx@s1.emqx.io”
cookie = “emqxsecretcookie”
data_dir = “/var/lib/emqx”
process_limit = 2097152
max_ports = 2097152
}
log {
file_handlers.default {
level = warning
file = “/var/log/emqx/emqx.log”
}
}
cluster {
name = emqxcl
discovery_strategy = static
static {
seeds = [“emqx@s1.emqx.io”, “emqx@s2.emqx.io”, “emqx@s3.emqx.io”]
}
}
listeners.tcp.default {
bind = “0.0.0.0:1883”
max_connections = 1024000
acceptors = 128
}
listeners.ssl.default {
bind = “0.0.0.0:8883”
max_connections = 512000
ssl_options {
keyfile = “/etc/emqx/certs/key.pem”
certfile = “/etc/emqx/certs/cert.pem”
cacertfile = “/etc/emqx/certs/cacert.pem”
}
}
listeners.ws.default {
bind = “0.0.0.0:8083”
max_connections = 1024000
websocket.mqtt_path = “/mqtt”
}
listeners.wss.default {
bind = “0.0.0.0:8084”
max_connections = 512000
websocket.mqtt_path = “/mqtt”
ssl_options {
keyfile = “/etc/emqx/certs/key.pem”
certfile = “/etc/emqx/certs/cert.pem”
cacertfile = “/etc/emqx/certs/cacert.pem”
}
}
dashboard {
listeners.http {
bind = 18083
}
default_username = “*"
default_password = "”
}
authorization {
deny_action = ignore
no_match = allow
cache = { enable = true }
sources = [
{
type = file
enable = true
path = “/etc/emqx/acl.conf”
}
]
}
在 EMQX 中启用的功能
启用了 WebHook
场景
具体问题
内存使用率持续增加触发告警,但是平均每台机器连接数只有10w,却使用了59G的内存。
执行命令:spawn(fun() → etop:start([{output, text}, {interval, 5}, {lines, 20}, {sort, memory}]) end).
执行命令:spawn(fun() → etop:start([{output, text}, {interval, 5}, {lines, 20}, {sort, msg_q}]) end).
数据桥接:
日志:
日志中有很多webhook的报错,不知道内存持续增长是否和这个有关