emqx内存大小该怎么调整

EMQX 性能问题
,
1

EMQX 版本

EMQX 版本 4.4.2

EMQX 安装部署方式

Linux通过安装包部署

EMQX 集群情况

emqx3个节点做的集群

服务器（运行 EMQX 的机器）硬件配置

核 32GB，CPU 型号：
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian

服务器操作系统和平台

centos7.5

测试场景

使用jemter工具，测试10万模拟设备连接到emqx集群中

具体问题

使用jemter工具，测试10万模拟设备连接到emqx集群中，emqx有日志说内存不足，请问有什么办法加大内存
warning日志如下：
2024-12-18T19:14:20.292804+08:00 [warning] [Dashboard] The initial default password for dashboard ‘admin’ user in emqx_dashboard.conf, For safety, it should be changed as soon as possible., Please use the ‘./bin/emqx_ctl admins’ CLI to change it., Then remove dashboard.default_user.login/password from emqx_dashboard.conf
2024-12-19T04:52:39.065909+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70%
2024-12-19T09:50:03.172888+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70.0%
2024-12-19T09:50:03.230487+08:00 [warning] [Dashboard] The initial default password for dashboard ‘admin’ user in emqx_dashboard.conf, For safety, it should be changed as soon as possible., Please use the ‘./bin/emqx_ctl admins’ CLI to change it., Then remove dashboard.default_user.login/password from emqx_dashboard.conf
2024-12-23T09:51:49.633507+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is deactivated
2024-12-26T15:39:35.147507+08:00 [warning] [SYSMON] busy_dist_port warning: suspid = <0.18515.22>, port = #Port<0.33>, [{proc_lib_initial_call,{emqx_connection,init,[‘Argument__1’,‘Argument__2’,‘Argument__3’,‘Argument__4’]}},{memory,21752},{total_heap_size,2585},{heap_size,1598},{stack_size,42},{min_heap_size,233},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{emqx_shared_sub,dispatch_per_qos,4,[{file,“emqx_shared_sub.erl”},{line,154}]},{emqx_shared_sub,dispatch,4,[{file,“emqx_shared_sub.erl”},{line,128}]},{emqx_broker,do_route,2,[{file,“emqx_broker.erl”},{line,248}]},{emqx_broker,‘-route/2-fun-0-’,3,[{file,“emqx_broker.erl”},{line,236}]},{lists,foldl,3,[{file,“lists.erl”},{line,1267}]},{emqx_channel,do_publish,3,[{file,“emqx_channel.erl”},{line,603}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,694}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,394}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,400}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,358}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]},{registered_name,},{status,suspended},{message_queue_len,0},{group_leader,<0.1588.0>},{priority,normal},{trap_exit,false},{reductions,107737},{last_calls,false},{catchlevel,3},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}], [{name,“tcp_inet”},{links,[<0.2040.0>,<0.1476.0>]},{id,264},{connected,<0.2040.0>},{input,101400961},{output,58},{os_pid,undefined}]
2024-12-26T18:07:06.022225+08:00 [warning] [SYSMON] long_schedule warning: pid = <0.20472.24>, info: [{timeout,256}, {in,undefined}, {out, {emqx_connection,recvloop, 2}}], [{proc_lib_initial_call,{emqx_connection,init,[‘Argument__1’,‘Argument__2’,‘Argument__3’,‘Argument__4’]}},{memory,13848},{total_heap_size,1597},{heap_size,610},{stack_size,9},{min_heap_size,233},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{emqx_connection,recvloop,2,[{file,“emqx_connection.erl”},{line,348}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]},{registered_name,},{status,waiting},{message_queue_len,0},{group_leader,<0.1588.0>},{priority,normal},{trap_exit,false},{reductions,1646569},{last_calls,false},{catchlevel,1},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}]
2024-12-27T02:26:16.299455+08:00 [warning] [SYSMON] long_schedule warning: pid = <0.15246.26>, info: [{timeout,259}, {in, {emqx_connection,recvloop, 2}}, {out, {emqx_connection,recvloop, 2}}], [{proc_lib_initial_call,{emqx_connection,init,[‘Argument__1’,‘Argument__2’,‘Argument__3’,‘Argument__4’]}},{memory,13848},{total_heap_size,1597},{heap_size,610},{stack_size,9},{min_heap_size,233},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{emqx_connection,recvloop,2,[{file,“emqx_connection.erl”},{line,348}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]},{registered_name,},{status,waiting},{message_queue_len,0},{group_leader,<0.1588.0>},{priority,normal},{trap_exit,false},{reductions,2106653},{last_calls,false},{catchlevel,1},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}]
2024-12-27T09:15:57.559636+08:00 [warning] [SYSMON] long_schedule warning: pid = <0.1819.0>, info: [{timeout,260}, {in, {emqx_cm, ‘-fun.mark_channel_disconnected/1-’, 1}}, {out, {emqx_misc,drain_down,2}}], [{proc_lib_initial_call,{emqx_cm,init,[‘Argument__1’]}},{memory,8704816},{total_heap_size,982318},{heap_size,121536},{stack_size,29},{min_heap_size,233},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{emqx_misc,drain_down,2,[{file,“emqx_misc.erl”},{line,197}]},{emqx_cm,handle_info,2,[{file,“emqx_cm.erl”},{line,499}]},{gen_server,try_dispatch,4,[{file,“gen_server.erl”},{line,695}]},{gen_server,handle_msg,6,[{file,“gen_server.erl”},{line,771}]},{proc_lib,init_p_do_apply,3,[{file,“proc_lib.erl”},{line,226}]}]},{registered_name,emqx_cm},{status,running},{message_queue_len,573},{group_leader,<0.1612.0>},{priority,normal},{trap_exit,false},{reductions,9277613},{last_calls,false},{catchlevel,2},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}]
2024-12-27T09:44:00.024918+08:00 [error] message=channel_closed driver=tcp socket=“#Port<0.43>” action=stopping
2024-12-27T09:44:00.025694+08:00 [error] message=channel_closed driver=tcp socket=“#Port<0.44>” action=stopping
2024-12-27T09:44:00.027319+08:00 [error] message=channel_closed driver=tcp socket=“#Port<0.110>” action=stopping
2024-12-27T09:46:39.900963+08:00 [warning] ekka_mnesia: still waiting for table(s): [emqx_resource,mqtt_app,emqx_rule,emqx_banned,mqtt_admin]
2024-12-27T09:47:09.905015+08:00 [warning] ekka_mnesia: still waiting for table(s): [emqx_resource,mqtt_app,emqx_rule,emqx_banned,mqtt_admin]
2024-12-27T09:47:39.908921+08:00 [warning] ekka_mnesia: still waiting for table(s): [emqx_resource,mqtt_app,emqx_rule,emqx_banned,mqtt_admin]
2024-12-27T09:48:09.912877+08:00 [warning] ekka_mnesia: still waiting for table(s): [emqx_resource,mqtt_app,emqx_rule,emqx_banned,mqtt_admin]
2024-12-27T09:48:27.908694+08:00 [warning] [Dashboard] The initial default password for dashboard ‘admin’ user in emqx_dashboard.conf, For safety, it should be changed as soon as possible., Please use the ‘./bin/emqx_ctl admins’ CLI to change it., Then remove dashboard.default_user.login/password from emqx_dashboard.conf
2024-12-27T11:37:53.438005+08:00 [warning] [SYSMON] long_schedule warning: port = #Port<0.32>, info: [{timeout,363}, {port_op,input}], [{name,“tcp_inet”},{links,[<0.1911.0>]},{id,256},{connected,<0.1911.0>},{input,0},{output,0},{os_pid,undefined}]
2024-12-27T21:53:17.843608+08:00 [warning] [SYSMON] busy_dist_port warning: suspid = <0.1845.0>, port = #Port<0.29>, [{proc_lib_initial_call,{emqx_pool,init,[‘Argument__1’]}},{memory,429144},{total_heap_size,53194},{heap_size,6772},{stack_size,38},{min_heap_size,233},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{mnesia_tm,async_send_dirty,6,[{file,“mnesia_tm.erl”},{line,2020}]},{mnesia_tm,dirty,2,[{file,“mnesia_tm.erl”},{line,1076}]},{emqx_cm,do_unregister_channel,1,[{file,“emqx_cm.erl”},{line,151}]},{lists,foreach,2,[{file,“lists.erl”},{line,1342}]},{emqx_pool,handle_cast,2,[{file,“emqx_pool.erl”},{line,108}]},{gen_server,try_dispatch,4,[{file,“gen_server.erl”},{line,695}]},{gen_server,handle_msg,6,[{file,“gen_server.erl”},{line,771}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]},{registered_name,emqx_pool_4},{status,suspended},{message_queue_len,25},{group_leader,<0.1612.0>},{priority,normal},{trap_exit,false},{reductions,890508},{last_calls,false},{catchlevel,3},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}], [{name,“tcp_inet”},{links,[<0.1779.0>,<0.1476.0>]},{id,232},{connected,<0.1779.0>},{input,33152849},{output,58},{os_pid,undefined}]
2025-01-02T17:50:04.806873+08:00 [warning] [SYSMON] busy_dist_port warning: suspid = <0.14634.20>, port = #Port<0.21>, [{proc_lib_initial_call,{emqx_connection,init,[‘Argument__1’,‘Argument__2’,‘Argument__3’,‘Argument__4’]}},{memory,29656},{total_heap_size,3573},{heap_size,2586},{stack_size,42},{min_heap_size,233},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{emqx_shared_sub,dispatch_per_qos,4,[{file,“emqx_shared_sub.erl”},{line,154}]},{emqx_shared_sub,dispatch,4,[{file,“emqx_shared_sub.erl”},{line,128}]},{emqx_broker,do_route,2,[{file,“emqx_broker.erl”},{line,248}]},{emqx_broker,‘-route/2-fun-0-’,3,[{file,“emqx_broker.erl”},{line,236}]},{lists,foldl,3,[{file,“lists.erl”},{line,1267}]},{emqx_channel,do_publish,3,[{file,“emqx_channel.erl”},{line,603}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,694}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,394}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,400}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,358}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]},{registered_name,},{status,suspended},{message_queue_len,0},{group_leader,<0.1588.0>},{priority,normal},{trap_exit,false},{reductions,37225},{last_calls,false},{catchlevel,3},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}], [{name,“tcp_inet”},{links,[<0.1721.0>,<0.1476.0>]},{id,168},{connected,<0.1721.0>},{input,108310314},{output,58},{os_pid,undefined}]
2025-01-03T16:31:56.111453+08:00 [error] Process: <0.2353.33> on node ‘emqx@172.29.247.179’, Context: maximum heap size reached, Max Heap Size: 10485760, Total Heap Size: 13134759, Kill: true, Error Logger: true, Message Queue Len: 0, GC Info: [{old_heap_block_size,5157867},{heap_block_size,7880076},{mbuf_size,97025},{recent_size,1754707},{stack_size,210},{old_heap_size,0},{heap_size,3581644},{bin_vheap_size,2363492},{bin_vheap_block_size,3389090},{bin_old_vheap_size,0},{bin_old_vheap_block_size,832883}]
2025-01-06T15:45:48.886409+08:00 [warning] [SYSMON] long_schedule warning: pid = <0.1893.0>, info: [{timeout,244}, {in,{emqx_pmon,find,2}}, {out, {emqx_cm, mark_channel_disconnected, 1}}], [{proc_lib_initial_call,{emqx_cm,init,[‘Argument__1’]}},{memory,11195704},{total_heap_size,1230721},{heap_size,196650},{stack_size,29},{min_heap_size,233},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{emqx_misc,drain_down,2,[{file,“emqx_misc.erl”},{line,197}]},{emqx_cm,handle_info,2,[{file,“emqx_cm.erl”},{line,499}]},{gen_server,try_dispatch,4,[{file,“gen_server.erl”},{line,695}]},{gen_server,handle_msg,6,[{file,“gen_server.erl”},{line,771}]},{proc_lib,init_p_do_apply,3,[{file,“proc_lib.erl”},{line,226}]}]},{registered_name,emqx_cm},{status,running},{message_queue_len,1858},{group_leader,<0.1612.0>},{priority,normal},{trap_exit,false},{reductions,12208390},{last_calls,false},{catchlevel,2},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}]
2025-01-06T15:45:50.218123+08:00 [error] Process: <0.20436.47> on node ‘emqx@172.29.247.179’, Context: maximum heap size reached, Max Heap Size: 10485760, Total Heap Size: 12178174, Kill: true, Error Logger: true, Message Queue Len: 0, GC Info: [{old_heap_block_size,4298223},{heap_block_size,7880076},{mbuf_size,48526},{recent_size,2079325},{stack_size,128},{old_heap_size,0},{heap_size,3533202},{bin_vheap_size,2455991},{bin_vheap_block_size,3800442},{bin_old_vheap_size,0},{bin_old_vheap_block_size,881975}]
2025-01-14T21:36:51.018513+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70%
2025-01-14T21:37:51.019408+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is deactivated
2025-01-14T21:38:51.248526+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70%
2025-01-14T21:50:52.471744+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is deactivated
2025-01-14T21:51:52.700094+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70%
2025-01-14T21:52:52.700568+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is deactivated
2025-01-14T22:21:56.133670+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70%
2025-01-14T22:22:56.137370+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is deactivated
2025-01-14T22:24:56.495507+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70%

2

4.4.2 已经过了维护周期了，可以升级一下。

这已经在说整个操作系统都内存不够了。
PS： emqx 的分配内存不需要配置。他是自动申请的。

3

我们升级了emqx版本，目前还是连不上，连接最大就是64,512个，emqx有报错日志：日志如下：
2025-01-21T11:02:51.325770+08:00 [warning] msg: config_key_not_recognized, unknown_config_keys: zone
2025-01-21T11:02:54.296900+08:00 [warning] msg: Stopping mria, reason: join
2025-01-21T11:02:54.849323+08:00 [warning] msg: config_key_not_recognized, unknown_config_keys: zone
2025-01-21T11:03:26.209240+08:00 [warning] msg: unexpected_api_access, request: #{pid => <0.4590.0>,port => 18083,scheme => <<“http”>>,version => ‘HTTP/1.1’,path => <<“/api/v4/auth”>>,host => <<“172.29.247.179”>>,peer => {{10,40,163,137},61865},bindings => #{},sock => {{172,29,247,179},18083},ref => ‘http:dashboard’,cert => undefined,headers => #{<<“accept”>> => <<“/”>>,<<“accept-encoding”>> => <<“gzip, deflate”>>,<<“accept-language”>> => <<“zh-CN,zh;q=0.9”>>,<<“cache-control”>> => <<“no-cache”>>,<<“connection”>> => <<“keep-alive”>>,<<“content-length”>> => <<“40”>>,<<“content-type”>> => <<“application/json”>>,<<“host”>> => <<“172.29.247.179:18083”>>,<<“origin”>> => <<“http://172.29.247.179:18083”>>,<<“referer”>> => <<“http://172.29.247.179:18083/">>,<<"user-agent”>> => <<“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36”>>},method => <<“POST”>>,body_length => 40,path_info => undefined,qs => <<>>,host_info => undefined,streamid => 1,has_body => true}
2025-01-21T11:03:35.572582+08:00 [warning] msg: unexpected_api_access, request: #{pid => <0.4603.0>,port => 18083,scheme => <<“http”>>,version => ‘HTTP/1.1’,path => <<“/api/v4/auth”>>,host => <<“172.29.247.179”>>,peer => {{10,40,163,137},61876},bindings => #{},sock => {{172,29,247,179},18083},ref => ‘http:dashboard’,cert => undefined,headers => #{<<“accept”>> => <<“/”>>,<<“accept-encoding”>> => <<“gzip, deflate”>>,<<“accept-language”>> => <<“zh-CN,zh;q=0.9”>>,<<“cache-control”>> => <<“no-cache”>>,<<“connection”>> => <<“keep-alive”>>,<<“content-length”>> => <<“40”>>,<<“content-type”>> => <<“application/json”>>,<<“host”>> => <<“172.29.247.179:18083”>>,<<“origin”>> => <<“http://172.29.247.179:18083”>>,<<“referer”>> => <<“http://172.29.247.179:18083/">>,<<"user-agent”>> => <<“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36”>>},method => <<“POST”>>,body_length => 40,path_info => undefined,qs => <<>>,host_info => undefined,streamid => 1,has_body => true}
2025-01-21T14:53:59.315642+08:00 [warning] msg: config_key_not_recognized, unknown_config_keys: zone
2025-01-21T15:35:48.450928+08:00 [error] supervisor: {esockd_connection_sup,<0.111738.0>}, errorContext: connection_shutdown, reason: #{max => 1000,reason => mailbox_overflow,value => 1728}, offender: [{pid,<0.111738.0>},{name,connection},{mfargs,{emqx_connection,start_link,[#{listener => {tcp,default},limiter => #{connection => #{initial => 0,rate => infinity,burst => 0}},enable_authn => true,zone => default}]}}]
2025-01-21T15:45:06.750887+08:00 [error] supervisor: {esockd_connection_sup,<0.152762.0>}, errorContext: connection_shutdown, reason: #{max => 1000,reason => mailbox_overflow,value => 1085}, offender: [{pid,<0.152762.0>},{name,connection},{mfargs,{emqx_connection,start_link,[#{listener => {tcp,default},limiter => #{connection => #{initial => 0,rate => infinity,burst => 0}},enable_authn => true,zone => default}]}}]
2025-01-21T15:55:14.431611+08:00 [error] supervisor: {esockd_connection_sup,<0.232496.0>}, errorContext: connection_shutdown, reason: #{max => 1000,reason => mailbox_overflow,value => 2083}, offender: [{pid,<0.232496.0>},{name,connection},{mfargs,{emqx_connection,start_link,[#{listener => {tcp,default},limiter => #{connection => #{initial => 0,rate => infinity,burst => 0}},enable_authn => true,zone => default}]}}]
2025-01-21T16:06:18.186762+08:00 [error] supervisor: {esockd_connection_sup,<0.325636.0>}, errorContext: connection_shutdown, reason: #{max => 1000,reason => mailbox_overflow,value => 1062}, offender: [{pid,<0.325636.0>},{name,connection},{mfargs,{emqx_connection,start_link,[#{listener => {tcp,default},limiter => #{connection => #{initial => 0,rate => infinity,burst => 0}},enable_authn => true,zone => default}]}}]
2025-01-21T16:07:07.658748+08:00 [error] supervisor: {esockd_connection_sup,<0.355126.0>}, errorContext: connection_shutdown, reason: #{max => 1000,reason => mailbox_overflow,value => 1192}, offender: [{pid,<0.355126.0>},{name,connection},{mfargs,{emqx_connection,start_link,[#{listener => {tcp,default},limiter => #{connection => #{initial => 0,rate => infinity,burst => 0}},enable_authn => true,zone => default}]}}]

4

目前是用的emqx版本是 emqx-5.7.2-el7-amd64.tar.gz，也还是3个节点，emqx的配置文件如下：
node {
name = “emqx@172.29.247.179”
cookie = “emqxsecretcookie”
data_dir = “data”
}

listeners.tcp.default {
bind = “0.0.0.0:1884”
max_connections = 1024000

proxy_protocol = true

}

gateway.coap {

mountpoint = “coap/”

connection_required = false

listeners.udp.default {
bind = “5683”
max_connections = 1024000
max_conn_rate = 10000
}
}

zone {
external {
max_mqueue_len = 0
}
internal {
max_mqueue_len = 0
}
}
cluster {
discovery_strategy = static
static {
seeds = [“emqx@172.29.247.161”, “emqx@172.29.247.172”, “emqx@172.29.247.179”]
}
}

log {
}
dashboard {
listeners.http {
bind = 18083
}
}

5

可以在社区内搜索一下这个关键字。以前也有人提过：

mailbox_overflow

6

应该是你测试的机器内存不足了吧，应该不是emqx，一台机器的端口号最大就是 65535，一个连接占一个端口

7

这个最大端口可以改吗？我想测试一下10万客户端连接的场景

8

改不了，想测试10万，你得多找几台机器测试