EMQX 版本
EMQX 版本 4.4.2
EMQX 安装部署方式
Linux通过安装包部署
EMQX 集群情况
emqx3个节点做的集群
服务器（运行 EMQX 的机器）硬件配置
核 32GB，CPU 型号：
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
服务器操作系统和平台
centos7.5
测试场景
使用jemter工具，测试10万模拟设备连接到emqx集群中
具体问题
使用jemter工具，测试10万模拟设备连接到emqx集群中，emqx有日志说内存不足，请问有什么办法加大内存
warning日志如下：
2024-12-18T19:14:20.292804+08:00 [warning] [Dashboard] The initial default password for dashboard ‘admin’ user in emqx_dashboard.conf, For safety, it should be changed as soon as possible., Please use the ‘./bin/emqx_ctl admins’ CLI to change it., Then remove dashboard.default_user.login/password from emqx_dashboard.conf
2024-12-19T04:52:39.065909+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70%
2024-12-19T09:50:03.172888+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70.0%
2024-12-19T09:50:03.230487+08:00 [warning] [Dashboard] The initial default password for dashboard ‘admin’ user in emqx_dashboard.conf, For safety, it should be changed as soon as possible., Please use the ‘./bin/emqx_ctl admins’ CLI to change it., Then remove dashboard.default_user.login/password from emqx_dashboard.conf
2024-12-23T09:51:49.633507+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is deactivated
2024-12-26T15:39:35.147507+08:00 [warning] [SYSMON] busy_dist_port warning: suspid = <0.18515.22>, port = #Port<0.33>, [{proc_lib_initial_call,{emqx_connection,init,[‘Argument__1’,‘Argument__2’,‘Argument__3’,‘Argument__4’]}},{memory,21752},{total_heap_size,2585},{heap_size,1598},{stack_size,42},{min_heap_size,233},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{emqx_shared_sub,dispatch_per_qos,4,[{file,“emqx_shared_sub.erl”},{line,154}]},{emqx_shared_sub,dispatch,4,[{file,“emqx_shared_sub.erl”},{line,128}]},{emqx_broker,do_route,2,[{file,“emqx_broker.erl”},{line,248}]},{emqx_broker,‘-route/2-fun-0-’,3,[{file,“emqx_broker.erl”},{line,236}]},{lists,foldl,3,[{file,“lists.erl”},{line,1267}]},{emqx_channel,do_publish,3,[{file,“emqx_channel.erl”},{line,603}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,694}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,394}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,400}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,358}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]},{registered_name,},{status,suspended},{message_queue_len,0},{group_leader,<0.1588.0>},{priority,normal},{trap_exit,false},{reductions,107737},{last_calls,false},{catchlevel,3},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}], [{name,“tcp_inet”},{links,[<0.2040.0>,<0.1476.0>]},{id,264},{connected,<0.2040.0>},{input,101400961},{output,58},{os_pid,undefined}]
2024-12-26T18:07:06.022225+08:00 [warning] [SYSMON] long_schedule warning: pid = <0.20472.24>, info: [{timeout,256}, {in,undefined}, {out, {emqx_connection,recvloop, 2}}], [{proc_lib_initial_call,{emqx_connection,init,[‘Argument__1’,‘Argument__2’,‘Argument__3’,‘Argument__4’]}},{memory,13848},{total_heap_size,1597},{heap_size,610},{stack_size,9},{min_heap_size,233},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{emqx_connection,recvloop,2,[{file,“emqx_connection.erl”},{line,348}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]},{registered_name,},{status,waiting},{message_queue_len,0},{group_leader,<0.1588.0>},{priority,normal},{trap_exit,false},{reductions,1646569},{last_calls,false},{catchlevel,1},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}]
2024-12-27T02:26:16.299455+08:00 [warning] [SYSMON] long_schedule warning: pid = <0.15246.26>, info: [{timeout,259}, {in, {emqx_connection,recvloop, 2}}, {out, {emqx_connection,recvloop, 2}}], [{proc_lib_initial_call,{emqx_connection,init,[‘Argument__1’,‘Argument__2’,‘Argument__3’,‘Argument__4’]}},{memory,13848},{total_heap_size,1597},{heap_size,610},{stack_size,9},{min_heap_size,233},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{emqx_connection,recvloop,2,[{file,“emqx_connection.erl”},{line,348}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]},{registered_name,},{status,waiting},{message_queue_len,0},{group_leader,<0.1588.0>},{priority,normal},{trap_exit,false},{reductions,2106653},{last_calls,false},{catchlevel,1},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}]
2024-12-27T09:15:57.559636+08:00 [warning] [SYSMON] long_schedule warning: pid = <0.1819.0>, info: [{timeout,260}, {in, {emqx_cm, ‘-fun.mark_channel_disconnected/1-’, 1}}, {out, {emqx_misc,drain_down,2}}], [{proc_lib_initial_call,{emqx_cm,init,[‘Argument__1’]}},{memory,8704816},{total_heap_size,982318},{heap_size,121536},{stack_size,29},{min_heap_size,233},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{emqx_misc,drain_down,2,[{file,“emqx_misc.erl”},{line,197}]},{emqx_cm,handle_info,2,[{file,“emqx_cm.erl”},{line,499}]},{gen_server,try_dispatch,4,[{file,“gen_server.erl”},{line,695}]},{gen_server,handle_msg,6,[{file,“gen_server.erl”},{line,771}]},{proc_lib,init_p_do_apply,3,[{file,“proc_lib.erl”},{line,226}]}]},{registered_name,emqx_cm},{status,running},{message_queue_len,573},{group_leader,<0.1612.0>},{priority,normal},{trap_exit,false},{reductions,9277613},{last_calls,false},{catchlevel,2},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}]
2024-12-27T09:44:00.024918+08:00 [error] message=channel_closed driver=tcp socket=“#Port<0.43>” action=stopping
2024-12-27T09:44:00.025694+08:00 [error] message=channel_closed driver=tcp socket=“#Port<0.44>” action=stopping
2024-12-27T09:44:00.027319+08:00 [error] message=channel_closed driver=tcp socket=“#Port<0.110>” action=stopping
2024-12-27T09:46:39.900963+08:00 [warning] ekka_mnesia: still waiting for table(s): [emqx_resource,mqtt_app,emqx_rule,emqx_banned,mqtt_admin]
2024-12-27T09:47:09.905015+08:00 [warning] ekka_mnesia: still waiting for table(s): [emqx_resource,mqtt_app,emqx_rule,emqx_banned,mqtt_admin]
2024-12-27T09:47:39.908921+08:00 [warning] ekka_mnesia: still waiting for table(s): [emqx_resource,mqtt_app,emqx_rule,emqx_banned,mqtt_admin]
2024-12-27T09:48:09.912877+08:00 [warning] ekka_mnesia: still waiting for table(s): [emqx_resource,mqtt_app,emqx_rule,emqx_banned,mqtt_admin]
2024-12-27T09:48:27.908694+08:00 [warning] [Dashboard] The initial default password for dashboard ‘admin’ user in emqx_dashboard.conf, For safety, it should be changed as soon as possible., Please use the ‘./bin/emqx_ctl admins’ CLI to change it., Then remove dashboard.default_user.login/password from emqx_dashboard.conf
2024-12-27T11:37:53.438005+08:00 [warning] [SYSMON] long_schedule warning: port = #Port<0.32>, info: [{timeout,363}, {port_op,input}], [{name,“tcp_inet”},{links,[<0.1911.0>]},{id,256},{connected,<0.1911.0>},{input,0},{output,0},{os_pid,undefined}]
2024-12-27T21:53:17.843608+08:00 [warning] [SYSMON] busy_dist_port warning: suspid = <0.1845.0>, port = #Port<0.29>, [{proc_lib_initial_call,{emqx_pool,init,[‘Argument__1’]}},{memory,429144},{total_heap_size,53194},{heap_size,6772},{stack_size,38},{min_heap_size,233},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{mnesia_tm,async_send_dirty,6,[{file,“mnesia_tm.erl”},{line,2020}]},{mnesia_tm,dirty,2,[{file,“mnesia_tm.erl”},{line,1076}]},{emqx_cm,do_unregister_channel,1,[{file,“emqx_cm.erl”},{line,151}]},{lists,foreach,2,[{file,“lists.erl”},{line,1342}]},{emqx_pool,handle_cast,2,[{file,“emqx_pool.erl”},{line,108}]},{gen_server,try_dispatch,4,[{file,“gen_server.erl”},{line,695}]},{gen_server,handle_msg,6,[{file,“gen_server.erl”},{line,771}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]},{registered_name,emqx_pool_4},{status,suspended},{message_queue_len,25},{group_leader,<0.1612.0>},{priority,normal},{trap_exit,false},{reductions,890508},{last_calls,false},{catchlevel,3},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}], [{name,“tcp_inet”},{links,[<0.1779.0>,<0.1476.0>]},{id,232},{connected,<0.1779.0>},{input,33152849},{output,58},{os_pid,undefined}]
2025-01-02T17:50:04.806873+08:00 [warning] [SYSMON] busy_dist_port warning: suspid = <0.14634.20>, port = #Port<0.21>, [{proc_lib_initial_call,{emqx_connection,init,[‘Argument__1’,‘Argument__2’,‘Argument__3’,‘Argument__4’]}},{memory,29656},{total_heap_size,3573},{heap_size,2586},{stack_size,42},{min_heap_size,233},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{emqx_shared_sub,dispatch_per_qos,4,[{file,“emqx_shared_sub.erl”},{line,154}]},{emqx_shared_sub,dispatch,4,[{file,“emqx_shared_sub.erl”},{line,128}]},{emqx_broker,do_route,2,[{file,“emqx_broker.erl”},{line,248}]},{emqx_broker,‘-route/2-fun-0-’,3,[{file,“emqx_broker.erl”},{line,236}]},{lists,foldl,3,[{file,“lists.erl”},{line,1267}]},{emqx_channel,do_publish,3,[{file,“emqx_channel.erl”},{line,603}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,694}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,394}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,400}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,358}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]},{registered_name,},{status,suspended},{message_queue_len,0},{group_leader,<0.1588.0>},{priority,normal},{trap_exit,false},{reductions,37225},{last_calls,false},{catchlevel,3},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}], [{name,“tcp_inet”},{links,[<0.1721.0>,<0.1476.0>]},{id,168},{connected,<0.1721.0>},{input,108310314},{output,58},{os_pid,undefined}]
2025-01-03T16:31:56.111453+08:00 [error] Process: <0.2353.33> on node ‘emqx@172.29.247.179’, Context: maximum heap size reached, Max Heap Size: 10485760, Total Heap Size: 13134759, Kill: true, Error Logger: true, Message Queue Len: 0, GC Info: [{old_heap_block_size,5157867},{heap_block_size,7880076},{mbuf_size,97025},{recent_size,1754707},{stack_size,210},{old_heap_size,0},{heap_size,3581644},{bin_vheap_size,2363492},{bin_vheap_block_size,3389090},{bin_old_vheap_size,0},{bin_old_vheap_block_size,832883}]
2025-01-06T15:45:48.886409+08:00 [warning] [SYSMON] long_schedule warning: pid = <0.1893.0>, info: [{timeout,244}, {in,{emqx_pmon,find,2}}, {out, {emqx_cm, mark_channel_disconnected, 1}}], [{proc_lib_initial_call,{emqx_cm,init,[‘Argument__1’]}},{memory,11195704},{total_heap_size,1230721},{heap_size,196650},{stack_size,29},{min_heap_size,233},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{emqx_misc,drain_down,2,[{file,“emqx_misc.erl”},{line,197}]},{emqx_cm,handle_info,2,[{file,“emqx_cm.erl”},{line,499}]},{gen_server,try_dispatch,4,[{file,“gen_server.erl”},{line,695}]},{gen_server,handle_msg,6,[{file,“gen_server.erl”},{line,771}]},{proc_lib,init_p_do_apply,3,[{file,“proc_lib.erl”},{line,226}]}]},{registered_name,emqx_cm},{status,running},{message_queue_len,1858},{group_leader,<0.1612.0>},{priority,normal},{trap_exit,false},{reductions,12208390},{last_calls,false},{catchlevel,2},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}]
2025-01-06T15:45:50.218123+08:00 [error] Process: <0.20436.47> on node ‘emqx@172.29.247.179’, Context: maximum heap size reached, Max Heap Size: 10485760, Total Heap Size: 12178174, Kill: true, Error Logger: true, Message Queue Len: 0, GC Info: [{old_heap_block_size,4298223},{heap_block_size,7880076},{mbuf_size,48526},{recent_size,2079325},{stack_size,128},{old_heap_size,0},{heap_size,3533202},{bin_vheap_size,2455991},{bin_vheap_block_size,3800442},{bin_old_vheap_size,0},{bin_old_vheap_block_size,881975}]
2025-01-14T21:36:51.018513+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70%
2025-01-14T21:37:51.019408+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is deactivated
2025-01-14T21:38:51.248526+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70%
2025-01-14T21:50:52.471744+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is deactivated
2025-01-14T21:51:52.700094+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70%
2025-01-14T21:52:52.700568+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is deactivated
2025-01-14T22:21:56.133670+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70%
2025-01-14T22:22:56.137370+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is deactivated
2025-01-14T22:24:56.495507+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70%