emqtt-bench压测时被强制Killed问题

问题描述

我在windows server 2008的服务器上安装了VMWare 15,然后创建了1个Broker,两个压测客户端
使用emqtt-bench进行连接数压测,发现最多只能创建20万个连接,再多就会被强制Killed

我对每个客户端设置了10个网卡,都使用NAT网络
我用其中一个客户端,打开多个ssh做压测时,当连接数达到20万时,就会强制关闭压测程序,在不同的客户端上试验,结果也是一样,我按文档中的系统调优进行了服务端与客户端设置,请问这到底是哪方面的问题?

环境信息

  • EMQ X 版本:emqx-centos7-4.2.10-x86_64 开源版
  • 操作系统及版本:Windows server 2008 VMWare 15 CentOS7.8
  • Broker:8核16G
  • 客户端:4核8G

相应的配置文件内容

系统优化参考文档设置

详细日志

控制台输出

[root@centos emqtt-bench]# ./emqtt_bench conn -h 192.168.132.129 -c 50000 --ifaddr 192.168.132.140
connected: 1962client(49619): connect error - {{shutdown,tcp_closed},
                                {tcp_closed,#Port<0.387>}}
client(49925): connect error - {{shutdown,tcp_closed},
                                {tcp_closed,#Port<0.81>}}
connected: 34618client(15383): connect error - {{shutdown,tcp_closed},
                                {tcp_closed,#Port<0.34623>}}
connected: 49997Killed

erlang.log.1

=====
===== LOGGING STARTED Mon Apr 19 11:57:03 CST 2021
=====
Exec: /opt/emqx/erts-10.7/bin/erlexec -boot /opt/emqx/releases/4.2.10/emqx -mode embedded -boot_var ERTS_LIB_DIR /opt/emqx/erts-10.7/../lib -mnesia dir "/opt/emqx/data/mnesia/emqx@192.168.132.129" -config /opt/emqx/data/configs/app.2021.04.19.11.57.12.config -args_file /opt/emqx/data/configs/vm.2021.04.19.11.57.12.args -vm_args /opt/emqx/data/configs/vm.2021.04.19.11.57.12.args -start_epmd false -epmd_module ekka_epmd -proto_dist ekka -- console
Root: /opt/emqx
/opt/emqx
Erlang/OTP 22 [erts-10.7] [source] [64-bit] [smp:8:8] [ds:8:8:8] [async-threads:4] [hipe]

Starting emqx on node emqx@192.168.132.129
Start http:management listener on 8081 successfully.
Start http:dashboard listener on 18083 successfully.
Start mqtt:tcp listener on 127.0.0.1:11883 successfully.
Start mqtt:tcp listener on 0.0.0.0:1883 successfully.
Start mqtt:ws listener on 0.0.0.0:8083 successfully.
Start mqtt:ssl listener on 0.0.0.0:8883 successfully.
Start mqtt:wss listener on 0.0.0.0:8084 successfully.
EMQ X Broker 4.2.10 is running now!
Eshell V10.7  (abort with ^G)
(emqx@192.168.132.129)1> 
===== ALIVE Mon Apr 19 12:12:37 CST 2021

===== ALIVE Mon Apr 19 12:27:37 CST 2021

===== ALIVE Mon Apr 19 12:42:37 CST 2021

===== ALIVE Mon Apr 19 12:57:37 CST 2021

===== ALIVE Mon Apr 19 13:12:37 CST 2021

===== ALIVE Mon Apr 19 13:27:37 CST 2021

===== Mon Apr 19 13:41:44 CST 2021
2021-04-19 13:41:44.808 [warning] [SYSMON] long_schedule warning: port = #Port<0.26>, info: [{timeout,435},
                                                  {port_op,dist_cmd}]
[{name,"tcp_inet"},
 {links,[<0.1843.0>,<0.1305.0>]},
 {id,208},
 {connected,<0.1843.0>},
 {input,1068003},
 {output,51},
 {os_pid,undefined}]

===== ALIVE Mon Apr 19 13:56:44 CST 2021

===== ALIVE Mon Apr 19 14:11:45 CST 2021

===== Mon Apr 19 14:21:44 CST 2021
2021-04-19 14:21:44.673 [warning] [SYSMON] busy_dist_port warning: suspid = <0.1556.0>, port = #Port<0.26>
[{initial_call,{proc_lib,init_p,5}},
 {current_function,{erlang,bif_return_trap,2}},
 {registered_name,emqx_pool_7},
 {status,suspended}, 
 {message_queue_len,6},
 {group_leader,<0.1424.0>},
 {priority,normal},
 {trap_exit,false},
 {reductions,192510},
 {last_calls,false},
 {catchlevel,3},
 {trace,0},
 {suspending,[]},
 {sequential_trace_token,[]},
 {error_handler,error_handler},
 {memory,700416},
 {total_heap_size,87358},
 {heap_size,10958},
 {stack_size,37},
 {min_heap_size,233}]
[{name,"tcp_inet"},
 {links,[<0.1843.0>,<0.1305.0>]},
 {id,208},
 {connected,<0.1843.0>},
 {input,34195357},
 {output,51},
 {os_pid,undefined}]
2021-04-19 14:23:04.185 [warning] [SYSMON] large_heap warning: pid = <0.1601.0>, info: [{old_heap_block_size,8912793},
                                             {heap_block_size,1199557},
                                             {mbuf_size,0},
                                             {stack_size,32},
                                             {old_heap_size,3650620},
                                             {heap_size,347835}]
[{initial_call,{proc_lib,init_p,5}},
 {current_function,{emqx_misc,drain_down,2}},
 {registered_name,emqx_cm},
 {status,running},
 {message_queue_len,10260},
 {group_leader,<0.1424.0>},
 {priority,normal},
 {trap_exit,false},
 {reductions,19597911},
 {last_calls,false},
 {catchlevel,2},
 {trace,0},
 {suspending,[]},
 {sequential_trace_token,[]},
 {error_handler,error_handler},
 {memory,96709368},
 {total_heap_size,10385191},
 {heap_size,1199557},
 {stack_size,27},
 {min_heap_size,233}]
2021-04-19 14:25:49.193 [warning] [SYSMON] long_schedule warning: pid = <0.9295.31>, info: [{timeout,359},
                                                 {in,{gen_server,rec_nodes,7}},
                                                 {out,
                                                  {emqx_connection,recvloop,
                                                   2}}]
[{initial_call,{proc_lib,init_p,5}},
 {current_function,{emqx_connection,recvloop,2}},
 {registered_name,[]},
 {status,waiting},
 {message_queue_len,0},
 {group_leader,<0.1397.0>},
 {priority,normal},
 {trap_exit,false},
 {reductions,2208},
 {last_calls,false},
 {catchlevel,1},
 {trace,0},
 {suspending,[]},
 {sequential_trace_token,[]},
 {error_handler,error_handler},
 {memory,34556},
 {total_heap_size,4184},
 {heap_size,2586},
 {stack_size,9}, 
 {min_heap_size,233}]
2021-04-19 14:25:49.201 [warning] [SYSMON] long_schedule warning: pid = <0.3272.22>, info: [{timeout,360},
                                                 {in,undefined},
                                                 {out,
                                                  {emqx_connection,recvloop,
                                                   2}}]
[{initial_call,{proc_lib,init_p,5}},
 {current_function,{emqx_connection,recvloop,2}},
 {registered_name,[]},
 {status,waiting}, 
 {message_queue_len,0},
 {group_leader,<0.1397.0>},
 {priority,normal},
 {trap_exit,false},
 {reductions,2656},
 {last_calls,false},
 {catchlevel,1},
 {trace,0},
 {suspending,[]},
 {sequential_trace_token,[]},
 {error_handler,error_handler},
 {memory,9028},
 {total_heap_size,993},
 {heap_size,376},
 {stack_size,9},
 {min_heap_size,233}]

===== ALIVE Mon Apr 19 14:40:49 CST 2021

===== ALIVE Mon Apr 19 14:55:49 CST 2021

emqx.log

2021-04-19 14:21:44.673 [warning] [SYSMON] busy_dist_port warning: suspid = <0.1556.0>, port = #Port<0.26>
[{initial_call,{proc_lib,init_p,5}},
 {current_function,{erlang,bif_return_trap,2}},
 {registered_name,emqx_pool_7},
 {status,suspended},
 {message_queue_len,6},
 {group_leader,<0.1424.0>},
 {priority,normal},
 {trap_exit,false},
 {reductions,192508},
 {last_calls,false},
 {catchlevel,3},
 {trace,0},
 {suspending,[]},
 {sequential_trace_token,[]},
 {error_handler,error_handler},
 {memory,700416},
 {total_heap_size,87358},
 {heap_size,10958},
 {stack_size,37},
 {min_heap_size,233}]
[{name,"tcp_inet"},
 {links,[<0.1843.0>,<0.1305.0>]},
 {id,208},
 {connected,<0.1843.0>},
 {input,34195357},
 {output,51},
 {os_pid,undefined}]
2021-04-19 14:23:04.185 [warning] [SYSMON] large_heap warning: pid = <0.1601.0>, info: [{old_heap_block_size,8912793},
                                             {heap_block_size,1199557},
                                             {mbuf_size,0},
                                             {stack_size,32},
                                             {old_heap_size,3650620},
                                             {heap_size,347835}]
[{initial_call,{proc_lib,init_p,5}},
 {current_function,{emqx_misc,drain_down,2}},
 {registered_name,emqx_cm},
 {status,running},
 {message_queue_len,2722},
 {group_leader,<0.1424.0>},
 {priority,normal},
 {trap_exit,false},
 {reductions,19393911},
 {last_calls,false},
 {catchlevel,2},
 {trace,0},
 {suspending,[]},
 {sequential_trace_token,[]},
 {error_handler,error_handler},
 {memory,95488164},
 {total_heap_size,10220873},
 {heap_size,1199557},
 {stack_size,27},
 {min_heap_size,233}]
2021-04-19 14:25:49.193 [warning] [SYSMON] long_schedule warning: pid = <0.9295.31>, info: [{timeout,359},
                                                 {in,{gen_server,rec_nodes,7}},
                                                 {out,
                                                  {emqx_connection,recvloop,
                                                   2}}]
[{initial_call,{proc_lib,init_p,5}},
 {current_function,{emqx_connection,recvloop,2}},
 {registered_name,[]},
 {status,waiting},
 {message_queue_len,0},
 {group_leader,<0.1397.0>},
 {priority,normal},
 {trap_exit,false},
 {reductions,2206},
 {last_calls,false},
 {catchlevel,1},
 {trace,0},
 {suspending,[]},
 {sequential_trace_token,[]},
 {error_handler,error_handler},
 {memory,34556},
 {total_heap_size,4184},
 {heap_size,2586},
 {stack_size,9},
 {min_heap_size,233}]
2021-04-19 14:25:49.201 [warning] [SYSMON] long_schedule warning: pid = <0.3272.22>, info: [{timeout,360},
                                                 {in,undefined},
                                                 {out,
                                                  {emqx_connection,recvloop,
                                                   2}}]
[{initial_call,{proc_lib,init_p,5}},
 {current_function,{emqx_connection,recvloop,2}},
 {registered_name,[]},
 {status,waiting},
 {message_queue_len,0},
 {group_leader,<0.1397.0>},
 {priority,normal},
 {trap_exit,false},
 {reductions,2654},
 {last_calls,false},
 {catchlevel,1},
 {trace,0},
 {suspending,[]},
 {sequential_trace_token,[]},
 {error_handler,error_handler},
 {memory,9028},
 {total_heap_size,993},
 {heap_size,376},
 {stack_size,9},
 {min_heap_size,233}]

相关截图

要不直接用 centos 系统? 不要用虚拟机了…