服务运行一段时间就down了

环境

  • EMQX 版本: [5.0.8]
  • 操作系统版本:Ubuntu 20.04.3 LTS

重现此问题的步骤

  1. 启动服务后,连接数达到930
  2. 大约10分钟左右就down了

预期行为

实际行为

[error] Error in process <0.3985.0> on node ‘emqx@127.0.0.1’ with exit value:, {badarg,[{os,cmd,[[112,115,32,45,111,32,112,99,112,117,44,112,109,101,109,32,“776490”]],[{file,“os.erl”},{line,419},{error_info,#{module => erl_kernel_errors}}]},{observer_cli,render_system_line,2,[{file,“observer_cli.erl”},{line,220}]},{observer_cli,redraw_running,8,[{file,“observer_cli.erl”},{line,194}]}]}
2023-12-09T13:53:17.014642+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:18.323144+08:00 [warning] exception: error, line: 116, mfa: minirest_handler:apply_callback/3, path: /nodes, reason: function_clause, stacktrace: [{emqx_mgmt_api_nodes,format,[‘emqx@127.0.0.1’,{error,{‘EXIT’,{{case_clause,{error,emfile}},[{emqx_vm,read_otp_version,0,[{file,“emqx_vm.erl”},{line,410}]},{emqx_mgmt,otp_rel,0,[{file,“emqx_mgmt.erl”},{line,533}]},{emqx_mgmt,node_info,0,[{file,“emqx_mgmt.erl”},{line,131}]}]}}}],[{file,“emqx_mgmt_api_nodes.erl”},{line,290}]},{emqx_mgmt_api_nodes,‘-list_nodes/1-lc$^0/1-0-’,1,[{file,“emqx_mgmt_api_nodes.erl”},{line,260}]},{emqx_mgmt_api_nodes,list_nodes,1,[{file,“emqx_mgmt_api_nodes.erl”},{line,260}]},{minirest_handler,apply_callback,3,[{file,“minirest_handler.erl”},{line,111}]},{minirest_handler,handle,2,[{file,“minirest_handler.erl”},{line,44}]},{minirest_handler,init,2,[{file,“minirest_handler.erl”},{line,27}]},{cowboy_handler,execute,2,[{file,“cowboy_handler.erl”},{line,41}]},{cowboy_stream_h,execute,3,[{file,“cowboy_stream_h.erl”},{line,318}]},{cowboy_stream_h,request_process,3,[{file,“cowboy_stream_h.erl”},{line,302}]},{proc_lib,init_p_do_apply,3,[{file,“proc_lib.erl”},{line,226}]}]
2023-12-09T13:53:20.331321+08:00 [warning] exception: error, line: 116, mfa: minirest_handler:apply_callback/3, path: /nodes, reason: function_clause, stacktrace: [{emqx_mgmt_api_nodes,format,[‘emqx@127.0.0.1’,{error,{‘EXIT’,{{case_clause,{error,emfile}},[{emqx_vm,read_otp_version,0,[{file,“emqx_vm.erl”},{line,410}]},{emqx_mgmt,otp_rel,0,[{file,“emqx_mgmt.erl”},{line,533}]},{emqx_mgmt,node_info,0,[{file,“emqx_mgmt.erl”},{line,131}]}]}}}],[{file,“emqx_mgmt_api_nodes.erl”},{line,290}]},{emqx_mgmt_api_nodes,‘-list_nodes/1-lc$^0/1-0-’,1,[{file,“emqx_mgmt_api_nodes.erl”},{line,260}]},{emqx_mgmt_api_nodes,list_nodes,1,[{file,“emqx_mgmt_api_nodes.erl”},{line,260}]},{minirest_handler,apply_callback,3,[{file,“minirest_handler.erl”},{line,111}]},{minirest_handler,handle,2,[{file,“minirest_handler.erl”},{line,44}]},{minirest_handler,init,2,[{file,“minirest_handler.erl”},{line,27}]},{cowboy_handler,execute,2,[{file,“cowboy_handler.erl”},{line,41}]},{cowboy_stream_h,execute,3,[{file,“cowboy_stream_h.erl”},{line,318}]},{cowboy_stream_h,request_process,3,[{file,“cowboy_stream_h.erl”},{line,302}]},{proc_lib,init_p_do_apply,3,[{file,“proc_lib.erl”},{line,226}]}]
2023-12-09T13:53:22.323482+08:00 [warning] exception: error, line: 116, mfa: minirest_handler:apply_callback/3, path: /nodes, reason: function_clause, stacktrace: [{emqx_mgmt_api_nodes,format,[‘emqx@127.0.0.1’,{error,{‘EXIT’,{{case_clause,{error,emfile}},[{emqx_vm,read_otp_version,0,[{file,“emqx_vm.erl”},{line,410}]},{emqx_mgmt,otp_rel,0,[{file,“emqx_mgmt.erl”},{line,533}]},{emqx_mgmt,node_info,0,[{file,“emqx_mgmt.erl”},{line,131}]}]}}}],[{file,“emqx_mgmt_api_nodes.erl”},{line,290}]},{emqx_mgmt_api_nodes,‘-list_nodes/1-lc$^0/1-0-’,1,[{file,“emqx_mgmt_api_nodes.erl”},{line,260}]},{emqx_mgmt_api_nodes,list_nodes,1,[{file,“emqx_mgmt_api_nodes.erl”},{line,260}]},{minirest_handler,apply_callback,3,[{file,“minirest_handler.erl”},{line,111}]},{minirest_handler,handle,2,[{file,“minirest_handler.erl”},{line,44}]},{minirest_handler,init,2,[{file,“minirest_handler.erl”},{line,27}]},{cowboy_handler,execute,2,[{file,“cowboy_handler.erl”},{line,41}]},{cowboy_stream_h,execute,3,[{file,“cowboy_stream_h.erl”},{line,318}]},{cowboy_stream_h,request_process,3,[{file,“cowboy_stream_h.erl”},{line,302}]},{proc_lib,init_p_do_apply,3,[{file,“proc_lib.erl”},{line,226}]}]
2023-12-09T13:53:24.323822+08:00 [warning] exception: error, line: 116, mfa: minirest_handler:apply_callback/3, path: /nodes, reason: function_clause, stacktrace: [{emqx_mgmt_api_nodes,format,[‘emqx@127.0.0.1’,{error,{‘EXIT’,{{case_clause,{error,emfile}},[{emqx_vm,read_otp_version,0,[{file,“emqx_vm.erl”},{line,410}]},{emqx_mgmt,otp_rel,0,[{file,“emqx_mgmt.erl”},{line,533}]},{emqx_mgmt,node_info,0,[{file,“emqx_mgmt.erl”},{line,131}]}]}}}],[{file,“emqx_mgmt_api_nodes.erl”},{line,290}]},{emqx_mgmt_api_nodes,‘-list_nodes/1-lc$^0/1-0-’,1,[{file,“emqx_mgmt_api_nodes.erl”},{line,260}]},{emqx_mgmt_api_nodes,list_nodes,1,[{file,“emqx_mgmt_api_nodes.erl”},{line,260}]},{minirest_handler,apply_callback,3,[{file,“minirest_handler.erl”},{line,111}]},{minirest_handler,handle,2,[{file,“minirest_handler.erl”},{line,44}]},{minirest_handler,init,2,[{file,“minirest_handler.erl”},{line,27}]},{cowboy_handler,execute,2,[{file,“cowboy_handler.erl”},{line,41}]},{cowboy_stream_h,execute,3,[{file,“cowboy_stream_h.erl”},{line,318}]},{cowboy_stream_h,request_process,3,[{file,“cowboy_stream_h.erl”},{line,302}]},{proc_lib,init_p_do_apply,3,[{file,“proc_lib.erl”},{line,226}]}]
2023-12-09T13:53:24.398700+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.398814+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.398874+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.398940+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.398979+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.399005+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.399050+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.399179+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.399106+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.399155+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.399185+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.399239+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.399257+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.399276+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.399324+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:24.399310+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:26.330263+08:00 [warning] exception: error, line: 116, mfa: minirest_handler:apply_callback/3, path: /nodes, reason: function_clause, stacktrace: [{emqx_mgmt_api_nodes,format,[‘emqx@127.0.0.1’,{error,{‘EXIT’,{{case_clause,{error,emfile}},[{emqx_vm,read_otp_version,0,[{file,“emqx_vm.erl”},{line,410}]},{emqx_mgmt,otp_rel,0,[{file,“emqx_mgmt.erl”},{line,533}]},{emqx_mgmt,node_info,0,[{file,“emqx_mgmt.erl”},{line,131}]}]}}}],[{file,“emqx_mgmt_api_nodes.erl”},{line,290}]},{emqx_mgmt_api_nodes,‘-list_nodes/1-lc$^0/1-0-’,1,[{file,“emqx_mgmt_api_nodes.erl”},{line,260}]},{emqx_mgmt_api_nodes,list_nodes,1,[{file,“emqx_mgmt_api_nodes.erl”},{line,260}]},{minirest_handler,apply_callback,3,[{file,“minirest_handler.erl”},{line,111}]},{minirest_handler,handle,2,[{file,“minirest_handler.erl”},{line,44}]},{minirest_handler,init,2,[{file,“minirest_handler.erl”},{line,27}]},{cowboy_handler,execute,2,[{file,“cowboy_handler.erl”},{line,41}]},{cowboy_stream_h,execute,3,[{file,“cowboy_stream_h.erl”},{line,318}]},{cowboy_stream_h,request_process,3,[{file,“cowboy_stream_h.erl”},{line,302}]},{proc_lib,init_p_do_apply,3,[{file,“proc_lib.erl”},{line,226}]}]
2023-12-09T13:53:40.621632+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:41.717345+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.424814+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.424945+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.425069+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.424937+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.424816+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.424917+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.425126+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.424987+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.424962+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.425026+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.425058+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.425159+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.425237+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.425265+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.425060+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:53:53.425242+08:00 [error] Accept error on 0.0.0.0:8883: emfile
2023-12-09T13:54:00.336376+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.336502+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.336599+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.336667+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.437599+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.437717+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.437669+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.437738+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.538739+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.539193+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.539033+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.538916+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.639674+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.639835+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.639759+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.639966+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.740693+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.740689+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.740809+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.740760+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.841614+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.841678+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.841752+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.841717+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.942677+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.942746+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.942780+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:00.942715+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.043707+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.043665+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.043755+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.043785+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.144712+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.144819+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.144867+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.144668+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.245654+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.245777+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.245809+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.245964+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.346687+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.346751+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.346651+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.346784+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.447641+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.447782+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.447676+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.447750+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.548622+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.548725+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.548667+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.548746+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.649645+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.649728+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.649690+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.649770+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.750673+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.750751+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.750836+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.750669+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.851688+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.851766+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.851799+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.851824+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.952713+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.952678+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.952767+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:01.952793+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.053671+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.053768+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.053799+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.053844+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.154669+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.154798+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.154827+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.154865+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.255697+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.255797+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.255852+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.255869+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.356720+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.356815+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.356854+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.356897+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.457628+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.457691+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.457718+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.457752+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.558624+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.558730+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.558781+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.558794+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.659602+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.659706+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.659662+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.659670+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.760668+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.760705+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.760746+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.760654+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.861669+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.861770+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.861764+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.861670+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.962699+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.962710+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.962821+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:02.962793+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:03.063702+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:03.063692+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:03.063787+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:03.063817+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:03.164688+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:03.164667+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:03.164753+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:03.164785+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:03.265673+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:03.265688+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors
2023-12-09T13:54:03.265757+08:00 [warning] Ranch acceptor reducing accept rate: out of file descriptors

是说文件描述符不够了。请参照该文档调优系统参数:系统调优 | EMQX 文档

系统调优那里的参数我基本都设了,请教一下,如果我多拉2个服务器,效果是不是好点,有一点服务器配置性能不太一样,是不是能提过整个的负载,主要是连接数提高了,有1000多个客户端,才出现这种情况的

这个能确定就是文件描述符的问题。你可以看看当前的 ulimit 值是多少。
如果用systemctl 启动的话,这个 limit 值是必须配置到 service 文件里的。