emqx客户端连接数上不去

EMQX 性能问题
,
1

EMQX 版本

4.4.7

EMQX 安装部署方式

集群部署

EMQX 集群情况

三个节点

服务器（运行 EMQX 的机器）硬件配置

节点规格为4核32GB

服务器操作系统和平台

节点os为aliyun_2_1903_x64_20G_alibase_20221102

服务端参数优化情况

已按照教程优化

压力机硬件配置

4核8G

压力机使用的测试工具

自建脚本

压力机参数优化情况

无

在 EMQX 中启用的功能

ACL，exhook，规则引擎

测试场景

压力机共6台，每台以每秒100的速率进行mqtt连接，经过slb接入emqx，最终每台压力机将建立5万连接。

具体问题

数量达到28万连接数时，emqx后台出现错误日志，dashbord接口全部超时。

  • emqx日志
    2023-04-10T18:36:41.893999+08:00 [error] 219801A40RAXF0123570@192.168.0.175:3700 [ExHook Svr] CALL emqx_exhook_v_1_hook_provider_client:on_client_connack(#{conninfo => #{clientid => <<“219801A40RAXF0123570”>>,keepalive => 60,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,peerhost => <<“192.168.0.175”>>,proto_name => <<“MQTT”>>,proto_ver => <<“4”>>,sockport => 8883,username => <<“219801A40RAXF0123570”>>},meta => #{cluster_name => “emqxcl”,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,sysdescr => “EMQ X Broker”,version => “4.4.7”},props => [],result_code => <<“success”>>}, #{channel => “default”,key_dispatch => <<“219801A40RAXF0123570”>>,pool_size => 4,timeout => 5000}) error: {deadline_exceeded,<<“Waiting for response timeout”>>}
    2023-04-10T18:36:41.894077+08:00 [error] 219801A40R95DML30955@10.242.0.1:57485 [ExHook Svr] CALL emqx_exhook_v_1_hook_provider_client:on_client_connack(#{conninfo => #{clientid => <<“219801A40R95DML30955”>>,keepalive => 60,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,peerhost => <<“10.242.0.1”>>,proto_name => <<“MQTT”>>,proto_ver => <<“4”>>,sockport => 8883,username => <<“219801A40R95DML30955”>>},meta => #{cluster_name => “emqxcl”,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,sysdescr => “EMQ X Broker”,version => “4.4.7”},props => [],result_code => <<“success”>>}, #{channel => “default”,key_dispatch => <<“219801A40R95DML30955”>>,pool_size => 4,timeout => 5000}) error: {deadline_exceeded,<<“Waiting for response timeout”>>}
    2023-04-10T18:36:41.654344+08:00 [warning] action: discard, file: emqx_cm.erl, line: 345, mfa: {emqx_cm,request_stepdown,3}, msg: session_stepdown_request_timeout, pid: <0.30699.343>, stale_channel: [{status,waiting},{message_queue_len,6},{current_stacktrace,[{gen,do_call,4,[{file,“gen.erl”},{line,214}]},{gen_server,call,3,[{file,“gen_server.erl”},{line,243}]},{emqx_session,subscribe,4,[{file,“emqx_session.erl”},{line,280}]},{emqx_channel,do_subscribe,3,[{file,“emqx_channel.erl”},{line,689}]},{lists,foldl,3,[{file,“lists.erl”},{line,1267}]},{emqx_channel,handle_info,2,[{file,“emqx_channel.erl”},{line,1022}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,694}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,394}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,358}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]}]
    2023-04-10T18:36:41.664695+08:00 [warning] action: discard, file: emqx_cm.erl, line: 345, mfa: {emqx_cm,request_stepdown,3}, msg: session_stepdown_request_timeout, pid: <0.30946.343>, stale_channel: [{status,waiting},{message_queue_len,6},{current_stacktrace,[{gen,do_call,4,[{file,“gen.erl”},{line,214}]},{gen_server,call,3,[{file,“gen_server.erl”},{line,243}]},{emqx_session,subscribe,4,[{file,“emqx_session.erl”},{line,280}]},{emqx_channel,do_subscribe,3,[{file,“emqx_channel.erl”},{line,689}]},{lists,foldl,3,[{file,“lists.erl”},{line,1267}]},{emqx_channel,handle_info,2,[{file,“emqx_channel.erl”},{line,1022}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,694}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,394}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,358}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]}]
    2023-04-10T18:36:41.900586+08:00 [error] 219801A40R95DML34922@10.242.0.1:13128 [ExHook Svr] CALL emqx_exhook_v_1_hook_provider_client:on_client_connected(#{clientinfo => #{anonymous => false,clientid => <<“219801A40R95DML34922”>>,cn => <<>>,dn => <<>>,is_superuser => false,mountpoint => <<>>,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,password => <<>>,peerhost => <<“10.242.0.1”>>,protocol => <<“mqtt”>>,sockport => 8883,username => <<“219801A40R95DML34922”>>},meta => #{cluster_name => “emqxcl”,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,sysdescr => “EMQ X Broker”,version => “4.4.7”}}, #{channel => “default”,key_dispatch => <<“219801A40R95DML34922”>>,pool_size => 4,timeout => 5000}) error: {deadline_exceeded,<<“Waiting for response timeout”>>}
    2023-04-10T18:36:41.897762+08:00 [error] 219801A40RAXF0240970@192.168.0.177:54599 [ExHook Svr] CALL emqx_exhook_v_1_hook_provider_client:on_client_connected(#{clientinfo => #{anonymous => false,clientid => <<“219801A40RAXF0240970”>>,cn => <<>>,dn => <<>>,is_superuser => false,mountpoint => <<>>,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,password => <<>>,peerhost => <<“192.168.0.177”>>,protocol => <<“mqtt”>>,sockport => 8883,username => <<“219801A40RAXF0240970”>>},meta => #{cluster_name => “emqxcl”,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,sysdescr => “EMQ X Broker”,version => “4.4.7”}}, #{channel => “default”,key_dispatch => <<“219801A40RAXF0240970”>>,pool_size => 4,timeout => 5000}) error: {deadline_exceeded,<<“Waiting for response timeout”>>}
    2023-04-10T18:36:41.896868+08:00 [warning] action: discard, file: emqx_cm.erl, line: 345, mfa: {emqx_cm,request_stepdown,3}, msg: session_stepdown_request_timeout, pid: <0.31093.343>, stale_channel: [{status,waiting},{message_queue_len,6},{current_stacktrace,[{gen,do_call,4,[{file,“gen.erl”},{line,214}]},{gen_server,call,3,[{file,“gen_server.erl”},{line,243}]},{emqx_session,subscribe,4,[{file,“emqx_session.erl”},{line,280}]},{emqx_channel,do_subscribe,3,[{file,“emqx_channel.erl”},{line,689}]},{lists,foldl,3,[{file,“lists.erl”},{line,1267}]},{emqx_channel,handle_info,2,[{file,“emqx_channel.erl”},{line,1022}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,694}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,394}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,358}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]}]
    2023-04-10T18:36:41.901504+08:00 [error] 219801A40RAXF0116125@10.242.0.1:26585 [ExHook Svr] CALL emqx_exhook_v_1_hook_provider_client:on_client_connected(#{clientinfo => #{anonymous => false,clientid => <<“219801A40RAXF0116125”>>,cn => <<>>,dn => <<>>,is_superuser => false,mountpoint => <<>>,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,password => <<>>,peerhost => <<“10.242.0.1”>>,protocol => <<“mqtt”>>,sockport => 8883,username => <<“219801A40RAXF0116125”>>},meta => #{cluster_name => “emqxcl”,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,sysdescr => “EMQ X Broker”,version => “4.4.7”}}, #{channel => “default”,key_dispatch => <<“219801A40RAXF0116125”>>,pool_size => 4,timeout => 5000}) error: {deadline_exceeded,<<“Waiting for response timeout”>>}
    2023-04-10T18:36:41.898794+08:00 [warning] 219801A40RDWEEQ37537@192.168.0.175:62140 action: discard, file: emqx_cm.erl, line: 345, mfa: {emqx_cm,request_stepdown,3}, msg: session_stepdown_request_timeout, pid: <0.6413.318>, stale_channel: [{status,waiting},{message_queue_len,6},{current_stacktrace,[{gen,do_call,4,[{file,“gen.erl”},{line,214}]},{gen_server,call,3,[{file,“gen_server.erl”},{line,243}]},{emqx_session,subscribe,4,[{file,“emqx_session.erl”},{line,280}]},{emqx_channel,do_subscribe,3,[{file,“emqx_channel.erl”},{line,689}]},{lists,foldl,3,[{file,“lists.erl”},{line,1267}]},{emqx_channel,handle_info,2,[{file,“emqx_channel.erl”},{line,1022}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,694}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,394}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,358}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]}]
    2023-04-10T18:36:41.909005+08:00 [error] 219801A40RAXF0218865@192.168.0.177:51890 [ExHook Svr] CALL emqx_exhook_v_1_hook_provider_client:on_client_connected(#{clientinfo => #{anonymous => false,clientid => <<“219801A40RAXF0218865”>>,cn => <<>>,dn => <<>>,is_superuser => false,mountpoint => <<>>,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,password => <<>>,peerhost => <<“192.168.0.177”>>,protocol => <<“mqtt”>>,sockport => 8883,username => <<“219801A40RAXF0218865”>>},meta => #{cluster_name => “emqxcl”,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,sysdescr => “EMQ X Broker”,version => “4.4.7”}}, #{channel => “default”,key_dispatch => <<“219801A40RAXF0218865”>>,pool_size => 4,timeout => 5000}) error: {deadline_exceeded,<<“Waiting for response timeout”>>}
    2023-04-10T18:36:41.929750+08:00 [error] 219801A40RQGD0135120@10.242.0.1:3652 [ExHook Svr] CALL emqx_exhook_v_1_hook_provider_client:on_client_connected(#{clientinfo => #{anonymous => false,clientid => <<“219801A40RQGD0135120”>>,cn => <<>>,dn => <<>>,is_superuser => false,mountpoint => <<>>,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,password => <<>>,peerhost => <<“10.242.0.1”>>,protocol => <<“mqtt”>>,sockport => 8883,username => <<“219801A40RQGD0135120”>>},meta => #{cluster_name => “emqxcl”,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,sysdescr => “EMQ X Broker”,version => “4.4.7”}}, #{channel => “default”,key_dispatch => <<“219801A40RQGD0135120”>>,pool_size => 4,timeout => 5000}) error: {deadline_exceeded,<<“Waiting for response timeout”>>}
    2023-04-10T18:36:41.921280+08:00 [error] 219801A40RAXF0231412@192.168.0.175:19831 [ExHook Svr] CALL emqx_exhook_v_1_hook_provider_client:on_client_connack(#{conninfo => #{clientid => <<“219801A40RAXF0231412”>>,keepalive => 60,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,peerhost => <<“192.168.0.175”>>,proto_name => <<“MQTT”>>,proto_ver => <<“4”>>,sockport => 8883,username => <<“219801A40RAXF0231412”>>},meta => #{cluster_name => “emqxcl”,node => <<“int-rd40@int-rd40-emqx-0.int-rd40-emqx-headless.iot.svc.cluster.local”>>,sysdescr => “EMQ X Broker”,version => “4.4.7”},props => [],result_code => <<“success”>>}, #{channel => “default”,key_dispatch => <<“219801A40RAXF0231412”>>,pool_size => 4,timeout => 5000}) error: {deadline_exceeded,<<“Waiting for response timeout”>>}
  • exhook日志
    2023-04-10 17:48:42.388 [grpc-default-worker-ELG-3-1] WARN io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler - Stream Error
    io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 49375
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:147)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:596)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:239)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:422)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:174)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
    at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
    at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
    2023-04-10 17:48:42.488 [grpc-default-worker-ELG-3-1] WARN io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler - Stream Error
    io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 49373
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:147)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:596)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:239)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:422)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:174)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
    at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
    at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
    2023-04-10 17:48:42.578 [grpc-default-worker-ELG-3-1] WARN io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler - Stream Error
    io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 49371
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:147)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:596)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:239)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:422)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:174)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
2

为了排除外部插件的原因，目前已经关闭了exhook和emqx_auth_mongo

3

意思是关闭了 exhook 之后，仍然连不上去吗？我看到你提供的 emqx 日志里，都是 exhook timeout 的凑无。

4

image
image2060×462 150 KB

这个是关闭exhook之后的日志

5

这个看起来像是使用重复的 clientid 登录的时候，尝试踢掉之前的 MQTT 连接的时候 timeout 了。

  • 首先是该 clientid 已经连接到 emqx 了，并且看日志是连接到了另外一个节点上面。
  • 然后那个 client 被卡住了，可能是某个插件导致的，那个节点上的 exhook 或者什么插件没关掉吗？
  • 升级到 4.4.16 可以解决踢连接 timeout 的问题，当 timeout 出现的时候，emqx 会强制杀死对面的那个连接。
6
  • 每台压测机生成的clientid都是唯一的，不可能出现重复的clientid。
  • exhook和emqx_auth_mongo都已经关闭了。
  • 目前用的4.4.7版本和我们线上的环境保持一致。