message_queue_too_long 异常导致服务端连接断开

环境

  • EMQX 版本:5.1.6
  • 操作系统版本:ubtu 20.0.4

重现此问题的步骤

1.压力测试,服务端当做设备端连接EMQTX,订阅了连接主题
2. 1秒1万条数据,服务端接收断开

出现异常:
2023-12-15T06:35:35.773310+00:00 [error] supervisor: ‘esockd_connection_sup - <0.2786.0>’, errorContext: connection_shutdown, reason: #{max => 1000,reason => message_queue_too_long,value => 5426}, offender: [{pid,<0.10555.3>},{name,connection},{mfargs,{emqx_connection,start_link,[#{enable_authn => true,limiter => undefined,listener => {tcp,default},zone => default}]}}]

我需要怎么处理这个问题,或者优化

优化/细分订阅端订阅的主题,使用共享订阅,或订阅更细粒度的主题。
看这条日志是因为这个客户端订阅的主题有过多的消息,短时间内大量的消息涌向同一个客户端会导致它的压力过大。

1 个赞

这个客户端是我的服务器,我做了扩展,所以在不考虑队列大小的情况我应该怎么优化,增加队列大小,或者说增加内存提升服务器性能?

MQTT 基本参数

增大 mqtt.max_mqueue_len 配置项,该项默认值为 1000。

设置为0和设置为100000都会打印异常,服务端还是断开了,实际设置没有效果,5.1.16版本

{“time”:1703147090879911,“pid”:“<0.2821.0>”,“msg”:“supervisor: ‘esockd_connection_sup - <0.2821.0>’, errorContext: connection_shutdown, reason: #{max => 1000,reason => message_queue_too_long,value => 2999}, offender: [{pid,<0.3303.0>},{name,connection},{mfargs,{emqx_connection,start_link,[#{enable_authn => true,limiter => undefined,listener => {tcp,default},zone => default}]}}]”,“level”:“error”,“error_logger”:{“type”:“supervisor_report”,“tag”:“error_report”},“domain”:[“supervisor_report”]}

客户端断开以后,重新连接会出现警告日志
emqx | 2023-12-21T17:02:25.258846+08:00 warning: {“time”:1703149345258846,“pid”:“<0.2773.0>”,“name”:“conn_congestion/inbound-devicehub-service/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.e30.i8ll_R7GzAHLZFr6LYbVjK48wur-ltjcBva74h4nUCg”,“msg”:“alarm_is_activated”,“message”:“connection congested: #{buffer => 4096,clientid => <<"inbound-devicehub-service">>,conn_state => connected,connected_at => 1703149257741,high_msgq_watermark => 8192,high_watermark => 1048576,memory => 394344,message_queue_len => 246,peername => <<"172.20.0.6:50706">>,pid => <<"<0.1717.1>">>,proto_name => <<"MQTT">>,proto_ver => 4,recbuf => 131072,recv_cnt => 3,recv_oct => 262,reductions => 286171,send_cnt => 369,send_oct => 307419,send_pend => 19924,sndbuf => 87040,sockname => <<"172.20.0.3:1883">>,socktype => tcp,username => <<"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.e30.i8ll_R7GzAHLZFr6LYbVjK48wur-ltjcBva74h4nUCg">>}”,“line”:418,“level”:“warning”,“mfa”:“emqx_alarm:do_actions/3”}
emqx | 2023-12-21T17:03:57.753092+08:00 warning: {“time”:1703149437753092,“pid”:“<0.2773.0>”,“name”:“conn_congestion/inbound-devicehub-service/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.e30.i8ll_R7GzAHLZFr6LYbVjK48wur-ltjcBva74h4nUCg”,“msg”:“alarm_is_deactivated”,“line”:424,“level”:“warning”,“mfa”:“emqx_alarm:do_actions/3”}

并且设置了不清除会话,重连上来队列消息也没重新下发,也没看到丢弃消息
$SYS/brokers/emqx@127.0.0.1/clients/+/disconnected是系统主题消息

设置