等待来自服务器的响应时超时

695067018 · 2023 年10 月 13 日 07:23

环境

EMQX 版本：5.0.26
操作系统版本：CentOS Linux release 7.9.2009

总共1000台左右的设备连接，批量发送30个mqtt消息指令qos为2的保留消息，就会出现超时
org.eclipse.paho.mqttv5.common.MqttException: 等待来自服务器的响应时超时
at org.eclipse.paho.mqttv5.client.internal.Token.waitForCompletion(Token.java:140)
at org.eclipse.paho.mqttv5.client.MqttToken.waitForCompletion(MqttToken.java:76)
at org.jeecg.modules.equ.handler.EquUpHandler.response(EquUpHandler.java:115)
at sun.reflect.GeneratedMethodAccessor586.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jeecg.modules.mqtt.mapping.MappingResolver.commandExec(MappingResolver.java:219)
at org.jeecg.modules.mqtt.mapping.MappingResolver$$FastClassBySpringCGLIB$$298a5cbb.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
at org.jeecg.modules.mqtt.mapping.MappingResolver$$EnhancerBySpringCGLIB$$32237434.commandExec()
at org.jeecg.modules.mqtt.mqtt.MqttService.messageArrived(MqttService.java:480)
at org.eclipse.paho.mqttv5.client.internal.CommsCallback.deliverMessage(CommsCallback.java:637)
at org.eclipse.paho.mqttv5.client.internal.CommsCallback.handleMessage(CommsCallback.java:481)
at org.eclipse.paho.mqttv5.client.internal.CommsCallback.run(CommsCallback.java:224)
at java.lang.Thread.run(Thread.java:750)

JimMoen · 2023 年10 月 13 日 09:20

推荐使用最新版 v5.3.0
使用 log trace 功能调查特定主题/客户端的 pub/sub 行为
保留消息具体的主题/payload大小？
- “批量” 是如何批量发布，单个客户端发布30条/还是每个客户端发布30条/30个客户端每个发布一条？
EMQX 日志麻烦提供下，特别是在出现超时情况时的日志。

695067018 · 2023 年10 月 13 日 09:43

2023-10-12T22:24:30.202577+08:00 [warning] msg: pubcomp_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 482, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 223
2023-10-12T22:24:35.223555+08:00 [warning] msg: pubcomp_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 482, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 227
2023-10-12T22:24:45.224205+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 217
2023-10-12T22:24:50.235452+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 218
2023-10-12T22:24:55.245438+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 219
2023-10-12T22:25:00.255821+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 221
2023-10-12T22:25:05.267803+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 222
2023-10-12T22:25:05.307520+08:00 [warning] msg: pubrec_packetId_inuse, mfa: emqx_channel:handle_in/2, line: 443, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 233
2023-10-12T22:25:10.278372+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 224
2023-10-12T22:25:15.290371+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 225
2023-10-12T22:25:20.302149+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 226
2023-10-12T22:25:25.314443+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 228
2023-10-12T22:25:25.353645+08:00 [warning] msg: pubrec_packetId_inuse, mfa: emqx_channel:handle_in/2, line: 443, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 237
2023-10-12T22:25:30.669527+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 229
2023-10-12T22:25:35.680013+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 230
2023-10-12T22:25:40.707890+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 231
2023-10-12T22:25:40.747578+08:00 [warning] msg: pubrec_packetId_inuse, mfa: emqx_channel:handle_in/2, line: 443, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 239
2023-10-12T22:25:45.721281+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 232
2023-10-12T22:25:45.760613+08:00 [warning] msg: pubrec_packetId_inuse, mfa: emqx_channel:handle_in/2, line: 443, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 241
2023-10-12T22:25:50.744287+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 234
2023-10-12T22:25:50.783614+08:00 [warning] msg: pubrec_packetId_inuse, mfa: emqx_channel:handle_in/2, line: 443, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 242
2023-10-12T22:25:55.755640+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 235
2023-10-12T22:26:00.768061+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 236
2023-10-12T22:26:05.778907+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 238
2023-10-12T22:26:05.818594+08:00 [warning] msg: pubcomp_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 482, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 227
2023-10-12T22:26:10.790871+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 240
2023-10-12T22:26:10.830609+08:00 [warning] msg: pubrec_packetId_inuse, mfa: emqx_channel:handle_in/2, line: 443, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 243
2023-10-12T22:26:15.801620+08:00 [warning] msg: puback_packetId_not_found, mfa: emqx_channel:handle_in/2, line: 428, peername: 10.88.75.22:46396, clientid: client-prod, packetId: 225

695067018 · 2023 年10 月 13 日 09:58

主题消息的大小大概200多字节，单个客户端发送30多条消息

JimMoen · 2023 年10 月 13 日 10:13

你可能需要对照着 mqtt 协议，调查下 client 在给 EMQX 回 puback/pubrec/pubcomp 时候的 packetid 是否正确。

可以先试着每次只发送一两个 qos2 的保留消息，并且降低发送频率

695067018 · 2023 年10 月 17 日 01:44

每次只发送一两个qos2的保留消息是没问题啊，批量发送就报错了，而且批量也不算多，30个左右。业务上要求的批量比30个还多。