qos0,qos1,qos2都有消息丢失

EMQX 错误报告
, ,
1
  • EMQX 版本：5.0.21
  • 操作系统版本：centos
    qos0,qos1,qos2都有消息丢失，但是日志里面没有任何error或warn的日志，请问如何查找消息丢失的原因
    image
    image953×609 17.8 KB

企业微信截图_17350122114705
企业微信截图_173501221147051920×894 69.2 KB

2

可以来一个完整的 emqx 日志看看么

3

emqx_20241224.zip (64.6 KB)
您好，日志已上传

4

emqx.log.1.zip (8.1 KB)
还有一个日志

5

环境

  • EMQX 版本：5.0.21
  • 操作系统版本：centos

消息丢失

2024-12-24T16:37:49.565554+08:00 [error] msg: failed_to_execute, mfa: emqx_hooks:safe_execute/2, line: 210, peername: 172.18.0.1:38488, clientid: uiot-web_Chrome_206qoXNg, exception: error, failed_call: {emqx_topic_metrics,on_message_delivered,[#{clientid => <<"uiot-web_Chrome_206qoXNg">>,enable_authn => true,is_bridge => false,is_superuser => true,listener => 'ws:default',mountpoint => undefined,peerhost => {172,18,0,1},protocol => mqtt,sockport => 8083,username => <<"platform">>,ws_cookie => [{<<"JSESSIONID">>,<<"MCvKSditHP2olwHTDAkvYl86maRrkGTb5-kf92gr">>}],zone => default},{message,<<0,6,42,0,7,53,77,87,244,67,178,4,29,171,66,244>>,0,<<"admin;cloud;1734934037">>,#{dup => false,retain => false},#{peerhost => {172,18,0,1},properties => #{},proto_ver => 4,protocol => mqtt,username => <<"platform">>},<<"/UIOT/iot/v1/device/gqOGm4376vR/analogChange">>,<<"{\"mrid\":\"gqOGm4376vR:Ia1\",\"quality\":1,\"value\":12,\"time\":0}">>,1735029469564,[]}]}, reason: badarg, stacktrace: [{ets,member,[emqx_topic_metrics,<<"/UIOT/iot/v1/device/gqOGm4376vR/analogChange">>],[{error_info,#{cause => id,module => erl_stdlib_errors}}]},{emqx_topic_metrics,is_registered,1,[{file,"emqx_topic_metrics.erl"},{line,182}]},{emqx_topic_metrics,on_message_delivered,2,[{file,"emqx_topic_metrics.erl"},{line,134}]},{emqx_hooks,safe_execute,2,[{file,"emqx_hooks.erl"},{line,200}]},{emqx_hooks,do_run_fold,3,[{file,"emqx_hooks.erl"},{line,180}]},{emqx_channel,do_deliver,2,[{file,"emqx_channel.erl"},{line,1126}]},{emqx_channel,handle_out,3,[{file,"emqx_channel.erl"},{line,1057}]},{emqx_ws_connection,with_channel,3,[{file,"emqx_ws_connection.erl"},{line,747}]},{cowboy_websocket,handler_call,6,[{file,"cowboy_websocket.erl"},{line,487}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]
2024-12-24T16:37:49.565465+08:00 [error] msg: failed_to_execute, mfa: emqx_hooks:safe_execute/2, line: 210, peername: 172.18.0.1:38450, clientid: uiot-web_Chrome_kxwDGBDh, exception: error, failed_call: {emqx_topic_metrics,on_message_delivered,[#{clientid => <<"uiot-web_Chrome_kxwDGBDh">>,enable_authn => true,is_bridge => false,is_superuser => true,listener => 'ws:default',mountpoint => undefined,peerhost => {172,18,0,1},protocol => mqtt,sockport => 8083,username => <<"platform">>,ws_cookie => [{<<"next-auth.session-token">>,<<"eyJhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIn0..MZSUOnXFr9qjoLXy.AX3HLeOvXEb5j4EtJWqjCQpz1vUUvLo4LHptsLV_G30g86nItjliSpPEtwJXbWjxMFdLlv1Yi0VOMRV2Liv24d4yLEDX0xH83b3rXwosd-FxRhGWOk9dX_SFXVSRj9PEICDdxYE4vZwDmrsE2mD92hNaFNxky4fNjg5iRJwWwMHA9sxDx6U2otqcfgQNJIy0aPMYePIMcSzEiOBJO-rRcD3HEDx21rjDJwVacD0jQsEtiTH1oyvZTgy43AMpwZKLsJIT7eJzVl49UW7TpT0.L_eLG1Et0gG3u68G_U-qrg">>},{<<"JSESSIONID">>,<<"g6c5nkhK3PukYp8LGTw6cD4yTQoSeyv5mjLA6ogC">>},{<<"autoDeployment">>,<<"67c0bb3f-5c0e-4195-a755-88956ae04946">>}],zone => default},{message,<<0,6,42,0,7,53,77,87,244,67,178,4,29,171,66,244>>,0,<<"admin;cloud;1734934037">>,#{dup => false,retain => false},#{peerhost => {172,18,0,1},properties => #{},proto_ver => 4,protocol => mqtt,username => <<"platform">>},<<"/UIOT/iot/v1/device/gqOGm4376vR/analogChange">>,<<"{\"mrid\":\"gqOGm4376vR:Ia1\",\"quality\":1,\"value\":12,\"time\":0}">>,1735029469564,[]}]}, reason: badarg, stacktrace: [{ets,member,[emqx_topic_metrics,<<"/UIOT/iot/v1/device/gqOGm4376vR/analogChange">>],[{error_info,#{cause => id,module => erl_stdlib_errors}}]},{emqx_topic_metrics,is_registered,1,[{file,"emqx_topic_metrics.erl"},{line,182}]},{emqx_topic_metrics,on_message_delivered,2,[{file,"emqx_topic_metrics.erl"},{line,134}]},{emqx_hooks,safe_execute,2,[{file,"emqx_hooks.erl"},{line,200}]},{emqx_hooks,do_run_fold,3,[{file,"emqx_hooks.erl"},{line,180}]},{emqx_channel,do_deliver,2,[{file,"emqx_channel.erl"},{line,1126}]},{emqx_channel,handle_out,3,[{file,"emqx_channel.erl"},{line,1057}]},{emqx_ws_connection,with_channel,3,[{file,"emqx_ws_connection.erl"},{line,747}]},{cowboy_websocket,handler_call,6,[{file,"cowboy_websocket.erl"},{line,487}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]
2024-12-24T16:37:49.565623+08:00 [error] msg: failed_to_execute, mfa: emqx_hooks:safe_execute/2, line: 210, peername: 172.18.0.1:39560, clientid: uiot-web_Chrome_YLoixwUn, exception: error, failed_call: {emqx_topic_metrics,on_message_delivered,[#{clientid => <<"uiot-web_Chrome_YLoixwUn">>,enable_authn => true,is_bridge => false,is_superuser => true,listener => 'ws:default',mountpoint => undefined,peerhost => {172,18,0,1},protocol => mqtt,sockport => 8083,username => <<"platform">>,ws_cookie => [{<<"autoDeployment">>,<<"6de96058-6aff-471b-964a-092bf3b8bca2">>},{<<"JSESSIONID">>,<<"hlf0CXKcPZBCyw979UZaJSW9jLsTE22jicT9AOhB">>}],zone => default},{message,<<0,6,42,0,7,53,77,87,244,67,178,4,29,171,66,244>>,0,<<"admin;cloud;1734934037">>,#{dup => false,retain => false},#{peerhost => {172,18,0,1},properties => #{},proto_ver => 4,protocol => mqtt,username => <<"platform">>},<<"/UIOT/iot/v1/device/gqOGm4376vR/analogChange">>,<<"{\"mrid\":\"gqOGm4376vR:Ia1\",\"quality\":1,\"value\":12,\"time\":0}">>,1735029469564,[]}]}, reason: badarg, stacktrace: [{ets,member,[emqx_topic_metrics,<<"/UIOT/iot/v1/device/gqOGm4376vR/analogChange">>],[{error_info,#{cause => id,module => erl_stdlib_errors}}]},{emqx_topic_metrics,is_registered,1,[{file,"emqx_topic_metrics.erl"},{line,182}]},{emqx_topic_metrics,on_message_delivered,2,[{file,"emqx_topic_metrics.erl"},{line,134}]},{emqx_hooks,safe_execute,2,[{file,"emqx_hooks.erl"},{line,200}]},{emqx_hooks,do_run_fold,3,[{file,"emqx_hooks.erl"},{line,180}]},{emqx_channel,do_deliver,2,[{file,"emqx_channel.erl"},{line,1126}]},{emqx_channel,handle_out,3,[{file,"emqx_channel.erl"},{line,1057}]},{emqx_ws_connection,with_channel,3,[{file,"emqx_ws_connection.erl"},{line,747}]},{cowboy_websocket,handler_call,6,[{file,"cowboy_websocket.erl"},{line,487}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]
2024-12-24T16:37:49.566002+08:00 [error] msg: failed_to_execute, mfa: emqx_hooks:safe_execute/2, line: 210, peername: 172.18.0.1:43420, clientid: uiot-web_Chrome_sd3d2vcr, exception: error, failed_call: {emqx_topic_metrics,on_message_delivered,[#{clientid => <<"uiot-web_Chrome_sd3d2vcr">>,enable_authn => true,is_bridge => false,is_superuser => true,listener => 'ws:default',mountpoint => undefined,peerhost => {172,18,0,1},protocol => mqtt,sockport => 8083,username => <<"platform">>,ws_cookie => [{<<"next-auth.session-token">>,<<"eyJhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIn0..MZSUOnXFr9qjoLXy.AX3HLeOvXEb5j4EtJWqjCQpz1vUUvLo4LHptsLV_G30g86nItjliSpPEtwJXbWjxMFdLlv1Yi0VOMRV2Liv24d4yLEDX0xH83b3rXwosd-FxRhGWOk9dX_SFXVSRj9PEICDdxYE4vZwDmrsE2mD92hNaFNxky4fNjg5iRJwWwMHA9sxDx6U2otqcfgQNJIy0aPMYePIMcSzEiOBJO-rRcD3HEDx21rjDJwVacD0jQsEtiTH1oyvZTgy43AMpwZKLsJIT7eJzVl49UW7TpT0.L_eLG1Et0gG3u68G_U-qrg">>},{<<"autoDeployment">>,<<"67c0bb3f-5c0e-4195-a755-88956ae04946">>},{<<"JSESSIONID">>,<<"gCbUkVfDJDDv8e8zj6BUOAB61lNmPmwt4I6je1Hm">>}],zone => default},{message,<<0,6,42,0,7,53,77,87,244,67,178,4,29,171,66,244>>,0,<<"admin;cloud;1734934037">>,#{dup => false,retain => false},#{peerhost => {172,18,0,1},properties => #{},proto_ver => 4,protocol => mqtt,username => <<"platform">>},<<"/UIOT/iot/v1/device/gqOGm4376vR/analogChange">>,<<"{\"mrid\":\"gqOGm4376vR:Ia1\",\"quality\":1,\"value\":12,\"time\":0}">>,1735029469564,[]}]}, reason: badarg, stacktrace: [{ets,member,[emqx_topic_metrics,<<"/UIOT/iot/v1/device/gqOGm4376vR/analogChange">>],[{error_info,#{cause => id,module => erl_stdlib_errors}}]},{emqx_topic_metrics,is_registered,1,[{file,"emqx_topic_metrics.erl"},{line,182}]},{emqx_topic_metrics,on_message_delivered,2,[{file,"emqx_topic_metrics.erl"},{line,134}]},{emqx_hooks,safe_execute,2,[{file,"emqx_hooks.erl"},{line,200}]},{emqx_hooks,do_run_fold,3,[{file,"emqx_hooks.erl"},{line,180}]},{emqx_channel,do_deliver,2,[{file,"emqx_channel.erl"},{line,1126}]},{emqx_channel,handle_out,3,[{file,"emqx_channel.erl"},{line,1057}]},{emqx_ws_connection,with_channel,3,[{file,"emqx_ws_connection.erl"},{line,747}]},{cowboy_websocket,handler_call,6,[{file,"cowboy_websocket.erl"},{line,487}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]
2024-12-24T16:37:49.565475+08:00 [error] msg: failed_to_execute, mfa: emqx_hooks:safe_execute/2, line: 210, peername: 172.18.0.1:38496, clientid: uiot-web_Chrome_jo48I5d9, exception: error, failed_call: {emqx_topic_metrics,on_message_delivered,[#{clientid => <<"uiot-web_Chrome_jo48I5d9">>,enable_authn => true,is_bridge => false,is_superuser => true,listener => 'ws:default',mountpoint => undefined,peerhost => {172,18,0,1},protocol => mqtt,sockport => 8083,username => <<"platform">>,ws_cookie => [{<<"autoDeployment">>,<<"c190d63a-6e34-44d9-a0a0-08d0841e9693">>},{<<"JSESSIONID">>,<<"vmLLqGBpCqXmAk3ducgCnmyI8GP5xfbDL3CML3hW">>}],zone => default},{message,<<0,6,42,0,7,53,77,87,244,67,178,4,29,171,66,244>>,0,<<"admin;cloud;1734934037">>,#{dup => false,retain => false},#{peerhost => {172,18,0,1},properties => #{},proto_ver => 4,protocol => mqtt,username => <<"platform">>},<<"/UIOT/iot/v1/device/gqOGm4376vR/analogChange">>,<<"{\"mrid\":\"gqOGm4376vR:Ia1\",\"quality\":1,\"value\":12,\"time\":0}">>,1735029469564,[]}]}, reason: badarg, stacktrace: [{ets,member,[emqx_topic_metrics,<<"/UIOT/iot/v1/device/gqOGm4376vR/analogChange">>],[{error_info,#{cause => id,module => erl_stdlib_errors}}]},{emqx_topic_metrics,is_registered,1,[{file,"emqx_topic_metrics.erl"},{line,182}]},{emqx_topic_metrics,on_message_delivered,2,[{file,"emqx_topic_metrics.erl"},{line,134}]},{emqx_hooks,safe_execute,2,[{file,"emqx_hooks.erl"},{line,200}]},{emqx_hooks,do_run_fold,3,[{file,"emqx_hooks.erl"},{line,180}]},{emqx_channel,do_deliver,2,[{file,"emqx_channel.erl"},{line,1126}]},{emqx_channel,handle_out,3,[{file,"emqx_channel.erl"},{line,1057}]},{emqx_ws_connection,with_channel,3,[{file,"emqx_ws_connection.erl"},{line,747}]},{cowboy_websocket,handler_call,6,[{file,"cowboy_websocket.erl"},{line,487}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]
2024-12-24T16:38:35.613091+08:00 [warning] msg: busy_port, mfa: emqx_sys_mon:handle_info/2, line: 178, portinfo: [{port,#Port<0.6>},{name,"0/1"},{links,[<0.1771.0>]},{id,48},{connected,<0.1771.0>},{input,0},{output,19990438},{os_pid,undefined}], procinfo: [{pid,<0.1771.0>},{memory,11816},{total_heap_size,1363},{heap_size,987},{stack_size,6},{min_heap_size,233},{initial_call,{erlang,apply,2}},{current_stacktrace,[{user,server_loop,2,[{file,"user.erl"},{line,131}]}]},{registered_name,user},{status,running},{message_queue_len,0},{group_leader,<0.1747.0>},{priority,normal},{trap_exit,true},{reductions,2304979},{last_calls,false},{catchlevel,0},{trace,0},{suspending,[]},{sequential_trace_token,[]},{error_handler,error_handler}]
6

第一个日志全是 debug 日志。不推荐在生产环境（大流量）下开 debug
这份日志完全看不出有消息丢弃的情况。一般如果有消息丢弃，都会打印日志的，目前还没有客户报过丢了消息没有日志原因的情况。
第二个日志是看起来像是你的 emqx 完全不正常了时在报一个错，只能说是一个完全不正常的现象，。但是看不出来是什么原因引起的，需要更前面完整的日志分析才能看得出来（或者你期间有什么其它的操作么）

7

您好，日志里面busy_port这条能看出什么问题吗：
2024-12-24T16:38:35.613091+08:00 [warning] msg: busy_port, mfa: emqx_sys_mon:handle_info/2, line: 178, portinfo: [{port,#Port<0.6>},{name,“0/1”},{links,[<0.1771.0>]},{id,48},{connected,<0.1771.0>},{input,0},{output,19990438},{os_pid,undefined}], procinfo: [{pid,<0.1771.0>},{memory,11816},{total_heap_size,1363},{heap_size,987},{stack_size,6},{min_heap_size,233},{initial_call,{erlang,apply,2}},{current_stacktrace,[{user,server_loop,2,[{file,“user.erl”},{line,131}]}]},{registered_name,user},{status,running},{message_queue_len,0},{group_leader,<0.1747.0>},{priority,normal},{trap_exit,true},{reductions,2304979},{last_calls,false},{catchlevel,0},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}]

8

emqx.log.1-1.zip (3.9 KB)
您好，有几条新日志，可以帮忙看一下吗
总是出现消息丢失，但是无法定位原因，日志也看不出

9

看不出来

10

这个 busyport 结合你第三部分的日志的 timeout，说明你的 websocket 一个端口（一个客户端）承受不了这么大的流量。你是不是订阅了#，这样的。
要不你就用共享订阅分担一下。不要一个客户端处理太多的消息。（好奇你现在的QPS是多少）