设备运行一段时间后无法连接到emqx服务器，剔除后又可以重新连接上

hades · 2024 年12 月 30 日 07:52

问题描述：

1）当前系统正常运行过程中，有些物联网设备由于功耗问题，会定期关机休眠。最近发现一些问题。有些设备运行一段时间后无法连接到服务器了。只要一链接上，就断开了。
2）系统端口，配置什么的可以排除，因为其他设备可以正常连接。
3）eqmx开启了会话功能，会话过期时间为120个小时。
4）一直连接的的设备没有问题。只有定期休眠的设备就会出现连接不上的情况。当连接不上时，从web界面剔除掉对应的客户端后，设备可以正常连接上来了。
5) 设备使用 clean Session 设置为false。
6）通过追踪日志发现，好像和持久化会话有点关系：

2024-12-28T16:44:03.747629+08:00 [SOCKET] 30142@39.144.46.89:16138 msg: emqx_connection_terminated, reason: [context: multiple_rank_y_for_rank_x, stacktrace: [{emqx_persistent_session_ds_stream_scheduler,‘-remove_fully_replayed_streams/1-fun-0-’,5,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,312}]},{maps,fold_1,4,[{file,“maps.erl”},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,“emqx_persistent_session_ds.erl”},{line,578}]},{emqx_channel,handle_timeout,3,[{file,“emqx_channel.erl”},{line,1439}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,811}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,472}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,434}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,251}]}], exception: error], username: bohuaimqtt

系统版本信息：

NAME="Ubuntu"

VERSION="18.04.5 LTS (Bionic Beaver)"

ID=ubuntu

ID_LIKE=debian

PRETTY_NAME="Ubuntu 18.04.5 LTS"

VERSION_ID="18.04"

HOME_URL="https://www.ubuntu.com/"

SUPPORT_URL="https://help.ubuntu.com/"

BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"

PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"

VERSION_CODENAME=bionic

UBUNTU_CODENAME=bionic

软件版本信息：

sysdescr  : EMQX
version   : 5.7.2
datetime  : 2024-12-30T15:40:37.739268802+08:00
uptime    : 9 days, 19 hours, 30 minutes, 1 seconds

日志详情
2024-12-28T16:44:03.634877+08:00 [MQTT] 30142@39.144.46.89:16138 msg: mqtt_packet_received, packet: CONNECT(Q0, R0, D0, ClientId=30142, ProtoName=MQTT, ProtoVsn=4, CleanStart=false, KeepAlive=60, Username=bohuaimqtt, Password=), username: bohuaimqtt
2024-12-28T16:44:03.635047+08:00 [AUTHN] 30142@39.144.46.89:16138 msg: authenticator_result, authenticator: password_based:built_in_database, result: {ok,#{is_superuser => false}}, username: bohuaimqtt
2024-12-28T16:44:03.635107+08:00 [AUTHN] 30142@39.144.46.89:16138 msg: authentication_result, reason: chain_result, result: {stop,{ok,#{is_superuser => false}}}, username: bohuaimqtt
2024-12-28T16:44:03.646754+08:00 [MQTT] 30142@39.144.46.89:16138 msg: mqtt_packet_sent, packet: CONNACK(Q0, R0, D0, AckFlags=1, ReasonCode=0), username: bohuaimqtt
2024-12-28T16:44:03.747629+08:00 [SOCKET] 30142@39.144.46.89:16138 msg: emqx_connection_terminated, reason: [context: multiple_rank_y_for_rank_x, stacktrace: [{emqx_persistent_session_ds_stream_scheduler,‘-remove_fully_replayed_streams/1-fun-0-’,5,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,312}]},{maps,fold_1,4,[{file,“maps.erl”},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,“emqx_persistent_session_ds.erl”},{line,578}]},{emqx_channel,handle_timeout,3,[{file,“emqx_channel.erl”},{line,1439}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,811}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,472}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,434}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,251}]}], exception: error], username: bohuaimqtt
2024-12-28T16:44:03.747840+08:00 [error] crasher: initial call: emqx_connection:init/4, pid: <0.29115657.2>, registered_name: [], error: {multiple_rank_y_for_rank_x,[{emqx_persistent_session_ds_stream_scheduler,‘-remove_fully_replayed_streams/1-fun-0-’,5,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,312}]},{maps,fold_1,4,[{file,“maps.erl”},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,“emqx_persistent_session_ds.erl”},{line,578}]},{emqx_channel,handle_timeout,3,[{file,“emqx_channel.erl”},{line,1439}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,811}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,472}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,434}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,251}]}]}, ancestors: [<0.2970.0>,<0.2969.0>,esockd_sup,<0.2561.0>], message_queue_len: 1, messages: [{timeout,#Ref<0.2848872681.718798849.6827>,{emqx_session,timer_bump_last_alive_at}}], links: [<0.2970.0>], dictionary: [{‘$logger_metadata$’,#{peername => “39.144.46.89:16138”,username => <<“bohuaimqtt”>>,clientid => <<“30142”>>}},{incoming_bytes,47},{outgoing_bytes,4},{recv_pkt,1},{send_pkt,1},{guid,{1735375443646638,268577252590857,0}}], trap_exit: false, status: running, heap_size: 6772, stack_size: 28, reductions: 21594; neighbours:
2024-12-28T16:44:24.154688+08:00 [MQTT] 30142@39.144.46.89:16139 msg: mqtt_packet_received, packet: CONNECT(Q0, R0, D0, ClientId=30142, ProtoName=MQTT, ProtoVsn=4, CleanStart=false, KeepAlive=60, Username=bohuaimqtt, Password=), username: bohuaimqtt
2024-12-28T16:44:24.154822+08:00 [AUTHN] 30142@39.144.46.89:16139 msg: authenticator_result, authenticator: password_based:built_in_database, result: {ok,#{is_superuser => false}}, username: bohuaimqtt
2024-12-28T16:44:24.154925+08:00 [AUTHN] 30142@39.144.46.89:16139 msg: authentication_result, reason: chain_result, result: {stop,{ok,#{is_superuser => false}}}, username: bohuaimqtt
2024-12-28T16:44:24.166302+08:00 [MQTT] 30142@39.144.46.89:16139 msg: mqtt_packet_sent, packet: CONNACK(Q0, R0, D0, AckFlags=1, ReasonCode=0), username: bohuaimqtt
2024-12-28T16:44:24.266579+08:00 [SOCKET] 30142@39.144.46.89:16139 msg: emqx_connection_terminated, reason: [context: multiple_rank_y_for_rank_x, stacktrace: [{emqx_persistent_session_ds_stream_scheduler,‘-remove_fully_replayed_streams/1-fun-0-’,5,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,312}]},{maps,fold_1,4,[{file,“maps.erl”},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,“emqx_persistent_session_ds.erl”},{line,578}]},{emqx_channel,handle_timeout,3,[{file,“emqx_channel.erl”},{line,1439}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,811}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,472}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,434}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,251}]}], exception: error], username: bohuaimqtt
2024-12-28T16:44:24.266770+08:00 [error] crasher: initial call: emqx_connection:init/4, pid: <0.29137295.2>, registered_name: , error: {multiple_rank_y_for_rank_x,[{emqx_persistent_session_ds_stream_scheduler,‘-remove_fully_replayed_streams/1-fun-0-’,5,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,312}]},{maps,fold_1,4,[{file,“maps.erl”},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,“emqx_persistent_session_ds.erl”},{line,578}]},{emqx_channel,handle_timeout,3,[{file,“emqx_channel.erl”},{line,1439}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,811}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,472}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,434}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,251}]}]}, ancestors: [<0.2970.0>,<0.2969.0>,esockd_sup,<0.2561.0>], message_queue_len: 1, messages: [{timeout,#Ref<0.2848872681.718798849.28087>,{emqx_session,timer_bump_last_alive_at}}], links: [<0.2970.0>], dictionary: [{‘$logger_metadata$’,#{peername => “39.144.46.89:16139”,username => <<“bohuaimqtt”>>,clientid => <<“30142”>>}},{incoming_bytes,47},{outgoing_bytes,4},{recv_pkt,1},{send_pkt,1},{guid,{1735375464166180,268577252612495,0}}], trap_exit: false, status: running, heap_size: 6772, stack_size: 28, reductions: 21615; neighbours:
2024-12-28T16:44:44.682767+08:00 [MQTT] 30142@39.144.46.89:16140 msg: mqtt_packet_received, packet: CONNECT(Q0, R0, D0, ClientId=30142, ProtoName=MQTT, ProtoVsn=4, CleanStart=false, KeepAlive=60, Username=bohuaimqtt, Password=), username: bohuaimqtt
2024-12-28T16:44:44.682907+08:00 [AUTHN] 30142@39.144.46.89:16140 msg: authenticator_result, authenticator: password_based:built_in_database, result: {ok,#{is_superuser => false}}, username: bohuaimqtt
2024-12-28T16:44:44.682978+08:00 [AUTHN] 30142@39.144.46.89:16140 msg: authentication_result, reason: chain_result, result: {stop,{ok,#{is_superuser => false}}}, username: bohuaimqtt
2024-12-28T16:44:44.694415+08:00 [MQTT] 30142@39.144.46.89:16140 msg: mqtt_packet_sent, packet: CONNACK(Q0, R0, D0, AckFlags=1, ReasonCode=0), username: bohuaimqtt
2024-12-28T16:44:44.794637+08:00 [SOCKET] 30142@39.144.46.89:16140 msg: emqx_connection_terminated, reason: [context: multiple_rank_y_for_rank_x, stacktrace: [{emqx_persistent_session_ds_stream_scheduler,‘-remove_fully_replayed_streams/1-fun-0-’,5,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,312}]},{maps,fold_1,4,[{file,“maps.erl”},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,“emqx_persistent_session_ds.erl”},{line,578}]},{emqx_channel,handle_timeout,3,[{file,“emqx_channel.erl”},{line,1439}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,811}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,472}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,434}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,251}]}], exception: error], username: bohuaimqtt
2024-12-28T16:44:44.794829+08:00 [error] crasher: initial call: emqx_connection:init/4, pid: <0.29160068.2>, registered_name: [], error: {multiple_rank_y_for_rank_x,[{emqx_persistent_session_ds_stream_scheduler,‘-remove_fully_replayed_streams/1-fun-0-’,5,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,312}]},{maps,fold_1,4,[{file,“maps.erl”},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,“emqx_persistent_session_ds.erl”},{line,578}]},{emqx_channel,handle_timeout,3,[{file,“emqx_channel.erl”},{line,1439}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,811}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,472}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,434}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,251}]}]}, ancestors: [<0.2970.0>,<0.2969.0>,esockd_sup,<0.2561.0>], message_queue_len: 1, messages: [{timeout,#Ref<0.2848872681.203161602.234975>,{emqx_session,timer_bump_last_alive_at}}], links: [<0.2970.0>], dictionary: [{‘$logger_metadata$’,#{peername => “39.144.46.89:16140”,username => <<“bohuaimqtt”>>,clientid => <<“30142”>>}},{incoming_bytes,47},{outgoing_bytes,4},{recv_pkt,1},{send_pkt,1},{guid,{1735375484694297,268577252635268,0}}], trap_exit: false, status: running, heap_size: 6772, stack_size: 28, reductions: 21616; neighbours:
2024-12-28T16:45:05.188756+08:00 [MQTT] 30142@39.144.46.89:16141 msg: mqtt_packet_received, packet: CONNECT(Q0, R0, D0, ClientId=30142, ProtoName=MQTT, ProtoVsn=4, CleanStart=false, KeepAlive=60, Username=bohuaimqtt, Password=), username: bohuaimqtt
2024-12-28T16:45:05.188940+08:00 [AUTHN] 30142@39.144.46.89:16141 msg: authenticator_result, authenticator: password_based:built_in_database, result: {ok,#{is_superuser => false}}, username: bohuaimqtt
2024-12-28T16:45:05.189002+08:00 [AUTHN] 30142@39.144.46.89:16141 msg: authentication_result, reason: chain_result, result: {stop,{ok,#{is_superuser => false}}}, username: bohuaimqtt
2024-12-28T16:45:05.200653+08:00 [MQTT] 30142@39.144.46.89:16141 msg: mqtt_packet_sent, packet: CONNACK(Q0, R0, D0, AckFlags=1, ReasonCode=0), username: bohuaimqtt
2024-12-28T16:45:05.300633+08:00 [SOCKET] 30142@39.144.46.89:16141 msg: emqx_connection_terminated, reason: [context: multiple_rank_y_for_rank_x, stacktrace: [{emqx_persistent_session_ds_stream_scheduler,‘-remove_fully_replayed_streams/1-fun-0-’,5,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,312}]},{maps,fold_1,4,[{file,“maps.erl”},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,“emqx_persistent_session_ds.erl”},{line,578}]},{emqx_channel,handle_timeout,3,[{file,“emqx_channel.erl”},{line,1439}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,811}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,472}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,434}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,251}]}], exception: error], username: bohuaimqtt
2024-12-28T16:45:05.300832+08:00 [error] crasher: initial call: emqx_connection:init/4, pid: <0.29180705.2>, registered_name: , error: {multiple_rank_y_for_rank_x,[{emqx_persistent_session_ds_stream_scheduler,‘-remove_fully_replayed_streams/1-fun-0-’,5,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,312}]},{maps,fold_1,4,[{file,“maps.erl”},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,“emqx_persistent_session_ds.erl”},{line,578}]},{emqx_channel,handle_timeout,3,[{file,“emqx_channel.erl”},{line,1439}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,811}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,472}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,434}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,251}]}]}, ancestors: [<0.2970.0>,<0.2969.0>,esockd_sup,<0.2561.0>], message_queue_len: 1, messages: [{timeout,#Ref<0.2848872681.718798849.78508>,{emqx_session,timer_bump_last_alive_at}}], links: [<0.2970.0>], dictionary: [{‘$logger_metadata$’,#{peername => “39.144.46.89:16141”,username => <<“bohuaimqtt”>>,clientid => <<“30142”>>}},{incoming_bytes,47},{outgoing_bytes,4},{recv_pkt,1},{send_pkt,1},{guid,{1735375505200527,268577252655905,0}}], trap_exit: false, status: running, heap_size: 6772, stack_size: 28, reductions: 21627; neighbours:

zhongwencool · 2024 年12 月 30 日 08:55

这个应该是一个明显的 bug，麻烦在 Issues · emqx/emqx · GitHub 上报一下哈。

hades · 2024 年12 月 30 日 10:15

好的

zhongwencool · 2025 年1 月 6 日 03:03

后续在这里跟踪：

github.com/emqx/emqx

设备运行一段时间后无法连接到emqx服务器，剔除后又可以重新连接上

已打开 09:12AM - 30 Dec 24 UTC

zhaoyou

BUG

### What happened? - 当前系统正常运行过程中，有些物联网设备由于功耗问题，会定期关机休眠。最近发现一些问题。有些设备运行一段时间后无法连接…到服务器了。只要一链接上，就断开了。 - 系统端口，配置什么的可以排除，因为其他设备可以正常连接。 - eqmx开启了会话功能，会话过期时间为120个小时。 - 一直连接的的设备没有问题。只有定期休眠的设备就会出现连接不上的情况。当连接不上时，从web界面剔除掉对应的客户端后，设备可以正常连接上来了。 - 设备使用 clean Session 设置为false。 - 通过追踪日志发现，好像和持久化会话有点关系： 2024-12-28T16:44:03.747629+08:00 [SOCKET] 30142@39.144.46.89:16138 msg: emqx_connection_terminated, reason: [context: multiple_rank_y_for_rank_x, stacktrace: [{emqx_persistent_session_ds_stream_scheduler,‘-remove_fully_replayed_streams/1-fun-0-’,5,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,312}]},{maps,fold_1,4,[{file,“maps.erl”},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,“emqx_persistent_session_ds_stream_scheduler.erl”},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,“emqx_persistent_session_ds.erl”},{line,578}]},{emqx_channel,handle_timeout,3,[{file,“emqx_channel.erl”},{line,1439}]},{emqx_connection,with_channel,3,[{file,“emqx_connection.erl”},{line,811}]},{emqx_connection,process_msg,2,[{file,“emqx_connection.erl”},{line,472}]},{emqx_connection,handle_recv,3,[{file,“emqx_connection.erl”},{line,434}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,251}]}], exception: error], username: bohuaimqtt ![image](https://github.com/user-attachments/assets/3a1a144f-132c-4a6f-9ed9-fecb37827cd7) ![image](https://github.com/user-attachments/assets/c6a2d812-be1b-47e8-8ff2-3c742668898e) ### What did you expect to happen? 设备应该开启后正常连接，而不是需要每次连接不上，手动剔除后才能重新连接。 ### How can we reproduce it (as minimally and precisely as possible)? _No response_ ### Anything else we need to know? _No response_ ### EMQX version <details> ```console $ ./bin/emqx_ctl broker # paste output here ``` sysdescr : EMQX version : 5.7.2 datetime : 2024-12-30T15:40:37.739268802+08:00 uptime : 9 days, 19 hours, 30 minutes, 1 seconds </details> ### OS version <details> ```console # On Linux: $ cat /etc/os-release # paste output here $ uname -a # paste output here ``` NAME="Ubuntu" VERSION="18.04.5 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.5 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic </details> ### Log files <details> 2024-12-28T16:44:03.634877+08:00 [MQTT] 30142@39.144.46.89:16138 msg: mqtt_packet_received, packet: CONNECT(Q0, R0, D0, ClientId=30142, ProtoName=MQTT, ProtoVsn=4, CleanStart=false, KeepAlive=60, Username=bohuaimqtt, Password=******), username: bohuaimqtt 2024-12-28T16:44:03.635047+08:00 [AUTHN] 30142@39.144.46.89:16138 msg: authenticator_result, authenticator: password_based:built_in_database, result: {ok,#{is_superuser => false}}, username: bohuaimqtt 2024-12-28T16:44:03.635107+08:00 [AUTHN] 30142@39.144.46.89:16138 msg: authentication_result, reason: chain_result, result: {stop,{ok,#{is_superuser => false}}}, username: bohuaimqtt 2024-12-28T16:44:03.646754+08:00 [MQTT] 30142@39.144.46.89:16138 msg: mqtt_packet_sent, packet: CONNACK(Q0, R0, D0, AckFlags=1, ReasonCode=0), username: bohuaimqtt 2024-12-28T16:44:03.747629+08:00 [SOCKET] 30142@39.144.46.89:16138 msg: emqx_connection_terminated, reason: [context: multiple_rank_y_for_rank_x, stacktrace: [{emqx_persistent_session_ds_stream_scheduler,'-remove_fully_replayed_streams/1-fun-0-',5,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,312}]},{maps,fold_1,4,[{file,"maps.erl"},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,"emqx_persistent_session_ds.erl"},{line,578}]},{emqx_channel,handle_timeout,3,[{file,"emqx_channel.erl"},{line,1439}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,811}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,434}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,251}]}], exception: error], username: bohuaimqtt 2024-12-28T16:44:03.747840+08:00 [error] crasher: initial call: emqx_connection:init/4, pid: <0.29115657.2>, registered_name: [], error: {multiple_rank_y_for_rank_x,[{emqx_persistent_session_ds_stream_scheduler,'-remove_fully_replayed_streams/1-fun-0-',5,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,312}]},{maps,fold_1,4,[{file,"maps.erl"},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,"emqx_persistent_session_ds.erl"},{line,578}]},{emqx_channel,handle_timeout,3,[{file,"emqx_channel.erl"},{line,1439}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,811}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,434}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,251}]}]}, ancestors: [<0.2970.0>,<0.2969.0>,esockd_sup,<0.2561.0>], message_queue_len: 1, messages: [{timeout,#Ref<0.2848872681.718798849.6827>,{emqx_session,timer_bump_last_alive_at}}], links: [<0.2970.0>], dictionary: [{'$logger_metadata$',#{peername => "39.144.46.89:16138",username => <<"bohuaimqtt">>,clientid => <<"30142">>}},{incoming_bytes,47},{outgoing_bytes,4},{recv_pkt,1},{send_pkt,1},{guid,{1735375443646638,268577252590857,0}}], trap_exit: false, status: running, heap_size: 6772, stack_size: 28, reductions: 21594; neighbours: 2024-12-28T16:44:24.154688+08:00 [MQTT] 30142@39.144.46.89:16139 msg: mqtt_packet_received, packet: CONNECT(Q0, R0, D0, ClientId=30142, ProtoName=MQTT, ProtoVsn=4, CleanStart=false, KeepAlive=60, Username=bohuaimqtt, Password=******), username: bohuaimqtt 2024-12-28T16:44:24.154822+08:00 [AUTHN] 30142@39.144.46.89:16139 msg: authenticator_result, authenticator: password_based:built_in_database, result: {ok,#{is_superuser => false}}, username: bohuaimqtt 2024-12-28T16:44:24.154925+08:00 [AUTHN] 30142@39.144.46.89:16139 msg: authentication_result, reason: chain_result, result: {stop,{ok,#{is_superuser => false}}}, username: bohuaimqtt 2024-12-28T16:44:24.166302+08:00 [MQTT] 30142@39.144.46.89:16139 msg: mqtt_packet_sent, packet: CONNACK(Q0, R0, D0, AckFlags=1, ReasonCode=0), username: bohuaimqtt 2024-12-28T16:44:24.266579+08:00 [SOCKET] 30142@39.144.46.89:16139 msg: emqx_connection_terminated, reason: [context: multiple_rank_y_for_rank_x, stacktrace: [{emqx_persistent_session_ds_stream_scheduler,'-remove_fully_replayed_streams/1-fun-0-',5,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,312}]},{maps,fold_1,4,[{file,"maps.erl"},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,"emqx_persistent_session_ds.erl"},{line,578}]},{emqx_channel,handle_timeout,3,[{file,"emqx_channel.erl"},{line,1439}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,811}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,434}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,251}]}], exception: error], username: bohuaimqtt 2024-12-28T16:44:24.266770+08:00 [error] crasher: initial call: emqx_connection:init/4, pid: <0.29137295.2>, registered_name: [], error: {multiple_rank_y_for_rank_x,[{emqx_persistent_session_ds_stream_scheduler,'-remove_fully_replayed_streams/1-fun-0-',5,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,312}]},{maps,fold_1,4,[{file,"maps.erl"},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,"emqx_persistent_session_ds.erl"},{line,578}]},{emqx_channel,handle_timeout,3,[{file,"emqx_channel.erl"},{line,1439}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,811}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,434}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,251}]}]}, ancestors: [<0.2970.0>,<0.2969.0>,esockd_sup,<0.2561.0>], message_queue_len: 1, messages: [{timeout,#Ref<0.2848872681.718798849.28087>,{emqx_session,timer_bump_last_alive_at}}], links: [<0.2970.0>], dictionary: [{'$logger_metadata$',#{peername => "39.144.46.89:16139",username => <<"bohuaimqtt">>,clientid => <<"30142">>}},{incoming_bytes,47},{outgoing_bytes,4},{recv_pkt,1},{send_pkt,1},{guid,{1735375464166180,268577252612495,0}}], trap_exit: false, status: running, heap_size: 6772, stack_size: 28, reductions: 21615; neighbours: 2024-12-28T16:44:44.682767+08:00 [MQTT] 30142@39.144.46.89:16140 msg: mqtt_packet_received, packet: CONNECT(Q0, R0, D0, ClientId=30142, ProtoName=MQTT, ProtoVsn=4, CleanStart=false, KeepAlive=60, Username=bohuaimqtt, Password=******), username: bohuaimqtt 2024-12-28T16:44:44.682907+08:00 [AUTHN] 30142@39.144.46.89:16140 msg: authenticator_result, authenticator: password_based:built_in_database, result: {ok,#{is_superuser => false}}, username: bohuaimqtt 2024-12-28T16:44:44.682978+08:00 [AUTHN] 30142@39.144.46.89:16140 msg: authentication_result, reason: chain_result, result: {stop,{ok,#{is_superuser => false}}}, username: bohuaimqtt 2024-12-28T16:44:44.694415+08:00 [MQTT] 30142@39.144.46.89:16140 msg: mqtt_packet_sent, packet: CONNACK(Q0, R0, D0, AckFlags=1, ReasonCode=0), username: bohuaimqtt 2024-12-28T16:44:44.794637+08:00 [SOCKET] 30142@39.144.46.89:16140 msg: emqx_connection_terminated, reason: [context: multiple_rank_y_for_rank_x, stacktrace: [{emqx_persistent_session_ds_stream_scheduler,'-remove_fully_replayed_streams/1-fun-0-',5,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,312}]},{maps,fold_1,4,[{file,"maps.erl"},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,"emqx_persistent_session_ds.erl"},{line,578}]},{emqx_channel,handle_timeout,3,[{file,"emqx_channel.erl"},{line,1439}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,811}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,434}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,251}]}], exception: error], username: bohuaimqtt 2024-12-28T16:44:44.794829+08:00 [error] crasher: initial call: emqx_connection:init/4, pid: <0.29160068.2>, registered_name: [], error: {multiple_rank_y_for_rank_x,[{emqx_persistent_session_ds_stream_scheduler,'-remove_fully_replayed_streams/1-fun-0-',5,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,312}]},{maps,fold_1,4,[{file,"maps.erl"},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,"emqx_persistent_session_ds.erl"},{line,578}]},{emqx_channel,handle_timeout,3,[{file,"emqx_channel.erl"},{line,1439}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,811}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,434}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,251}]}]}, ancestors: [<0.2970.0>,<0.2969.0>,esockd_sup,<0.2561.0>], message_queue_len: 1, messages: [{timeout,#Ref<0.2848872681.203161602.234975>,{emqx_session,timer_bump_last_alive_at}}], links: [<0.2970.0>], dictionary: [{'$logger_metadata$',#{peername => "39.144.46.89:16140",username => <<"bohuaimqtt">>,clientid => <<"30142">>}},{incoming_bytes,47},{outgoing_bytes,4},{recv_pkt,1},{send_pkt,1},{guid,{1735375484694297,268577252635268,0}}], trap_exit: false, status: running, heap_size: 6772, stack_size: 28, reductions: 21616; neighbours: 2024-12-28T16:45:05.188756+08:00 [MQTT] 30142@39.144.46.89:16141 msg: mqtt_packet_received, packet: CONNECT(Q0, R0, D0, ClientId=30142, ProtoName=MQTT, ProtoVsn=4, CleanStart=false, KeepAlive=60, Username=bohuaimqtt, Password=******), username: bohuaimqtt 2024-12-28T16:45:05.188940+08:00 [AUTHN] 30142@39.144.46.89:16141 msg: authenticator_result, authenticator: password_based:built_in_database, result: {ok,#{is_superuser => false}}, username: bohuaimqtt 2024-12-28T16:45:05.189002+08:00 [AUTHN] 30142@39.144.46.89:16141 msg: authentication_result, reason: chain_result, result: {stop,{ok,#{is_superuser => false}}}, username: bohuaimqtt 2024-12-28T16:45:05.200653+08:00 [MQTT] 30142@39.144.46.89:16141 msg: mqtt_packet_sent, packet: CONNACK(Q0, R0, D0, AckFlags=1, ReasonCode=0), username: bohuaimqtt 2024-12-28T16:45:05.300633+08:00 [SOCKET] 30142@39.144.46.89:16141 msg: emqx_connection_terminated, reason: [context: multiple_rank_y_for_rank_x, stacktrace: [{emqx_persistent_session_ds_stream_scheduler,'-remove_fully_replayed_streams/1-fun-0-',5,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,312}]},{maps,fold_1,4,[{file,"maps.erl"},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,"emqx_persistent_session_ds.erl"},{line,578}]},{emqx_channel,handle_timeout,3,[{file,"emqx_channel.erl"},{line,1439}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,811}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,434}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,251}]}], exception: error], username: bohuaimqtt 2024-12-28T16:45:05.300832+08:00 [error] crasher: initial call: emqx_connection:init/4, pid: <0.29180705.2>, registered_name: [], error: {multiple_rank_y_for_rank_x,[{emqx_persistent_session_ds_stream_scheduler,'-remove_fully_replayed_streams/1-fun-0-',5,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,312}]},{maps,fold_1,4,[{file,"maps.erl"},{line,416}]},{emqx_persistent_session_ds_stream_scheduler,remove_fully_replayed_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,302}]},{emqx_persistent_session_ds_stream_scheduler,renew_streams,1,[{file,"emqx_persistent_session_ds_stream_scheduler.erl"},{line,128}]},{emqx_persistent_session_ds,handle_timeout,3,[{file,"emqx_persistent_session_ds.erl"},{line,578}]},{emqx_channel,handle_timeout,3,[{file,"emqx_channel.erl"},{line,1439}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,811}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,434}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,251}]}]}, ancestors: [<0.2970.0>,<0.2969.0>,esockd_sup,<0.2561.0>], message_queue_len: 1, messages: [{timeout,#Ref<0.2848872681.718798849.78508>,{emqx_session,timer_bump_last_alive_at}}], links: [<0.2970.0>], dictionary: [{'$logger_metadata$',#{peername => "39.144.46.89:16141",username => <<"bohuaimqtt">>,clientid => <<"30142">>}},{incoming_bytes,47},{outgoing_bytes,4},{recv_pkt,1},{send_pkt,1},{guid,{1735375505200527,268577252655905,0}}], trap_exit: false, status: running, heap_size: 6772, stack_size: 28, reductions: 21627; neighbours: </details>