EMOX Broker4.4.16 假死,运行两天后会出现

环境

  • EMQX 版本: Broker4.4.16
  • 操作系统版本: centos 7.6 , debian docker 容器运行

日志信息:

2023-12-24T02:20:17.764454+00:00 [warning] 445-3866/90:02:8A:14:EB:4A@5.19.30.22:33430 [Channel] The PUBACK PacketId 1 is not found.
2023-12-24T02:20:45.438367+00:00 [warning] [Alarm Handler] Alarm high_cpu_usage is activated, 97.34276268987739% cpu usage
2023-12-24T02:23:35.940836+00:00 [warning] 445-1828/90:02:8A:14:C2:B6@5.19.13.78:56216 [Channel] The PUBACK PacketId 35 is not found.
2023-12-24T02:23:45.458040+00:00 [warning] [Alarm Handler] Alarm high_cpu_usage is deactivated
2023-12-24T02:25:05.337526+00:00 [error] [Pool] Error: badarg, [{ets,delete,[emqx_channel_conn,{<<“445-4026/90:02:8A:14:6F:1C”>>,<0.22055.1>}],[{error_info,#{cause => id,module => erl_stdlib_errors}}]},{emqx_cm,do_unregister_channel,1,[{file,“emqx_cm.erl”},{line,156}]},{emqx_cm,clean_down,1,[{file,“emqx_cm.erl”},{line,561}]},{lists,foreach,2,[{file,“lists.erl”},{line,1342}]},{emqx_pool,handle_cast,2,[{file,“emqx_pool.erl”},{line,108}]},{gen_server,try_dispatch,4,[{file,“gen_server.erl”},{line,695}]},{gen_server,handle_msg,6,[{file,“gen_server.erl”},{line,771}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]
2023-12-24T02:25:05.338043+00:00 [error] [Pool] Error: badarg, [{ets,lookup,[emqx_submon,<0.6186.2>],[{error_info,#{cause => id,module => erl_stdlib_errors}}]},{emqx_broker_helper,clean_down,1,[{file,“emqx_broker_helper.erl”},{line,157}]},{lists,foreach,2,[{file,“lists.erl”},{line,1342}]},{emqx_pool,handle_cast,2,[{file,“emqx_pool.erl”},{line,108}]},{gen_server,try_dispatch,4,[{file,“gen_server.erl”},{line,695}]},{gen_server,handle_msg,6,[{file,“gen_server.erl”},{line,771}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]
2023-12-24T02:25:05.338538+00:00 [error] [Pool] Error: badarg, [{ets,delete,[emqx_channel_conn,{<<“445-732/90:02:8A:14:25:4E”>>,<0.30882.1>}],[{error_info,#{cause => id,module => erl_stdlib_errors}}]},{emqx_cm,do_unregister_channel,1,[{file,“emqx_cm.erl”},{line,156}]},{emqx_cm,clean_down,1,[{file,“emqx_cm.erl”},{line,561}]},{lists,foreach,2,[{file,“lists.erl”},{line,1342}]},{emqx_pool,handle_cast,2,[{file,“emqx_pool.erl”},{line,108}]},{gen_server,try_dispatch,4,[{file,“gen_server.erl”},{line,695}]},{gen_server,handle_msg,6,[{file,“gen_server.erl”},{line,771}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,236}]}]
Stop http:management listener on 0.0.0.0:9099 successfully.
Starting emqx on node a736927ebf6e@172.19.0.2
Start mqtt:tcp:internal listener on 127.0.0.1:11883 successfully.
Start mqtt:tcp:external listener on 0.0.0.0:1883 successfully.
Start mqtt:ws:external listener on 0.0.0.0:8083 successfully.
Start mqtt:ssl:external listener on 0.0.0.0:8883 successfully.
Start mqtt:wss:external listener on 0.0.0.0:8084 successfully.
Start http:management listener on 9099 successfully.
Start http:dashboard listener on 18083 successfully.
EMQX Broker 4.4.16 is running now!
2023-12-24T02:28:17.235553+00:00 [warning] 445-1892/90:02:8A:14:EB:25@5.19.14.23:51722 [Channel] The PUBACK PacketId 7 is not found.
2023-12-24T02:28:19.987483+00:00 [warning] 445-2540/90:02:8A:14:3D:98@5.19.20.26:37732 [Channel] The PUBACK PacketId 7 is not found.
2023-12-24T02:28:20.675670+00:00 [warning] 445-641/90:02:8A:14:3D:73@5.19.59.23:40616 [Channel] The PUBACK PacketId 7 is not found.
2023-12-24T02:28:20.724885+00:00 [warning] 445-539/90:02:8A:13:C5:A6@5.19.58.23:58734 [Channel] The PUBACK PacketId 7 is not found.
2023-12-24T02:28:21.256382+00:00 [warning] 445-112/90:02:8A:14:3D:4C@5.19.55.27:43056 [Channel] The PUBACK PacketId 7 is not found.
2023-12-24T02:29:45.053224+00:00 [warning] 445-3037/90:02:8A:14:C3:86@5.19.24.33:58430 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T02:29:47.493308+00:00 [warning] 445-1288/90:02:8A:15:0B:2F@5.19.64.183:47286 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T02:31:15.586003+00:00 [warning] [Alarm Handler] Alarm high_cpu_usage is activated, 88.84592754475918% cpu usage
2023-12-24T02:33:12.744586+00:00 [warning] 445-3715/90:02:8A:15:0B:B0@5.19.29.41:44440 [Channel] The PUBACK PacketId 4 is not found.
2023-12-24T02:33:13.366496+00:00 [warning] 445-2592/90:02:8A:14:0C:B1@5.19.20.78:49208 [Channel] The PUBACK PacketId 4 is not found.
2023-12-24T02:33:39.882029+00:00 [warning] 445-3179/90:02:8A:15:0C:C5@5.19.25.56:41370 [Channel] The PUBACK PacketId 6 is not found.
2023-12-24T02:33:49.066138+00:00 [warning] 445-2577/90:02:8A:14:12:2C@5.19.20.63:55468 [Channel] The PUBACK PacketId 2 is not found.
2023-12-24T02:33:51.603591+00:00 [warning] 445-1747/90:02:8A:14:11:F5@5.19.12.125:57638 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T02:33:58.850660+00:00 [warning] 445-2510/90:02:8A:14:C1:0A@5.19.19.93:52494 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T02:33:59.840053+00:00 [warning] 445-1239/90:02:8A:14:0C:66@5.19.64.134:53648 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T02:34:00.952894+00:00 [warning] 445-2943/90:02:8A:14:2F:5E@5.19.23.57:46720 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T02:34:09.708806+00:00 [warning] 445-2030/90:02:8A:15:0C:60@5.19.14.161:45260 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T02:34:10.285251+00:00 [warning] 445-2448/90:02:8A:14:C2:1F@5.19.19.31:57322 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T02:34:15.588672+00:00 [warning] [Alarm Handler] Alarm high_cpu_usage is deactivated
2023-12-24T02:34:44.027771+00:00 [warning] 445-2203/90:02:8A:14:C1:5E@5.19.16.44:41958 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T02:34:52.667081+00:00 [warning] 445-1020/90:02:8A:14:2F:70@5.19.63.33:46504 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T02:34:52.668733+00:00 [warning] 445-2769/90:02:8A:14:C3:56@5.19.21.133:50144 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T02:36:56.810304+00:00 [warning] 445-1562/90:02:8A:14:26:AB@5.19.11.69:52166 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T02:37:45.752092+00:00 [warning] 445-703/90:02:8A:14:0D:8A@5.19.60.55:46932 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T02:37:46.365157+00:00 [warning] 445-1257/90:02:8A:15:3D:18@5.19.64.152:54076 [Channel] The PUBACK PacketId 3 is not found.
Starting emqx on node a736927ebf6e@172.19.0.2
Start mqtt:tcp:internal listener on 127.0.0.1:11883 successfully.
Start mqtt:tcp:external listener on 0.0.0.0:1883 successfully.
Start mqtt:ws:external listener on 0.0.0.0:8083 successfully.
Start mqtt:ssl:external listener on 0.0.0.0:8883 successfully.
Start mqtt:wss:external listener on 0.0.0.0:8084 successfully.
Start http:management listener on 9099 successfully.
Start http:dashboard listener on 18083 successfully.
EMQX Broker 4.4.16 is running now!
2023-12-24T13:04:16.963162+00:00 [warning] 445-3441/90:02:8A:14:EB:47@5.19.27.21:48626 [Channel] The PUBACK PacketId 6 is not found.
2023-12-24T13:04:44.680072+00:00 [warning] [Alarm Handler] Alarm high_cpu_usage is activated, 95.04118339216674% cpu usage
2023-12-24T13:07:33.142360+00:00 [warning] 445-1196/90:02:8A:15:3C:FE@5.19.64.91:50932 [Channel] The PUBACK PacketId 6 is not found.
2023-12-24T13:08:44.688207+00:00 [warning] [Alarm Handler] Alarm high_cpu_usage is deactivated
2023-12-24T13:12:53.940845+00:00 [warning] 445-337/90:02:8A:14:25:BC@5.19.56.79:45064 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:14:17.287932+00:00 [warning] 445-1466/90:02:8A:14:30:FA@5.19.65.101:41280 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:14:17.738879+00:00 [warning] 445-1066/90:02:8A:14:0F:04@5.19.63.79:51748 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:14:18.056604+00:00 [warning] 445-798/90:02:8A:14:30:37@5.19.61.50:49750 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:14:30.769070+00:00 [warning] 445-489/90:02:8A:14:0D:66@5.19.57.102:45700 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:14:37.254989+00:00 [warning] 445-123/90:02:8A:14:30:40@5.19.55.38:48630 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:14:41.749548+00:00 [warning] 445-606/90:02:8A:14:30:EF@5.19.58.90:56776 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:14:41.967434+00:00 [warning] 445-1418/90:02:8A:14:30:DA@5.19.65.53:44182 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:14:55.997355+00:00 [warning] 445-1433/90:02:8A:14:0C:81@5.19.65.68:54538 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:14:56.245880+00:00 [warning] 445-1177/90:02:8A:15:3B:F9@5.19.64.72:59464 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:14:56.706728+00:00 [warning] 445-897/90:02:8A:14:2F:22@5.19.62.38:54198 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:14:58.875835+00:00 [warning] 445-607/90:02:8A:14:30:98@5.19.58.91:53948 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:15:00.718940+00:00 [warning] 445-1294/90:02:8A:14:de:67@5.19.64.189:50308 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:15:01.464841+00:00 [warning] 445-277/90:02:8A:15:0A:C2@5.19.55.192:39758 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:15:01.780389+00:00 [warning] 445-450/90:02:8A:14:0D:05@5.19.57.63:43854 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:15:02.349099+00:00 [warning] 445-230/90:02:8A:15:0C:01@5.19.55.145:58430 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:15:04.162943+00:00 [warning] 445-810/90:02:8A:14:31:11@5.19.61.62:44810 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:15:05.032605+00:00 [warning] 445-632/90:02:8A:14:2E:C4@5.19.58.116:47384 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:15:06.055887+00:00 [warning] 445-752/90:02:8A:14:30:7C@5.19.60.104:53554 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:15:10.145147+00:00 [warning] 445-679/90:02:8A:14:25:AB@5.19.60.31:45484 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:15:10.789646+00:00 [warning] 445-1182/90:02:8A:14:de:46@5.19.64.77:43076 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:15:53.462204+00:00 [warning] 445-649/90:02:8A:14:0E:EA@5.19.59.31:60878 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:16:04.037193+00:00 [warning] 445-916/90:02:8A:14:30:12@5.19.62.57:52676 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:16:05.560043+00:00 [warning] 445-1092/90:02:8A:14:30:62@5.19.63.105:53450 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:16:40.701910+00:00 [warning] 445-498/90:02:8A:14:26:77@5.19.57.111:40736 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:16:44.081547+00:00 [warning] 445-213/90:02:8A:15:0C:2C@5.19.55.128:49908 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:16:45.968089+00:00 [warning] 445-991/90:02:8A:14:26:AF@5.19.62.132:56626 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:16:56.103662+00:00 [warning] 445-139/90:02:8A:14:0D:75@5.19.55.54:50780 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:17:01.692967+00:00 [warning] 445-1421/90:02:8A:14:30:E5@5.19.65.56:37672 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:17:02.695188+00:00 [warning] 445-861/90:02:8A:14:0E:46@5.19.61.113:56532 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:17:27.184013+00:00 [warning] 445-194/90:02:8A:15:0D:0F@5.19.55.109:33386 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:17:27.629451+00:00 [warning] 445-698/90:02:8A:14:0D:81@5.19.60.50:34572 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:17:36.447475+00:00 [warning] 445-1251/90:02:8A:15:3B:F0@5.19.64.146:59008 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:17:36.507170+00:00 [warning] 445-1472/90:02:8A:14:26:CD@5.19.65.107:41320 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:17:36.675255+00:00 [warning] 445-140/90:02:8A:15:0C:96@5.19.55.55:33912 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:17:43.544123+00:00 [warning] 445-1231/90:02:8A:14:de:9C@5.19.64.126:55946 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:17:45.477338+00:00 [warning] 445-266/90:02:8A:14:0F:05@5.19.55.181:60218 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:17:46.761904+00:00 [warning] 445-768/90:02:8A:14:30:9F@5.19.60.120:50534 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:18:03.158883+00:00 [warning] 445-566/90:02:8A:14:30:3C@5.19.58.50:44912 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:18:03.334911+00:00 [warning] 445-718/90:02:8A:14:26:27@5.19.60.70:54388 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:18:05.737872+00:00 [warning] 445-581/90:02:8A:14:25:E5@5.19.58.65:54598 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T13:18:24.102012+00:00 [warning] 445-586/90:02:8A:14:25:E9@5.19.58.70:48610 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T23:54:03.484911+00:00 [warning] 445-2725/90:02:8A:14:5A:C2@5.19.21.89:57692 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T23:54:29.079848+00:00 [warning] 445-2779/90:02:8A:14:BF:E6@5.19.21.143:54448 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T23:54:41.315796+00:00 [warning] 445-2685/90:02:8A:14:6F:55@5.19.21.49:40390 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T23:54:42.079945+00:00 [warning] 445-2775/90:02:8A:14:C1:37@5.19.21.139:45386 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T23:54:56.696607+00:00 [warning] 445-2733/90:02:8A:14:C2:89@5.19.21.97:39770 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T23:55:13.320429+00:00 [warning] 445-2769/90:02:8A:14:C3:56@5.19.21.133:55952 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T23:55:25.961866+00:00 [warning] 445-2688/90:02:8A:14:6F:7F@5.19.21.52:43536 [Channel] The PUBACK PacketId 2 is not found.
2023-12-24T23:55:59.565506+00:00 [warning] 445-2687/90:02:8A:14:6F:7D@5.19.21.51:54020 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T23:55:59.861370+00:00 [warning] 445-2740/90:02:8A:14:5C:47@5.19.21.104:38300 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T23:56:06.308661+00:00 [warning] 445-2704/90:02:8A:14:C1:ED@5.19.21.68:43460 [Channel] The PUBACK PacketId 3 is not found.
2023-12-24T23:56:15.119255+00:00 [warning] 445-2688/90:02:8A:14:6F:7F@5.19.21.52:43536 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:39:53.264524+00:00 [warning] 445-4215/90:02:8A:14:C1:3A@5.19.32.70:52384 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:39:58.226809+00:00 [warning] 445-4267/90:02:8A:14:6E:CE@5.19.32.122:54428 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:40:04.636797+00:00 [warning] 445-4258/90:02:8A:14:6E:C4@5.19.32.113:54584 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:40:04.768844+00:00 [warning] 445-4249/90:02:8A:14:C2:67@5.19.32.104:55918 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:40:25.783728+00:00 [warning] 445-4221/90:02:8A:14:C2:17@5.19.32.76:51002 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:40:29.930340+00:00 [warning] 445-4256/90:02:8A:14:6F:09@5.19.32.111:36618 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:40:47.490455+00:00 [warning] 445-4289/90:02:8A:14:6F:B5@5.19.32.144:37626 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:40:47.808159+00:00 [warning] 445-4197/90:02:8A:14:C1:46@5.19.32.52:49450 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:40:58.792587+00:00 [warning] 445-4213/90:02:8A:14:C3:83@5.19.32.68:36648 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:41:11.003535+00:00 [warning] 445-4196/90:02:8A:14:C2:B8@5.19.32.51:45534 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:41:17.927848+00:00 [warning] 445-4195/90:02:8A:14:C1:EC@5.19.32.50:53428 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:41:28.680226+00:00 [warning] 445-4262/90:02:8A:14:C2:BA@5.19.32.117:52338 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:41:32.762034+00:00 [warning] 445-4206/90:02:8A:14:C2:B0@5.19.32.61:57436 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:41:40.709040+00:00 [warning] 445-4275/90:02:8A:14:70:08@5.19.32.130:51982 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:41:42.108484+00:00 [warning] 445-4199/90:02:8A:14:C1:07@5.19.32.54:50046 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:41:48.450443+00:00 [warning] 445-4254/90:02:8A:14:6E:D9@5.19.32.109:54196 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:41:49.182534+00:00 [warning] 445-4266/90:02:8A:14:6E:ED@5.19.32.121:60448 [Channel] The PUBACK PacketId 2 is not found.
2023-12-25T00:42:36.188704+00:00 [warning] 445-4282/90:02:8A:14:70:10@5.19.32.137:48638 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:42:38.340472+00:00 [warning] 445-4266/90:02:8A:14:6E:ED@5.19.32.121:60448 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:43:43.556233+00:00 [warning] 445-4207/90:02:8A:14:C3:7B@5.19.32.62:36058 [Channel] The PUBACK PacketId 2 is not found.
2023-12-25T00:43:56.049971+00:00 [warning] 445-4273/90:02:8A:14:70:23@5.19.32.128:58322 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T00:44:32.713827+00:00 [warning] 445-4207/90:02:8A:14:C3:7B@5.19.32.62:36058 [Channel] The PUBACK PacketId 3 is not found.
2023-12-25T03:19:33.566617+00:00 [warning] [CM] kicked_an_unknown_session 445-2767/90:02:8A:14:12:23

重现此问题的步骤

  1. xxx
  2. xxx
  3. xxx

预期行为

实际行为

请问对 EMQX 容器分配的资源是怎么样的。假死过程中,是否发现有内存,或者 CPU 耗尽的情况

EMQX容器分配资源没有限制,与宿主机内存是一样的。假死过程中,日志看到有cpu报警 使用率 达到 97% 以上。

宿主机内存 :8G

2023-12-24T02:20:45.438367+00:00 [warning] [Alarm Handler] Alarm high_cpu_usage is activated, 97.34276268987739% cpu usage