emqtt_bench 订阅消息,出现了client(14): EXIT for {shutdown,tcp_closed}不知道是什么原因

./emqtt_bench pub -t test_topic -h XX -c 10 -I 10 -q 0

./emqtt_bench sub -t test_topic -h xx -c 100 -q 2

31s recv total=1287697 rate=40320.00/sec
client(14): EXIT for {shutdown,tcp_closed}
32s recv total=1327386 rate=39689.00/sec
client(23): EXIT for {shutdown,tcp_closed}
33s recv total=1365955 rate=38569.00/sec
client(33): EXIT for {shutdown,tcp_closed}
34s recv total=1394705 rate=28750.00/sec

可以看下 EMQX 这边的日志。

291456, Total Heap Size: 9060719, Kill: true, Error Logger: true, Message Queue Len: 40, GC Info: [{old_heap_block_size,3581853},{heap_block_size,5472277},{mbuf_size,6630},{recent_size,1940954},{stack_size,45},{old_heap_size,0},{heap_size,2487358},{bin_vheap_size,1045764},{bin_vheap_block_size,1511148},{bin_old_vheap_size,0},{bin_old_vheap_block_size,416441}]
2022-11-14T06:26:23.908222+00:00 [warning] line: 417, mfa: emqx_alarm:do_actions/3, msg: alarm_is_deactivated, name: <<“conn_congestion/izbp1arkb2a36tvo0ciosdz_bench_sub_47_4054741830/undefined”>>
2022-11-14T06:26:24.152125+00:00 [error] Process: <0.4304.0> on node ‘emqx@IP’, Context: maximum heap size reached, Max Heap Size: 6291456, Total Heap Size: 8109159, Kill: true, Error Logger: true, Message Queue Len: 20, GC Info: [{old_heap_block_size,2072833},{heap_block_size,6025584},{mbuf_size,10742},{recent_size,57436},{stack_size,34},{old_heap_size,1860645},{heap_size,1727312},{bin_vheap_size,301216},{bin_vheap_block_size,466976},{bin_old_vheap_size,918148},{bin_old_vheap_block_size,832883}]

“>>,conn_state => connected,connected_at => 1668411166552,high_msgq_watermark => 8192,high_watermark => 1048576,memory => 88792,message_queue_len => 0,peername => <<"47.98.44.74:55989">>,pid => <<"<0.32675.1>">>,proto_name => <<"MQTT">>,proto_ver => 5,recbuf => 131072,recv_cnt => 2,recv_oct =>”…>>, mfa: emqx_alarm:do_actions/3, msg: alarm_is_activated, name: <<“conn_congestion/izbp1arkb2a36tvo0ciosdz_bench_sub_155_1833159068/undefined”>>
2022-11-14T07:32:53.032249+00:00 [warning] line: 411, message: <<"connection congested: #{buffer => 4096,clientid => <<"izbp1arkb2a36tvo0ciosdz_bench_sub_194_693873829">>,conn_state => connected,connected_at => 1668411164953,high_msgq_watermark => 8192,high_watermark => 1048576,memory => 55304,message_queue_len => 0,peername => <<"47.98.44.74:43136">>,pid => <<"<0.32182.1>">>,proto_name => <<"MQTT">>,proto_ver => 5,recbuf => 131072,recv_cnt => 2,recv_oct => "…>>, mfa: emqx_alarm:do_actions/3, msg: alarm_is_activated, name: <<“conn_congestion/izbp1arkb2a36tvo0ciosdz_bench_sub_194_693873829/undefined”>>
!!!
WARNING: Default (insecure) Erlang cookie is in use.
WARNING: Configure node.cookie in /opt/emqx/etc/emqx.conf or override from environment variable EMQX_NODE__COOKIE
NOTE: Use the same config value for all nodes in the cluster.
!!!
[warning] unknown_env_vars: [“EMQX_CLUSTER__DISCOVERY”]
rpc.port_discovery = EMQX_RPC__PORT_DISCOVERY = manual
log.file_handlers.default.enable = EMQX_LOG__FILE_HANDLERS__DEFAULT__ENABLE = false
log.console_handler.enable = EMQX_LOG__CONSOLE_HANDLER__ENABLE = true
node.name = EMQX_NODE__NAME = emqx@127.0.0.1
2022-11-14T07:33:12.908978+00:00 [warning] unknown_env_vars: [“EMQX_CLUSTER__DISCOVERY”]
2022-11-14T07:33:12.988591+00:00 [warning] line: 411, message: <<“77.52% mem usage”>>, mfa: emqx_alarm:do_actions/3, msg: alarm_is_activated, name: high_system_memory_usage
Listener ssl:default on 0.0.0.0:8883 started.
Listener tcp:default on 0.0.0.0:1883 started.
Listener ws:default on 0.0.0.0:8083 started.
Listener wss:default on 0.0.0.0:8084 started.
Listener http:dashboard on :18083 started.
EMQX 5.0.9 is running now!

到达了客户端进程的最大堆栈限制,所以触发了强制关闭策略,相关的参数是 force_shutdown_policy

造成这个现象的原因通常是客户端消费能力跟不上生产能力,导致消息在 EMQX 这边堆积过多。

设置连接/会话进程可使用的最大消息队列长度和堆大小,超出限制时将强制关闭进程:

zone.external.force_shutdown_policy = 8000|800MB

5.0好像没有,4.X是有的?这个参数改名了吗

有的呢,etc/emqx-example.conf 里面包含了所有的配置项,你可以在里面搜索 force_shutdown,然后把相关的配置项拷贝到 etc/emqx.conf 中使其生效。

1 个赞

你好,你是看哪条错误信息关键字看出来的,另外我的现象的客户端有shutdown,tcp_closed信息,服务端也卡死了,这个是什么原因呢

有这条日志:

[error] Process: <0.4304.0> on node ‘emqx@IP’, Context: maximum heap size reached, Max Heap Size: 6291456, Total Heap Size: 8109159, Kill: true

出现 tcp_closed 最好能结合看下 EMQX 的日志,信息太少不好判断。

谢谢