EMQX webhook推送到HTTP服务 偶尔会出现java.io.IOException: Connection reset by peer

环境

  • EMQX 版本:4.2.5
  • 操作系统版本:Alibaba Cloud Linux 2.1903 LTS 64位

重现此问题的步骤

  1. 通过webhook将数据转发到Netty服务

预期行为

netty能正常消费数据

实际行为

netty服务错误日志

2024-10-16 09:13:22.328 |-WARN [WORK_4] io.netty.channel.DefaultChannelPipeline [1152] -| An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:256)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:750)

EMQX错误日志

2024-09-29 15:31:35.129 [error] <<“12623502”>>@117.132.198.84:59474 [Channel] Unexpected info: {#Ref<0.3092544574.1911029761.91327>,
{error,{shutdown,econnrefused}}}
2024-09-29 15:31:35.120 [error] <<“12627745”>>@:61511 Take action <<“data_to_webserver_1727418900324028508”>> failed, continue next action, reason: {exit,{timeout,{gen_server,call,[<0.17137.1>,{post,{<<“/bridgingKafka/bsaAlertorData”>>,[{<<“content-type”>>,<<“application/json”>>}],<<"{"ts":1727595089116,"prodect_id":"1030","id":"DAT_0","engine_id_source":"data_12627745","engine_id":"12627745","datapoints":{"ver":"V1.0.201215","gas1":[{"1":"0,2,1,0,,,,,,"},{"2":"0,2,1,0,,,,,,"},{"3":"0,2,1,0,,,,,,"},{"4":"0,2,0,1,,,,,,"}],"date":"2024-09-29 15:27:20","csq":"23,99"}}">>},5000},6000]}},[{gen_server,call,3,[{file,“gen_server.erl”},{line,223}]},{emqx_web_hook_actions,‘-on_action_create_data_to_webserver/2-fun-3-’,8,[{file,“emqx_web_hook_actions.erl”},{line,223}]},{emqx_rule_runtime,take_action,5,[{file,“emqx_rule_runtime.erl”},{line,235}]},{emqx_rule_runtime,‘-take_actions/4-lc$^0/1-0-’,4,[{file,“emqx_rule_runtime.erl”},{line,227}]},{emqx_rule_runtime,do_apply_rule,2,[{file,“emqx_rule_runtime.erl”},{line,110}]},{emqx_rule_runtime,apply_rules,2,[{file,“emqx_rule_runtime.erl”},{line,52}]},{emqx_rule_events,on_message_publish,2,[{file,“emqx_rule_events.erl”},{line,81}]},{emqx_hooks,safe_execute,2,[{file,“emqx_hooks.erl”},{line,164}]}]}

emqx 这边日志里第一条是 econnrefused,表示服务端不再监听该端口(可能挂了),第二条 timeout 是发送请求超时,没有收到 HTTP Response。

那请问 java.io.IOException: Connection reset by peer异常是EMQX的这个报错导致的吗?netty服务是能正常处理数据,但是也会频繁出现这个报错

TCP reset 一般是说对面 TCP 连接的状态已经不一样了,提示连接已经断开,只是本地这一侧没有及时感知到。