设备上线早高峰会出现客户端被踢下线的报错,通常客户端能够重连成功,但有一次重连失败

这个客户端订阅了全量设备上线通知,期间cpu内存和网络资源正常。看下为啥会重连,以及偶现的重连失败原因
服务端日志:
{“time”:“2026-10-20T07:00:25.388376+08:00”,“level”:“info”,“msg”:“terminate”,“username”:“detector-prod”,“reason”:“{shutdown,#{max => 1000,reason => message_queue_too_long,value => 1678}}”,“pid”:“<0.14088.2802>”,“peername”:“隐私”,“mfa”:“{emqx_connection,terminate,2}”,“line”:678,“clientid”:“detector-prod_1789576441982_3pOg”}
客户端自动重连失败日志:
{“timestamp”:“2026-10-20T07:01:35.901906705+08:00”,“level”:“ERROR”,“logger”:“reactor.core.publisher.Operators”,“message”:“Operator called default onErrorDropped”,“thread”:“parallel-1”,“trace_id”:“b0410222435662d6f0e95f918abd9e93”,“trace_flags”:“01”,“span_id”:“7e2be7294c78d495”,“stack_trace”:“reactor.core.Exceptions$ErrorCallbackNotImplemented: java.lang.RuntimeException: Client is not connected (32104)\nCaused by: java.lang.RuntimeException: Client is not connected (32104)\n\tat com.sgm.vehiclestatusdetector.common.config.MqttClientConfig.sendDetectMessage(MqttClientConfig.java:161)\n\tat com.sgm.vehiclestatusdetector.service.impl.DetectServiceImpl.lambda$detectVehicleOnlineStatus$0(DetectServiceImpl.java:50)\n\tat reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:49)\n\tat reactor.core.publisher.Mono.subscribe(Mono.java:4576)\n\tat reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:430)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.lambda$onNext$1(TracingSubscriber.java:64)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.withActiveSpan(TracingSubscriber.java:97)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.withActiveSpan(TracingSubscriber.java:91)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onNext(TracingSubscriber.java:64)\n\tat reactor.core.publisher.FluxLimitRequest$FluxLimitRequestSubscriber.onNext(FluxLimitRequest.java:99)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.lambda$onNext$1(TracingSubscriber.java:64)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.withActiveSpan(TracingSubscriber.java:97)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.withActiveSpan(TracingSubscriber.java:91)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onNext(TracingSubscriber.java:64)\n\tat reactor.core.publisher.FluxInterval$IntervalRunnable.run(FluxInterval.java:124)\n\tat reactor.core.scheduler.PeriodicWorkerTask.call(PeriodicWorkerTask.java:59)\n\tat reactor.core.scheduler.PeriodicWorkerTask.run(PeriodicWorkerTask.java:73)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: org.eclipse.paho.mqttv5.common.MqttException: Client is not connected\n\tat org.eclipse.paho.mqttv5.client.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:32)\n\tat org.eclipse.paho.mqttv5.client.internal.ClientComms.sendNoWait(ClientComms.java:231)\n\tat org.eclipse.paho.mqttv5.client.MqttAsyncClient.publish(MqttAsyncClient.java:1530)\n\tat com.sgm.vehiclestatusdetector.common.config.MqttClientConfig.sendDetectMessage(MqttClientConfig.java:148)\n\t… 22 common frames omitted\n”,“class”:“reactor.util.Loggers$Slf4JLogger”,“method”:“error”,“caller_file_name”:“Loggers.java”,“line”:324}
使用的是java paho客户端代码如下,使用了自动重连:
@Slf4j
@Configuration
public class MqttClientConfig {

@Value("${mqtt.brokerUrl}")
private String brokerUrl;
@Value("${mqtt.username}")
private String username;
@Value("${mqtt.password}")
private String password;
@Value("${mqtt.connectTimeout}")
private int connectTimeout;
@Value("${mqtt.keepalive}")
private int keepalive;
@Value("${mqtt.subTopics}")
private String subTopics;
@Value("${mqtt.detectTopicSuffix}")
private String detectTopicSuffix;


private final MqttMessage detectMessage = new MqttMessage("1".getBytes(), 0, false, new MqttProperties());
private final Sinks.Many<String> eventSink = Sinks.many().multicast().directBestEffort();
private MqttAsyncClient mqttAsyncClient;

@Autowired
private MetricsConfig metricsConfig;


@Bean
public Sinks.Many<String> mqttMessageSink() {
    return eventSink;
}

@Bean
public MqttAsyncClient mqttClient() throws MqttException {
    MqttAsyncClient client = new MqttAsyncClient(brokerUrl, MqttClientIdGenerator.generateClientId(username), new MemoryPersistence());
    client.setCallback(new MqttCallback() {
        @Override
        public void disconnected(MqttDisconnectResponse disconnectResponse) {
            log.error("MQTT回调: 连接断开 {}", disconnectResponse.toString());
        }

        @Override
        public void mqttErrorOccurred(MqttException exception) {
            log.error("MQTT回调: MQTT异常: {}", exception.toString());
            System.out.println("MQTT异常: " + exception.toString());
        }

        @Override
        public void messageArrived(String topic, MqttMessage message) {
            log.info("MQTT回调: 收到MQTT消息, payload: {}, topic: {}", message, topic);
            String vin = topic.split("/")[0]; // 提取VIN号
            eventSink.tryEmitNext(vin);
        }

        @Override
        public void deliveryComplete(IMqttToken token) {
            log.info("MQTT回调: 消息投递成功: {}", token.getMessageId());
        }

        @Override
        public void connectComplete(boolean reconnect, String serverURI) {
            if (reconnect) {
                log.info("MQTT回调: 重连成功:{}", serverURI);
                subscribeTopics(client, subTopics);
            } else {
                log.info("MQTT回调: 连接成功: {}", serverURI);
                subscribeTopics(client, subTopics);
            }
        }

        @Override
        public void authPacketArrived(int reasonCode, MqttProperties properties) {
            log.info("MQTT回调: Auth packet arrived: {}", properties);
        }
    });
    MqttConnectionOptions options = getMqttConnectionOptions();
    IMqttToken connectToken = client.connect(options);
    try{
        connectToken.waitForCompletion();
    }catch (Exception e){
        e.printStackTrace();
    }
    this.mqttAsyncClient = client;
    return client;
}

private MqttConnectionOptions getMqttConnectionOptions() {
    MqttConnectionOptions options = new MqttConnectionOptions();
    options.setAutomaticReconnect(true);
    options.setCleanStart(true);
    options.setKeepAliveInterval(keepalive);
    options.setConnectionTimeout(connectTimeout);
    options.setUserName(username);
    options.setPassword(password.getBytes());
    return options;
}

private void subscribeTopics(MqttAsyncClient client, String topics) {
    String[] topicsArray = topics.split(",");
    if (client == null || topicsArray.length == 0) {
        log.warn("MQTT订阅异常: MQTT client is null or no topics to subscribe.");
        return;
    }
    int[] qosLevels = new int[topicsArray.length];
    Arrays.fill(qosLevels, 1);
    try {
        IMqttToken token = client.subscribe(topicsArray, qosLevels);
        token.waitForCompletion();
    } catch (MqttException e) {
        log.error("MQTT订阅异常: Failed to subscribe topics: ", e);
        throw new RuntimeException("MQTT Topic订阅异常");
    }
}

public Flux<String> getFluxFilterByVin(String vin) {
    return eventSink.asFlux()
            .filter(v -> v.equals(vin));
}


public void sendDetectMessage(String vin) {
    try {
        String topic = vin.concat(detectTopicSuffix);
        mqttAsyncClient.publish(topic, detectMessage, null, new MqttActionListener() {
            @Override
            public void onSuccess(IMqttToken asyncActionToken) {
            }
            @Override
            public void onFailure(IMqttToken asyncActionToken, Throwable exception) {
                metricsConfig.incrementMqttDetectSendFailCounter();
                log.error("MQTT PUBLISH FAIL, topic:{}", topic, exception);
            }
        });
    } catch (MqttException e) {
        metricsConfig.incrementMqttDetectSendFailCounter();
        log.error("MQTT PUBLISH FAIL 发送探测消息异常, vin: {}", vin, e);
        throw new RuntimeException(e);
    }
}

}

以下来自 Gemin Carbon 4 high 的分析,我觉得非常的对,特别是对 Paho 的分析,是我做不出来的。比我强。值得看看

这个问题非常典型,是 EMQX 服务端单连接进程保护机制 与 Eclipse Paho MQTTv5 Java 客户端(1.2.5)并发重连缺陷 + 业务回调写法问题 在早高峰流量洪峰下叠加导致的。

下面从 EMQX 服务端源码、Paho MQTTv5 Java 客户端源码、业务 Reactor/回调代码 三个维度做完整溯源,并给出可直接落地的改造方案。


一、核心结论速览

  1. 为什么整机 CPU、内存、网络都正常,早高峰仍会被踢下线?
    • 服务端日志 reason => message_queue_too_long, max => 1000, value => 1678 触发的是 EMQX 单连接 Erlang 进程的邮箱(Mailbox)积压保护(配置项 force_shutdown.max_mailbox_size,默认 1000),而不是整机资源耗尽,也不是会话层的 mqueue。
    • 早高峰海量车机并发上线属于典型的 多核扇入单进程(N-to-1 Fan-in):所有车机连接进程分散在多核 Scheduler 上并发向这唯一一个 detector-prod 连接进程投递消息;同时客户端在单回调线程里同步打 log.info、用 directBestEffort() 做 O(K) 广播过滤、使用 QoS 1 订阅,且将高频探测下发(Publish)与全量上线订阅(Subscribe)复用在同一条连接上,导致该 Erlang 连接进程邮箱瞬间突破 1000 被强制关闭 Socket。
  2. 为什么平时能自动重连成功,但偶现一次“永久重连失败”?
    • 从异常栈行号(MqttAsyncClient.java:1530 → ClientComms.java:231)可精确锁定客户端版本为 org.eclipse.paho:org.eclipse.paho.mqttv5.client:1.2.5。
    • Paho MQTTv5 1.2.5 存在 3 个源码级并发竞态 Bug(核心是 CommsCallback.stop() 的 while (isRunning()) 假等待缺陷,以及 CONNACK 异步回调 stopReconnectCycle() 与瞬时二次断连 startReconnectCycle() 的无锁竞态)。当早高峰“刚重连收到 CONNACK 又瞬间被洪峰踢断”,或旧回调线程被 connectComplete 中的 token.waitForCompletion() 阻塞超过 1s 时,会导致 reconnectTimer 被误取消置空 或 ConnectBG 重连线程死循环卡死在 CONNECTING 状态,客户端从此永久停留在未连接状态(32104)。
    • 同时,业务侧未配置离线缓冲(DisconnectedBufferOptions),且 sendDetectMessage 将 MqttException(32104) 包装成 RuntimeException 抛入未做异常兜底的 Reactor FluxInterval -> flatMap 流,触发 Operators.onErrorDropped 直接终止了探测流。

二、问题一源码分析:为什么早高峰会被 EMQX 踢下线?

1. EMQX 源码触发链路还原

从日志字段 "mfa":"{emqx_connection,terminate,2}","line":678 及错误名 message_queue_too_long 可以确认服务端版本位于 EMQX 5.0.23 ~ 5.6.0 区间(注:EMQX 5.6.1+ 在 PR #12766 中将该原因重命名为 mailbox_overflow)。

在 EMQX 中,每一条 MQTT 客户端连接对应一个独立的 Erlang 进程(日志中为 <0.14088.2802>):

  1. 周期性触发 OOM/Mailbox 检查:
    在 emqx_connection.erl 中,连接进程每收到 active_n(默认 100)个报文进入 tcp_passive,或每异步写出超过 active_n(默认 100)个报文收到 {inet_reply, Sock, ok} 时,都会执行 check_oom/1:
    check_oom(State = #state{channel = Channel}) ->
        ShutdownPolicy = emqx_config:get_zone_conf(
            emqx_channel:info(zone, Channel), [force_shutdown]
        ),
        case emqx_utils:check_oom(ShutdownPolicy) of
            {shutdown, Reason} ->
                %% 立即触发 terminate/2 回调断开连接
                erlang:exit({shutdown, Reason});
            _ ->
                ok
        end,
        State.
    
  2. 检查 Erlang 进程邮箱长度:
    emqx_utils:check_oom/2 调用 erlang:process_info(Pid, [message_queue_len, total_heap_size]):
    check_oom(Pid, #{max_mailbox_size := MaxQLen, max_heap_size := MaxHeapSize}) ->
        case process_info(Pid, [message_queue_len, total_heap_size]) of
            [{message_queue_len, QLen}, {total_heap_size, HeapSize}] ->
                do_check_oom([
                    {QLen, MaxQLen, message_queue_too_long}, %% 5.6.1+ 为 mailbox_overflow
                    {HeapSize, MaxHeapSize, proc_heap_too_large}
                ])
        end.
    
    对应配置项 force_shutdown.max_mailbox_size 默认值正是 1000。注意 emqx_schema.erl 的官方注释明确强调:

    Note: "message queue" here refers to the "message mailbox" of the Erlang process, not the mqueue of QoS 1 and QoS 2.
    (这里的消息队列指 Erlang 连接进程的内部邮箱,而非 QoS 1/2 的会话队列 mqueue。)

  3. 直接关闭底层连接(不发 MQTT DISCONNECT 报文):
    当检测到 QLen = 1678 > 1000 时,check_oom 立即执行 erlang:exit({shutdown, #{max => 1000, reason => message_queue_too_long, value => 1678}}),进入 emqx_connection:terminate/2(第 678 行打印 terminate 日志),直接调用 close_socket_ok(State) 关闭底层 TCP Socket。

2. 为什么 CPU/内存/网络正常,单进程邮箱却积压到 1678?

这是服务端多核扇入模型与客户端 3 个代码瓶颈共同作用的结果:

  • 多核 N 对 1 扇入(CPU 正常但单核单进程饱和):
    早上 7:00 车机集中点火上线,成千上万个车机连接进程分布在 EMQX 所有 CPU 核心(Scheduler)上并行向 <0.14088.2802> 发送 {deliver, Topic, Msg},而 <0.14088.2802> 作为单个 Erlang 进程同一时刻最多只能在一个 CPU 核心上串行消费。因此整机多核 CPU、内存和网卡带宽都很充裕,但这一个订阅者进程的入队速率远超单核消费速率。
  • Paho 单回调线程瓶颈与 INBOUND_QUEUE_SIZE = 10 反压:
    查看 Paho 源码 CommsCallback.java#L56 与 #L382-L391:
    • Paho 内部接收缓冲队列硬编码为 INBOUND_QUEUE_SIZE = 10;
    • 业务订阅使用了 QoS 1(Arrays.fill(qosLevels, 1)),而 Paho 只有在 mqttCallback.messageArrived() 执行完毕后才会发送 PUBACK(CommsCallback.java#L484-L488);
    • 业务在 messageArrived 和 deliveryComplete 中每条消息都同步打 log.info,并且通过 Sinks.many().multicast().directBestEffort() + eventSink.asFlux().filter(v -> v.equals(vin)) 分发——当同时探测 K 辆车时,每一次 tryEmitNext(vin) 都会在唯一的 MQTT Call 回调线程上同步遍历全部 K 个订阅者执行 filter(时间复杂度 O(K))。
    • 一旦 MQTT Call 线程稍有卡顿(日志 I/O、GC 或 O(K) 遍历),messageQueue 满 10 条就会让网络读线程 MQTT Rec 阻塞在 spaceAvailable.wait(200),停止读取 TCP 缓冲区,进而通过 TCP 滑动窗口反压回 EMQX 连接进程。
  • 读写复用同一条 MQTT 连接加剧邮箱膨胀:
    该客户端不仅接收全量上线通知,还用同一个 mqttAsyncClient 高频调用 sendDetectMessage 发送探测报文。在 EMQX 侧,同一个进程 <0.14088.2802> 的邮箱里同时涌入:海量下行 {deliver, ...}、每次异步发包的 {inet_reply, Sock, ok}、客户端回传的 QoS 1 PUBACK 以及客户端发来的大量探测 PUBLISH(还需在该进程内同步执行路由),导致瞬时邮箱长度轻松突破 1000。

三、问题二源码分析:为什么偶现“自动重连永久失败”?

根据报错堆栈行号可确认使用的是 org.eclipse.paho.mqttv5.client:1.2.5(自 2020 年起未发新正式版):

日志显示 07:00:25.388 被踢下线,70.5 秒后(07:01:35.901)依然处于未连接状态。通过逐行分析 Paho 1.2.5 源码,自动重连偶发永久失效有 3 条确凿的源码级缺陷链路:

根因 1(最核心 Bug):CommsCallback.stop() 假等待 + stopReconnectCycle() 误杀新重连定时器

  1. Paho 的 CommsCallback.stop() 存在逻辑 Bug,根本不会等待回调线程退出:
    查看 CommsCallback.java#L137-L159 与 #L653-L660:
    // CommsCallback.java:140-155
    synchronized (lifecycle) {
        target_state = State.STOPPED; // (1) 先把 target_state 设为了 STOPPED
    }
    if (!Thread.currentThread().equals(callbackThread)) {
        synchronized (workAvailable) {
            workAvailable.notifyAll();
        }
        // 注释写着 Wait for the thread to finish,但 isRunning() 判断的是 (... && target_state == State.RUNNING)!
        while (isRunning()) { // (2) 第一轮判断直接为 false!0ms 直接返回,根本没等 callbackThread 结束!
            try { Thread.sleep(100); } catch (Exception e) { }
            clientState.notifyQueueLock();
        }
    }
    
  2. CONNACK 异步成功回调反向取消了二次断连新建的 reconnectTimer:
    完整竞态时序如下:
    [MQTT Rec 线程] 收到第1次重连的 CONNACK
       ├─ ClientState.java:1078 同步将 conState 置为 CONNECTED
       └─ ClientState.java:1079 将 conToken 异步入队 completeQueue 交给 [MQTT Call 线程]
    [EMQX / 网络] 早高峰洪峰瞬间再次踢断或闪断连接!
    [MQTT Rec 线程] 捕获 EOFException -> 执行 shutdownConnection (此时 wasConnected == true)
       ├─ 调用 callback.stop() -> 因上述 Bug 立即返回,旧 [MQTT Call 线程] 仍在运行!
       ├─ 将 conState 置为 DISCONNECTED
       └─ 调用 MqttReconnectCallback.disconnected -> startReconnectCycle() 新建了 reconnectTimer
    [旧 MQTT Call 线程] 紧接着才消费到刚才队列里的 conToken!
       ├─ 执行 ConnectActionListener.onSuccess -> MqttReconnectActionListener.onSuccess (MqttAsyncClient.java:1675)
       ├─ 调用 stopReconnectCycle() -> 将刚刚新建的 reconnectTimer 直接 cancel() 并置为 null!!
       └─ 继续调用 connectComplete -> subscribeTopics 抛 32104,被 ConnectActionListener.java:176 静默吞掉
    ==> 最终状态:conState = DISCONNECTED,reconnectTimer = null,自动重连永久死亡!
    
    此外,MqttAsyncClient.java#L1593-L1615 中 stopReconnectCycle() 加了 synchronized (clientLock),而 startReconnectCycle() 甚至完全没有加锁,进一步放大了该竞态窗口。

根因 2:CommsCallback.start() 死循环 + 用户在 connectComplete 中调用 token.waitForCompletion()

  1. 复用的 CommsCallback 单例死循环卡死 ConnectBG:
    ClientComms 在整个生命周期复用同一个 CommsCallback 实例(ClientComms.java#L118)。因为 stop() 不等旧 MQTT Call 线程退出,若旧 MQTT Call 线程因阻塞操作或日志/GC 卡顿超过首次重连间隔(1000ms)仍未执行到 run() 末尾的 current_state = State.STOPPED,新重连线程 ConnectBG 调用 CommsCallback.start() 时:
    synchronized (lifecycle) {
        if (current_state == State.STOPPED) { // <--- 为 false!整个启动块被跳过!
            ...
            target_state = State.RUNNING;
            new Thread(this).start();
        }
    }
    while (!isRunning()) { // <--- target_state 保持 STOPPED,ConnectBG 陷入无限死循环!
        try { Thread.sleep(100); } catch (Exception e) { }
    }
    
    此时 conState 卡在 CONNECTING;即便 15 秒后 EMQX 因 idle_timeout 断开空连接触发下一次重连并唤醒旧 ConnectBG,苏醒的旧 ConnectBG 也会因 conToken 冲突抛出 32201 并在 ClientComms.java#L800-L802 调用 shutdownConnection 把正在建立的新连接直接拆毁。
  2. 业务在 connectComplete 回调线程中同步 token.waitForCompletion() 的死锁雷区:
    connectComplete 本身就运行在唯一的 MQTT Call 回调线程上。在里面调用无超时的 token.waitForCompletion() 阻塞了 MQTT Call 线程——一旦在收到 SUBACK 之前有超过 10 条 PUBLISH 报文到达(例如改为 cleanStart=false、存在保留消息、或分多次 subscribe),MQTT Rec 会在第 11 条报文处阻塞于 spaceAvailable.wait(200) 等 MQTT Call 腾空间,而 MQTT Call 卡在 waitForCompletion() 等 MQTT Rec 读 SUBACK,形成 100% 双线程死锁。

根因 3:attemptReconnect() 单次定时任务吞掉同步异常不续期 + 业务 Reactor 流崩溃

  1. 单次 TimerTask 断链:
    在 MqttAsyncClient.java#L1578-L1599 中,reconnectTimer.schedule(new ReconnectTask(), reconnectDelay) 是单次调度,只有进入异步 MqttReconnectActionListener.onFailure 才会调用 rescheduleReconnectCycle 续期。如果 attemptReconnect() 调 connect() 时因状态竞态同步抛出任何 MqttException(如 32102 Disconnecting / 32110 Connect in progress),catch 块只打印 FINE 级别日志而不重新调度定时器,重连链条直接断死。
  2. 业务代码层面的 3 个放大器:
    • 未配置 DisconnectedBufferOptions:只要在重连窗口或指数退避期(1s -> 2s -> 4s ... -> 128s,外加 connectTimeout 默认 30s,两次握手超时即可超过 70 秒)调用 publish,ClientComms.sendNoWait 就会立即抛 32104。
    • Reactor 流直接终止:sendDetectMessage 捕获 MqttException 后抛出 new RuntimeException(e),而 DetectServiceImpl 的 FluxInterval -> flatMap(MonoRunnable) 未加 .onErrorResume(...) 且 .subscribe() 未传错误处理回调,导致触发 Operators.onErrorDropped(ErrorCallbackNotImplemented)并直接取消上游定时流。
    • 共享可变 MqttMessage 线程不安全:private final MqttMessage detectMessage = new MqttMessage(..., new MqttProperties()) 被 parallel-* 多线程并发传入 publish,而 Paho 内部会修改 MqttMessage 及其 MqttProperties 字段。

四、完整解决方案

1. 架构与 EMQX 服务端优化(消除早高峰被踢根源)

  1. 改用共享订阅(Shared Subscription)分摊全量上线通知(最核心)
    不要让单个连接扛全量车机上线洪峰。将配置中的 mqtt.subTopics 改为共享订阅前缀:
    • 例如原主题为 +/online,改为 $share/vehicle-detector/+/online。
    • 多实例部署(或单实例内建立 2~4 个订阅 Client)即可在 EMQX 侧将消息均匀分发到多个连接进程,彻底消除单进程邮箱被打爆的问题。
  2. 读写连接分离
    将 “订阅全量上线通知的 Consumer Client” 与 “下发探测指令的 Publisher Client” 拆成两个独立的 MqttAsyncClient Bean,避免上下行报文、路由计算和 {inet_reply, ...} 挤占同一个 EMQX 连接进程邮箱。
  3. 降低上线通知订阅 QoS
    结合业务场景(本身已有主动探测兜底),车辆上线事件属于瞬态通知,订阅端建议由 QoS 1 改为 QoS 0,省去早高峰每秒数千次 PUBACK 往返及 Paho INBOUND_QUEUE_SIZE = 10 的回调确认阻塞。
  4. EMQX 配置适度放宽邮箱阈值
    在 emqx.conf(或 Dashboard → 管理 → MQTT 高级设置)适度调大 max_mailbox_size 吸收早高峰瞬时毛刺:
    force_shutdown {
      enable = true
      max_mailbox_size = 10000   # 默认 1000,早高峰汇聚订阅场景建议调至 5000~10000
      max_heap_size = 64MB
    }
    

2. 客户端 SDK 与业务代码改造方案

路线 A(长期推荐):切换至 hivemq-mqtt-client

org.eclipse.paho.mqttv5.client 自 1.2.5(2020 年)后长期未发正式版。你的项目本身基于 Project Reactor,强烈建议迁移到原生基于 Netty + RxJava/Reactor 背压设计的 com.hivemq:hivemq-mqtt-client(支持 Mqtt5ReactorClient,无 INBOUND_QUEUE_SIZE=10 卡死问题,内置无竞态的自动重连与重订阅能力)。

路线 B(短期最快落地):保留 Paho 1.2.5,修复回调阻塞 + O(1) 分发 + 离线缓冲 + 重连看门狗

如果不更换依赖,建议按以下 5 点修改 MqttClientConfig 与业务调用侧:

  1. 去掉 connectComplete 中的 token.waitForCompletion():改为异步 subscribe(..., MqttActionListener),绝不阻塞 MQTT Call 线程。
  2. 将 VIN 匹配从 O(K) 全局广播改为 ConcurrentHashMap 精确 O(1) 路由,并将高频 log.info 降级为 log.debug,彻底解放 MQTT Call 线程。
  3. 开启 DisconnectedBufferOptions:允许断连/重连窗口期暂存探测消息,不再直接抛 32104。
  4. 每次 publish 构造独立的 MqttMessage 实例(复用不可变的 byte[] payload)。
  5. 增加 @Scheduled 连接看门狗(Watchdog):针对 Paho 1.2.5 偶发 reconnectTimer 静默失活或卡死在 CONNECTING 的 Bug,定时巡检并在断连超时后主动拉起 reconnect()(若底层状态机卡死则自动重建实例)。

参考改造代码:

@Slf4j
@Configuration
@EnableScheduling
public class MqttClientConfig {

    private static final byte[] DETECT_PAYLOAD = "1".getBytes(StandardCharsets.UTF_8);

    @Value("${mqtt.brokerUrl}")
    private String brokerUrl;
    @Value("${mqtt.username}")
    private String username;
    @Value("${mqtt.password}")
    private String password;
    @Value("${mqtt.connectTimeout:10}")
    private int connectTimeout;
    @Value("${mqtt.keepalive:60}")
    private int keepalive;
    @Value("${mqtt.subTopics}")
    private String subTopics;
    @Value("${mqtt.detectTopicSuffix}")
    private String detectTopicSuffix;

    @Autowired
    private MetricsConfig metricsConfig;

    // 按 VIN 精确路由,避免在 Paho 单回调线程上做 O(K) 全局广播过滤
    private final ConcurrentHashMap<String, Sinks.Many<String>> vinWaiters = new ConcurrentHashMap<>();

    private volatile MqttAsyncClient mqttAsyncClient;
    private volatile long disconnectedSinceMs = 0L;

    @Bean
    public MqttAsyncClient mqttClient() throws MqttException {
        this.mqttAsyncClient = createAndConnectClient();
        return this.mqttAsyncClient;
    }

    private synchronized MqttAsyncClient createAndConnectClient() throws MqttException {
        String clientId = MqttClientIdGenerator.generateClientId(username);
        MqttAsyncClient client = new MqttAsyncClient(brokerUrl, clientId, new MemoryPersistence());

        // 开启断连内存缓冲,防止短暂重连期间 publish 直接抛 32104
        DisconnectedBufferOptions bufferOpts = new DisconnectedBufferOptions();
        bufferOpts.setBufferEnabled(true);
        bufferOpts.setBufferSize(5000);
        bufferOpts.setPersistBuffer(false);
        bufferOpts.setDeleteOldestMessages(true);
        client.setBufferOpts(bufferOpts);

        client.setCallback(new MqttCallback() {
            @Override
            public void disconnected(MqttDisconnectResponse disconnectResponse) {
                disconnectedSinceMs = System.currentTimeMillis();
                log.error("MQTT回调: 连接断开 {}", disconnectResponse);
            }

            @Override
            public void mqttErrorOccurred(MqttException exception) {
                log.error("MQTT回调: MQTT异常", exception);
            }

            @Override
            public void messageArrived(String topic, MqttMessage message) {
                // 高频路径切勿逐条打 log.info,改为 debug
                if (log.isDebugEnabled()) {
                    log.debug("MQTT回调: 收到消息, topic: {}", topic);
                }
                int slashIdx = topic.indexOf('/');
                String vin = slashIdx > 0 ? topic.substring(0, slashIdx) : topic;
                // O(1) 精确唤醒正在探测该 VIN 的订阅者
                Sinks.Many<String> sink = vinWaiters.get(vin);
                if (sink != null) {
                    sink.tryEmitNext(vin);
                }
            }

            @Override
            public void deliveryComplete(IMqttToken token) {
                log.debug("MQTT回调: 消息投递成功: {}", token.getMessageId());
            }

            @Override
            public void connectComplete(boolean reconnect, String serverURI) {
                disconnectedSinceMs = 0L;
                log.info("MQTT回调: {}成功: {}", reconnect ? "重连" : "连接", serverURI);
                // 严禁在此调用 token.waitForCompletion() 阻塞 MQTT Call 线程!改为纯异步订阅
                subscribeTopicsAsync(client, subTopics);
            }

            @Override
            public void authPacketArrived(int reasonCode, MqttProperties properties) {
                log.debug("MQTT回调: Auth packet arrived: {}", properties);
            }
        });

        MqttConnectionOptions options = getMqttConnectionOptions();
        try {
            // 仅在应用启动初始化时带超时等待首次连接
            client.connect(options).waitForCompletion(connectTimeout * 1000L);
        } catch (Exception e) {
            log.error("MQTT初始连接失败,交由后台自动重连与看门狗恢复", e);
            disconnectedSinceMs = System.currentTimeMillis();
        }
        return client;
    }

    private MqttConnectionOptions getMqttConnectionOptions() {
        MqttConnectionOptions options = new MqttConnectionOptions();
        options.setAutomaticReconnect(true);
        options.setCleanStart(true);
        options.setKeepAliveInterval(keepalive);
        options.setConnectionTimeout(connectTimeout);
        options.setMaxReconnectDelay(30_000); // 最大重连退避缩短至 30s(默认 128s)
        options.setUserName(username);
        options.setPassword(password.getBytes(StandardCharsets.UTF_8));
        return options;
    }

    private void subscribeTopicsAsync(MqttAsyncClient client, String topics) {
        String[] topicsArray = Arrays.stream(topics.split(","))
                .map(String::trim)
                .filter(s -> !s.isEmpty())
                .toArray(String[]::new);
        if (topicsArray.length == 0) {
            return;
        }
        int[] qosLevels = new int[topicsArray.length];
        Arrays.fill(qosLevels, 0); // 建议上线通知降为 QoS 0;若必须 QoS 1 亦可设为 1

        try {
            client.subscribe(topicsArray, qosLevels, null, new MqttActionListener() {
                @Override
                public void onSuccess(IMqttToken asyncActionToken) {
                    log.info("MQTT异步订阅成功: {}", Arrays.toString(topicsArray));
                }

                @Override
                public void onFailure(IMqttToken asyncActionToken, Throwable exception) {
                    log.error("MQTT异步订阅失败: {}", Arrays.toString(topicsArray), exception);
                }
            });
        } catch (MqttException e) {
            log.error("MQTT发起异步订阅异常", e);
        }
    }

    /**
     * 看门狗兜底:规避 Paho 1.2.5 reconnectTimer 被误取消或 ConnectBG 卡死在 CONNECTING 的并发缺陷
     */
    @Scheduled(fixedDelay = 15_000L)
    public synchronized void mqttConnectionWatchdog() {
        MqttAsyncClient client = this.mqttAsyncClient;
        if (client == null || client.isConnected()) {
            disconnectedSinceMs = 0L;
            return;
        }
        long now = System.currentTimeMillis();
        if (disconnectedSinceMs == 0L) {
            disconnectedSinceMs = now;
            return;
        }
        long downMs = now - disconnectedSinceMs;
        if (downMs < 20_000L) {
            return;
        }
        try {
            log.warn("MQTT看门狗: 已断连 {} ms,尝试主动触发 reconnect()", downMs);
            client.reconnect();
        } catch (MqttException e) {
            // 若卡在 CONNECT_IN_PROGRESS(32110) / DISCONNECTING(32102) 超过 60s,说明命中了 Paho 僵尸状态 Bug,重建客户端
            if (downMs > 60_000L) {
                log.error("MQTT看门狗: 断连已超 {} ms 且状态机异常(rc={}),重建 MqttAsyncClient 实例", downMs, e.getReasonCode(), e);
                try {
                    client.close(true);
                } catch (Exception ignore) {
                }
                try {
                    this.mqttAsyncClient = createAndConnectClient();
                    disconnectedSinceMs = System.currentTimeMillis();
                } catch (Exception rebuildEx) {
                    log.error("MQTT看门狗: 重建客户端失败", rebuildEx);
                }
            }
        }
    }

    public Flux<String> getFluxFilterByVin(String vin) {
        return vinWaiters
                .computeIfAbsent(vin, k -> Sinks.many().multicast().directBestEffort())
                .asFlux()
                .doFinally(signal -> vinWaiters.remove(vin));
    }

    public void sendDetectMessage(String vin) {
        String topic = vin.concat(detectTopicSuffix);
        // 每次发送创建独立 MqttMessage,避免多线程共享可变对象与 MqttProperties
        MqttMessage msg = new MqttMessage(DETECT_PAYLOAD, 0, false, new MqttProperties());
        try {
            mqttAsyncClient.publish(topic, msg, null, new MqttActionListener() {
                @Override
                public void onSuccess(IMqttToken asyncActionToken) {
                }

                @Override
                public void onFailure(IMqttToken asyncActionToken, Throwable exception) {
                    metricsConfig.incrementMqttDetectSendFailCounter();
                    log.error("MQTT PUBLISH FAIL, topic:{}", topic, exception);
                }
            });
        } catch (MqttException e) {
            metricsConfig.incrementMqttDetectSendFailCounter();
            log.warn("MQTT PUBLISH FAIL 发送探测消息异常, vin: {}, rc: {}", vin, e.getReasonCode());
            // 不要向无容错的定时流抛未捕获 RuntimeException;若调用方需要感知异常,也务必在 Reactor flatMap 内加 onErrorResume
        }
    }
}

同时检查 DetectServiceImpl.detectVehicleOnlineStatus 中的 Reactor 链路,确保内部 flatMap 捕获单次发送异常而不中断上游 Flux.interval:

Flux.interval(Duration.ofSeconds(1))
    .take(maxRetries)
    .flatMap(tick -> Mono.fromRunnable(() -> mqttClientConfig.sendDetectMessage(vin))
        .onErrorResume(ex -> {
            log.warn("单次探测发送失败,等待下次重试, vin={}", vin, ex);
            return Mono.empty();
        }))
    .subscribe(
        unused -> {},
        err -> log.error("车辆探测流异常结束, vin={}", vin, err)
    );