用emqtt-bench性能压测，databoard会崩溃504，压测发现并发连接数超过1000就崩溃

EMQX
1

环境信息

  • EMQX 版本：emqx4.4.4
  • 操作系统及版本：centos7
  • 其他

问题描述

通过下面压测可以看出
[root@localhost emqtt-bench]# ./emqtt_bench sub -t topic111 -h ***** -p 1883 -c 1000 -q 1 -u emqx2 -P ******
Start with 4 workers, addrs pool size: 1 and req interval: 40 ms

1s sub total=87 rate=86.74/sec
1s connect_succ total=87 rate=86.74/sec
2s sub total=184 rate=96.90/sec
2s connect_succ total=184 rate=96.90/sec
3s sub total=283 rate=99.00/sec
3s connect_succ total=283 rate=99.00/sec
4s sub total=380 rate=97.00/sec
4s connect_succ total=380 rate=97.00/sec
5s sub total=476 rate=96.00/sec
5s connect_succ total=476 rate=96.00/sec
6s sub total=576 rate=100.00/sec
6s connect_succ total=576 rate=100.00/sec
7s sub total=672 rate=96.00/sec
7s connect_succ total=672 rate=96.00/sec
8s sub total=770 rate=98.00/sec
8s connect_succ total=770 rate=98.00/sec
9s sub total=868 rate=98.00/sec
9s connect_succ total=868 rate=98.00/sec
10s sub total=965 rate=97.00/sec
10s connect_succ total=965 rate=97.00/sec
11s sub total=996 rate=31.00/sec
11s connect_succ total=996 rate=31.00/sec
2m45s sub total=997 rate=0.01/sec
2m45s connect_succ total=997 rate=0.01/sec

配置文件及日志

image
image1183×116 5.29 KB

image
image755×416 8.66 KB

2

你好，能提供下对应的 emqx.log 么

3

我在emqx.log没找到相关信息，

4

你中间某个 Proxy 或在 emqx 机器的文件句柄数不够？ EMQX 本身是不会报告 504, Gateway Time-out 的

5

我是用haproxy+docker-compose搭建的集群

6

可是为什么没有日志呢？emqx.log什么日志都没有

7

client(13): EXIT for {shutdown,tcp_closed}
client(223): EXIT for {shutdown,tcp_closed}
client(220): EXIT for {shutdown,tcp_closed}
client(463): EXIT for {shutdown,tcp_closed}
client(337): EXIT for {shutdown,tcp_closed}
client(318): EXIT for {shutdown,tcp_closed}
client(205): EXIT for {shutdown,tcp_closed}
100个publisher，每秒10条消息，500个subscriber，结果docker容器退出了。。。

8

可能请求都没有到 emqx 就被拒绝了，ha 机器有调优么？文件句柄是多少？

9

可是为什么docker emqx容器会被自动删掉、、这种情况应该怎么避免呢？

10

@Rory

11

docker-compose file 分享一下？

12

version: ‘3.9’

x-default-emqx: &default-emqx

image: emqx/emqx:4.4.4

env_file:

  - conf.cluster.env

healthcheck:

  test: ["CMD", "/opt/emqx/bin/emqx_ctl", "status"]

  interval: 5s

  timeout: 25s

  retries: 5

volumes:

  - ./dockeremqx/emqx.conf:/opt/emqx/etc/emqx.conf

  - ./dockeremqx/acl.conf:/opt/emqx/etc/acl.conf

  - ./dockeremqx/emqx_auth_mysql.conf:/opt/emqx/etc/plugins/emqx_auth_mysql.conf

  - ./dockeremqx/loaded_plugins:/opt/emqx/data/loaded_plugins

services:

haproxy:

container_name: haproxy

image: haproxy:2.4

privileged: true

depends_on:

  - emqx1

  - emqx2

volumes:

  - ./haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg

  - ./certs_ha/certs:/usr/local/etc/haproxy/certs

ports:

  - "18083:18083"

  - "1883:1883"

- “8883:8883”

  - "8083:8083"

- “8084:8084”

  - "9090:9090"

networks:

  - emqx_bridge

working_dir: /usr/local/etc/haproxy

command:

  - bash

  - -c

  - |

    set -x

    cat /usr/local/etc/haproxy/certs/cert.pem /usr/local/etc/haproxy/certs/key.pem > /tmp/emqx.pem

    haproxy -f /usr/local/etc/haproxy/haproxy.cfg

emqx1:

<<: *default-emqx

container_name: node1.emqx.io

environment:

  - "EMQX_HOST=node1.emqx.io"

ports:

  - "18081:18083"

  - "8881:8083"

  - "1882:1883"

networks:

  emqx_bridge:

    aliases:

    - node1.emqx.io

emqx2:

<<: *default-emqx

container_name: node2.emqx.io

environment:

  - "EMQX_HOST=node2.emqx.io"

ports:

  - "18082:18083"

  - "8882:8083"

  - "1881:1883"

networks:

  emqx_bridge:

    aliases:

    - node2.emqx.io

networks:

emqx_bridge:

driver: bridge

name: emqx_bridge

# ipam:

#   driver: default

#   config:

#     - subnet: 172.100.239.0/24

#       gateway: 172.100.239.1
13

请问一下，这个压测导致docker容器退出的问题，我还有个以为，就是emqx容器里的滚动日志怎么关联本地的文件夹，在容器退出时，也能保证日志不丢呢？