Haproxy负载k8s部署的EMQX的问题

参考官网链接配置haproxy
用 HAProxy 负载均衡 EMQX 集群 | EMQX文档
配置后报错:
[root@k8s-master-prod haproxy]# haproxy -f /usr/local/haproxy/haproxy.cfg -c
[NOTICE] (39622) : haproxy version is 2.8.4-a4ebf9d
[NOTICE] (39622) : path to executable is /usr/bin/haproxy
[ALERT] (39622) : config : parsing [/usr/local/haproxy/haproxy.cfg:46] : ‘stick’: invalid arg 2 in converter ‘mqtt_field_value’ : expected ‘)’
[ALERT] (39622) : config : Error(s) found in configuration file : /usr/local/haproxy/haproxy.cfg
[ALERT] (39622) : config : Fatal errors found in configuration.
Haproxy版本用的是2.84,请问如何解除这些报错啊~

以下是我完整的配置:
frontend mqtt_dubbo
bind *:38000
mode tcp
tcp-request inspect-delay 10s
tcp-request content reject unless { req.payload(0,0), mqtt_is_valid }
default_backend mqtt_backend_dubbo
backend mqtt_backend_dubbo
mode tcp
stick-table type string len 32 size 100k expire 30m
stick on req.payload(0,0), mqtt_field_value(connect, client_identifier)
server emqx111 192.168.127.129:30784
server emqx222 192.168.127.130:30784

我试了下,是可以的。
感觉应该是你的格式复制错了。 :thinking:

我是粘贴复制官方的 这不科学啊
[root@k8s-master-prod haproxy]# haproxy -f /usr/local/haproxy/haproxy.cfg -c
[NOTICE] (35070) : haproxy version is 2.8.4-a4ebf9d
[NOTICE] (35070) : path to executable is /usr/bin/haproxy
[ALERT] (35070) : config : parsing [/usr/local/haproxy/haproxy.cfg:39] : ‘stick’: invalid arg 2 in converter ‘mqtt_field_value’ : expected ‘)’
[ALERT] (35070) : config : Error(s) found in configuration file : /usr/local/haproxy/haproxy.cfg
[ALERT] (35070) : config : Fatal errors found in configuration.
您在 /var/spool/mail/root 中有新邮件

推荐你可以用这个试试:

把他下到你本地,把 docker 里面的 emqx 版本和 haproxy 版本都改成你用的。然后按 readme操作 应该也是 ok 的。

这个是可以的 官方代码框复制就是有问题

然后经过验证,为啥我MQTT连接数到了2万8就卡住不网上叠加了呢,无论是k8s还是本地二进制起的。。。这算是一个问题么

搜索EMQ系统调优

我是按照那篇文档来调整的,但就是上不去连接数
系统调优 | EMQX文档
这是我优化的配置,看看哪里不对
[root@k8s-slave-prod01 emqx]# cat /etc/sysctl.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.file-max = 1000000000
fs.nr_open = 1000000000
[root@k8s-dev-slave ~]# cat /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.overcommit_memory = 1
net.ipv4.conf.all.route_localnet = 1
vm.panic_on_oom = 0
fs.inotify.max_user_watches = 89100
fs.file-max = 1000000000
fs.nr_open = 1000000000
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_buckets = 1048576
net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
net.netfilter.nf_conntrack_tcp_timeout_established = 21600
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_max_tw_buckets = 1048576
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 1638400
net.core.netdev_max_backlog = 16384
net.core.rmem_max = 262144000
net.core.wmem_max = 262144000
net.core.optmem_max = 16777216
net.ipv4.tcp_fin_timeout = 15
net.ipv4.ip_local_port_range = 3000 65535
net.ipv4.udp_mem = 74583000 499445000 749166000
net.ipv4.tcp_mem= 378150000 504200000 756300000
net.ipv4.tcp_rmem=1024 4096 16777216
net.ipv4.tcp_wmem=1024 4096 16777216
net.ipv4.tcp_fin_timeout=15
net.core.rmem_default=262144000
net.core.wmem_default=262144000
net.ipv4.tcp_fastopen=3

[root@k8s-slave-prod01 emqx]# cat /etc/security/limits.conf
emqx soft nofile 510000
emqx hard nofile 510000

  • soft nofile 1048576
  • hard nofile 1048576
  • soft nproc 655350
  • hard nproc 655350
  • hard memlock unlimited
  • soft memlock unlimited

[root@k8s-slave-prod01 emqx]# cat emqx.conf
node {
name = “emqx@127.0.0.1
cookie = “emqxsecretcookie”
data_dir = “/var/lib/emqx”
max_ports = 2097152
}
cluster {
name = emqxcl
discovery_strategy = singleton
}
dashboard {
listeners {
http.bind = 18083
# https.bind = 18084
https {
ssl_options {
certfile = “${EMQX_ETC_DIR}/certs/cert.pem”
keyfile = “${EMQX_ETC_DIR}/certs/key.pem”
}
}
}
}
mqtt {
max_packet_size = 1MB
max_clientid_len = 65535
max_topic_levels = 128
max_qos_allowed = 2
max_topic_alias = 65535
retain_available = true
shared_subscription = true
shared_subscription_strategy = round_robin
}
session {
max_subscriptions = infinity
await_rel_timeout = 600s
max_mqueue_len = 5000
}
flapping_detect {
enable = true
max_count = 15
window_time = “1m”
ban_time = “5m”
}
durable_sessions {
enable = true
}
listeners.tcp.default {
bind = “0.0.0.0:1883”
max_conn_rate = “10000/s”
messages_rate = “10000/s”
bytes_rate = “1240KB/s”
max_connections = 1000000
acceptors = “64”
}

顺便补充在k8s用operator部署的,容器内核参数
initContainers:
- name: busybox
image: Harbor
securityContext:
runAsUser: 0
runAsGroup: 0
capabilities:
add:
- SYS_ADMIN
drop:
- ALL
command:
- /bin/sh
- -c
- |
mount -o remount rw /proc/sys
sysctl -w net.core.somaxconn=1638400
sysctl -w net.ipv4.ip_local_port_range=“10000 65535”
sysctl -w kernel.core_uses_pid=0
sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w fs.nr_open=1000000000
sysctl -w fs.file-max=1000000000
sysctl -w net.ipv4.ip_local_port_range=‘1025 65534’
sysctl -w net.ipv4.udp_mem=‘74583000 499445000 749166000’
sysctl -w net.ipv4.tcp_max_sync_backlog=163840
sysctl -w net.core.netdev_max_backlog=163840
sysctl -w net.core.optmem_max=16777216
sysctl -w net.ipv4.tcp_rmem=‘1024 4096 16777216’
sysctl -w net.ipv4.tcp_wmem=‘1024 4096 16777216’
sysctl -w net.ipv4.tcp_max_tw_buckets=1048576
sysctl -w net.ipv4.tcp_fin_timeout=15
sysctl -w net.core.rmem_default=262144000
sysctl -w net.core.wmem_default=262144000
sysctl -w net.core.rmem_max=262144000
sysctl -w net.core.wmem_max=262144000
sysctl -w net.ipv4.tcp_mem=‘378150000 504200000 756300000’
sysctl -w net.netfilter.nf_conntrack_max=1048576
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=21600

嗯看起来没问题

有没有一些解决思路。。连接数的确卡在2w8就不动了,单机、容器都是

直连 emqx 看看,排除一下 haproxy 问题,ps 不要把压测客户端放在 emqx 的机器上

刚才自己排除了 问题在压测客户端 好嘛真没想到 贼六~
7d14e2ca3cc6b4b417fe736f7a4452f5

还有就是现在不能用 emqtt_bench 这个压测工具按照文档部署有问题 官网文档也是存在一些争议 只能用 mqttx-cli 来压测,也希望能够完善一下emqtt_bench 吧

如果源码编译用 emqtt_bench 有困难的话,推荐使用docker:
https://hub.docker.com/r/emqx/emqtt-bench

docker pull emqx/emqtt-bench
docker run -it emqx/emqtt-bench conn --help

好的,收到

也同样的看下那个测试工具的github和文档,一些内容的确有些问题呢,比如编译elang要求是25版本以上等等吧,其余都还挺好,其余没啥了,问题解决了,负载功能没问题了。