求助: 开源版5.8.9 listeners.tcp.default.max_conn_rate 无法准确限速

环境说明:

操作系统: ubuntu 22.04
内核版本: 5.15.0-119-generic 
系统架构: x86_64
软件版本: 开源版本 5.8.9
测试工具: eMQTT-Bench:latest

采取以下两种方案进行10/s限速测试:

  • 方案一: 测试秒级限速准确率, 创建客户端间隔50ms,即每秒20个连接,总计20个客户端
  • 方案二: 测试长时间限速准确率,创建客户端间隔50ms,即每秒20个连接,总计400个客户端
    测试结果:
  • 方案一: 15~17个客户端连接成功.
  • 方案二: 204个客户端成功连接.
    测试过程中第一秒客户端连接数和限速存在加大的差异.
    以下为测试日志:
ubt@abc:/tmp/emqx-bench/bin$ ./emqtt_bench conn --prefix=bench_sub_$RANDOM_$N --ssl=true -c 20 -i 50
Start with 8 workers, addrs pool size: 1 and req interval: 400 ms 

client(8): connect error - {ssl_closed,
                               {sslsocket,
                                   {gen_tcp,#Port<0.17>,tls_connection,
                                       undefined},
                                   [<0.224.0>,<0.219.0>]}}
client(14): connect error - {ssl_closed,
                             {sslsocket,
                              {gen_tcp,#Port<0.20>,tls_connection,undefined},
                              [<0.223.0>,<0.218.0>]}}
client(8): EXIT for {shutdown,ssl_closed}
client(14): EXIT for {shutdown,ssl_closed}
1s connect_succ total=15 rate=14.97/sec
1s connect_fail total=2 rate=2.00/sec
2s connect_succ total=17 rate=2.00/sec
3s connect_succ total=18 rate=1.00/sec
^C
ubt@abc:/tmp/emqx-bench/bin$ 
ubt@abc:/tmp/emqx-bench/bin$ 
ubt@abc:/tmp/emqx-bench/bin$ ./emqtt_bench conn --prefix=bench_sub_$RANDOM_$N --ssl=true -c 400 -i 50
Start with 8 workers, addrs pool size: 1 and req interval: 400 ms 

client(152): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.14>,tls_connection,undefined},
                               [<0.231.0>,<0.216.0>]}}
...
client(353): EXIT for {shutdown,ssl_closed}
1s connect_succ total=17 rate=16.97/sec
1s connect_fail total=7 rate=6.99/sec
client(4): connect error - {ssl_closed,
                               {sslsocket,
                                   {gen_tcp,#Port<0.31>,tls_connection,
                                       undefined},
                                   [<0.313.0>,<0.310.0>]}}
...
client(155): EXIT for {shutdown,ssl_closed}
2s connect_succ total=25 rate=8.00/sec
2s connect_fail total=15 rate=8.00/sec
client(206): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.50>,tls_connection,undefined},
                               [<0.421.0>,<0.413.0>]}}
...
client(108): EXIT for {shutdown,ssl_closed}
3s connect_succ total=37 rate=12.00/sec
3s connect_fail total=27 rate=12.00/sec
client(359): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.69>,tls_connection,undefined},
                               [<0.558.0>,<0.550.0>]}}
...
client(60): EXIT for {shutdown,ssl_closed}
4s connect_succ total=45 rate=8.00/sec
4s connect_fail total=35 rate=8.00/sec
client(211): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.91>,tls_connection,undefined},
                               [<0.663.0>,<0.661.0>]}}
...
client(163): EXIT for {shutdown,ssl_closed}
5s connect_succ total=57 rate=12.00/sec
5s connect_fail total=47 rate=12.00/sec
client(114): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.112>,tls_connection,undefined},
                               [<0.803.0>,<0.794.0>]}}
...
client(215): EXIT for {shutdown,ssl_closed}
6s connect_succ total=65 rate=8.00/sec
6s connect_fail total=55 rate=8.00/sec
client(366): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.125>,tls_connection,undefined},
                               [<0.875.0>,<0.874.0>]}}
...
client(318): EXIT for {shutdown,ssl_closed}
7s connect_succ total=76 rate=11.00/sec
7s connect_fail total=68 rate=13.00/sec
client(369): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.149>,tls_connection,undefined},
                               [<0.1027.0>,<0.1020.0>]}}
...
client(320): EXIT for {shutdown,ssl_closed}
8s connect_succ total=84 rate=8.00/sec
8s connect_fail total=76 rate=8.00/sec
client(371): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.165>,tls_connection,undefined},
                               [<0.1130.0>,<0.1123.0>]}}
...
client(273): EXIT for {shutdown,ssl_closed}
9s connect_succ total=96 rate=12.00/sec
9s connect_fail total=88 rate=12.00/sec
client(274): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.194>,tls_connection,undefined},
                               [<0.1281.0>,<0.1273.0>]}}
...
client(325): EXIT for {shutdown,ssl_closed}
10s connect_succ total=104 rate=8.00/sec
10s connect_fail total=96 rate=8.00/sec
client(276): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.209>,tls_connection,undefined},
                               [<0.1380.0>,<0.1375.0>]}}
...
client(128): EXIT for {shutdown,ssl_closed}
11s connect_succ total=116 rate=12.00/sec
11s connect_fail total=108 rate=12.00/sec
client(129): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.235>,tls_connection,undefined},
                               [<0.1519.0>,<0.1515.0>]}}
...
client(80): EXIT for {shutdown,ssl_closed}
12s connect_succ total=124 rate=8.00/sec
12s connect_fail total=116 rate=8.00/sec
client(31): connect error - {ssl_closed,
                             {sslsocket,
                              {gen_tcp,#Port<0.246>,tls_connection,undefined},
                              [<0.1614.0>,<0.1606.0>]}}
...
client(83): EXIT for {shutdown,ssl_closed}
13s connect_succ total=136 rate=12.00/sec
13s connect_fail total=128 rate=12.00/sec
client(134): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.273>,tls_connection,undefined},
                               [<0.1767.0>,<0.1765.0>]}}
...
client(85): EXIT for {shutdown,ssl_closed}
14s connect_succ total=144 rate=8.00/sec
14s connect_fail total=136 rate=8.00/sec
client(336): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.291>,tls_connection,undefined},
                               [<0.1859.0>,<0.1857.0>]}}
...
client(188): EXIT for {shutdown,ssl_closed}
15s connect_succ total=156 rate=12.00/sec
15s connect_fail total=148 rate=12.00/sec
client(339): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.316>,tls_connection,undefined},
                               [<0.2003.0>,<0.1994.0>]}}
...
client(90): EXIT for {shutdown,ssl_closed}
16s connect_succ total=164 rate=8.00/sec
16s connect_fail total=156 rate=8.00/sec
client(391): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.325>,tls_connection,undefined},
                               [<0.2089.0>,<0.2081.0>]}}
...
client(243): EXIT for {shutdown,ssl_closed}
17s connect_succ total=176 rate=12.00/sec
17s connect_fail total=168 rate=12.00/sec
client(144): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.351>,tls_connection,undefined},
                               [<0.2245.0>,<0.2240.0>]}}
...
client(195): EXIT for {shutdown,ssl_closed}
18s connect_succ total=184 rate=8.00/sec
18s connect_fail total=176 rate=8.00/sec
client(246): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.369>,tls_connection,undefined},
                               [<0.2331.0>,<0.2326.0>]}}
...
client(398): EXIT for {shutdown,ssl_closed}
19s connect_succ total=196 rate=12.00/sec
19s connect_fail total=188 rate=12.00/sec
client(249): connect error - {ssl_closed,
                              {sslsocket,
                               {gen_tcp,#Port<0.394>,tls_connection,undefined},
                               [<0.2484.0>,<0.2481.0>]}}
...
client(350): EXIT for {shutdown,ssl_closed}
20s connect_succ total=204 rate=8.00/sec
20s connect_fail total=196 rate=8.00/sec

日志中对部分报错日志进行裁剪,避免超出发帖限制

另外还进行了几次不同的限速值的测试:

  • 限速10/s 客户端创建间隔: 50ms 总客户端: 400
1s connect_succ total=17 rate=16.97/sec
1s connect_fail total=7 rate=6.99/sec
2s connect_succ total=25 rate=8.00/sec
2s connect_fail total=15 rate=8.00/sec
3s connect_succ total=37 rate=12.00/sec
3s connect_fail total=27 rate=12.00/sec
4s connect_succ total=45 rate=8.00/sec
4s connect_fail total=35 rate=8.00/sec
5s connect_succ total=57 rate=12.00/sec
5s connect_fail total=47 rate=12.00/sec
6s connect_succ total=65 rate=8.00/sec
6s connect_fail total=55 rate=8.00/sec
7s connect_succ total=76 rate=11.00/sec
7s connect_fail total=68 rate=13.00/sec
8s connect_succ total=84 rate=8.00/sec
8s connect_fail total=76 rate=8.00/sec
9s connect_succ total=96 rate=12.00/sec
9s connect_fail total=88 rate=12.00/sec
10s connect_succ total=104 rate=8.00/sec
10s connect_fail total=96 rate=8.00/sec
11s connect_succ total=116 rate=12.00/sec
11s connect_fail total=108 rate=12.00/sec
12s connect_succ total=124 rate=8.00/sec
12s connect_fail total=116 rate=8.00/sec
13s connect_succ total=136 rate=12.00/sec
13s connect_fail total=128 rate=12.00/sec
14s connect_succ total=144 rate=8.00/sec
14s connect_fail total=136 rate=8.00/sec
15s connect_succ total=156 rate=12.00/sec
15s connect_fail total=148 rate=12.00/sec
16s connect_succ total=164 rate=8.00/sec
16s connect_fail total=156 rate=8.00/sec
17s connect_succ total=176 rate=12.00/sec
17s connect_fail total=168 rate=12.00/sec
18s connect_succ total=184 rate=8.00/sec
18s connect_fail total=176 rate=8.00/sec
19s connect_succ total=196 rate=12.00/sec
19s connect_fail total=188 rate=12.00/sec
20s connect_succ total=204 rate=8.00/sec
20s connect_fail total=196 rate=8.00/sec
  • 限速20/s 客户端创建间隔: 25ms 总客户端 800
1s connect_succ total=36 rate=35.93/sec
1s connect_fail total=4 rate=3.99/sec
2s connect_succ total=54 rate=18.00/sec
2s connect_fail total=26 rate=22.00/sec
3s connect_succ total=74 rate=20.00/sec
3s connect_fail total=46 rate=20.00/sec
4s connect_succ total=94 rate=20.00/sec
4s connect_fail total=66 rate=20.00/sec
5s connect_succ total=114 rate=20.00/sec
5s connect_fail total=86 rate=20.00/sec
6s connect_succ total=134 rate=20.00/sec
6s connect_fail total=106 rate=20.00/sec
7s connect_succ total=154 rate=20.00/sec
7s connect_fail total=126 rate=20.00/sec
8s connect_succ total=174 rate=20.00/sec
8s connect_fail total=146 rate=20.00/sec
9s connect_succ total=194 rate=20.00/sec
9s connect_fail total=166 rate=20.00/sec
10s connect_succ total=214 rate=20.00/sec
10s connect_fail total=186 rate=20.00/sec
11s connect_succ total=234 rate=20.00/sec
11s connect_fail total=206 rate=20.00/sec
12s connect_succ total=254 rate=20.00/sec
12s connect_fail total=226 rate=20.00/sec
13s connect_succ total=274 rate=20.00/sec
13s connect_fail total=246 rate=20.00/sec
14s connect_succ total=294 rate=20.00/sec
14s connect_fail total=266 rate=20.00/sec
15s connect_succ total=314 rate=20.00/sec
15s connect_fail total=286 rate=20.00/sec
16s connect_succ total=334 rate=20.00/sec
16s connect_fail total=306 rate=20.00/sec
17s connect_succ total=354 rate=20.00/sec
17s connect_fail total=326 rate=20.00/sec
18s connect_succ total=374 rate=20.00/sec
18s connect_fail total=346 rate=20.00/sec
19s connect_succ total=394 rate=20.00/sec
19s connect_fail total=366 rate=20.00/sec
20s connect_succ total=414 rate=20.00/sec
20s connect_fail total=386 rate=20.00/sec
  • 限速80/s 客户端创建间隔: 10ms 总客户端: 2000
1s connect_succ total=104 rate=103.79/sec
2s connect_succ total=200 rate=96.00/sec
3s connect_succ total=296 rate=96.00/sec
4s connect_succ total=392 rate=96.00/sec
4s connect_fail total=8 rate=2.00/sec
5s connect_succ total=464 rate=72.00/sec
5s connect_fail total=32 rate=24.00/sec
6s connect_succ total=544 rate=80.00/sec
6s connect_fail total=48 rate=16.00/sec
7s connect_succ total=624 rate=80.00/sec
7s connect_fail total=72 rate=24.00/sec
8s connect_succ total=704 rate=80.00/sec
8s connect_fail total=88 rate=16.00/sec
9s connect_succ total=784 rate=79.92/sec
9s connect_fail total=104 rate=15.98/sec
10s connect_succ total=864 rate=80.08/sec
10s connect_fail total=128 rate=24.02/sec
11s connect_succ total=944 rate=80.00/sec
11s connect_fail total=144 rate=16.00/sec
12s connect_succ total=1016 rate=72.00/sec
12s connect_fail total=168 rate=24.00/sec
13s connect_succ total=1104 rate=88.00/sec
13s connect_fail total=184 rate=16.00/sec
14s connect_succ total=1176 rate=72.00/sec
14s connect_fail total=208 rate=24.00/sec
15s connect_succ total=1256 rate=80.00/sec
15s connect_fail total=224 rate=16.00/sec
16s connect_succ total=1336 rate=80.00/sec
16s connect_fail total=248 rate=24.00/sec
17s connect_succ total=1416 rate=80.00/sec
17s connect_fail total=264 rate=16.00/sec
18s connect_succ total=1496 rate=80.00/sec
18s connect_fail total=288 rate=24.00/sec
19s connect_succ total=1576 rate=80.00/sec
19s connect_fail total=304 rate=16.00/sec
20s connect_succ total=1656 rate=80.00/sec
20s connect_fail total=320 rate=16.00/sec
21s connect_succ total=1672 rate=16.00/sec
21s connect_fail total=328 rate=8.00/sec

启动日志里的“8 workers、req interval: 400 ms”表示每个 worker 间隔约 400ms,8 个 worker 近乎同时启动,并不是整个工具严格每 50ms 发起一个连接。

尤其是 -c 20 这一轮,按当前源码,20 个客户端分配给 8 个 worker 的数量为 2、2、2、2、2、2、2、6。因此发起时间大致是:0ms 约 8 个,400ms 约 8 个,随后在 800、1200、1600、2000ms 各约 1 个。也就是说,这 20 个连接并不是 1 秒内均匀发完的。按这个节奏和 10/s 令牌桶推演,第一秒约 15 成功、2 失败,随后成功数累计到 17、18,与日志一致。

-c 400 能被 8 整除,流量更接近每 400ms 一批 8 个,因此 1 秒统计窗口里出现 8、12 交替并不奇怪。相邻两个稳定窗口合起来约 20 个,平均仍约为 10/s。启动后的前几秒还可能消耗之前积累的令牌,所以短时成功率超过配置值,并不等于限流失效。

要把发起节奏调平滑,可以在测试环境用单 scheduler 跑一轮,并将 BROKER 替换为实际地址:

ERL_ZFLAGS=“+S 1:1” ./emqtt_bench conn -h BROKER -p 1883 -c 1200 -i 50

确认启动日志显示 1 worker、间隔 50ms。建议持续观察约 60 秒,跳过前 10 秒启动阶段,用后续成功数之差除以实际时间。单 worker 能消除批量发起的影响,但不会消除令牌桶本身的突发能力。

标题说的的是 tcp.default,命令却带了 --ssl=true。测试明文 TCP 监听器时去掉 SSL;测试 TLS 时,检查对应 ssl 监听器的配置并明确指定端口。–ssl=true 不会自动把默认端口切到 8883。

抱歉,测试环境部署在公网对外开放,为了安全删除命令行中的服务地址/端口以及账号密码信息,遗漏了–ssl=true参数造成误解,请见谅.

以下为配置ERL_ZFLAGS=“+S 1:1”的执行日志,第一秒的时候限速和配置值还是存在一定的差异

  • 限速10/s
ubt@ubt:/tmp/emqx/bin$ ERL_ZFLAGS="+S 1:1" ./emqtt_bench conn --prefix=bench_sub_$RANDOM_$N -c 500 -i 10|grep 'connect_'
1s connect_succ total=19 rate=18.96/sec
1s connect_fail total=66 rate=65.87/sec
2s connect_succ total=29 rate=10.00/sec
2s connect_fail total=147 rate=81.00/sec
3s connect_succ total=39 rate=10.00/sec
3s connect_fail total=228 rate=81.00/sec
4s connect_succ total=49 rate=10.00/sec
4s connect_fail total=308 rate=79.92/sec
5s connect_succ total=59 rate=10.00/sec
5s connect_fail total=389 rate=81.08/sec
6s connect_succ total=65 rate=6.00/sec
6s connect_fail total=435 rate=46.00/sec
^C
ubt@ubt:/tmp/emqx/bin$ ERL_ZFLAGS="+S 1:1" ./emqtt_bench conn --prefix=bench_sub_$RANDOM_$N -c 500 -i 20|grep 'connect_'
1s connect_succ total=19 rate=18.96/sec
1s connect_fail total=26 rate=25.95/sec
2s connect_succ total=29 rate=10.00/sec
2s connect_fail total=64 rate=38.00/sec
3s connect_succ total=38 rate=9.00/sec
3s connect_fail total=102 rate=38.00/sec
4s connect_succ total=48 rate=10.00/sec
4s connect_fail total=140 rate=38.00/sec
5s connect_succ total=58 rate=10.00/sec
5s connect_fail total=178 rate=38.00/sec
6s connect_succ total=68 rate=10.00/sec
6s connect_fail total=215 rate=36.96/sec
7s connect_succ total=78 rate=10.00/sec
7s connect_fail total=253 rate=38.04/sec
8s connect_succ total=88 rate=10.00/sec
8s connect_fail total=291 rate=38.00/sec
9s connect_succ total=98 rate=10.00/sec
9s connect_fail total=328 rate=37.00/sec
10s connect_succ total=108 rate=10.00/sec
10s connect_fail total=366 rate=38.00/sec
11s connect_succ total=113 rate=5.00/sec
11s connect_fail total=387 rate=21.00/sec
  • 限速20/s
ubt@ubt:/tmp/emqx/bin$ ERL_ZFLAGS="+S 1:1" ./emqtt_bench conn --prefix=bench_sub_$RANDOM_$N -c 500 -i 10|grep 'connect_'
1s connect_succ total=38 rate=37.89/sec
1s connect_fail total=46 rate=45.86/sec
2s connect_succ total=57 rate=19.00/sec
2s connect_fail total=118 rate=72.00/sec
3s connect_succ total=76 rate=19.00/sec
3s connect_fail total=190 rate=72.00/sec
4s connect_succ total=96 rate=20.00/sec
4s connect_fail total=261 rate=70.93/sec
5s connect_succ total=116 rate=20.00/sec
5s connect_fail total=331 rate=70.07/sec
6s connect_succ total=128 rate=12.00/sec
6s connect_fail total=372 rate=41.00/sec
^C
ubt@ubt:/tmp/emqx/bin$ ERL_ZFLAGS="+S 1:1" ./emqtt_bench conn --prefix=bench_sub_$RANDOM_$N -c 500 -i 20|grep 'connect_'
1s connect_succ total=37 rate=36.93/sec
1s connect_fail total=8 rate=7.98/sec
2s connect_succ total=56 rate=19.00/sec
2s connect_fail total=36 rate=28.00/sec
3s connect_succ total=76 rate=20.00/sec
3s connect_fail total=64 rate=27.97/sec
4s connect_succ total=96 rate=20.00/sec
4s connect_fail total=92 rate=28.03/sec
5s connect_succ total=116 rate=20.00/sec
5s connect_fail total=119 rate=27.00/sec
6s connect_succ total=136 rate=20.00/sec
6s connect_fail total=147 rate=28.00/sec
7s connect_succ total=156 rate=20.00/sec
7s connect_fail total=175 rate=28.00/sec
8s connect_succ total=176 rate=20.00/sec
8s connect_fail total=202 rate=27.00/sec
9s connect_succ total=196 rate=20.00/sec
9s connect_fail total=230 rate=28.00/sec
10s connect_succ total=215 rate=19.00/sec
10s connect_fail total=258 rate=28.00/sec
11s connect_succ total=226 rate=11.00/sec
11s connect_fail total=274 rate=16.00/sec

已在另外一张帖子中活动官方解答,再次感谢