emqx 4.x使用redis 做为身份验证和权限控制后端时的bug

错误报告

环境

  • EMQX 版本: 4.4.15
  • 操作系统版本:Ubuntu 20

重现此问题的步骤

  1. 创建redis cluster 5 或 6,其中包含500万用于emqx认证的用户,这可能不是主要原因,但redis的slots分片不连续,有三个shard,每个shard上有300个左右的不连续slots range。
  2. 创建emqx集群(3节点),静态手动的集群,emqx_ctl cluster join方式
  3. 修改/etc/emqx/plugins/emqx_auth_redis.conf,配置使用redis cluster 5或6做为后端
  4. 通过dashboard或者emqx_ctl plugins reload emqx_auth_redis启动和加载redis 插件
  5. 问题出现:
    4.1 问题1: redis cluster 的分片的master节点上产生数千上万的tcp连接(是redis节点间的内部连接,但在启用emqx redis插件前不会出现这些数量的连接, 只要停止emqx redis插件或者停止emqx服务,这些连接就会消失)
    4.2 问题2: 在正确的配置了redis验证和授权后,出现授权不符合预期的情况。即:A. 设置了emqx.conf中的allow_anonymous = false 和 acl_nomatch = deny, B. 设置了acl.conf的最后一条改为{deny, all}. 但是在这个情况下,在用户连接上emqx服务后,如果redis后端服务停止了,这个连接上的用户就具有了不受限的pub/sub权限,可以往任意topic上发布或监听。
    4.3 问题3: 在以上设置下,如果在redis的key中只有验证密码的key而没有acl授权的key, 这个用户也具有不受限的pub/sub权限。

预期行为

实际行为


功能请求

描述你需要的功能

为什么你需要这个功能


其他

你好,

  • 问题1:很多连接的事情看起来像是 Redis 的问题,我没有见过类似问题也没搜到相关的现象,你能继续调查一下吗?主要是看看 emqx 到 redis 一共有多少连接,然后试试其他的程序连接 redis cluster,看看会不会出现同样的现象。
  • 问题2 和 问题 3 我一会儿尝试一下本地复现。


root@emqxbackup001:/tmp# netstat -ntap | grep ESTAB| grep 6379 | awk ‘{print $NF}’ | sort | uniq -c | sort -k1n
1 -
17320 22338/emqxbackup001
20000 34823/redis-server
root@emqxbackup001:/tmp# ps -p 22338 -fl ww
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 S emqx 22338 22264 0 80 0 - 952111 poll_s Mar10 pts/1 28:00 emqxbackup001 -P 2097152 -Q 1048576 -e 256000 -spp true -A 4 -IOt 4 -SDio 8 – -root /usr/lib/emqx -progname usr/bin/emqx – -home /var/lib/emqx – -boot /usr/lib/emqx/releases/4.4.15/start -mode embedded -boot_var ERTS_LIB_DIR /usr/lib/emqx/erts-12.3.2.2/…/lib -mnesia dir “/var/lib/emqx/mnesia/emqxbackup001@172.19.178.149” -config /var/lib/emqx/configs/app.2023.03.10.11.46.10.config -kernel net_ticktime 120 -shutdown_time 30000 -pa /var/lib/emqx/patches -pa /usr/lib/emqx/releases/4.4.15/consolidated -setcookie yunduolalxqemqxsecretcookie -name emqxbackup001@172.19.178.149 – -vm_args /var/lib/emqx/configs/vm.2023.03.10.11.46.10.args -start_epmd false -epmd_module ekka_epmd -proto_dist ekka – console –

root@emqxbackup001:/tmp# redis-cli -c info

Server

redis_version:7.0.9
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:c8bd85726a87e0b
redis_mode:cluster
os:Linux 5.4.0-144-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:9.4.0
process_id:34823
process_supervised:no
run_id:c17a756e5ee860daf3f83ce761384cb0433bd6a5
tcp_port:6379
server_time_usec:1678688942088602
uptime_in_seconds:9419
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:967342
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
io_threads_active:0

Clients

connected_clients:1
cluster_connections:4
maxclients:20000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

Memory

used_memory:411705016
used_memory_human:392.63M
used_memory_rss:423288832
used_memory_rss_human:403.68M
used_memory_peak:582526336
used_memory_peak_human:555.54M
used_memory_peak_perc:70.68%
used_memory_overhead:133034320
used_memory_startup:2128440
used_memory_dataset:278670696
used_memory_dataset_perc:68.04%
allocator_allocated:411819608
allocator_active:412536832
allocator_resident:420356096
total_system_memory:16239435776
total_system_memory_human:15.12G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:717224
allocator_rss_ratio:1.02
allocator_rss_bytes:7819264
rss_overhead_ratio:1.01
rss_overhead_bytes:2932736
mem_fragmentation_ratio:1.03
mem_fragmentation_bytes:11623688
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:0
mem_cluster_links:4352
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

Persistence

loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1678682161
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:4
rdb_current_bgsave_time_sec:-1
rdb_saves:7
rdb_last_cow_size:1318912
rdb_last_load_keys_expired:675
rdb_last_load_keys_loaded:1221403
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

Stats

total_connections_received:30436
total_commands_processed:1310415
instantaneous_ops_per_sec:0
total_net_input_bytes:325894934
total_net_output_bytes:6949828
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:1723512
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:715
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:1672
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:7758
total_forks:7
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:1
dump_payload_sanitizations:0
total_reads_processed:246911
total_writes_processed:216477
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:30420
reply_buffer_expands:0

Replication

role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:61f5635f980e1a84ce9d8763b8405444f02b2a96
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:338328733
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:83.684921
used_cpu_user:98.213107
used_cpu_sys_children:1.531550
used_cpu_user_children:16.020206
used_cpu_sys_main_thread:83.625955
used_cpu_user_main_thread:98.195934

Modules

Errorstats

errorstat_MOVED:count=1

Cluster

cluster_enabled:1

Keyspace

db0:keys=1217632,expires=1214980,avg_ttl=4716023399

root@emqxbackup001:/tmp# redis-cli -c cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3
cluster_size:3
cluster_current_epoch:516
cluster_my_epoch:516
cluster_stats_messages_ping_sent:9437
cluster_stats_messages_pong_sent:19715
cluster_stats_messages_update_sent:23
cluster_stats_messages_sent:29175
cluster_stats_messages_ping_received:8835
cluster_stats_messages_pong_received:9435
cluster_stats_messages_update_received:11
cluster_stats_messages_received:18281
total_cluster_links_buffer_limit_exceeded:0

root@emqxbackup001:/tmp# redis-cli -c cluster nodes
d595aefc6b29ada3d84594a8c304df9ab4bc2916 172.19.123.114:6379@16379 master - 0 1678688988411 515 connected 2727-2815 2817-2882 2884-2994 3005 3009 3011 3014-3015 3021 3025 3027 3029 3034 3038-3039 3041 3043 3046-3047 3055-3056 3060 3062-3063 3065 3069 3075-3077 3079 3082-3083 3086 3088-3090 3096 3099-3100 3102 3104 3110 3113-3117 3122 3124 3126-3127 3129 3136-3138 3141-3142 3147 3151 3159-3160 3163-3165 3168-3169 3174 3176 3180 3182 3184-3185 3189 3191 3195 3197-3198 3201 3203-3205 3207 3215 3218 3222 3225 3227 3229-3230 3232-3233 3237 3241 3249 3251 3253-3254 3256-3257 3262 3265 3269 3271 3274-3277 3279 3281-3282 3286 3299 3302-3303 3305-3306 3308-3309 3314 3321 3323 3328 3332 3335 3339-3340 3344-3347 3350 3354 3357 3361 3364-3365 3369 3372 3381 3385-3387 3389 3396 3401 3403-3406 3411 3419-3420 3426-3427 3435-3439 3443 3446-3447 3449 3452 3454 3462-3463 3465 3467 3469-3470 3474 3486 3491-3492 3494 3498 3500 3504-3506 3508 3510 3522 3530-3531 3533 3536-3537 3539 3543 3546 3553-3554 3562-3563 3566-3567 3569 3572 3575 3579-3581 3584 3586 3591-3592 3594-3595 3598 3600 3602-3604 3608 3610 3615-3616 3620-3621 3627-3628 3631 3636-3638 3644-3645 3649-3650 3654-3656 3658-3659 3662 3668 3671 3674 3676 3682-3683 3690 3692 3697 3700 3702-3703 3705 3707 3709 3712 3720-3721 3725-3726 3731-3734 3736 3738-3739 3743-3744 3755-3756 3758 3761-3762 3766-3767 3772 3774 3778-3779 3781 3783-3786 3788-3789 3795-3796 3798 3801 3807 3809-3811 3819 3822 3828 3831-3832 3841 3845 3850 3856 3860 3862 3864 3868-3869 3872 3874-3876 3878-3879 3883 3885-3886 3888 3892 3900-3901 3906-3907 3909 3913 3916-3919 3934 3936-3937 3939 3941 3945 3953-3956 3960-3962 3965 3967 3970 3975-3977 3980-3982 3984 3986 3994 3996-3997 4004-4005 4007 4010 4015 4019 4022-4023 4026 4028 4032-4033 4037 4040-4041 4044 4047 4049-4051 4054 4059 4061 4066-4071 4073 4076-4077 4083 4088-4090 4092 4096 4098-4099 4101-4102 4105 4109 4111-4112 4115 4117 4119-4120 4122-4123 4126 4129 4131 4135 4146-4148 4153 4157-4158 4161-4162 4167-4168 4170 4176-4178 4187 4189 4193-4194 4202-4204 4206 4208-4209 4211 4215 4217 4223-4224 4226 4233 4235 4239 4241 4250 4257-4258 4260 4262 4265 4267-4268 4270 4273-4274 4276 4279 4286 4288 4297 4299 4301-4303 4305 4311 4313 4316 4318 4326-4327 4330 4334-4335 4340 4343-4344 4350-4351 4357-4359 4363 4365 4367 4369 4373 4376 4380 4382 4389 4394 4397 4400 4408-4409 4414-4415 4419 4421-4423 4425-4426 4431 4433 4435-4436 4440 4450-4451 4453-4454 4456 4458 4467 4469 4472 4477 4482-4483 4491 4494-4496 4498 4500 4502-4503 4507 4512 4525 4530 4532-4533 4535 4537-4540 4545-4546 4549-4550 4554-4558 4561-4565 4567-4568 4570 4573 4577-4579 4582 4587-4588 4590 4592-4594 4596 4599-4600 4602 4609 4612 4615 4618 4625-4626 4628 4631-4635 4637 4641 4645 4648-4651 4653 4655 4658 4661-4662 4664 4668 4671 4676-4677 4680-4681 4684 4686-4688 4694 4696 4701-4702 4704 4706 4709-4711 4717-4719 4721 4723 4727 4732 4734 4736 4739 4741 4750-4751 4754 4756 4761 4776-4777 4784-4785 4787-4788 4790 4798 4807-4808 4817-4818 4820 4823-4825 4829-4831 4836 4841-4842 4849 4854-4855 4859-4861 4863-4868 4872 4874 4882-4883 4889 4891 4894 4898-4903 4905 4909 4911 4913-4915 4919 4921-4922 4930 4932 4936-4937 4942 4946 4948 4950 4952 4956 4961 4964 4968 4971 4973-4974 4976-4977 4985 4996 4999 5001 5008 5011 5013 5016 5022 5026-5028 5030 5032-5035 5038 5040-5041 5043 5047-5048 5050 5053 5059 5061 5069-5070 5073-5074 5078 5081-5083 5091 5093-5095 5099-5100 5107-5108 5110 5116 5120 5123 5125 5127 5129-5130 5132 5135 5140 5145-5146 5148 5151 5154 5156 5161-5162 5164 5168-5169 5176-5178 5180-5181 5186 5189 5195 5198 5205-5206 5208 5211-5212 5214 5221 5225 5239-5241 5244-5245 5247 5249 5252 5254 5256 5258 5261-5262 5266 5268 5271 5273-5274 5280 5283-5285 5287 5292 5294 5300 5303-5305 5307 5309-5310 5312-5313 5315-5317 5321 5324 5328 5330 5335 5343 5348 5350-5352 5357 5361 5367 5372 5375 5379-5380 5382-5383 5386-5388 5392 5396 5398-5399 5401-5402 5407 5413-5415 5417 5419 5421 5423-5428 5430-5432 5434-5435 5437-5439 5446-5448 5451 5453 5458 5462-5463 5467 5470 5473 5475 5479 5483 5485-5487 5491 5493-5495 5497-5498 5507 5509-5510 5514 5516-5518 5522-5523 5526 5532 5534 5541 5543 5552 5555 5558-5559 5561 5565-5566 5568-5569 5571-5572 5574-5575 5577 5580-5581 5584-5585 5589 5592 5596 5602 5606 5608-5610 5613 5616-5617 5622 5627 5635 5641-5644 5648-5649 5651 5653 5658-5659 5664 5666 5668-5671 5673-5674 5676 5681 5683 5690-5692 5697 5703-5705 5708 5714-5716 5718-5719 5721 5725 5727-5728 5731-5732 5734-5735 5737 5742 5747 5749 5757 5759 5761-5762 5765-5767 5769-5770 5772-5774 5776 5778 5780-5782 5787-5788 5792 5795 5797 5799 5803 5806-5808 5815-5817 5820-5822 5830-5831 5833 5837 5849-5850 5852-5855 5860-5861 5863 5865 5867 5874 5877 5879 5888 5900-5901 5904 5912 5917-5918 5920 5925 5929 5934-5936 5942-5943 5947 5954 5964-5966 5972-5973 5975 5980 5982-5983 5988-5989 5992 5994-5996 5998-5999 6002 6004 6012-6014 6022-6023 6026 6028 6030 6032-6033 6039-6040 6049-6051 6056 6059-6061 6065 6067-6069 6075-6080 6084 6088 6091 6094 6102-6103 6105 6108 6110-6112 6114 6116-6118 6130-6131 6133 6137 6140-6141 6147 6150 6153 6158-6159 6162 6166 6168 6171 6177 6186 6191 6201-6202 6205-6207 6211 6216-6217 6223 6225-6226 6228-6229 6231 6234 6239 6242 6244-6246 6248-6250 6272-6273 6275-6277 6280 6283 6285 6289 6291-6292 6297 6301-6302 6307-6309 6312 6314 6324 6327 6329 6332-6333 6340-6341 6347 6349-6350 6352 6364 6366 6369 6377 6380-6382 6384 6386 6389-6390 6393 6397-6399 6403-6404 6406 6410-6411 6414-6415 6420-6421 6423 6425 6431-6432 6438-6440 6442 6447 6449-6450 6456 6460 6465-6467 6482 6485 6487-6488 6490-6491 6494 6496-6497 6500 6503 6506 6510-6511 6514-6515 6517 6519-6520 6522 6524 6528 6530-6531 6538 6543-6547 6551-6553 6555-6556 6559 6563 6565 6568-6570 6576 6579 6587-6588 6591-6593 6595 6597-6598 6601 6608-6609 6611 6614-6616 6618-6619 6621-6622 6625 6627 6629 6638 6644-6646 6655 6658 6662 6664 6666-6667 6672 6674 6680-6681 6687 6693 6697 6700 6702 6704-6705 6708 6718-6719 6722 6724 6726 6731-6732 6736 6741 6747 6752-6753 6755 6757 6766 6768-6769 6772 6774-6775 6781 6783 6786-6788 6790 6794 6799-6800 6805-6807 6809 6816 6818 6821 6828-6836 6838-6846 6848-6851 6853-6856 6858-6881 6883-6887 6889-6892 6896-6905 6907-6914 6916-6921 6924-6926 6928-6932 6934-6938 6940-6947 6949-6957 6959 6962-6967 6971-6973 6975-6987 6989-6991 6993-6997 6999-7000 7003-7012 7015-7024 7028-7030 7032-7034 7036 7038-7039 7041-7047 7049-7050 7052-7058 7060-7061 7065 7067-7071 7073 7075-7079 7081-7084 7086-7094 7096-7100 7102-7106 7108 7110 7112 7114-7117 7119 7121 7124 7126-7129 7131-7149 7151-7169 7174-7175 7177-7178 7180-7181 7183-7185 7187 7189-7197 7199-7208 7213 7215-7216 7218-7221 7223-7224 7226-7235 7237-7239 7241-7242 7245 7247-7248 7250-7251 7253-7255 7259-7271 7273-7289 7291-7293 7296 7298-7310 7312-7314 7316-7322 7324-7328 7330-7334 7336-7344 7346-7348 7350-7353 7355-7369 7371-7387 7389-7390 7392-7393 7395-7397 7400-7401 7403-7410 7412-7415 7417-7424 7427 7429-7435 7437-7440 7442-7444 7446-7450 7452-7462 7464-7479 7481-7493 7495-7498 7500 7502 7504 7506 7508 7510-7513 7515-7521 7523-7526 7528-7532 7536-7538 7540-7546 7548-7550 7554-7556 7558-7560 7562-7563 7566-7584 7586-7595 7598 7600-7604 7607-7608 7612-7629 7631 7633-7634 7636-7638 7640-7642 7644-7648 7650-7658 7661 7663 7665-7668 7670-7681 7683-7684 7686-7692 7694-7718 7720 7722 7724-7728 7730 7732-7743 7745-7746 7748-7754 7757-7764 7766-7768 7770-7789 7791-7802 7804-7805 7807 7809-7813 7817-7820 7822-7824 7826-7828 7830-7831 7833-7834 7836-7842 7844-7848 7850-7853 7856-7864 7869-7888 7891-7894 7897-7898 7900 7902-7903 7905-7913 7915-7918 7920-7927 7930-7931 7933-7934 7936-7947 7949 7951-7958 7960-7966 7968-7971 7973 7975-7977 7979-7986 7988-7997 7999-8004 8007-8008 8010-8014 8016-8018 8020-8036 8038-8039 8041-8046 8049-8060 8063-8071 8073-8074 8076-8077 8080-8087 8089 8091-8095 8097-8109 8111-8134 8137-8140 8143-8152 8154-8158 8160-8161 8163 8165-8167 8169-8176 8180-8184 8186-10922
0a05e3ea0f79ecf2acd73477d03e4ef1075b6f95 172.19.178.149:6379@16379 myself,master - 0 1678688982000 516 connected 0-2726 2816 2883 2995-3004 3006-3008 3010 3012-3013 3016-3020 3022-3024 3026 3028 3030-3033 3035-3037 3040 3042 3044-3045 3048-3054 3057-3059 3061 3064 3066-3068 3070-3074 3078 3080-3081 3084-3085 3087 3091-3095 3097-3098 3101 3103 3105-3109 3111-3112 3118-3121 3123 3125 3128 3130-3135 3139-3140 3143-3146 3148-3150 3152-3158 3161-3162 3166-3167 3170-3173 3175 3177-3179 3181 3183 3186-3188 3190 3192-3194 3196 3199-3200 3202 3206 3208-3214 3216-3217 3219-3221 3223-3224 3226 3228 3231 3234-3236 3238-3240 3242-3248 3250 3252 3255 3258-3261 3263-3264 3266-3268 3270 3272-3273 3278 3280 3283-3285 3287-3298 3300-3301 3304 3307 3310-3313 3315-3320 3322 3324-3327 3329-3331 3333-3334 3336-3338 3341-3343 3348-3349 3351-3353 3355-3356 3358-3360 3362-3363 3366-3368 3370-3371 3373-3380 3382-3384 3388 3390-3395 3397-3400 3402 3407-3410 3412-3418 3421-3425 3428-3434 3440-3442 3444-3445 3448 3450-3451 3453 3455-3461 3464 3466 3468 3471-3473 3475-3485 3487-3490 3493 3495-3497 3499 3501-3503 3507 3509 3511-3521 3523-3529 3532 3534-3535 3538 3540-3542 3544-3545 3547-3552 3555-3561 3564-3565 3568 3570-3571 3573-3574 3576-3578 3582-3583 3585 3587-3590 3593 3596-3597 3599 3601 3605-3607 3609 3611-3614 3617-3619 3622-3626 3629-3630 3632-3635 3639-3643 3646-3648 3651-3653 3657 3660-3661 3663-3667 3669-3670 3672-3673 3675 3677-3681 3684-3689 3691 3693-3696 3698-3699 3701 3704 3706 3708 3710-3711 3713-3719 3722-3724 3727-3730 3735 3737 3740-3742 3745-3754 3757 3759-3760 3763-3765 3768-3771 3773 3775-3777 3780 3782 3787 3790-3794 3797 3799-3800 3802-3806 3808 3812-3818 3820-3821 3823-3827 3829-3830 3833-3840 3842-3844 3846-3849 3851-3855 3857-3859 3861 3863 3865-3867 3870-3871 3873 3877 3880-3882 3884 3887 3889-3891 3893-3899 3902-3905 3908 3910-3912 3914-3915 3920-3933 3935 3938 3940 3942-3944 3946-3952 3957-3959 3963-3964 3966 3968-3969 3971-3974 3978-3979 3983 3985 3987-3993 3995 3998-4003 4006 4008-4009 4011-4014 4016-4018 4020-4021 4024-4025 4027 4029-4031 4034-4036 4038-4039 4042-4043 4045-4046 4048 4052-4053 4055-4058 4060 4062-4065 4072 4074-4075 4078-4082 4084-4087 4091 4093-4095 4097 4100 4103-4104 4106-4108 4110 4113-4114 4116 4118 4121 4124-4125 4127-4128 4130 4132-4134 4136-4145 4149-4152 4154-4156 4159-4160 4163-4166 4169 4171-4175 4179-4186 4188 4190-4192 4195-4201 4205 4207 4210 4212-4214 4216 4218-4222 4225 4227-4232 4234 4236-4238 4240 4242-4249 4251-4256 4259 4261 4263-4264 4266 4269 4271-4272 4275 4277-4278 4280-4285 4287 4289-4296 4298 4300 4304 4306-4310 4312 4314-4315 4317 4319-4325 4328-4329 4331-4333 4336-4339 4341-4342 4345-4349 4352-4356 4360-4362 4364 4366 4368 4370-4372 4374-4375 4377-4379 4381 4383-4388 4390-4393 4395-4396 4398-4399 4401-4407 4410-4413 4416-4418 4420 4424 4427-4430 4432 4434 4437-4439 4441-4449 4452 4455 4457 4459-4466 4468 4470-4471 4473-4476 4478-4481 4484-4490 4492-4493 4497 4499 4501 4504-4506 4508-4511 4513-4524 4526-4529 4531 4534 4536 4541-4544 4547-4548 4551-4553 4559-4560 4566 4569 4571-4572 4574-4576 4580-4581 4583-4586 4589 4591 4595 4597-4598 4601 4603-4608 4610-4611 4613-4614 4616-4617 4619-4624 4627 4629-4630 4636 4638-4640 4642-4644 4646-4647 4652 4654 4656-4657 4659-4660 4663 4665-4667 4669-4670 4672-4675 4678-4679 4682-4683 4685 4689-4693 4695 4697-4700 4703 4705 4707-4708 4712-4716 4720 4722 4724-4726 4728-4731 4733 4735 4737-4738 4740 4742-4749 4752-4753 4755 4757-4760 4762-4775 4778-4783 4786 4789 4791-4797 4799-4806 4809-4816 4819 4821-4822 4826-4828 4832-4835 4837-4840 4843-4848 4850-4853 4856-4858 4862 4869-4871 4873 4875-4881 4884-4888 4890 4892-4893 4895-4897 4904 4906-4908 4910 4912 4916-4918 4920 4923-4929 4931 4933-4935 4938-4941 4943-4945 4947 4949 4951 4953-4955 4957-4960 4962-4963 4965-4967 4969-4970 4972 4975 4978-4984 4986-4995 4997-4998 5000 5002-5007 5009-5010 5012 5014-5015 5017-5021 5023-5025 5029 5031 5036-5037 5039 5042 5044-5046 5049 5051-5052 5054-5058 5060 5062-5068 5071-5072 5075-5077 5079-5080 5084-5090 5092 5096-5098 5101-5106 5109 5111-5115 5117-5119 5121-5122 5124 5126 5128 5131 5133-5134 5136-5139 5141-5144 5147 5149-5150 5152-5153 5155 5157-5160 5163 5165-5167 5170-5175 5179 5182-5185 5187-5188 5190-5194 5196-5197 5199-5204 5207 5209-5210 5213 5215-5220 5222-5224 5226-5238 5242-5243 5246 5248 5250-5251 5253 5255 5257 5259-5260 5263-5265 5267 5269-5270 5272 5275-5279 5281-5282 5286 5288-5291 5293 5295-5299 5301-5302 5306 5308 5311 5314 5318-5320 5322-5323 5325-5327 5329 5331-5334 5336-5342 5344-5347 5349 5353-5356 5358-5360 5362-5366 5368-5371 5373-5374 5376-5378 5381 5384-5385 5389-5391 5393-5395 5397 5400 5403-5406 5408-5412 5416 5418 5420 5422 5429 5433 5436 5440-5445 5449-5450 5452 5454-5457 5459-5461 5464-5466 5468-5469 5471-5472 5474 5476-5478 5480-5482 5484 5488-5490 5492 5496 5499-5506 5508 5511-5513 5515 5519-5521 5524-5525 5527-5531 5533 5535-5540 5542 5544-5551 5553-5554 5556-5557 5560 5562-5564 5567 5570 5573 5576 5578-5579 5582-5583 5586-5588 5590-5591 5593-5595 5597-5601 5603-5605 5607 5611-5612 5614-5615 5618-5621 5623-5626 5628-5634 5636-5640 5645-5647 5650 5652 5654-5657 5660-5663 5665 5667 5672 5675 5677-5680 5682 5684-5689 5693-5696 5698-5702 5706-5707 5709-5713 5717 5720 5722-5724 5726 5729-5730 5733 5736 5738-5741 5743-5746 5748 5750-5756 5758 5760 5763-5764 5768 5771 5775 5777 5779 5783-5786 5789-5791 5793-5794 5796 5798 5800-5802 5804-5805 5809-5814 5818-5819 5823-5829 5832 5834-5836 5838-5848 5851 5856-5859 5862 5864 5866 5868-5873 5875-5876 5878 5880-5887 5889-5899 5902-5903 5905-5911 5913-5916 5919 5921-5924 5926-5928 5930-5933 5937-5941 5944-5946 5948-5953 5955-5963 5967-5971 5974 5976-5979 5981 5984-5987 5990-5991 5993 5997 6000-6001 6003 6005-6011 6015-6021 6024-6025 6027 6029 6031 6034-6038 6041-6048 6052-6055 6057-6058 6062-6064 6066 6070-6074 6081-6083 6085-6087 6089-6090 6092-6093 6095-6101 6104 6106-6107 6109 6113 6115 6119-6129 6132 6134-6136 6138-6139 6142-6146 6148-6149 6151-6152 6154-6157 6160-6161 6163-6165 6167 6169-6170 6172-6176 6178-6185 6187-6190 6192-6200 6203-6204 6208-6210 6212-6215 6218-6222 6224 6227 6230 6232-6233 6235-6238 6240-6241 6243 6247 6251-6271 6274 6278-6279 6281-6282 6284 6286-6288 6290 6293-6296 6298-6300 6303-6306 6310-6311 6313 6315-6323 6325-6326 6328 6330-6331 6334-6339 6342-6346 6348 6351 6353-6363 6365 6367-6368 6370-6376 6378-6379 6383 6385 6387-6388 6391-6392 6394-6396 6400-6402 6405 6407-6409 6412-6413 6416-6419 6422 6424 6426-6430 6433-6437 6441 6443-6446 6448 6451-6455 6457-6459 6461-6464 6468-6481 6483-6484 6486 6489 6492-6493 6495 6498-6499 6501-6502 6504-6505 6507-6509 6512-6513 6516 6518 6521 6523 6525-6527 6529 6532-6537 6539-6542 6548-6550 6554 6557-6558 6560-6562 6564 6566-6567 6571-6575 6577-6578 6580-6586 6589-6590 6594 6596 6599-6600 6602-6607 6610 6612-6613 6617 6620 6623-6624 6626 6628 6630-6637 6639-6643 6647-6654 6656-6657 6659-6661 6663 6665 6668-6671 6673 6675-6679 6682-6686 6688-6692 6694-6696 6698-6699 6701 6703 6706-6707 6709-6717 6720-6721 6723 6725 6727-6730 6733-6735 6737-6740 6742-6746 6748-6751 6754 6756 6758-6765 6767 6770-6771 6773 6776-6780 6782 6784-6785 6789 6791-6793 6795-6798 6801-6804 6808 6810-6815 6817 6819-6820 6822-6827 6837 6847 6852 6857 6882 6888 6893-6895 6906 6915 6922-6923 6927 6933 6939 6948 6958 6960-6961 6968-6970 6974 6988 6992 6998 7001-7002 7013-7014 7025-7027 7031 7035 7037 7040 7048 7051 7059 7062-7064 7066 7072 7074 7080 7085 7095 7101 7107 7109 7111 7113 7118 7120 7122-7123 7125 7130 7150 7170-7173 7176 7179 7182 7186 7188 7198 7209-7212 7214 7217 7222 7225 7236 7240 7243-7244 7246 7249 7252 7256-7258 7272 7290 7294-7295 7297 7311 7315 7323 7329 7335 7345 7349 7354 7370 7388 7391 7394 7398-7399 7402 7411 7416 7425-7426 7428 7436 7441 7445 7451 7463 7480 7494 7499 7501 7503 7505 7507 7509 7514 7522 7527 7533-7535 7539 7547 7551-7553 7557 7561 7564-7565 7585 7596-7597 7599 7605-7606 7609-7611 7630 7632 7635 7639 7643 7649 7659-7660 7662 7664 7669 7682 7685 7693 7719 7721 7723 7729 7731 7744 7747 7755-7756 7765 7769 7790 7803 7806 7808 7814-7816 7821 7825 7829 7832 7835 7843 7849 7854-7855 7865-7868 7889-7890 7895-7896 7899 7901 7904 7914 7919 7928-7929 7932 7935 7948 7950 7959 7967 7972 7974 7978 7987 7998 8005-8006 8009 8015 8019 8037 8040 8047-8048 8061-8062 8072 8075 8078-8079 8088 8090 8096 8110 8135-8136 8141-8142 8153 8159 11565 11868 13971 14416 14471 14483 15572
f147cf1e672348d4b7dd284de55d6cd4385bd19b 172.19.178.148:6379@16379 master - 0 1678688989413 514 connected 8162 8164 8168 8177-8179 8185 10923-11564 11566-11867 11869-13970 13972-14415 14417-14470 14472-14482 14484-15571 15573-16383
root@emqxbackup001:/tmp#

这个情况我也测试了redis 7 + emqx 5.0.20, 产生的tcp连接数一样非常多,我设置的redis maxclients 2万个,一加载插件就爆满.

我把不连续的slots整理后,再重新启用redis,连接数就正常了.

root@emqxbackup002:/var/lib/redis# redis-cli --cluster call 172.19.178.148:6379 dbsize

Calling dbsize
172.19.178.148:6379: 1229918
172.19.123.114:6379: 1219197
172.19.178.149:6379: 1219935
root@emqxbackup002:/var/lib/redis# redis-cli cluster nodes
72c7ea3175a375e2971b2909bd2fd4cffdda464e 172.19.123.114:6379@16379 master - 0 1678695731499 519 connected 0-5460
41f64fde5807c7dc963a4e0a09cc83c110ec5c46 172.19.178.149:6379@16379 master - 0 1678695729490 520 connected 5461-10921
f147cf1e672348d4b7dd284de55d6cd4385bd19b 172.19.178.148:6379@16379 myself,master - 0 1678695729000 517 connected 10922-16383
root@emqxbackup002:/var/lib/redis# netstat -ntap | awk ‘/:6379/{print $NF}’ | sort | uniq -c | sort -k1n
7 -
24 27679/emqxbackup002
25 26290/redis-server
root@emqxbackup002:/var/lib/redis# ps -p 27679 -fl ww
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
4 S emqx 27679 1 12 80 0 - 932544 poll_s 14:26 ? 14:08 emqxbackup002 -Bd -spp true -A 4 -IOt 4 -SDio 8 -e 262144 -zdbbl 8192 -Q 1048576 -P 2097152 – -root /usr/lib/emqx -progname /usr/bin/emqx – -home /var/lib/emqx – -noshell -noshell -noinput -boot /usr/lib/emqx/releases/5.0.20/start -boot_var RELEASE_LIB /usr/lib/emqx/lib -boot_var ERTS_LIB_DIR /usr/lib/emqx/lib -mode embedded -config /var/lib/emqx/configs/app.2023.03.13.14.26.15.config -stdlib restricted_shell emqx_restricted_shell -shutdown_time 30000 -pa /var/lib/emqx/patches -mnesia dump_log_write_threshold 5000 -mnesia dump_log_time_threshold 60000 -os_mon start_disksup false -pa /usr/lib/emqx/releases/5.0.20/consolidated -kernel net_ticktime 120 -setcookie yunduolalxqemqxsecretcookie -name emqxbackup002@172.19.178.148 -mnesia dir “/var/lib/emqx/mnesia/emqxbackup002@172.19.178.148” – -start_epmd false -epmd_module ekka_epmd -proto_dist ekka – foreground -emqx_data_dir /var/lib/emqx –
root@emqxbackup002:/var/lib/redis#