emqx集群prometheus接口响应慢

zzdld · 2023 年9 月 20 日 02:30

emqx5.0.23集群部署在k8s环境，启动3core+18replica,整个集群添加300万连接，每节点连接均匀分布，部分节点/api/v5/prometheus/stats接口响应慢，超10s

blankalupo · 2023 年9 月 20 日 03:04

High CPU Usage after upgrade from emqx 4.3.8 to emqx 5.0.19

opened 01:08PM - 14 Sep 23 UTC

ctellechea2001

perf #triage/wait

### What happened? We updated in the lower environment (DEV,SBX,UAT) the vers…ion from 4.3.8 to 5.0.19 a two month ago. Everything was fine in the lower enviroment. Today we did the same change in PRD (Logically the traffic and connections in PRD is higher than the lower enviroments) and we discovered that the CPU usage on each node have been increased. ![image](https://github.com/emqx/emqx/assets/45426180/bea34195-b8d1-41d2-abd0-b28e8d8f87e2) After the migration to 5.0.19 with the same cpu resources assigned as the 4.3.8 we started with the following errors: `line: 416, message: <<"VM is overloaded on node: line: 422, name: runq_overload ` We needed to change the current Kubernetes EMQX Cluster Nodes resources limits. We removed the 1000m cpu limits and we increased the cpu request from 500m to 1. After that we stop receiving some random emqx connections issues. ![image](https://github.com/emqx/emqx/assets/45426180/9a72190f-350c-4565-a405-0f67a27f5482) Do you know if that is normal that High CPU Increase in the emqx 5.0.19 or is necessary to change something? We have 920 connections in PRD in a cluster of tree nodes ### What did you expect to happen? We need to discover the High cpu usage on emqx 5.0.19 ### How can we reproduce it (as minimally and precisely as possible)? emqx 5.0.19 ### Anything else we need to know? _No response_ ### EMQX version <details> ```console $ ./bin/emqx_ctl broker # paste output here ``` </details> ### OS version <details> ```console # On Linux: $ cat /etc/os-release # paste output here $ uname -a # paste output here # On Windows: C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture # paste output here ``` </details> ### Log files <details> </details>

这是个已知问题，可以参考这个进行修改

zzdld · 2023 年9 月 20 日 07:02

您好我的问题是promethus接口响应慢不是cpu消耗高啊？

zzdld · 2023 年9 月 20 日 08:40

我是要加监控啊，怎么能直接关掉呢

blankalupo · 2023 年9 月 20 日 08:45

这几个是非业务相关的指标，是 Erlang VM 的内部指标，采集没有多大的意义，反而会增加耗时，把这几个去掉，应该就能解决你这里的问题了

zzdld · 2023 年9 月 20 日 08:56

请问用operator2.1方式部署是通过edit方式将这些加到 bootstrapConfig中吗，会引起集群重启吗？

zzdld · 2023 年9 月 21 日 01:40

帮忙解答一下谢谢

blankalupo · 2023 年9 月 21 日 01:58

operator 的问题，建议可以先暂时去 operator 的 github 上提 issue

zzdld · 2023 年9 月 21 日 07:46

不是问题，我在咨询按照上面的配置修改后会不会造成集群重启

zzdld · 2023 年10 月 26 日 12:30

我用这个方法试了不行，可能是因为我用的使用 Prometheus+Grafana 监控 EMQX 集群 | EMQX Operator2.1.1 文档
这个方法建立的监控，所以直接调PUT /api/v5/prometheus 接口进行修改不能成功？