出现 large_heap warning日志

EMQX
,
1

5.0.23集群运行在k8s容器中，连接500w连接，运行一段时间后出现以下warning，每隔一段时间出现一次[warning] msg: large_heap, mfa: emqx_sys_mon:handle_info/2, line: 164, info: [{old_heap_block_size,31936144},{heap_block_size,4298223},{mbuf_size,0},{stack_size,97},{old_heap_size,18414130},{heap_size,73}], procinfo: [{pid,<0.7931.4221>},{memory,289875912},{total_heap_size,36234367},{heap_size,4298223},{stack_size,102},{min_heap_size,233},{proc_lib_initial_call,{cowboy_stream_h,request_process,[‘Argument__1’,‘Argument__2’,‘Argument__3’]}},{initial_call,{proc_lib,init_p,5}},{current_stacktrace,[{erlang,port_control,3,},{prim_inet,ctl_cmd,3,},{prim_inet,peername,1,},{prometheus_vm_dist_collector,‘-dist_info/5-lc$^0/1-0-’,2,[{file,“prometheus_vm_dist_collector.erl”},{line,412}]},{prometheus_vm_dist_collector,‘-dist_info/5-lc$^0/1-0-’,2,[{file,“prometheus_vm_dist_collector.erl”},{line,409}]},{prometheus_vm_dist_collector,dist_info,5,[{file,“prometheus_vm_dist_collector.erl”},{line,409}]},{prometheus_vm_dist_collector,‘-dist_info/0-lc$^1/1-1-’,2,[{file,“prometheus_vm_dist_collector.erl”},{line,391}]},{prometheus_vm_dist_collector,‘-dist_info/0-lc$^1/1-1-’,2,[{file,“prometheus_vm_dist_collector.erl”},{line,391}]},{prometheus_vm_dist_collector,metrics1,0,[{file,“prometheus_vm_dist_collector.erl”},{line,292}]},{prometheus_vm_dist_collector,metrics,0,[{file,“prometheus_vm_dist_collector.erl”},{line,286}]},{prometheus_vm_dist_collector,collect_mf,2,[{file,“prometheus_vm_dist_collector.erl”},{line,271}]},{prometheus_collector,collect_mf,3,[{file,“prometheus_collector.erl”},{line,156}]},{prometheus_registry,‘-collect/2-lc$^0/1-0-’,3,[{file,“prometheus_registry.erl”},{line,86}]},{prometheus_registry,collect,2,[{file,“prometheus_registry.erl”},{line,87}]},{prometheus_text_format,format,1,[{file,“prometheus_text_format.erl”},{line,71}]},{emqx_prometheus_api,stats,2,[{file,“emqx_prometheus_api.erl”},{line,101}]},{minirest_handler,apply_callback,3,[{file,“minirest_handler.erl”},{line,111}]},{minirest_handler,handle,2,[{file,“minirest_handler.erl”},{line,44}]},{
minirest_handler,init,2,[{file,“minirest_handler.erl”},{line,27}]},{cowboy_handler,execute,2,[{file,“cowboy_handler.erl”},{line,41}]},{cowboy_stream_h,execute,3,[{file,“cowboy_stream_h.erl”},{line,318}]},{cowboy_stream_h,request_process,3,[{file,“cowboy_stream_h.erl”},{line,302}]}]},{registered_name,},{status,running},{message_queue_len,0},{group_leader,<0.1975.0>},{priority,normal},{trap_exit,false},{reductions,40592829},{last_calls,false},{catchlevel,6},{trace,0},{suspending,},{sequential_trace_token,},{error_handler,error_handler}]

2

连接数很大的情况下，prometheus pull 数据的接口在处理时，进程会申请到很多内存。
参考配置手册中的 sysmon.vm.large_heap 配置项： 告警阈值

sysmon.vm.large_heap

类型: disabled | bytesize

默认值: 32MB

启用后，当一个 Erlang 进程申请了大量内存，系统会触发一条带有 large_heap 关键字的 warning 级别日志。同时还会发布一条主题为 $SYS/sysmon/busy_dist_port 的 MQTT 系统消息。

3

所以这不是问题咯，正常就是要申请这么大的内存？

4

5.0.23上申请大内存，也解决不了这个告警。你得从配置里关掉：

prometheus {
mnesia_collector = disabled
vm_dist_collector = disabled
vm_memory_collector = disabled
vm_msacc_collector = disabled
vm_statistics_collector = disabled
vm_system_info_collector = disabled
}