环境信息
-
EMQX 版本：k8s 环境，容器镜像 emqx/emqx，查看 emqx 版本为 4.3.11
-
操作系统及版本：宿主机 网易云 centos8
-
其他：
1.mysql版本号为 8.0.27；
2.emqx 为 statefulset，mysql 也为 statefulset
问题描述
k8s 上部署了 emqx 和 mysql8 的有状态服务。其中 mysql 服务地址为：10.233.47.222。
根据 emqx 官网提供的 mysql 认证初始化脚本，创建了数据库 mqtt，两张表 mqtt_user，mqtt_acl。
创建 emqx 需要的 ConfigMap，用来定义环境变量，内容如下：
kind: ConfigMap
apiVersion: v1
metadata:
name: emqx-config
namespace: k8s-demo
annotations:
kubesphere.io/alias-name: emqx-config
kubesphere.io/creator: admin
kubesphere.io/description: emqx-config
data:
EMQX_ACL_NOMATCH: deny
EMQX_ALLOW_ANONYMOUS: 'false'
EMQX_AUTH__MYSQL__DATABASE: mqtt
EMQX_AUTH__MYSQL__PASSWORD: emqx123
EMQX_AUTH__MYSQL__PASSWORD_HASH: plain
EMQX_AUTH__MYSQL__SERVER: '10.233.47.222:3306'
EMQX_AUTH__MYSQL__USERNAME: mqtt
EMQX_LOADED_PLUGINS: emqx_auth_mysql
EMQX_RPC__PORT_DISCOVERY: k8s
update_etc_resolv.sh: >-
echo "$(sed 's/options ndots:5/#options ndots:5/g' /etc/resolv.conf)" >
/etc/resolv.conf
在 emqx pod 上设置了 EMQX_ 开头的环境变量，用以加载 mysql 认证插件，如下所示：
启动容器前，再次检查 emqx 的环境变量，并将环境变量中配置的 mysql 服务地址、数据库、口令等，在除 emqx 之外的其他容器中连接 10.233.47.222 ，都能成功连接。
然后启动 emqx 容器，容器可以启动成功，但是后台日志显示连接 mysql 超时。后台日志如下：
登录 emqx 的 dashboard，在插件管理页面，启动 mysql 认证插件，提示报错，报错信息如下：
{emqx_auth_mysql,{bad_return,{{emqx_auth_mysql_app,start,[normal,[]]},{‘EXIT’,{{badmatch,{error,{shutdown,{failed_to_start_child,emqx_auth_mysql,{shutdown,{failed_to_start_child,worker_sup,{shutdown,{failed_to_start_child,{worker,1},timeout}}}}}}}},[{emqx_auth_mysql_app,start,2,[{file,“emqx_auth_mysql_app.erl”},{line,38}]},{application_master,start_it_old,4,[{file,“application_master.erl”},{line,277}]}]}}}}}
在 emqx 容器终端使用命令 “busybox-extras telnet 10.233.47.222 3306” 去检查 emqx 能否访问到 mysql，也是可以 telnet 通的，如下图：
配置文件及日志
emqx_mysql_auth.conf 配置文件
因为是在 pod 中配置了环境变量，这里只是列出配置文件中数据库连接部分。
auth.mysql.server = 10.233.47.222:3306
auth.mysql.pool = 8
auth.mysql.username = mqtt
auth.mysql.password = emqx123
auth.mysql.database = mqtt
auth.mysql.password_hash = plain
后台日志
EMQX_LOADED_PLUGINS=emqx_auth_mysql
acl_nomatch = "deny"
allow_anonymous = "false"
listener.ssl.external.acceptors = "32"
listener.ssl.external.max_connections = "102400"
listener.tcp.external.acceptors = "64"
listener.tcp.external.max_connections = "1024000"
listener.ws.external.acceptors = "16"
listener.ws.external.max_connections = "102400"
listener.wss.external.acceptors = "16"
listener.wss.external.max_connections = "102400"
log.to = "console"
node.max_ets_tables = "2097152"
node.max_ports = "1048576"
node.name = "emqx-0@10.233.113.73"
node.process_limit = "2097152"
rpc.port_discovery = "manual"
Starting emqx on node emqx-0@10.233.113.73
Start mqtt:tcp:internal listener on 127.0.0.1:11883 successfully.
Start mqtt:tcp:external listener on 0.0.0.0:1883 successfully.
Start mqtt:ws:external listener on 0.0.0.0:8083 successfully.
Start mqtt:ssl:external listener on 0.0.0.0:8883 successfully.
Start mqtt:wss:external listener on 0.0.0.0:8084 successfully.
Start http:management listener on 8081 successfully.
Start http:dashboard listener on 18083 successfully.
2022-03-22T10:17:29.407988+08:00 [error] crasher: initial call: mysql_conn:init/1, pid: <0.2027.0>, registered_name: [], exit: {timeout,[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,401}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.2026.0>,<0.2025.0>,<0.2023.0>,emqx_auth_mysql_sup,<0.2021.0>], message_queue_len: 0, messages: [], links: [<0.2026.0>], dictionary: [], trap_exit: false, status: running, heap_size: 376, stack_size: 28, reductions: 382; neighbours:
2022-03-22T10:17:29.408553+08:00 [error] [MySQL] Can't connect to MySQL server: timeout
2022-03-22T10:17:29.408798+08:00 [error] Supervisor: {<0.2025.0>,ecpool_worker_sup}. Context: start_error. Reason: timeout. Offender: id={worker,1},pid=undefined.
2022-03-22T10:17:29.409004+08:00 [error] crasher: initial call: ecpool_worker:init/1, pid: <0.2026.0>, registered_name: [], exit: {timeout,[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,401}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.2025.0>,<0.2023.0>,emqx_auth_mysql_sup,<0.2021.0>], message_queue_len: 1, messages: [{'EXIT',<0.2027.0>,timeout}], links: [<0.2025.0>], dictionary: [], trap_exit: true, status: running, heap_size: 1598, stack_size: 28, reductions: 4381; neighbours:
2022-03-22T10:17:29.409484+08:00 [error] Supervisor: {<0.2023.0>,ecpool_pool_sup}. Context: start_error. Reason: {shutdown,{failed_to_start_child,{worker,1},timeout}}. Offender: id=worker_sup,pid=undefined.
2022-03-22T10:17:29.410735+08:00 [error] Supervisor: {local,emqx_auth_mysql_sup}. Context: start_error. Reason: {shutdown,{failed_to_start_child,worker_sup,{shutdown,{failed_to_start_child,{worker,1},timeout}}}}. Offender: id=emqx_auth_mysql,pid=undefined.
2022-03-22T10:17:29.411123+08:00 [error] crasher: initial call: application_master:init/4, pid: <0.2020.0>, registered_name: [], exit: {{bad_return,{{emqx_auth_mysql_app,start,[normal,[]]},{'EXIT',{{badmatch,{error,{shutdown,{failed_to_start_child,emqx_auth_mysql,{shutdown,{failed_to_start_child,worker_sup,{shutdown,{failed_to_start_child,{worker,1},timeout}}}}}}}},[{emqx_auth_mysql_app,start,2,[{file,"emqx_auth_mysql_app.erl"},{line,38}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,277}]}]}}}},[{application_master,init,4,[{file,"application_master.erl"},{line,138}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.2019.0>], message_queue_len: 1, messages: [{'EXIT',<0.2021.0>,normal}], links: [<0.2019.0>,<0.1435.0>], dictionary: [], trap_exit: true, status: running, heap_size: 610, stack_size: 28, reductions: 235; neighbours:
2022-03-22T10:17:29.417045+08:00 [error] [Plugins] Load plugin emqx_auth_mysql failed, cannot start plugin emqx_auth_mysql for {bad_return,{{emqx_auth_mysql_app,start,[normal,[]]},{'EXIT',{{badmatch,{error,{shutdown,{failed_to_start_child,emqx_auth_mysql,{shutdown,{failed_to_start_child,worker_sup,{shutdown,{failed_to_start_child,{worker,1},timeout}}}}}}}},[{emqx_auth_mysql_app,start,2,[{file,"emqx_auth_mysql_app.erl"},{line,38}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,277}]}]}}}}
EMQ X Broker 4.3.11 is running now!
2022-03-22T10:18:24.139539+08:00 [warning] [Alarm Handler] Alarm high_system_memory_usage is activated, System memory usage is higher than 70%
