emqx5.x版本配置的失败和疑问

环境信息

  • EMQX 版本:5.0.6
  • 操作系统及版本:linux
  • 其他:采用docker容器化部署的方式

问题描述

我把emqx的配置文件暴露出来之后,启动的过程中就报错了,报错信息和日志信息在下面展示,这个是其中的一个问题,另外我还有一些疑问。

1、原本我在4.x版本的时候,用的是文件配置文件的方式来配置我的权限校验和一些retainer的参数,但在我4.x版本迁移到5.x版本的过程中,我原本的两个配置文件失效了,如下:


下面是我的docker编译文件的内容:

我应该是把这些配置文件打入到镜像中了,但升级镜像之后,这些配置就失效了,这是第一个疑问。

2、第二个我看4.x版本升级5.x版本的指南中,一些原本需要的配置文件方式变成了参数配置,那这个是不是我上面配置文件失效的原因?于是我就把我原本配置文件的方式改为了调整emqx.conf的方式来配置参数,我的做法和理解是否正确?但如果正确的话,我的配置写入进去之后,并没有起来服务。

希望得到一些帮助和支持,以及对文档的疑问,谢谢!

配置文件及日志

日志如下:

[error]: {concat_error,<<"failed_to_concat [{<<\"listeners\">>,\n                   [{#{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 72},\n                       '$hcTyp' => key,'$hcVal' => <<\"http\">>},\n                     #{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 73},\n                       '$hcTyp' => object,\n                       '$hcVal' =>\n                           [{#{'$hcMeta' =>\n                                   #{filename => \"/opt/emqx/etc/emqx.conf\",\n                                     line => 73},\n                               '$hcTyp' => key,'$hcVal' => <<\"bind\">>},\n                             #{'$hcMeta' =>\n                                   #{filename => \"/opt/emqx/etc/emqx.conf\",\n                                     line => 73},\n                               '$hcTyp' => integer,'$hcVal' => 18083}}]}}]},\n                  {<<\"default_username\">>,<<\"admin\">>},\n                  {<<\"default_password\">>,<<\"public\">>},\n                  <<\"authentication\">>,\n                  [[{#{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 82},\n                       '$hcTyp' => key,'$hcVal' => <<\"mechanism\">>},\n                     #{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 82},\n                       '$hcTyp' => string,'$hcVal' => <<\"password_based\">>}},\n                    {#{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 83},\n                       '$hcTyp' => key,'$hcVal' => <<\"backend\">>},\n                     #{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 83},\n                       '$hcTyp' => string,'$hcVal' => <<\"mysql\">>}},\n                    {#{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 84},\n
'$hcTyp' => key,'$hcVal' => <<\"enable\">>},\n                     #{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 84},\n                       '$hcTyp' => bool,'$hcVal' => true}},\n                    {#{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 86},\n                       '$hcTyp' => key,\n                       '$hcVal' => <<\"password_hash_algorithm\">>},\n                     #{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 86},\n                       '$hcTyp' => object,\n                       '$hcVal' =>\n                           [{#{'$hcMeta' =>\n                                   #{filename => \"/opt/emqx/etc/emqx.conf\",\n                                     line => 87},\n                               '$hcTyp' => key,'$hcVal' => <<\"name\">>},\n                             #{'$hcMeta' =>\n                                   #{filename => \"/opt/emqx/etc/emqx.conf\",\n                                     line => 87},\n                               '$hcTyp' => string,'$hcVal' => <<\"sha256\">>}},\n                            {#{'$hcMeta' =>\n                                   #{filename => \"/opt/emqx/etc/emqx.conf\",\n                                     line => 88},\n                               '$hcTyp' => key,\n                               '$hcVal' => <<\"salt_position\">>},\n                             #{'$hcMeta' =>\n                                   #{filename => \"/opt/emqx/etc/emqx.conf\",\n                                     line => 88},\n                               '$hcTyp' => string,\n                               '$hcVal' => <<\"suffix\">>}}]}},\n                    {#{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 91},\n                       '$hcTyp' => key,'$hcVal' => <<\"database\">>},\n                     #{'$hcMeta' =>\n                           #{filename => \"/opt/em
qx/etc/emqx.conf\",line => 91},\n                       '$hcTyp' => string,'$hcVal' => <<\"emqx\">>}},\n                    {#{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 92},\n                       '$hcTyp' => key,'$hcVal' => <<\"username\">>},\n                     #{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 92},\n                       '$hcTyp' => string,'$hcVal' => <<\"sucheon_ds\">>}},\n                    {#{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 93},\n                       '$hcTyp' => key,'$hcVal' => <<\"password\">>},\n                     #{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 93},\n                       '$hcTyp' => string,'$hcVal' => <<\"sucheonDSAdmin!\">>}},\n                    {#{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 94},\n                       '$hcTyp' => key,'$hcVal' => <<\"server\">>},\n                     #{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 94},\n                       '$hcTyp' => string,\n                       '$hcVal' =>\n                           <<\"rm-wz9gx93q8k763av6v9o.mysql.rds.aliyuncs.com:3306\">>}},\n                    {#{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 95},\n                       '$hcTyp' => key,'$hcVal' => <<\"query\">>},\n                     #{'$hcMeta' =>\n                           #{filename => \"/opt/emqx/etc/emqx.conf\",line => 95},\n                       '$hcTyp' => string,\n                       '$hcVal' =>\n                           <<\"SELECT password_hash, salt, is_superuser FROM users where username = ${username} LIMIT 1\">>}}]]] at_line 71 in_file /opt/emqx/etc/emqx.conf">>}
ERROR: call_hocon_failed: -s emqx_conf_schema -c /opt/emqx/etc/emqx.conf multi_get node.data_dir node.name node.cookie node.db_backend cluster.proto_dist

配置文件如下:

## NOTE:
## Configs in this file might be overridden by:
## 1. Environment variables which start with 'EMQX_' prefix
## 2. File $EMQX_NODE__DATA_DIR/configs/cluster-override.conf
## 3. File $EMQX_NODE__DATA_DIR/configs/local-override.conf
##
## The *-override.conf files are overwritten at runtime when changes
## are made from EMQX dashboard UI, management HTTP API, or CLI.
## All configuration details can be found in emqx.conf.example

node {
  name = "emqx@127.0.0.1"
  cookie = emqxsecretcookie
  data_dir = "data"
  etc_dir = "etc"
}

log {
  file_handlers.default {
    level = warning
    file = "log/emqx.log"
  }
}

cluster {
  name = emqxcl
  discovery_strategy = manual
}


listeners.tcp.default {
  bind = "0.0.0.0:1883"
  max_connections = 1024000
}

listeners.ssl.default {
  bind = "0.0.0.0:8883"
  max_connections = 512000
  ssl_options {
    keyfile = "etc/certs/key.pem"
    certfile = "etc/certs/cert.pem"
    cacertfile = "etc/certs/cacert.pem"
  }
}

listeners.ws.default {
  bind = "0.0.0.0:8083"
  max_connections = 1024000
  websocket.mqtt_path = "/mqtt"
}

listeners.wss.default {
  bind = "0.0.0.0:8084"
  max_connections = 512000
  websocket.mqtt_path = "/mqtt"
  ssl_options {
    keyfile = "etc/certs/key.pem"
    certfile = "etc/certs/cert.pem"
    cacertfile = "etc/certs/cacert.pem"
  }
}

# listeners.quic.default {
#  enabled = true
#  bind = "0.0.0.0:14567"
#  max_connections = 1024000
#  keyfile = "etc/certs/key.pem"
#  certfile = "etc/certs/cert.pem"
#}

dashboard {
    listeners.http {
        bind = 18083
    }
    default_username = "admin"
    default_password = "public"
}

# 配置认证方式为mysql
authentication [
  {
  mechanism = password_based
  backend = mysql
  enable = true

  password_hash_algorithm {
    name = sha256
    salt_position = suffix
  }

  database = emqx
  username = "xxxxx"
  password = "xxxxx"
  server = "rm-xxxxxx.mysql.rds.aliyuncs.com:3306"
  query = "SELECT password_hash, salt, is_superuser FROM users where username = ${username} LIMIT 1"
}
]

retainer.backend  {
## Value: ram | disc | disc_only
##  - ram: memory only
##  - disc: both memory and disc
##  - disc_only: disc only
storage_type = disc_only
# 消息保留的数量上限
max_retained_messages = 1000
}

retainer = {
# 设置消息失效时间为2分钟
msg_expiry_interval = "2m"
# 设置消息清楚时间为5分钟
msg_clear_interval = "5m"

}


include emqx_enterprise.conf

5.0 采用了全新的配置系统,不兼容 4.x 的配置,建议你先看看 5.0 的文档

是的,已经看了,然后我使用的新的配置启动不起来,报错信息如上所示,这个该如何排查

你的配置里面 authentication 和 retainer 都配置错了,部分格式和字段在 5.0 里面都是不能解析的。
如果确实需要修改 emqx.conf ,建议用 5.0 安装好后 etc 目录下的 emqx.conf 做为模板。
目前建议尽量通过 API 或者 dashboard 进行配置的修改,所有配置项都可以通过 API 查看 (swagger 格式,每个字段都有解释说明)

感谢,另外咨询个问题,如果我在面板中进行修改的话,如果保证docker的持久化,需要把那些存储卷挂载出来?

如果 container 一直存在,也可以不用把数据目录 (emqx 安装目录里面的 data 子目录) 挂载出来。
然后使用 docker 的话,一定要保证每次启动时的 ip 都是不变的,建议 node name 最好使用域名,如果确实需要使用 ip 做为 node name,就一定要保证每次启动时的 ip 都是一样的,否则可能会出现数据丢失

明白了,感谢指导!

请问配置文件中如何使用变量?

请问可以只是单独把配置文件emqx.conf挂载出来吗? 我老是失败

什么叫单独把配置文件emqx.conf挂载,如果是容器里面,直接用环境来指定就好了,etc 目录保持只读

就是将容器内配置文件挂载到外部 想通过外部修改配置文件来改变容器内的配置文件

不能这样干,emqx 5.0 强制了不让改etc目录。
你可以用环境变量来改配置
或者不用容器,直接用rpm/deb包装。

好的 麻烦您了 感谢