k8s operator 模式部署emqx如何配置ssl

pasty · 2022 年7 月 20 日 03:49

环境信息

EMQX 版本：4.4
操作系统及版本：operator版本1.2
其他

问题描述

emqx在k8s部署怎么开启ssl，官方文档我没有找到示例

配置文件及日志

pasty · 2022 年7 月 20 日 03:54

apiVersion: apps.emqx.io/v1beta2
kind: EmqxBroker
metadata:
  name: emqx
spec:
  serviceAccountName: "emqx"
  image: emqx/emqx:4.4.3
  replicas: 3
  labels:
    cluster: emqx
  storage:
    storageClassName: local
    resources:
      requests:
        storage: 20Gi
    accessModes:
    - ReadWriteOnce
  emqxTemplate:
    listener:
      type: NodePort
      ports:
        mqtt: 1883
        mqtts: 8883
        ws: 8083
        wss: 8084
        dashboard: 18083
        api: 8081
    acl:
      - permission: allow
        username: "dashboard"
        action: subscribe
        topics:
          filter:
            - "$SYS/#"
            - "#"
      - permission: allow
        ipaddress: "127.0.0.1"
        topics:
          filter:
            - "$SYS/#"
          equal:
            - "#"
      - permission: deny
        action: subscribe
        topics:
          filter:
            - "$SYS/#"
          equal:
            - "#"
      - permission: allow
    plugins:
      - name: emqx_management
        enable: true
      - name: emqx_recon
        enable: true
      - name: emqx_retainer
        enable: true
      - name: emqx_dashboard
        enable: true
      - name: emqx_telemetry
        enable: true
      - name: emqx_rule_engine
        enable: true
      - name: emqx_bridge_mqtt
        enable: false
    modules:
      - name: emqx_mod_acl_internal
        enable: true
      - name: emqx_mod_presence
        enable: true

这是我的yaml

Rory · 2022 年7 月 20 日 09:40

建议使用 apps.emqx.io/v1beta3，apps.emqx.io/v1beta3 提供了 .spec.emqxTemplate.volume 和 .spec.emqxTemplate.volumeMount 字段，可以将你的 ssl 证书作为 secret 挂载进去

pasty · 2022 年7 月 20 日 10:52

apiVersion: apps.emqx.io/v1beta3
kind: EmqxBroker
metadata:
  name: emqx3
  labels:
    "foo1": "bar1"
spec:
  emqxTemplate:
    image: emqx/emqx:4.4.5
    volume:
      - name: volume-ays4bb
        secret:
          secretName: mqtt-serverkey
          items:
            - key: mqtt-server.crt
              path: /etc/certs/
            - key: mqtt_server.key
              path: /etc/certs/
            - key: server_ca.crt
              path: /etc/certs/
    volumeMount:
      - name: volume-ays4bb
        readOnly: true
        mountPath: /etc/certs/

error: error validating “emqx-v3.yaml”: error validating data: [ValidationError(EmqxBroker.spec.emqxTemplate): unknown field “volume” in io.emqx.apps.v1beta3.EmqxBroker.spec.emqxTemplate, ValidationError(EmqxBroker.spec.emqxTemplate): unknown field “volumeMount” in io.emqx.apps.v1beta3.EmqxBroker.spec.emqxTemplate]; if you choose to ignore these errors, turn validation off with --validate=false

是我写法不正确吗

Rory · 2022 年7 月 20 日 13:13

volume => extraVolumes

volumeMount => extraVolumeMounts

不好意思我上个回复写错了

pasty · 2022 年7 月 22 日 01:54

apiVersion: apps.emqx.io/v1beta3
kind: EmqxBroker
metadata:
  name: emqx3
  labels:
    "foo1": "bar1"
spec:
  emqxTemplate:
    image: emqx/emqx:4.4.5
    extraVolumes:
      - name: mqtt-serverkey
        secret:
          secretName: mqtt-serverkey
          items:
            - key: mqtt-server.crt
              path: mqtt-server.crt
            - key: mqtt_server.key
              path: mqtt_server.key
            - key: server_ca.crt
              path: server_ca.crt
      - name: emqx
        configMap:
          name: emqx
          defaultMode: 420
    extraVolumeMounts:
      - name: mqtt-serverkey
        readOnly: true
        mountPath: /opt/emqx/etc/certs_ssl/
      - name: emqx
        readOnly: true
        mountPath: /opt/emqx/etc/emqx.conf
        subPath: emqx.conf

把证书挂载进去后，配置文件通过修改证书目录同样挂载到容器内，进容器内发现配置文件是修改正常了的，但是我通过mqtt客户端连接emq开启ssl连接失败，下面是我截取的报错日志

2022-07-22T01:47:03.505547+00:00 [error] 10.233.106.0:43482 [MQTT] , Parse failed for malformed_utf8_string, [{emqx_frame,parse_utf8_string,2,[{file,"emqx_frame.erl"},{line,513}]},{emqx_frame,parse_packet,3,[{file,"emqx_frame.erl"},{line,226}]},{emqx_frame,parse_frame,4,[{file,"emqx_frame.erl"},{line,201}]},{emqx_connection,parse_incoming,3,[{file,"emqx_connection.erl"},{line,655}]},{emqx_connection,handle_msg,2,[{file,"emqx_connection.erl"},{line,648}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,394}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,358}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}], Frame data:<<22,3,3,0,193,1,0,0,189,3,3,98,218,1,152,51,59,25,175,204,32,107,11,205,115,252,193,211,159,53,99,130,75,134,2,63,211,237,137,234,218,70,64,0,0,86,192,36,192,40,0,61,192,38,192,42,0,107,0,106,192,10,192,20,0,53,192,5,192,15,0,57,0,56,192,35,192,39,0,60,192,37,192,41,0,103,0,64,192,9,192,19,0,47,192,4,192,14,0,...>>

 2022-07-22T01:47:03.700271+00:00 [error] 10.233.106.0:50729 [MQTT] , Parse failed for malformed_utf8_string, [{emqx_frame,parse_utf8_string,2,[{file,"emqx_frame.erl"},{line,513}]},{emqx_frame,parse_packet,3,[{file,"emqx_frame.erl"},{line,226}]},{emqx_frame,parse_frame,4,[{file,"emqx_frame.erl"},{line,201}]},{emqx_connection,parse_incoming,3,[{file,"emqx_connection.erl"},{line,655}]},{emqx_connection,handle_msg,2,[{file,"emqx_connection.erl"},{line,648}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,394}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,358}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}], Frame data:<<22,3,3,0,193,1,0,0,189,3,3,98,218,1,152,21,200,26,251,247,77,107,80,164,127,194,75,141,196,36,249,164,120,84,17,10,191,189,78,239,65,42,5,0,0,86,192,36,192,40,0,61,192,38,192,42,0,107,0,106,192,10,192,20,0,53,192,5,192,15,0,57,0,56,192,35,192,39,0,60,192,37,192,41,0,103,0,64,192,9,192,19,0,47,192,4,192,14,0,...>>

 2022-07-22T01:47:08.820116+00:00 [error] [Plugins] Write File "/mounted/plugins/data/loaded_plugins" Error: erofs
 2022-07-22T01:49:42.564479+00:00 [error] supervisor: 'esockd_connection_sup - <0.2534.0>', errorContext: connection_shutdown, reason: {ssl_error,{options,{certfile,"etc/certs_ssl/mqtt_server.crt",{error,enoent}}}}, offender: [{pid,<0.29268.0>},{name,connection},{mfargs,{emqx_connection,start_link,[[{deflate_options,[]},{max_conn_rate,500},{active_n,100},{zone,external},{proxy_address_header,<<>>},{proxy_port_header,<<>>},{supported_subprotocols,[]}]]}}]

 2022-07-22T01:49:43.212073+00:00 [error] supervisor: 'esockd_connection_sup - <0.2534.0>', errorContext: connection_shutdown, reason: {ssl_error,{options,{certfile,"etc/certs_ssl/mqtt_server.crt",{error,enoent}}}}, offender: [{pid,<0.29272.0>},{name,connection},{mfargs,{emqx_connection,start_link,[[{deflate_options,[]},{max_conn_rate,500},{active_n,100},{zone,external},{proxy_address_header,<<>>},{proxy_port_header,<<>>},{supported_subprotocols,[]}]]}}]

DDDHuang · 2022 年7 月 26 日 02:31

看下这个文件的权限是否正常，日志报错是文件读取异常了，看下权限