k8s部署集群一段时间后未进行操作但是出现emqx-listeners升级失败警告

pwc · 2024 年1 月 8 日 02:08

https://docs.emqx.com/zh/emqx-operator/latest/deployment/on-alibaba-cloud.html我基于该文档部署了一个3节点集群，使用一段时间后发出警告，警告如下图

，但是本人未进行过操作，请问这是什么原因？是否使用该方式部署集群会自动升级emqx-listeners？

zhongwencool · 2024 年1 月 8 日 02:27

github.com/emqx/emqx-operator

Permanent reconcile of emqx-listeners loadbalancer

opened 09:23AM - 21 Dec 23 UTC

thyrasel

bug

**Describe the bug** I have setup EMQX in my local K3s dev cluster and also in …AWS EKS. In AWS EKS it is running fine but in K3s the emqx-listeners loadbalancer service, daemonset and emqx-listener pods are frequently recreated. This results in disconnects of the MQTT clients. If I set the type to ClusterIP it works fine but I need an external IP. I also tried to use metallb instead of k3s standardlb but with the same result. I have also seen that sometimes the service config object slightly changes when it gets recreated as the order of the ports is changed. In the logs I can see the following error message "spec.loadBalancerClass: Invalid value: \"null\"". emqx-operator logs ``` emqx-operator-controller-manager-55bd6f9676-6tcrp {"level":"error","ts":"2023-12-20T15:21:51Z","msg":"Reconciler error","controller":"emqx","controllerGroup":"apps.emqx.io","controllerKind":"EMQX","eMQX":{"name":"emqx","namespace":"phoenix-test"}," ││ namespace":"phoenix-test","name":"emqx","reconcileID":"a344a71c-c724-4a95-9212-8488617c5b28","error":"failed to create or update services: failed to update Service emqx-listeners: Service \"emqx-listeners\" is invalid: spec.loadBalancerClass: Inval ││ id value: \"null\": may not change once set","errorVerbose":"Service \"emqx-listeners\" is invalid: spec.loadBalancerClass: Invalid value: \"null\": may not change once set\nfailed to update Service emqx-listeners\ngithub.com/emqx/emqx-operator/int ││ ernal/handler.(*Handler).Update\n\t/workspace/internal/handler/handler.go:146\ngithub.com/emqx/emqx-operator/internal/handler.(*Handler).CreateOrUpdate\n\t/workspace/internal/handler/handler.go:125\ngithub.com/emqx/emqx-operator/internal/handler.(* ││ Handler).CreateOrUpdateList\n\t/workspace/internal/handler/handler.go:60\ngithub.com/emqx/emqx-operator/controllers/apps/v2beta1.(*addSvc).reconcile\n\t/workspace/controllers/apps/v2beta1/add_svc.go:34\ngithub.com/emqx/emqx-operator/controllers/app ││ s/v2beta1.(*EMQXReconciler).Reconcile\n\t/workspace/controllers/apps/v2beta1/emqx_controller.go:133\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/c ││ ontroller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/i ││ nternal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/ ││ sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\nfailed to create or update services\ngithub.com/emqx/emqx-operator/controllers/apps/v2beta1.(*addSvc).re ││ concile\n\t/workspace/controllers/apps/v2beta1/add_svc.go:35\ngithub.com/emqx/emqx-operator/controllers/apps/v2beta1.(*EMQXReconciler).Reconcile\n\t/workspace/controllers/apps/v2beta1/emqx_controller.go:133\nsigs.k8s.io/controller-runtime/pkg/inter ││ nal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io ││ /controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller ││ .go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:159 ││ 8","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/c ││ ontroller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"} ``` emqx-listeners service ``` Name: emqx-listeners Namespace: phoenix-test Labels: apps.emqx.io/instance=emqx apps.emqx.io/managed-by=emqx-operator Annotations: apps.emqx.io/last-applied: UEsDBBQACAAIAAAAAAAAAAAAAAAAAAAAAAAIAAAAb3JpZ2luYWy8VVtz8jYQ/S/7bFMbDBi99TrTmV5oSzOdxH1Yy+tUg5BUSSZJGf/3jmwDhtCkM+2XN1u7Onv2clYHQCPuyDqhFT... Selector: apps.emqx.io/db-role=core,apps.emqx.io/instance=emqx,apps.emqx.io/managed-by=emqx-operator Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 10.43.105.215 IPs: 10.43.105.215 LoadBalancer Ingress: 172.17.85.245 Port: tcp-default 1883/TCP TargetPort: 1883/TCP NodePort: tcp-default 31203/TCP Endpoints: 10.42.0.139:1883,10.42.0.140:1883 Port: ws-default 8083/TCP TargetPort: 8083/TCP NodePort: ws-default 30595/TCP Endpoints: 10.42.0.139:8083,10.42.0.140:8083 Port: wss-default 8084/TCP TargetPort: 8084/TCP NodePort: wss-default 31907/TCP Endpoints: 10.42.0.139:8084,10.42.0.140:8084 Port: ssl-default 8883/TCP TargetPort: 8883/TCP NodePort: ssl-default 32631/TCP Endpoints: 10.42.0.139:8883,10.42.0.140:8883 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning UnAvailableLoadBalancer 26m (x5 over 32m) There are no available nodes for LoadBalancer Normal UpdatedLoadBalancer 26m (x6 over 33m) Updated LoadBalancer with new IPs: [] -> [172.17.85.245] Normal EnsuringLoadBalancer 3m2s (x61 over 33m) service-controller Ensuring load balancer Normal AppliedDaemonSet 3m2s (x61 over 33m) Applied LoadBalancer DaemonSet kube-system/svclb-emqx-listeners-c1057f0a ``` emqx-listeners daemonset ``` Name: svclb-emqx-listeners-c1057f0a Selector: app=svclb-emqx-listeners-c1057f0a Node-Selector: <none> Labels: objectset.rio.cattle.io/hash=8c4a41d9478c436ff8b5f2af3a960b16b4f6d74b svccontroller.k3s.cattle.io/nodeselector=false svccontroller.k3s.cattle.io/svcname=emqx-listeners svccontroller.k3s.cattle.io/svcnamespace=phoenix-test Annotations: deprecated.daemonset.template.generation: 37 objectset.rio.cattle.io/applied: H4sIAAAAAAAA/8xW3W7jNhN9lQ9zLSnSRkm0Ar6LIAmKoF3HsL29WRjBiBrFrCmSJUfaGIHfvaDyUyV11kGBZgNfGCRnzpDnzEDnDtDK38l5aTSUgNb6gz6DCNZS11DCOVJr9JwYIm... objectset.rio.cattle.io/id: objectset.rio.cattle.io/owner-gvk: /v1, Kind=Service objectset.rio.cattle.io/owner-name: emqx-listeners objectset.rio.cattle.io/owner-namespace: phoenix-test Desired Number of Nodes Scheduled: 1 Current Number of Nodes Scheduled: 1 Number of Nodes Scheduled with Up-to-date Pods: 1 Number of Nodes Scheduled with Available Pods: 1 Number of Nodes Misscheduled: 0 Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed Pod Template: Labels: app=svclb-emqx-listeners-c1057f0a svccontroller.k3s.cattle.io/svcname=emqx-listeners svccontroller.k3s.cattle.io/svcnamespace=phoenix-test Service Account: svclb Containers: lb-tcp-8083: Image: rancher/klipper-lb:v0.4.4 Port: 8083/TCP Host Port: 8083/TCP Environment: SRC_PORT: 8083 SRC_RANGES: 0.0.0.0/0 DEST_PROTO: TCP DEST_PORT: 8083 DEST_IPS: 10.43.105.215 Mounts: <none> lb-tcp-8084: Image: rancher/klipper-lb:v0.4.4 Port: 8084/TCP Host Port: 8084/TCP Environment: SRC_PORT: 8084 SRC_RANGES: 0.0.0.0/0 DEST_PROTO: TCP DEST_PORT: 8084 DEST_IPS: 10.43.105.215 Mounts: <none> lb-tcp-8883: Image: rancher/klipper-lb:v0.4.4 Port: 8883/TCP Host Port: 8883/TCP Environment: SRC_PORT: 8883 SRC_RANGES: 0.0.0.0/0 DEST_PROTO: TCP DEST_PORT: 8883 DEST_IPS: 10.43.105.215 Mounts: <none> lb-tcp-1883: Image: rancher/klipper-lb:v0.4.4 Port: 1883/TCP Host Port: 1883/TCP Environment: SRC_PORT: 1883 SRC_RANGES: 0.0.0.0/0 DEST_PROTO: TCP DEST_PORT: 1883 DEST_IPS: 10.43.105.215 Mounts: <none> Volumes: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 34m daemonset-controller Created pod: svclb-emqx-listeners-c1057f0a-4d9bx Normal SuccessfulDelete 33m daemonset-controller Deleted pod: svclb-emqx-listeners-c1057f0a-4d9bx Normal SuccessfulCreate 33m daemonset-controller Created pod: svclb-emqx-listeners-c1057f0a-m27ml Normal SuccessfulDelete 32m daemonset-controller Deleted pod: svclb-emqx-listeners-c1057f0a-m27ml Normal SuccessfulCreate 32m daemonset-controller Created pod: svclb-emqx-listeners-c1057f0a-45nj6 Normal SuccessfulDelete 28m daemonset-controller Deleted pod: svclb-emqx-listeners-c1057f0a-45nj6 Normal SuccessfulCreate 28m daemonset-controller Created pod: svclb-emqx-listeners-c1057f0a-54wkz Normal SuccessfulDelete 28m daemonset-controller Deleted pod: svclb-emqx-listeners-c1057f0a-54wkz Normal SuccessfulCreate 28m daemonset-controller Created pod: svclb-emqx-listeners-c1057f0a-cvls4 Normal SuccessfulDelete 27m daemonset-controller Deleted pod: svclb-emqx-listeners-c1057f0a-cvls4 Normal SuccessfulCreate 27m daemonset-controller Created pod: svclb-emqx-listeners-c1057f0a-spzph Normal SuccessfulDelete 27m daemonset-controller Deleted pod: svclb-emqx-listeners-c1057f0a-spzph Normal SuccessfulCreate 27m daemonset-controller Created pod: svclb-emqx-listeners-c1057f0a-bdb8l Normal SuccessfulDelete 26m daemonset-controller Deleted pod: svclb-emqx-listeners-c1057f0a-bdb8l Normal SuccessfulCreate 26m daemonset-controller Created pod: svclb-emqx-listeners-c1057f0a-xqz92 Normal SuccessfulDelete 26m daemonset-controller Deleted pod: svclb-emqx-listeners-c1057f0a-xqz92 Normal SuccessfulCreate 26m daemonset-controller Created pod: svclb-emqx-listeners-c1057f0a-jbdj5 Normal SuccessfulDelete 25m daemonset-controller Deleted pod: svclb-emqx-listeners-c1057f0a-jbdj5 Normal SuccessfulCreate 22m (x5 over 25m) daemonset-controller (combined from similar events): Created pod: svclb-emqx-listeners-c1057f0a-8v9m6 Normal SuccessfulDelete 4m9s (x23 over 25m) daemonset-controller (combined from similar events): Deleted pod: svclb-emqx-listeners-c1057f0a-6twvk ``` **To Reproduce** Deploy emqx operator in K3s with listenersServiceTemplate->spec->type set to LoadBalancer. As per [documentation](https://docs.emqx.com/en/emqx-operator/latest/tasks/configure-emqx-service.html). **Expected behavior** EMQX-Operator does not trigger reconcile of emqx-listeners service when using loadbalancer. **Anything else we need to know?**: **Environment details:**: - Kubernetes version: v1.28.4+k3s2 - Cloud-provider/provisioner: k3s - emqx-operator version: 2.2.5 - Install method: helm

这个问题有 github issue,
可以在这里看到最新的进展。

pwc · 2024 年1 月 8 日 02:34

好的，我跟踪一下