emqx版本：4.4.19
stop的服务器节点全部释放了，强制删除stop的节点失败
日志报错如下
2024-10-14T15:32:53.030212+08:00 [error] [Ctl] CMD Error:{error,{no_exists,emqx_sn_registry}}, Stacktrace:[{ekka_mnesia,ensure_ok,1,[{file,"ekka_mnesia.erl"},{line,285}]},{emqx_mgmt_cli,cluster,1,[{file,"emqx_mgmt_cli.erl"},{line,165}]},{emqx_ctl,run_command,2,[{file,"emqx_ctl.erl"},{line,104}]},{erpc,execute_call,4,[{file,"erpc.erl"},{line,392}]}]
控制台执行./bin/emqx eval 'io:format(“~p”,[ets:tab2list(schema)])'如下
root@iot-app-emqx-004-eu:/usr/local/bin/emqx# ./bin/emqx eval 'io:format("~p",[ets:tab2list(schema)])'
[{schema,emqx_shared_subscription,
[{name,emqx_shared_subscription},
{type,bag},
{ram_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,emqx_shared_subscription},
{attributes,[group,topic,subpid]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,{{1701341552438559519,-576460752303422847,1},
'emqx@172.26.203.182'}},
{version,{{11,7},{'emqx@172.26.203.247',{1728,872379,572514}}}}]},
{schema,emqx_trie,
[{name,emqx_trie},
{type,ordered_set},
{ram_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,emqx_trie},
{attributes,[key,count]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true},
{write_concurrency,true}]}]},
{cookie,{{1701341552436843354,-576460752303423033,1},
'emqx@172.26.203.182'}},
{version,{{11,7},{'emqx@172.26.203.247',{1728,872379,564071}}}}]},
{schema,emqx_resource,
[{name,emqx_resource},
{type,set},
{ram_copies,[]},
{disc_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[{3,ordered}]},
{snmp,[]},
{local_content,false},
{record_name,resource},
{attributes,[id,type,config,created_at,description]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,{{1701341552426838426,-576460752303423199,1},
'emqx@172.26.203.182'}},
{version,{{11,7},{'emqx@172.26.203.247',{1728,872379,516317}}}}]},
{schema,schema,
[{name,schema},
{type,set},
{ram_copies,[]},
{disc_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,schema},
{attributes,[table,cstruct]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,{{1701341552409927640,-576460752303423385,1},
'emqx@172.26.203.182'}},
{version,{{22,1},{'emqx@172.26.203.247',{1728,872379,467465}}}}]},
{schema,mqtt_app,
[{name,mqtt_app},
{type,set},
{ram_copies,[]},
{disc_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,mqtt_app},
{attributes,[id,secret,name,desc,status,expired]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,{{1701341552432766352,-576460752303423007,1},
'emqx@172.26.203.182'}},
{version,{{11,7},{'emqx@172.26.203.247',{1728,872379,543464}}}}]},
{schema,emqx_exclusive_subscription,
[{name,emqx_exclusive_subscription},
{type,set},
{ram_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,exclusive_subscription},
{attributes,[topic,clientid]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true},
{write_concurrency,true}]}]},
{cookie,{{1703476462406285875,-576460752303419518,1},
'emqx@172.26.203.224'}},
{version,{{5,0},{'emqx@172.26.203.247',{1728,872380,816361}}}}]},
{schema,emqx_rule_action,
[{name,emqx_rule_action},
{type,set},
{ram_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[{4,ordered},{5,ordered}]},
{snmp,[]},
{local_content,false},
{record_name,action},
{attributes,[name,category,for,app,types,module,on_create,
on_destroy,hidden,params_spec,title,description]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,{{1701341552425002880,-576460752303423295,1},
'emqx@172.26.203.182'}},
{version,{{11,7},{'emqx@172.26.203.247',{1728,872379,508567}}}}]},
{schema,emqx_rule,
[{name,emqx_rule},
{type,set},
{ram_copies,[]},
{disc_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[{3,ordered}]},
{snmp,[]},
{local_content,false},
{record_name,rule},
{attributes,[id,for,rawsql,is_foreach,fields,doeach,incase,
conditions,on_action_failed,actions,enabled,created_at,
description,state]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,{{1701341552422937381,-576460752303423391,1},
'emqx@172.26.203.182'}},
{version,{{11,7},{'emqx@172.26.203.247',{1728,872379,496861}}}}]},
{schema,emqx_route,
[{name,emqx_route},
{type,bag},
{ram_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,route},
{attributes,[topic,dest]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true},
{write_concurrency,true}]}]},
{cookie,{{1701341552442089363,-576460752303422655,1},
'emqx@172.26.203.182'}},
{version,{{11,7},{'emqx@172.26.203.247',{1728,872379,590227}}}}]},
{schema,emqx_mod_delayed,
[{name,emqx_mod_delayed},
{type,ordered_set},
{ram_copies,[]},
{disc_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,true},
{record_name,delayed_message},
{attributes,[key,msg]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,{{1701341552430750778,-576460752303423071,1},
'emqx@172.26.203.182'}},
{version,{{2,20},{'emqx@172.26.203.247',{1728,872379,534431}}}}]},
{schema,emqx_retainer,
[{name,emqx_retainer},
{type,set},
{ram_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,retained},
{attributes,[topic,msg,expiry_time]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{dets,[{auto_save,1000}]},
{ets,[compressed,
{read_concurrency,true},
{write_concurrency,true}]}]},
{cookie,{{1702452077341087169,-576460752292414747,1},
'emqx@172.26.203.182'}},
{version,{{8,7},{'emqx@172.26.203.247',{1728,872381,634604}}}}]},
{schema,emqx_activated_alarm,
[{name,emqx_activated_alarm},
{type,set},
{ram_copies,[]},
{disc_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,true},
{record_name,activated_alarm},
{attributes,[name,details,message,activate_at]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,{{1701341552446063272,-576460752303422559,1},
'emqx@172.26.203.182'}},
{version,{{2,20},{'emqx@172.26.203.247',{1728,872380,836013}}}}]},
{schema,emqx_deactivated_alarm,
[{name,emqx_deactivated_alarm},
{type,ordered_set},
{ram_copies,[]},
{disc_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,true},
{record_name,deactivated_alarm},
{attributes,[activate_at,name,details,message,deactivate_at]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,{{1701341552448120166,-576460752303422527,1},
'emqx@172.26.203.182'}},
{version,{{2,20},{'emqx@172.26.203.247',{1728,872380,845994}}}}]},
{schema,emqx_sn_registry,
[{name,emqx_sn_registry},
{type,set},
{ram_copies,['emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,emqx_sn_registry},
{attributes,[key,value]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,{{1701341552421172953,-576460752303423487,1},
'emqx@172.26.203.182'}},
{version,{{8,3},{'emqx@172.26.203.218',{1701,350681,25226}}}}]},
{schema,emqx_channel_registry,
[{name,emqx_channel_registry},
{type,bag},
{ram_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,channel},
{attributes,[chid,pid]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true},
{write_concurrency,true}]}]},
{cookie,{{1701341552452742226,-576460752303422431,1},
'emqx@172.26.203.182'}},
{version,{{11,7},{'emqx@172.26.203.247',{1728,872380,874205}}}}]},
{schema,emqx_telemetry,
[{name,emqx_telemetry},
{type,set},
{ram_copies,[]},
{disc_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,true},
{record_name,telemetry},
{attributes,[id,uuid,enabled]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,{{1701341552419108391,-576460752303423257,1},
'emqx@172.26.203.182'}},
{version,{{2,20},{'emqx@172.26.203.247',{1728,872379,489228}}}}]},
{schema,emqx_banned,
[{name,emqx_banned},
{type,set},
{ram_copies,[]},
{disc_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,banned},
{attributes,[who,by,reason,at,until]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,{{1701341552443851608,-576460752303422937,1},
'emqx@172.26.203.182'}},
{version,{{11,7},{'emqx@172.26.203.247',{1728,872380,824225}}}}]},
{schema,mqtt_admin,
[{name,mqtt_admin},
{type,set},
{ram_copies,[]},
{disc_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,mqtt_admin},
{attributes,[username,password,tags]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true},
{write_concurrency,true}]}]},
{cookie,{{1701341552434772219,-576460752303422911,1},
'emqx@172.26.203.182'}},
{version,{{11,7},{'emqx@172.26.203.247',{1728,872379,553178}}}}]},
{schema,emqx_resource_type,
[{name,emqx_resource_type},
{type,set},
{ram_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[{3,ordered}]},
{snmp,[]},
{local_content,false},
{record_name,resource_type},
{attributes,[name,provider,params_spec,on_create,on_status,
on_destroy,title,description]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,{{1701341552428894392,-576460752303423161,1},
'emqx@172.26.203.182'}},
{version,{{11,7},{'emqx@172.26.203.247',{1728,872379,526539}}}}]},
{schema,emqx_routing_node,
[{name,emqx_routing_node},
{type,set},
{ram_copies,['emqx@172.26.203.247','emqx@172.26.203.230',
'emqx@172.26.203.225','emqx@172.26.203.224',
'emqx@172.26.203.218','emqx@172.26.203.204',
'emqx@172.26.203.183','emqx@172.26.203.182']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,routing_node},
{attributes,[name,const]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,{{1701341552440277308,-576460752303422751,1},
'emqx@172.26.203.182'}},
{version,{{11,7},{'emqx@172.26.203.247',{1728,872379,582875}}}}]}]ok
我猜测可能是你现在正在 running 的两个节点也没有正常工作。看看这两个节点的日志，是不是还在打印等待其他节点的日志？
running的节点能够正常工作，mqtt client也能正常访问，集群使用没问题问题，目前上面有很多在线的设备正常使用，查看日志并没有一直报错的日志
我在社区看到一个类似的问题：Unable to force-leave stale nodes · Issue #12850 · emqx/emqx · GitHub ，但是我们报的错误不一样
你的问题和你提到的 GitHub 帖子里的问题是一回事儿。这样就比较麻烦了，你按照 William( qzhuyan) 给出的方案试试看：
这里你的 Tab = emqx_sn_registry。
你那些 stopped 节点还能拉起来吗？完全没办法拉起来的话，试试看使用同样的节点名启动新的节点，注意 emqx 版本号要跟原来一样。然后再加回集群去，最后 force-leave 掉它再释放机器。
stopped的节点已经完全释放了，无法再次拉起，需要使用相同的ip在添加一次再移除吗
你要么按照 GitHub 帖子里的试试，要么用我这个办法使用相同 IP 地址把老的节点拉起来，注意 emqx 版本号必须是跟之前老节点的一样。