rootcood/datacoord/querycoord/indexcoord启动报错，没有对public/default空间有相应的权限Request is denied: namespace=public/default

[lb405828825]

前提参考：issue: #38401

日志：
time="2024-12-12T10:34:10Z" level=error msg="[Failed to create producer]" error="server error: ServiceNotReady: Namespace bundle for topic (persistent://public/default/cpic-milvus-
rootcoord-dml_0) not served by this instance:broker:8080. Please redo the lookup. Request is denied: namespace=public/default" topic="persistent://public/default/cpic-milvus-rootco
ord-dml_0"

执行命令：docker stack deploy -c milvus-coord.yaml milvus --with-registry-auth

在这一步报错，rootcood/datacoord/querycoord/indexcoord restart always failed，日志显示就是role在public/default没有对应权限。可是需要赋哪些role赋予什么样的权限呢？

已经对 role: admin、others、rootcoord、querycoord、indexcoord、datacoord赋予了[produce,consume,functions,sources,sinks,packages] 还是不行

[yhmo]

看日志是调用pulsar的client接口时没有权限增删topic，这应该跟milvus自己的role/privilege没关系。这个应该只跟你在milvus.yaml中pulsar的配置相关。你的pulsar是不是有权限设置的。

[lb405828825]

你好，我发现pulsar-broker 的容器不断的产生如下错误日志
milvus_broker_dockerlogs-1213.log
，或许这才是进一步的原因：
2024-12-13T02:25:26,704+0000 [pulsar-io-3-16] WARN org.apache.pulsar.broker.service.ServerCnx - [/10.0.18.54:35508][persistent://public/default/cpic-milvus-rootcoord-dml_2][cpic-milvus-dataNode-131-cpic-milvus-rootcoord-dml_2_454534945721548834v0] Failed to create consumer: consumerId=9, Namespace bundle for topic (persistent://public/default/cpic-milvus-rootcoord-dml_2) not served by this instance:broker:8080. Please redo the lookup. Request is denied: namespace=public/default
2024-12-13T02:25:27,938+0000 [pulsar-load-manager-1-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Writing local data to metadata store because maximum change Infinity% exceeded threshold 10%; time since last report written is 300.081 seconds
2024-12-13T02:25:27,939+0000 [pulsar-load-manager-1-1] WARN org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Error writing broker data on metadata store
java.util.concurrent.CompletionException: java.lang.IllegalStateException: Lock was not in valid state: Released
at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:413) ~[?:?]
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2118) ~[?:?]
at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.writeBrokerDataOnZooKeeper(ModularLoadManagerImpl.java:1125) ~[org.apache.pulsar-pulsar-broker-3.1.0.jar:3.1.0]
at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.writeBrokerDataOnZooKeeper(ModularLoadManagerImpl.java:1110) ~[org.apache.pulsar-pulsar-broker-3.1.0.jar:3.1.0]
at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerWrapper.writeLoadReportOnZookeeper(ModularLoadManagerWrapper.java:124) ~[org.apache.pulsar-pulsar-broker-3.1.0.jar:3.1.0]
at org.apache.pulsar.broker.loadbalance.LoadReportUpdaterTask.run(LoadReportUpdaterTask.java:39) ~[org.apache.pulsar-pulsar-broker-3.1.0.jar:3.1.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.lang.IllegalStateException: Lock was not in valid state: Released
at org.apache.pulsar.metadata.coordination.impl.ResourceLockImpl.lambda$updateValue$0(ResourceLockImpl.java:80) ~[org.apache.pulsar-pulsar-metadata-3.1.0.jar:3.1.0]
at org.apache.pulsar.common.util.FutureUtil$Sequencer.sequential(FutureUtil.java:198) ~[org.apache.pulsar-pulsar-common-3.1.0.jar:3.1.0]
at org.apache.pulsar.metadata.coordination.impl.ResourceLockImpl.updateValue(ResourceLockImpl.java:77) ~[org.apache.pulsar-pulsar-metadata-3.1.0.jar:3.1.0]
... 11 more

[yhmo]

应该是pulsar broker部署方式有问题(可能数量太少了)，等待 @LoveEachDay 的评论

[lb405828825]

我是通过官方地址https://milvus.io/tools/sizing 这个来定broker 2个。
另外我把另外一个broker的容器的docker logs日志也上传上来了，等2位大佬帮忙看看。
@yhmo @LoveEachDay
milvus_broker-2-dockerlog-1213.log

2024-12-13T02:25:12,842+0000 [broker-topic-workers-OrderedExecutor-4-0] ERROR org.apache.pulsar.broker.admin.v2.PersistentTopics - [null] Failed to delete subscription cpic-milvus-dataNode-131-cpic-milvus-rootcoord-dml_2_454534945721548834v0 from topic persistent://public/default/cpic-milvus-rootcoord-dml_2
org.apache.pulsar.broker.web.RestException: Subscription cpic-milvus-dataNode-131-cpic-milvus-rootcoord-dml_2_454534945721548834v0 not found for topic persistent://public/default/cpic-milvus-rootcoord-dml_2
at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.lambda$internalDeleteSubscriptionForNonPartitionedTopicAsync$165(PersistentTopicsBase.java:1609) ~[org.apache.pulsar-pulsar-broker-3.1.0.jar:3.1.0]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
at org.apache.pulsar.broker.service.BrokerService$2.lambda$openLedgerComplete$5(BrokerService.java:1704) ~[org.apache.pulsar-pulsar-broker-3.1.0.jar:3.1.0]
at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:787) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614) ~[?:?]
at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:795) ~[?:?]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?]
at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.2.jar:4.16.2]
at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:113) ~[org.apache.bookkeeper-bookkeeper-common-4.16.2.jar:4.16.2]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
2024-12-13T02:25:12,871+0000 [broker-topic-workers-OrderedExecutor-4-0] INFO org.eclipse.jetty.server.RequestLog - 10.0.18.54 - - [13/Dec/2024:02:25:11 +0000] "DELETE /admin/v2/persistent/public/default/cpic-milvus-rootcoord-dml_2/subscription/cpic-milvus-dataNode-131-cpic-milvus-rootcoord-dml_2_454534945721548834v0?force=true HTTP/1.1" 404 175 "-" "None" 1081
......
2024-12-13T02:26:32,051+0000 [pulsar-io-3-3] ERROR org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/__change_events] [null] Failed to create producer: {"errorMsg":"Namespace bundle for topic (persistent://public/default/__change_events) not served by this instance:broker:8080. Please redo the lookup. Request is denied: namespace=public/default","reqId":584862220398092166, "remote":"broker/10.0.18.45:6650", "local":"/10.0.18.45:58786"}
2024-12-13T02:26:32,052+0000 [pulsar-io-3-3] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/__change_events] [null] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException$LookupException: {"errorMsg":"Namespace bundle for topic (persistent://public/default/__change_events) not served by this instance:broker:8080. Please redo the lookup. Request is denied: namespace=public/default","reqId":584862220398092166, "remote":"broker/10.0.18.45:6650", "local":"/10.0.18.45:58786"}

[yhmo]

但是看你前面的截图里broker只有一个？

[lb405828825]

global模式，2个哦，2个broker的日志，可以看上面提供的日志。我怀疑是不是这两个broker容器启动有啥异常导致的？我目前把2个broker 容器docker stop之后 滚动重启，目前rootcoord容器启动了，但datacoord还会报权限错误，只能反复去重启pulsar-broker等容器来试图启动集群.......等待大佬们帮忙解答.