Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Failure in loading metadata when used with OS plugin implementing IndexStorePlugin #17202

Open
jitendra-titaniam opened this issue Jan 30, 2025 · 1 comment
Open

[BUG] Failure in loading metadata when used with OS plugin implementing IndexStorePlugin #17202

jitendra-titaniam opened this issue Jan 30, 2025 · 1 comment
Labels
bug Something isn't working Storage Issues and PRs relating to data and metadata storage untriaged

Comments

@jitendra-titaniam
Copy link

jitendra-titaniam commented Jan 30, 2025 •
edited
Loading

Describe the bug

Opensearch IndexStorePlugin allows plugin developers to provide custom implementation of org.apache.lucene.store.Directory.
Plugin developer can write their own implementation of Directory.createOutput(String filename, IOContext context) and Directory.openInput(String filename, IOContext context) methods.

Portal26 (AWS Partner) builds and sells a plugin for OpenSearch. The index store custom plugin adds encryption at rest for the data written to the file system. We have supported the custom index.store.type=titaniam in the index settings and implemented EncryptedFSDirectory to store the data as encrypted.

Current implementation of Store.readMetadataSnapshot is directly creating instance of NIOFSDirectory instead of using DirectoryFactory provided by IndexStorePlugin. This result in CorruptIndexException while listing store metadata for shard.

Related component

Indexing:Replication

To Reproduce

Can be reproduced with 2 node cluster with number_of_replicas 1.

Steps to reproduce the behavior:

  1. Create a Opensearch Plugin of type IndexStorePlugin. Implement getDirectoryFactories(). For index.store.type of titaniam this method returns an subclass of FsDirectoryFactory that creates custom implementation of Directory rather than an FSDirectory or NIOFSDirectory.
  2. On 2 nodes cluster of opensearch, install the plugin and restart Opensearch
  3. Create index
PUT lockbox-index-encrypted
{
  "mappings": {
    "properties": {
      "eventtime": {
        "type": "date"
      },
      "agentid": {
        "type": "keyword"
      },
      "cve": {
        "type": "keyword"
      }
    }
  },
  "settings": {
    "index.store.type": "titaniam",
    "number_of_replicas": 1
  }
}
  1. Ingest document
POST /lockbox-index-encrypted/_doc/1
{
  "eventtime": "2020-01-01",
  "agentid": "afabd04a-37af-4642-bce4-0048cc6a893b",
  "cve": "CVE-2077-1514"
}
  1. See that search works
GET lockbox-index-encrypted/_search
  1. Restart the nodes

Following warning is shown on one of the nodes

2024-11-25T10:03:12,083][WARN ][o.o.g.G.InternalReplicaShardAllocator] [opensearch-node2] [lockbox-index-encrypted][0]: failed to list shard for shard_store on node [1AQEeLNrSGaO6bVTWKVeTQ]
org.opensearch.action.FailedNodeException: Failed node [1AQEeLNrSGaO6bVTWKVeTQ]
	at org.opensearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:308) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:282) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService$6.handleException(TransportService.java:903) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1526) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1640) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1614) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService$8.onFailure(TransportService.java:1119) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:896) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:54) [opensearch-2.11.1.jar:2.11.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.opensearch.transport.RemoteTransportException: [opensearch-node2][172.24.0.5:9300][internal:cluster/nodes/indices/shard/store[n]]
Caused by: org.opensearch.OpenSearchException: Failed to list store metadata for shard [[lockbox-index-encrypted][0]]
	at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.nodeOperation(TransportNodesListShardStoreMetadata.java:157) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.nodeOperation(TransportNodesListShardStoreMetadata.java:85) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:200) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:328) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:324) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:113) ~[?:?]
	at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService$8.doRun(TransportService.java:1108) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.11.1.jar:2.11.1]
	... 3 more
Caused by: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=-872359702 vs expected footer=-1071082520 (resource=NIOFSIndexInput(path="/var/lib/lockboxdata-os211-n2/osv211/nodes/0/indices/XY2Y0jZCQ3CF8m-VcFD7tQ/0/index/_0.cfe"))
	at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:584) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:534) ~[lucene-core-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.opensearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:1232) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:1180) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:1154) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:1125) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.store.Store$MetadataSnapshot.<init>(Store.java:1048) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.store.Store.readMetadataSnapshot(Store.java:595) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.listStoreMetadata(TransportNodesListShardStoreMetadata.java:214) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.nodeOperation(TransportNodesListShardStoreMetadata.java:155) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.nodeOperation(TransportNodesListShardStoreMetadata.java:85) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:200) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:328) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:324) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:113) ~[?:?]
	at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService$8.doRun(TransportService.java:1108) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.11.1.jar:2.11.1]
	... 3 more
[2024-11-25T10:03:12,516][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-node2] Detected cluster change event for destination migration

Expected behavior

Metadata listing should use IndexStorePlugin to get the Directory instance.

Store.readMetadataSnapshot should use DirectoryFactory provided by the IndexStorePlugin to read the metadata.

Additional Details

Plugins
None except the custom Plugin implementing IndexStorePlugin

Host/Environment (please complete the following information):

  • OS: Linux
  • Version 2.11.1

Additional context
Similar direct access to file system was used in Knn instead of using IndexStorePlugin abstraction. This is fixed now.
opensearch-project/k-NN#1228
@jitendra-titaniam jitendra-titaniam added bug Something isn't working untriaged labels Jan 30, 2025
@github-actions github-actions bot added the Indexing:Replication Issues and PRs related to core replication framework eg segrep label Jan 30, 2025
@andrross andrross added the Storage Issues and PRs relating to data and metadata storage label Jan 31, 2025
@andrross andrross removed the Indexing:Replication Issues and PRs related to core replication framework eg segrep label Jan 31, 2025
@jitendra-titaniam
Copy link
Author

We see same error with OpenSearch Version 2.18.0. (Our customer is using Version 2.18.0).

[2025-02-03T06:36:29,056][WARN ][o.o.g.G.InternalReplicaShardAllocator_ShardId=[[lockbox-index-encrypted][0]]] [opensearch-node1-218] failed to list shard for shard_store on node [CYYMLF0ASlWLdZhKrf8Isw]
org.opensearch.action.FailedNodeException: Failed node [CYYMLF0ASlWLdZhKrf8Isw]
	at org.opensearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:314) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:288) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.TransportService$9.handleException(TransportService.java:1741) [opensearch-2.18.0.jar:2.18.0]
	at com.titaniamlabs.os.search.TitaniamTransportResponseHandler.handleException(TitaniamTransportResponseHandler.java:70) [os-v218-all-1.0-SNAPSHOT.jar:?]
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1527) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.NativeMessageHandler.lambda$handleException$5(NativeMessageHandler.java:454) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.NativeMessageHandler.handleException(NativeMessageHandler.java:452) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.NativeMessageHandler.handlerResponseError(NativeMessageHandler.java:444) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:172) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:796) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.18.0.jar:2.18.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.114.Final.jar:4.1.114.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.114.Final.jar:4.1.114.Final]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.transport.RemoteTransportException: [opensearch-node2-218][192.168.208.5:9300][internal:cluster/nodes/indices/shard/store[n]]
Caused by: org.opensearch.OpenSearchException: Failed to list store metadata for shard [[lockbox-index-encrypted][0]]
	at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.nodeOperation(TransportNodesListShardStoreMetadata.java:152) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.nodeOperation(TransportNodesListShardStoreMetadata.java:73) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:200) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:334) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:330) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:119) ~[?:?]
	at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:44) ~[?:?]
	at org.opensearch.performanceanalyzer.transport.RTFPerformanceAnalyzerTransportRequestHandler.messageReceived(RTFPerformanceAnalyzerTransportRequestHandler.java:63) ~[?:?]
	at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.NativeMessageHandler$RequestHandler.doRun(NativeMessageHandler.java:487) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1005) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.18.0.jar:2.18.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=242800877 vs expected footer=-1071082520 (resource=NIOFSIndexInput(path="/var/lib/lockboxdata-os218-n2/osv218/nodes/0/indices/Yg8PfFSfRhGxGM0J3dITyw/0/index/_0.cfe"))
	at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:585) ~[lucene-core-9.12.0.jar:9.12.0 e913796758de3d9b9440669384b29bec07e6a5cd - 2024-09-25 16:37:02]
	at org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:535) ~[lucene-core-9.12.0.jar:9.12.0 e913796758de3d9b9440669384b29bec07e6a5cd - 2024-09-25 16:37:02]
	at org.opensearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:1235) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:1183) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:1157) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:1128) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.index.store.Store$MetadataSnapshot.<init>(Store.java:1051) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.index.store.Store.readMetadataSnapshot(Store.java:597) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.indices.store.TransportNodesListShardStoreMetadataHelper.listShardMetadataInternal(TransportNodesListShardStoreMetadataHelper.java:111) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.listStoreMetadata(TransportNodesListShardStoreMetadata.java:158) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.nodeOperation(TransportNodesListShardStoreMetadata.java:150) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.nodeOperation(TransportNodesListShardStoreMetadata.java:73) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:200) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:334) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:330) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:119) ~[?:?]
	at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:44) ~[?:?]
	at org.opensearch.performanceanalyzer.transport.RTFPerformanceAnalyzerTransportRequestHandler.messageReceived(RTFPerformanceAnalyzerTransportRequestHandler.java:63) ~[?:?]
	at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.transport.NativeMessageHandler$RequestHandler.doRun(NativeMessageHandler.java:487) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1005) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.18.0.jar:2.18.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working Storage Issues and PRs relating to data and metadata storage untriaged
Projects
Storage Project Board
Status: 🆕 New
Milestone
No milestone
Development

No branches or pull requests
2 participants
@andrross @jitendra-titaniam