[BUG] Cross cluster replication fails to allocate shard on follower cluster

[borutlukic]

What is the bug?
Replication does not start. Shard fails to allocate.

How can one reproduce the bug?
Steps to reproduce the behavior:

PUT _plugins/_replication/proxy-2024.11/_start
{
"leader_alias": "main-cluster",
"leader_index": "proxy-2024.11",
"use_roles":{
"leader_cluster_role": "all_access",
"follower_cluster_role": "all_access"
}
}

4. See error
   GET _plugins/_replication/proxy-2024.11/_status
   {
   "status": "FAILED",
   "reason": "",
   "leader_alias": "prod-mon-elk-muc",
   "leader_index": "proxy-2024.11",
   "follower_index": "proxy-2024.11"
   }

What is the expected behavior?
Replication should start

What is your host/environment?

• OS: Ubuntu 22
• Opensearch 2.18 docker image
• Plugins: just what is part of opensearch docker image

Do you have any screenshots?
N/A

Do you have any additional context?
Opensearch logs give lots java stack traces, but they all end with java.lang.IllegalStateException: confined:

Example stack log:
[2024-11-25T16:51:05,128][ERROR][o.o.r.r.RemoteClusterRepository] [opensearch-node-114] Restore of [proxy-2024.11][0] failed due to java.lang.IllegalStateException: confined
at org.apache.lucene.store.MemorySegmentIndexInput.ensureAccessible(MemorySegmentIndexInput.java:103)
at org.apache.lucene.store.MemorySegmentIndexInput.buildSlice(MemorySegmentIndexInput.java:461)
at org.apache.lucene.store.MemorySegmentIndexInput.clone(MemorySegmentIndexInput.java:425)
at org.apache.lucene.store.MemorySegmentIndexInput$SingleSegmentImpl.clone(MemorySegmentIndexInput.java:530)
at org.opensearch.replication.repository.RestoreContext.openInput(RestoreContext.kt:39)
at org.opensearch.replication.repository.RemoteClusterRestoreLeaderService.openInputStream(RemoteClusterRestoreLeaderService.kt:76)
at org.opensearch.replication.action.repository.TransportGetFileChunkAction$shardOperation$1.invoke(TransportGetFileChunkAction.kt:59)
at org.opensearch.replication.action.repository.TransportGetFileChunkAction$shardOperation$1.invoke(TransportGetFileChunkAction.kt:57)
at org.opensearch.replication.util.ExtensionsKt.performOp(Extensions.kt:55)
at org.opensearch.replication.util.ExtensionsKt.performOp$default(Extensions.kt:52)
at org.opensearch.replication.action.repository.TransportGetFileChunkAction.shardOperation(TransportGetFileChunkAction.kt:57)
at org.opensearch.replication.action.repository.TransportGetFileChunkAction.shardOperation(TransportGetFileChunkAction.kt:33)
at org.opensearch.action.support.single.shard.TransportSingleShardAction.lambda$asyncShardOperation$0(TransportSingleShardAction.java:131)
at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74)
at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1005)
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.lang.Thread.run(Thread.java:1583)

Followed by:
[2024-11-25T16:51:05,135][ERROR][o.o.r.r.RemoteClusterRepository] [opensearch-node-114] Releasing leader resource failed due to NotSerializableExceptionWrapper[wrong_thread_exception: Attempted access outside owning thread]
at jdk.internal.foreign.MemorySessionImpl.wrongThread(MemorySessionImpl.java:315)
at jdk.internal.misc.ScopedMemoryAccess$ScopedAccessError.newRuntimeException(ScopedMemoryAccess.java:113)
at jdk.internal.foreign.MemorySessionImpl.checkValidState(MemorySessionImpl.java:219)
at jdk.internal.foreign.ConfinedSession.justClose(ConfinedSession.java:83)
at jdk.internal.foreign.MemorySessionImpl.close(MemorySessionImpl.java:242)
at jdk.internal.foreign.MemorySessionImpl$1.close(MemorySessionImpl.java:88)
at org.apache.lucene.store.MemorySegmentIndexInput.close(MemorySegmentIndexInput.java:514)
at org.opensearch.replication.repository.RestoreContext.close(RestoreContext.kt:52)
at org.opensearch.replication.repository.RemoteClusterRestoreLeaderService.removeLeaderClusterRestore(RemoteClusterRestoreLeaderService.kt:142)
at org.opensearch.replication.action.repository.TransportReleaseLeaderResourcesAction.shardOperation(TransportReleaseLeaderResourcesAction.kt:48)
at org.opensearch.replication.action.repository.TransportReleaseLeaderResourcesAction.shardOperation(TransportReleaseLeaderResourcesAction.kt:31)
at org.opensearch.action.support.single.shard.TransportSingleShardAction.lambda$asyncShardOperation$0(TransportSingleShardAction.java:131)
at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74)
at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1005)
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.lang.Thread.run(Thread.java:1583)

[borutlukic]

Setting:
"plugins.replication.follower.index.recovery.chunk_size": "1gb",
"plugins.replication.follower.index.recovery.max_concurrent_file_chunks": "1"

Seems to fix the issue. It appears that if the files are too large on the primary cluster, replication fails to start unless recovery.chunk_size is big enough to transfer files in one go.

[borutlukic]

It appears that setting 'plugins.replication.follower.index.recovery.chunk_size' to the max (which is 1gb) I can get all but one index to replicate. Would it be possible to raise the limit to higher than 1gb? As there appears to be something strange happening when the transfer of files needs to happen in chunks. The chunk with offset >0 will fail to get transferred with the 'java.lang.IllegalStateException: confined'

[dblock]

[Catch All Triage - 1, 2, 3, 4]

[Lavisx]

I am experiencing the same problem. After upgrading to OpenSearch 2.18.0 from 2.12.0, I am unable to create the replication. My index has a 10GB shard, and it cannot be transferred.