SystemInitializers should not give upon the first failed attempt

[rymsha]

A store timeout laves system repository in broken state so the entire cluster cannot boot up properly

Initializer should retry the init in case of a failure

2024-05-30T18:06:57,231 ERROR c.e.x.c.i.s.SecurityInitializer - Unable to initialize role: role:system.auditlog
com.enonic.xp.repository.IndexException: Failed to store document with id [role:system.auditlog] in index [search-system-repo] branch master
	at com.enonic.xp.repo.impl.elasticsearch.storage.StorageDaoImpl.store(StorageDaoImpl.java:118)
	at com.enonic.xp.repo.impl.storage.IndexDataServiceImpl.store(IndexDataServiceImpl.java:105)
	at com.enonic.xp.repo.impl.storage.NodeStorageServiceImpl.store(NodeStorageServiceImpl.java:107)
	at com.enonic.xp.repo.impl.storage.NodeStorageServiceImpl.store(NodeStorageServiceImpl.java:76)
	at com.enonic.xp.repo.impl.node.CreateNodeCommand.execute(CreateNodeCommand.java:98)
	at com.enonic.xp.repo.impl.node.NodeServiceImpl.doCreate(NodeServiceImpl.java:417)
	at com.enonic.xp.repo.impl.node.NodeServiceImpl.create(NodeServiceImpl.java:405)
	at com.enonic.xp.core.impl.security.SecurityServiceImpl.lambda$createRole$28(SecurityServiceImpl.java:756)
	at com.enonic.xp.context.ContextImpl.callWith(ContextImpl.java:100)
	at com.enonic.xp.core.impl.security.SecurityServiceImpl.callWithContext(SecurityServiceImpl.java:1084)
	at com.enonic.xp.core.impl.security.SecurityServiceImpl.createRole(SecurityServiceImpl.java:756)
	at com.enonic.xp.core.impl.security.SecurityInitializer.addRole(SecurityInitializer.java:331)
	at com.enonic.xp.core.impl.security.SecurityInitializer.initializeRoles(SecurityInitializer.java:263)
	at com.enonic.xp.core.impl.security.SecurityInitializer.lambda$doInitialize$0(SecurityInitializer.java:117)
	at com.enonic.xp.context.ContextImpl.runWith(ContextImpl.java:84)
	at com.enonic.xp.core.impl.security.SecurityInitializer.doInitialize(SecurityInitializer.java:114)
	at com.enonic.xp.init.Initializer.initialize(Initializer.java:51)
Caused by: org.elasticsearch.ElasticsearchTimeoutException: Timeout waiting for task.
	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:70)
	at com.enonic.xp.repo.impl.elasticsearch.storage.StorageDaoImpl.store(StorageDaoImpl.java:111)
	... 243 common frames omitted

2024-05-30T18:06:57,274 INFO  c.e.x.c.i.s.SecurityInitializer - Added user:system:su as member of role:system.admin
2024-05-30T18:06:57,278 INFO  c.e.x.c.i.s.SecurityInitializer - Initializing [/keys] folder
2024-05-30T18:07:07,436 ERROR c.e.x.c.i.s.SecurityServiceActivator - bundle com.enonic.xp.core.security:7.14.1 (73)[com.enonic.xp.core.impl.security.SecurityServiceActivator(148)] : The activate method has thrown an exception
org.apache.felix.log.LogException: com.enonic.xp.repository.IndexException: Failed to store document with id [95b432c2-8429-47cb-8fb5-bd8a015148f6] in index [search-system-repo] branch master
	at com.enonic.xp.repo.impl.elasticsearch.storage.StorageDaoImpl.store(StorageDaoImpl.java:118)
	at com.enonic.xp.repo.impl.storage.IndexDataServiceImpl.store(IndexDataServiceImpl.java:105)
	at com.enonic.xp.repo.impl.storage.NodeStorageServiceImpl.store(NodeStorageServiceImpl.java:107)
	at com.enonic.xp.repo.impl.storage.NodeStorageServiceImpl.store(NodeStorageServiceImpl.java:76)
	at com.enonic.xp.repo.impl.node.CreateNodeCommand.execute(CreateNodeCommand.java:98)
	at com.enonic.xp.repo.impl.node.NodeServiceImpl.doCreate(NodeServiceImpl.java:417)
	at com.enonic.xp.repo.impl.node.NodeServiceImpl.create(NodeServiceImpl.java:405)
	at com.enonic.xp.core.impl.security.SecurityInitializer.initializeKeys(SecurityInitializer.java:182)
	at com.enonic.xp.core.impl.security.SecurityInitializer.lambda$doInitialize$0(SecurityInitializer.java:122)
	at com.enonic.xp.context.ContextImpl.runWith(ContextImpl.java:84)
	at com.enonic.xp.core.impl.security.SecurityInitializer.doInitialize(SecurityInitializer.java:114)
	at com.enonic.xp.init.Initializer.initialize(Initializer.java:51)
	at com.enonic.xp.core.impl.security.SecurityServiceActivator.activate(SecurityServiceActivator.java:43)
Caused by: org.apache.felix.log.LogException: org.elasticsearch.ElasticsearchTimeoutException: Timeout waiting for task.
	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:70)
	at com.enonic.xp.repo.impl.elasticsearch.storage.StorageDaoImpl.store(StorageDaoImpl.java:111)
	... 238 common frames omitted

[rymsha]

Additionaly based on experience xp.init.maxTries should be 900 (which gives ~15 minutes) #10615