rabbitmq clustering is getting failed

[nagarjunakurra]

i have created a 3 node cluster for rabbitmq but clustering getting failed.config file is as follows

cluster_formation.peer_discovery_backend = classic_config
cluster_formation.classic_config.nodes.1 = rabbit@rabbitmq-01
cluster_formation.classic_config.nodes.2 = rabbit@rabbitmq-02
cluster_formation.classic_config.nodes.3 = rabbit@rabbitmq-03
mnesia_table_loading_retry_timeout = 60000
mnesia_table_loading_retry_limit = 15

error is as follows

| 2022-01-12 11:10:16.412 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-02: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:16.413 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-03: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:16.413 [error] <0.272.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 9 retries left...
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:16.914 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-02: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:16.915 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-03: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:16.915 [error] <0.272.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 8 retries left...
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:17.416 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-02: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:17.417 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-03: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:17.417 [error] <0.272.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 7 retries left...
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:17.918 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-02: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:17.919 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-03: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:17.919 [error] <0.272.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 6 retries left...
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:18.420 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-02: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:18.421 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-03: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:18.421 [error] <0.272.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 5 retries left...
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:18.922 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-02: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:18.923 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-03: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:18.923 [error] <0.272.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 4 retries left...
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:19.423 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-02: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:19.424 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-03: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:19.424 [error] <0.272.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 3 retries left...
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:19.925 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-02: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:19.926 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-03: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:19.926 [error] <0.272.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 2 retries left...
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:20.426 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-02: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:20.427 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-03: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:20.427 [error] <0.272.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 1 retries left...
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:20.928 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-02: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:20.929 [warning] <0.272.0> Could not auto-cluster with node rabbit@rabbitmq-03: {error,tables_not_present}
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:20.929 [error] <0.272.0> Trying to join discovered peers failed. Will retry after a delay of 500 ms, 0 retries left...
qa_rabbitmq-node-1.1.gw0pyyy7ylay@manager-2 | 2022-01-12 11:10:21.430 [warning] <0.272.0> Could not successfully contact any node of: rabbit@rabbitmq-02,rabbit@rabbitmq-03 (as in Erlang distribution). Starting as a blank standalone node...

any help would save me out of the problem
Thanks in Advance

[wglambert]

This issue seems similar rabbitmq/rabbitmq-peer-discovery-aws#11

But I would try asking a rabbitmq-specific resource for help, given the comment here rabbitmq/rabbitmq-peer-discovery-aws#11 (comment)

[nagarjunakurra]

we are running the above configuration in docker swarm mode on-premisis servers.and we are using rabbitmq:3.8.10-management image can any please help me with this