Cluster setup and troubleshooting

[wickstargazer]

Hi!

I am running into these logs on AWS ECS

Tried to associate with unreachable remote address [akka.tcp://document-transactional-system@9c188c0eb1fc48e09885acf4c9fd340a.document-batch-system.batch-import-system:5110]. 
Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. 
Reason: [Association failed with akka.tcp://document-transactional-system@9c188c0eb1fc48e09885acf4c9fd340a.document-batch-system.batch-import-system:5110] 
Caused by: [System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known

also this

Dropping message [Akka.Actor.ActorSelectionMessage] for non-local recipient

I am using lighthouse and also service-discovery on AWS to help with the dynamic dns. However it keeps reporting the above error.

1. How do I fix it?
2. How do I suppress it?

Lastly, I am also wondering how would I know if the stream that I started crashed and restart it when the actor restarts or the container restarts?

I am having a bit of a hard time with the circuit break and supervision strategy with the massive amount of snapshots in the DB that is accumulated over the crashes. Would be awesome to get some guidance here.

[Aaronontheweb]

Caused by: [System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known

This means you attempted to connect to a service DNS for which there was no resolvable hostname - this is a DNS configuration error. Happens at the network layer above Akka.NET.

Dropping message [Akka.Actor.ActorSelectionMessage] for non-local recipient

This means that Akka.Remote received the message but the hostname the message was addressed to was different than Akka.Remote's binding address - you do not want to suppress these messages because it means your Akka.Remote configuration is wrong either on the sender side or the receiver side. Set the akka.remote.dot-netty.public-hostname of akka.remote to match the same domain name as what the sender is using and that should resolve this issue.

[wickstargazer]

i am trying to set the hostname to 0.0.0.0 and my public-hostname is set from the dynamic name ECS assign to from the service-discovery feature. i am assigning it in code like so..

var remoteConfig = $"akka.remote.dot-netty.tcp.public-hostname = {taskId}.document-batch-system.batch-import-system\n";
                config = ConfigurationFactory.ParseString(remoteConfig)
                    .WithFallback(config);
                var remoteIpConfig = $"akka.remote.dot-netty.tcp.hostname = 0.0.0.0\n";

taskId comes from a meta-data endpoint called within the code.
will keep an eye out to see if its still happening

[Aaronontheweb]

Lastly, I am also wondering how would I know if the stream that I started crashed and restart it when the actor restarts or the container restarts?

Are you referring to a user-defined stream here? Can you give us some more information?

[wickstargazer]

yes its a user defined stream to query tags from a journal. But when the system/actor dies, that stream is lost?
How do i make sure the stream is also restored and the tags keeps getting into the stream so i can continue sending the progress of the job

 var readJournal = Akka.Persistence.Query.PersistenceQuery.Get(Context.System)
                    .ReadJournalFor<Akka.Persistence.Query.Sql.SqlReadJournal>("akka.persistence.query.journal.sql");
Source<EventEnvelope, Akka.NotUsed> itemSource = readJournal.EventsByTag(message.BatchId);

thats in a receive actor.