🐛 Kship gets stuck and doesn't recover

[blabadi]

Describe the bug

2021-04-16 | 18:48:24.158 | 1 | KafkaConsumerDestination{consumerDestinationName='song_analysis', partitions=1, dlqName='kship-song-staging-01-dlq'}.container-0-C-1 | DEBUG | org.springframework.web.reactive.function.client.ExchangeFunctions | [54549d1] HTTP POST https://ego.staging.icgc-argo.org/api/oauth/token
2021-04-16 | 18:48:24.180 | 1 | reactor-http-epoll-4 | DEBUG | org.springframework.http.codec.FormHttpMessageWriter | [54549d1] Writing form fields [grant_type] (content masked)
2021-04-16 | 18:48:24.185 | 1 | reactor-http-epoll-4 | DEBUG | org.springframework.web.reactive.function.client.ExchangeFunctions | [54549d1] Response 307 TEMPORARY_REDIRECT
2021-04-16 | 18:48:24.186 | 1 | KafkaConsumerDestination{consumerDestinationName='song_analysis', partitions=1, dlqName='kship-song-staging-01-dlq'}.container-0-C-1 | ERROR | org.icgcargo.kship.ConsumerToHttp | failed to send message
org.springframework.web.reactive.function.client.WebClientResponseException: 307 Cannot get token, expected 2xx HTTP Status code
	at org.springframework.web.reactive.function.client.WebClientResponseException.create(WebClientResponseException.java:212)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ⇢ Request to POST null [DefaultWebClient]
Stack trace:....
2021-04-16 | 18:48:24.187 | 1 | KafkaConsumerDestination{consumerDestinationName='song_analysis', partitions=1, dlqName='kship-song-staging-01-dlq'}.container-0-C-1 | DEBUG | org.springframework.retry.backoff.ExponentialBackOffPolicy | Sleeping for 10000
2021-04-16 | 18:48:34.188 | 1 | KafkaConsumerDestination{consumerDestinationName='song_analysis', partitions=1, dlqName='kship-song-staging-01-dlq'}.container-0-C-1 | DEBUG | org.springframework.cloud.stream.messaging.DirectWithAttributesChannel | preSend on channel 'bean 'input'', message: GenericMessage [payload=byte[5329], headers={kafka_offset=24562, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@493891b4, deliveryAttempt=1131, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=0, kafka_receivedTopic=song_analysis, kafka_receivedTimestamp=1618587594978, kafka_groupId=kship-song-staging-grp}]
2021-04-16 | 18:48:34.188 | 1 | KafkaConsumerDestination{consumerDestinationName='song_analysis', partitions=1, dlqName='kship-song-staging-01-dlq'}.container-0-C-1 | DEBUG | org.springframework.cloud.stream.binding.StreamListenerMessageHandler | org.springframework.cloud.stream.binding.StreamListenerMessageHandler@7ae9a33a received message: GenericMessage [payload=byte[5329], headers={kafka_offset=24562, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@493891b4, deliveryAttempt=1131, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=0, kafka_receivedTopic=song_analysis, kafka_receivedTimestamp=1618587594978, contentType=application/json, kafka_groupId=kship-song-staging-grp}]
2021-04-16 | 18:48:34.189 | 1 | KafkaConsumerDestination{consumerDestinationName='song_analysis', partitions=1, dlqName='kship-song-staging-01-dlq'}.container-0-C-1 | DEBUG | org.icgcargo.kship.ConsumerToHttp | received a message : {
  "analysisId" : "474d9ae5-2eaa-49a7-8d9a-e52eaa79a7d5",
  "studyId" : "OCCAMS-GB",
  "state" : "UNPUBLISHED",
....
  }
}
2021-04-16 | 18:48:34.190 | 1 | KafkaConsumerDestination{consumerDestinationName='song_analysis', partitions=1, dlqName='kship-song-staging-01-dlq'}.container-0-C-1 | DEBUG | org.springframework.web.reactive.function.client.ExchangeFunctions | [5932ff16] HTTP POST https://ego.staging.icgc-argo.org/api/oauth/token
2021-04-16 | 18:48:34.191 | 1 | reactor-http-epoll-4 | DEBUG | org.springframework.http.codec.FormHttpMessageWriter | [5932ff16] Writing form fields [grant_type] (content masked)

the above is the last log statements kship produced before a maintenance weekend.

It seems that kship is stuck waiting a response from ego to obtain an oauth2 token before calling the gateway
this happened due to recent maintenance that took ego down

possible fix: add timeout to avoid being stuck if the root cause is being stuck on the actual http call to ego