Groups code over federation hammers servers causing 429s and marking the server down

[ara4n]

e.g. 5 clients hitting POST /_matrix/client/r0/publicised_groups at the same time results in matrix.org hammering tfrmatrix.net over federation which it 429s and is then marked as down.

matrix@hera:~/synapse/var$ tail -1000000 homeserver.log| egrep 'POST-O-1063[0-5]'
2018-07-17 13:46:42,486 - synapse.http.outbound - 164 - INFO - POST-1167164 - {POST-O-10630} [tfrmatrix.net] Sending request: POST matrix://tfrmatrix.net/_matrix/federation/v1/get_groups_publicised
2018-07-17 13:46:42,488 - synapse.http.outbound - 164 - INFO - POST-1167189 - {POST-O-10631} [tfrmatrix.net] Sending request: POST matrix://tfrmatrix.net/_matrix/federation/v1/get_groups_publicised
2018-07-17 13:46:42,488 - synapse.http.outbound - 164 - INFO - POST-1167166 - {POST-O-10632} [tfrmatrix.net] Sending request: POST matrix://tfrmatrix.net/_matrix/federation/v1/get_groups_publicised
2018-07-17 13:46:42,489 - synapse.http.outbound - 164 - INFO - POST-1167202 - {POST-O-10633} [tfrmatrix.net] Sending request: POST matrix://tfrmatrix.net/_matrix/federation/v1/get_groups_publicised
2018-07-17 13:46:42,490 - synapse.http.outbound - 164 - INFO - POST-1167192 - {POST-O-10634} [tfrmatrix.net] Sending request: POST matrix://tfrmatrix.net/_matrix/federation/v1/get_groups_publicised
2018-07-17 13:46:47,825 - synapse.http.outbound - 246 - INFO - POST-1167192 - {POST-O-10634} [tfrmatrix.net] Result: 429 Unknown Status
2018-07-17 13:46:47,828 - synapse.http.outbound - 246 - INFO - POST-1167189 - {POST-O-10631} [tfrmatrix.net] Result: 429 Unknown Status
2018-07-17 13:46:49,256 - synapse.http.outbound - 246 - INFO - POST-1167164 - {POST-O-10630} [tfrmatrix.net] Result: 429 Unknown Status
2018-07-17 13:46:49,258 - synapse.http.outbound - 246 - INFO - POST-1167202 - {POST-O-10633} [tfrmatrix.net] Result: 429 Unknown Status
2018-07-17 13:46:49,260 - synapse.http.outbound - 246 - INFO - POST-1167166 - {POST-O-10632} [tfrmatrix.net] Result: 429 Unknown Status

[clokep]

Communities/groups are being removed, see #11584.