Pools sometimes not paying out miners after they disconnect

[kbs1]

On xdag.org solo mining pool, DE2, we were performing maintenance on the node (reloading the daemon after changing the community fund address).
That means that during this time, miners were redirected to DE3 pool. This pool had it's config changed to 0-98-0-2.

After DE2 synced, the auto failover system took DE2 miners back. This is performed as a normal miner disconnection event.

Shortly after the miners were redirected back, I checked DE3 with "account" command to see if the new "community fee" distribution works OK.
I expected to see almost zero balance on all found main blocks. This was the command output:

xdag> account
dOR+nTSFrY77Sk+2kfAUbejWwXpp0aOb       1003.520000000  key 0
4TakIlRE613ayfhfcvv50872j1EGSN51       1003.520000000  key 0
AxhgWG+Oe1ByESco6i8BLtF7MvsF5YpW          0.000000012  key 0
lkUmKFCfeIsrSouvVtw3kziff+qVUhvY          0.000000009  key 0
pUGELtSbl0i6vqhjVxAd9kcDD57etN/7          0.000000000  key 0
TxnqVuETv56cBtNdjbJCJOoCmyxL9FvP          0.000000000  key 0
............

As we can see, first 2 shown "accounts" were not paid out correctly. Checking in explorer, it can be seen only the "community fund" transfer was made. In logs, the following was found:

grep -B 10 -A 10 -i xfer xdag.log
.....
2019-04-01 01:28:32.592 [7f7cb27fc700:INFO]  Create: 00000000016c8e4e8739f2829be5b1f7d92da1909273f97e5b44638c8a358b3e t=17285a4ffff pos=1
2019-04-01 01:28:32.592 [7f7cb27fc700:INFO]  MAIN  : 000000000000032f20d88a26b29e5010e87808d04345dea225ac9afb1c54f410 t=17285a3ffff pos=b400
2019-04-01 01:28:32.592 [7f7cb27fc700:INFO]  Good +: 00000000016c8e4e8739f2829be5b1f7d92da1909273f97e5b44638c8a358b3e t=17285a4ffff pos=fffffffffffffffe
2019-04-01 01:28:32.592 [7f7cb27fc700:INFO]  Mintop: 000000000005980d87a3be93ad4f64b37e8633f1331b22526e1d451ea69c6ff0 t=17285a4ffff pos=fffffffffffffffe
2019-04-01 01:28:32.609 [7f88c7a57700:INFO]  Pretop: 0000000000007a738bf4eddb473df7f777bec01b5841a8dd92bc1ce4c3e3487d t=17285a4ffff pos=fffffffffffffffe
2019-04-01 01:28:32.609 [7f88c7a57700:INFO]  Diff  : 9485c3fb4c343419ae029f89946 (+2173346058f69f8bcd0fb)
2019-04-01 01:28:32.609 [7f88c7a57700:INFO]  Good  : 0000000000007a738bf4eddb473df7f777bec01b5841a8dd92bc1ce4c3e3487d t=17285a4ffff pos=fffffffffffffffe
2019-04-01 01:28:32.611 [7f88c6a55700:INFO]  Pretop: 00000000000018578c93d976931e30cd64127d9dd9cd5ee6b7a500cceb305927 t=17285a4ffff pos=fffffffffffffffe
2019-04-01 01:28:32.611 [7f88c6a55700:INFO]  Diff  : 9485c4821dd7530fd598070a060 (+a844d77f4f1e34f34d815)
2019-04-01 01:28:32.611 [7f88c6a55700:INFO]  Good  : 00000000000018578c93d976931e30cd64127d9dd9cd5ee6b7a500cceb305927 t=17285a4ffff pos=fffffffffffffffe
***** [OK!] ***** 2019-04-01 01:28:33.490 [7f7cb2ffd700:MESS]  Xfer : from pUGELtSbl0i6vqhjVxAd9kcDD57etN/7 to YIGCwAVmrO5WMztjNI9a1sScPfVBRHDZ xfer 1003.519999999 XDAG
***** [OK!] ***** 2019-04-01 01:28:33.490 [7f7cb2ffd700:MESS]  Xfer : from pUGELtSbl0i6vqhjVxAd9kcDD57etN/7 to ajMoIX5GZOIlWRwkPWjkv5mlY1cSIadA xfer 20.479999999 XDAG
2019-04-01 01:28:33.490 [7f7cb2ffd700:INFO]  Create: 5942f3d9b2d3d9d226d648d97d0ab38f30ef4b3f8c01fea993a24120dd127a16 t=17285a505f5 pos=1
2019-04-01 01:28:33.490 [7f7cb2ffd700:INFO]  Pool: miner lAdk6Boe6FNwjWUOkaeu5huRj5OwkbWK is removed from miners list
***** [DISCONNECTED!] ***** 2019-04-01 01:28:33.490 [7f7cb2ffd700:INFO]  Pool: miner YIGCwAVmrO5WMztjNI9a1sScPfVBRHDZ is removed from miners list
2019-04-01 01:28:33.490 [7f7cb2ffd700:INFO]  Pool: miner 4vYeA0Hq/ej3T2lEUqPIFuPOjlu7T48H is removed from miners list
2019-04-01 01:28:33.490 [7f7cb2ffd700:INFO]  Pool: miner 8FCx+XqB1jXM0htaDRKQD3Nx72Y9azPY is removed from miners list
2019-04-01 01:28:33.490 [7f7cb2ffd700:INFO]  Pool: miner 1WYQMSxj8Vz4OsNslx3c0Nbu2+KTVviB is removed from miners list
2019-04-01 01:28:33.490 [7f7cb2ffd700:INFO]  Pool: miner GkECAohQVqoOzcKPi1JOZl6exwNdLz5W is removed from miners list
2019-04-01 01:28:33.490 [7f7cb2ffd700:INFO]  Pool: miner i0BRSVL3HQLfZpxd0anmWDnqYDq70KM9 is removed from miners list
2019-04-01 01:28:33.490 [7f7cb2ffd700:INFO]  Pool: miner 2t8rKbrR9ERqBRJnKVb0qvFtVfSdzGrr is removed from miners list
2019-04-01 01:28:33.490 [7f7cb2ffd700:INFO]  Pool: miner kllHUGcgV3lWjAcgM1o9JZreqeSshnfk is removed from miners list
--
2019-04-01 01:30:40.418 [7f7cb17fa700:INFO]  QueryB: t=16f3e400000 dt=100000
2019-04-01 01:30:40.450 [7f88c7a57700:INFO]  Good  : 0000000000000d446ce81c13ec1ed23102e068a565f5dd1748247b0b96b9f8b9 t=17285a6ffff pos=fffffffffffffffe
2019-04-01 01:30:40.512 [7f88c8a59700:INFO]  Good  : 51c339de7f4018e51936d6af305267f5ee869b74097b7df05995bfbc060b1c15 t=17285a22851 pos=c600
2019-04-01 01:30:40.666 [7f7cb17fa700:INFO]  QueryB: t=16f3e500000 dt=100000
2019-04-01 01:30:40.865 [7f7cb17fa700:INFO]  QueryB: t=16f3e600000 dt=100000
2019-04-01 01:30:40.930 [7f7cb27fc700:INFO]  Create: 00000000015280797f7bac6c573a70e173ef8040608ff63feafbd3b1555936da t=17285a6ffff pos=1
2019-04-01 01:30:40.930 [7f7cb27fc700:INFO]  Good +: 00000000015280797f7bac6c573a70e173ef8040608ff63feafbd3b1555936da t=17285a6ffff pos=fffffffffffffffe
2019-04-01 01:30:40.930 [7f7cb27fc700:INFO]  Mintop: 00000000000007f065b4d25ac4218f2ae6efe6c28c88bcc5bf4047cbc44b025a t=17285a6ffff pos=fffffffffffffffe
2019-04-01 01:30:41.062 [7f7cb17fa700:INFO]  QueryB: t=16f3e700000 dt=100000
2019-04-01 01:30:41.283 [7f7cb17fa700:INFO]  QueryB: t=16f3e800000 dt=100000
***** [MISSING PAYOUT!] ***** 2019-04-01 01:30:41.498 [7f7cb2ffd700:MESS]  Xfer : from 4TakIlRE613ayfhfcvv50872j1EGSN51 to ajMoIX5GZOIlWRwkPWjkv5mlY1cSIadA xfer 20.479999999 XDAG
2019-04-01 01:30:41.499 [7f7cb2ffd700:INFO]  Create: ccb022bfb66eee50b38f5442e8d8860fa5633816d23ecc37ac386b949f2f0387 t=17285a705fe pos=1
2019-04-01 01:30:41.499 [7f7cb2ffd700:INFO]  Paid  : 000000000000555475de4806518ff6ced3f9fb725ff8c9da5deb445422a436e1 t=1728598ffff res=0
2019-04-01 01:30:41.513 [7f7cb17fa700:INFO]  QueryB: t=16f3e900000 dt=100000
2019-04-01 01:30:41.546 [7f7cb3fff700:INFO]  Good +: ccb022bfb66eee50b38f5442e8d8860fa5633816d23ecc37ac386b949f2f0387 t=17285a705fe pos=0
2019-04-01 01:30:41.739 [7f7cb17fa700:INFO]  QueryB: t=16f3ea00000 dt=100000
2019-04-01 01:30:41.973 [7f7cb17fa700:INFO]  QueryB: t=16f3eb00000 dt=100000
2019-04-01 01:30:42.213 [7f7cb17fa700:INFO]  QueryB: t=16f3ec00000 dt=100000
2019-04-01 01:30:42.426 [7f7cb17fa700:INFO]  QueryB: t=16f3ed00000 dt=100000
2019-04-01 01:30:42.651 [7f7cb17fa700:INFO]  QueryB: t=16f3ee00000 dt=100000
2019-04-01 01:30:42.896 [7f7cb17fa700:INFO]  QueryB: t=16f3ef00000 dt=100000
--
2019-04-01 01:31:44.038 [7f7cb27fc700:INFO]  Pretop: 00000000009bbb7fef88d19dc09efd2b41e76d1d3584f1f36216d0c51c65d776 t=17285a7ffff pos=fffffffffffffffe
2019-04-01 01:31:44.038 [7f7cb27fc700:INFO]  Diff  : 9485c8c7235c8f3919ce612f633 (+1a4d30aca977da4db94)
2019-04-01 01:31:44.038 [7f7cb27fc700:INFO]  Good +: 00000000009bbb7fef88d19dc09efd2b41e76d1d3584f1f36216d0c51c65d776 t=17285a7ffff pos=fffffffffffffffe
2019-04-01 01:31:44.038 [7f7cb27fc700:INFO]  Mintop: 00000000009bbb7fef88d19dc09efd2b41e76d1d3584f1f36216d0c51c65d776 t=17285a7ffff pos=fffffffffffffffe
2019-04-01 01:31:44.075 [7f7cb9fff700:INFO]  MAIN  : 00000000000007f065b4d25ac4218f2ae6efe6c28c88bcc5bf4047cbc44b025a t=17285a6ffff pos=9200
2019-04-01 01:31:44.118 [7f7cb17fa700:INFO]  QueryB: t=16f56800000 dt=100000
2019-04-01 01:31:44.157 [7f88c6a55700:INFO]  Pretop: 000000000000634cecaccd7cfc79b3fb49caac11920c74f8aee8372c4e502360 t=17285a7ffff pos=fffffffffffffffe
2019-04-01 01:31:44.157 [7f88c6a55700:INFO]  Diff  : 9485c8f048afa1957d082a3d744 (+293fa043090cb1a35bca5)
2019-04-01 01:31:44.157 [7f88c6a55700:INFO]  Good  : 000000000000634cecaccd7cfc79b3fb49caac11920c74f8aee8372c4e502360 t=17285a7ffff pos=fffffffffffffffe
2019-04-01 01:31:44.317 [7f7cb17fa700:INFO]  QueryB: t=16f56900000 dt=100000
***** [MISSING PAYOUT!] ***** 2019-04-01 01:31:44.503 [7f7cb2ffd700:MESS]  Xfer : from dOR+nTSFrY77Sk+2kfAUbejWwXpp0aOb to ajMoIX5GZOIlWRwkPWjkv5mlY1cSIadA xfer 20.479999999 XDAG
2019-04-01 01:31:44.504 [7f7cb2ffd700:INFO]  Create: f7b9f7498d970b327a99ac15b3324ceb54afd36ef361ab70e4ae86215c3eb736 t=17285a80203 pos=1
2019-04-01 01:31:44.504 [7f7cb2ffd700:INFO]  Paid  : 0000000000004b3c9ba3d1697ac1d6e86d14f091b64f4afb8ead85349d7ee474 t=1728599ffff res=0
2019-04-01 01:31:44.542 [7f7cb17fa700:INFO]  QueryB: t=16f56a00000 dt=100000
2019-04-01 01:31:44.548 [7f88c7a57700:INFO]  Good  : 00000000000099ae56c4c23778cf788900a7eb47ee8aac7336b074baf27047f3 t=17285a7ffff pos=fffffffffffffffe
2019-04-01 01:31:44.550 [7f7cb3fff700:INFO]  Good +: f7b9f7498d970b327a99ac15b3324ceb54afd36ef361ab70e4ae86215c3eb736 t=17285a80203 pos=0
2019-04-01 01:31:44.718 [7f7cb17fa700:INFO]  QueryB: t=16f56b00000 dt=100000
2019-04-01 01:31:44.887 [7f7cb17fa700:INFO]  QueryB: t=16f56c00000 dt=100000
2019-04-01 01:31:45.058 [7f7cb17fa700:INFO]  QueryB: t=16f56d00000 dt=100000
2019-04-01 01:31:45.143 [7f7cb27fc700:INFO]  Mining: start from beginning because of pre-top block changed
2019-04-01 01:31:45.143 [7f7cb27fc700:INFO]  Mintop: 000000000000634cecaccd7cfc79b3fb49caac11920c74f8aee8372c4e502360 t=17285a7ffff pos=fffffffffffffffe

Please check the lines marked with stars. As you can see, after the strongest miner disconnected, he was no longer paid out. Please remember this was a solo mining config, so there SHOULD have been only one miner paid out per block found.
The miners were redirected to DE3, meaning the pool saw them as coming from one IP, but that probably doesn't play any role in this issue.