fix: connection dropped for refresh-data [TCTC-6839] #600
Conversation
Codecov ReportAll modified lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## release/v0.9 #600 +/- ##
==============================================
Coverage 100.00% 100.00%
==============================================
Files 19 19
Lines 839 845 +6
==============================================
+ Hits 839 845 +6
☔ View full report in Codecov by Sentry. |
|
on merge pas tant qu'on n'est pas sur que ça fix le souci du client |
Sanix-Darker
force-pushed
the
fix/connection-dropped
branch
2 times, most recently
from
a377fa6 to
4fd67a8
Compare
Sanix-Darker
force-pushed
the
fix/connection-dropped
branch
3 times, most recently
from
1e0ba0c to
36f76d4
Compare
peakina/io/ftp/ftp_utils.py
Outdated
| log.warning(f"File '{file_path}' not available inside : '{dir_path}' !") | ||
|
|
||
| parsed_url = urllib.parse.urlsplit(url) | ||
| path_without_file = parsed_url.path.rsplit("/", 1)[0] | ||
| modified_url = urllib.parse.urlunsplit( | ||
| (parsed_url.scheme, parsed_url.netloc, path_without_file, "", "") | ||
| ) | ||
|
|
||
| files_available = ", ".join(ftp_listdir(modified_url)[:15]) | ||
| # we list only 15 as maximum | ||
| log.warning(f"List of files : ({files_available})") |
There was a problem hiding this comment.
ça serait plus simple et lisible d'avoir qu'un seul log
There was a problem hiding this comment.
Updates here : a72fc9c
Sanix-Darker
force-pushed
the
fix/connection-dropped
branch
from
36f76d4 to
d8118be
Compare
but for each ftp-client it was defined at 3s, Setting it to 10s is too small when we have a long list of files that could takes too much time to resolve, the value should be increased, in this commit, am tripling the default fomr FTPS class that was at 60s to 180s. Later on, we should find a way to set a FF that will forward that extremly HUGE value to the lib itself.
Sanix-Darker
force-pushed
the
fix/connection-dropped
branch
from
8fa1ca9 to
e874a86
Compare
| _DEFAULT_MAX_TIMEOUT_SECONDS = 30 | ||
| _DEFAULT_MAX_RETRY = 7 |
There was a problem hiding this comment.
tu augmente quand même le timeout et les retry?
There was a problem hiding this comment.
le retry etait 4
le timeout etait 3
Tu suggeres que je remettes ces valeurs ?
Je me disais que ces valeurs ne sont pas deconantes non plus.
et si tu regardes la valeur du timeout par defaut de la calsse FTP, il etait a 60s : https://github.com/toucantoco/peakina/blob/main/peakina/io/ftp/ftp_utils.py#L27
| ssh_client.close() | ||
| # In cae of Exception, we don't want to raise it | ||
| with suppress(AttributeError): | ||
| ssh_client.close() |
There was a problem hiding this comment.
je loggerait quand même le cas ou on arrive pas à close du coup
WHAT
connection droppedon remote file (SFTP).This is not a "new issue", my proposition on this PR is to increase the timeout, the retry and
suppress the .close() error and let see on live prod how it behave.
HOW