Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker artifacts tests is quite inefficient - takes a lot of time to try and fetch non-existants logs #9532

Open
mykaul opened this issue Dec 11, 2024 · 1 comment

Comments

@mykaul
Copy link
Contributor

mykaul commented Dec 11, 2024

(perhaps it's not a SCT issue?)
See https://jenkins.scylladb.com/job/releng-testing/job/artifacts/job/artifacts-docker-test/11/consoleFull for example:

00:10:03.784  Start collect logs for cluster loader-set
00:10:03.784  No nodes found for loader-set cluster. Logs will not be collected
00:10:03.784  There are no logs collected for loader-set
00:10:03.784  Start collect logs for cluster monitor-set
00:10:03.784  Collecting logs on host: monitor-node-b588aee0-2af6-496e-86da-b932c20894b9-0
00:10:39.311  'request': Number of retries exceeded!
00:10:49.550  Create prometheus snapshot [try #1]
00:11:25.258  'request': Number of retries exceeded!
00:11:35.455  Create prometheus snapshot [try #2]
00:12:11.075  'request': Number of retries exceeded!
00:12:19.534  'create_prometheus_snapshot': Number of retries exceeded!
00:12:19.534  Create prometheus snapshot failed HTTPConnectionPool(host='127.0.0.1', port=9090): Max retries exceeded with url: /api/v1/status/config (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x77b6e8b8fca0>: Failed to establish a new connection: [Errno 111] Connection refused')).
00:12:19.534  Use prometheus data directory
00:12:19.534  /home/jenkins/sct-monitoring/scylla-monitoring-data
00:12:19.534  Unable to archive log `scylla-monitoring-data' to `/home/jenkins/sct-monitoring/prometheus_data_20241211_044304.tar.gz'
00:12:19.534  Dir with scylla monitoring stack was not found
00:12:19.534  None, None, None
00:12:19.534  Monitoring stack were not collected
00:12:19.535  Dir with scylla monitoring stack was not found
00:12:19.535  Monitoring version was not found

We are wasting >2 minutes doing nothing, trying to collects logs we'll never have in this test.

The whole test takes 13 minutes, I'm sure we can cut it substantially.

CC @yaronkaikov

@mykaul mykaul removed their assignment Dec 11, 2024
@mykaul
Copy link
Contributor Author

mykaul commented Dec 11, 2024

I see that the log collection is part of SCT (

LOGGER.info("Start collect logs for cluster %s", log_collector.cluster_log_type)
)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant