Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Published reports incomplete when running all tests at one #23

Open
sysarch-repo opened this issue Aug 9, 2024 · 2 comments
Open

Published reports incomplete when running all tests at one #23

sysarch-repo opened this issue Aug 9, 2024 · 2 comments

Comments

@sysarch-repo
Copy link

sysarch-repo commented Aug 9, 2024 •
edited
Loading

Acceptance tests may comprise of hundred test cases and it therefore is more convenient to run them as one Jenkins job. This can be achieved by adding "all" to the test list, i.e.:

ubuntu@xt1:~/xtest/hello-world$ cat site.yml 
---
- hosts:
    - 127.0.0.1
  roles:
    - role: nti.xtesting
      project: hello_world
      registry_deploy: true
      repo: 127.0.0.1
      dport: 5000
      suites:
        - container: hello-world-tests
          tests:
            - all

The console output in Jenkins is showing the following expected result:

Started by upstream project "[hello_world-latest-daily](http://127.0.0.1:8080/job/hello_world-latest-daily/)" build number [11](http://127.0.0.1:8080/job/hello_world-latest-daily/11)
originally caused by:
 Started by user [admin](http://127.0.0.1:8080/user/admin)
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building in workspace /var/jenkins_home/workspace/hello_world-127_0_0_1-hello-world-tests-latest-all
[hello_world-127_0_0_1-hello-world-tests-latest-all] $ /bin/sh -xe /tmp/jenkins12684654906665076593.sh
+ set +x
2024-08-09 17:46:58,397 - xtesting.ci.run_tests - INFO - Deployment description:

+-------------------------+-----------------------------------------------------------+
|         ENV VAR         |                           VALUE                           |
+-------------------------+-----------------------------------------------------------+
|         CI_LOOP         |                           daily                           |
|          DEBUG          |                           false                           |
|     DEPLOY_SCENARIO     |                  os-nosdn-nofeature-noha                  |
|      INSTALLER_TYPE     |                          unknown                          |
|        BUILD_TAG        |                        RTE2QIGHRW2J                       |
|        NODE_NAME        |                        hello_world                        |
|       TEST_DB_URL       |            http://10.0.3.15:8000/api/v1/results           |
|     TEST_DB_EXT_URL     |            http://10.0.3.15:8000/api/v1/results           |
|     S3_ENDPOINT_URL     |                   http://10.0.3.15:9000/                   |
|        S3_DST_URL       |     s3://xtesting/RTE2QIGHRW2J/hello_world-127_0_0_1-     |
|                         |               hello-world-tests-latest-all-3              |
|       HTTP_DST_URL      |      http://10.0.3.15:8181/RTE2QIGHRW2J/hello_world-      |
|                         |          127_0_0_1-hello-world-tests-latest-all-3         |
+-------------------------+-----------------------------------------------------------+

2024-08-09 17:46:58,413 - xtesting.ci.run_tests - INFO - TESTS TO BE EXECUTED:

+---------------------+-------------------------------------------+---------------------+
|        TIERS        |                DESCRIPTION                |      TESTCASES      |
+---------------------+-------------------------------------------+---------------------+
|     Basic tests     |     This is hello world basic testing     |     Test1 Test2     |
+---------------------+-------------------------------------------+---------------------+

2024-08-09 17:46:58,429 - xtesting.ci.run_tests - INFO - Loading test case 'Test1'...
2024-08-09 17:46:58,477 - xtesting.ci.run_tests - INFO - Running test case 'Test1'...
Hello world!
2024-08-09 17:46:58,477 - xtesting.ci.run_tests - INFO - Test result:

+-------------------+---------------------+------------------+----------------+
|     TEST CASE     |       PROJECT       |     DURATION     |     RESULT     |
+-------------------+---------------------+------------------+----------------+
|       Test1       |     hello_world     |      00:00       |      PASS      |
+-------------------+---------------------+------------------+----------------+

2024-08-09 17:46:58,822 - xtesting.core.testcase - INFO - All artifacts were successfully published: 

http://10.0.3.15:8181/RTE2QIGHRW2J/hello_world-127_0_0_1-hello-world-tests-latest-all-3/xtesting.log

2024-08-09 17:46:58,844 - xtesting.core.testcase - INFO - The results were successfully pushed to DB: 

http://10.0.3.15:8000/api/v1/results/66b65612f58ff39d3d7ddb7a

2024-08-09 17:46:58,861 - xtesting.ci.run_tests - INFO - Loading test case 'Test2'...
2024-08-09 17:46:58,873 - xtesting.ci.run_tests - INFO - Running test case 'Test2'...
Hello world!
2024-08-09 17:46:58,873 - xtesting.ci.run_tests - INFO - Test result:

+-------------------+---------------------+------------------+----------------+
|     TEST CASE     |       PROJECT       |     DURATION     |     RESULT     |
+-------------------+---------------------+------------------+----------------+
|       Test2       |     hello_world     |      00:00       |      PASS      |
+-------------------+---------------------+------------------+----------------+

2024-08-09 17:46:58,924 - xtesting.core.testcase - INFO - All artifacts were successfully published: 

http://10.0.3.15:8181/RTE2QIGHRW2J/hello_world-127_0_0_1-hello-world-tests-latest-all-3/xtesting.log

2024-08-09 17:46:58,940 - xtesting.core.testcase - INFO - The results were successfully pushed to DB: 

http://10.0.3.15:8000/api/v1/results/66b65612f58ff39d3d7ddb7b

2024-08-09 17:46:58,945 - xtesting.ci.run_tests - INFO - Xtesting report:

+-------------------+---------------------+---------------------+------------------+----------------+
|     TEST CASE     |       PROJECT       |         TIER        |     DURATION     |     RESULT     |
+-------------------+---------------------+---------------------+------------------+----------------+
|       Test1       |     hello_world     |     Basic tests     |      00:00       |      PASS      |
|       Test2       |     hello_world     |     Basic tests     |      00:00       |      PASS      |
+-------------------+---------------------+---------------------+------------------+----------------+

2024-08-09 17:46:58,948 - xtesting.ci.run_tests - INFO - Execution exit value: Result.EX_OK
Finished: SUCCESS

The JSON files accessible via

  • http://10.0.3.15:8000/api/v1/results/66b65612f58ff39d3d7ddb7a
  • http://10.0.3.15:8000/api/v1/results/66b65612f58ff39d3d7ddb7b
    show test individual information (which is ok), example for Test2:
    {"project_name": "hello_world", "scenario": "os-nosdn-nofeature-noha", "stop_date": "2024-08-09 17:46:58", "case_name": "Test2", "build_tag": "RTE2QIGHRW2J", "version": "unknown", "pod_name": "hello_world", "criteria": "PASS", "installer": "unknown", "_id": "66b65612f58ff39d3d7ddb7b", "start_date": "2024-08-09 17:46:58", "details": {"links": ["http://10.0.3.15:8181/RTE2QIGHRW2J/hello_world-127_0_0_1-hello-world-tests-latest-all-3/xtesting.log"]}}

But the published log file accessible via

2024-08-09 17:46:58,397 - xtesting.ci.run_tests - INFO - Deployment description:

+-------------------------+-----------------------------------------------------------+
|         ENV VAR         |                           VALUE                           |
+-------------------------+-----------------------------------------------------------+
|         CI_LOOP         |                           daily                           |
|          DEBUG          |                           false                           |
|     DEPLOY_SCENARIO     |                  os-nosdn-nofeature-noha                  |
|      INSTALLER_TYPE     |                          unknown                          |
|        BUILD_TAG        |                        RTE2QIGHRW2J                       |
|        NODE_NAME        |                        hello_world                        |
|       TEST_DB_URL       |            http://10.0.3.15:8000/api/v1/results           |
|     TEST_DB_EXT_URL     |            http://10.0.3.15:8000/api/v1/results           |
|     S3_ENDPOINT_URL     |                   http://10.0.3.15:9000                   |
|        S3_DST_URL       |     s3://xtesting/RTE2QIGHRW2J/hello_world-127_0_0_1-     |
|                         |               hello-world-tests-latest-all-3              |
|       HTTP_DST_URL      |      http://10.0.3.15:8181/RTE2QIGHRW2J/hello_world-      |
|                         |          127_0_0_1-hello-world-tests-latest-all-3         |
+-------------------------+-----------------------------------------------------------+

2024-08-09 17:46:58,413 - xtesting.ci.run_tests - INFO - TESTS TO BE EXECUTED:

+---------------------+-------------------------------------------+---------------------+
|        TIERS        |                DESCRIPTION                |      TESTCASES      |
+---------------------+-------------------------------------------+---------------------+
|     Basic tests     |     This is hello world basic testing     |     Test1 Test2     |
+---------------------+-------------------------------------------+---------------------+

2024-08-09 17:46:58,429 - xtesting.ci.run_tests - INFO - Loading test case 'Test1'...
2024-08-09 17:46:58,477 - xtesting.ci.run_tests - INFO - Running test case 'Test1'...
2024-08-09 17:46:58,477 - xtesting.ci.run_tests - INFO - Test result:

+-------------------+---------------------+------------------+----------------+
|     TEST CASE     |       PROJECT       |     DURATION     |     RESULT     |
+-------------------+---------------------+------------------+----------------+
|       Test1       |     hello_world     |      00:00       |      PASS      |
+-------------------+---------------------+------------------+----------------+

2024-08-09 17:46:58,822 - xtesting.core.testcase - INFO - All artifacts were successfully published: 

http://10.0.3.15:8181/RTE2QIGHRW2J/hello_world-127_0_0_1-hello-world-tests-latest-all-3/xtesting.log

2024-08-09 17:46:58,844 - xtesting.core.testcase - INFO - The results were successfully pushed to DB: 

http://10.0.3.15:8000/api/v1/results/66b65612f58ff39d3d7ddb7a

2024-08-09 17:46:58,861 - xtesting.ci.run_tests - INFO - Loading test case 'Test2'...
2024-08-09 17:46:58,873 - xtesting.ci.run_tests - INFO - Running test case 'Test2'...
2024-08-09 17:46:58,873 - xtesting.ci.run_tests - INFO - Test result:

+-------------------+---------------------+------------------+----------------+
|     TEST CASE     |       PROJECT       |     DURATION     |     RESULT     |
+-------------------+---------------------+------------------+----------------+
|       Test2       |     hello_world     |      00:00       |      PASS      |
+-------------------+---------------------+------------------+----------------+

The expectation is that when running all tests at once, the log file should be generated and published once, after all tests have been executed.
@collivier
Copy link
Owner

Thank you @sysarch-repo for your feedbacks.

I'm not clear about your benefits here. By calling with "all", xtesting basically schedules all the tests one by one sequentially. If you list them all in the yaml files, Jenkins or GitLab CI/CD is rather in charge of the schedulling which allows you more flexibility especially to run them in parallel.

If I understand correctly, your request would be not to create 1 job per testcase but a single 'all' job ; the test case publications (DB and S3) would remain unchanged. That would result in a bit of extra logic in jenkins job template.

I must also check if it would work with a mutiple container use.

@sysarch-repo
Copy link
Author

sysarch-repo commented Sep 18, 2024 •
edited
Loading

@collivier
thank you for your comments.
I believe the issue is using the same URL
http://10.0.3.15:8181/RTE2QIGHRW2J/hello_world-127_0_0_1-hello-world-tests-latest-all-3/xtesting.log
across the individual tests. This probably creates a race condition resulting in the truncated log file.
I further assume this root cause is found in the xtesting Python library rather than in the CI framework.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@collivier @sysarch-repo