Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get clearer hints what to do, if iterations/repetitions were reduced automatically #195

Open
mawHBT opened this issue Oct 10, 2022 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@mawHBT
Copy link
Collaborator

mawHBT commented Oct 10, 2022

What feature do you want to see added?

If a VM-execution exceeds the timeout, iterations/repetitions are automatically reduced for the next VM-execution. But current VM-execution is marked as erroneous. This can be misleading if you don't get a hint, that it is "erroneous" because it exceeded the timeout. e.g. you just set two VMs to execute and each one is marked with error, you maybe will think there is a general problem.
Maybe just increasing timeout or reducing iterations/repetitions would solve your problem. Also you could set more VMs, so iterations/repetitions are reduced after the first executions and you get successful executions afterwards.
The message in case of automatically reduction should mention that, telling you that maybe the "error" is not really one and what you should try next.

Upstream changes

No response

@mawHBT mawHBT added the enhancement New feature or request label Oct 10, 2022
@mawHBT
Copy link
Collaborator Author

mawHBT commented Oct 11, 2022

How should build-state be, if measurements give no results? Right now, e.g. if you set just 2 VMs and measurement-execution will exceed the timeout in each one, you will see 4 VM-runs marked with error. Build-state will be green in this case. This is ok, since no real error occurred. But you will not notice, that measurements did not produce results and that you should reconfigure (increasing VMs or timeout or decreasing repetitions/iterations directly). Maybe build-state should be unstable, if any test produces no measurement-results or even if configuration is changed, so that you will notice? (Should this be an extra issue?)

@DaGeRe DaGeRe self-assigned this Oct 12, 2022
@DaGeRe
Copy link
Contributor

DaGeRe commented Oct 12, 2022

To solve this issue, two main steps need to be executed:

  • Saving the information, how iterations were reduced (in Peass)
  • Displaying them in Peass-CI

I'll take care of this in the next days.

Setting the build state to unstable would not be a good idea from my point of view, since is something that might happen (and that will happen frequently, if you just take a default configuration for all test cases). If you experience this issue frequently, I'd recommend to change the configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants