since running all mutations even if just a couple of source files are changed

[steoj]

We are using Stryker.NET (and Stryker JS) in a fairly large application, where the backend C#/.NET part involves approximately 500 files, 20 000 lines of code and ballpark 1300 xUnit tests. Running a full stryker.NET analysis on this takes about 8 hours on a MacBook pro M3.
Obviously, waiting this long for (otherwise exelent) feedback on the quality of our tests is not ideal. Trying to run any of thits in a github workflow times out on the 6hr limit.

We are trying to use the since feature, with reference to the last commit in main, hoping for Stryker to run only on the *.cs files changed since main, but Stryker.NET insists on running all mutations because a non-cs file is changed:

10819 mutants will be tested because: Non-CSharp files in test project were changed

Command line for running stryker:

dotnet stryker --since:$latest_commit --config-file:stryker-since.json --output report

Config file:

{
  "$schema": "https://json.schemastore.org/stryker-config",
  "stryker-config": {
    "mutation-level": "Advanced",
    "additional-timeout": 5000,
    "coverage-analysis": "perTest",
    "verbosity": "info",
    "reporters": [
      "Progress",
      "Html",
      "cleartext",
      "json",
      "markdown"
    ],
    "solution": "backend.sln",
    "report-file-name": "mutation-report",
    "break-on-initial-test-failure": false,
    "project-info": {
      "name": "github.com/<org>/spts-app",
      "version": "main"
    },
    "mutate": [
      "**/*.cs",
      "!**/Spts.Test/",
      "!**/*Configuration.cs"
    ],
    "since": {
      "enabled": true,
      "target": "main",
      "ignore-changes-in": [
        "**/Spts.Test/"
      ]
    }
  }
}

Any config settings we are forgetting?

[dupdob]

hi
what problem are you trying to resolve here?
For clarification, Stryker re-runs every mutations when a non source file is changed because Stryker cannot identify what depends on said file(s) and thus what actually needs to be tested again.

That being said, what is your intent using since? I.e what are you trying to achieve?

[steoj]

Thanks for the request for clarification.
The real underlying challenge here is to give the developer a timely response on the quality of the tests they are writing.
To do this, I need to run stryker in a pipeline. In our case this is github workflows. As a stryker run using since (the changed files since a reference branch - main) in this case causes 6 checkers and 5 runners on a Mac /M3 to work for 7-8 hours to get through all mutants, a github workflow attempting to do the same with one checker an one runner will time out every time after 6 hours.

What I really need is feedback on the changed code only. Ideally only on the lines that have been changed as a compromise to having a full report on all files when a non-cs file is changed.

One approach is to script the diff files and pass them to the stryker run as mutate config like:

dotnet stryker --config-file:stryker-diff.json -m file1.cs -m file2.cs --output report

A similar approach for the Stryker JS in a github workflow, completes in 12-15 (depending on the amount of chnanges) minutes which is acceptable.

Stryker.net mutates whole files and even changing two files only, needs about an hour to run.

Hope this clarifies my objective. Any suggestions on what to do?

[dupdob]

Thanks. As of today, there is no feature allowing to pick specific source files to be mutated/tested again. The priority for Stryker.Net is report accuracy over speed, that is why it often default to pessimistic view.
Do you know which non cs file(s) is/are changed and triggered a full retest? As of today, there is no way to completely ignore some file(s), but adding this feature could make sense. Note that it would be at the expense of correctness, as explained earlier.
But again, this would be logical in a workflow when you want to get a quick assessment.

Alternatively, I suggest dropping the additional timeout settings, it will speed up things a bit.
You could also consider using a lower mutation level for a quick assessment, and use advanced when you need/want a global assessment on the code base.

[steoj]

We use the same repo for both backend and frontend. Any .ts file (or .yaml, .json, .tsx etc) will trigger a full run it seems.
Changing the config to include:

    "test-projects": [
      "**/Spts.Test/Spts.Test.csproj"
    ],

and remove the solution reference seems to ignore files not touched by the test project,

Although I can understand the priority of accuracy over speed in most cases, I notice that when the feedback comes too late, it is often ignored. We could force waiting for stryker report on changes but that would greatly reduce our throughput I belive.

[dupdob]

To be specific: any changed non CSharp of Fsharp file located in one of the test projects will trigger a full retest.
Does it match your experience?

You are the second person to signal that since is too slow for PRs. The underlying question is: can we improve since or is a new option required?

[rouke-broersma]

I think the main issue is that the ignore-changes-in glob provided by the user is incomplete.

Glob according to config file:

Glob that would actually target files as described:

See: https://globster.xyz

[steoj]

What is sort of working for us is running this is a workflow for pull-requests only:

  - steps:
      - name: Get changes
        run: |
          git fetch origin main
          base_commit=$(git rev-parse origin/main)
          changed_files=$(git diff --name-only $base_commit HEAD -- "**/*.cs")

          formatted_files=""
          for file in $changed_files; do
              # Ignore any file under backend/Spts.Test
              if [[ "$file" != backend/Spts.Test/* ]]; then
                  # Extract just the file name and prepend '**/'
                  filename=$(basename "$file")
                  formatted_files="$formatted_files -m **/$filename"
              fi
          done

          echo "CHANGED_FILES=$formatted_files" >> $GITHUB_ENV
        shell: /usr/bin/bash -e {0}

      - name: Run stryker.net
        run: |
          dotnet stryker --config-file:stryker-diff.json ${{ env.CHANGED_FILES }} --output report

When run in a pull-request producing 4 filenames changed, it finishes in about one hour.
I actually had to increase the additional timeout to make the majority of the tests not time out, but reducing the mutation level might be a good idea.
Really would be a good idea to filter on lines as well.

[rouke-broersma]

We are open to improvements to since mode. Back when this feature was introduced we did not have line change information from the library we were using for git. I believe since one of our contributors has made line change information available in the library, but they have not had the time to actually use this information in stryker.