Exit 1 with "FAILED" and no explanation after forced fail test

[WillGibson]

I thought it might be because I have skipped a few tests for now because they don't play ball when running with mutmut, but I commented them out and the result was the same.

Should this line be...

if runner.run_forced_fail() != 0

It's not obvious to me what the purpose or expected behaviour of the forced failed run is and it doesn't seem to be mentioned in the documentation.

I don't think our code resets os.environ anywhere.

Steps to replicate

# Setup
git clone [email protected]:uktrade/platform-tools.git
git checkout 3562d91
poetry install

# Run
rm -rf mutants # In case you already have stuff in the mutants directory
poetry run mutmut run

Expected behaviour

Mutations are tested

Actual behaviour

Exit code: 1 ➜ rm -rf mutants && poetry run mutmut run
...
Running forced fail test
...
=== 609 passed, 4 skipped in 23.40s ===
    exit code 0
FAILED
Exit code: 1 ➜ 

[boxed]

Forced fail is a quick check to see if the mutants are being run. It's quite easy to get into a situation where the original code is first in the import list and then all mutants will survive. Which might be what happens in your case. During forced fail, ideally only one test should be run and that should immediately fail.

[WillGibson]

I just updated the commit hash in the steps to replicate, it was the wrong one.

[WillGibson]

Hi @boxed. I've forked our repository and brought this little challenge on holiday with me for the bits where I just want to stay out of the midday sun ;-)

If you were to run the following commands, the problem manifests fairly quickly...

# Setup
git clone [email protected]:WillGibson/platform-tools.git
git checkout add-mutation-testing
poetry install

# Run
rm -rf mutants # In case you already have stuff in the mutants directory
poetry run mutmut run

To keep the blast radius small, I think I have configured it so that the only test file in play is tests/platform_helper/providers/test_ecs.py and the only file that mutmut is trying to mutate is dbt_platform_helper/providers/ecs.py.

I am not sure what you mean by "It's quite easy to get into a situation where the original code is first in the import list and then all mutants will survive.".

I'm hoping you can have a look and explain further?

[boxed]

I am not sure what you mean by "It's quite easy to get into a situation where the original code is first in the import list and then all mutants will survive.".

I meant when developing mutmut. I have many times broken mutmut in some subtle way and suddenly it executes the original code when it should be running the mutated code. Any config done by users could also have this effect as it's very sensitive to changes for python path.

Writing mutmut 3.0 was quite a pain due to this brittleness. Mutmut 2 was much more robust in this way, but it was also extremely slow so I think it's worth it...

[WillGibson]

I would like to get to the bottom of it. So that we can use Mutmut 3, which as you say is fast, and to help make it easier for others to start using it.

[boxed]

Help is greatly appreciated. I have very limited time with family and working at a startup...

[WillGibson]

Are there instructions anywhere for working on Mutmut, similar to a CONTRIBUTING.md.

I would like to fork Mutmut then run my local copy of that against our offending code to see if that helps me figure it out.

[boxed]

No there is no such documentation.

[WillGibson]

How do you do it?

Sketchy instructions are fine, I will tidy them up as a start to a contributing page.

[boxed]

I read the code, and I know how it's supposed to work because I wrote it all :P

The basic idea is using "mutant schemata", which is an industry term for generating all the mutants up front and renames the original function. Then it replaces the original functions with a "trampoline", which will call either the original function or a mutant depending on some command for which mutant should be active. In this case this is controlled by an environment variable.

The execution model is based on fork, which gives pretty strong isolation guarantees, and is super fast to avoid a lot of overhead in starting a new process and importing everything.

There's also some code that detects which tests hit which functions, so only relevant tests are run, which can cut execution time with a lot.

[WillGibson]

I've made a small pull request to get myself started.

Next I will move onto creating an initial CONTRIBUTING.rst.

Then I will try and figure out how to make Mutmut work with the https://github.com/uktrade/platform-tools tests and hopefully make Mutmut handle it them as they are. If I can't do that, I will at least document what is needs to be changed in the codebase under test.

[WillGibson]

Pull request to add initial contributing documentation.

[Julien-Delavisse]

Hello !

I seem to have this problem too.

I can't get mutmut >= 3.0.0 to work.

I made a mini repo with a single function and a single test here: https://github.com/Julien-Delavisse/mutmut-v3-tests

you can do a git clone https://github.com/Julien-Delavisse/mutmut-v3-tests && pip install pytest mutmut && pytest && mutmut run to see the initial problem.

I thought the problem might come from a particular version of python or mutmut, so I made a little script to test mutmut against its version and the version of python, all isolated in Docker.

It turns out that mutmut doesn't work on this very simple repo with version >= 3.0.0, so the python version doesn't make any difference.

You can run the ./test.sh script if you like, but I've left the results.log log file to share the results quickly.

I took a quick look at the mutmut source code, wait and see 😅

[boxed]

The common thread here seems to be to by src style layouts. Something I don't personally use in my projects.

[boxed]

@Julien-Delavisse Thanks for this reproduction.

[boxed]

@Julien-Delavisse I looked into your project, but I'm afraid your reproduction isn't very useful. You do import src.hello_world, but that's not how you're supposed to do that import in a src-type layout. If you change it to the correct import hello_world, mutmut runs as it should. The mistake is in your project, not in mutmut.

[Julien-Delavisse]

Hi @boxed !

Thanks already for your very useful project 😉

I modified my project to go in your direction, but it does not change my results that mutmut >= 3.0.0 does not work (but that mutmut < 3.0.0 works).

What modifications have you made to make this example project work?

[boxed]

Ah. I'm running from the main branch, not the latest released version. If you do that your simple project works, but I believe not the other projects in this issue.

But please, anyone affected, do try with the latest code from master.

[Julien-Delavisse]

I made a git clone https://github.com/boxed/mutmut && pip install -e ./mutmut (commit 9f4b070) and now I have this error like others people :

I'm curious about the different results between us. Do you have any ideas?

⠋ Generating mutantssrc/helloworld.py
     also copying tests
     also copying test
     also copying tests.py
     also copying setup.cfg
     also copying pyproject.toml

    done in 1ms
⠙ Listing all tests python -m pytest  -vv -x -q --collect-only --rootdir=.
======================================================== test session starts =========================================================
platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0 -- /home/user/.venvs/mutmut/bin/python3.12
cachedir: .pytest_cache
rootdir: /home/user/repositories/live/github.com/Julien-Delavisse/mutmut-v3-tests/mutants
configfile: pyproject.toml
testpaths: tests
collected 1 item                                                                                                                     

<Dir mutants>
  <Dir tests>
    <Module test_helloworld.py>
      <Function test_hello_world>

===================================================== 1 test collected in 0.00s ======================================================
    exit code 0

⠹ Running clean testspython -m pytest  -vv -x -q --import-mode=append --rootdir=.
======================================================== test session starts =========================================================
platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0 -- /home/user/.venvs/mutmut/bin/python3.12
cachedir: .pytest_cache
rootdir: /home/user/repositories/live/github.com/Julien-Delavisse/mutmut-v3-tests/mutants
configfile: pyproject.toml
testpaths: tests
collected 1 item                                                                                                                     

tests/test_helloworld.py::test_hello_world PASSED                                                                              [100%]

========================================================= 1 passed in 0.00s ==========================================================
    exit code 0

    done
⠸ Running forced fail testpython -m pytest  -vv -x -q --import-mode=append --rootdir=.
======================================================== test session starts =========================================================
platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0 -- /home/user/.venvs/mutmut/bin/python3.12
cachedir: .pytest_cache
rootdir: /home/user/repositories/live/github.com/Julien-Delavisse/mutmut-v3-tests/mutants
configfile: pyproject.toml
testpaths: tests
collected 1 item                                                                                                                     

tests/test_helloworld.py::test_hello_world FAILED                                                                              [100%]

============================================================== FAILURES ==============================================================
__________________________________________________________ test_hello_world __________________________________________________________

    def test_hello_world():
>       assert helloworld.hello_world() == 1

tests/test_helloworld.py:4: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/helloworld.py:56: in hello_world
    result = _mutmut_trampoline(x_hello_world__mutmut_orig, x_hello_world__mutmut_mutants, *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

orig = <function x_hello_world__mutmut_orig at 0x73f07c92d760>
mutants = {'x_hello_world__mutmut_1': <function x_hello_world__mutmut_1 at 0x73f07c92d800>}, args = (), kwargs = {}
os = <module 'os' (frozen)>, mutant_under_test = 'fail'
MutmutProgrammaticFailException = <class 'mutmut.__main__.MutmutProgrammaticFailException'>

    def _mutmut_trampoline(orig, mutants, *args, **kwargs):
        import os
        mutant_under_test = os.environ['MUTANT_UNDER_TEST']
        if mutant_under_test == 'fail':
            from mutmut.__main__ import MutmutProgrammaticFailException
>           raise MutmutProgrammaticFailException('Failed programmatically')
E           mutmut.__main__.MutmutProgrammaticFailException: Failed programmatically

src/helloworld.py:9: MutmutProgrammaticFailException
====================================================== short test summary info =======================================================
FAILED tests/test_helloworld.py::test_hello_world - mutmut.__main__.MutmutProgrammaticFailException: Failed programmatically
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================================= 1 failed in 0.03s ==========================================================
    exit code 1

    done
Running mutation testing
⠴ 1/1  🎉 0 🫥 1  ⏰ 0  🤔 0  🙁 0  🔇 0
0.00 mutations/second

[WillGibson]

I am still experiencing the same issue with the code/branch where I'm trying to get Mutmut working with our codebase.