Add mergiraf to the evaluated tools

[wetneb]

Thank you so much for this huge evaluation effort. I haven't managed to run it entirely myself because of some mismatching dependencies on my end, but I have made a little script which should (hopefully) add mergiraf to the evaluated tools. I have tested the script interactively and it seems to respect the contract, but I am not sure if I have done everything that is needed to evaluate it alongside the other tools.

[mernst]

Thank you for this patch. I just learned of mergiraf a week ago, and I had put "add mergiraf" to our to-do list. I appreciate your help in doing that.

You are right that the full pipeline is challenging to run. I'm sorry about that.

This patch looks great overall. I have a few comments for you.

[mernst]

I have fixed a few things. Now, a new "expected" value for the merge results must be committed. (Tests are currently failing because there is no "expected" value for Mergiraf results.)

[benedikt-schesch]

The results have been posted to this branch and visible in results/combined/tables/all/table_summary.tex. It looks very promising, unhandled merges go from 51% to 37% of merges compared to git but the number of incorrect merges go from 3% to 6%. At least the tool is already much better than Spork and Intellimerge but it would be great to look in the results file in results/combined/result_adjusted.csv and look for merges where the tool gives an incorrect result but git does not. With src/python/replay_merge.py you can replay each merge to see the differences.

[benedikt-schesch]

Small test is still failling which is a bit concerning, I updated the results and on my local machine it passes but in the Github CI/CD it seems to fail. It fails only because the hash of the commit is different which makes me suspect that your tool might not be fully platform agnostic i.e. the output of the tool might be slightly different on different platforms like for example bash sorting functions. Do you think that could be the case?

[wetneb]

Thanks a lot for this preliminary report! That's a good question, I don't know Rust well enough to know for sure. It could be that there are some differences in hashing, or line endings (CRLF/LF). The Git version is also likely to make a difference. I don't know how doable it would be to extract an example merge scenario which would behave differently on the two architectures - probably not that straightforward?

[benedikt-schesch]

I think git is fine with this, would be very concerning if not. I will try to isolate which one of the merges is affected by this.

[benedikt-schesch]

It seems that all the merges in the pedrovgs/Algorithms repository (as seen in the CI/CD pipeline) have a hash different from the expected one. This discrepancy occurs for all merges that start with index 2 in that repository. This makes me suspect that some character or other element in the repository is not being handled properly by mergiraf.

To reproduce the issue, you could either:

1. Find a machine where the test fails (if you have a MacBook, it often shows different behavior), or
2. Use act to locally run the GitHub Actions CI/CD pipeline and investigate why the merge produces a different hash.

I’ve encountered this problem before, and here’s the general debugging workflow I followed:

Steps to Diagnose:

1. Keep the Merge Workspaces
   Set DELETE_WORKDIRS to False in this file. (Ideally, this should be a flag, not a variable.) This will ensure the merges are not cleaned up after testing.

2. Run the Small Test
   Execute make small-test to reproduce the behavior.

3. Test on Different Machines or CI/CD with act
   Run the test on:

   • A passing machine and a failing machine, or
   • Your local machine using act or directly through CI/CD.

4. Analyze the Hash Mismatch
   On the problematic merge, run the following command:

   sha256sum <(export LC_ALL=C; export LC_COLLATE=C; cd \
   <local_repo_path>; \
   find . -type f -not -path '*/\\.git*' -exec sha256sum {} \; | sort)

   Replace <local_repo_path> with the actual path to the local repository.

   Compare the hash values between a passing and a failing machine:

   • Use diff to find differences between the two sets of merge outputs.
   • Alternatively, identify which specific file is causing the hash mismatch by applying the same command to individual files.

Typically, the source of the issue becomes apparent from the diff. For reference, my repository for running GitHub Actions locally is nektos/act.