markdown-link-check gives false positive for SPIE articles

[hugobuddel]

markdown-link-check gives false positive for SPIE articles on https://www.spiedigitallibrary.org since December 21th 2023.

To reproduce, 3 times the same article, chosen for its shot title:

echo "https://dx.doi.org/10.1117/12.2565546" >> README.md
echo "https://www.spiedigitallibrary.org/conference-proceedings-of-spie/11442/2565546/The-use-of-machine-learning-algorithms-for-image-recognition/10.1117/12.2565546.full" >> README.md
echo "https://www.spiedigitallibrary.org/conference-proceedings-of-spie/11442/2565546/The-use-of-machine-learning-algorithms-for-image-recognition/10.1117/12.2565546.full#_=_" >> README.md

markdown-link-check README.md 

Result:

FILE: README.md
  [✖] https://dx.doi.org/10.1117/12.2565546
  [✖] https://www.spiedigitallibrary.org/conference-proceedings-of-spie/11442/2565546/The-use-of-machine-learning-algorithms-for-image-recognition/10.1117/12.2565546.full
  [✖] https://www.spiedigitallibrary.org/conference-proceedings-of-spie/11442/2565546/The-use-of-machine-learning-algorithms-for-image-recognition/10.1117/12.2565546.full#_=

  3 links checked.

  ERROR: 3 dead links found!
  [✖] https://dx.doi.org/10.1117/12.2565546 → Status: 302
  [✖] https://www.spiedigitallibrary.org/conference-proceedings-of-spie/11442/2565546/The-use-of-machine-learning-algorithms-for-image-recognition/10.1117/12.2565546.full → Status: 302
  [✖] https://www.spiedigitallibrary.org/conference-proceedings-of-spie/11442/2565546/The-use-of-machine-learning-algorithms-for-image-recognition/10.1117/12.2565546.full#_= → Status: 302

While the links work fine with curl:

$ curl -v https://dx.doi.org/10.1117/12.2565546
[...]
< HTTP/2 302 
[...]
< location: https://www.spiedigitallibrary.org/conference-proceedings-of-spie/11442/2565546/The-use-of-machine-learning-algorithms-for-image-recognition/10.1117/12.2565546.full
[...]

$ curl -v -o /dev/null https://www.spiedigitallibrary.org/conference-proceedings-of-spie/11442/2565546/The-use-of-machine-learning-algorithms-for-image-recognition/10.1117/12.2565546.full
[...]
< HTTP/2 200

However, the [...]/12.2565546.full does some complicated redirection scheme in firefox (as witnessed by the network monitor), and then ultimately gets redirected again to [...]/12.2565546.full#_=, which then renders.

My hypothesis is that this same redirect scheme occurs with markdown-link-check, and this gets seen as a circular redirection.

This only happened since December 21th 2023, as can be seen in one of our github actions: https://github.com/AstarVienna/ScopeSim/actions/workflows/markdown_link_check.yml