Detect and merge duplicate items

[qwertea]

(Another non bug, I think; feel free to close if this is out of scope.)

BBC Science & other BBC feeds seem to randomly duplicate items¹, and it's annoying to see a duplicate of an item you already read marked as new. It would be nice if duplicates were automatically detected and pruned/merged.

feed: https://feeds.bbci.co.uk/news/science_and_environment/rss.xml
archived feed in case the duplicate is removed later: https://files.catbox.moe/p76r5v.rss

---

1. At least, that's what I think is going on here.

[jocmp]

Thanks for the info @qwertea. I have noticed this with a few feeds. In this case the feed has the same link twice, example: <link>https://www.bbc.com/news/articles/cvgeydkz08go</link>. While the link is the same, the guid is just slightly different between the two. Note the different between the extra hash at the end.

- https://www.bbc.com/news/articles/cvgeydkz08go#0
+ https://www.bbc.com/news/articles/cvgeydkz08go#3

I think In this case I can skip duplicates if the link is the same between multiple items.