fix: add recursively merge of documents by their parents if they meet the threshold

[davidsbatista]

Related Issues

• fixes #146

Proposed Changes:

• Updated the run() method to have a recursive merging process. It recursively groups documents by their parents and merges them if they meet the threshold, continuing up the hierarchy until no more merges are possible. Previously, this was hard-coded to a 2-level hierarchy

• The function that checks for the input was not being called

How did you test it?

• ran unit tests locally and did manual verification
• CI tests
• added more tests, and bumped test coverage to 93%, previously it was below 80%

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[coveralls]

Pull Request Test Coverage Report for Build 12654383306

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.4%) to 83.545%

---

Totals
Change from base Build 12624157864:	0.4%
Covered Lines:	2041
Relevant Lines:	2443

---

💛 - Coveralls

[julian-risch]

I have one change request about the try_merge_level. Other than that the PR looks good to me.
There are cases where we check multiple times whether we can merge the same child_docs over and over in try_merge_level.
Imagine we start try_merge_level with 4 documents.
2 documents from a parent A and two from a parent B. Let's say when we calculate merge scores, documents from parent A get merged. documents from parent B don't get merged.
Now, there were some merges made, so we're not done yet. We go into the recursion and check the documents for B again. Let's prevent this from happening. I suggest we change two lines for this but haven't tried it. Please check or maybe you have another idea.

[julian-risch]

The class docstring also needs to be updated.
When I run the example code, i get three levels of documents now.

from haystack import Document
from haystack_experimental.components.splitters import HierarchicalDocumentSplitter
from haystack_experimental.components.retrievers.auto_merging_retriever import AutoMergingRetriever
from haystack.document_stores.in_memory import InMemoryDocumentStore

# create a hierarchical document structure with 2 levels, where the parent document has 3 children
text = "The sun rose early in the morning. It cast a warm glow over the trees. Birds began to sing."
original_document = Document(content=text)
builder = HierarchicalDocumentSplitter(block_sizes=[10, 3], split_overlap=0, split_by="word")
docs = builder.run([original_document])["documents"]


>>> len([d for d in docs if d.meta["__level"]==0])
1
>>> len([d for d in docs if d.meta["__level"]==1])
2
>>> len([d for d in docs if d.meta["__level"]==2])
7

which contradicts the comment # create a hierarchical document structure with 2 levels, where the parent document has 3 children

[davidsbatista]

I wrote it not counting the tree's root, the original document, which is always present - but we should count it as a level.

[julian-risch]

Found one more line that needs to be changed in my opinion. As all tests pass already, I would suggest adding a test to cover that change too.

[julian-risch]

Thanks for the quick call! Looks good to me now!