feat(loaders): document loader pdf

[Tachikoma000]

Add PDF Loader to Document Loaders

This PR implements a PDF loader as part of the document loaders module in Rig. It allows users to easily load and process PDF documents for use in RAG systems and other document processing tasks.

Changes

• Implemented PdfLoader struct in src/document_loaders/pdf.rs
• Added PdfLoader to the document_loaders module
• Implemented DocumentLoader trait for PdfLoader
• Used the lopdf crate for PDF parsing
• Added error handling for file operations and PDF parsing
• Updated Cargo.toml with the lopdf dependency
• Added unit tests for PdfLoader
• Updated documentation with PdfLoader usage examples

Implementation Details

The PdfLoader uses the lopdf crate to parse PDF files and extract text content. It handles potential errors such as file not found or parsing errors. The extracted text is converted into DocumentEmbeddings for further processing in Rig.

Testing

Unit tests have been added to ensure the PdfLoader correctly loads PDF files and handles various edge cases. The tests cover:

• Loading a valid PDF file
• Handling a non-existent file
• Processing a PDF with multiple pages
• Dealing with empty PDF files

Documentation

Code files are commented and some docstrings added

Related Issue

Closes #24

Checklist

• Code follows the project's coding style
• Tests have been added and all tests pass
• Documentation has been updated
• Commit messages are clear and descriptive
• Changes have been reviewed for potential performance impacts

Additional Notes

This implementation focuses on text extraction from PDFs. Future enhancements could include handling PDFs with complex layouts or embedded images.

[0xMochan]

I haven't done a formal review but I noticed a couple of things when I looked over this!

• This PR adds a lot of dependencies. We probably want to throw this as a cargo feature on the main rig library so that not all users download these deps if they install the library.
  • Also, tokio likely shouldn't be one of the dependencies since the library can work with multiple async backends.
  • Perhaps another route would be for a new crate rig-pdf (or rig-loaders) instead of keeping this in the main library would be more suitable.
• It seems like there is also a CSV loader but I don't really see that documented in the PR.
• There's some code in the mod.rs file which I think we are trying to avoid. It could be spun out into a document.rs file.
• There shouldn't be any println code in our library to avoid erroneous output when people use our library. It should be using the tracing module so that logging can be turned off and on at will.
• I think some of the Error typing could be simplified using thiserror types for the return values.

These are just some first thought comments, I think stopher will do a main review of this. Looking forward to trying this out locally!

[cvauclair]

This is being (re-)implemented in #55