Add all Contentfile metadata to chunk responses

[shanbady]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/6829

Description (What does it do?)

Currently, when retrieving contentfile chunks via the vector content files endpoint, we return the payload that was initially stored with the embedding document - however, for learning resources, since there is a 1-1 relation with what is in qdrant, we pull the serialized resources from the database and render that.

This PR makes it so that the vector contentfile endpoint does the same. What prompted this was the need for having the edx_block_id in the response.

How can this be tested?

1. Checkout this branch
2. make sure you have some contentfiles locally otherwise populate them python manage.py backpopulate_mitxonline_files
3. generate embeddings for contentfiles via python manage.py generate_embeddings --all or python manage.py generate_embeddings --resource-ids <comma separated resource ids>
4. visit your local contentfiles vector endpoint and inspect the response. compare this to the production response - see that there is more info including the edx_block_id and database specific ids.

[gemini-code-assist]

Hello @shanbady, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

This pull request addresses issue #6829, aiming to enhance the vector content files endpoint by including all Contentfile metadata in the chunk responses. Currently, the endpoint returns the initial payload stored with the embedding document. This PR modifies the endpoint to serialize resources from the database, similar to how learning resources are handled, to include additional information like the edx_block_id in the response. The changes primarily involve modifying the _content_file_vector_hits function in vector_search/utils.py to fetch and serialize Contentfile data from the database before returning it.

Highlights

• Feature Enhancement: The primary goal is to include all Contentfile metadata, such as edx_block_id, in the responses from the vector content files endpoint.
• Data Serialization: The PR updates the _content_file_vector_hits function to fetch Contentfile objects from the database using run_readable_id and key and serialize them using ContentFileSerializer.
• Payload Enrichment: The serialized Contentfile data is then added to the original payload before being returned, ensuring that the response includes comprehensive metadata.

Changelog

• vector_search/utils.py
  • Imported ContentFile model and ContentFileSerializer (lines 9-13).
  • Modified _content_file_vector_hits function (lines 373-388) to fetch Contentfile objects from the database based on run_readable_id and key from the search result payloads.
  • Serialized the fetched Contentfile objects using ContentFileSerializer (line 382).
  • Updated the original payload with the serialized data, excluding the 'content' field, before returning the result (lines 381-387).

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

---

A chunk of data, incomplete,
Missing fields, a coder's defeat.
But now with metadata's grace,
Full context shines in its place,
A richer response, oh so sweet!

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The changes in this pull request aim to enhance the vector content files endpoint by including all Contentfile metadata in the responses. This addresses the need for having more information, such as the edx_block_id, in the response. The changes look good overall, but I have a suggestion regarding the efficiency of the database query.

Summary of Findings

• Efficiency of database query in loop: The current implementation performs a database query within a loop, which can be inefficient. Consider optimizing this by fetching all required data in a single query or using a more efficient data structure.

Assessment

The pull request introduces a change to the vector content files endpoint to include all Contentfile metadata in the responses. The changes seem reasonable and address the issue of missing metadata, such as the edx_block_id. However, there are some potential efficiency concerns with the database query within the loop that should be addressed. I recommend addressing these comments before merging, and users should have others review and approve this code before merging.