Support Multimodal inference with multiple images and PDF-s in NATIVE engine
#1424
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xrdaukar
changed the title
NATIVE engine
taenin
approved these changes
Feb 12, 2025
| @@ -43,7 +43,7 @@ def _get_engine(config: InferenceConfig) -> BaseInferenceEngine: | |||
| def infer_interactive( | |||
| config: InferenceConfig, | |||
| *, | |||
| input_image_bytes: Optional[bytes] = None, | |||
| input_image_bytes: Optional[list[bytes]] = None, | |||
There was a problem hiding this comment.
More of a meta comment about this PR: This looks like a breaking change for VLM functionality. Should we consider a larger version bump for our next pypi package?
There was a problem hiding this comment.
We may be lifting one arbitrary/un-documented constraint. The previous code might by broken/not-functional if multi-image inputs are passed to it (most datasets are single-image) => it's probably OK to do regular version increment.
Co-authored-by: Matthew Persons <[email protected]>
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
-- Update NATIVE inference engine to support prediction with multiple images.
-- Add utils to load PDF-s as N images (1 image per page).
-- Add new target
[file_formats]to load libraries required for PDF processing-- Update CLI
inferto support PDF inputs-- Add sample 4-page PDF to
testdata-- Tested with Llama Vision only for now (will handle other model types and other engines separately)
Related issues
Towards OPE-994, OPE-355
Before submitting
Reviewers
At least one review from a member of
oumi-ai/oumi-staffis required.