Added Multimodal visual qna using two advanced models-BLIP & CLIP #1188
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ls/Blip and Clip models
…imodal VQA.(using CLIP and VizWiz dataset).ipynb
…sing(Using_BLIP).ipynb
…ls/src/Output/img
|
Thank you for submitting your pull request! 🙌 We'll review it as soon as possible. If there are any specific instructions or feedback regarding your PR, we'll provide them here. Thanks again for your contribution! 😊 |
|
@Niketkumardheeryan Please review as time is less |
|
@Panchadip-128 expecting more modular code with proper comments |
…ls/VQA_image_processing(Using_BLIP).ipynb
|
@Niketkumardheeryan Done sir that one file also i made the code into modules |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
his repository contains an implementation of a Visual Question Answering (VQA) model built using the BLIP (Bootstrapping Language-Image Pre-training) framework. This model can understand image content and answer questions related to the provided images. A tool that reads an image based on ML algorithms( BLIP model) and implements Visual QA which answers questions based on user prompts for the image, deployed through Gradio web application
Overview
This project focuses on developing Visual Question Answering (VQA) systems using two models: CLIP (Contrastive Language-Image Pretraining) and BLIP (Bootstrapped Language-Image Pretraining). The goal of VQA is to answer questions about an image by jointly learning from both textual and visual information. This project demonstrates how to utilize the CLIP and BLIP models for VQA tasks, and it includes the training, validation, and testing procedures, as well as metrics for evaluation.
Data Overview:
The dataset used for this project is sourced from the VizWiz 2023 VQA challenge. It contains three main components:
train.json: The training set with image URLs, questions, and answers. val.json: The validation set. test.json: The test set without answer labels for evaluation purposes. Each entry in the dataset includes:
Image: The visual input for the model. Question: The textual question that the model needs to answer. Answers: A list of possible answers (for training/validation) and their confidences. The dataset is preprocessed to create a balanced set by stratifying based on the answerable and answer_type labels.
CLIP-Based VQA Model
Overview:
The CLIP-based VQA model uses the pre-trained CLIP model to extract both visual and textual embeddings. The two embeddings are concatenated to form a single vector, which is then passed through a fully connected network to predict the answer.