InstructLab and Deepsearch #106
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jjasghar
force-pushed
the
jjasghar/deepsearch
branch
5 times, most recently
from
2787217 to
ba8aaed
Compare
This is the proposal to start integrating th document conversion system Deepsearch from IBM Research and InstructLab Co-authored-by: Ming Zhao <[email protected]> Co-authored-by: BJ Hargrave <[email protected]> Signed-off-by: JJ Asghar <[email protected]>
jjasghar
force-pushed
the
jjasghar/deepsearch
branch
from
ba8aaed to
c2ec3e1
Compare
hickeyma
requested changes
Jul 4, 2024
|
|
||
| # DeepSearch + InstructLab Integration Proposal | ||
|
|
||
| <https://github.com/DS4SD> |
There was a problem hiding this comment.
Moved below where DeepSearch is first mentioned
Comment on lines
+8
to
+12
| Managing submissions for the open-source InstructLab project has revealed a significant bottleneck in processing | ||
| knowledge documents. For the InstructLab backend to effectively utilize these documents, they must be in markdown | ||
| format. Currently, we only accept Wikipedia articles, but the built-in conversion tool is inadequate. Internally at | ||
| IBM, and other companies, many knowledge submissions are in multiple document formats, including PDF format, | ||
| necessitating conversion to markdown before being used in InstructLab. |
There was a problem hiding this comment.
Suggested change
| Managing submissions for the open-source InstructLab project has revealed a significant bottleneck in processing | |
| knowledge documents. For the InstructLab backend to effectively utilize these documents, they must be in markdown | |
| format. Currently, we only accept Wikipedia articles, but the built-in conversion tool is inadequate. Internally at | |
| IBM, and other companies, many knowledge submissions are in multiple document formats, including PDF format, | |
| necessitating conversion to markdown before being used in InstructLab. | |
| Managing taxonomy submissions in InstructLab has revealed an issue in processing knowledge documents. For the InstructLab to handle these documents, they must be in [markdown](https://en.wikipedia.org/wiki/Markdown) format. However, many knowledge submissions are in multiple document formats, including PDF, which necessitating conversion to markdown before being used by InstructLab. |
Comment on lines
+14
to
+16
| Existing open-source methods, such as PanDoc, are inconsistent. While they preserve text, they struggle with parsing | ||
| tables and special symbols, as evidenced by issues in PR #1154 of the taxonomy repo in the InstructLab project. Other | ||
| open-source solutions have similar shortcomings. |
There was a problem hiding this comment.
Suggested change
| Existing open-source methods, such as PanDoc, are inconsistent. While they preserve text, they struggle with parsing | |
| tables and special symbols, as evidenced by issues in PR #1154 of the taxonomy repo in the InstructLab project. Other | |
| open-source solutions have similar shortcomings. | |
| Existing open source tools, such as [Pandoc](https://pandoc.org/), are inconsistent. While they preserve text, they struggle with parsing tables and special symbols, as evidenced by issues in [PR #1154](https://github.com/instructlab/taxonomy/pull/1154) of the taxonomy repo in the InstructLab. Other open-source solutions have similar shortcomings. |
Comment on lines
+20
to
+23
| IBM's DeepSearch software excels in document conversion, outperforming traditional open-source methods. Utilizing a | ||
| computer vision model layer, it accurately parses content in the files, including titles, headers, and tables. | ||
| Additionally, it automatically implements RAG layers for models, which could benefit the InstructLab process in | ||
| the future. |
There was a problem hiding this comment.
Suggested change
| IBM's DeepSearch software excels in document conversion, outperforming traditional open-source methods. Utilizing a | |
| computer vision model layer, it accurately parses content in the files, including titles, headers, and tables. | |
| Additionally, it automatically implements RAG layers for models, which could benefit the InstructLab process in | |
| the future. | |
| IBM's [DeepSearch](https://github.com/DS4SD) software excels in document conversion, outperforming traditional open source methods. Utilizing a computer vision model layer, it accurately parses content in the files, including titles, headers, and tables. Additionally, it automatically implements RAG layers for models, which could benefit the InstructLab process in the future. |
| @@ -0,0 +1,52 @@ | |||
|
|
|||
| # DeepSearch + InstructLab Integration Proposal | |||
There was a problem hiding this comment.
Suggested change
| # DeepSearch + InstructLab Integration Proposal | |
| # Document Conversion Proposal |
Comment on lines
+30
to
+33
| ### Open-Source Conversion | ||
|
|
||
| - Implement a basic document conversion tool in the UI using an open-source method such as PanDoc. This tool will be | ||
| lightweight and easily hosted, ensuring it can be used and improved by the community. |
There was a problem hiding this comment.
Suggested change
| ### Open-Source Conversion | |
| - Implement a basic document conversion tool in the UI using an open-source method such as PanDoc. This tool will be | |
| lightweight and easily hosted, ensuring it can be used and improved by the community. | |
| - Open source conversion: Implement a basic document conversion tool in the InstructLab UI using an open source tool such as [Pandoc](https://pandoc.org/). This tool will be lightweight and easily hosted, ensuring it can be used and improved by the community. |
Comment on lines
+35
to
+39
| ### DeepSearch Integration | ||
|
|
||
| - Enable the UI to switch the conversion endpoint to DeepSearch, allowing high-fidelity markdown conversions for | ||
| backend use. This approach maintains an open-source version while benefiting from DeepSearch's superior | ||
| conversion capabilities. |
There was a problem hiding this comment.
Suggested change
| ### DeepSearch Integration | |
| - Enable the UI to switch the conversion endpoint to DeepSearch, allowing high-fidelity markdown conversions for | |
| backend use. This approach maintains an open-source version while benefiting from DeepSearch's superior | |
| conversion capabilities. | |
| - [DeepSearch](https://github.com/DS4SD) conversion: Enable the InstructLab UI to switch the conversion endpoint to DeepSearch, allowing high-fidelity markdown conversions for backend use. This approach uses the open source version of Deepsearch while benefiting from DeepSearch's superior conversion capabilities. |
Comment on lines
+41
to
+43
| IBM Research and the DeepSearch team will host the DeepSearch endpoint for the open-source community. This | ||
| arrangement benefits the community by streamlining contributions and provides data and exposure for the DeepSearch | ||
| project. IBM's contribution underscores its commitment to supporting and improving open-source projects. |
There was a problem hiding this comment.
Suggested change
| IBM Research and the DeepSearch team will host the DeepSearch endpoint for the open-source community. This | |
| arrangement benefits the community by streamlining contributions and provides data and exposure for the DeepSearch | |
| project. IBM's contribution underscores its commitment to supporting and improving open-source projects. | |
| IBM Research and the DeepSearch team will host the DeepSearch endpoint for the InstructLab community. This | |
| arrangement benefits the community by providing a means to handle different document formats. |
Comment on lines
+45
to
+48
| This integration will highlight the value of DeepSearch, highlighting their potential for those integrating | ||
| InstructLab into their workflows. If the volume of community requests becomes unsustainable for the DeepSearch team, | ||
| we hope for ample notification to allow the community to find alternative solutions. By then, we anticipate that the | ||
| open-source versions will have improved sufficiently, or the value of the integration will justify continued support. |
There was a problem hiding this comment.
Suggested change
| This integration will highlight the value of DeepSearch, highlighting their potential for those integrating | |
| InstructLab into their workflows. If the volume of community requests becomes unsustainable for the DeepSearch team, | |
| we hope for ample notification to allow the community to find alternative solutions. By then, we anticipate that the | |
| open-source versions will have improved sufficiently, or the value of the integration will justify continued support. | |
| If the volume of community requests becomes unsustainable for the DeepSearch team to manage, we hope for ample notification to allow the community to find alternative solutions. By then, we anticipate that the open source conversion tools will have improved sufficiently, or the value of the integration will justify continued support. |
Comment on lines
+50
to
+52
| By adopting this two-pronged approach, we ensure the integrity of the open-source project while leveraging IBM's | ||
| advanced DeepSearch capabilities. This strategy balances community collaboration with innovative technology, | ||
| fostering innovation and improvement in document processing for the InstructLab project. |
There was a problem hiding this comment.
Suggested change
| By adopting this two-pronged approach, we ensure the integrity of the open-source project while leveraging IBM's | |
| advanced DeepSearch capabilities. This strategy balances community collaboration with innovative technology, | |
| fostering innovation and improvement in document processing for the InstructLab project. | |
| By adopting this two-pronged approach, we ensure the integrity of the open source project while leveraging advanced DeepSearch capabilities. This strategy balances community collaboration with innovative technology, | |
| fostering innovation and improvement in document processing for the InstructLab community. |
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the proposal to start integrating th document conversion system Deepsearch from IBM Research and InstructLab