Companion example for nemo/trulens blog #1131
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* bump versions in quickstarts * bump version * remove openai references in function definitions page * gemini example * headers * second example: semantic evals * updates, add rag triad * update top header
…#696) * add aliases for selectors for main method args and main method return * break down * refine --------- Co-authored-by: Josh Reini <[email protected]>
* exposed AzureOpenAI provider * added docs * Update CONTRIBUTING.md * typo in mkdocs.yml --------- Co-authored-by: Josh Reini <[email protected]>
* first * typos * typehint
* import llama only if needed * use optional imports instead --------- Co-authored-by: Piotr Mardziel <[email protected]>
* fix * typo * don't print external if internal is available
Co-authored-by: Piotr Mardziel <[email protected]>
* adjust docstring for select_context * langchain select_context, update quickstarts * undo app name change * remove dev cell * generalized langchain select_context (#711) * generalized langchain select_context * typo * typo in string * update langchain example to pass app in select_context --------- Co-authored-by: Josh Reini <[email protected]> * comments, clarity updates to quickstarts * add lib-independent select_context * update lc li quickstarts --------- Co-authored-by: Piotr Mardziel <[email protected]>
* add optional * bug class_info fix
* update configs * bugfix * dont add class info to dicts
* Fix correctness prompt Fixes #718 * Update base.py
* Bump suggested notebook versions * Combine notebooks and py files --------- Co-authored-by: Shayak Sen <[email protected]>
* Bump suggested notebook versions * Combine notebooks and py files * Update __init__.py --------- Co-authored-by: Shayak Sen <[email protected]>
* debug * display python version * python version * PromptTemplate update import * bad escape fix * add msg to exception * pass kwargs in Groundedness * pass kwargs with GroundTruthAgreement * give default value to ground_truth_imp * migrate db on reset
* fix example notebook * fixes * remove commented out
* always use prompt instead of messages * use messages in base * use prompt in bedrock * move score to top of cot template, request entire template be used * remove dev * add TODO
* update langchain instrumentation page * include instrumented methods * llama-index instrumentation updates * update the overview * change path to instrumentation overview * add some more info in appendices and line space --------- Co-authored-by: Piotr Mardziel <[email protected]>
* add instructions and text wrapping * format * clean up github scripts and update README sources * typo --------- Co-authored-by: Josh Reini <[email protected]>
Co-authored-by: joshreini1 <[email protected]>
* fix * remove redundant --------- Co-authored-by: Josh Reini <[email protected]>
* adjusted * fix typo
Co-authored-by: joshreini1 <[email protected]>
* add instructions and text wrapping * format * debugging * making obj arg no longer required * remove obj and add documentation for WithClassInfo * remove IPython from most notebooks and organize imports * fix test errors * forgot warning --------- Co-authored-by: Josh Reini <[email protected]>
* update notebooks to test * rehack * update langchain requirement * add core lowerbound
* fix instance check on generic types * remove original check --------- Co-authored-by: Piotr Mardziel <[email protected]>
Co-authored-by: Josh Reini <[email protected]>
* Update feedback.py * use name
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
dosubot
bot
added
the
size:XXL
piotrm0
reviewed
May 13, 2024
| @@ -0,0 +1,1250 @@ | |||
| { | |||
There was a problem hiding this comment.
Line #24. "text1":"{question_lens}",\
The nemo persons suggested that using the variables version of the lenses here (for example {context_lens} ) is a bit confusing. Because of this, there is the other notebook that does it a bit simpler by defining python methods for each of these flows.
Reply via ReviewNB
| @@ -0,0 +1,1250 @@ | |||
| { | |||
There was a problem hiding this comment.
Line #64. if $groundedness_result < 0.7
Confused here. $groundedness_result, $answerrelevance_result, $contextrelevance_result have not been defined. Also unclear when the prior two flows get executed. Perhaps the code text is oudated?
Reply via ReviewNB
There was a problem hiding this comment.
Updated to include $groundedness_result, $answerrelevance_result, $contextrelevance_result generation.
Not sure what you mean by "Also unclear when the prior two flows get executed."?
There was a problem hiding this comment.
This config specification determines what "flows" get executed when the bot wants to output something:
output:
flows:
- check language match
# triad defined seperately so hopefully they can be executed in parallel
- check rag triad groundedness
- check rag triad relevance
- check rag triad qs_relevance
- bot: explain usage and capabilitiesThe last flow in the notebook is check rag triad context_relevance which is now not executed as it is not in the list. Even if it were in the list, it is unclear how the values that were set by the other flows are set or whether they are available in that flow (I'm uncertain if colang will make the variables defined in one flow available in another). The other issue is even if variables were available, you would need to make sure the 2 flows that compute $groundedness_result and $answerrelevance_result are executed before they are used in the check rag triad context_relevance flow.
dosubot
bot
added
size:XL
retreivers -> retrievers
* unify groundedness start * remove groundedness.py * groundedness nli moves * remove custom aggregator * groundedness aggregator to user code * move agg to trulens side by default (groundedness) * remove extra code * remove hf key setting * remove hf import * add comment about aggregation for context relevance * update init * remove unneeded import * use generate_score_and_reasons for groundedness internally * f-strings for groundedness prompts * docstring * docstrings formatting * groundedness reasons template * remove redundant prompt * update quickstarts * llama-index notebooks * rag triad helper update * oai assistant nb * update readme * models notebooks updates * iterate nbs * mongo, pinecone nbs * update huggingface docstring * remove outdated docstring selector notes * more docstring cleaning
* open ai streaming adjustments in cost tracking * notes * delete outputs
Co-authored-by: joshreini1 <[email protected]> Co-authored-by: Josh Reini <[email protected]>
* Update selecting_components.md * Update MultiQueryRetrievalLangchain.ipynb * Update random_evaluation.ipynb * Update canopy_quickstart.ipynb
Co-authored-by: joshreini1 <[email protected]>
* update comprehensiveness + nb * nb expansion * fix typo * meetingbank transcript data * oss models in app * test * benchmarking gpt-3.5-turbo, gpt-4-turbo, and gpt-4o * update path * comprehensiveness benchmark * updated summarization_eval nb * fix normalization * show improvement in comprehensiveness feedback functions --------- Co-authored-by: Daniel <[email protected]>
* version bump * simpler lc quickstart * update installs and imports * update langchain instrumentation docs * remove groundedness ref from providers.md * build docs fixes * remove key cell * fix docs build * firx formatting for stock.md * remove extra spaces * undo format change * update docstrings for hugs and base provider * openai docstring updates * hugs docstring update * update context relevance hugs docstring * more docstring updates * remove can be changed messages from openai provider docstrings
Co-authored-by: joshreini1 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Other details that are good to know but need not be announced: