Skip to content

Commit

Permalink
Address Gagan's comments
Browse files Browse the repository at this point in the history
  • Loading branch information
evpearce committed Nov 27, 2024
1 parent e835806 commit 3e00647
Show file tree
Hide file tree
Showing 3 changed files with 21 additions and 26 deletions.
8 changes: 4 additions & 4 deletions backend/promptfoo/intent_config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ providers:
prompts: file://promptfoo_test_runner.py:create_prompt

tests:
- description: "questions directed towards the database look ups should have only 1 question -1"
- description: "questions directed towards the database lookups should have only 1 question -1"
vars:
system_prompt_template: "intent-system"
user_prompt_template: "intent"
Expand All @@ -30,7 +30,7 @@ tests:
- type: javascript
value: JSON.parse(output).questions.length === 0

- description: "verify that the correct company name is determinded from the chat history"
- description: "verify that the correct company name is determined from the chat history"
vars:
system_prompt_template: "intent-system"
user_prompt_template: "intent"
Expand All @@ -51,11 +51,11 @@ tests:
user_prompt_template: "intent"
user_prompt_args:
chat_history: []
question: "Compare Ryanair emmissions to other companies in the industry"
question: "Compare Ryanair emissions to other companies in the industry"
assert:
- type: javascript
value: JSON.parse(output).questions[0].includes("Ryanair")
- type: llm-rubric
value: The 1st item in the questions array contains a question about finding the emissions for Ryanair
- type: llm-rubric
value: The 2nd item in the questions array contains a question about finding the emissions for companiesin the industry
value: The 2nd item in the questions array contains a question about finding the emissions for companies in the industry
10 changes: 4 additions & 6 deletions backend/src/prompts/templates/intent-system.j2
Original file line number Diff line number Diff line change
@@ -1,16 +1,14 @@
You are an expert in determinging the intent behind a user's question, breaking down complex questions into multiple simplier questions and forming stand alone questions based on context from chat history.
You are an expert in determining the intent behind a user's question, breaking down complex questions into multiple simpler questions and forming standalone questions based on context from chat history.

You will be given a question and may also be provided with chat history which is to be used as context if provided.


- First determin the user intent of the question by taking key points from the question and gaining context from the chat history.
- Second, if the question does not mentions csv, dataset or database, using the user intent try to split the question up into multiple questions with singular objectives.
- First determine the user intent of the question by taking key points from the question and gaining context from the chat history.
- Second, if the question does not mention csv, dataset or database, using the user intent try to split the question up into multiple questions with singular objectives.

Output your result in the following json format:

{
"question": "string of the original question",
"user_intent": "string of the intent of the user's question",
"questions": array of singular objective questions or if the question mentions csv, dataset or database an empty array
}

}
29 changes: 13 additions & 16 deletions backend/src/supervisors/supervisor.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,28 +12,25 @@
no_agent_response = "I am sorry, but I was unable to find an agent to solve this task"


async def process_question(question) -> None:
try:
(agent_name, answer, status) = await solve_task(question, get_scratchpad())
update_scratchpad(agent_name, question, answer)
if status == "error":
raise Exception(answer)
except Exception as error:
update_scratchpad(error=str(error))


async def solve_all(intent_json) -> None:
questions = intent_json["questions"]

if len(questions) == 0:
try:
question = intent_json["question"]
(agent_name, answer, status) = await solve_task(question, get_scratchpad())
update_scratchpad(agent_name, question, answer)
if status == "error":
raise Exception(answer)
except Exception as error:
update_scratchpad(error=str(error))
question = intent_json["question"]
await process_question(question)
else:
for question in questions:
try:
(agent_name, answer, status) = await solve_task(question, get_scratchpad())
update_scratchpad(agent_name, question, answer)
if status == "error":
logger.info("status == error")
raise Exception(answer)
except Exception as error:
update_scratchpad(error=str(error))
await process_question(question)


async def solve_task(task, scratchpad, attempt=0) -> Tuple[str, str, str]:
Expand Down

0 comments on commit 3e00647

Please sign in to comment.