Fs 102/create report agent #34

stevenhillcox-sl · 2024-11-25T13:52:53Z

Description

https://scottlogic.atlassian.net.mcas.ms/browse/FS-102

Changelog

Add a new report agent to form and execute prompts relating to ESG report generation
Add new user/system prompt templates for ESG report generation
Add report agent into report_director.py flow

… the same way. We may want to think about creating some kind of 'SystemAgent' class that doesn't use invokve with an utternace in the future. Ignore an annoying pyright error that is a non-issue

…emove the example output which was restricting the prompts flexability

…nt getter model of the other agents. updated prompts to more accurately describe relationships and updated generate cypher query to be less restricted through copying examples which was causing bad cypher queries to be generated. added one BDD test based on the new dataset

…tfoo to have reusable prompt generator for all promptfoo tests.

# Conflicts: # .env.example # backend/src/director.py # backend/src/prompts/templates/generate-knowledge-graph-model.j2 # backend/src/utils/dynamic_knowledge_graph.py

IMladjenovic · 2024-11-25T17:57:38Z

backend/promptfoo/create_esg_report_config.yaml

+prompts: file://promptfoo_test_runner.py:create_prompt
+
+tests:
+  - description: "test model prompt references all csv headers in result using valid json format"


Do we need to add an assert for this test to check the output against?

I mainly used this to test that our LLMs were giving sensible responses. I am not entirely sure if there's anything more I can test at this point until we learn more about the ESG capabilities.
I suppose you could theoretically check that the output is formatted as markdown, but because almost anything is valid markdown, I am not sure how I would implement that.
Happy to hear any suggestions?

Yeah I agree, not much to test here. In that case, I'd just update the test description to description: sample test for prompt development or something along those lines

IMladjenovic · 2024-11-25T18:06:15Z

backend/src/agents/esg_report_agent.py

+    async def invoke(self, utterance: str) -> str:
+        user_prompt = engine.load_prompt(
+            "create-esg-report-user-prompt",
+            document_text=utterance)


It feels a bit funny that we need to pass in the file via utterance and we need to annotate with @agent, our base Agent class is purpose built around the core "chat" style functionality of InferESG.

This is Something for another ticket entirely - but it feels like we need a new concept here, possibly creating a SystemAgent or ReportAgent base class that makes a cleaner separation from the Agent class - or possibly work the other way and change Agent class as it is now into ChatAgent.

What you've built here looks good and matches the designs we agreed on for this ticket

I actually tried renaming the variable to something more sensible, but the linter didn't seem to like it.

IMladjenovic · 2024-11-26T11:02:25Z

Can we hook this up to the API and report director and test with swagger? http://localhost:8250/docs

IMladjenovic · 2024-11-26T11:04:26Z

Can we hook this up to the API and report director and test with swagger? http://localhost:8250/docs

Ah, we are still waiting for Mike's ticket to go in to get the report director changes

evpearce · 2024-11-26T11:18:32Z

backend/promptfoo/create_esg_report_config.yaml

+      user_prompt_template: "create-esg-report-user-prompt"
+      system_prompt_template: "create-esg-report-system-prompt"
+      user_prompt_args:
+        document_text: "Published September 2024  Carbon Reduction Plan 


I tried running your prompt with the kingfisher business report (https://scottlogic.atlassian.net/wiki/spaces/FS/pages/4422729729/Kingfisher+Responsible+Business+Report) it cut off the response after the 2nd bullet point and social and didn't even get to Governance. The response is probably too long.

Also it completely doesn't work for the McDonald's impact report (https://scottlogic.atlassian.net/wiki/spaces/FS/pages/4422696964/McDonald+s+Impact+Report+2023+24)

When I run this, I am seeing it work for mcdonalds without any changes required. Not sure what's different - were you using the large mistral model when you tested?

evpearce · 2024-11-26T11:26:48Z

backend/promptfoo/create_esg_report_config.yaml

+
+prompts: file://promptfoo_test_runner.py:create_prompt
+
+tests:


could we add a few basic tests
for example:

has the 3 headings "Environmental", "Social" and "Governance"

for the AWS report the percentages should be easy enough to test since it's 100% environmental

another test for another report eg kingfisher

any content that might be wanting to be tested?

I've added tests for the first bullet point.

For testing the specific content we expect to see in the report, I would lean towards not locking ourselves into expecting anything particular at the moment. Once this is in we need to take stock of where we are, what we are seeing and present it to Arbdn to get feedback on the direction. Trying to capture specifics in these reports via tests is hard because this could change as we learn more from Arbdn

…oo tests

IMladjenovic · 2024-11-29T14:42:27Z

backend/src/utils/config.py

@@ -61,6 +63,7 @@ def load_env(self):
            self.files_directory = os.getenv("FILES_DIRECTORY", default_files_directory)
            self.answer_agent_llm = os.getenv("ANSWER_AGENT_LLM")
            self.intent_agent_llm = os.getenv("INTENT_AGENT_LLM")
+            self.report_agent_llm = os.getenv("ESG_REPORT_AGENT_LLM")


Ivan to fix this (remove ESG_)

Steven Hillcox and others added 21 commits November 22, 2024 11:09

WIP Create Report Agent

ee7fba6

WIP Update prompts and add a simple unit test

b6c58ef

pull all cahnges across

9feb3e6

fix linting

9eb18db

remove typo

4fe0729

new agent doesn't work like other agents, so shouldn't be included in…

1653b93

… the same way. We may want to think about creating some kind of 'SystemAgent' class that doesn't use invokve with an utternace in the future. Ignore an annoying pyright error that is a non-issue

minor fix

10e29a9

minor tweaks to prompt

78e68b4

small improvement to the prompt to model dates as relationships and r…

f0ee65e

…emove the example output which was restricting the prompts flexability

add datasets from abondned branch. slight improvements to prompts

deed7bc

fix env files and cleanup a test as per MR comments

9011bfd

update file name. improve comments

ec0b234

rework dkg not as agent but as util

bad342b

improve prompts from PR feedback. Setup promptfoo tests. rework promp…

ec5e6b5

…tfoo to have reusable prompt generator for all promptfoo tests.

prompt improvements

b95750f

Tidy up + promptfoo setup

2406789

Merge branch 'main' into FS-102/create-report-agent

9beeea2

# Conflicts: # .env.example # backend/src/director.py # backend/src/prompts/templates/generate-knowledge-graph-model.j2 # backend/src/utils/dynamic_knowledge_graph.py

Wire up ESG agent

176410f

Fix linting

bc1842f

Remove files copied over in rebase

e978487

IMladjenovic reviewed Nov 25, 2024

View reviewed changes

Update promptfoo test description

5fda58f

evpearce reviewed Nov 26, 2024

View reviewed changes

IMladjenovic added 3 commits November 28, 2024 13:31

Merge branch 'main' into FS-102/create-report-agent

273d546

hook report agent into report director

8310083

refactor esg_report_agent to report_agent everywhere. improve promptf…

67aa954

…oo tests

fix typing issue

c6d9947

IMladjenovic reviewed Nov 29, 2024

View reviewed changes

IMladjenovic added 7 commits November 29, 2024 15:44

fix env variable name

dc0f33c

fix test

67e2bd3

fix linting

093f8cd

attempt to fix tests again

cef4e0b

attempt to fix tests again

743db6c

attempt to fix tests again

93d2c24

fix tests

6025fca

evpearce approved these changes Dec 2, 2024

View reviewed changes

slight improvement to prompt

43f7066

IMladjenovic merged commit df60207 into main Dec 2, 2024
4 checks passed

IMladjenovic deleted the FS-102/create-report-agent branch December 2, 2024 13:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fs 102/create report agent #34

Fs 102/create report agent #34

stevenhillcox-sl commented Nov 25, 2024 •

edited by IMladjenovic

Loading

IMladjenovic Nov 25, 2024

stevenhillcox-sl Nov 26, 2024

IMladjenovic Nov 26, 2024

stevenhillcox-sl Nov 26, 2024

IMladjenovic Nov 25, 2024

stevenhillcox-sl Nov 26, 2024

IMladjenovic commented Nov 26, 2024

IMladjenovic commented Nov 26, 2024

evpearce Nov 26, 2024

evpearce Nov 26, 2024

IMladjenovic Nov 28, 2024

evpearce Nov 26, 2024

IMladjenovic Nov 28, 2024

IMladjenovic Nov 29, 2024


		prompts: file://promptfoo_test_runner.py:create_prompt

		tests:

Fs 102/create report agent #34

Fs 102/create report agent #34

Conversation

stevenhillcox-sl commented Nov 25, 2024 • edited by IMladjenovic Loading

Description

Changelog

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

IMladjenovic commented Nov 26, 2024

IMladjenovic commented Nov 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevenhillcox-sl commented Nov 25, 2024 •

edited by IMladjenovic

Loading