-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fs 102/create report agent #34
Changes from 22 commits
ee7fba6
b6c58ef
9feb3e6
9eb18db
4fe0729
1653b93
10e29a9
78e68b4
f0ee65e
deed7bc
9011bfd
efe55aa
ec0b234
bad342b
ec5e6b5
b95750f
2406789
9beeea2
176410f
bc1842f
e978487
5fda58f
273d546
8310083
67aa954
c6d9947
dc0f33c
67e2bd3
093f8cd
cef4e0b
743db6c
93d2c24
6025fca
43f7066
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,110 @@ | ||
description: "Test ESG Report Prompt" | ||
|
||
providers: | ||
- id: mistral:mistral-large-latest | ||
config: | ||
temperature: 0 | ||
|
||
prompts: file://promptfoo_test_runner.py:create_prompt | ||
|
||
tests: | ||
- description: "Sample test to aid in ESG report generation development" | ||
vars: | ||
user_prompt_template: "create-esg-report-user-prompt" | ||
system_prompt_template: "create-esg-report-system-prompt" | ||
user_prompt_args: | ||
document_text: "Published September 2024 Carbon Reduction Plan | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I tried running your prompt with the kingfisher business report (https://scottlogic.atlassian.net/wiki/spaces/FS/pages/4422729729/Kingfisher+Responsible+Business+Report) it cut off the response after the 2nd bullet point and social and didn't even get to Governance. The response is probably too long. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also it completely doesn't work for the McDonald's impact report (https://scottlogic.atlassian.net/wiki/spaces/FS/pages/4422696964/McDonald+s+Impact+Report+2023+24) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When I run this, I am seeing it work for mcdonalds without any changes required. Not sure what's different - were you using the large mistral model when you tested? |
||
Supplier name: Amazon Web Services EU SARL (UK Branch) (“AWS UK”) | ||
Publication date: September 30, 2024 | ||
Commitment to Achieving Net Zero | ||
AWS UK, as part of Amazon.com, Inc. (“Amazon”), is committed to achieving net -zero | ||
emissions by 2040. In 2019, Amazon co -founded The Climate Pledge, a public commitment | ||
to innovate, use our scale for good and go faster to address the urgency of the climate crisis | ||
to reach net -zero carbon across the entire organization by 2040. Since committing to the | ||
Pledge, we’ve changed how we conduct our business and the running of our operations, and | ||
we’ve increased funding and implementation of new technologies and services that | ||
decarbonize and help preserve the natural world, alon gside the ambitious goals outlined in | ||
The Climate Pledge. We’re fully committed to our goals and our work to build a better planet. | ||
Baseline Emissions Footprint | ||
Base Year emissions are a record of the greenhouse gases that have been produced in the | ||
past an d are the reference point against which emissions reduction can be measured. | ||
Baseline Year: 2020 | ||
Additional Details relating to the Baseline Emissions calculations: | ||
AWS UK utilized January 1, 2020 to December 31, 2020 as the baseline year for emissions | ||
reporting under this Carbon Reduction Plan. Our plan includes emissions data from relevant | ||
affiliate companies helping to provide AWS UK’s services to our customers. We ’ve included both | ||
location -based and market -based method Scope 2 emissions in the following tables. AWS UK | ||
benefits from contractual arrangements entered into by our affiliate(s) for renewable electricity | ||
and/or renewable attributes that are reflected in t he market -based data set. More information | ||
about our corporate carbon footprint and methodology can be found on our website . | ||
Our baseline year does not include Scope 1 emissions. In 2022 we updated our methodology | ||
and Scope 1 emissions are now included in total emissions for AWS UK | ||
|
||
Published September 2024 Baseline year emissions: | ||
EMISSIONS TOTAL (tCO 2e) | ||
Scope 1 0 | ||
Scope 2 61,346 – Location -based method | ||
2,813 – Market -based method | ||
Scope 3 (Included | ||
Sources) 3,770 | ||
Total Emissions 65,116 – Location -based method | ||
6,583 – Market -based method | ||
Current Emissions Reporting | ||
Reporting Year: 202 3 (January 1, 202 3 to December 31, 202 3) | ||
EMISSIONS TOTAL (tCO 2e) | ||
Scope 1 2,23 3 | ||
Scope 2 126,755 – Location -based method | ||
0 – Market -based method | ||
Scope 3 (Included | ||
Sources) 13,188 | ||
Total Emissions 142,17 6 – Location -based method | ||
15,42 1 – Market -based method | ||
|
||
Published September 2024 Emissions Reduction Targets | ||
In 2019, we set an ambitious goal to match 100% of the electricity we use with renewable | ||
energy by 2030. This goal includes all data centres , logistics facilities, physical stores, and | ||
corporate offices, as well as on -site charg ing points and our financially integrated subsidiaries. | ||
We are proud to have achieved this goal in 2023, seven years early, with 100% of the electricity | ||
consum ed by Amazon matched with renewable energy sources. | ||
Amazon continue s to be transparent and share our progress to reach net -zero carbon in our | ||
annual Sustainability Report , which also includes details on how we measure carbon . | ||
Carbon Reduction Projects | ||
Completed Carbon Reduction Initiatives | ||
Amazon continues to take actions across our operations to drive carbon reduction around the | ||
world, including in the UK. As of January 202 4, Amazon’s renewable energy portfolio includes | ||
243 wind and solar farms and 2 70 rooftop solar projects, totalling 513 projects and 28 | ||
gigawatts of renewable energy capacity. This includes several utility -scale renewable energy | ||
projects located within the UK: | ||
•In 2019, Amazon announced our first power purchase agreement in the UK, located in | ||
Kintyre Peninsula, Scotland. The “Amazon Wind Farm Scotland – Beinn an Tuirc 3” | ||
began o perating in 2021, providing 50 megawatts (MW) of new renewable capacity to | ||
the electricity grid with expected generation of 168,000 megawatt hours (MWh) of | ||
clean energy annually. That’s enough to power 46,000 UK homes every year. | ||
•In December 2020, Amazon a nnounced a two -phase renewable energy project located | ||
in South Lanarkshire, Scotland, the Kennoxhead wind farm. Kennoxhead will be the | ||
largest single -site onshore wind project in the UK, enabled through corporate | ||
procurement. Once fully operational, Kenno xhead will produce 129 MW of renewable | ||
capacity and is expected to generate 439,000 MWh of clean energy annually. Phase 1 | ||
(60 MW) began operating in 2022, and Phase 2 (69 MW) will begin operations in 2024 . | ||
•In 2022, Amazon announced its first project in Nor thern Ireland, a 16 MW onshore | ||
windfarm in Co Antrim. | ||
•In 2022, Amazon also announced a new 473 MW offshore wind farm, Moray West, | ||
located off the coast of Scotland . Amazon expects completion of Moray West in 2024. | ||
This is Amazon’s largest project in Scotland and the largest corporate renewable | ||
energy deal announced by any company in the UK to date. | ||
•In 2023, Amazon announced a new 47 MW solar farm, Warl ey located in Essex. | ||
This project is expected to be operational in 2024. | ||
|
||
Published September 2024 Declaration and Sign Off | ||
This Carbon Reduction Plan has been completed in accordance with PPN 06/21 and | ||
associated guidance and reporting standard for Carbon Reduction Plans. | ||
Emiss ions have been reported and recorded in accordance with the published reporting | ||
standard for Carbon Reduction Plans and the GHG Reporting Protocol corporate standard1 | ||
and uses the appropri ate Government emission conversion factors for greenhouse gas | ||
company reporting2. | ||
Scope 1 and Scope 2 emissions have been reported in accordance with S ECR requirements, | ||
and the required subset of Scope 3 emissions have been reported in accordance with the | ||
published reporting standard for Carbon Reduction Plans and the Corporate Value Chain | ||
(Scope 3) Standard3. | ||
This Carbon Reduction Plan has been reviewe d and signed off by the board of directors (or | ||
equivalent management body)." |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
from src.agents import Agent, agent | ||
from src.prompts import PromptEngine | ||
|
||
engine = PromptEngine() | ||
|
||
|
||
@agent( | ||
name="ESGReportAgent", | ||
description="This agent is responsible for generating an ESG report", | ||
tools=[], | ||
) | ||
class ESGReportAgent(Agent): | ||
async def invoke(self, utterance: str) -> str: | ||
user_prompt = engine.load_prompt( | ||
"create-esg-report-user-prompt", | ||
document_text=utterance) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It feels a bit funny that we need to pass in the file via This is Something for another ticket entirely - but it feels like we need a new concept here, possibly creating a What you've built here looks good and matches the designs we agreed on for this ticket There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I actually tried renaming the variable to something more sensible, but the linter didn't seem to like it. |
||
|
||
system_prompt = engine.load_prompt( | ||
"create-esg-report-system-prompt", | ||
) | ||
|
||
return await self.llm.chat(self.model, system_prompt=system_prompt, user_prompt=user_prompt) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
The user will provide a report from a company. Your goal is to analyse the document and respond answering the following questions in the format described below: | ||
|
||
Basic: | ||
|
||
1. What is the name of the company that this document refers to? | ||
2. What year or years does the information refer too? | ||
3. Summarise in one sentence what the document is about? | ||
|
||
ESG (Environment, Social, Governance: | ||
1. Which aspects of ESG does this document primarily discuss, respond with a percentage of each topic covered by the document. | ||
2. What aspects of ESG are not discussed in the document? | ||
|
||
Environmental: | ||
|
||
1. What environmental goals does this document describe? | ||
2. What beneficial environmental claims does the company make? | ||
3. What potential environment greenwashing can you identify that should be fact checked? | ||
4. What environmental regulations, standards or certifications can you identify in the document? | ||
|
||
Social: | ||
|
||
1. What social goals does this document describe? | ||
2. What beneficial societal claims does the company make? | ||
3. What potential societal greenwashing can you identify that should be fact checked? | ||
4. What societal regulations, standards or certifications can you identify in the document? | ||
|
||
Governance: | ||
|
||
1. What governance goals does this document describe? | ||
2. What beneficial governance claims does the company make? | ||
3. What potential governance greenwashing can you identify that should be fact checked? | ||
4. What governance regulations, standards or certifications can you identify in the document? | ||
|
||
Conclusion: | ||
|
||
1. What is your conclusion about the claims and potential greenwashing in this document? | ||
2. What are your recommended next steps to verify any of the claims in this document? | ||
|
||
The report should be formatted as markdown. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
Generate an ESG report using the following document: | ||
|
||
{{ document_text }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
import pytest | ||
|
||
from src.agents.esg_report_agent import ESGReportAgent | ||
from src.llm.factory import get_llm | ||
|
||
mock_model = "mockmodel" | ||
mock_llm = get_llm("mockllm") | ||
|
||
@pytest.mark.asyncio | ||
async def test_invoke_calls_llm(mocker): | ||
esg_agent = ESGReportAgent(llm_name="mockllm", model=mock_model) | ||
mock_response = "A Test Report" | ||
|
||
mock_llm.chat = mocker.AsyncMock(return_value=mock_response) | ||
|
||
response = await esg_agent.invoke("Test Document") | ||
|
||
assert response == mock_response | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could we add a few basic tests
for example:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've added tests for the first bullet point.
For testing the specific content we expect to see in the report, I would lean towards not locking ourselves into expecting anything particular at the moment. Once this is in we need to take stock of where we are, what we are seeing and present it to Arbdn to get feedback on the direction. Trying to capture specifics in these reports via tests is hard because this could change as we learn more from Arbdn