Merge branch 'daniel/prompt-opt' into trec-exp-adding-rubric

examples/experimental/mipro_fewshot_optimized

@@ -0,0 +1,58 @@
+{
+  "generate_score_and_reasons.predictor": {
+    "lm": null,
+    "traces": [],
+    "train": [],
+    "demos": [
+      {
+        "augmented": true,
+        "input": "{\"source\":\"The us senate has failed to pass a republican proposal to repeal president barack obama's signature healthcare policy without replacing it. The 45-55 vote marks the second defeat for republicans aiming to pass a bill this week to undo the health policy. The vote came a day after the senate rejected a plan to repeal and replace the health law with a republican plan. Senators will now consider a \\\"skinny\\\" repeal, which would scale back some of the more controversial provisions. The \\\"skinny\\\" plan would eliminate unpopular parts of obamacare - the affordable care act - including the individual mandate requiring all americans to have health insurance coverage as well as a tax on medical devices. If that measure passes, the senate and house of representatives would then be tasked with finalising a bill that could still change during negotiations. If successful, the full house and senate would again have to approve the measure. Seven republicans opposed the repeal-only plan on wednesday, which included a two-year delay on a replacement in the hope of finding agreement before that time elapses. The defeat is the latest blow for president donald trump, who put repealing obamacare at the centre of his election campaign, and underscores sharp divisions within the republican party over a replacement plan. A proposal to repeal and replace obamacare with the better care reconciliation act (bcra), a republican plan that was crafted over last two months, collapsed on tuesday night after nine republicans voted against it. Republicans have long railed against obamacare as government overreach, criticising the system for introducing government-run marketplaces, where premiums have risen sharply for some people. The party's proposed alternative included steep cuts to medicaid, a healthcare programme for the poor and disabled. And it removes obamacare's individual mandate requiring all americans to have health insurance or pay a tax penalty. About 20 million people gained health insurance under former president obama's affordable care act. The non-partisan congressional budgetary office (cbo) found the bill would strip 22 million americans of health insurance over the next decade. Other attempts to replace obamacare have failed in recent weeks due to republican divisions. In an early morning tweet on wednesday, mr trump lambasted republican senator lisa murkowski of alaska for opposing the party's plan, saying she \\\"really let the republicans, and our country, down yesterday. Too bad!\\\" Mrs murkowski was one of two republicans earlier on tuesday to vote against allowing the debate on healthcare legislation reform to go forward. But her party eked out a victory after vice president mike pence cast the tie-breaking vote which allowed discussion to take place. Senator john mccain, who was recently diagnosed with an aggressive brain tumour, received a standing ovation as he returned to congress to cast his \\\"yes\\\" vote. President trump tweeted his thanks to the arizona senator for playing \\\"such a vital role\\\" in the vote.\",\"statement\":\"The us senate has rejected a republican plan to repeal president barack obama's healthcare law.\"}",
+        "reasoning": "```json\n{\n  \"score\": 1.0,\n  \"reasons\": \"The statement is directly supported by the source context, which explicitly mentions that \\\"The us senate has failed to pass a republican proposal to repeal president barack obama's signature healthcare policy without replacing it.\\\" and also states that \\\"A proposal to repeal and replace obamacare with the better care reconciliation act (bcra), a republican plan that was crafted over last two months, collapsed on tuesday night after nine republicans voted against it.\\\"\"\n}\n```",
+        "output": "```json\n{\n  \"score\": 1.0,\n  \"reasons\": \"The statement is directly supported by the source context, which explicitly mentions that \\\"The us senate has failed to pass a republican proposal to repeal president barack obama's signature healthcare policy without replacing it.\\\" and also states that \\\"A proposal to repeal and replace obamacare with the better care reconciliation act (bcra), a republican plan that was crafted over last two months, collapsed on tuesday night after nine republicans voted against it.\\\"\"\n}\n```"
+      },
+      {
+        "augmented": true,
+        "input": "{\"source\":\"Labour has launched an probe into complaints about members connected to a row over the move to change a primary school to welsh-medium education. A blogger posted screenshots suggesting a carmarthenshire councillor shared a facebook post by a far-right group. The post said: \\\"when a dog attacks a child it is put down. Shouldn't we do the same with sex predators?\\\" Tegwen devichand said she did not remember sharing the post and opposed the death penalty. Ms devichand, who is labour deputy group leader at carmarthenshire council, said: \\\"i couldn't put a horse down, let alone a human being.\\\" The post which ms devichand appears to have shared was initially posted by far-right group britain first. Ms devichand opposes plans to remove english-stream education at ysgol llangennech which were approved by carmarthenshire council in january. \\\"people should have choice, it shouldn't be forced on them,\\\" she said. The decision has been controversial with one labour councillor kim thomas describing it as \\\"segregation... at worst, apartheid\\\". The decision has been controversial with one labour councillor kim thomas describing it as \\\"segregation... at worst, apartheid\\\". Plaid cymru has accused labour members of working with ukip to undermine the council's decision. She also said labour had not contacted her about the matter and she had not been told the party was investigating. Ms devichand said she was complaining to the police about a \\\"hate campaign\\\" against her. On sunday labour said a different member connected to the row had been suspended. Plaid cymru carmarthen east and dinefwr mp jonathan edwards said over the course of the weekend \\\"we have seen labour members and an elected councillor exposed for sharing the propaganda of ultra-right groups. \\\"this is in addition to another elected councillor branding labour's own education policy as'apartheid'.\\\" \\\"launching an investigation into the llanelli labour party is a welcome first step, but the party's head office must take swift and immediate action to stamp this out this type of divisive campaigning.\\\" A welsh labour spokeswoman said: \\\"the labour party takes all complaints of this nature very seriously. The local party in llanelli have made us aware of complaints and an investigation is under way.\\\"\",\"statement\":\"Plaid cymru has called for an investigation into the welsh labour party after a councillor was accused of sharing propaganda.\"}",
+        "reasoning": "```json\n{\n  \"score\": 0.8,\n  \"reasons\": \"The statement is supported because Plaid Cymru has indeed called for an investigation into the Welsh Labour party after a councillor was accused of sharing propaganda, as stated in the source context. However, the score is not 1.0 because the specific details and quotes from Jonathan Edwards are not directly mentioned in the original statement.\"\n}\n```",
+        "output": "```json\n{\n  \"score\": 0.8,\n  \"reasons\": \"The statement is supported because Plaid Cymru has indeed called for an investigation into the Welsh Labour party after a councillor was accused of sharing propaganda, as stated in the source context. However, the score is not 1.0 because the specific details and quotes from Jonathan Edwards are not directly mentioned in the original statement.\"\n}\n```"
+      },
+      {
+        "augmented": true,
+        "input": "{\"source\":\"Ben murdoch-masila scored two tries as salford fought back to end castleford tigers' unbeaten start to the season. Luke gale put tigers ahead, grounding a loose ball after salford failed to handle a high kick. Murdoch-masila's try reduced the deficit to two as gareth o'brien missed the conversion, but greg minkin powered over to extend castleford's lead. Murdoch-masila scored his second try to get salford within two points, before o'brien's penalty and drop-goal won it. Castleford had been top of super league before the game after winning their opening four matches, but wigan's draw with huddersfield means they fall to second in the table. Tigers are the leading scorers in the competition this season, but they struggled to break down a resilient salford defence. O'brien missed his first attempt at a drop-goal, but castleford knocked on from the restart to allow the former tigers loanee a second effort. His kick earned salford their third win of the season, taking them up to fifth in the early table. Salford head coach ian watson told bbc radio manchester:. \\\"i just thought our team mentality and our group unity in what we set out to do this week was brilliant. \\\"i thought our middles were phenomenal, they controlled our pack. \\\"he [ben murdoch-masila] is a tough kid and he's a great player as well, we just need to get him up fully fit now and get him running properly.\\\" Castleford head coach daryl powell told bbc radio leeds:. \\\"i'm disappointed, it was a poor performance. \\\"it was a tight game, conditions dictated a fair bit, but i thought we just went into ourselves way too much in the second half. \\\"it was just one of those games where we just couldn't quite get going, but i thought they played really well. \\\"if we'd have got a draw at the end it would have probably been a travesty, i thought they deserved to win the game.\\\" Salford: o'brien; johnson, welham, sa'u, j carney; lui, dobson; walne, tomkins, mossop, murdoch-masila, griffin, flanagan. Replacements: brining, krasniqi, lopczak, t carney. Castleford: hardaker, minikin, webster, shenton, eden; roberts, gale; lynch, mcshane, massey, sene-lefao, mcmeeken, milner. Replacements: millington, moores, cook, springer.\",\"statement\":\"Ben murdoch-masila scored two tries as salford came from behind to beat super league leaders castleford.\"}",
+        "reasoning": "```json\n{\n  \"score\": 1.0,\n  \"reasons\": \"The statement is fully supported by the source context, as it accurately reports Ben Murdoch-Masila scoring two tries and Salford coming from behind to beat Castleford, which is explicitly stated in the article.\"\n}\n```",
+        "output": "```json\n{\n  \"score\": 1.0,\n  \"reasons\": \"The statement is fully supported by the source context, as it accurately reports Ben Murdoch-Masila scoring two tries and Salford coming from behind to beat Castleford, which is explicitly stated in the article.\"\n}\n```"
+      },
+      {
+        "query": "A senior member of france's opposition ump party has admitted there were \"anomalies\" in the accounts of nicolas sarkozy's 2012 presidential campaign. Jerome lavrilleux said the former president was not aware of the \"slip\". It comes after a lawyer for an event organiser accused the ump of ordering almost 11m euros ($15m; \u00e2\u00a39m) worth of fake invoices to cover overspends. Reports say the scandal could damage the party and complicate any efforts by mr sarkozy to be re-elected. Police searched the headquarters of the ump in paris on monday afternoon. \"there have been anomalies,\" mr lavrilleux, a deputy director of mr sarkozy's 2012 presidential campaign, told france's bfm tv. With tears in his eyes. \"there was no wrongdoing, there was a terrible spiral, a train going at high speed and people who should have pulled the emergency alarm and didn't, and i was probably one of them,\" he said. He said mr sarkozy, ump chairman jean-francois cope and batien millot, the founder of the events company bygmalion, were not aware of the actions taken. It comes after patrick maisonneuve, the lawyer for bygmalion, told the same television station that the centre-right ump had ordered fake invoices to cover exploding campaign costs. He said it was made clear to bygmalion that if it did not comply, it would not be paid. The ump's chairman was already facing growing speculation that he will have to step down after the french newspaper liberation reported this month that the party paid some 20m euros to a unit of bygmalion, founded by two associates of mr cope, to organise campaign events. Mr cope told bmf tv earlier on monday that he knew nothing of any wrongdoing. He said he had put his trust in the people who were responsible for the party's accounts and did not think he had to look over their shoulders. Mr cope was already expected to face challenges for the leadership of the party after the ump's poor result in the european parliament elections. It came second to the far-right front national. Mr sarkozy, who lost the presidency to francois hollande in 2012, is rumoured to be preparing to make a comeback in 2017. He has not commented on the scandal over his 2012 campaign finances.",
+        "expected_response": "The deputy director of nicolas sarkozy's 2012 presidential campaign has said his party's finances were in a `` terrible spiral''.",
+        "expected_score": 1.0,
+        "dspy_uuid": "3924fad8-bf06-47cc-965b-055168326394",
+        "dspy_split": "train"
+      },
+      {
+        "query": "Wales will use the pain of past failures as inspiration to reach the euro 2016 finals, says chris gunter. A point in bosnia-herzegovina on saturday or at home to andorra three days later would see wales end their 57-year absence from major tournaments. Gunter, 26, believes the current side can deliver where previous generations have fallen short. \"it's been a hard time. We aren't the biggest nation but we've probably underachieved,\" said the defender. \"i remember romania [1993] and the russia campaign [2003], the closer we get over the years, the more these questions seem to come. \"it's nice to be involved in the right sort of scenarios towards the end of the group. We played belgium when they celebrated getting to the [2014] world cup and we've put ourselves in that position now.\" Wales are a point clear at the summit of group b, with the top two teams qualifying automatically for euro 2016. Even if they lose their final two qualifiers - unlikely as andorra have never won a competitive game away from home - wales would still reach next summer's tournament if israel fail to win both of their remaining fixtures against cyprus and group favourites belgium. With the insurance of a home game against andorra on tuesday, the odds of qualification are overwhelmingly in wales' favour. But gunter is mindful of previous failures and says wales will be aiming to seal their place in france with a positive result against bosnia on saturday. \"the near misses are there as a reminder that in football even the most unlikely thing can happen until the job is done,\" he added. \"we've worked so hard for so long, it'd be a massive mistake to get complacent and think the job is done. \"if we start thinking we'll be celebrating on saturday night we'll be caught out. It's a really tough game. We have done well because we've always prepared in the right way. \"until we've qualified we have to concentrate as anything can happen.\" Gareth bale has been integral to wales' recent success, scoring six of their nine goals in this campaign, and a calf injury had made the real madrid forward a doubt for the forthcoming double-header. But he returned for his club side against atletico madrid on sunday and is expected to feature against bosnia and andorra. \"gareth is here, he's fit, he's done the same as everybody else. It's brilliant news for us and the gaffer [chris coleman] as well,\" said gunter. \"i don't think if he's fit and here we'll look beyond the game at the weekend. It'd be a brave man to tell him he's not playing saturday! \"if he's fit i'm sure he'll be involved and hopefully in the same form as the rest of the campaign.\"",
+        "expected_response": "Wales defender chris gunter says it would be a `` massive mistake'' to get complacent as they close in on euro 2016.",
+        "expected_score": 1.0,
+        "dspy_uuid": "b9df7de7-7290-4f2e-afd0-7f8f72bff3d8",
+        "dspy_split": "train"
+      }
+    ],
+    "signature": {
+      "instructions": "Based on the provided code, it appears that you're working on a DSPy module named `GroundednessDSPy`. The task is to evaluate if every sentence in the statement (except trivial ones) is supported or entailed by the source context. \n\nTo accomplish this, we'll need to:\n\n1. Define the signature of the DSPy module.\n2. Implement the logic for groundedness analysis.\n\nHere's how you could complete these tasks based on your code snippet:\n\n```python\nclass GroundednessSignature(dspy.Signature):\n    def __init__(self):\n        super().__init__(\n            input_fields=[\n                dspy.InputField(\n                    prefix=\"Source\",\n                    desc=\"The context that may or may not support the statement.\",",
+      "fields": [
+        {
+          "prefix": "Input:",
+          "description": "${input}"
+        },
+        {
+          "prefix": "Reasoning: Let's think step by step in order to",
+          "description": "${produce the output}. We ..."
+        },
+        {
+          "prefix": "Output:",
+          "description": "${output}"
+        }
+      ]
+    }
+  }
+}

examples/experimental/mipro_zeroshot_optimized

@@ -0,0 +1,25 @@
+{
+  "generate_score_and_reasons.predictor": {
+    "lm": null,
+    "traces": [],
+    "train": [],
+    "demos": [],
+    "signature": {
+      "instructions": "Your task is to evaluate if every sentence in the statement (except the trivial ones like stylistic sentences) is supported or entailed by the source context.\nGenerate a score the scale of 0.0 to 1.0 with reasons for the score.",
+      "fields": [
+        {
+          "prefix": "Input:",
+          "description": "${input}"
+        },
+        {
+          "prefix": "Reasoning: Let's think step by step in order to",
+          "description": "${produce the output}. We ..."
+        },
+        {
+          "prefix": "Output:",
+          "description": "${output}"
+        }
+      ]
+    }
+  }
+}