added quantemp project info

avishekanand · Jul 2, 2024 · 19dcd4e · 19dcd4e
1 parent f806f1f
commit 19dcd4e
Show file tree

Hide file tree

Showing 18 changed files with 126 additions and 135 deletions.
diff --git a/_projects/quantemp_project.md b/_projects/quantemp_project.md
@@ -25,18 +25,81 @@ evaluation set for numerical claim verification.
 [Paper](https://arxiv.org/abs/2403.17169)
 
 
+# QuanTemp
 
-{% raw %}
-
-```html
-<div class="row justify-content-sm-center">
-  <div class="col-sm-8 mt-3 mt-md-0">
-    {% include figure.liquid path="assets/img/6.jpg" title="example image" class="img-fluid rounded z-depth-1" %}
-  </div>
-  <div class="col-sm-4 mt-3 mt-md-0">
-    {% include figure.liquid path="assets/img/11.jpg" title="example image" class="img-fluid rounded z-depth-1" %}
-  </div>
-</div>
+A benchmark of Quantitative and Temporal Claims. This repository contains data and code for Numerical Claims. It includes the first set of real-world numerical claims, a fine-grained taxonomy for numerical claims (statistical, temporal, comparison, and interval), and inference code for reproducing the best results in the paper. The repository also includes trained models and collected evidence snippets.
+
+
+## Example Dataset
+
+```json
+{
+        "country_of_origin": "usa",
+        "label": "Conflicting",
+        "url": "https://www.politifact.com/factchecks/2010/aug/08/donald-carcieri/carcieri-says-tax-repeal-spawned-new-business/",
+        "lang": "en",
+        "claim": "Repealing the sales tax on boats in Rhode Island has spawned 2,000 companies, 7,000 jobs and close to $2 billion a year in sales activity.",
+        "doc": "The furor over U.S. Sen. John Kerry\u2019s yacht being docked on the tax-free shores of Rhode Island -- not in Massachusetts where he lives -- has subsided now that the senator\u2019s team has promised he will pay the $500,000 in owed taxes.But Rhode Island Governor Donald Carcieri did his part to make sure the underlying issue of Rhode Island\u2019s tax exemption for boats is not forgotten.",
+        "taxonomy_label": "statistical",
+        "label_original": "half-true"
+    }
+```
+
+The pipeline consists of claim decomposition, evidence retrieval and stance detection steps.
+
+##Setup
+
+You can simply do a pip install to setup. 
+
+```bash
+pip install -r requirements.txt
 ```
 
-{% endraw %}
+Download the models from [here](https://drive.google.com/drive/folders/1FmaelDhJ7QwsRTs8H0B4vYliw_qjL7P-?usp=sharing) and dump it into models folder
+
+
+The following is the project structure We release the BM25 results in this repo. However, for those interested the corpus is at the link below to encourage reporducability and need not be downloaded from search engines again. We release this as search results may drift over time.
+
+[Link to Corpus](https://drive.google.com/drive/folders/1GYzSK0oU2MiaKbyBO3hE8kO4gdmxDjCv?usp=drive_link)
+
+
+#File Structure
+
+The file structure for the resources are in the following structure
+
+```
+├── data
+│   ├── decomposed_questions
+│   │   ├── test_claimdecomp.csv
+│   │   └── test_programfc.json
+│   ├── raw_data
+│   │   ├── test_claims_quantemp.json
+│   │   ├── train_claims_quantemp.json
+│   │   └── val_claims_quantemp.json
+│   └── bm25_scored_evidence
+│       ├── bm25_top_100_claimdecomp.json
+│       
+├── code
+│    ├── data_processing
+│        ├──
+│    │   ├── test_claimdecomp.csv
+│    │   └── test_programfc.json
+│    ├── utils
+│    │   ├── load_veracity_predictor.py
+│    ├── nli_inference
+│    │   ├── veracity_prediction.py
+├── requirements.txt
+└── README.md
+```
+
+To reproduce results on paper for finqa-roberta-large (ELASTIC) run
+
+```bash
+python3 code/nli_inference/veracity_prediction.py --test_path data/raw_data/test_claims_quantemp.json --bm25_evidence_path data/bm25_scored_evidence/bm25_top_100_claimdecomp.json --base_model roberta-large-mnli --model_path models/finqa_roberta_claimdecomp_early_stop_2/model_weights.zip --questions_path data/decomposed_questions/test/test_claimdecomp.csv --output_path finqa_roberta_claimdecomp
+```
+
+followed by
+
+```bash
+python3 code/evaluation/eval_veracity_prediction.py --output_path output/finqa_roberta_claimdecomp.csv
+```
diff --git a/_projects/suwr_project.md b/_projects/suwr_project.md
@@ -1,80 +1,8 @@
 ---
 layout: page
-title: project 9
-description: another project with an image 🎉
+title: SUWR
+description: Interpretability by design 
 img: assets/img/6.jpg
 importance: 4
 category: fun
 ---
-
-Every project has a beautiful feature showcase page.
-It's easy to include images in a flexible 3-column grid format.
-Make your photos 1/3, 2/3, or full width.
-
-To give your project a background in the portfolio page, just add the img tag to the front matter like so:
-
-    ---
-    layout: page
-    title: project
-    description: a project with a background image
-    img: /assets/img/12.jpg
-    ---
-
-<div class="row">
-    <div class="col-sm mt-3 mt-md-0">
-        {% include figure.liquid loading="eager" path="assets/img/1.jpg" title="example image" class="img-fluid rounded z-depth-1" %}
-    </div>
-    <div class="col-sm mt-3 mt-md-0">
-        {% include figure.liquid loading="eager" path="assets/img/3.jpg" title="example image" class="img-fluid rounded z-depth-1" %}
-    </div>
-    <div class="col-sm mt-3 mt-md-0">
-        {% include figure.liquid loading="eager" path="assets/img/5.jpg" title="example image" class="img-fluid rounded z-depth-1" %}
-    </div>
-</div>
-<div class="caption">
-    Caption photos easily. On the left, a road goes through a tunnel. Middle, leaves artistically fall in a hipster photoshoot. Right, in another hipster photoshoot, a lumberjack grasps a handful of pine needles.
-</div>
-<div class="row">
-    <div class="col-sm mt-3 mt-md-0">
-        {% include figure.liquid loading="eager" path="assets/img/5.jpg" title="example image" class="img-fluid rounded z-depth-1" %}
-    </div>
-</div>
-<div class="caption">
-    This image can also have a caption. It's like magic.
-</div>
-
-You can also put regular text between your rows of images.
-Say you wanted to write a little bit about your project before you posted the rest of the images.
-You describe how you toiled, sweated, _bled_ for your project, and then... you reveal its glory in the next row of images.
-
-<div class="row justify-content-sm-center">
-    <div class="col-sm-8 mt-3 mt-md-0">
-        {% include figure.liquid path="assets/img/6.jpg" title="example image" class="img-fluid rounded z-depth-1" %}
-    </div>
-    <div class="col-sm-4 mt-3 mt-md-0">
-        {% include figure.liquid path="assets/img/11.jpg" title="example image" class="img-fluid rounded z-depth-1" %}
-    </div>
-</div>
-<div class="caption">
-    You can also have artistically styled 2/3 + 1/3 images, like these.
-</div>
-
-The code is simple.
-Just wrap your images with `<div class="col-sm">` and place them inside `<div class="row">` (read more about the <a href="https://getbootstrap.com/docs/4.4/layout/grid/">Bootstrap Grid</a> system).
-To make images responsive, add `img-fluid` class to each; for rounded corners and shadows use `rounded` and `z-depth-1` classes.
-Here's the code for the last row of images above:
-
-{% raw %}
-
-```html
-<div class="row justify-content-sm-center">
-  <div class="col-sm-8 mt-3 mt-md-0">
-    {% include figure.liquid path="assets/img/6.jpg" title="example image" class="img-fluid rounded z-depth-1" %}
-  </div>
-  <div class="col-sm-4 mt-3 mt-md-0">
-    {% include figure.liquid path="assets/img/11.jpg" title="example image" class="img-fluid rounded z-depth-1" %}
-  </div>
-</div>
-```
-
-{% endraw %}
diff --git a/_site/404.html b/_site/404.html
@@ -593,9 +593,9 @@ <h1 class="post-title">Page not found</h1>
               description: "An open-domain benchmark to verify claims with quantities and temporal expressions",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/quantemp_project/";
-                },},{id: "projects-project-9",
-              title: 'project 9',
-              description: "another project with an image 🎉",
+                },},{id: "projects-suwr",
+              title: 'SUWR',
+              description: "Interpretability by design",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/suwr_project/";
                 },},{

diff --git a/_site/_pages/dropdown/index.html b/_site/_pages/dropdown/index.html
@@ -592,9 +592,9 @@ <h1 class="post-title">submenus</h1>
               description: "An open-domain benchmark to verify claims with quantities and temporal expressions",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/quantemp_project/";
-                },},{id: "projects-project-9",
-              title: 'project 9',
-              description: "another project with an image 🎉",
+                },},{id: "projects-suwr",
+              title: 'SUWR',
+              description: "Interpretability by design",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/suwr_project/";
                 },},{

diff --git a/_site/assets/jupyter/blog.ipynb.html b/_site/assets/jupyter/blog.ipynb.html
@@ -3,7 +3,7 @@
 <html lang="en">
 <head><meta charset="utf-8"/>
 <meta content="width=device-width, initial-scale=1.0" name="viewport"/>
-<title>jekyll-jupyter-notebook20240702-37017-2j1pyw</title><script src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.1.10/require.min.js"></script>
+<title>jekyll-jupyter-notebook20240702-37017-2bmyzw</title><script src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.1.10/require.min.js"></script>
 <style type="text/css">
     pre { line-height: 125%; }
 td.linenos .normal { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }

diff --git a/_site/blog/2022/index.html b/_site/blog/2022/index.html
@@ -601,9 +601,9 @@ <h1 class="post-title">
               description: "An open-domain benchmark to verify claims with quantities and temporal expressions",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/quantemp_project/";
-                },},{id: "projects-project-9",
-              title: 'project 9',
-              description: "another project with an image 🎉",
+                },},{id: "projects-suwr",
+              title: 'SUWR',
+              description: "Interpretability by design",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/suwr_project/";
                 },},{

diff --git a/_site/blog/2024/index.html b/_site/blog/2024/index.html
@@ -601,9 +601,9 @@ <h1 class="post-title">
               description: "An open-domain benchmark to verify claims with quantities and temporal expressions",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/quantemp_project/";
-                },},{id: "projects-project-9",
-              title: 'project 9',
-              description: "another project with an image 🎉",
+                },},{id: "projects-suwr",
+              title: 'SUWR',
+              description: "Interpretability by design",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/suwr_project/";
                 },},{

diff --git a/_site/blog/index.html b/_site/blog/index.html
@@ -741,9 +741,9 @@ <h3>
               description: "An open-domain benchmark to verify claims with quantities and temporal expressions",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/quantemp_project/";
-                },},{id: "projects-project-9",
-              title: 'project 9',
-              description: "another project with an image 🎉",
+                },},{id: "projects-suwr",
+              title: 'SUWR',
+              description: "Interpretability by design",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/suwr_project/";
                 },},{

diff --git a/_site/cv/index.html b/_site/cv/index.html
@@ -1519,9 +1519,9 @@ <h6 class="ml-1 ml-md-4" style="font-size: 0.95rem; font-style: italic">Quantum
               description: "An open-domain benchmark to verify claims with quantities and temporal expressions",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/quantemp_project/";
-                },},{id: "projects-project-9",
-              title: 'project 9',
-              description: "another project with an image 🎉",
+                },},{id: "projects-suwr",
+              title: 'SUWR',
+              description: "Interpretability by design",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/suwr_project/";
                 },},{

diff --git a/_site/feed.xml b/_site/feed.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="4.3.3">Jekyll</generator><link href="http://localhost:4000/feed.xml" rel="self" type="application/atom+xml" /><link href="http://localhost:4000/" rel="alternate" type="text/html" hreflang="en" /><updated>2024-07-02T09:54:13+02:00</updated><id>http://localhost:4000/feed.xml</id><title type="html">Avishek Anand</title><subtitle>This is the homepage&gt;</subtitle><entry><title type="html">Google Gemini updates: Flash 1.5, Gemma 2 and Project Astra</title><link href="http://localhost:4000/blog/2024/google-gemini-updates-flash-15-gemma-2-and-project-astra/" rel="alternate" type="text/html" title="Google Gemini updates: Flash 1.5, Gemma 2 and Project Astra" /><published>2024-05-14T00:00:00+02:00</published><updated>2024-05-14T00:00:00+02:00</updated><id>http://localhost:4000/blog/2024/google-gemini-updates-flash-15-gemma-2-and-project-astra</id><content type="html" xml:base="http://localhost:4000/blog/2024/google-gemini-updates-flash-15-gemma-2-and-project-astra/"><![CDATA[]]></content><author><name></name></author><summary type="html"><![CDATA[We’re sharing updates across our Gemini family of models and a glimpse of Project Astra, our vision for the future of AI assistants.]]></summary></entry><entry><title type="html">Displaying External Posts on Your al-folio Blog</title><link href="http://localhost:4000/blog/2022/displaying-external-posts-on-your-al-folio-blog/" rel="alternate" type="text/html" title="Displaying External Posts on Your al-folio Blog" /><published>2022-04-24T01:20:09+02:00</published><updated>2022-04-24T01:20:09+02:00</updated><id>http://localhost:4000/blog/2022/displaying-external-posts-on-your-al-folio-blog</id><content type="html" xml:base="http://localhost:4000/blog/2022/displaying-external-posts-on-your-al-folio-blog/"><![CDATA[]]></content><author><name></name></author></entry></feed>
+<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="4.3.3">Jekyll</generator><link href="http://localhost:4000/feed.xml" rel="self" type="application/atom+xml" /><link href="http://localhost:4000/" rel="alternate" type="text/html" hreflang="en" /><updated>2024-07-02T17:19:28+02:00</updated><id>http://localhost:4000/feed.xml</id><title type="html">Avishek Anand</title><subtitle>This is the homepage&gt;</subtitle><entry><title type="html">Google Gemini updates: Flash 1.5, Gemma 2 and Project Astra</title><link href="http://localhost:4000/blog/2024/google-gemini-updates-flash-15-gemma-2-and-project-astra/" rel="alternate" type="text/html" title="Google Gemini updates: Flash 1.5, Gemma 2 and Project Astra" /><published>2024-05-14T00:00:00+02:00</published><updated>2024-05-14T00:00:00+02:00</updated><id>http://localhost:4000/blog/2024/google-gemini-updates-flash-15-gemma-2-and-project-astra</id><content type="html" xml:base="http://localhost:4000/blog/2024/google-gemini-updates-flash-15-gemma-2-and-project-astra/"><![CDATA[]]></content><author><name></name></author><summary type="html"><![CDATA[We’re sharing updates across our Gemini family of models and a glimpse of Project Astra, our vision for the future of AI assistants.]]></summary></entry><entry><title type="html">Displaying External Posts on Your al-folio Blog</title><link href="http://localhost:4000/blog/2022/displaying-external-posts-on-your-al-folio-blog/" rel="alternate" type="text/html" title="Displaying External Posts on Your al-folio Blog" /><published>2022-04-24T01:20:09+02:00</published><updated>2022-04-24T01:20:09+02:00</updated><id>http://localhost:4000/blog/2022/displaying-external-posts-on-your-al-folio-blog</id><content type="html" xml:base="http://localhost:4000/blog/2022/displaying-external-posts-on-your-al-folio-blog/"><![CDATA[]]></content><author><name></name></author></entry></feed>
diff --git a/_site/index.html b/_site/index.html
@@ -1688,9 +1688,9 @@ <h2>
               description: "An open-domain benchmark to verify claims with quantities and temporal expressions",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/quantemp_project/";
-                },},{id: "projects-project-9",
-              title: 'project 9',
-              description: "another project with an image 🎉",
+                },},{id: "projects-suwr",
+              title: 'SUWR',
+              description: "Interpretability by design",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/suwr_project/";
                 },},{

diff --git a/_site/news/index.html b/_site/news/index.html
@@ -654,9 +654,9 @@ <h1 class="post-title">News</h1>
               description: "An open-domain benchmark to verify claims with quantities and temporal expressions",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/quantemp_project/";
-                },},{id: "projects-project-9",
-              title: 'project 9',
-              description: "another project with an image 🎉",
+                },},{id: "projects-suwr",
+              title: 'SUWR',
+              description: "Interpretability by design",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/suwr_project/";
                 },},{

diff --git a/_site/people/index.html b/_site/people/index.html
@@ -938,9 +938,9 @@ <h1 class="post-title">team</h1>
               description: "An open-domain benchmark to verify claims with quantities and temporal expressions",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/quantemp_project/";
-                },},{id: "projects-project-9",
-              title: 'project 9',
-              description: "another project with an image 🎉",
+                },},{id: "projects-suwr",
+              title: 'SUWR',
+              description: "Interpretability by design",
               section: "Projects",handler: () => {
                   window.location.href = "/projects/suwr_project/";
                 },},{
Original file line number	Diff line number	Diff line change
		@@ -1 +1 @@
		<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="4.3.3">Jekyll</generator><link href="http://localhost:4000/feed.xml" rel="self" type="application/atom+xml" /><link href="http://localhost:4000/" rel="alternate" type="text/html" hreflang="en" /><updated>2024-07-02T09:54:13+02:00</updated><id>http://localhost:4000/feed.xml</id><title type="html">Avishek Anand</title><subtitle>This is the homepage></subtitle><entry><title type="html">Google Gemini updates: Flash 1.5, Gemma 2 and Project Astra</title><link href="http://localhost:4000/blog/2024/google-gemini-updates-flash-15-gemma-2-and-project-astra/" rel="alternate" type="text/html" title="Google Gemini updates: Flash 1.5, Gemma 2 and Project Astra" /><published>2024-05-14T00:00:00+02:00</published><updated>2024-05-14T00:00:00+02:00</updated><id>http://localhost:4000/blog/2024/google-gemini-updates-flash-15-gemma-2-and-project-astra</id><content type="html" xml:base="http://localhost:4000/blog/2024/google-gemini-updates-flash-15-gemma-2-and-project-astra/"><![CDATA[]]></content><author><name></name></author><summary type="html"><![CDATA[We’re sharing updates across our Gemini family of models and a glimpse of Project Astra, our vision for the future of AI assistants.]]></summary></entry><entry><title type="html">Displaying External Posts on Your al-folio Blog</title><link href="http://localhost:4000/blog/2022/displaying-external-posts-on-your-al-folio-blog/" rel="alternate" type="text/html" title="Displaying External Posts on Your al-folio Blog" /><published>2022-04-24T01:20:09+02:00</published><updated>2022-04-24T01:20:09+02:00</updated><id>http://localhost:4000/blog/2022/displaying-external-posts-on-your-al-folio-blog</id><content type="html" xml:base="http://localhost:4000/blog/2022/displaying-external-posts-on-your-al-folio-blog/"><![CDATA[]]></content><author><name></name></author></entry></feed>
		<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="4.3.3">Jekyll</generator><link href="http://localhost:4000/feed.xml" rel="self" type="application/atom+xml" /><link href="http://localhost:4000/" rel="alternate" type="text/html" hreflang="en" /><updated>2024-07-02T17:19:28+02:00</updated><id>http://localhost:4000/feed.xml</id><title type="html">Avishek Anand</title><subtitle>This is the homepage></subtitle><entry><title type="html">Google Gemini updates: Flash 1.5, Gemma 2 and Project Astra</title><link href="http://localhost:4000/blog/2024/google-gemini-updates-flash-15-gemma-2-and-project-astra/" rel="alternate" type="text/html" title="Google Gemini updates: Flash 1.5, Gemma 2 and Project Astra" /><published>2024-05-14T00:00:00+02:00</published><updated>2024-05-14T00:00:00+02:00</updated><id>http://localhost:4000/blog/2024/google-gemini-updates-flash-15-gemma-2-and-project-astra</id><content type="html" xml:base="http://localhost:4000/blog/2024/google-gemini-updates-flash-15-gemma-2-and-project-astra/"><![CDATA[]]></content><author><name></name></author><summary type="html"><![CDATA[We’re sharing updates across our Gemini family of models and a glimpse of Project Astra, our vision for the future of AI assistants.]]></summary></entry><entry><title type="html">Displaying External Posts on Your al-folio Blog</title><link href="http://localhost:4000/blog/2022/displaying-external-posts-on-your-al-folio-blog/" rel="alternate" type="text/html" title="Displaying External Posts on Your al-folio Blog" /><published>2022-04-24T01:20:09+02:00</published><updated>2022-04-24T01:20:09+02:00</updated><id>http://localhost:4000/blog/2022/displaying-external-posts-on-your-al-folio-blog</id><content type="html" xml:base="http://localhost:4000/blog/2022/displaying-external-posts-on-your-al-folio-blog/"><![CDATA[]]></content><author><name></name></author></entry></feed>