Skip to content

Commit

Permalink
Merge from MantisAI/fix/failing-CICD
Browse files Browse the repository at this point in the history
Fix/failing cicd
  • Loading branch information
infopz authored Oct 4, 2023
2 parents 787e957 + 3caba0b commit 21302ac
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 5 deletions.
4 changes: 3 additions & 1 deletion .github/workflows/checks.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
name: CI Code Checks
on: [pull_request]
permissions:
contents: write
jobs:
build:
name: code checks
Expand Down Expand Up @@ -41,4 +43,4 @@ jobs:
folder: coverage

- name: Type checking with mypy
run: mypy --config setup.cfg src
run: mypy --config setup.cfg src
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ We can use the following five metrics to consider difference categories of error
|Incorrect (INC)|the output of a system and the golden annotation don’t match|
|Partial (PAR)|system and the golden annotation are somewhat “similar” but not the same|
|Missing (MIS)|a golden annotation is not captured by a system|
|Spurius (SPU)|system produces a response which doesn’t exit in the golden annotation|
|Spurious (SPU)|system produces a response which doesn’t exit in the golden annotation|

These five metrics can be measured in four different ways:

Expand Down Expand Up @@ -172,7 +172,7 @@ __Putting all together:__
|Incorrect|2|0|2|3|
|Partial|0|2|0|0|
|Missed|1|1|1|1|
|Spurius|1|1|1|1|
|Spurious|1|1|1|1|
|Precision|0.5|0.66|0.5|0.33|
|Recall|0.5|0.66|0.5|0.33|
|F1|0.5|0.66|0.5|0.33|
Expand All @@ -182,9 +182,9 @@ __Putting all together:__

In scenarios IV and VI the entity type of the `true` and `pred` does not match, in both cases we only scored against
the true entity, not the predicted one. You can argue that the predicted entity could also be scored as spurious,
but according to the definition of `spurius`:
but according to the definition of `spurious`:

* Spurius (SPU) : system produces a response which does not exist in the golden annotation;
* Spurious (SPU) : system produces a response which does not exist in the golden annotation;

In this case there exists an annotation, but with a different entity type, so we assume it's only incorrect.

Expand Down

0 comments on commit 21302ac

Please sign in to comment.