Extractive QA Datasets

[pintersebastian]

Hi! In the assignment description for the Extractive QA part, it says we should use the top 1 re-ranking result on the MS MARCO test set, but I have problems finding said set.

Anybody got a link or can point me to the set?

The closest thing i found was: https://microsoft.github.io/msmarco/Datasets.html

Kind regards,
Sebastian

[ldosen00]

Hello,
If you go back a bit on the website you were on you can see the leaderboards here: https://microsoft.github.io/msmarco/
There you can choose the top result.

Best,
Luka

[pintersebastian]

Thanks for the reply!

I have already seen the ranking, but unfortunately i don't see any link where we can download a dataset, I really must have missed it.

Kind Regards,
Sebastian

[pi-pa]

Sorry for the confusion, only take this part (taken from the instructions) concerning the evaluation of Exercise 2.3 into account:

Evaluate both your top-1 (or more) MSMARCO passage results from the best re-ranking model using msmarco-fira-21.qrels.qa-answers.tsv (only evaluate the overlap of pairs that are in the result and the qrels) + the provided FiRA gold-label pairs msmarco-fira-21.qrels.qa-tuples.tsv using the provided qa evaluation methods in core_metrics.py with the MSMARCO-FiRA-2021 QA labels.

[pintersebastian]

I am sry, I am still slightly confused...

So, for clarification, there are 2 things to do in regards to evaluation:

1. Use the re-ranking result of our OWN model from part 2 and use the top-1 most relevant document (according to our best performing models) and msmarco-fira-21.qrels.qa-answers.tsv as labels.
2. We use msmarco-fira-21.qrels.qa-tuples.tsv and msmarco-fira-21.qrels.qa-answers.tsv as labels.

Both are evaluated using the qa-method in core_metrics.py ?

Sry if i am asking same questions twice, but the wording really confuses me.

Kind Regards,
Sebastian

[whoopi24]

Hi Sebastian,
I am not a tutor, so I am not 100% sure if my suggestion is correct. I have read the following discussion #101 and actually I would answer both your questions with "yes". This means that 1. is just a subset of 2., right?

I was wondering if 2. is really the correct approach since it will take a lot of time to run the QA pipeline for 52606 observations. Maybe some of the tutors could be so nice and answer soon, since the deadline is in less than a week.

Thank you and best regards,
Marina