Part 1 data format and content

[waffelbaellchen]

Hello,

just to avoid misunderstandings, would it be possible to provide further information on the format and content of some files provided for part 1 of the assignment? I hope that I haven't overlooked any further details, but I am currently stewing over the format and content of the file fira-22.baseline-qrels.tsv

• is it possible that this file is accidentally not tab-separated but space-separated?
• what does the column hardcoded-Q0 refer to?
• what does the column relevance-grade refer to? Does it maybe show the simple majority voting or heuristic based on the raw judgements?

Thanks a lot in advance for any feedback!

[sophiaalthammer]

Hello,

no worries, to your questions:

• yes the file is space-separated
• the hardcoded Q0 column doesnt refer to anything, this is the officel TREC qrels (label file) format, which is commonly used for IR qrel files
• the relevance grade is the relevance grade attained from the raw judgements: if there is perfect agreement, then the relevance grade is the one which everyone annotated, if there is not perfect agreement, but there exists a majority label in the voting, it is that majority label, otherwise if there is no agreement and no majority, the highest relevane label is chosen as heuristic

I hope that answers your questions!

Best,
Sophia