'dpr_all_documents' is not defined

[golubovic]

Hi,

I experience issue with global variable ‘dpr_all_documents’ involving tokenizer parallelism, please see logs below. This issue has been raised before for DPR repo.

Note that:

1. all_docs size has value as expected (I use test document(s) of 5 entries, for testing purposes rather than wiki dataset, please see the log below)
2. validation_workers is set to 1 in dense_retreiver.yaml (saying that, that setting isn't a problem, I have set it to one just as a safety measure)
3. I have tried setting TOKENIZERS_PARALLELISM=false (doesn't make a difference). NOTE: Transformers library "0.8.0rc4" has issue with this setting not taking effect currently
4. I have tried downgrading transformers and tokenizers library to previous versions (no success), good article/comment by [Allohvk] on what is going on with RUST tokenizers used by Huggingface can be found in here https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning
5. I have tried refactoring dpr_all_documents and passing it as a regular method/function parameter and removing ‘global’ definition, that however results in ‘KeyError’ exception for the given id_prefix of the defined datasource in default_sources.yaml

Please let me know if you have any questions.

Thanks,
Mladen

Logs:
[2023-09-21 07:35:40,997][dpr.models.hf_models][INFO] - Initializing HF BERT Encoder. cfg_name=bert-base-uncased
[2023-09-21 07:35:43,260][dpr.models.hf_models][INFO] - Initializing HF BERT Encoder. cfg_name=bert-base-uncased
[2023-09-21 07:35:44,405][root][INFO] - Loading saved model state ...
[2023-09-21 07:35:44,611][root][INFO] - Selecting standard question encoder
[2023-09-21 07:35:44,677][root][INFO] - Encoder vector_size=768
[2023-09-21 07:35:44,677][root][INFO] - qa_dataset: dpr_ds_retreiving_questions
[2023-09-21 07:35:44,680][root][INFO] - questions len 6
[2023-09-21 07:35:44,680][root][INFO] - questions_text len 0
[2023-09-21 07:35:44,680][root][INFO] - Local Index class <class 'dpr.indexer.faiss_indexers.DenseFlatIndexer'>
[2023-09-21 07:35:44,680][root][INFO] - Using special token None
[2023-09-21 07:35:45,875][root][INFO] - Total encoded queries tensor torch.Size([6, 768])
[2023-09-21 07:35:45,877][root][INFO] - ctx_sources: <class 'dpr.data.retriever_data.CsvCtxSrc'>
[2023-09-21 07:35:45,877][root][INFO] - id_prefixes per dataset: ['ds_default_sources_yaml_prefix:']
[2023-09-21 07:35:45,877][root][INFO] - ctx_files_patterns: ['/Users/directory/Developer/DPR-main/checkpoints/generated_embeddings_0']
[2023-09-21 07:35:45,878][root][INFO] - Embeddings files id prefixes: ['ds_default_sources_yaml_prefix:']
[2023-09-21 07:35:45,878][root][INFO] - Reading all passages data from files: ['/Users/directory/Developer/DPR-main/checkpoints/generated_embeddings_0']
[2023-09-21 07:35:45,878][root][INFO] - Reading file /Users/directory/Developer/DPR-main/checkpoints/generated_embeddings_0
[2023-09-21 07:35:45,880][root][INFO] - data indexed 5
[2023-09-21 07:35:45,880][root][INFO] - Total data indexed 5
[2023-09-21 07:35:45,880][root][INFO] - Data indexing completed.
[2023-09-21 07:35:45,880][root][INFO] - Serializing index to /Users/directory/Developer/DPR-main/checkpoints/faiss_index_ctx
[2023-09-21 07:35:45,883][root][INFO] - index search time: 0.002260 sec.
[2023-09-21 07:35:45,884][dpr.data.retriever_data][INFO] - Reading file /Users/directory/Developer/DPR-main/dpr/downloads/data/wikipedia_split/psgs_w100-s.tsv
[2023-09-21 07:35:45,885][root][INFO] - Loaded ctx data: 5
[2023-09-21 07:35:45,885][root][INFO] - validating passages. size=5
[2023-09-21 07:35:45,885][dpr.data.qa_validation][INFO] - all_docs size 5
[2023-09-21 07:35:45,885][dpr.data.qa_validation][INFO] - dpr_all_documents size 5
[2023-09-21 07:35:45,925][dpr.data.qa_validation][INFO] - Matching answers in top docs...
2023-09-21 07:35:49,689 [INFO] faiss.loader: Loading faiss with AVX2 support.
2023-09-21 07:35:49,717 [INFO] faiss.loader: Successfully loaded faiss with AVX2 support.
/Users/directory/Developer/DPR-main/dense_retriever.py:472: UserWarning:
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
@hydra.main(config_path="conf", config_name="dense_retriever")
Error executing job with overrides: []
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/Users/directory/.pyenv/versions/3.9.12/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/Users/directory/.pyenv/versions/3.9.12/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/Users/directory/Developer/DPR-main/dpr/data/qa_validation.py", line 127, in check_answer
doc = dpr_all_documents[doc_id]
NameError: name 'dpr_all_documents' is not defined
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/directory/Developer/DPR-main/dense_retriever.py", line 628, in main
questions_doc_hits = validate(
File "/Users/directory/Developer/DPR-main/dense_retriever.py", line 309, in validate
match_stats = calculate_matches(passages, answers, result_ctx_ids, workers_num, match_type)
File "/Users/directory/Developer/DPR-main/dpr/data/qa_validation.py", line 68, in calculate_matches
scores = processes.map(get_score_partial, questions_answers_docs)
File "/Users/directory/.pyenv/versions/3.9.12/lib/python3.9/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/Users/directory/.pyenv/versions/3.9.12/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
NameError: name 'dpr_all_documents' is not defined

[Hannibal046]

The dpr_all_documents is defined here and it works on my side:
https://github.com/facebookresearch/DPR/blob/a31212dc0a54dfa85d8bfa01e1669f149ac832b7/dpr/data/qa_validation.py#L56C1-L57

[golubovic]

Issue with dpr_all_documents arises when running densre_retreiver.py with small input dataset. Log example above gives input dataset of six questions in total. When dataset is very small this issue surfaces out and dpr_all_documents is not available to all processes which try to access it.

Simple (and not optimal) workaround is to pass the variable in calculate_matches function as additional parameter (please see below). Of course that implies inefficient use of memory as a consequence.

get_score_partial = partial(check_answer, match_type=match_type, tokenizer=tokenizer,dpr_all_documents=dpr_all_documents)