Issue when more than one user accesses the service

[clang88]

Mateo has issues when more than one person or session is used for "Evaluate" at the same time. It usually outputs an error like the following (but sometimes others, like "File not found".

TypeError: 'NoneType' object is not iterable
Traceback:

File "/home/user/mateo-demo/src/mateo_st/pages/03_📏_Evaluate.py", line 432, in _compute_metrics
    result = _compute_metric(
File "/home/user/.local/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 194, in wrapper
    return cached_func(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 225, in __call__
    return self._get_or_create_cached_value(args, kwargs)
File "/home/user/.local/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 248, in _get_or_create_cached_value
    return self._handle_cache_miss(cache, value_key, func_args, func_kwargs)
File "/home/user/.local/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 302, in _handle_cache_miss
    computed_value = self._info.func(*func_args, **func_kwargs)
File "/home/user/mateo-demo/src/mateo_st/pages/03_📏_Evaluate.py", line 391, in _compute_metric
    raise exc
File "/home/user/mateo-demo/src/mateo_st/pages/03_📏_Evaluate.py", line 363, in _compute_metric
    metric.compute(
File "/home/user/.local/lib/python3.10/site-packages/evaluate/module.py", line 450, in compute
    self.add_batch(**inputs)
File "/home/user/.local/lib/python3.10/site-packages/evaluate/module.py", line 524, in add_batch
    elif set(self.selected_feature_format) != {"references", "predictions"}:

We use MATEO at the University with students, so during class it commonly happens that more than 1 person tries to use the system at the same time. I have not been able to get around to check the code yet, but it appears to be something with the uploaded files getting lost/overwritten by the second session. Is this a known issue or perhaps something that is caused by Streamlit?

[BramVanroy]

Hi there thanks for your interest. I have not experienced this before, even with many concurrent users.

• are you using an existing, hosted instance of mateo (which link), or your own (docker) instance?
• which metrics cause this? It is possible that the system goes out of memory when you are using large models/files on limited d hardware
• can you provide test files for me to test with

[BramVanroy]

Also note that files can't get overwritten since everything is processed in-memory.

[clang88]

Hi, thanks for the quick reply and clarification!

• are you using an existing, hosted instance of mateo (which link), or your own (docker) instance?
  • I observed it on the hugging face spaces (the basic free tier with 18GB of RAM) and the https://mateo.ivdnt.org/ hosted MATEO. I just reproduced it on my Laptop with the CPU branch and 16GB Ram running the most recent version in Docker on WSL2; indeed it seems to crash when two different files are evaluated at the same time with COMET. RAM usage goes to around 14GB in Task manager but according to docker stats the consumption of MATEO is at around 3.392 GB (with 7.668 being the limit)
• which metrics cause this? It is possible that the system goes out of memory when you are using large models/files on limited d hardware
  • I have only observed this with COMET so far, however, the non-neural methods finish so fast I can't properly stage a simultaneous run.

You can find the test files below:

Vienna_Environmental.en-de.GoogleTranslate.de.txt
Vienna_Environmental.en-de.OpusCAT.de.txt
Vienna_Environmental.en-de.OpusCAT_finetuned.de.txt
Vienna_Environmental.en-de.OpusCAT_finetuned_augmented.de.txt
Vienna_Environmental.en-de.test.reference.de.txt
Vienna_Environmental.en-de.test.source.en.txt

P.S.: While reproducing on my laptop right now, I ran into the same issue some students reported to me:

FileNotFoundError: [Errno 2] No such file or directory: '/home/mateo_user/.cache/huggingface/metrics/comet/Unbabel/wmt22-comet-da/default_experiment-e45ab135-7f14-458d-a1a5-a0beee82b98c-1-0.arrow'
Traceback:

File "/home/mateo_user/mateo/mateo-demo/src/mateo_st/pages/03_📏_Evaluate.py", line 436, in _compute_metrics
    result = _compute_metric(
             ^^^^^^^^^^^^^^^^
File "/home/mateo_user/.local/lib/python3.11/site-packages/streamlit/runtime/caching/cache_utils.py", line 212, in wrapper
    return cached_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mateo_user/.local/lib/python3.11/site-packages/streamlit/runtime/caching/cache_utils.py", line 243, in __call__
    return self._get_or_create_cached_value(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mateo_user/.local/lib/python3.11/site-packages/streamlit/runtime/caching/cache_utils.py", line 268, in _get_or_create_cached_value
    return self._handle_cache_miss(cache, value_key, func_args, func_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mateo_user/.local/lib/python3.11/site-packages/streamlit/runtime/caching/cache_utils.py", line 324, in _handle_cache_miss
    computed_value = self._info.func(*func_args, **func_kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mateo_user/mateo/mateo-demo/src/mateo_st/pages/03_📏_Evaluate.py", line 395, in _compute_metric
    raise exc
File "/home/mateo_user/mateo/mateo-demo/src/mateo_st/pages/03_📏_Evaluate.py", line 367, in _compute_metric
    metric.compute(
File "/home/mateo_user/.local/lib/python3.11/site-packages/evaluate/module.py", line 476, in compute
    os.remove(file_path)

Also attached, a video of me reproducing the issue on https://mateo.ivdnt.org/

2024-12-16_20-03-37.mp4

P.S.: The docker logs don't mention anything relating to the crash it seems.

[BramVanroy]

Thanks for all the info! I'm a bit busy now but will have a look as soon as possible.