Fails at Downloading faiss index from hf://datasets/fondant-ai/datacomp-small-clip/faiss

[cydal]

Currently experiencing issues. I run the default code without making any changes. I have attempted on different systems, both local and cloud, I have also tried with GPU and CPU. I get the same error -

Starting pipeline run...
segment_images Pulling
download_images Pulling
caption_images Pulling
retrieve_from_faiss_by_prompt Pulling
generatepromptscomponent Pulling
segment_images Pulled
retrieve_from_faiss_by_prompt Pulled
download_images Pulled
caption_images Pulled
generatepromptscomponent Pulled
Container controlnet-pipeline-generatepromptscomponent-1 Recreate
Container controlnet-pipeline-generatepromptscomponent-1 Recreated
Container controlnet-pipeline-retrieve_from_faiss_by_prompt-1 Recreate
Container controlnet-pipeline-retrieve_from_faiss_by_prompt-1 Recreated
Container controlnet-pipeline-download_images-1 Recreate
Container controlnet-pipeline-download_images-1 Recreated
Container controlnet-pipeline-caption_images-1 Recreate
Container controlnet-pipeline-caption_images-1 Recreated
Container controlnet-pipeline-segment_images-1 Recreate
Container controlnet-pipeline-segment_images-1 Recreated
Attaching to caption_images-1, download_images-1, generatepromptscomponent-1, retrieve_from_faiss_by_prompt-1, segment_images-1
generatepromptscomponent-1 | [2024-03-28 07:37:04,289 | fondant.cli | INFO] Component GeneratePromptsComponent found in module main
generatepromptscomponent-1 | [2024-03-28 07:37:04,292 | fondant.component.executor | INFO] Skipping component execution
generatepromptscomponent-1 | [2024-03-28 07:37:04,293 | fondant.component.executor | INFO] Matching execution detected for component. The last execution of the component originated from controlnet-pipeline-20240328073537.
generatepromptscomponent-1 | [2024-03-28 07:37:04,294 | fondant.component.executor | INFO] Saving output manifest to /artifacts/controlnet-pipeline/controlnet-pipeline-20240328073700/generatepromptscomponent/manifest.json
generatepromptscomponent-1 | [2024-03-28 07:37:04,294 | fondant.component.executor | INFO] Writing cache key with manifest reference to /artifacts/controlnet-pipeline/cache/70ed9242b116d6a0a0af9e074cf8308f.txt
generatepromptscomponent-1 exited with code 0
retrieve_from_faiss_by_prompt-1 | [2024-03-28 07:37:06,311 | faiss.loader | INFO] Loading faiss with AVX2 support.
retrieve_from_faiss_by_prompt-1 | [2024-03-28 07:37:06,327 | faiss.loader | INFO] Successfully loaded faiss with AVX2 support.
retrieve_from_faiss_by_prompt-1 | [2024-03-28 07:37:07,972 | fondant.cli | INFO] Component RetrieveFromFaissByPrompt found in module main
retrieve_from_faiss_by_prompt-1 | [2024-03-28 07:37:07,976 | fondant.component.executor | INFO] Previous component generatepromptscomponent run was cached. Cached pipeline id: controlnet-pipeline-20240328073537
retrieve_from_faiss_by_prompt-1 | [2024-03-28 07:37:07,977 | fondant.component.executor | INFO] No matching execution for component detected
retrieve_from_faiss_by_prompt-1 | [2024-03-28 07:37:07,977 | root | INFO] Executing component
retrieve_from_faiss_by_prompt-1 | [2024-03-28 07:37:07,977 | main | INFO] Downloading faiss index from hf://datasets/fondant-ai/datacomp-small-clip/faiss
retrieve_from_faiss_by_prompt-1 | Traceback (most recent call last):
retrieve_from_faiss_by_prompt-1 | File "/opt/conda/bin/fondant", line 8, in
retrieve_from_faiss_by_prompt-1 | sys.exit(entrypoint())
retrieve_from_faiss_by_prompt-1 | File "/opt/conda/lib/python3.10/site-packages/fondant/cli.py", line 89, in entrypoint
retrieve_from_faiss_by_prompt-1 | args.func(args)
retrieve_from_faiss_by_prompt-1 | File "/opt/conda/lib/python3.10/site-packages/fondant/cli.py", line 711, in execute
retrieve_from_faiss_by_prompt-1 | executor.execute(component)
retrieve_from_faiss_by_prompt-1 | File "/opt/conda/lib/python3.10/site-packages/fondant/component/executor.py", line 335, in execute
retrieve_from_faiss_by_prompt-1 | output_manifest = self._run_execution(component_cls, input_manifest)
retrieve_from_faiss_by_prompt-1 | File "/opt/conda/lib/python3.10/site-packages/fondant/component/executor.py", line 290, in _run_execution
retrieve_from_faiss_by_prompt-1 | state = component.setup()
retrieve_from_faiss_by_prompt-1 | File "/component/src/./main.py", line 65, in setup
retrieve_from_faiss_by_prompt-1 | cluster = LocalCluster(
retrieve_from_faiss_by_prompt-1 | File "/opt/conda/lib/python3.10/site-packages/distributed/deploy/local.py", line 251, in init
retrieve_from_faiss_by_prompt-1 | workers = {i: worker for i in range(n_workers)}
retrieve_from_faiss_by_prompt-1 | TypeError: 'float' object cannot be interpreted as an integer
retrieve_from_faiss_by_prompt-1 exited with code 1
Finished pipeline run.
service "retrieve_from_faiss_by_prompt" didn't complete successfully: exit 1

[cydal]

The latest instance is NVIDIA RTX A6000 with 48GB GPU RAM, 500GB hard disk space, 6vCPUS, and 55GB RAM running on Ubuntu 22.04

[mrchtr]

Hi @cydal, the pipeline from the main branch doesn't support a GPU usage right now. There is this #15 open PR to enable the GPU. Would you up for giving that pipeline a try? We will merge it soon!

[cydal]

Thank you! Works fine on the GPU