Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

I got an error in open lm installation #297

Open
orhanerday opened this issue Jul 27, 2024 · 2 comments
Open

I got an error in open lm installation #297

orhanerday opened this issue Jul 27, 2024 · 2 comments

Comments

@orhanerday
Copy link

Environment

  • Transformers and Tokenizer versions: transformers-4.43.3 tokenizers-0.19.1
  • Platform: Python 3 Google Compute Engine backend (GPU)
  • Hardware accelerator: A100 GPU
  • Python version: Python 3.10.12

I got an error in open lm installation

I ran
pip install git+https://github.com/mlfoundations/open_lm.git

Here is the error traceback I received:


Collecting pyyaml>=5.1 (from datasets->open_lm==0.0.34)
Using cached PyYAML-5.4.1.tar.gz (175 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'error
@orhanerday
Copy link
Author

Here is the whole execution process

image


Collecting git+https://github.com/mlfoundations/open_lm.git
  Cloning https://github.com/mlfoundations/open_lm.git to /tmp/pip-req-build-gdj_yt4y
  Resolved https://github.com/mlfoundations/open_lm.git to commit 9bb92ef1689333534b7057942a20d18a46d1fa52
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (2.3.1+cu121)
Collecting xformers>=0.0.22 (from open_lm==0.0.34)
  Using cached xformers-0.0.27.post2-cp310-cp310-manylinux2014_x86_64.whl.metadata (1.0 kB)
Collecting tiktoken (from open_lm==0.0.34)
  Using cached tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting wandb (from open_lm==0.0.34)
  Using cached wandb-0.17.5-py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (10 kB)
Collecting webdataset (from open_lm==0.0.34)
  Using cached webdataset-0.2.86-py3-none-any.whl.metadata (29 kB)
Collecting pandas==2.1.4 (from open_lm==0.0.34)
  Using cached pandas-2.1.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (18 kB)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (2024.6.1)
Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (4.66.4)
Collecting jsonlines (from open_lm==0.0.34)
  Using cached jsonlines-4.0.0-py3-none-any.whl.metadata (1.6 kB)
Collecting boto3 (from open_lm==0.0.34)
  Using cached boto3-1.34.149-py3-none-any.whl.metadata (6.6 kB)
Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (9.4.0)
Collecting zstandard (from open_lm==0.0.34)
  Using cached zstandard-0.23.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting pysimdjson (from open_lm==0.0.34)
  Using cached pysimdjson-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.9 kB)
Requirement already satisfied: cloudpathlib in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (0.18.1)
Collecting datasets (from open_lm==0.0.34)
  Using cached datasets-2.20.0-py3-none-any.whl.metadata (19 kB)
Collecting multiprocess>=0.70.11 (from open_lm==0.0.34)
  Using cached multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB)
Collecting dill (from open_lm==0.0.34)
  Using cached dill-0.3.8-py3-none-any.whl.metadata (10 kB)
Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (0.23.5)
Collecting pre-commit (from open_lm==0.0.34)
  Using cached pre_commit-3.7.1-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting ray[all] (from open_lm==0.0.34)
  Using cached ray-2.33.0-cp310-cp310-manylinux2014_x86_64.whl.metadata (13 kB)
Collecting loguru (from open_lm==0.0.34)
  Using cached loguru-0.7.2-py3-none-any.whl.metadata (23 kB)
Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (4.42.4)
Collecting s3fs (from open_lm==0.0.34)
  Using cached s3fs-2024.6.1-py3-none-any.whl.metadata (1.6 kB)
Collecting wikipedia (from open_lm==0.0.34)
  Using cached wikipedia-1.4.0.tar.gz (27 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: ipython in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (7.34.0)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.23.5-py3-none-any.whl.metadata (28 kB)
Requirement already satisfied: numpy<2,>=1.22.4 in /usr/local/lib/python3.10/dist-packages (from pandas==2.1.4->open_lm==0.0.34) (1.25.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas==2.1.4->open_lm==0.0.34) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas==2.1.4->open_lm==0.0.34) (2024.1)
Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas==2.1.4->open_lm==0.0.34) (2024.1)
Collecting torch (from open_lm==0.0.34)
  Using cached torch-2.4.0-cp310-cp310-manylinux1_x86_64.whl.metadata (26 kB)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->open_lm==0.0.34) (3.15.4)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch->open_lm==0.0.34) (4.12.2)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->open_lm==0.0.34) (1.13.1)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->open_lm==0.0.34) (3.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->open_lm==0.0.34) (3.1.4)
Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch->open_lm==0.0.34)
  Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch->open_lm==0.0.34)
  Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch->open_lm==0.0.34)
  Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cudnn-cu12==9.1.0.70 (from torch->open_lm==0.0.34)
  Using cached nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cublas-cu12==12.1.3.1 (from torch->open_lm==0.0.34)
  Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cufft-cu12==11.0.2.54 (from torch->open_lm==0.0.34)
  Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-curand-cu12==10.3.2.106 (from torch->open_lm==0.0.34)
  Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch->open_lm==0.0.34)
  Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch->open_lm==0.0.34)
  Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-nccl-cu12==2.20.5 (from torch->open_lm==0.0.34)
  Using cached nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-nvtx-cu12==12.1.105 (from torch->open_lm==0.0.34)
  Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)
Collecting triton==3.0.0 (from torch->open_lm==0.0.34)
  Using cached triton-3.0.0-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.3 kB)
Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch->open_lm==0.0.34)
  Using cached nvidia_nvjitlink_cu12-12.5.82-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting botocore<1.35.0,>=1.34.149 (from boto3->open_lm==0.0.34)
  Using cached botocore-1.34.149-py3-none-any.whl.metadata (5.7 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3->open_lm==0.0.34)
  Using cached jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.11.0,>=0.10.0 (from boto3->open_lm==0.0.34)
  Using cached s3transfer-0.10.2-py3-none-any.whl.metadata (1.7 kB)
Collecting pyarrow>=15.0.0 (from datasets->open_lm==0.0.34)
  Using cached pyarrow-17.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (3.3 kB)
Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets->open_lm==0.0.34) (0.6)
Collecting requests>=2.32.2 (from datasets->open_lm==0.0.34)
  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting xxhash (from datasets->open_lm==0.0.34)
  Using cached xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
Collecting fsspec (from open_lm==0.0.34)
  Using cached fsspec-2024.5.0-py3-none-any.whl.metadata (11 kB)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->open_lm==0.0.34) (3.9.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets->open_lm==0.0.34) (24.1)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets->open_lm==0.0.34) (6.0.1)
Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (71.0.4)
Collecting jedi>=0.16 (from ipython->open_lm==0.0.34)
  Using cached jedi-0.19.1-py2.py3-none-any.whl.metadata (22 kB)
Requirement already satisfied: decorator in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (4.4.2)
Requirement already satisfied: pickleshare in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (0.7.5)
Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (5.7.1)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (3.0.47)
Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (2.16.1)
Requirement already satisfied: backcall in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (0.2.0)
Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (0.1.7)
Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (4.9.0)
Requirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonlines->open_lm==0.0.34) (23.2.0)
Collecting torchmetrics<1.3.3,>=0.10.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-1.3.2-py3-none-any.whl.metadata (19 kB)
Collecting torch-optimizer<0.4,>=0.3.0 (from mosaicml->open_lm==0.0.34)
  Using cached torch_optimizer-0.3.0-py3-none-any.whl.metadata (55 kB)
Requirement already satisfied: torchvision<0.18.2,>=0.13.1 in /usr/local/lib/python3.10/dist-packages (from mosaicml->open_lm==0.0.34) (0.18.1+cu121)
INFO: pip is looking at multiple versions of mosaicml to determine which version is compatible with other requirements. This could take a while.
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.23.4-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.23.3-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.23.2-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.23.1-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.23.0-py3-none-any.whl.metadata (28 kB)
Collecting torchvision<0.18.1,>=0.13.1 (from mosaicml->open_lm==0.0.34)
  Using cached torchvision-0.18.0-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.22.0-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.21.3-py3-none-any.whl.metadata (28 kB)
Collecting torchvision<0.17.3,>=0.13.1 (from mosaicml->open_lm==0.0.34)
  Using cached torchvision-0.17.2-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)
INFO: pip is still looking at multiple versions of mosaicml to determine which version is compatible with other requirements. This could take a while.
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.21.2-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.21.1-py3-none-any.whl.metadata (28 kB)
Collecting torchmetrics<1.3.2,>=0.10.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-1.3.1-py3-none-any.whl.metadata (19 kB)
Collecting torchvision<0.17.2,>=0.13.1 (from mosaicml->open_lm==0.0.34)
  Using cached torchvision-0.17.1-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.21.0-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.20.1-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.20.0-py3-none-any.whl.metadata (28 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Using cached mosaicml-0.19.1-py3-none-any.whl.metadata (28 kB)
Collecting torchmetrics<1.3.1,>=0.10.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-1.3.0.post0-py3-none-any.whl.metadata (20 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.19.0-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.18.2-py3-none-any.whl.metadata (27 kB)
Collecting torchvision<0.17,>=0.13.1 (from mosaicml->open_lm==0.0.34)
  Using cached torchvision-0.16.2-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.18.1-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.17.2-py3-none-any.whl.metadata (27 kB)
Collecting torchmetrics<1.1,>=0.10.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-1.0.3-py3-none-any.whl.metadata (21 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.17.1-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.17.0-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.16.4-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.16.3-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.16.2-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.16.1-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.16.0-py3-none-any.whl.metadata (28 kB)
Collecting torchvision<0.16,>=0.13.1 (from mosaicml->open_lm==0.0.34)
  Using cached torchvision-0.15.2-cp310-cp310-manylinux1_x86_64.whl.metadata (11 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.15.1-py3-none-any.whl.metadata (28 kB)
Collecting torchmetrics<0.12,>=0.10.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-0.11.4-py3-none-any.whl.metadata (15 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.15.0-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.14.1-py3-none-any.whl.metadata (28 kB)
Collecting torchmetrics<0.11.4,>=0.10.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-0.11.3-py3-none-any.whl.metadata (15 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.14.0-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.13.5-py3-none-any.whl.metadata (28 kB)
Collecting torchvision<0.15,>=0.11.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchvision-0.14.1-cp310-cp310-manylinux1_x86_64.whl.metadata (11 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.13.4-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.13.3-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.13.2-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.13.1-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.12.1-py3-none-any.whl.metadata (27 kB)
Collecting torchmetrics<0.10.0,>=0.7.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-0.9.3-py3-none-any.whl.metadata (17 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.12.0-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.11.1-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.11.0-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.10.1-py3-none-any.whl.metadata (27 kB)
Collecting torchmetrics<0.8,>=0.7.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-0.7.3-py3-none-any.whl.metadata (20 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.10.0-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.9.0-py3-none-any.whl.metadata (27 kB)
Collecting torch-optimizer<0.2,>=0.1.0 (from mosaicml->open_lm==0.0.34)
  Using cached torch_optimizer-0.1.0-py3-none-any.whl.metadata (53 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.8.2-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.8.1-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.8.0-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.7.1-py3-none-any.whl.metadata (26 kB)
Collecting pyyaml>=5.1 (from datasets->open_lm==0.0.34)
  Using cached PyYAML-5.4.1.tar.gz (175 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error

@AegisAurora
Copy link

Getting the same error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@orhanerday @AegisAurora