Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Training loss goes to 0 and eval loss goes to nan #67

Open
ZeguanXiao opened this issue May 7, 2024 · 5 comments
Open

Training loss goes to 0 and eval loss goes to nan #67

ZeguanXiao opened this issue May 7, 2024 · 5 comments

Comments

@ZeguanXiao
Copy link

When using the script in the README to finetune llama2, the training loss goes to 0 and the eval loss goes to nan randomly.
@HZQ950419
Copy link
Collaborator

Hi,

What GPUs are you using to finetune llama2? I used to have this issue with V100s, but works well with 3090s, A100s.

@ZeguanXiao
Copy link
Author

I use A100 and L40. The issue occurs randomly, meaning that under the same settings, sometimes it happens and sometimes it doesn't.

@ZeguanXiao
Copy link
Author

@HZQ950419 Can you share your Python environment configuration? The issue may related to certain versions of transformers or tokenizers etc.

@HZQ950419
Copy link
Collaborator

Hi, the packages are showing below:

_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
accelerate                0.21.0                   pypi_0    pypi
aiofiles                  23.1.0                   pypi_0    pypi
aiohttp                   3.8.4                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
altair                    4.2.2                    pypi_0    pypi
anyio                     3.6.2                    pypi_0    pypi
appdirs                   1.4.4                    pypi_0    pypi
asttokens                 2.2.1                    pypi_0    pypi
async-timeout             4.0.2                    pypi_0    pypi
attrs                     22.2.0                   pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
bitsandbytes              0.37.2                   pypi_0    pypi
black                     23.3.0                   pypi_0    pypi
ca-certificates           2023.01.10           h06a4308_0  
certifi                   2022.12.7        py39h06a4308_0  
charset-normalizer        3.1.0                    pypi_0    pypi
click                     8.1.3                    pypi_0    pypi
cmake                     3.26.1                   pypi_0    pypi
contourpy                 1.0.7                    pypi_0    pypi
cpm-kernels               1.0.11                   pypi_0    pypi
cycler                    0.11.0                   pypi_0    pypi
datasets                  2.10.1                   pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
dill                      0.3.6                    pypi_0    pypi
entrypoints               0.4                      pypi_0    pypi
evaluate                  0.4.0                    pypi_0    pypi
executing                 1.2.0                    pypi_0    pypi
fastapi                   0.95.0                   pypi_0    pypi
ffmpy                     0.3.0                    pypi_0    pypi
filelock                  3.10.6                   pypi_0    pypi
fire                      0.5.0                    pypi_0    pypi
fonttools                 4.39.3                   pypi_0    pypi
frozenlist                1.3.3                    pypi_0    pypi
fsspec                    2023.3.0                 pypi_0    pypi
gradio                    3.23.0                   pypi_0    pypi
h11                       0.14.0                   pypi_0    pypi
httpcore                  0.16.3                   pypi_0    pypi
httpx                     0.23.3                   pypi_0    pypi
huggingface-hub           0.16.4                   pypi_0    pypi
idna                      3.4                      pypi_0    pypi
importlib-resources       5.12.0                   pypi_0    pypi
ipython                   8.11.0                   pypi_0    pypi
jedi                      0.18.2                   pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
joblib                    1.2.0                    pypi_0    pypi
jsonschema                4.17.3                   pypi_0    pypi
kiwisolver                1.4.4                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.4.2                h6a678d5_6  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libstdcxx-ng              11.2.0               h1234567_1  
linkify-it-py             2.0.0                    pypi_0    pypi
lit                       16.0.0                   pypi_0    pypi
markdown-it-py            2.2.0                    pypi_0    pypi
markupsafe                2.1.2                    pypi_0    pypi
matplotlib                3.7.1                    pypi_0    pypi
matplotlib-inline         0.1.6                    pypi_0    pypi
mdit-py-plugins           0.3.3                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
multidict                 6.0.4                    pypi_0    pypi
multiprocess              0.70.14                  pypi_0    pypi
mypy-extensions           1.0.0                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
networkx                  3.0                      pypi_0    pypi
numpy                     1.24.2                   pypi_0    pypi
nvidia-cublas-cu11        11.10.3.66               pypi_0    pypi
nvidia-cuda-cupti-cu11    11.7.101                 pypi_0    pypi
nvidia-cuda-nvrtc-cu11    11.7.99                  pypi_0    pypi
nvidia-cuda-runtime-cu11  11.7.99                  pypi_0    pypi
nvidia-cudnn-cu11         8.5.0.96                 pypi_0    pypi
nvidia-cufft-cu11         10.9.0.58                pypi_0    pypi
nvidia-curand-cu11        10.2.10.91               pypi_0    pypi
nvidia-cusolver-cu11      11.4.0.1                 pypi_0    pypi
nvidia-cusparse-cu11      11.7.4.91                pypi_0    pypi
nvidia-nccl-cu11          2.14.3                   pypi_0    pypi
nvidia-nvtx-cu11          11.7.91                  pypi_0    pypi
openssl                   1.1.1t               h7f8727e_0  
orjson                    3.8.9                    pypi_0    pypi
packaging                 23.0                     pypi_0    pypi
pandas                    1.5.3                    pypi_0    pypi
parso                     0.8.3                    pypi_0    pypi
pathspec                  0.11.1                   pypi_0    pypi
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    9.4.0                    pypi_0    pypi
pip                       23.0.1           py39h06a4308_0  
platformdirs              3.2.0                    pypi_0    pypi
prompt-toolkit            3.0.38                   pypi_0    pypi
protobuf                  4.24.3                   pypi_0    pypi
psutil                    5.9.4                    pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
pyarrow                   11.0.0                   pypi_0    pypi
pydantic                  1.10.7                   pypi_0    pypi
pydub                     0.25.1                   pypi_0    pypi
pygments                  2.14.0                   pypi_0    pypi
pyparsing                 3.0.9                    pypi_0    pypi
pyrsistent                0.19.3                   pypi_0    pypi
python                    3.9.16               h7a1cb2a_2  
python-dateutil           2.8.2                    pypi_0    pypi
python-multipart          0.0.6                    pypi_0    pypi
pytz                      2023.2                   pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
readline                  8.2                  h5eee18b_0  
regex                     2023.3.23                pypi_0    pypi
requests                  2.28.2                   pypi_0    pypi
responses                 0.18.0                   pypi_0    pypi
rfc3986                   1.5.0                    pypi_0    pypi
safetensors               0.3.1                    pypi_0    pypi
scikit-learn              1.2.2                    pypi_0    pypi
scipy                     1.10.1                   pypi_0    pypi
semantic-version          2.10.0                   pypi_0    pypi
sentencepiece             0.1.97                   pypi_0    pypi
setuptools                65.6.3           py39h06a4308_0  
six                       1.16.0                   pypi_0    pypi
sniffio                   1.3.0                    pypi_0    pypi
sqlite                    3.41.1               h5eee18b_0  
stack-data                0.6.2                    pypi_0    pypi
starlette                 0.26.1                   pypi_0    pypi
sympy                     1.11.1                   pypi_0    pypi
termcolor                 2.2.0                    pypi_0    pypi
threadpoolctl             3.1.0                    pypi_0    pypi
tk                        8.6.12               h1ccaba5_0  
tokenize-rt               5.0.0                    pypi_0    pypi
tokenizers                0.14.0                   pypi_0    pypi
tomli                     2.0.1                    pypi_0    pypi
toolz                     0.12.0                   pypi_0    pypi
torch                     2.0.0                    pypi_0    pypi
tqdm                      4.65.0                   pypi_0    pypi
traitlets                 5.9.0                    pypi_0    pypi
transformers              4.34.0.dev0              pypi_0    pypi
triton                    2.0.0                    pypi_0    pypi
typing-extensions         4.5.0                    pypi_0    pypi
tzdata                    2022g                h04d1e81_0  
uc-micro-py               1.0.1                    pypi_0    pypi
urllib3                   1.26.15                  pypi_0    pypi
uvicorn                   0.21.1                   pypi_0    pypi
wcwidth                   0.2.6                    pypi_0    pypi
websockets                10.4                     pypi_0    pypi
wheel                     0.38.4           py39h06a4308_0  
xxhash                    3.2.0                    pypi_0    pypi
xz                        5.2.10               h5eee18b_1  
yarl                      1.8.2                    pypi_0    pypi
zipp                      3.15.0                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_0

@ZeguanXiao
Copy link
Author

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@HZQ950419 @ZeguanXiao