Granite Model Error

[sbreddy2020]

When I tried using ibm-granite/granite-3b-code-base in Jupyter notebook , runtime used (CUDA/CPU) , It got stuck here , PFA screenshot.

Please look into it and advise solution sooner

[mayank31398]

I don't think this has anything to do with the model.
Might be one of 2 reasons:

1. your google drive ran out of memory (since you are running on colab)
2. the drive to colab interconnect is slow.

I am not entirely sure but it works locally on our machines.

[mayank31398]

We have dedicated servers but it works on my Macbook as well. (bit slow on Mac though)
4090s should be fine.
For the 3b model in 16-bit precision, you should need ~6GB of mem.
Can you try specifying torch_dtype=torch.float16 in the from_pretrained function?
that might help

default torch_dtype is torch.float32.

[sbreddy2020]

Thanks again for quick response. Tried granite-3b--code-base model on windows 10 laptop from Google Colab. Got 16GB RAM, crashed while model.eval(), Given torch.dtype=torch.float16 too before model. eval(). Still getting error as machine RAM exhausted. I expect all ML models whoever develops should be usable or compatible to Windows 10 because most of the present generation uses these config laptops or desktops. Do you agree Sir?

[mayank31398]

Can you try without colab?
I don't know how to debug this.
It works fine on local machines.

[sbreddy2020]

I tried locally using Pycharm IDE,
Got stuck here

D:\BHASKARA\UpGrad\AI Research\IBMGranite\venv\lib\site-packages\huggingface_hub\file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
warnings.warn(
Downloading shards: 0%| | 0/2 [00:00<?, ?it/s]

[mayank31398]

Hmm, I have no clue. I haven't seen this issue before
Also, this has nothing to do with our code either.
Seems like a HuggingFace download issue.