Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

error when I train the model #3

Open
czy-0326 opened this issue Feb 23, 2021 · 4 comments
Open

error when I train the model #3

czy-0326 opened this issue Feb 23, 2021 · 4 comments

Comments

@czy-0326
Copy link

2021-02-23 22:06:27,598 INFO Start training
kitti_models/3DSSD_openPCDet(3DSSD)

epochs: 0it [00:00, ?it/s]
2021-02-23 22:06:28,232 INFO End training kitti_models/3DSSD_openPCDet(3DSSD)
When I run the command python train.py ,it doesnot work ,the problem is in line 3.

@czy-0326
Copy link
Author

when I delete the ckpt folder,it can continue ,but I meet the new problem
2021-02-23 22:25:04,173 INFO Start training kitti_models/3DSSD_openPCDet(3DSSD)
epochs: 0%| | 0/80 [00:01<?, ?it/s]
Traceback (most recent call last): | 0/3712 [00:00<?, ?it/s]
File "train.py", line 198, in
main()
File "train.py", line 170, in main
merge_all_iters_to_one_epoch=args.merge_all_iters_to_one_epoch
File "/media/czy/新加卷/3DSSD-pytorch-openPCDet-main/tools/train_utils/train_utils.py", line 93, in train_model
dataloader_iter=dataloader_iter
File "/media/czy/新加卷/3DSSD-pytorch-openPCDet-main/tools/train_utils/train_utils.py", line 40, in train_one_epoch
loss.backward()
File "/home/czy/anaconda3/envs/3dssd/lib/python3.7/site-packages/torch/tensor.py", line 195, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/czy/anaconda3/envs/3dssd/lib/python3.7/site-packages/torch/autograd/init.py", line 99, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: Expected isFloatingType(grads[i].type().scalarType()) to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)

@Mian0429
Copy link

when I delete the ckpt folder,it can continue ,but I meet the new problem
2021-02-23 22:25:04,173 INFO Start training kitti_models/3DSSD_openPCDet(3DSSD)
epochs: 0%| | 0/80 [00:01<?, ?it/s]
Traceback (most recent call last): | 0/3712 [00:00<?, ?it/s]
File "train.py", line 198, in
main()
File "train.py", line 170, in main
merge_all_iters_to_one_epoch=args.merge_all_iters_to_one_epoch
File "/media/czy/新加卷/3DSSD-pytorch-openPCDet-main/tools/train_utils/train_utils.py", line 93, in train_model
dataloader_iter=dataloader_iter
File "/media/czy/新加卷/3DSSD-pytorch-openPCDet-main/tools/train_utils/train_utils.py", line 40, in train_one_epoch
loss.backward()
File "/home/czy/anaconda3/envs/3dssd/lib/python3.7/site-packages/torch/tensor.py", line 195, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/czy/anaconda3/envs/3dssd/lib/python3.7/site-packages/torch/autograd/init.py", line 99, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: Expected isFloatingType(grads[i].type().scalarType()) to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)

I met the same error as you when I train on cu101+pytorch1.4+python3.6, and it works when I use cu90+pytorch1.1+python3.6. I don't know why but it works.

@GYE19970220
Copy link

same error,any answers?

1 similar comment
@3623687277
Copy link

same error,any answers?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants