Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Go error for a very large tar file #14

Open
rvencu opened this issue Dec 29, 2022 · 1 comment
Open

Go error for a very large tar file #14

rvencu opened this issue Dec 29, 2022 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@rvencu
Copy link

rvencu commented Dec 29, 2022

I tried to split a 2.3TB tar file into many 1GB tar files. With tarp I got this error after 55 shards:

panic: runtime error: index out of range [1] with length 0

goroutine 19 [running]:
github.com/tmbdev/tarp/dpipes.FnameSplit({0xc0002201e0?, 0xc0005dc030?})
        /root/tarp/dpipes/rawtario.go:36 +0xd2
github.com/tmbdev/tarp/dpipes.Aggregate(0x0?, 0xc0003ec060)
        /root/tarp/dpipes/rawtario.go:55 +0x137
created by github.com/tmbdev/tarp/dpipes.TarSource.func1
        /root/tarp/dpipes/tario.go:55 +0x9e

At first I thought the tar is damaged but trying the old tarproc in python (installed with pip install tarproc) I was able to split the big tar. The speed dropped from 32 shards per minute to 18 shards per minute but I am happy my problem is solved.

@tmbdev tmbdev self-assigned this Aug 2, 2023
@tmbdev tmbdev added the bug Something isn't working label Aug 2, 2023
@tmbdev
Copy link
Collaborator

tmbdev commented Aug 2, 2023

Thanks; I'll try to reproduce it. It looks like there may be an unexpected filename in there somewhere. The Python code is a bit more forgiving.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants