Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Base.cp seems to have a bug when copying large files #56537

Open
sriharshakandala opened this issue Nov 12, 2024 · 3 comments
Open

Base.cp seems to have a bug when copying large files #56537

sriharshakandala opened this issue Nov 12, 2024 · 3 comments
Labels
io Involving the I/O subsystem: libuv, read, write, etc.

Comments

@sriharshakandala
Copy link

sriharshakandala commented Nov 12, 2024

Issue

Base.cp seems to have a bug when copying large files (~17 GB).

For example,

cp ifs_oper_T1279_2016080100.nc atmos_dyamond_summer/ifs_oper_T1279_2016080100.nc

works fine, but,

Base.cp("ifs_oper_T1279_2016080100.nc", "atmos_dyamond_summer/ifs_oper_T1279_2016080100.nc")

tries to create a huge file that eventually uses up all the space on the hard drive before crashing.

However, using

run(`cp ifs_oper_T1279_2016080100.nc atmos_dyamond_summer/ifs_oper_T1279_2016080100.nc`)

instead works correctly.

A more detailed description of the issue can be found CliMA/ClimaArtifacts#64 and CliMA/ClimaArtifacts#69

@giordano giordano added the io Involving the I/O subsystem: libuv, read, write, etc. label Nov 12, 2024
@stevengj
Copy link
Member

stevengj commented Nov 13, 2024

Base.cp calls sendfile, which calls jl_fs_sendfile which calls uv_fs_sendfile from libuv, which calls an OS routine. What operating system are you on?

@fredrikekre
Copy link
Member

Possibly related to #30723 and/or #39868

@sriharshakandala
Copy link
Author

sriharshakandala commented Nov 13, 2024

Base.cp calls sendfile, which calls jl_fs_sendfile which calls uv_fs_sendfile from libuv, which calls an OS routine. What operating system are you on?

This is on macOS Sequoia 15.0.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
io Involving the I/O subsystem: libuv, read, write, etc.
Projects
None yet
Development

No branches or pull requests

4 participants