New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stri_read_lines: maximal supported file size is ~0.67 GB #396

Open

gagolews opened this issue Aug 21, 2020 · 1 comment

Owner

gagolews commented Aug 21, 2020

This function should be rewritten if larger files are to be supported.

stri_write_lines is most likely affected as well

This is due to how these are currently implemented, see #395 for the underlying issue (stri_encode)

gagolews mentioned this issue

memory allocation error if strings too long #138

Closed

Owner Author

gagolews commented Aug 24, 2020

some code to play with:

Rscript -e '
x <- paste0(sample(letters, 100, replace = TRUE), collapse = "")
lines <- rep.int(x, 1e7)
writeLines(lines, "bigfile.txt")
print(file.size("bigfile.txt")/1024/1024)
lines2 <- stringi::stri_read_lines("bigfile.txt")
stopifnot(identical(lines, lines2))
'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment