Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider using SXSSWorkbook for reading xlsx files #1016

Open
koperagen opened this issue Jan 17, 2025 · 0 comments
Open

Consider using SXSSWorkbook for reading xlsx files #1016

koperagen opened this issue Jan 17, 2025 · 0 comments
Labels
files reading/writing from/to files help wanted Extra attention is needed, feel free to help :)
Milestone

Comments

@koperagen
Copy link
Collaborator

koperagen commented Jan 17, 2025

xssfworkbook keeps whole file in memory, which for even relatively small dataframes or 100k rows can lead to a very significant memory consumption. sxssfworkbook implements streaming writing and flushes rows on disk as you write them. For comparison, max heap writing dataFrameOf('a'..'z').fill(1_000_000) { it }:
xssfworkbook 15 gb
sxssfworkbook 1.6 gb

The only concern is some potential incompatibilities mentioned in javadoc here:
https://poi.apache.org/apidocs/dev/org/apache/poi/xssf/streaming/SXSSFWorkbook.html

Other than that, from user perspective df.writeExcel(file) will benefit from much smaller memory consumption without lose of functionality:
https://poi.apache.org/components/spreadsheet/
https://stackoverflow.com/questions/33047512/hssfworkbook-vs-xssfworkbook-vs-sxssfworkbook-apache-poi

Check some other JVM DF libraries, do they use it?

@koperagen koperagen added files reading/writing from/to files help wanted Extra attention is needed, feel free to help :) labels Jan 17, 2025
@koperagen koperagen modified the milestones: Backlog, 0.16.0 Jan 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
files reading/writing from/to files help wanted Extra attention is needed, feel free to help :)
Projects
None yet
Development

No branches or pull requests

1 participant