You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We should look into refactoring ApplicationParquetCleaner as currently, we hook a new callback into Spark for each export made during reading. A long-running application could register a large number of callbacks. Unclear if this would create a performance issue (eg: max number of callbacks) but it could be better designed.
Also, note that the name ApplicationParquetCleaner is now outdated and should be changed as we also support JSON files for reading operations.
The text was updated successfully, but these errors were encountered:
We should look into refactoring
ApplicationParquetCleaner
as currently, we hook a new callback into Spark for each export made during reading. A long-running application could register a large number of callbacks. Unclear if this would create a performance issue (eg: max number of callbacks) but it could be better designed.Also, note that the name ApplicationParquetCleaner is now outdated and should be changed as we also support JSON files for reading operations.
The text was updated successfully, but these errors were encountered: