You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Because Delta scans work by using a subclass of ParquetFileFormat within a normal Hadoop relation, Comet will see this and simply replace it with a CometParquetFileFormat, losing all the Delta-specific things in it's own subclass, such as deletion vector support and column mapping
Steps to reproduce
Don't have exact steps right now, noticed this randomly while testing things out and kind of expected this to be a problem.
Expected behavior
If doing a Delta scan, it should not be eligible to be converted to a CometScan. The check for ParquetFileFormat probably needs to be an exact class comparison that doesn't include subclasses.
Longer term it would be interesting if it is possible to delegate the necessary behavior to custom file formats, but all the work trying to push down the Parquet scans to datafusion might make that impossible unless a different approach like using delta-rs directly is used such as in #174
Additional context
No response
The text was updated successfully, but these errors were encountered:
Describe the bug
Because Delta scans work by using a subclass of
ParquetFileFormat
within a normal Hadoop relation, Comet will see this and simply replace it with aCometParquetFileFormat
, losing all the Delta-specific things in it's own subclass, such as deletion vector support and column mappingSteps to reproduce
Don't have exact steps right now, noticed this randomly while testing things out and kind of expected this to be a problem.
Expected behavior
If doing a Delta scan, it should not be eligible to be converted to a CometScan. The check for
ParquetFileFormat
probably needs to be an exact class comparison that doesn't include subclasses.Longer term it would be interesting if it is possible to delegate the necessary behavior to custom file formats, but all the work trying to push down the Parquet scans to datafusion might make that impossible unless a different approach like using delta-rs directly is used such as in #174
Additional context
No response
The text was updated successfully, but these errors were encountered: