Skip to content

Commit

Permalink
fix(pyspark): run pre-execute hooks for to_delta
Browse files Browse the repository at this point in the history
  • Loading branch information
deepyaman committed Jul 18, 2024
1 parent eaa1301 commit 635d98c
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions ibis/backends/pyspark/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -870,14 +870,15 @@ def to_delta(
An integer to effect a specific row limit. A value of `None` means
"no limit". The default is in `ibis/config.py`.
**kwargs
PySpark Delta Lake table write arguments.
https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrameWriter.save.html
Additional keyword arguments passed to
[pyspark.sql.DataFrameWriter](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrameWriter.html).
"""
if self.mode == "streaming":
raise NotImplementedError(
"Writing to a Delta Lake table in streaming mode is not supported"
)
self._run_pre_execute_hooks(expr)
df = self._session.sql(self.compile(expr, params=params, limit=limit))
df.write.format("delta").save(os.fspath(path), **kwargs)

Expand Down

0 comments on commit 635d98c

Please sign in to comment.