Skip to content

Commit

Permalink
[SPARK-50959][ML][PYTHON] Swallow the exception of JavaWrapper.__del__
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

``` python
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.linalg import Vectors
from pyspark.sql import SparkSession

spark = SparkSession.builder.master("local[*]").getOrCreate()

dataset = spark.createDataFrame(
    [(Vectors.dense([0.0]), 0.0),
     (Vectors.dense([0.4]), 1.0),
     (Vectors.dense([0.5]), 0.0),
     (Vectors.dense([0.6]), 1.0),
     (Vectors.dense([1.0]), 1.0)] * 10,
    ["features", "label"])
lr = LogisticRegression()
model = lr.fit(dataset)
```

The above code will raise exception at the end. Even I remove the `try_remote_del` from https://github.com/apache/spark/blob/master/python/pyspark/ml/wrapper.py#L58 , the issue is still there.

``` console
Exception ignored in: <function JavaWrapper.del_ at 0x70b8caf2f920>_
Traceback (most recent call last):
  File "/home/xxx/work.d/spark/spark-master/python/pyspark/ml/util.py", line 254, in wrapped
  File "/home/xxx/work.d/spark/spark-master/python/pyspark/ml/wrapper.py", line 60, in __del__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <function JavaWrapper.del_ at 0x70b8caf2f920>_
Traceback (most recent call last):
  File "/home/xxx/work.d/spark/spark-master/python/pyspark/ml/util.py", line 254, in wrapped
  File "/home/xxx/work.d/spark/spark-master/python/pyspark/ml/wrapper.py", line 60, in __del__
ImportError: sys.meta_path is None, Python is likely shutting down
```

Looks like the `__del__` function of JavaWrapper is called after python shutting down and the python related modules is not functional.

### Why are the changes needed?

Fix the bug, good user experience.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?

CI passes

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #49615 from wbo4958/del.bug.

Authored-by: Bobby Wang <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 74cf4d8)
Signed-off-by: Ruifeng Zheng <[email protected]>
  • Loading branch information
wbo4958 authored and zhengruifeng committed Jan 23, 2025
1 parent 7aed7ec commit 79cdb11
Showing 1 changed file with 9 additions and 6 deletions.
15 changes: 9 additions & 6 deletions python/pyspark/ml/wrapper.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,12 +57,15 @@ def __init__(self, java_obj: Optional["JavaObject"] = None):

@try_remote_del
def __del__(self) -> None:
from pyspark.core.context import SparkContext

if SparkContext._active_spark_context and self._java_obj is not None:
SparkContext._active_spark_context._gateway.detach( # type: ignore[union-attr]
self._java_obj
)
try:
from pyspark.core.context import SparkContext

if SparkContext._active_spark_context and self._java_obj is not None:
SparkContext._active_spark_context._gateway.detach( # type: ignore[union-attr]
self._java_obj
)
except Exception:
pass

@classmethod
def _create_from_java_class(cls: Type[JW], java_class: str, *args: Any) -> JW:
Expand Down

0 comments on commit 79cdb11

Please sign in to comment.