Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix input table jpy restriction #5260

Merged
merged 12 commits into from
Mar 21, 2024
Merged

Fix input table jpy restriction #5260

Changes from 3 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
0f386a9
Fix input table jpy restriction
arman-ddl Mar 16, 2024
e44039f
Remove unnecessary jpy type
arman-ddl Mar 16, 2024
d9db20b
Merge branch 'main' into arman_input_table_fix
arman-ddl Mar 18, 2024
cb2792d
Wrap java object with the most specific wrapper
jmao-denver Mar 20, 2024
6e9e9e3
Fix docstring format issue
jmao-denver Mar 20, 2024
0a1cc20
Apply suggestions from code review
jmao-denver Mar 20, 2024
42765af
Respond to review comments
jmao-denver Mar 20, 2024
302321f
Tiny coding formatting issue
jmao-denver Mar 20, 2024
b143ab7
Update py/server/deephaven/_wrapper.py
jmao-denver Mar 20, 2024
672a2ed
Add one more test case
jmao-denver Mar 20, 2024
319a68d
Empty Commit to force rerun of CI
jmao-denver Mar 20, 2024
47e866c
Respond to latest review comments
jmao-denver Mar 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 17 additions & 15 deletions py/server/deephaven/_wrapper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,22 +102,16 @@ def _is_direct_initialisable(cls) -> bool:
return False


def _lookup_wrapped_class(j_obj: jpy.JType) -> List[type]:
""" Returns the wrapper class for the specified Java object. """
def _lookup_wrapped_class(j_obj: jpy.JType) -> List[JObjectWrapper]:
""" Returns the wrapper classes for the specified Java object. """
# load every module in the deephaven package so that all the wrapper classes are loaded and available to wrap
# the Java objects returned by calling resolve()
global _has_all_wrappers_imported
if not _has_all_wrappers_imported:
_recursive_import(__package__.partition(".")[0])
_has_all_wrappers_imported = True

wcs = []
for wc in _di_wrapper_classes:
j_clz = wc.j_object_type
if j_clz.jclass.isInstance(j_obj):
wcs.append(wc)

return wcs
return [wc for wc in _di_wrapper_classes if wc.j_object_type.jclass.isInstance(j_obj)]


def javaify(obj: Any) -> Optional[jpy.JType]:
Expand Down Expand Up @@ -164,10 +158,17 @@ def pythonify(j_obj: Any) -> Optional[Any]:


def _wrap_with_subclass(j_obj: jpy.JType, cls: type) -> Optional[JObjectWrapper]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably, this is slower than the existing behavior. The only place I can imagine where that might matter is something like a PartitionedTable.transform with a Python function. Does the Java constituent Table get wrapped automatically as a Python Table wrapper, and if so , is it slow enough to matter?

@jmao-denver you might need to test this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, they are not. When requested (e.g. via. constituent_tables()), the constituent tables are explicitly wrapped in Table, not through this auto wrapping facility.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that addresses my concern. I am talking about when you pass a Python function (adapted to Java) to transform.
deephaven.table.PartitionedTable.transform

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some quick/minimal testing, ~2 - 4 % slower, the more constituent tables, the slower, which is kinda expected (more wrapping):

def transform_func(t: Table) -> Table:
    return t.update("f = a + b")

t = empty_table(100_000_000).update(["a = i", "b = i + 1", "c = i % 10_000", "d = `text`", "e = i % 1000000"])
pt = t.partition_by(by=["c"])
print("num of constituent tables: ", len(pt.constituent_tables))
print("constituent table size: ", pt.constituent_tables[0].size)
with make_user_exec_ctx():
    st = time.process_time_ns()
    transformed_pt = pt.transform(transform_func)
    print("transform time: ", (time.process_time_ns() - st)/10**9)
    self.assertIsNotNone(transformed_pt)

Before:

num of constituent tables:  10000
constituent table size:  10000
transform time:  42.321798386

num of constituent tables:  1000
constituent table size:  100000
transform time:  20.706249172

After:

num of constituent tables:  10000
constituent table size:  10000
transform time:  44.08636249

num of constituent tables:  1000
constituent table size:  100000
transform time:  21.192759225

""" Returns a wrapper instance for the specified Java object if the specified class is a subclass of the wrapper
class. Otherwise, returns None. """
subclasses = cls.__subclasses__()
for subclass in subclasses:
""" Returns a wrapper instance for the specified Java object by trying the entire subclasses' hierarchy. The
function employs the DFS strategy to try most specific subclass first. If no matching wrapper class is found,
jmao-denver marked this conversation as resolved.
Show resolved Hide resolved
returns None.

The premises for this function are as follows:
- The subclasses all share the same class attribute `j_object_type` (guaranteed by subclassing JObjectWrapper)
- The subclasses are all direct initialisable (guaranteed by subclassing JObjectWrapper)
- The subclasses are all distinct from each other and check for their uniqueness in the initializer (__init__), e.g.
InputTable checks for the presence of the INPUT_TABLE_ATTRIBUTE attribute on the Java object.
"""
for subclass in cls.__subclasses__():
try:
if (wrapper := _wrap_with_subclass(j_obj, subclass)) is not None:
return wrapper
Expand All @@ -176,7 +177,8 @@ def _wrap_with_subclass(j_obj: jpy.JType, cls: type) -> Optional[JObjectWrapper]
continue
return None

def wrap_j_object(j_obj: jpy.JType) -> Union[JObjectWrapper, jpy.JType]:

def wrap_j_object(j_obj: jpy.JType) -> Optional[Union[JObjectWrapper, jpy.JType]]:
""" Wraps the specified Java object as an instance of the most specific custom wrapper class if one is available,
otherwise returns the raw Java object. """
if j_obj is None:
Expand All @@ -191,7 +193,7 @@ def wrap_j_object(j_obj: jpy.JType) -> Union[JObjectWrapper, jpy.JType]:
except:
continue

return j_obj
return j_obj


def unwrap(obj: Any) -> Union[jpy.JType, Any]:
Expand Down
Loading