Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Remaining filter behavior change after velox switch to meta velox #3778

Closed
zhli1142015 opened this issue Nov 20, 2023 · 1 comment · Fixed by #3791
Closed

[VL] Remaining filter behavior change after velox switch to meta velox #3778

zhli1142015 opened this issue Nov 20, 2023 · 1 comment · Fixed by #3791
Labels
bug Something isn't working performance

Comments

@zhli1142015
Copy link
Contributor

zhli1142015 commented Nov 20, 2023

Backend

VL (Velox)

Bug description

Before
image

After
image

Query:

spark.sql("""
select count(ss_item_sk), ss_addr_sk, ss_ext_sales_price, ss_sold_date_sk from store_sales where IsNull(ss_addr_sk) and IsNotNull(ss_item_sk)
group by ss_addr_sk, ss_ext_sales_price, ss_sold_date_sk
""").collect
Plan Node: 
-- Project[expressions: (n2_4:INTEGER, murmur3hash(42,"n0_1","n0_2","n0_3")), (n2_5:INTEGER, "n0_1"), (n2_6:DECIMAL(7,2), "n0_2"), (n2_7:INTEGER, "n0_3"), (n2_8:BIGINT, "n1_3")] -> n2_4:INTEGER, n2_5:INTEGER, n2_6:DECIMAL(7,2), n2_7:INTEGER, n2_8:BIGINT
  -- Aggregation[PARTIAL [n0_1, n0_2, n0_3] n1_3 := count("n0_0")] -> n0_1:INTEGER, n0_2:DECIMAL(7,2), n0_3:INTEGER, n1_3:BIGINT
    -- TableScan[table: hive_table, range filters: [(ss_item_sk, Filter(IsNotNull, deterministic, null not allowed))], remaining filter: (isnull("ss_addr_sk"))] -> n0_0:INTEGER, n0_1:INTEGER, n0_2:DECIMAL(7,2), n0_3:INTEGER

Before isnull is added in https://github.com/oap-project/velox/blob/update/velox/expression/ExprToSubfieldFilter.cpp#L509, but not now.

else if (call.name() == "is_null" || call.name() == "isnull")

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

No response

@zhli1142015
Copy link
Contributor Author

cc @rui-mo

@zhouyuan zhouyuan changed the title Remaining filter behavior change after velox switch to meta velox [VL] Remaining filter behavior change after velox switch to meta velox Nov 21, 2023
@zhli1142015 zhli1142015 linked a pull request Nov 21, 2023 that will close this issue
@github-project-automation github-project-automation bot moved this from Todo to Done in Gluten 1.1.0 Nov 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working performance
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants