Skip to content

Commit

Permalink
more
Browse files Browse the repository at this point in the history
  • Loading branch information
@viirya
viirya committed Sep 17, 2024
1 parent 902587c commit f708bb4
Show file tree
Hide file tree
Showing 7 changed files with 331 additions and 347 deletions.
2 changes: 2 additions & 0 deletions spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1000,7 +1000,9 @@ class CometSparkSessionExtensions
CometColumnarShuffle,
_) =>
s.withNewChildren(Seq(child))
}

eliminatedPlan.transformUp {
case op: RowToColumnarExec
if op.output.forall(a => QueryPlanSerde.rowColumnarSupportedDataType(a.dataType)) =>
CometRowToColumnarExec(op.output, op.child)
Expand Down
83 changes: 39 additions & 44 deletions spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q98/explain.txt
View file
Original file line number Diff line number Diff line change
@@ -1,30 +1,29 @@
== Physical Plan ==
* ColumnarToRow (26)
+- CometProject (25)
+- CometSort (24)
+- CometColumnarExchange (23)
+- CometRowToColumnar (22)
+- * Project (21)
+- Window (20)
+- * ColumnarToRow (19)
+- CometSort (18)
+- CometExchange (17)
+- CometHashAggregate (16)
+- CometExchange (15)
+- CometHashAggregate (14)
+- CometProject (13)
+- CometBroadcastHashJoin (12)
:- CometProject (7)
: +- CometBroadcastHashJoin (6)
: :- CometFilter (2)
: : +- CometScan parquet spark_catalog.default.store_sales (1)
: +- CometBroadcastExchange (5)
: +- CometFilter (4)
: +- CometScan parquet spark_catalog.default.item (3)
+- CometBroadcastExchange (11)
+- CometProject (10)
+- CometFilter (9)
+- CometScan parquet spark_catalog.default.date_dim (8)
* ColumnarToRow (25)
+- CometProject (24)
+- CometSort (23)
+- CometColumnarExchange (22)
+- * Project (21)
+- Window (20)
+- * ColumnarToRow (19)
+- CometSort (18)
+- CometExchange (17)
+- CometHashAggregate (16)
+- CometExchange (15)
+- CometHashAggregate (14)
+- CometProject (13)
+- CometBroadcastHashJoin (12)
:- CometProject (7)
: +- CometBroadcastHashJoin (6)
: :- CometFilter (2)
: : +- CometScan parquet spark_catalog.default.store_sales (1)
: +- CometBroadcastExchange (5)
: +- CometFilter (4)
: +- CometScan parquet spark_catalog.default.item (3)
+- CometBroadcastExchange (11)
+- CometProject (10)
+- CometFilter (9)
+- CometScan parquet spark_catalog.default.date_dim (8)


(1) Scan parquet spark_catalog.default.store_sales
Expand Down Expand Up @@ -124,54 +123,50 @@ Arguments: [sum(_w0#15) windowspecdefinition(i_class#9, specifiedwindowframe(Row
Output [7]: [i_item_desc#7, i_category#10, i_class#9, i_current_price#8, itemrevenue#14, ((_w0#15 * 100) / _we0#16) AS revenueratio#17, i_item_id#6]
Input [8]: [i_item_desc#7, i_category#10, i_class#9, i_current_price#8, itemrevenue#14, _w0#15, i_item_id#6, _we0#16]

(22) CometRowToColumnar
Input [7]: [i_item_desc#7, i_category#10, i_class#9, i_current_price#8, itemrevenue#14, revenueratio#17, i_item_id#6]
Arguments: [i_item_desc#7, i_category#10, i_class#9, i_current_price#8, itemrevenue#14, revenueratio#17, i_item_id#6]

(23) CometColumnarExchange
(22) CometColumnarExchange
Input [7]: [i_item_desc#7, i_category#10, i_class#9, i_current_price#8, itemrevenue#14, revenueratio#17, i_item_id#6]
Arguments: rangepartitioning(i_category#10 ASC NULLS FIRST, i_class#9 ASC NULLS FIRST, i_item_id#6 ASC NULLS FIRST, i_item_desc#7 ASC NULLS FIRST, revenueratio#17 ASC NULLS FIRST, 5), ENSURE_REQUIREMENTS, CometColumnarShuffle, [plan_id=3]

(24) CometSort
(23) CometSort
Input [7]: [i_item_desc#7, i_category#10, i_class#9, i_current_price#8, itemrevenue#14, revenueratio#17, i_item_id#6]
Arguments: [i_item_desc#7, i_category#10, i_class#9, i_current_price#8, itemrevenue#14, revenueratio#17, i_item_id#6], [i_category#10 ASC NULLS FIRST, i_class#9 ASC NULLS FIRST, i_item_id#6 ASC NULLS FIRST, i_item_desc#7 ASC NULLS FIRST, revenueratio#17 ASC NULLS FIRST]

(25) CometProject
(24) CometProject
Input [7]: [i_item_desc#7, i_category#10, i_class#9, i_current_price#8, itemrevenue#14, revenueratio#17, i_item_id#6]
Arguments: [i_item_desc#7, i_category#10, i_class#9, i_current_price#8, itemrevenue#14, revenueratio#17], [i_item_desc#7, i_category#10, i_class#9, i_current_price#8, itemrevenue#14, revenueratio#17]

(26) ColumnarToRow [codegen id : 3]
(25) ColumnarToRow [codegen id : 3]
Input [6]: [i_item_desc#7, i_category#10, i_class#9, i_current_price#8, itemrevenue#14, revenueratio#17]

===== Subqueries =====

Subquery:1 Hosting operator id = 1 Hosting Expression = ss_sold_date_sk#3 IN dynamicpruning#4
BroadcastExchange (31)
+- * ColumnarToRow (30)
+- CometProject (29)
+- CometFilter (28)
+- CometScan parquet spark_catalog.default.date_dim (27)
BroadcastExchange (30)
+- * ColumnarToRow (29)
+- CometProject (28)
+- CometFilter (27)
+- CometScan parquet spark_catalog.default.date_dim (26)


(27) Scan parquet spark_catalog.default.date_dim
(26) Scan parquet spark_catalog.default.date_dim
Output [2]: [d_date_sk#11, d_date#12]
Batched: true
Location [not included in comparison]/{warehouse_dir}/date_dim]
PushedFilters: [IsNotNull(d_date), GreaterThanOrEqual(d_date,1999-02-22), LessThanOrEqual(d_date,1999-03-24), IsNotNull(d_date_sk)]
ReadSchema: struct<d_date_sk:int,d_date:date>

(28) CometFilter
(27) CometFilter
Input [2]: [d_date_sk#11, d_date#12]
Condition : (((isnotnull(d_date#12) AND (d_date#12 >= 1999-02-22)) AND (d_date#12 <= 1999-03-24)) AND isnotnull(d_date_sk#11))

(29) CometProject
(28) CometProject
Input [2]: [d_date_sk#11, d_date#12]
Arguments: [d_date_sk#11], [d_date_sk#11]

(30) ColumnarToRow [codegen id : 1]
(29) ColumnarToRow [codegen id : 1]
Input [1]: [d_date_sk#11]

(31) BroadcastExchange
(30) BroadcastExchange
Input [1]: [d_date_sk#11]
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [plan_id=4]

Expand Down
67 changes: 33 additions & 34 deletions spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q98/simplified.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,37 +4,36 @@ WholeStageCodegen (3)
CometProject [i_item_desc,i_category,i_class,i_current_price,itemrevenue,revenueratio]
CometSort [i_item_desc,i_category,i_class,i_current_price,itemrevenue,revenueratio,i_item_id]
CometColumnarExchange [i_category,i_class,i_item_id,i_item_desc,revenueratio] #1
CometRowToColumnar [i_item_desc,i_category,i_class,i_current_price,itemrevenue,revenueratio,i_item_id]
WholeStageCodegen (2)
Project [i_item_desc,i_category,i_class,i_current_price,itemrevenue,_w0,_we0,i_item_id]
InputAdapter
Window [_w0,i_class]
WholeStageCodegen (1)
ColumnarToRow
InputAdapter
CometSort [i_item_desc,i_category,i_class,i_current_price,itemrevenue,_w0,i_item_id]
CometExchange [i_class] #2
CometHashAggregate [i_item_desc,i_category,i_class,i_current_price,itemrevenue,_w0,i_item_id,sum,sum(UnscaledValue(ss_ext_sales_price))]
CometExchange [i_item_id,i_item_desc,i_category,i_class,i_current_price] #3
CometHashAggregate [i_item_id,i_item_desc,i_category,i_class,i_current_price,sum,ss_ext_sales_price]
CometProject [ss_ext_sales_price,i_item_id,i_item_desc,i_current_price,i_class,i_category]
CometBroadcastHashJoin [ss_ext_sales_price,ss_sold_date_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category,d_date_sk]
CometProject [ss_ext_sales_price,ss_sold_date_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
CometBroadcastHashJoin [ss_item_sk,ss_ext_sales_price,ss_sold_date_sk,i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
CometFilter [ss_item_sk,ss_ext_sales_price,ss_sold_date_sk]
CometScan parquet spark_catalog.default.store_sales [ss_item_sk,ss_ext_sales_price,ss_sold_date_sk]
SubqueryBroadcast [d_date_sk] #1
BroadcastExchange #4
WholeStageCodegen (1)
ColumnarToRow
InputAdapter
CometProject [d_date_sk]
CometFilter [d_date_sk,d_date]
CometScan parquet spark_catalog.default.date_dim [d_date_sk,d_date]
CometBroadcastExchange [i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category] #5
CometFilter [i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
CometScan parquet spark_catalog.default.item [i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
CometBroadcastExchange [d_date_sk] #6
CometProject [d_date_sk]
CometFilter [d_date_sk,d_date]
CometScan parquet spark_catalog.default.date_dim [d_date_sk,d_date]
WholeStageCodegen (2)
Project [i_item_desc,i_category,i_class,i_current_price,itemrevenue,_w0,_we0,i_item_id]
InputAdapter
Window [_w0,i_class]
WholeStageCodegen (1)
ColumnarToRow
InputAdapter
CometSort [i_item_desc,i_category,i_class,i_current_price,itemrevenue,_w0,i_item_id]
CometExchange [i_class] #2
CometHashAggregate [i_item_desc,i_category,i_class,i_current_price,itemrevenue,_w0,i_item_id,sum,sum(UnscaledValue(ss_ext_sales_price))]
CometExchange [i_item_id,i_item_desc,i_category,i_class,i_current_price] #3
CometHashAggregate [i_item_id,i_item_desc,i_category,i_class,i_current_price,sum,ss_ext_sales_price]
CometProject [ss_ext_sales_price,i_item_id,i_item_desc,i_current_price,i_class,i_category]
CometBroadcastHashJoin [ss_ext_sales_price,ss_sold_date_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category,d_date_sk]
CometProject [ss_ext_sales_price,ss_sold_date_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
CometBroadcastHashJoin [ss_item_sk,ss_ext_sales_price,ss_sold_date_sk,i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
CometFilter [ss_item_sk,ss_ext_sales_price,ss_sold_date_sk]
CometScan parquet spark_catalog.default.store_sales [ss_item_sk,ss_ext_sales_price,ss_sold_date_sk]
SubqueryBroadcast [d_date_sk] #1
BroadcastExchange #4
WholeStageCodegen (1)
ColumnarToRow
InputAdapter
CometProject [d_date_sk]
CometFilter [d_date_sk,d_date]
CometScan parquet spark_catalog.default.date_dim [d_date_sk,d_date]
CometBroadcastExchange [i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category] #5
CometFilter [i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
CometScan parquet spark_catalog.default.item [i_item_sk,i_item_id,i_item_desc,i_current_price,i_class,i_category]
CometBroadcastExchange [d_date_sk] #6
CometProject [d_date_sk]
CometFilter [d_date_sk,d_date]
CometScan parquet spark_catalog.default.date_dim [d_date_sk,d_date]
Loading

0 comments on commit f708bb4

Please sign in to comment.