[SPARK-50792][SQL][FOLLOWUP] Improve the push down information for binary #49555
Conversation
|
ping @cloud-fan cc @sunxiaoguang |
| case BinaryType => | ||
| assert(value.isInstanceOf[Array[Byte]]) | ||
| val bytes = value.asInstanceOf[Array[Byte]] | ||
| bytes.map("%02X".format(_)).mkString("X'", "", "'") |
There was a problem hiding this comment.
Seems like X literal isn't universal and depends on dialect (#49452)
There was a problem hiding this comment.
Yes. you said right. The output information is just a generic represent like the others.
There was a problem hiding this comment.
How about to use ApacheHex.encodeHexString(binary, ... as in other places in the file.
There was a problem hiding this comment.
where else do we use encodeHexString?
There was a problem hiding this comment.
Anyway, this is for display only, we just need a popular binary value pretty string format.
There was a problem hiding this comment.
Spark SQL already parsed the format X'123456'.
The binary literal show the format 0x123456.
override def toString: String = value match {
case null => "null"
case binary: Array[Byte] => "0x" + ApacheHex.encodeHexString(binary, false)
Which one is the better?
|
+1, LGTM. Merging to master/4.0. |
…nary ### What changes were proposed in this pull request? This PR proposes to improve the push down information for binary. ### Why are the changes needed? Before this PR. the push down information for binary looks like below. `PushedFilters: [binary_col IS NOT NULL, binary_col = [B6e5af973]` After this PR, the push down information will be: `PushedFilters: [binary_col IS NOT NULL, binary_col = X'123456']` ### Does this PR introduce _any_ user-facing change? 'Yes'. Fix the bug. ### How was this patch tested? GA ### Was this patch authored or co-authored using generative AI tooling? 'No'. Closes #49555 from beliefer/SPARK-50792_followup. Authored-by: beliefer <[email protected]> Signed-off-by: Max Gekk <[email protected]> (cherry picked from commit 7e3de64) Signed-off-by: Max Gekk <[email protected]>
|
@MaxGekk @cloud-fan @HyukjinKwon Thank you for all! |
What changes were proposed in this pull request?
This PR proposes to improve the push down information for binary.
Why are the changes needed?
Before this PR. the push down information for binary looks like below.
PushedFilters: [binary_col IS NOT NULL, binary_col = [B@6e5af973]After this PR, the push down information will be:
PushedFilters: [binary_col IS NOT NULL, binary_col = X'123456']Does this PR introduce any user-facing change?
'Yes'.
Fix the bug.
How was this patch tested?
GA
Was this patch authored or co-authored using generative AI tooling?
'No'.