Skip to content

Commit

Permalink
feat: Add from_json Spark function
Browse files Browse the repository at this point in the history
  • Loading branch information
zhli1142015 committed Dec 2, 2024
1 parent ac5c15e commit 006efc5
Show file tree
Hide file tree
Showing 7 changed files with 910 additions and 1 deletion.
15 changes: 15 additions & 0 deletions velox/docs/functions/spark/json.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,18 @@ JSON Functions
SELECT json_object_keys(''); -- NULL
SELECT json_object_keys(1); -- NULL
SELECT json_object_keys('"hello"'); -- NULL

.. spark:function:: from_json(jsonString) -> [json object]
Casting a JSON text to a supported type returns the value represented by this
JSON text. The JSON text must represent a valid value of the type it is casted
to, or null will be returned. Casting to ARRAY and MAP is supported when the
element type of the array is one of the supported types, or when the key type of
the map is VARCHAR and value type of the map is one of the supported types. When
casting from JSON to ROW, only JSON object are supported. Cast from JSON object
to ROW uses case sensitive match for the JSON keys.
Behaviors of the casts are shown with the examples below:::

SELECT from_json('{"a": 1}', 'ROW(a INT)'); -- {a=1}
SELECT from_json('["name", "age", "id"]', 'array<string>'); -- ['name', 'age', 'id']
SELECT from_json('{"a": 1, "b": 2}', 'map<string,int>'); -- {a=1, b=2}
4 changes: 4 additions & 0 deletions velox/functions/sparksql/Register.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@
#include "velox/functions/sparksql/Uuid.h"
#include "velox/functions/sparksql/specialforms/AtLeastNNonNulls.h"
#include "velox/functions/sparksql/specialforms/DecimalRound.h"
#include "velox/functions/sparksql/specialforms/FromJson.h"
#include "velox/functions/sparksql/specialforms/MakeDecimal.h"
#include "velox/functions/sparksql/specialforms/SparkCastExpr.h"

Expand Down Expand Up @@ -152,6 +153,9 @@ void registerAllSpecialFormGeneralFunctions() {
exec::registerFunctionCallToSpecialForm(
AtLeastNNonNullsCallToSpecialForm::kAtLeastNNonNulls,
std::make_unique<AtLeastNNonNullsCallToSpecialForm>());
exec::registerFunctionCallToSpecialForm(
FromJsonCallToSpecialForm::kFromJson,
std::make_unique<FromJsonCallToSpecialForm>());
}

namespace {
Expand Down
3 changes: 2 additions & 1 deletion velox/functions/sparksql/specialforms/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,11 @@
velox_add_library(
velox_functions_spark_specialforms
AtLeastNNonNulls.cpp
FromJson.cpp
DecimalRound.cpp
MakeDecimal.cpp
SparkCastExpr.cpp
SparkCastHooks.cpp)

velox_link_libraries(velox_functions_spark_specialforms fmt::fmt
velox_expression)
velox_functions_json velox_expression)
Loading

0 comments on commit 006efc5

Please sign in to comment.