New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add timestamp_micros, timestamp_millis, unix_micros, unix_millis Spark functions #9448

Closed

zhli1142015 wants to merge 7 commits into facebookincubator:main from zhli1142015:micros_millis

Contributor

zhli1142015 commented Apr 11, 2024

Doc:
https://spark.apache.org/docs/latest/api/sql/index.html#timestamp_micros
https://spark.apache.org/docs/latest/api/sql/index.html#timestamp_millis
https://spark.apache.org/docs/latest/api/sql/index.html#unix_micros
https://spark.apache.org/docs/latest/api/sql/index.html#unix_millis

Code:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L665

facebook-github-bot added the CLA Signed label

netlify bot commented Apr 11, 2024 •

edited

Loading

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`b99f5db`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/6630375a2c8f85000878b329

Contributor Author

zhli1142015 commented Apr 11, 2024

cc @PHILO-HE and @rui-mo , thanks.

zhli1142015 force-pushed the micros_millis branch from 19111a1 to d4ea225 Compare

April 11, 2024 22:28

PHILO-HE reviewed

View reviewed changes

Contributor

PHILO-HE left a comment

Basically looks good! Some small suggestions. Thanks!

velox/functions/sparksql/tests/DateTimeFunctionsTest.cpp Outdated Show resolved Hide resolved

velox/functions/sparksql/Register.cpp Outdated Show resolved Hide resolved

rui-mo reviewed

View reviewed changes

Collaborator

rui-mo left a comment

Thanks. Just two nits.

velox/docs/functions/spark/datetime.rst Outdated Show resolved Hide resolved

velox/docs/functions/spark/datetime.rst Outdated Show resolved Hide resolved

zhli1142015 force-pushed the micros_millis branch 3 times, most recently from 7f525a7 to 222335d Compare

April 17, 2024 03:55

Contributor Author

zhli1142015 commented Apr 17, 2024

@mbasmanova , could you help review this PR? thanks.

zhli1142015 requested a review from rui-mo

April 18, 2024 02:41

Contributor

FelixYBW commented Apr 19, 2024

@mbasmanova Looks good to us. Can you help to take a look?

Contributor Author

zhli1142015 commented Apr 22, 2024

@Yuhta , would you also please help to review this PR? Thanks

Yuhta reviewed

View reviewed changes

velox/functions/sparksql/DateTimeFunctions.h Show resolved Hide resolved

Contributor

Yuhta commented Apr 23, 2024

Can we run spark expression fuzzer on the PR for 20 min?

Contributor

FelixYBW commented Apr 23, 2024

Can we run spark expression fuzzer on the PR for 20 min?

Spark Fuzzer is not ready yet. Rui is working on it. Currently it compares with DuckDB result.

Contributor

Yuhta commented Apr 23, 2024

@FelixYBW Only aggregation fuzzer is comparing the result, scalar expression fuzzer is just checking the fast path is working correctly

Contributor Author

zhli1142015 commented Apr 24, 2024

Can we run spark expression fuzzer on the PR for 20 min?
Done, thanks.

Stack trace has been disabled. Use --velox_exception_user_stacktrace_enabled=true to enable it.
W0424 12:23:44.772893 1524259 Expr.cpp:1812] Failed to generate SQL: Exception: VeloxUserError
Error Source: USER
Error Code: UNSUPPORTED
Reason: Type not supported yet: VARBINARY
Retriable: False
Function: appendSqlLiteral
File: /var/git/velox/velox/expression/ConstantExpr.cpp
Line: 228
Stack trace:
Stack trace has been disabled. Use --velox_exception_user_stacktrace_enabled=true to enable it.
E0424 12:23:44.956534 1524259 ExpressionFuzzerVerifier.cpp:373] Total iterations: 5239044
E0424 12:23:44.956560 1524259 ExpressionFuzzerVerifier.cpp:374] Total failed: 283195
[==========] Running 0 tests from 0 test suites.
[==========] 0 tests from 0 test suites ran. (0 ms total)
[  PASSED  ] 0 tests.

Contributor

facebook-github-bot commented Apr 24, 2024

@Yuhta has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Contributor

Yuhta commented Apr 24, 2024

Some UBSAN error:

Note: Google Test filter = DateTimeFunctionsTest.millisToTimestamp
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from DateTimeFunctionsTest
[ RUN      ] DateTimeFunctionsTest.millisToTimestamp

buck-out/v2/gen/fbcode/05af3bec81fdae30/velox/type/__velox_timestamp__/buck-headers/velox/type/Timestamp.h:199:35: runtime error: signed integer overflow: -9223372036854776 * 1000 cannot be represented in type 'long'
    #0 0x7f39bdccce5c in facebook::velox::Timestamp::fromMillis(long) fbcode/velox/type/Timestamp.h:199
    #1 0x7f39bdda9470 in call<long> fbcode/velox/functions/sparksql/DateTimeFunctions.h:777
    #2 0x7f39bdda9470 in callImpl fbcode/velox/core/SimpleFunctionMetadata.h:930
    #3 0x7f39bdda9470 in call fbcode/velox/core/SimpleFunctionMetadata.h:870
    #4 0x7f39bdda9470 in doApplyNotNull<1UL, long, 0> fbcode/velox/expression/SimpleFunctionAdapter.h:841
    #5 0x7f39bdda9470 in doApplyNotNull<0UL, facebook::velox::exec::FlatVectorReader<long>, 0> fbcode/velox/expression/SimpleFunctionAdapter.h:831
    #6 0x7f39bdda9470 in operator()<int> fbcode/velox/expression/SimpleFunctionAdapter.h:651
    #7 0x7f39bdda9470 in operator()<int> fbcode/velox/expression/EvalCtx.h:102
    #8 0x7f39bdda9470 in void facebook::velox::SelectivityVector::applyToSelected<void facebook::velox::exec::EvalCtx::applyToSelectedNoThrow<void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MillisToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::iterate<facebook::velox::exec::FlatVectorReader<long>>(facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MillisToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::ApplyContext&, facebook::velox::exec::FlatVectorReader<long>&) const::'lambda1'(auto)>(facebook::velox::SelectivityVector const&, auto)::'lambda'(auto)>(auto) const fbcode/velox/vector/SelectivityVector.h:429
    #9 0x7f39bdda8dcf in void facebook::velox::exec::EvalCtx::applyToSelectedNoThrow<void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MillisToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::iterate<facebook::velox::exec::FlatVectorReader<long>>(facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MillisToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::ApplyContext&, facebook::velox::exec::FlatVectorReader<long>&) const::'lambda1'(auto)>(facebook::velox::SelectivityVector const&, auto) fbcode/velox/expression/EvalCtx.h:100
    #10 0x7f39bdda08f5 in void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MillisToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::ApplyContext::applyToSelectedNoThrow<void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MillisToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::iterate<facebook::velox::exec::FlatVectorReader<long>>(facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MillisToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::ApplyContext&, facebook::velox::exec::FlatVectorReader<long>&) const::'lambda1'(auto)>(auto) fbcode/velox/expression/SimpleFunctionAdapter.h:195
    #11 0x7f39bdd9fe86 in void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MillisToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::iterate<facebook::velox::exec::FlatVectorReader<long>>(facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MillisToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::ApplyContext&, facebook::velox::exec::FlatVectorReader<long>&) const fbcode/velox/expression/SimpleFunctionAdapter.h:645
    #12 0x7f39bdd8c0f8 in void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MillisToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::unpackSpecializeForAllEncodings<1, facebook::velox::exec::FlatVectorReader<long>>(facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MillisToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::ApplyContext&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector>>> const&, facebook::velox::exec::FlatVectorReader<long>&) const fbcode/velox/expression/SimpleFunctionAdapter.h:473
    #13 0x7f39bdd8b509 in void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MillisToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::unpackSpecializeForAllEncodings<0>(facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MillisToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::ApplyContext&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector>>> const&) const fbcode/velox/expression/SimpleFunctionAdapter.h:489
    #14 0x7f39bdd8a1ba in facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MillisToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::apply(facebook::velox::SelectivityVector const&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector>>>&, std::shared_ptr<facebook::velox::Type const> const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&) const fbcode/velox/expression/SimpleFunctionAdapter.h:362
    #15 0x7f39c4b588a2 in facebook::velox::exec::Expr::applyFunction(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&) fbcode/velox/expression/Expr.cpp:1522
    #16 0x7f39c4b510d0 in facebook::velox::exec::Expr::evalFlatNoNullsImpl(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&, facebook::velox::exec::ExprSet const*) fbcode/velox/expression/Expr.cpp:773
    #17 0x7f39c4b4f67b in facebook::velox::exec::Expr::evalFlatNoNulls(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&, facebook::velox::exec::ExprSet const*) fbcode/velox/expression/Expr.cpp:737
    #18 0x7f39c4b5d662 in facebook::velox::exec::Expr::eval(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&, facebook::velox::exec::ExprSet const*) fbcode/velox/expression/Expr.cpp:792
    #19 0x7f39c4b25757 in facebook::velox::exec::ExprSet::eval(int, int, bool, facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector>>>&) fbcode/velox/expression/Expr.cpp:1928
    #20 0x6d1331 in facebook::velox::exec::ExprSet::eval(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector>>>&) fbcode/velox/expression/Expr.h:683
    #21 0x6cfe89 in facebook::velox::functions::test::FunctionBaseTest::evaluate(facebook::velox::exec::ExprSet&, std::shared_ptr<facebook::velox::RowVector> const&, std::optional<facebook::velox::SelectivityVector> const&) fbcode/velox/functions/prestosql/tests/utils/FunctionBaseTest.h:244
    #22 0x6cf5ce in std::shared_ptr<facebook::velox::BaseVector> facebook::velox::functions::test::FunctionBaseTest::evaluateImpl<facebook::velox::exec::ExprSet>(std::shared_ptr<facebook::velox::core::ITypedExpr const> const&, std::shared_ptr<facebook::velox::RowVector> const&, std::optional<facebook::velox::SelectivityVector> const&) fbcode/velox/functions/prestosql/tests/utils/FunctionBaseTest.h:303
    #23 0x6ceb77 in facebook::velox::functions::test::FunctionBaseTest::evaluate(std::shared_ptr<facebook::velox::core::ITypedExpr const> const&, std::shared_ptr<facebook::velox::RowVector> const&, std::optional<facebook::velox::SelectivityVector> const&) fbcode/velox/functions/prestosql/tests/utils/FunctionBaseTest.h:76
    #24 0x6cd7f6 in facebook::velox::functions::test::FunctionBaseTest::evaluate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::shared_ptr<facebook::velox::RowVector> const&, std::optional<facebook::velox::SelectivityVector> const&) fbcode/velox/functions/prestosql/tests/utils/FunctionBaseTest.h:86
    #25 0xb774a8 in std::shared_ptr<facebook::velox::SimpleVector<facebook::velox::Timestamp>> facebook::velox::functions::test::FunctionBaseTest::evaluate<facebook::velox::SimpleVector<facebook::velox::Timestamp>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::shared_ptr<facebook::velox::RowVector> const&, std::optional<facebook::velox::SelectivityVector> const&, std::shared_ptr<facebook::velox::Type const> const&) fbcode/velox/functions/prestosql/tests/utils/FunctionBaseTest.h:119
    #26 0xb69658 in std::optional<facebook::velox::Timestamp> facebook::velox::functions::test::FunctionBaseTest::evaluateOnce<facebook::velox::Timestamp>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::shared_ptr<facebook::velox::RowVector>, std::optional<facebook::velox::SelectivityVector> const&, std::shared_ptr<facebook::velox::Type const> const&) fbcode/velox/functions/prestosql/tests/utils/FunctionBaseTest.h:201
    #27 0xf53fa2 in std::optional<facebook::velox::Timestamp> facebook::velox::functions::test::FunctionBaseTest::evaluateOnce<facebook::velox::Timestamp, long>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::optional<long> const&) fbcode/velox/functions/prestosql/tests/utils/FunctionBaseTest.h:172
    #28 0xf56c18 in facebook::velox::functions::sparksql::test::(anonymous namespace)::DateTimeFunctionsTest_millisToTimestamp_Test::TestBody()::$_39::operator()(long) const fbcode/velox/functions/sparksql/tests/DateTimeFunctionsTest.cpp:1012
    #29 0xf56710 in facebook::velox::functions::sparksql::test::(anonymous namespace)::DateTimeFunctionsTest_millisToTimestamp_Test::TestBody() fbcode/velox/functions/sparksql/tests/DateTimeFunctionsTest.cpp:1042
    #30 0x7f39c84d973e in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) fbsource/src/gtest.cc:2675
    #31 0x7f39c84d8fc4 in testing::Test::Run() fbsource/src/gtest.cc:2692
    #32 0x7f39c84debff in testing::TestInfo::Run() fbsource/src/gtest.cc:2841
    #33 0x7f39c84e6bb6 in testing::TestSuite::Run() fbsource/src/gtest.cc:3020
    #34 0x7f39c852251b in testing::internal::UnitTestImpl::RunAllTests() fbsource/src/gtest.cc:5925
    #35 0x7f39c852157b in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) fbsource/src/gtest.cc:2675
    #36 0x7f39c8520ab9 in testing::UnitTest::Run() fbsource/src/gtest.cc:5489
    #37 0x7f39c83b29a0 in RUN_ALL_TESTS() fbsource/gtest/gtest.h:2317
    #38 0x7f39c83b262f in main fbcode/common/gtest/LightMain.cpp:20
    #39 0x7f39b722c656 in __libc_start_call_main /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #40 0x7f39b722c717 in [__libc_start_main@GLIBC_2.2.5](mailto:__libc_start_main@GLIBC_2.2.5) /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../csu/libc-start.c:409:3
    #41 0x6a49b0 in _start /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/x86_64/start.S:116

SUMMARY: UndefinedBehaviorSanitizer: signed-integer-overflow buck-out/v2/gen/fbcode/05af3bec81fdae30/velox/type/__velox_timestamp__/buck-headers/velox/type/Timestamp.h:199:35 in 

Test was never completed. The test process might have crashed.

zhli1142015 force-pushed the micros_millis branch 3 times, most recently from d6129f9 to 2dda331 Compare

April 25, 2024 08:34

Contributor

facebook-github-bot commented Apr 25, 2024

@Yuhta has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Yuhta approved these changes

View reviewed changes

Contributor

Yuhta commented Apr 25, 2024

One more error:

Note: Google Test filter = DateTimeFunctionsTest.microsToTimestamp
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from DateTimeFunctionsTest
[ RUN      ] DateTimeFunctionsTest.microsToTimestamp

buck-out/v2/gen/fbcode/05af3bec81fdae30/velox/type/__velox_timestamp__/buck-headers/velox/type/Timestamp.h:223:35: runtime error: signed integer overflow: -9223372036855 * 1000000 cannot be represented in type 'long'
    #0 0x7fb103fa524c in facebook::velox::Timestamp::fromMicros(long) fbcode/velox/type/Timestamp.h:223
    #1 0x7fb104081860 in call<long> fbcode/velox/functions/sparksql/DateTimeFunctions.h:754
    #2 0x7fb104081860 in callImpl fbcode/velox/core/SimpleFunctionMetadata.h:930
    #3 0x7fb104081860 in call fbcode/velox/core/SimpleFunctionMetadata.h:870
    #4 0x7fb104081860 in doApplyNotNull<1UL, long, 0> fbcode/velox/expression/SimpleFunctionAdapter.h:841
    #5 0x7fb104081860 in doApplyNotNull<0UL, facebook::velox::exec::FlatVectorReader<long>, 0> fbcode/velox/expression/SimpleFunctionAdapter.h:831
    #6 0x7fb104081860 in operator()<int> fbcode/velox/expression/SimpleFunctionAdapter.h:651
    #7 0x7fb104081860 in operator()<int> fbcode/velox/expression/EvalCtx.h:102
    #8 0x7fb104081860 in void facebook::velox::SelectivityVector::applyToSelected<void facebook::velox::exec::EvalCtx::applyToSelectedNoThrow<void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MicrosToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::iterate<facebook::velox::exec::FlatVectorReader<long>>(facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MicrosToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::ApplyContext&, facebook::velox::exec::FlatVectorReader<long>&) const::'lambda1'(auto)>(facebook::velox::SelectivityVector const&, auto)::'lambda'(auto)>(auto) const fbcode/velox/vector/SelectivityVector.h:429
    #9 0x7fb1040811bf in void facebook::velox::exec::EvalCtx::applyToSelectedNoThrow<void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MicrosToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::iterate<facebook::velox::exec::FlatVectorReader<long>>(facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MicrosToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::ApplyContext&, facebook::velox::exec::FlatVectorReader<long>&) const::'lambda1'(auto)>(facebook::velox::SelectivityVector const&, auto) fbcode/velox/expression/EvalCtx.h:100
    #10 0x7fb104078ce5 in void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MicrosToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::ApplyContext::applyToSelectedNoThrow<void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MicrosToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::iterate<facebook::velox::exec::FlatVectorReader<long>>(facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MicrosToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::ApplyContext&, facebook::velox::exec::FlatVectorReader<long>&) const::'lambda1'(auto)>(auto) fbcode/velox/expression/SimpleFunctionAdapter.h:195
    #11 0x7fb104078276 in void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MicrosToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::iterate<facebook::velox::exec::FlatVectorReader<long>>(facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MicrosToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::ApplyContext&, facebook::velox::exec::FlatVectorReader<long>&) const fbcode/velox/expression/SimpleFunctionAdapter.h:645
    #12 0x7fb1040644e8 in void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MicrosToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::unpackSpecializeForAllEncodings<1, facebook::velox::exec::FlatVectorReader<long>>(facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MicrosToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::ApplyContext&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector>>> const&, facebook::velox::exec::FlatVectorReader<long>&) const fbcode/velox/expression/SimpleFunctionAdapter.h:473
    #13 0x7fb1040638f9 in void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MicrosToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::unpackSpecializeForAllEncodings<0>(facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MicrosToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::ApplyContext&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector>>> const&) const fbcode/velox/expression/SimpleFunctionAdapter.h:489
    #14 0x7fb1040625aa in facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::MicrosToTimestampFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, facebook::velox::Timestamp, facebook::velox::ConstantChecker<long>, long>>::apply(facebook::velox::SelectivityVector const&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector>>>&, std::shared_ptr<facebook::velox::Type const> const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&) const fbcode/velox/expression/SimpleFunctionAdapter.h:362
    #15 0x7fb10b104562 in facebook::velox::exec::Expr::applyFunction(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&) fbcode/velox/expression/Expr.cpp:1522
    #16 0x7fb10b0fcd90 in facebook::velox::exec::Expr::evalFlatNoNullsImpl(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&, facebook::velox::exec::ExprSet const*) fbcode/velox/expression/Expr.cpp:773
    #17 0x7fb10b0fb33b in facebook::velox::exec::Expr::evalFlatNoNulls(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&, facebook::velox::exec::ExprSet const*) fbcode/velox/expression/Expr.cpp:737
    #18 0x7fb10b109322 in facebook::velox::exec::Expr::eval(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&, facebook::velox::exec::ExprSet const*) fbcode/velox/expression/Expr.cpp:792
    #19 0x7fb10b0d1417 in facebook::velox::exec::ExprSet::eval(int, int, bool, facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector>>>&) fbcode/velox/expression/Expr.cpp:1932
    #20 0x6db1b1 in facebook::velox::exec::ExprSet::eval(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector>>>&) fbcode/velox/expression/Expr.h:683
    #21 0x6d9d09 in facebook::velox::functions::test::FunctionBaseTest::evaluate(facebook::velox::exec::ExprSet&, std::shared_ptr<facebook::velox::RowVector> const&, std::optional<facebook::velox::SelectivityVector> const&) fbcode/velox/functions/prestosql/tests/utils/FunctionBaseTest.h:244
    #22 0x6d944e in std::shared_ptr<facebook::velox::BaseVector> facebook::velox::functions::test::FunctionBaseTest::evaluateImpl<facebook::velox::exec::ExprSet>(std::shared_ptr<facebook::velox::core::ITypedExpr const> const&, std::shared_ptr<facebook::velox::RowVector> const&, std::optional<facebook::velox::SelectivityVector> const&) fbcode/velox/functions/prestosql/tests/utils/FunctionBaseTest.h:303
    #23 0x6d89f7 in facebook::velox::functions::test::FunctionBaseTest::evaluate(std::shared_ptr<facebook::velox::core::ITypedExpr const> const&, std::shared_ptr<facebook::velox::RowVector> const&, std::optional<facebook::velox::SelectivityVector> const&) fbcode/velox/functions/prestosql/tests/utils/FunctionBaseTest.h:76
    #24 0x6d7676 in facebook::velox::functions::test::FunctionBaseTest::evaluate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::shared_ptr<facebook::velox::RowVector> const&, std::optional<facebook::velox::SelectivityVector> const&) fbcode/velox/functions/prestosql/tests/utils/FunctionBaseTest.h:86
    #25 0xb81328 in std::shared_ptr<facebook::velox::SimpleVector<facebook::velox::Timestamp>> facebook::velox::functions::test::FunctionBaseTest::evaluate<facebook::velox::SimpleVector<facebook::velox::Timestamp>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::shared_ptr<facebook::velox::RowVector> const&, std::optional<facebook::velox::SelectivityVector> const&, std::shared_ptr<facebook::velox::Type const> const&) fbcode/velox/functions/prestosql/tests/utils/FunctionBaseTest.h:119
    #26 0xb734d8 in std::optional<facebook::velox::Timestamp> facebook::velox::functions::test::FunctionBaseTest::evaluateOnce<facebook::velox::Timestamp>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::shared_ptr<facebook::velox::RowVector>, std::optional<facebook::velox::SelectivityVector> const&, std::shared_ptr<facebook::velox::Type const> const&) fbcode/velox/functions/prestosql/tests/utils/FunctionBaseTest.h:201
    #27 0xf6e342 in std::optional<facebook::velox::Timestamp> facebook::velox::functions::test::FunctionBaseTest::evaluateOnce<facebook::velox::Timestamp, long>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::optional<long> const&) fbcode/velox/functions/prestosql/tests/utils/FunctionBaseTest.h:172
    #28 0xf6db98 in facebook::velox::functions::sparksql::test::(anonymous namespace)::DateTimeFunctionsTest_microsToTimestamp_Test::TestBody()::$_38::operator()(long) const fbcode/velox/functions/sparksql/tests/DateTimeFunctionsTest.cpp:975
    #29 0xf6d690 in facebook::velox::functions::sparksql::test::(anonymous namespace)::DateTimeFunctionsTest_microsToTimestamp_Test::TestBody() fbcode/velox/functions/sparksql/tests/DateTimeFunctionsTest.cpp:1005
    #30 0x7fb10e8e873e in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) fbsource/src/gtest.cc:2675
    #31 0x7fb10e8e7fc4 in testing::Test::Run() fbsource/src/gtest.cc:2692
    #32 0x7fb10e8edbff in testing::TestInfo::Run() fbsource/src/gtest.cc:2841
    #33 0x7fb10e8f5bb6 in testing::TestSuite::Run() fbsource/src/gtest.cc:3020
    #34 0x7fb10e93151b in testing::internal::UnitTestImpl::RunAllTests() fbsource/src/gtest.cc:5925
    #35 0x7fb10e93057b in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) fbsource/src/gtest.cc:2675
    #36 0x7fb10e92fab9 in testing::UnitTest::Run() fbsource/src/gtest.cc:5489
    #37 0x7fb10e7c19a0 in RUN_ALL_TESTS() fbsource/gtest/gtest.h:2317
    #38 0x7fb10e7c162f in main fbcode/common/gtest/LightMain.cpp:20
    #39 0x7fb0fd62c656 in __libc_start_call_main /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #40 0x7fb0fd62c717 in [__libc_start_main@GLIBC_2.2.5](mailto:__libc_start_main@GLIBC_2.2.5) /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../csu/libc-start.c:409:3
    #41 0x6ae830 in _start /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/x86_64/start.S:116

SUMMARY: UndefinedBehaviorSanitizer: signed-integer-overflow buck-out/v2/gen/fbcode/05af3bec81fdae30/velox/type/__velox_timestamp__/buck-headers/velox/type/Timestamp.h:223:35 in 

Test was never completed. The test process might have crashed.

zhli1142015 force-pushed the micros_millis branch from 2dda331 to d70279c Compare

April 25, 2024 23:41

Contributor

facebook-github-bot commented Apr 26, 2024

@Yuhta has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

zhli1142015 added 2 commits

April 28, 2024 15:36


          Add timestamp_micros, timestamp_millis, unix_micros, unix_millis Spar…

282d8fb

…k functions


          address comments

2d3a1ff

zhli1142015 added 4 commits

April 28, 2024 15:36


          address comments

773f15a


          address comments

07020ee


          try fix UBSAN error

b21f819


          fix more UBSAN error

f549b6c

zhli1142015 force-pushed the micros_millis branch from d70279c to f549b6c Compare

April 28, 2024 07:37

Contributor Author

zhli1142015 commented Apr 29, 2024 •

edited

Loading

Resolved code conflict with main branch. @Yuhta could you please help trigger the Meta CI again? Thanks.

Contributor

facebook-github-bot commented Apr 29, 2024

@Yuhta has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Yuhta reviewed

View reviewed changes

Contributor

Yuhta left a comment

Some lint warnings

velox/functions/sparksql/DateTimeFunctions.h Outdated Show resolved Hide resolved

velox/functions/sparksql/DateTimeFunctions.h Outdated Show resolved Hide resolved

velox/functions/sparksql/DateTimeFunctions.h Outdated Show resolved Hide resolved


          address comments

b99f5db

Contributor

facebook-github-bot commented Apr 30, 2024

@Yuhta has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot closed this in

6e253f7

facebook-github-bot added the Merged label

Contributor

facebook-github-bot commented Apr 30, 2024

@Yuhta merged this pull request in 6e253f7.

conbench-facebook bot commented Apr 30, 2024

Conbench analyzed the 1 benchmark run on commit 6e253f7b.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

Joe-Abraham pushed a commit to Joe-Abraham/velox that referenced this pull request


          Add timestamp_micros, timestamp_millis, unix_micros, unix_millis Spar…

20c7203

…k functions (facebookincubator#9448)

Summary:
Doc:
https://spark.apache.org/docs/latest/api/sql/index.html#timestamp_micros
https://spark.apache.org/docs/latest/api/sql/index.html#timestamp_millis
https://spark.apache.org/docs/latest/api/sql/index.html#unix_micros
https://spark.apache.org/docs/latest/api/sql/index.html#unix_millis

Code:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L665

Pull Request resolved: facebookincubator#9448

Reviewed By: mbasmanova

Differential Revision: D56519466

Pulled By: Yuhta

fbshipit-source-id: e5b8369c72c2ce597a815e8d0d1c191d126b8345

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed Merged