Skip to content

Commit

Permalink
Fix HdfsMiniCluster search path (facebookincubator#10947)
Browse files Browse the repository at this point in the history
Summary:
PR facebookincubator#10446 upgraded the Hadoop version from 2.10.1 to 3.3.0, but the Hadoop seach path was not modified, which caused the `velox_hdfs_file_test` CI of the later submitted PR to turn red.
e.g. https://github.com/facebookincubator/velox/actions/runs/10752319167/job/29820507912?pr=10946
https://github.com/facebookincubator/velox/actions/runs/10735750279/job/29773707267?pr=10939
```
331/354 Test facebookincubator#338: velox_hdfs_file_test ...................................................***Exception: SegFault  0.20 sec
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240907 16:17:40.657701 65194 RegisterAbfsFileSystem.cpp:41] Register ABFS
Running main() from /__w/velox/velox/_build/release/_deps/gtest-src/googletest/src/gtest_main.cc
[==========] Running 22 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 20 tests from HdfsFileSystemTest
E20240907 16:17:40.6[685](https://github.com/facebookincubator/velox/actions/runs/10752319167/job/29820507912?pr=10946#step:11:686)77 65194 Exceptions.h:67] Line: /__w/velox/velox/velox/connectors/hive/storage_adapters/hdfs/tests/HdfsMiniCluster.cpp:71, Function:HdfsMiniCluster, Expression:  Failed to find minicluster executable hadoop', Source: RUNTIME, ErrorCode: INVALID_STATE
unknown file: Failure
C++ exception with description "Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Failed to find minicluster executable hadoop'
Retriable: False
Function: HdfsMiniCluster
File: /__w/velox/velox/velox/connectors/hive/storage_adapters/hdfs/tests/HdfsMiniCluster.cpp
Line: 71
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox11filesystems4test15HdfsMiniClusterC1Ev
# 4  _ZN18HdfsFileSystemTest14SetUpTestSuiteEv
# 5  _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS_9TestSuiteEvEET0_PT_MS4_FS3_vEPKc
# 6  _ZN7testing9TestSuite3RunEv
# 7  _ZN7testing8internal12UnitTestImpl11RunAllTestsEv
# 8  _ZN7testing8UnitTest3RunEv
# 9  main
# 10 __libc_start_call_main
# 11 __libc_start_main
# 12 _start
" thrown in SetUpTestSuite().
```

CC: majetideepak assignUser kgpai

Pull Request resolved: facebookincubator#10947

Reviewed By: pedroerp

Differential Revision: D62381299

Pulled By: mbasmanova

fbshipit-source-id: 8876cd79fdc9a7e756c02f506a79384d8b07de48
  • Loading branch information
wypb authored and facebook-github-bot committed Sep 10, 2024
1 parent cc10dc2 commit 4744994
Showing 1 changed file with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@

namespace facebook::velox::filesystems::test {
static const std::string miniClusterExecutableName{"hadoop"};
static const std::string hadoopSearchPath{":/usr/local/hadoop-2.10.1/bin"};
static const std::string hadoopSearchPath{":/usr/local/hadoop/bin"};
static const std::string jarCommand{"jar"};
static const std::string miniclusterJar{
"/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.1-tests.jar"};
"/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.0-tests.jar"};
static const std::string miniclusterCommand{"minicluster"};
static const std::string noMapReduceOption{"-nomr"};
static const std::string formatNameNodeOption{"-format"};
Expand Down

0 comments on commit 4744994

Please sign in to comment.