Skip to content

Commit

Permalink
Fix 'Too many open files' error in JoinFuzzer (facebookincubator#8702)
Browse files Browse the repository at this point in the history
Summary:
Pull Request resolved: facebookincubator#8702

By default, Hive Connector caches 20'000 file handles. This leads to "Too many
open files" errors on some systems. Limit the number of cached file handles to
1'000.

Reviewed By: Yuhta

Differential Revision: D53535323

fbshipit-source-id: 2a995bd596aa28c2f8182ed5a8a640ba9806cabb
  • Loading branch information
mbasmanova authored and facebook-github-bot committed Feb 7, 2024
1 parent 02a4f6c commit a9e7cd5
Showing 1 changed file with 6 additions and 2 deletions.
8 changes: 6 additions & 2 deletions velox/exec/tests/JoinFuzzer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -140,11 +140,15 @@ class JoinFuzzer {
JoinFuzzer::JoinFuzzer(size_t initialSeed)
: vectorFuzzer_{getFuzzerOptions(), pool_.get()} {
filesystems::registerLocalFileSystem();

// Make sure not to run out of open file descriptors.
const std::unordered_map<std::string, std::string> hiveConfig = {
{connector::hive::HiveConfig::kNumCacheFileHandles, "1000"}};
auto hiveConnector =
connector::getConnectorFactory(
connector::hive::HiveConnectorFactory::kHiveConnectorName)
->newConnector(kHiveConnectorId, std::make_shared<core::MemConfig>());
connector::registerConnector(hiveConnector);
->newConnector(
kHiveConnectorId, std::make_shared<core::MemConfig>(hiveConfig));

seed(initialSeed);
}
Expand Down

0 comments on commit a9e7cd5

Please sign in to comment.