Performance issue on HdfsReadFile

[dingqiaoyi]

Can we reuse the Hdfs file handle here ?

velox/velox/connectors/hive/storage_adapters/hdfs/HdfsReadFile.cpp

Lines 33 to 54 in 70fe9cf

	void HdfsReadFile::preadInternal(uint64_t offset, uint64_t length, char* pos)
	const {
	checkFileReadParameters(offset, length);
	auto file = hdfsOpenFile(hdfsClient_, filePath_.data(), O_RDONLY, 0, 0, 0);
	VELOX_CHECK_NOT_NULL(
	file,
	"Unable to open file {}. got error: {}",
	filePath_,
	hdfsGetLastError());
	seekToPosition(file, offset);
	uint64_t totalBytesRead = 0;
	while (totalBytesRead < length) {
	auto bytesRead = hdfsRead(hdfsClient_, file, pos, length - totalBytesRead);
	VELOX_CHECK(bytesRead >= 0, "Read failure in HDFSReadFile::preadInternal.")
	totalBytesRead += bytesRead;
	pos += bytesRead;
	}
	if (hdfsCloseFile(hdfsClient_, file) == -1) {
	LOG(ERROR) << "Unable to close file, errno: " << errno;
	}
	}

[majetideepak]

@dingqiaoyi there is some discussion on this in the PR that added HDFS support here #1427 (comment)

[dingqiaoyi]

Thanks for your help and I will look into it. Thanks!