Skip to content

Commit

Permalink
Add faulty file system for io failure injection
Browse files Browse the repository at this point in the history
Add faulty file system to allow us to inject failure for low-level file operations in unit and fuzzer test.
It is implemented under velox filesystem. It is wrapper on top of the real file system, it either inject
errors or delegate the file operations to the underlying file system. We extend TempDirectoryPath or
TempFilePath to allow user to specify whether enable fault injection or not in test. If it does, it returns
the file or directory path with faulty file system scheme prefix (faulty:). This allows the velox file system
to open the file through faulty file system and by removing the faulty fs scheme prefix, it can delegate
the file operation to the corresponding underlying file system.

We support three kinds of file fault injections: (1) error injection which throws for a set (or all) file operation
types; (2) delay injection for a set (or all) file operation types; (3) custom injection with user provided fault
injection hook. We define the base structure FaultFileOperation to capture the file fault injection parameters
and each file api will extend with its own operation type.

This PR only supports fault injection for read file operations. Next we will extend to other file operation types
as well as fs operation types.
  • Loading branch information
xiaoxmeng committed Apr 12, 2024
1 parent a3b4849 commit 15be2c8
Show file tree
Hide file tree
Showing 14 changed files with 1,003 additions and 130 deletions.
6 changes: 2 additions & 4 deletions velox/common/file/File.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -238,10 +238,8 @@ LocalWriteFile::LocalWriteFile(
{
if (shouldThrowOnFileAlreadyExists) {
FILE* exists = fopen(buf.get(), "rb");
VELOX_CHECK(
!exists,
"Failure in LocalWriteFile: path '{}' already exists.",
path);
VELOX_CHECK_NULL(
exists, "Failure in LocalWriteFile: path '{}' already exists.", path);
}
}
auto* file = fopen(buf.get(), "ab");
Expand Down
8 changes: 4 additions & 4 deletions velox/common/file/File.h
Original file line number Diff line number Diff line change
Expand Up @@ -225,14 +225,14 @@ class InMemoryWriteFile final : public WriteFile {
std::string* file_;
};

// Current implementation for the local version is quite simple (e.g. no
// internal arenaing), as local disk writes are expected to be cheap. Local
// files match against any filepath starting with '/'.

/// Current implementation for the local version is quite simple (e.g. no
/// internal arenaing), as local disk writes are expected to be cheap. Local
/// files match against any filepath starting with '/'.
class LocalReadFile final : public ReadFile {
public:
explicit LocalReadFile(std::string_view path);

/// TODO: deprecate this after creating local file all through velox fs interface.
explicit LocalReadFile(int32_t fd);

~LocalReadFile();
Expand Down
3 changes: 2 additions & 1 deletion velox/common/file/tests/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,10 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
add_subdirectory(utils)

add_library(velox_file_test_utils TestUtils.cpp)
target_link_libraries(velox_file_test_utils PUBLIC velox_file)
target_link_libraries(velox_file_test_utils PUBLIC velox_file velox_file_test_lib)

add_executable(velox_file_test FileTest.cpp UtilsTest.cpp)
add_test(velox_file_test velox_file_test)
Expand Down
Loading

0 comments on commit 15be2c8

Please sign in to comment.