Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DAOS-16556 client: Do not land. Test only! #15347

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

DAOS-16556 client: Do not land. Test only! #15347

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
138ae60
DAOS-16556 client: Do not land. Test only!
wiliamhuang Oct 19, 2024
f35cc5c
empty commit to trigger rebuild
wiliamhuang Oct 20, 2024
2557ce5
Revert "DAOS-16563 client: mark pool/cont handle as g2l after fork (#…
wiliamhuang Oct 20, 2024
ce71dd6
Revert "DAOS-15583 client: introduce whitelist mode into libpil4dfs (…
wiliamhuang Oct 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 7 additions & 14 deletions docs/user/filesystem.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1021,24 +1021,17 @@ libpil4dfs intercepting summary for ops on DFS:
[op_sum ] 5003
```

### Bypassing function interception in libpil4dfs
libpil4dfs enhances I/O performance by bypassing the fuse kernel when going over dfuse for I/O intensive workloads. In some scenarios however, for short-running applications (e.g., simple linux commands like cat, mkdir, chmod, etc.), there is no enough incentive to justify initializing the DAOS environment in user space with libpil4dfs, since this is relatively expensive. Such overhead is particularly noticeable for processes that complete within tens or hundreds of milliseconds and run frequently.
To address this issue, DAOS can disable function interception by libpil4dfs for specific executables/commands listed below:

"arch", "as", "awk", "basename", "bc", "cal", "cat", "chmod", "chown", "clang", "clear", "cmake", "cmake3", "cp", "cpp", "daos", "daos_agent", "daos_engine", "daos_server", "df", "dfuse", "dmg", "expr", "f77", "f90", "f95", "file", "gawk", "gcc", "gfortran", "gmake", "go", "gofmt", "grep", "g++", "head", "link", "ln", "ls", "kill", "m4", "make", "mkdir", "mktemp", "mv", "nasm", "yasm", "nm", "numactl", "patchelf", "ping", "pkg-config", "ps", "pwd", "ranlib", "readelf", "readlink", "rename", "rm", "rmdir", "rpm", "sed", "seq", "size", "sleep", "sort", "ssh", "stat", "strace", "strip", "su", "sudo", "tail", "tee", "telnet", "time", "top", "touch", "tr", "truncate", "uname", "vi", "vim", "whoami", "yes"

Also some scripting tools for package management, configuration and compiling,
"autoconf", "configure", "dnf", "dnf-3", "libtool", "libtoolize", "lsb_release", "meson", "scons", scons-3"

In addition, DAOS provides an environment variable (D_IL_BYPASS_LIST) to disable function interception by libpil4dfs for specific applications that are set in that env with the following syntax:
```
$ export D_IL_BYPASS_LIST="app_a:app_b:app_c:app_d"
```

### Turn on compatible mode in libpil4dfs
Fake file descriptor (FD) is used in regular mode in libpil4dfs.so for efficiency. open() returns fake fd to applications. In cases of some APIs are not intercepted, applications could crash with the error "Bad File Descriptor". Compatible mode is provided to work around such situations.
Setting env "D_IL_COMPATIBLE=1" turns on compatible mode. Kernel fd allocated by dfuse instead of fake fd will be returned to applications. This mode provides better compatibility with degraded performance in open, openat, and opendir, etc. Please start dfuse with "--disable-caching" to disable caching before using compatible mode.

### Child Process Inheritance

Normally child processes inherit environmental variables from parent processes. In rare cases, e.g.
scons, envs are striped off when calling execve(). It might be useful to force pil4dfs related env
set in child processes by setting env "D_IL_ENFORCE_EXEC_ENV=1". This flag is 0 if not set.


### Directory caching

To improve performance, directories are cached in a hash table. The size of this hash table could
Expand Down
29 changes: 0 additions & 29 deletions src/client/api/init.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -361,32 +361,3 @@ daos_fini(void)
D_MUTEX_UNLOCK(&module_lock);
return rc;
}

/**
* Re-initialize the DAOS library in the child process after fork.
*/
int
daos_reinit(void)
{
int rc;

rc = daos_eq_lib_reset_after_fork();
if (rc)
return rc;

daos_dti_reset();

/**
* Mark all pool and container handles owned by the parent process as if they were created
* in the child processes with g2l to avoid confusing the DAOS engines.
*/
rc = dc_pool_mark_all_slave();
if (rc)
return rc;

rc = dc_cont_mark_all_slave();
if (rc)
return rc;

return 0;
}
15 changes: 6 additions & 9 deletions src/client/dfuse/il/int_posix.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -820,11 +820,10 @@ child_hdlr(void)
{
int rc;

rc = daos_reinit();
rc = daos_eq_lib_reset_after_fork();
if (rc)
DL_WARN(rc, "daos_reinit() failed in child process");

/** Reset event queue */
DL_WARN(rc, "daos_eq_lib_init() failed in child process");
daos_dti_reset();
ioil_eqh = ioil_iog.iog_main_eqh = DAOS_HDL_INVAL;

if (ioil_iog.iog_eq_count_max) {
Expand Down Expand Up @@ -883,12 +882,10 @@ check_ioctl_on_open(int fd, struct fd_entry *entry, int flags)
D_GOTO(err, rc = daos_der2errno(rc));
}
ioil_iog.iog_main_eqh = ioil_eqh;
}

rc = pthread_atfork(NULL, NULL, &child_hdlr);
if (rc)
DFUSE_LOG_WARNING("Failed to install atfork handler: " DF_RC, DP_RC(rc));
rc = 0;
rc = pthread_atfork(NULL, NULL, &child_hdlr);
D_ASSERT(rc == 0);
}
}

d_list_for_each_entry(pool, &ioil_iog.iog_pools_head, iop_pools) {
Expand Down
Loading
Loading