DAOS-16501 build: Add libsanitize

Add documentation on ASan lib usaged and limitations.

Signed-off-by: Cedric Koch-Hofer <[email protected]>

docs/dev/development.md

@@ -122,6 +122,57 @@ scons reduces the clutter from compiler setup.
 Additionally, the tool supports options to filter by directory and file names and specify a lower
 bound value to report.
 
+### Address Sanitizer
+
+To debug memory corruptions, leaks, and other issues, DAOS can be compiled with the
+[AddressSanitizer (ASan)](https://github.com/google/sanitizers/wiki/AddressSanitizer) library.
+
+#### Building with ASan
+
+To build instrumented libraries, executables, RPMs, etc., use the `SANITIZERS=address` flag with the
+`scons` command.  This option is also managed by the CI system through a Jenkins configuration
+variable.
+
+ASan is supported with both Clang and GCC compilers. However, some compiler-specific configurations,
+such as maximum function stack size, may lead to unexpected behavior in the instrumented
+binaries.
+
+#### Customizing ASan Behavior
+ASan behavior can be configured using the `ASAN_OPTIONS` environment variable. For example, you can
+add the following entry to the `env_vars` section of the `daos_server.yml` configuration file:
+```yaml
+engines:
+- ..
+  env_vars:
+  ...
+  - ASAN_OPTIONS=atexit=1:print_stats=1:log_path=/tmp/daos_engine0.asan:disable_coredump=1:handle_segv=2:handle_abort=2:handle_sigfpe=2:handle_sigill=2:handle_sigbus=2:use_sigaltstack=1
+```
+For detailed information, see the
+[Sanitizer Common Flags](https://github.com/google/sanitizers/wiki/SanitizerCommonFlags) and
+[AddressSanitizer Flags](https://github.com/google/sanitizers/wiki/AddressSanitizerFlags)
+documentation pages.
+
+#### Known Issues
+
+ASan support in DAOS is still experimental and has the following known issues:
+1. **Library Path Issues**  
+   ASan's instrumentation of `dlopen()` removes `RPATH` and `RUNPATH` entries passed to the
+   compiler. As a result, libraries like `librdb` may not be found if DAOS is built with SCons and
+   installed in a non-standard location.  
+   **Workaround**: Use the `LD_LIBRARY_PATH` environment variable or the `ldconfig` command.  
+   More details are available in [Bug 27790](https://bugs.llvm.org/show_bug.cgi?id=27790).
+
+2. **Missing Statistics on SIGKILL**  
+   Stopping an application with `kill -s SIGKILL` prevents ASan from printing statistics, such as
+   detected memory leaks. For instance, this occurs when stopping DAOS engines using:  
+   ```bash
+   dmg system stop --force
+   ```
+
+3. **Integration with Test Framework**  
+   ASan is not fully integrated with the DAOS regression testing framework. Some tests, such as
+   those using Valgrind, may fail due to false positives.
+
 ## Go dependencies
 
 Developers contributing Go code may need to change the external dependencies