Error handling, how to do it?

[cedricchevalier19]

Error handling is notoriously hard to do well, especially in HPC, when dealing with multiple threads, accelerators, and, in our case, MPI ranks.

We can list here our thoughts on designing the safest API (and still efficient) API.

We must avoid exceptions (usually, C++ developers do not oppose this one!).
But then, how do you deal with errors?

• We can use int return type, like MPI. I'm not too fond of this way as it is easy not to check the return value.
• We can abort if we think the error is not recoverable. As a library, we should avoid this brutal termination.
• We can rely on specialized data structures for error handling like std::expected, but unfortunately, it is C++23. For some methods, we can use std::optional, even if it is not strictly speaking error handling.

The library must not check global coherency and never initiate more communications than the user wants (so there is no collective to ensure every rank is okay).

[cwpearson]

I think I disagree, I'm leaning towards exceptions though I'm open to being convinced of another path

1. Kokkos throws, so we cannot save people from handling exceptions merely by not throwing them ourselves.
   a. ...unless we try...catch every Kokkos call and converting it to an error value
2. C++ Core Guidelines suggest exceptions for error handling
3. Google C++ guidelines famously say "no exceptions," but it's merely because none of their code supports exceptions so they're not going to start now. They also say they'd probably do things differently if they started from scratch.
4. Best to complain noisily about an error state right when it happens.
5. In light of (4), we would need to add [[nodiscard]] to our error returns so errors don't get silently ignored. However, most of the time people will just want their program to die if there is a communication error, so they will need to handle every single error, so...why not just use exceptions and save them the trouble and lines of code?
6. If someone wants to write a resiliency layer on top of us, they they will need to handle errors either way, and I don't see why they'd care whether they check fields of error values or catch exceptions.

[cedricchevalier19]

I am really not convinced by exceptions, in particular when dealing with GPU or threads. But perhaps I am biased by also using Rust.

But I think the main point here is how to deal with error in a distributed environment. https://elib.dlr.de/144193/1/Testing%20HPC%20C%2B%2B%20software%20with%20GoogleTest.pdf shows how are it is to use google tests for MPI applications.

[cwpearson]

A good reference. Do you think it's possible to handle those problems by using the public interaface of google test like I started doing here: #32, or do we really need a fork of Google Test?

[cedricchevalier19]

I think we should follow your path and keep things as simple as possible. It is probably better to document things we cannot test than do a wrong code that does not test them.

[cwpearson]

I think the accelerator point is a good one. As long as we recognize that Rust has ? to simplify Result and we do not, I could live with something like std::expected. In many cases, our API will just return an error or nothing (std::expected<void, err>?). I guess we'd have to roll our own, expected, but hopefully it wouldn't be too bad.

I agree we need a reasonably-usable MPI testing framework.

I don't know that distributed error handling is really in scope for the project.

[dssgabriel]

I would also avoid exceptions and prefer to go the std::expected/std::optional route. It is unclear how exceptions would work when dealing with asynchronous/non-blocking stuff.

We can probably use std::variant from C++20 to implement our own KokkosComm::expected. It should be pretty straightforward, and we can always have a look at how LLVM/clang and GNU/gcc do it 👍

[aprokop]

Just putting it here for reference: mpi-forum/mpi-issues#288. There is also some conversation on error handling. Seems like people generally lean towards exceptions.

[cwpearson]

Excellent, thanks.

[cedricchevalier19]

Thanks for this interesting link.

However, I do not exactly have the same conclusion, it seems exceptions are not well suited for collective operations.

[devreal]

KokkosComm should provide an abort mechanism that maps to the communication library's abort (like MPI_Abort). Otherwise, it's impossible to reliably tear down a job if an error happens.

I agree that exceptions are difficult (think collective destruction of windows for example, which stops exception unwinding in a collective call). I don't like having a mechanism similar to C return codes (which std::expected<> ultimately would be). It's adding a lot of cost (mostly in terms of code overhead) to handle events that are unlikely.

A mechanism similar to error callbacks (similar to custom error handlers in MPI) could be useful. The application can either throw (if it knows that the callsite can handle exceptions), abort, or force the return of an error code / set a flag to defer handling. It's the most flexible way for users but might require the development of best practices to educate users.

[cedricchevalier19]

KokkosComm should provide an abort mechanism that maps to the communication library's abort (like MPI_Abort). Otherwise, it's impossible to reliably tear down a job if an error happens.

It is a good idea. We will add this to the list #10.

I agree that exceptions are difficult (think collective destruction of windows for example, which stops exception unwinding in a collective call). I don't like having a mechanism similar to C return codes (which std::expected<> ultimately would be). It's adding a lot of cost (mostly in terms of code overhead) to handle events that are unlikely.

Here I really miss Rust's ? syntax.

I really don't see how to properly handle things with exceptions between host and device.

As for the error callbacks, I think, at least for now, we will focus on a simpler way to handle errors.

[dssgabriel]

I agree that exceptions are difficult (think collective destruction of windows for example, which stops exception unwinding in a collective call). I don't like having a mechanism similar to C return codes (which std::expected<> ultimately would be). It's adding a lot of cost (mostly in terms of code overhead) to handle events that are unlikely.

Here I really miss Rust's ? syntax.

Would it be feasible to use something like TartanLlama's expected?
This would make error handling much more straightforward, with code being both easy to read and write.

A mechanism similar to error callbacks (similar to custom error handlers in MPI) could be useful.

I also think it is better to leave this aside for now

[devreal]

Would it be feasible to use something like TartanLlama's expected?

[...]

A mechanism similar to error callbacks (similar to custom error handlers in MPI) could be useful.
I also think it is better to leave this aside for now

Isn't the or_else a callback to handle errors? Granted, it's localized vs communicator-wide callbacks. It's a similar overhead in terms of coding for a rare error case. And it would lead us to continuation passing, which is potentially error-prone and not at all straight-forward to parse for humans.

[cwpearson]

Can I ask a dumb question - does anyone have an example of an application that checks and does something upon an MPI error? The applications I'm familiar with just treat all MPI errors as fatal. Is the only real use case here some kind of resilience thing?