enable Half in mpi #1759

yhmtsai · 2024-12-30T16:01:04Z

This PR enables half precision in distributed environment by adding custom operation.

one-side operation like accumulation and fetch_and_op does not support custom operation.

Note. Newer version of mpi might support half precision natively (also for one-side operation) if the administrator build it with compiler supporting native half precision and enable the option.

TODO:

enable the rest distributed function with half
~~put the custom operation in gko::comm?~~ it does not grow along with #nodes -> create/free when necessary

MarcelKoch

I'm mainly concerned about using device buffers for the custom operations, and maybe moving the operations into a private header.

MarcelKoch · 2025-01-09T09:18:54Z

include/ginkgo/core/base/mpi.hpp

+
+
+template <typename ValueType>
+inline void sum(void* input, void* output, int* len, MPI_Datatype* datatype)


Do these functions work with device buffers? Or is it maybe necessary to copy the device buffers first to the host and then do the operation?

I have the same concern.
However, I think mpi will not use device memory to handle the reduction operation. Even in frontier, they have additional unit to handle not use gpu to sum cross nodes.

MarcelKoch · 2025-01-09T09:41:07Z

include/ginkgo/core/distributed/vector.hpp

+    mpi::op_manager sum_op_;
+    mpi::op_manager norm_sum_op_;


I don't think the vector needs to store these. We can either just create them when needed, and delete them directly afterward. There should not be a large overhead associated to that. Or we use globals (probably in the form of returning a static variable). I would not mind using that, since we only need to have one operation per value type anyway.

I have tried the static variable already but it did not work out.
We call MPI_finalize to finish, but the static variable will be destroyed when the process leave main scope, which is after MPI_finalize and MPI complains that.

I tried some simple benchmarking of creating/freeing the MPI_Op, and it seems to not scale with the number of processes. I guess that is to be expected, as it is not a collective operation. So I would suggest to just create the ops on-the-fly when necessary.

BTW here is the benchmark: https://gist.github.com/MarcelKoch/cbbee408bbd4d53689ef6b7aad404b04

thanks for profiling! just to be sure, you mean the runtime is not scale with the number of processes, right?

Yes, it has a fixed overhead that doesn't depend on the number of processes.

include/ginkgo/core/base/mpi.hpp

include/ginkgo/core/distributed/vector.hpp

MarcelKoch · 2025-01-09T09:58:11Z

include/ginkgo/core/distributed/vector.hpp


    static std::unique_ptr<target_type> create_empty(const source_type* source)
    {
        return target_type::create(source->get_executor(),
                                   source->get_communicator());
    }
+
+    // Allow to create_empty of the same type
+    // For distributed case, next<next<V>> will be V in the candicated list.


You mean if half is disabled, right? But shouldn't that also lead to issues with Dense?

yes.
I do not think so because Dense will use

ginkgo/include/ginkgo/core/base/temporary_conversion.hpp

Line 102 in b18607e

static std::unique_ptr<TargetType> create_empty(const SourceType* source)

to call create(exec) function and Dense can convert_to itself, which satisfies the enable_if condition.

Co-authored-by: Marcel Koch <[email protected]>

MarcelKoch

I would suggest removing the heap allocation for predefined mpi ops, rest looks good.

MarcelKoch · 2025-01-16T11:13:00Z

core/mpi/mpi_op.hpp

+}  // namespace detail
+
+
+using op_manager = std::shared_ptr<MPI_Op>;


Maybe just store the MPI_Op in a struct, so that you don't need to allocate/free anything for predefined MPI_Ops.
You could also keep the unique_ptr for the custom op, and only use the struct for the predefined ones.

MarcelKoch · 2025-01-16T11:26:01Z

core/test/utils.hpp

@@ -396,6 +396,9 @@ using TwoValueIndexTypes = add_to_cartesian_type_product_t<
 using ValueLocalGlobalIndexTypesBase =


this is now unused.

MarcelKoch · 2025-01-16T11:27:27Z

common/cuda_hip/distributed/assembly_kernels.cpp

@@ -90,7 +90,7 @@ void count_non_owning_entries(
        num_parts, local_part, row_part_ptrs.get_data(), send_count.get_data());
 }

-GKO_INSTANTIATE_FOR_EACH_VALUE_AND_LOCAL_GLOBAL_INDEX_TYPE_BASE(
+GKO_INSTANTIATE_FOR_EACH_VALUE_AND_LOCAL_GLOBAL_INDEX_TYPE(


The base version of that macro is now unused.

yhmtsai added the 1:ST:WIP This PR is a work in progress. Not ready for review. label Dec 30, 2024

yhmtsai self-assigned this Dec 30, 2024

ginkgo-bot added reg:testing This is related to testing. type:solver This is related to the solvers type:preconditioner This is related to the preconditioners mod:all This touches all Ginkgo modules. labels Dec 30, 2024

yhmtsai added 1:ST:ready-for-review This PR is ready for review and removed 1:ST:WIP This PR is a work in progress. Not ready for review. labels Jan 2, 2025

yhmtsai requested a review from a team January 2, 2025 08:30

MarcelKoch self-requested a review January 7, 2025 08:03

MarcelKoch requested changes Jan 9, 2025

View reviewed changes

yhmtsai and others added 6 commits January 14, 2025 10:57

WIP: half distributed

802b96b

WIP: change the instantiate type and implement the draft

5cd924d

implement and test

b3c461a

enable vector half

a8674dc

enable half in distributed matrix/vector/pgm/gmres

2b867fc

move the half custom op

8739d81

Co-authored-by: Marcel Koch <[email protected]>

yhmtsai force-pushed the half_mpi branch from 68a7045 to 8739d81 Compare January 14, 2025 09:57

yhmtsai added 2 commits January 15, 2025 08:47

add support when mpi support half natively

f88ecd4

change the implementation type because MPICH use int for MPI_Op

cd683f1

yhmtsai added the 1:ST:run-full-test label Jan 15, 2025

yhmtsai added 2 commits January 15, 2025 17:51

remove WIP native half mpi

1dddb15

update format

d802ba9

yhmtsai requested a review from MarcelKoch January 15, 2025 16:53

MarcelKoch requested changes Jan 16, 2025

View reviewed changes

pratikvn self-requested a review January 17, 2025 09:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enable Half in mpi #1759

enable Half in mpi #1759

yhmtsai commented Dec 30, 2024 •

edited

Loading

MarcelKoch left a comment

MarcelKoch Jan 9, 2025

yhmtsai Jan 10, 2025

MarcelKoch Jan 9, 2025

yhmtsai Jan 10, 2025

MarcelKoch Jan 13, 2025

MarcelKoch Jan 13, 2025

yhmtsai Jan 13, 2025

MarcelKoch Jan 13, 2025

MarcelKoch Jan 9, 2025

yhmtsai Jan 13, 2025

MarcelKoch left a comment

MarcelKoch Jan 16, 2025

MarcelKoch Jan 16, 2025

MarcelKoch Jan 16, 2025



		template <typename ValueType>
		inline void sum(void* input, void* output, int* len, MPI_Datatype* datatype)

		} // namespace detail


		using op_manager = std::shared_ptr<MPI_Op>;

		@@ -396,6 +396,9 @@ using TwoValueIndexTypes = add_to_cartesian_type_product_t<
		using ValueLocalGlobalIndexTypesBase =

enable Half in mpi #1759

Are you sure you want to change the base?

enable Half in mpi #1759

Conversation

yhmtsai commented Dec 30, 2024 • edited Loading

MarcelKoch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcelKoch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yhmtsai commented Dec 30, 2024 •

edited

Loading