Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GPU offloading #2

Draft
wants to merge 126 commits into
base: main
Choose a base branch
from
Draft

GPU offloading #2

wants to merge 126 commits into from

Conversation

benegee
Copy link
Owner

@benegee benegee commented Nov 18, 2024

Very rough summary:

Features

Important changes

  • Adapt.jl, adapt_to for several types
  • AbstractHeterogeneousContainer:
    # Containers that support heterogenous computing via KernelAbstractions.jl
    # should be subtypes of this type.
    #
    # The first type parameter must be `Array` by default and if
    # `Adapt.adapt_structure(to, container)` is called then this must be
    # `typeof(to)`. This is used in downstream code, e.g. for calling
    # `wrap_array` with an appropriate type after `resize!`ing.
    #
    # The second type parameter determines if KA.jl is used.
    # By default, each container must initialize this to `false`.
    # However, when `Adapt.adapt_structure` is called on the container, it must
    # be changed to `true`.
  • VecOfArray
    # Wraps a Vector of Arrays, forwards `getindex` to the underlying Vector.
    # Implements `Adapt.adapt_structure` to allow offloading to the GPU which is
    # not possible for a plain Vector of Arrays.
    struct VecOfArrays{T <: AbstractArray}
    arrays::Vector{T}
    end
    used for P4estMPICache
    mutable struct P4estMPICache{BufferType <: DenseVector, VecInt <: DenseVector{<:Integer}}
    mpi_neighbor_ranks::Vector{Int}
    mpi_neighbor_interfaces::VecOfArrays{VecInt}
    mpi_neighbor_mortars::VecOfArrays{VecInt}
    mpi_send_buffers::VecOfArrays{BufferType}
    mpi_recv_buffers::VecOfArrays{BufferType}
    mpi_send_requests::Vector{MPI.Request}
    mpi_recv_requests::Vector{MPI.Request}
    n_elements_by_rank::OffsetArray{Int, 1, Array{Int, 1}}
    n_elements_global::Int
    first_element_global_id::Int
    end
  • Trixi.Indexing module instead of symbols

    Trixi.jl/src/solvers/dg.jl

    Lines 755 to 778 in 33903d9

    # For some mesh types, elements next to a surface may have local coordinate systems
    # that are not aligned so the nodes may have to be indexed differently.
    # `IndexInfo` is used to describe how the nodes should be indexed.
    # For example, in 2d a `Tuple` with two `IndexInfo` objects, one for each dimension,
    # would be used.
    # `first` or `last` indicates that the corresponding index is constant and is either
    # the first or the last one. This effectively encodes the position of the surface
    # with respect to the local coordinate system. The other `IndexInfo` object(s)
    # encode if the index in the corresponding dimension is running forward or backward.
    #
    # The Enum is wrapped in a module and exported so that the enum values do not pollute
    # the global namespace and can only be accessed via `Indexing.value`.
    module Indexing
    @enum IndexInfo begin
    first
    last
    i_forward
    i_backward
    j_forward
    j_backward
    end
    export IndexInfo
    end
    using .Indexing

Todos

Johannes Markert and others added 30 commits April 8, 2024 13:04
…mework/Trixi.jl into feature-t8code-curved-geometry
it refers to level of refinement in lat lon direction, not number of
tree as in the p4est version
…mework/Trixi.jl into feature-t8code-curved-geometry
@benegee
Copy link
Owner Author

benegee commented Nov 25, 2024

Concerning Trixi.Indexing:

Some containers contain symbols, e.g. node_indices in P4estInterfaceContainer. This can lead to errors like

KernelError: passing and using non-bitstype argument 
Argument 8 to your kernel function is of type Symbol, which is not isbits:

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

JuliaFormatter

[JuliaFormatter] reported by reviewdog 🐶

x_neg = BoundaryConditionCoupled(2, (Indexing.last, Indexing.i_forward), Float64,


[JuliaFormatter] reported by reviewdog 🐶

x_pos = BoundaryConditionCoupled(2, (Indexing.first, Indexing.i_forward), Float64,


[JuliaFormatter] reported by reviewdog 🐶

x_neg = BoundaryConditionCoupled(1, (Indexing.last, Indexing.i_forward), Float64,


[JuliaFormatter] reported by reviewdog 🐶

x_pos = BoundaryConditionCoupled(1, (Indexing.first, Indexing.i_forward), Float64,


[JuliaFormatter] reported by reviewdog 🐶

boundary_conditions1 = (x_neg = BoundaryConditionCoupled(2, (Indexing.last, Indexing.i_forward), Float64,


[JuliaFormatter] reported by reviewdog 🐶

x_pos = BoundaryConditionCoupled(2, (Indexing.first, Indexing.i_forward), Float64,


[JuliaFormatter] reported by reviewdog 🐶

boundary_conditions2 = (x_neg = BoundaryConditionCoupled(1, (Indexing.last, Indexing.i_forward), Float64,


[JuliaFormatter] reported by reviewdog 🐶

x_pos = BoundaryConditionCoupled(1, (Indexing.first, Indexing.i_forward), Float64,


[JuliaFormatter] reported by reviewdog 🐶

l2_error, linf_error = calc_error_norms(u_ode_cpu, sol.t[end], analyzer, semi_cpu,


[JuliaFormatter] reported by reviewdog 🐶

if !(typeof(semi) <: SemidiscretizationHyperbolic) && !(typeof(semi.mesh) <: P4estMesh)


[JuliaFormatter] reported by reviewdog 🐶

performance_counter::PerformanceCounter) where {Mesh, Equations,


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

neighbor_ids = unsafe_wrap_or_alloc(to, _neighbor_ids, size(interfaces.neighbor_ids))
node_indices = unsafe_wrap_or_alloc(to, _node_indices, size(interfaces.node_indices))


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

interfaces.node_indices[side, interface_id] = (Indexing.first, surface_index1,


[JuliaFormatter] reported by reviewdog 🐶

interfaces.node_indices[side, interface_id] = (Indexing.last, surface_index1,


[JuliaFormatter] reported by reviewdog 🐶

interfaces.node_indices[side, interface_id] = (surface_index1, Indexing.first,


[JuliaFormatter] reported by reviewdog 🐶

interfaces.node_indices[side, interface_id] = (surface_index1, Indexing.last,


[JuliaFormatter] reported by reviewdog 🐶

surface_index2, Indexing.first)


[JuliaFormatter] reported by reviewdog 🐶

surface_index2, Indexing.last)


[JuliaFormatter] reported by reviewdog 🐶

boundaries.node_indices[boundary_id] = (Indexing.first, Indexing.i_forward, Indexing.j_forward)


[JuliaFormatter] reported by reviewdog 🐶

boundaries.node_indices[boundary_id] = (Indexing.last, Indexing.i_forward, Indexing.j_forward)


[JuliaFormatter] reported by reviewdog 🐶

boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.first, Indexing.j_forward)


[JuliaFormatter] reported by reviewdog 🐶

boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.last, Indexing.j_forward)


[JuliaFormatter] reported by reviewdog 🐶

boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.j_forward, Indexing.first)


[JuliaFormatter] reported by reviewdog 🐶

boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.j_forward, Indexing.last)


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

_prolong2mortars!(backend, cache, u, mesh, equations, mortar_l2, surface_integral, dg)


[JuliaFormatter] reported by reviewdog 🐶

_calc_surface_integral!(backend, du, u, mesh, equations, surface_integral, dg, cache)


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

@kernel function interface_flux_kernel!(surface_flux_values, equations, surface_flux, nodes,


[JuliaFormatter] reported by reviewdog 🐶

@kernel function prolong2boundaries_kernel!(u_boundaries, neighbor_ids, _node_indices, u,


[JuliaFormatter] reported by reviewdog 🐶

boundary_interp_factor_1, boundary_interp_factor_2,


[JuliaFormatter] reported by reviewdog 🐶

surface_flux_values[v, l, m, 1, element] *
boundary_interp_factor_1)


[JuliaFormatter] reported by reviewdog 🐶

surface_flux_values[v, l, m, 2,
element] *
boundary_interp_factor_2)


[JuliaFormatter] reported by reviewdog 🐶

surface_flux_values[v, l, m, 3, element] *
boundary_interp_factor_1)


[JuliaFormatter] reported by reviewdog 🐶

surface_flux_values[v, l, m, 4,
element] *
boundary_interp_factor_2)


[JuliaFormatter] reported by reviewdog 🐶

surface_flux_values[v, l, m, 5, element] *
boundary_interp_factor_1)


[JuliaFormatter] reported by reviewdog 🐶

surface_flux_values[v, l, m, 6,
element] *
boundary_interp_factor_2)


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

@inline function _prolong2mpimortars!(backend::Backend, cache, u,


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}},


[JuliaFormatter] reported by reviewdog 🐶

_calc_mpi_mortar_flux!(backend, surface_flux_values, mesh, nonconservative_terms, equations,


[JuliaFormatter] reported by reviewdog 🐶

mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}},


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

mutable struct P4estMPICache{BufferType <: DenseVector, VecInt <: DenseVector{<:Integer}}


[JuliaFormatter] reported by reviewdog 🐶

mpi_neighbor_interfaces = Adapt.adapt_structure(to, mpi_cache.mpi_neighbor_interfaces)


[JuliaFormatter] reported by reviewdog 🐶

return P4estMPICache{BufferType, VecInt}(mpi_neighbor_ranks, mpi_neighbor_interfaces,


[JuliaFormatter] reported by reviewdog 🐶

mesh::Union{StructuredMesh{3}, P4estMesh{3}, T8codeMesh{3}},


[JuliaFormatter] reported by reviewdog 🐶

0.07601887903440395,


[JuliaFormatter] reported by reviewdog 🐶

4.848310144356219,

@@ -37,7 +38,7 @@ semi = SemidiscretizationHyperbolic(mesh, equations, initial_condition, solver,
# ODE solvers, callbacks etc.

tspan = (0.0, 5.0)
ode = semidiscretize(semi, tspan)
ode = semidiscretize(semi, tspan; adapt_to=CuArray)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
ode = semidiscretize(semi, tspan; adapt_to=CuArray)
ode = semidiscretize(semi, tspan; adapt_to = CuArray)

Comment on lines +15 to +26
function initial_condition_taylor_green_vortex(x, t, equations::CompressibleEulerEquations3D)
A = 1.0 # magnitude of speed
Ms = 0.1 # maximum Mach number

rho = 1.0
v1 = A * sin(x[1]) * cos(x[2]) * cos(x[3])
v2 = -A * cos(x[1]) * sin(x[2]) * cos(x[3])
v3 = 0.0
p = (A / Ms)^2 * rho / equations.gamma # scaling to get Ms
p = p + 1.0/16.0 * A^2 * rho * (cos(2*x[1])*cos(2*x[3]) + 2*cos(2*x[2]) + 2*cos(2*x[1]) + cos(2*x[2])*cos(2*x[3]))

return prim2cons(SVector(rho, v1, v2, v3, p), equations)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
function initial_condition_taylor_green_vortex(x, t, equations::CompressibleEulerEquations3D)
A = 1.0 # magnitude of speed
Ms = 0.1 # maximum Mach number
rho = 1.0
v1 = A * sin(x[1]) * cos(x[2]) * cos(x[3])
v2 = -A * cos(x[1]) * sin(x[2]) * cos(x[3])
v3 = 0.0
p = (A / Ms)^2 * rho / equations.gamma # scaling to get Ms
p = p + 1.0/16.0 * A^2 * rho * (cos(2*x[1])*cos(2*x[3]) + 2*cos(2*x[2]) + 2*cos(2*x[1]) + cos(2*x[2])*cos(2*x[3]))
return prim2cons(SVector(rho, v1, v2, v3, p), equations)
function initial_condition_taylor_green_vortex(x, t,
equations::CompressibleEulerEquations3D)
A = 1.0 # magnitude of speed
Ms = 0.1 # maximum Mach number
rho = 1.0
v1 = A * sin(x[1]) * cos(x[2]) * cos(x[3])
v2 = -A * cos(x[1]) * sin(x[2]) * cos(x[3])
v3 = 0.0
p = (A / Ms)^2 * rho / equations.gamma # scaling to get Ms
p = p +
1.0 / 16.0 * A^2 * rho *
(cos(2 * x[1]) * cos(2 * x[3]) + 2 * cos(2 * x[2]) + 2 * cos(2 * x[1]) +
cos(2 * x[2]) * cos(2 * x[3]))
return prim2cons(SVector(rho, v1, v2, v3, p), equations)

Comment on lines +31 to +32
solver = DGSEM(polydeg=3, surface_flux=flux_lax_friedrichs,
volume_integral=VolumeIntegralFluxDifferencing(flux_lax_friedrichs))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
solver = DGSEM(polydeg=3, surface_flux=flux_lax_friedrichs,
volume_integral=VolumeIntegralFluxDifferencing(flux_lax_friedrichs))
solver = DGSEM(polydeg = 3, surface_flux = flux_lax_friedrichs,
volume_integral = VolumeIntegralFluxDifferencing(flux_lax_friedrichs))

volume_integral=VolumeIntegralFluxDifferencing(flux_lax_friedrichs))

coordinates_min = (-1.0, -1.0, -1.0) .* pi
coordinates_max = ( 1.0, 1.0, 1.0) .* pi

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
coordinates_max = ( 1.0, 1.0, 1.0) .* pi
coordinates_max = (1.0, 1.0, 1.0) .* pi

Comment on lines +39 to +41
mesh = P4estMesh(trees_per_dimension, polydeg=1,
coordinates_min=coordinates_min, coordinates_max=coordinates_max,
initial_refinement_level=2)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
mesh = P4estMesh(trees_per_dimension, polydeg=1,
coordinates_min=coordinates_min, coordinates_max=coordinates_max,
initial_refinement_level=2)
mesh = P4estMesh(trees_per_dimension, polydeg = 1,
coordinates_min = coordinates_min, coordinates_max = coordinates_max,
initial_refinement_level = 2)

coupling_function31)
boundary_conditions_y_pos3 = BoundaryConditionCoupled(1, (:i_forward, :begin), Float64,
boundary_conditions_y_pos3 = BoundaryConditionCoupled(1, (Indexing.i_forward, Indexing.first), Float64,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
boundary_conditions_y_pos3 = BoundaryConditionCoupled(1, (Indexing.i_forward, Indexing.first), Float64,
boundary_conditions_y_pos3 = BoundaryConditionCoupled(1,
(Indexing.i_forward, Indexing.first),
Float64,

@@ -157,13 +158,13 @@ coupling_function43 = (x, u, equations_other, equations_own) -> u
coupling_function42 = (x, u, equations_other, equations_own) -> u

# Define the coupling boundary conditions and the system it is coupled to.
boundary_conditions_x_neg4 = BoundaryConditionCoupled(3, (:end, :i_forward), Float64,
boundary_conditions_x_neg4 = BoundaryConditionCoupled(3, (Indexing.last, Indexing.i_forward), Float64,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
boundary_conditions_x_neg4 = BoundaryConditionCoupled(3, (Indexing.last, Indexing.i_forward), Float64,
boundary_conditions_x_neg4 = BoundaryConditionCoupled(3,
(Indexing.last, Indexing.i_forward),
Float64,

coupling_function43)
boundary_conditions_x_pos4 = BoundaryConditionCoupled(3, (:begin, :i_forward), Float64,
boundary_conditions_x_pos4 = BoundaryConditionCoupled(3, (Indexing.first, Indexing.i_forward), Float64,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
boundary_conditions_x_pos4 = BoundaryConditionCoupled(3, (Indexing.first, Indexing.i_forward), Float64,
boundary_conditions_x_pos4 = BoundaryConditionCoupled(3,
(Indexing.first, Indexing.i_forward),
Float64,

coupling_function43)
boundary_conditions_y_neg4 = BoundaryConditionCoupled(2, (:i_forward, :end), Float64,
boundary_conditions_y_neg4 = BoundaryConditionCoupled(2, (Indexing.i_forward, Indexing.last), Float64,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
boundary_conditions_y_neg4 = BoundaryConditionCoupled(2, (Indexing.i_forward, Indexing.last), Float64,
boundary_conditions_y_neg4 = BoundaryConditionCoupled(2,
(Indexing.i_forward, Indexing.last),
Float64,

coupling_function42)
boundary_conditions_y_pos4 = BoundaryConditionCoupled(2, (:i_forward, :begin), Float64,
boundary_conditions_y_pos4 = BoundaryConditionCoupled(2, (Indexing.i_forward, Indexing.first), Float64,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
boundary_conditions_y_pos4 = BoundaryConditionCoupled(2, (Indexing.i_forward, Indexing.first), Float64,
boundary_conditions_y_pos4 = BoundaryConditionCoupled(2,
(Indexing.i_forward, Indexing.first),
Float64,

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

JuliaFormatter

[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

neighbor_ids = unsafe_wrap_or_alloc(to, _neighbor_ids, size(interfaces.neighbor_ids))
node_indices = unsafe_wrap_or_alloc(to, _node_indices, size(interfaces.node_indices))


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

interfaces.node_indices[side, interface_id] = (Indexing.first, surface_index1,


[JuliaFormatter] reported by reviewdog 🐶

interfaces.node_indices[side, interface_id] = (Indexing.last, surface_index1,


[JuliaFormatter] reported by reviewdog 🐶

interfaces.node_indices[side, interface_id] = (surface_index1, Indexing.first,


[JuliaFormatter] reported by reviewdog 🐶

interfaces.node_indices[side, interface_id] = (surface_index1, Indexing.last,


[JuliaFormatter] reported by reviewdog 🐶

surface_index2, Indexing.first)


[JuliaFormatter] reported by reviewdog 🐶

surface_index2, Indexing.last)


[JuliaFormatter] reported by reviewdog 🐶

boundaries.node_indices[boundary_id] = (Indexing.first, Indexing.i_forward, Indexing.j_forward)


[JuliaFormatter] reported by reviewdog 🐶

boundaries.node_indices[boundary_id] = (Indexing.last, Indexing.i_forward, Indexing.j_forward)


[JuliaFormatter] reported by reviewdog 🐶

boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.first, Indexing.j_forward)


[JuliaFormatter] reported by reviewdog 🐶

boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.last, Indexing.j_forward)


[JuliaFormatter] reported by reviewdog 🐶

boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.j_forward, Indexing.first)


[JuliaFormatter] reported by reviewdog 🐶

boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.j_forward, Indexing.last)


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

_prolong2mortars!(backend, cache, u, mesh, equations, mortar_l2, surface_integral, dg)


[JuliaFormatter] reported by reviewdog 🐶

_calc_surface_integral!(backend, du, u, mesh, equations, surface_integral, dg, cache)


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

@kernel function interface_flux_kernel!(surface_flux_values, equations, surface_flux, nodes,


[JuliaFormatter] reported by reviewdog 🐶

@kernel function prolong2boundaries_kernel!(u_boundaries, neighbor_ids, _node_indices, u,


[JuliaFormatter] reported by reviewdog 🐶

boundary_interp_factor_1, boundary_interp_factor_2,


[JuliaFormatter] reported by reviewdog 🐶

surface_flux_values[v, l, m, 1, element] *
boundary_interp_factor_1)


[JuliaFormatter] reported by reviewdog 🐶

surface_flux_values[v, l, m, 2,
element] *
boundary_interp_factor_2)


[JuliaFormatter] reported by reviewdog 🐶

surface_flux_values[v, l, m, 3, element] *
boundary_interp_factor_1)


[JuliaFormatter] reported by reviewdog 🐶

surface_flux_values[v, l, m, 4,
element] *
boundary_interp_factor_2)


[JuliaFormatter] reported by reviewdog 🐶

surface_flux_values[v, l, m, 5, element] *
boundary_interp_factor_1)


[JuliaFormatter] reported by reviewdog 🐶

surface_flux_values[v, l, m, 6,
element] *
boundary_interp_factor_2)


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

@inline function _prolong2mpimortars!(backend::Backend, cache, u,


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}},


[JuliaFormatter] reported by reviewdog 🐶

_calc_mpi_mortar_flux!(backend, surface_flux_values, mesh, nonconservative_terms, equations,


[JuliaFormatter] reported by reviewdog 🐶

mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}},


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

mutable struct P4estMPICache{BufferType <: DenseVector, VecInt <: DenseVector{<:Integer}}


[JuliaFormatter] reported by reviewdog 🐶

mpi_neighbor_interfaces = Adapt.adapt_structure(to, mpi_cache.mpi_neighbor_interfaces)


[JuliaFormatter] reported by reviewdog 🐶

return P4estMPICache{BufferType, VecInt}(mpi_neighbor_ranks, mpi_neighbor_interfaces,


[JuliaFormatter] reported by reviewdog 🐶

mesh::Union{StructuredMesh{3}, P4estMesh{3}, T8codeMesh{3}},


[JuliaFormatter] reported by reviewdog 🐶

0.07601887903440395,


[JuliaFormatter] reported by reviewdog 🐶

4.848310144356219,


function initial_condition_taylor_green_vortex(x, t,
equations::CompressibleEulerEquations3D)
A = 1.0 # magnitude of speed

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
A = 1.0 # magnitude of speed
A = 1.0 # magnitude of speed

Comment on lines +17 to +22
v1 = A * sin(x[1]) * cos(x[2]) * cos(x[3])
v2 = -A * cos(x[1]) * sin(x[2]) * cos(x[3])
v3 = 0.0
p = (A / Ms)^2 * rho / equations.gamma # scaling to get Ms
p = p + 1.0/16.0 * A^2 * rho * (cos(2*x[1])*cos(2*x[3]) +
2*cos(2*x[2]) + 2*cos(2*x[1]) + cos(2*x[2])*cos(2*x[3]))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
v1 = A * sin(x[1]) * cos(x[2]) * cos(x[3])
v2 = -A * cos(x[1]) * sin(x[2]) * cos(x[3])
v3 = 0.0
p = (A / Ms)^2 * rho / equations.gamma # scaling to get Ms
p = p + 1.0/16.0 * A^2 * rho * (cos(2*x[1])*cos(2*x[3]) +
2*cos(2*x[2]) + 2*cos(2*x[1]) + cos(2*x[2])*cos(2*x[3]))
v1 = A * sin(x[1]) * cos(x[2]) * cos(x[3])
v2 = -A * cos(x[1]) * sin(x[2]) * cos(x[3])
v3 = 0.0
p = (A / Ms)^2 * rho / equations.gamma # scaling to get Ms
p = p +
1.0 / 16.0 * A^2 * rho *
(cos(2 * x[1]) * cos(2 * x[3]) +
2 * cos(2 * x[2]) + 2 * cos(2 * x[1]) + cos(2 * x[2]) * cos(2 * x[3]))

Comment on lines +31 to +32
solver = DGSEM(polydeg=5, surface_flux=volume_flux,
volume_integral=VolumeIntegralFluxDifferencing(volume_flux))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
solver = DGSEM(polydeg=5, surface_flux=volume_flux,
volume_integral=VolumeIntegralFluxDifferencing(volume_flux))
solver = DGSEM(polydeg = 5, surface_flux = volume_flux,
volume_integral = VolumeIntegralFluxDifferencing(volume_flux))

volume_integral=VolumeIntegralFluxDifferencing(volume_flux))

coordinates_min = (-1.0, -1.0, -1.0) .* pi
coordinates_max = ( 1.0, 1.0, 1.0) .* pi

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
coordinates_max = ( 1.0, 1.0, 1.0) .* pi
coordinates_max = (1.0, 1.0, 1.0) .* pi

Comment on lines +40 to +42
mesh = P4estMesh(trees_per_dimension, polydeg=1,
coordinates_min=coordinates_min, coordinates_max=coordinates_max,
periodicity=true, initial_refinement_level=initial_refinement_level)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
mesh = P4estMesh(trees_per_dimension, polydeg=1,
coordinates_min=coordinates_min, coordinates_max=coordinates_max,
periodicity=true, initial_refinement_level=initial_refinement_level)
mesh = P4estMesh(trees_per_dimension, polydeg = 1,
coordinates_min = coordinates_min, coordinates_max = coordinates_max,
periodicity = true, initial_refinement_level = initial_refinement_level)

performance_counter = PerformanceCounter()

cache::Cache,
performance_counter::PerformanceCounter) where {Mesh, Equations,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
performance_counter::PerformanceCounter) where {Mesh, Equations,
performance_counter::PerformanceCounter) where {
Mesh,
Equations,

BoundaryConditions,
SourceTerms,
Solver,
Cache}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
Cache}
Cache
}

SemidiscretizationHyperbolic{typeof(mesh), typeof(equations),
typeof(initial_condition),
typeof(_boundary_conditions), typeof(source_terms),
typeof(solver), typeof(cache)}(mesh, equations,
initial_condition,
_boundary_conditions,
source_terms, solver,
cache)
cache, performance_counter)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
cache, performance_counter)
cache,
performance_counter)

initial_condition,
boundary_conditions,
source_terms, solver,
cache, performance_counter)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
cache, performance_counter)
cache,
performance_counter)

@@ -752,6 +762,37 @@ function compute_coefficients!(u, func, t, mesh::AbstractMesh{3}, equations, dg:
end
end

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

JuliaFormatter

[JuliaFormatter] reported by reviewdog 🐶

@inline function _prolong2mpimortars!(backend::Backend, cache, u,


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}},


[JuliaFormatter] reported by reviewdog 🐶

_calc_mpi_mortar_flux!(backend, surface_flux_values, mesh, nonconservative_terms, equations,


[JuliaFormatter] reported by reviewdog 🐶

mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}},


[JuliaFormatter] reported by reviewdog 🐶


[JuliaFormatter] reported by reviewdog 🐶

mutable struct P4estMPICache{BufferType <: DenseVector, VecInt <: DenseVector{<:Integer}}


[JuliaFormatter] reported by reviewdog 🐶

mpi_neighbor_interfaces = Adapt.adapt_structure(to, mpi_cache.mpi_neighbor_interfaces)


[JuliaFormatter] reported by reviewdog 🐶

return P4estMPICache{BufferType, VecInt}(mpi_neighbor_ranks, mpi_neighbor_interfaces,


[JuliaFormatter] reported by reviewdog 🐶

mesh::Union{StructuredMesh{3}, P4estMesh{3}, T8codeMesh{3}},


[JuliaFormatter] reported by reviewdog 🐶

0.07601887903440395,


[JuliaFormatter] reported by reviewdog 🐶

4.848310144356219,

_contravariant_vectors = Adapt.adapt_structure(to, elements._contravariant_vectors)
_inverse_jacobian = Adapt.adapt_structure(to, elements._inverse_jacobian)
_surface_flux_values = Adapt.adapt_structure(to, elements._surface_flux_values)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change

Comment on lines +308 to +309
neighbor_ids = unsafe_wrap_or_alloc(to, _neighbor_ids, size(interfaces.neighbor_ids))
node_indices = unsafe_wrap_or_alloc(to, _node_indices, size(interfaces.node_indices))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
neighbor_ids = unsafe_wrap_or_alloc(to, _neighbor_ids, size(interfaces.neighbor_ids))
node_indices = unsafe_wrap_or_alloc(to, _node_indices, size(interfaces.node_indices))
neighbor_ids = unsafe_wrap_or_alloc(to, _neighbor_ids,
size(interfaces.neighbor_ids))
node_indices = unsafe_wrap_or_alloc(to, _node_indices,
size(interfaces.node_indices))

neighbor_ids = Adapt.adapt_structure(to, boundaries.neighbor_ids)
node_indices = Adapt.adapt_structure(to, boundaries.node_indices)
name = boundaries.name

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change

neighbor_ids = unsafe_wrap_or_alloc(to, _neighbor_ids, size(mortars.neighbor_ids))
node_indices = unsafe_wrap_or_alloc(to, _node_indices, size(mortars.node_indices))


Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change

@@ -94,28 +94,28 @@ end

if faces[side] == 0
# Index face in negative x-direction
interfaces.node_indices[side, interface_id] = (:begin, surface_index1,
interfaces.node_indices[side, interface_id] = (Indexing.first, surface_index1,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
interfaces.node_indices[side, interface_id] = (Indexing.first, surface_index1,
interfaces.node_indices[side, interface_id] = (Indexing.first,
surface_index1,

Comment on lines +517 to +519
surface_flux_values[v, l, m, 2,
element] *
boundary_interp_factor_2)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
surface_flux_values[v, l, m, 2,
element] *
boundary_interp_factor_2)
surface_flux_values[v, l, m, 2,
element] *
boundary_interp_factor_2)

Comment on lines +523 to +524
surface_flux_values[v, l, m, 3, element] *
boundary_interp_factor_1)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
surface_flux_values[v, l, m, 3, element] *
boundary_interp_factor_1)
surface_flux_values[v, l, m, 3, element] *
boundary_interp_factor_1)

Comment on lines +528 to +530
surface_flux_values[v, l, m, 4,
element] *
boundary_interp_factor_2)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
surface_flux_values[v, l, m, 4,
element] *
boundary_interp_factor_2)
surface_flux_values[v, l, m, 4,
element] *
boundary_interp_factor_2)

Comment on lines +534 to +535
surface_flux_values[v, l, m, 5, element] *
boundary_interp_factor_1)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
surface_flux_values[v, l, m, 5, element] *
boundary_interp_factor_1)
surface_flux_values[v, l, m, 5, element] *
boundary_interp_factor_1)

Comment on lines +539 to +541
surface_flux_values[v, l, m, 6,
element] *
boundary_interp_factor_2)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
surface_flux_values[v, l, m, 6,
element] *
boundary_interp_factor_2)
surface_flux_values[v, l, m, 6,
element] *
boundary_interp_factor_2)

Comment on lines +799 to +800
_calc_surface_integral!(backend, du, u, mesh, equations, surface_integral, dg, cache)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
_calc_surface_integral!(backend, du, u, mesh, equations, surface_integral, dg, cache)
_calc_surface_integral!(backend, du, u, mesh, equations, surface_integral, dg,
cache)

end
end

@inline function _prolong2mpimortars!(backend::Backend, cache, u,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
@inline function _prolong2mpimortars!(backend::Backend, cache, u,
@inline function _prolong2mpimortars!(backend::Backend, cache, u,

surface_integral, dg::DG, cache)
return nothing
end
end # @muladd

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
end # @muladd
end # @muladd

end

@inline function _prolong2mpimortars!(backend::Nothing, cache, u,
mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}},

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}},
mesh::Union{ParallelP4estMesh{3},
ParallelT8codeMesh{3}},

@@ -416,6 +448,16 @@
nonconservative_terms, equations,
mortar_l2::LobattoLegendreMortarL2,
surface_integral, dg::DG, cache)
backend = backend_or_nothing(cache.mpi_mortars)
_calc_mpi_mortar_flux!(backend, surface_flux_values, mesh, nonconservative_terms, equations,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
_calc_mpi_mortar_flux!(backend, surface_flux_values, mesh, nonconservative_terms, equations,
_calc_mpi_mortar_flux!(backend, surface_flux_values, mesh, nonconservative_terms,
equations,


function Adapt.adapt_structure(to, mpi_cache::P4estMPICache)
mpi_neighbor_ranks = mpi_cache.mpi_neighbor_ranks
mpi_neighbor_interfaces = Adapt.adapt_structure(to, mpi_cache.mpi_neighbor_interfaces)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
mpi_neighbor_interfaces = Adapt.adapt_structure(to, mpi_cache.mpi_neighbor_interfaces)
mpi_neighbor_interfaces = Adapt.adapt_structure(to,
mpi_cache.mpi_neighbor_interfaces)

@assert eltype(mpi_send_buffers) == eltype(mpi_recv_buffers)
BufferType = eltype(mpi_send_buffers)
VecInt = eltype(mpi_neighbor_interfaces)
return P4estMPICache{BufferType, VecInt}(mpi_neighbor_ranks, mpi_neighbor_interfaces,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
return P4estMPICache{BufferType, VecInt}(mpi_neighbor_ranks, mpi_neighbor_interfaces,
return P4estMPICache{BufferType, VecInt}(mpi_neighbor_ranks,
mpi_neighbor_interfaces,

end

@inline function _apply_jacobian!(::Nothing, du,
mesh::Union{StructuredMesh{3}, P4estMesh{3}, T8codeMesh{3}},

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
mesh::Union{StructuredMesh{3}, P4estMesh{3}, T8codeMesh{3}},
mesh::Union{StructuredMesh{3}, P4estMesh{3},
T8codeMesh{3}},

0.00021710076010951073,
0.0004386796338203878,
0.00020836270267103122,
0.07601887903440395,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
0.07601887903440395,
0.07601887903440395

0.02980358831035801,
0.048476331898047564,
0.02200137344113612,
4.848310144356219,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
4.848310144356219,
4.848310144356219


# start simulation with tiny final time to trigger precompilation
duration_precompile = @elapsed trixi_include(elixir_path,
tspan=(0.0, 1e-14))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
tspan=(0.0, 1e-14))
tspan = (0.0, 1e-14))

end

# start the real simulation
duration_elixir = @elapsed trixi_include(elixir_path, maxiters=maxiters)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
duration_elixir = @elapsed trixi_include(elixir_path, maxiters=maxiters)
duration_elixir = @elapsed trixi_include(elixir_path, maxiters = maxiters)

# reduce metrics per rank
open("metrics.out", "w") do io
for (key, _) in gathered_metrics[1]
println(io, key, ": ", mapreduce(x->x[key], min, gathered_metrics))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
println(io, key, ": ", mapreduce(x->x[key], min, gathered_metrics))
println(io, key, ": ", mapreduce(x -> x[key], min, gathered_metrics))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants