-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GPU offloading #2
base: main
Are you sure you want to change the base?
Conversation
…mework/Trixi.jl into feature-t8code-curved-geometry
it refers to level of refinement in lat lon direction, not number of tree as in the p4est version
…mework/Trixi.jl into feature-t8code-curved-geometry
…ubed-sphere-setup
MPI.jl's reduce currently does not work for custom operators (such as Trixi's min/max) on ARM
…ubed-sphere-setup
This reverts commit 6607023.
Concerning Some containers contain symbols, e.g.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit
JuliaFormatter
[JuliaFormatter] reported by reviewdog 🐶
x_neg = BoundaryConditionCoupled(2, (Indexing.last, Indexing.i_forward), Float64, |
[JuliaFormatter] reported by reviewdog 🐶
x_pos = BoundaryConditionCoupled(2, (Indexing.first, Indexing.i_forward), Float64, |
[JuliaFormatter] reported by reviewdog 🐶
x_neg = BoundaryConditionCoupled(1, (Indexing.last, Indexing.i_forward), Float64, |
[JuliaFormatter] reported by reviewdog 🐶
x_pos = BoundaryConditionCoupled(1, (Indexing.first, Indexing.i_forward), Float64, |
[JuliaFormatter] reported by reviewdog 🐶
boundary_conditions1 = (x_neg = BoundaryConditionCoupled(2, (Indexing.last, Indexing.i_forward), Float64, |
[JuliaFormatter] reported by reviewdog 🐶
x_pos = BoundaryConditionCoupled(2, (Indexing.first, Indexing.i_forward), Float64, |
[JuliaFormatter] reported by reviewdog 🐶
boundary_conditions2 = (x_neg = BoundaryConditionCoupled(1, (Indexing.last, Indexing.i_forward), Float64, |
[JuliaFormatter] reported by reviewdog 🐶
x_pos = BoundaryConditionCoupled(1, (Indexing.first, Indexing.i_forward), Float64, |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/callbacks_step/analysis.jl
Line 624 in d942046
l2_error, linf_error = calc_error_norms(u_ode_cpu, sol.t[end], analyzer, semi_cpu, |
[JuliaFormatter] reported by reviewdog 🐶
if !(typeof(semi) <: SemidiscretizationHyperbolic) && !(typeof(semi.mesh) <: P4estMesh) |
[JuliaFormatter] reported by reviewdog 🐶
performance_counter::PerformanceCounter) where {Mesh, Equations, |
[JuliaFormatter] reported by reviewdog 🐶
Cache} |
[JuliaFormatter] reported by reviewdog 🐶
cache, performance_counter) |
[JuliaFormatter] reported by reviewdog 🐶
cache, performance_counter) |
[JuliaFormatter] reported by reviewdog 🐶
Line 764 in d942046
[JuliaFormatter] reported by reviewdog 🐶
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/containers.jl
Lines 308 to 309 in d942046
neighbor_ids = unsafe_wrap_or_alloc(to, _neighbor_ids, size(interfaces.neighbor_ids)) | |
node_indices = unsafe_wrap_or_alloc(to, _node_indices, size(interfaces.node_indices)) |
[JuliaFormatter] reported by reviewdog 🐶
[JuliaFormatter] reported by reviewdog 🐶
[JuliaFormatter] reported by reviewdog 🐶
interfaces.node_indices[side, interface_id] = (Indexing.first, surface_index1, |
[JuliaFormatter] reported by reviewdog 🐶
interfaces.node_indices[side, interface_id] = (Indexing.last, surface_index1, |
[JuliaFormatter] reported by reviewdog 🐶
interfaces.node_indices[side, interface_id] = (surface_index1, Indexing.first, |
[JuliaFormatter] reported by reviewdog 🐶
interfaces.node_indices[side, interface_id] = (surface_index1, Indexing.last, |
[JuliaFormatter] reported by reviewdog 🐶
surface_index2, Indexing.first) |
[JuliaFormatter] reported by reviewdog 🐶
surface_index2, Indexing.last) |
[JuliaFormatter] reported by reviewdog 🐶
boundaries.node_indices[boundary_id] = (Indexing.first, Indexing.i_forward, Indexing.j_forward) |
[JuliaFormatter] reported by reviewdog 🐶
boundaries.node_indices[boundary_id] = (Indexing.last, Indexing.i_forward, Indexing.j_forward) |
[JuliaFormatter] reported by reviewdog 🐶
boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.first, Indexing.j_forward) |
[JuliaFormatter] reported by reviewdog 🐶
boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.last, Indexing.j_forward) |
[JuliaFormatter] reported by reviewdog 🐶
boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.j_forward, Indexing.first) |
[JuliaFormatter] reported by reviewdog 🐶
boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.j_forward, Indexing.last) |
[JuliaFormatter] reported by reviewdog 🐶
typeof(_u), |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d.jl
Line 457 in d942046
_prolong2mortars!(backend, cache, u, mesh, equations, mortar_l2, surface_integral, dg) |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d.jl
Lines 801 to 802 in d942046
_calc_surface_integral!(backend, du, u, mesh, equations, surface_integral, dg, cache) | |
[JuliaFormatter] reported by reviewdog 🐶
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Line 264 in d942046
@kernel function interface_flux_kernel!(surface_flux_values, equations, surface_flux, nodes, |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Line 353 in d942046
@kernel function prolong2boundaries_kernel!(u_boundaries, neighbor_ids, _node_indices, u, |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Line 504 in d942046
boundary_interp_factor_1, boundary_interp_factor_2, |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Lines 512 to 513 in d942046
surface_flux_values[v, l, m, 1, element] * | |
boundary_interp_factor_1) |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Lines 517 to 519 in d942046
surface_flux_values[v, l, m, 2, | |
element] * | |
boundary_interp_factor_2) |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Lines 523 to 524 in d942046
surface_flux_values[v, l, m, 3, element] * | |
boundary_interp_factor_1) |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Lines 528 to 530 in d942046
surface_flux_values[v, l, m, 4, | |
element] * | |
boundary_interp_factor_2) |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Lines 534 to 535 in d942046
surface_flux_values[v, l, m, 5, element] * | |
boundary_interp_factor_1) |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Lines 539 to 541 in d942046
surface_flux_values[v, l, m, 6, | |
element] * | |
boundary_interp_factor_2) |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Line 591 in d942046
end # @muladd |
[JuliaFormatter] reported by reviewdog 🐶
@inline function _prolong2mpimortars!(backend::Backend, cache, u, |
[JuliaFormatter] reported by reviewdog 🐶
end # @muladd |
[JuliaFormatter] reported by reviewdog 🐶
mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}}, |
[JuliaFormatter] reported by reviewdog 🐶
_calc_mpi_mortar_flux!(backend, surface_flux_values, mesh, nonconservative_terms, equations, |
[JuliaFormatter] reported by reviewdog 🐶
mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}}, |
[JuliaFormatter] reported by reviewdog 🐶
end # @muladd |
[JuliaFormatter] reported by reviewdog 🐶
mutable struct P4estMPICache{BufferType <: DenseVector, VecInt <: DenseVector{<:Integer}} |
[JuliaFormatter] reported by reviewdog 🐶
mpi_neighbor_interfaces = Adapt.adapt_structure(to, mpi_cache.mpi_neighbor_interfaces) |
[JuliaFormatter] reported by reviewdog 🐶
return P4estMPICache{BufferType, VecInt}(mpi_neighbor_ranks, mpi_neighbor_interfaces, |
[JuliaFormatter] reported by reviewdog 🐶
mesh::Union{StructuredMesh{3}, P4estMesh{3}, T8codeMesh{3}}, |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/test/test_t8code_3d.jl
Line 349 in d942046
0.07601887903440395, |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/test/test_t8code_3d.jl
Line 356 in d942046
4.848310144356219, |
@@ -37,7 +38,7 @@ semi = SemidiscretizationHyperbolic(mesh, equations, initial_condition, solver, | |||
# ODE solvers, callbacks etc. | |||
|
|||
tspan = (0.0, 5.0) | |||
ode = semidiscretize(semi, tspan) | |||
ode = semidiscretize(semi, tspan; adapt_to=CuArray) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
ode = semidiscretize(semi, tspan; adapt_to=CuArray) | |
ode = semidiscretize(semi, tspan; adapt_to = CuArray) |
function initial_condition_taylor_green_vortex(x, t, equations::CompressibleEulerEquations3D) | ||
A = 1.0 # magnitude of speed | ||
Ms = 0.1 # maximum Mach number | ||
|
||
rho = 1.0 | ||
v1 = A * sin(x[1]) * cos(x[2]) * cos(x[3]) | ||
v2 = -A * cos(x[1]) * sin(x[2]) * cos(x[3]) | ||
v3 = 0.0 | ||
p = (A / Ms)^2 * rho / equations.gamma # scaling to get Ms | ||
p = p + 1.0/16.0 * A^2 * rho * (cos(2*x[1])*cos(2*x[3]) + 2*cos(2*x[2]) + 2*cos(2*x[1]) + cos(2*x[2])*cos(2*x[3])) | ||
|
||
return prim2cons(SVector(rho, v1, v2, v3, p), equations) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
function initial_condition_taylor_green_vortex(x, t, equations::CompressibleEulerEquations3D) | |
A = 1.0 # magnitude of speed | |
Ms = 0.1 # maximum Mach number | |
rho = 1.0 | |
v1 = A * sin(x[1]) * cos(x[2]) * cos(x[3]) | |
v2 = -A * cos(x[1]) * sin(x[2]) * cos(x[3]) | |
v3 = 0.0 | |
p = (A / Ms)^2 * rho / equations.gamma # scaling to get Ms | |
p = p + 1.0/16.0 * A^2 * rho * (cos(2*x[1])*cos(2*x[3]) + 2*cos(2*x[2]) + 2*cos(2*x[1]) + cos(2*x[2])*cos(2*x[3])) | |
return prim2cons(SVector(rho, v1, v2, v3, p), equations) | |
function initial_condition_taylor_green_vortex(x, t, | |
equations::CompressibleEulerEquations3D) | |
A = 1.0 # magnitude of speed | |
Ms = 0.1 # maximum Mach number | |
rho = 1.0 | |
v1 = A * sin(x[1]) * cos(x[2]) * cos(x[3]) | |
v2 = -A * cos(x[1]) * sin(x[2]) * cos(x[3]) | |
v3 = 0.0 | |
p = (A / Ms)^2 * rho / equations.gamma # scaling to get Ms | |
p = p + | |
1.0 / 16.0 * A^2 * rho * | |
(cos(2 * x[1]) * cos(2 * x[3]) + 2 * cos(2 * x[2]) + 2 * cos(2 * x[1]) + | |
cos(2 * x[2]) * cos(2 * x[3])) | |
return prim2cons(SVector(rho, v1, v2, v3, p), equations) |
solver = DGSEM(polydeg=3, surface_flux=flux_lax_friedrichs, | ||
volume_integral=VolumeIntegralFluxDifferencing(flux_lax_friedrichs)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
solver = DGSEM(polydeg=3, surface_flux=flux_lax_friedrichs, | |
volume_integral=VolumeIntegralFluxDifferencing(flux_lax_friedrichs)) | |
solver = DGSEM(polydeg = 3, surface_flux = flux_lax_friedrichs, | |
volume_integral = VolumeIntegralFluxDifferencing(flux_lax_friedrichs)) |
volume_integral=VolumeIntegralFluxDifferencing(flux_lax_friedrichs)) | ||
|
||
coordinates_min = (-1.0, -1.0, -1.0) .* pi | ||
coordinates_max = ( 1.0, 1.0, 1.0) .* pi |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
coordinates_max = ( 1.0, 1.0, 1.0) .* pi | |
coordinates_max = (1.0, 1.0, 1.0) .* pi |
mesh = P4estMesh(trees_per_dimension, polydeg=1, | ||
coordinates_min=coordinates_min, coordinates_max=coordinates_max, | ||
initial_refinement_level=2) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
mesh = P4estMesh(trees_per_dimension, polydeg=1, | |
coordinates_min=coordinates_min, coordinates_max=coordinates_max, | |
initial_refinement_level=2) | |
mesh = P4estMesh(trees_per_dimension, polydeg = 1, | |
coordinates_min = coordinates_min, coordinates_max = coordinates_max, | |
initial_refinement_level = 2) |
coupling_function31) | ||
boundary_conditions_y_pos3 = BoundaryConditionCoupled(1, (:i_forward, :begin), Float64, | ||
boundary_conditions_y_pos3 = BoundaryConditionCoupled(1, (Indexing.i_forward, Indexing.first), Float64, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
boundary_conditions_y_pos3 = BoundaryConditionCoupled(1, (Indexing.i_forward, Indexing.first), Float64, | |
boundary_conditions_y_pos3 = BoundaryConditionCoupled(1, | |
(Indexing.i_forward, Indexing.first), | |
Float64, |
@@ -157,13 +158,13 @@ coupling_function43 = (x, u, equations_other, equations_own) -> u | |||
coupling_function42 = (x, u, equations_other, equations_own) -> u | |||
|
|||
# Define the coupling boundary conditions and the system it is coupled to. | |||
boundary_conditions_x_neg4 = BoundaryConditionCoupled(3, (:end, :i_forward), Float64, | |||
boundary_conditions_x_neg4 = BoundaryConditionCoupled(3, (Indexing.last, Indexing.i_forward), Float64, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
boundary_conditions_x_neg4 = BoundaryConditionCoupled(3, (Indexing.last, Indexing.i_forward), Float64, | |
boundary_conditions_x_neg4 = BoundaryConditionCoupled(3, | |
(Indexing.last, Indexing.i_forward), | |
Float64, |
coupling_function43) | ||
boundary_conditions_x_pos4 = BoundaryConditionCoupled(3, (:begin, :i_forward), Float64, | ||
boundary_conditions_x_pos4 = BoundaryConditionCoupled(3, (Indexing.first, Indexing.i_forward), Float64, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
boundary_conditions_x_pos4 = BoundaryConditionCoupled(3, (Indexing.first, Indexing.i_forward), Float64, | |
boundary_conditions_x_pos4 = BoundaryConditionCoupled(3, | |
(Indexing.first, Indexing.i_forward), | |
Float64, |
coupling_function43) | ||
boundary_conditions_y_neg4 = BoundaryConditionCoupled(2, (:i_forward, :end), Float64, | ||
boundary_conditions_y_neg4 = BoundaryConditionCoupled(2, (Indexing.i_forward, Indexing.last), Float64, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
boundary_conditions_y_neg4 = BoundaryConditionCoupled(2, (Indexing.i_forward, Indexing.last), Float64, | |
boundary_conditions_y_neg4 = BoundaryConditionCoupled(2, | |
(Indexing.i_forward, Indexing.last), | |
Float64, |
coupling_function42) | ||
boundary_conditions_y_pos4 = BoundaryConditionCoupled(2, (:i_forward, :begin), Float64, | ||
boundary_conditions_y_pos4 = BoundaryConditionCoupled(2, (Indexing.i_forward, Indexing.first), Float64, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
boundary_conditions_y_pos4 = BoundaryConditionCoupled(2, (Indexing.i_forward, Indexing.first), Float64, | |
boundary_conditions_y_pos4 = BoundaryConditionCoupled(2, | |
(Indexing.i_forward, Indexing.first), | |
Float64, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit
JuliaFormatter
[JuliaFormatter] reported by reviewdog 🐶
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/containers.jl
Lines 308 to 309 in 9c857e4
neighbor_ids = unsafe_wrap_or_alloc(to, _neighbor_ids, size(interfaces.neighbor_ids)) | |
node_indices = unsafe_wrap_or_alloc(to, _node_indices, size(interfaces.node_indices)) |
[JuliaFormatter] reported by reviewdog 🐶
[JuliaFormatter] reported by reviewdog 🐶
[JuliaFormatter] reported by reviewdog 🐶
interfaces.node_indices[side, interface_id] = (Indexing.first, surface_index1, |
[JuliaFormatter] reported by reviewdog 🐶
interfaces.node_indices[side, interface_id] = (Indexing.last, surface_index1, |
[JuliaFormatter] reported by reviewdog 🐶
interfaces.node_indices[side, interface_id] = (surface_index1, Indexing.first, |
[JuliaFormatter] reported by reviewdog 🐶
interfaces.node_indices[side, interface_id] = (surface_index1, Indexing.last, |
[JuliaFormatter] reported by reviewdog 🐶
surface_index2, Indexing.first) |
[JuliaFormatter] reported by reviewdog 🐶
surface_index2, Indexing.last) |
[JuliaFormatter] reported by reviewdog 🐶
boundaries.node_indices[boundary_id] = (Indexing.first, Indexing.i_forward, Indexing.j_forward) |
[JuliaFormatter] reported by reviewdog 🐶
boundaries.node_indices[boundary_id] = (Indexing.last, Indexing.i_forward, Indexing.j_forward) |
[JuliaFormatter] reported by reviewdog 🐶
boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.first, Indexing.j_forward) |
[JuliaFormatter] reported by reviewdog 🐶
boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.last, Indexing.j_forward) |
[JuliaFormatter] reported by reviewdog 🐶
boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.j_forward, Indexing.first) |
[JuliaFormatter] reported by reviewdog 🐶
boundaries.node_indices[boundary_id] = (Indexing.i_forward, Indexing.j_forward, Indexing.last) |
[JuliaFormatter] reported by reviewdog 🐶
typeof(_u), |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d.jl
Line 457 in 9c857e4
_prolong2mortars!(backend, cache, u, mesh, equations, mortar_l2, surface_integral, dg) |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d.jl
Lines 801 to 802 in 9c857e4
_calc_surface_integral!(backend, du, u, mesh, equations, surface_integral, dg, cache) | |
[JuliaFormatter] reported by reviewdog 🐶
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Line 264 in 9c857e4
@kernel function interface_flux_kernel!(surface_flux_values, equations, surface_flux, nodes, |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Line 353 in 9c857e4
@kernel function prolong2boundaries_kernel!(u_boundaries, neighbor_ids, _node_indices, u, |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Line 504 in 9c857e4
boundary_interp_factor_1, boundary_interp_factor_2, |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Lines 512 to 513 in 9c857e4
surface_flux_values[v, l, m, 1, element] * | |
boundary_interp_factor_1) |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Lines 517 to 519 in 9c857e4
surface_flux_values[v, l, m, 2, | |
element] * | |
boundary_interp_factor_2) |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Lines 523 to 524 in 9c857e4
surface_flux_values[v, l, m, 3, element] * | |
boundary_interp_factor_1) |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Lines 528 to 530 in 9c857e4
surface_flux_values[v, l, m, 4, | |
element] * | |
boundary_interp_factor_2) |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Lines 534 to 535 in 9c857e4
surface_flux_values[v, l, m, 5, element] * | |
boundary_interp_factor_1) |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Lines 539 to 541 in 9c857e4
surface_flux_values[v, l, m, 6, | |
element] * | |
boundary_interp_factor_2) |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/src/solvers/dgsem_p4est/dg_3d_gpu.jl
Line 591 in 9c857e4
end # @muladd |
[JuliaFormatter] reported by reviewdog 🐶
@inline function _prolong2mpimortars!(backend::Backend, cache, u, |
[JuliaFormatter] reported by reviewdog 🐶
end # @muladd |
[JuliaFormatter] reported by reviewdog 🐶
mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}}, |
[JuliaFormatter] reported by reviewdog 🐶
_calc_mpi_mortar_flux!(backend, surface_flux_values, mesh, nonconservative_terms, equations, |
[JuliaFormatter] reported by reviewdog 🐶
mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}}, |
[JuliaFormatter] reported by reviewdog 🐶
end # @muladd |
[JuliaFormatter] reported by reviewdog 🐶
mutable struct P4estMPICache{BufferType <: DenseVector, VecInt <: DenseVector{<:Integer}} |
[JuliaFormatter] reported by reviewdog 🐶
mpi_neighbor_interfaces = Adapt.adapt_structure(to, mpi_cache.mpi_neighbor_interfaces) |
[JuliaFormatter] reported by reviewdog 🐶
return P4estMPICache{BufferType, VecInt}(mpi_neighbor_ranks, mpi_neighbor_interfaces, |
[JuliaFormatter] reported by reviewdog 🐶
mesh::Union{StructuredMesh{3}, P4estMesh{3}, T8codeMesh{3}}, |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/test/test_t8code_3d.jl
Line 349 in 9c857e4
0.07601887903440395, |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/test/test_t8code_3d.jl
Line 356 in 9c857e4
4.848310144356219, |
|
||
function initial_condition_taylor_green_vortex(x, t, | ||
equations::CompressibleEulerEquations3D) | ||
A = 1.0 # magnitude of speed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
A = 1.0 # magnitude of speed | |
A = 1.0 # magnitude of speed |
v1 = A * sin(x[1]) * cos(x[2]) * cos(x[3]) | ||
v2 = -A * cos(x[1]) * sin(x[2]) * cos(x[3]) | ||
v3 = 0.0 | ||
p = (A / Ms)^2 * rho / equations.gamma # scaling to get Ms | ||
p = p + 1.0/16.0 * A^2 * rho * (cos(2*x[1])*cos(2*x[3]) + | ||
2*cos(2*x[2]) + 2*cos(2*x[1]) + cos(2*x[2])*cos(2*x[3])) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
v1 = A * sin(x[1]) * cos(x[2]) * cos(x[3]) | |
v2 = -A * cos(x[1]) * sin(x[2]) * cos(x[3]) | |
v3 = 0.0 | |
p = (A / Ms)^2 * rho / equations.gamma # scaling to get Ms | |
p = p + 1.0/16.0 * A^2 * rho * (cos(2*x[1])*cos(2*x[3]) + | |
2*cos(2*x[2]) + 2*cos(2*x[1]) + cos(2*x[2])*cos(2*x[3])) | |
v1 = A * sin(x[1]) * cos(x[2]) * cos(x[3]) | |
v2 = -A * cos(x[1]) * sin(x[2]) * cos(x[3]) | |
v3 = 0.0 | |
p = (A / Ms)^2 * rho / equations.gamma # scaling to get Ms | |
p = p + | |
1.0 / 16.0 * A^2 * rho * | |
(cos(2 * x[1]) * cos(2 * x[3]) + | |
2 * cos(2 * x[2]) + 2 * cos(2 * x[1]) + cos(2 * x[2]) * cos(2 * x[3])) |
solver = DGSEM(polydeg=5, surface_flux=volume_flux, | ||
volume_integral=VolumeIntegralFluxDifferencing(volume_flux)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
solver = DGSEM(polydeg=5, surface_flux=volume_flux, | |
volume_integral=VolumeIntegralFluxDifferencing(volume_flux)) | |
solver = DGSEM(polydeg = 5, surface_flux = volume_flux, | |
volume_integral = VolumeIntegralFluxDifferencing(volume_flux)) |
volume_integral=VolumeIntegralFluxDifferencing(volume_flux)) | ||
|
||
coordinates_min = (-1.0, -1.0, -1.0) .* pi | ||
coordinates_max = ( 1.0, 1.0, 1.0) .* pi |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
coordinates_max = ( 1.0, 1.0, 1.0) .* pi | |
coordinates_max = (1.0, 1.0, 1.0) .* pi |
mesh = P4estMesh(trees_per_dimension, polydeg=1, | ||
coordinates_min=coordinates_min, coordinates_max=coordinates_max, | ||
periodicity=true, initial_refinement_level=initial_refinement_level) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
mesh = P4estMesh(trees_per_dimension, polydeg=1, | |
coordinates_min=coordinates_min, coordinates_max=coordinates_max, | |
periodicity=true, initial_refinement_level=initial_refinement_level) | |
mesh = P4estMesh(trees_per_dimension, polydeg = 1, | |
coordinates_min = coordinates_min, coordinates_max = coordinates_max, | |
periodicity = true, initial_refinement_level = initial_refinement_level) |
performance_counter = PerformanceCounter() | ||
|
||
cache::Cache, | ||
performance_counter::PerformanceCounter) where {Mesh, Equations, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
performance_counter::PerformanceCounter) where {Mesh, Equations, | |
performance_counter::PerformanceCounter) where { | |
Mesh, | |
Equations, |
BoundaryConditions, | ||
SourceTerms, | ||
Solver, | ||
Cache} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
Cache} | |
Cache | |
} |
SemidiscretizationHyperbolic{typeof(mesh), typeof(equations), | ||
typeof(initial_condition), | ||
typeof(_boundary_conditions), typeof(source_terms), | ||
typeof(solver), typeof(cache)}(mesh, equations, | ||
initial_condition, | ||
_boundary_conditions, | ||
source_terms, solver, | ||
cache) | ||
cache, performance_counter) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
cache, performance_counter) | |
cache, | |
performance_counter) |
initial_condition, | ||
boundary_conditions, | ||
source_terms, solver, | ||
cache, performance_counter) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
cache, performance_counter) | |
cache, | |
performance_counter) |
@@ -752,6 +762,37 @@ function compute_coefficients!(u, func, t, mesh::AbstractMesh{3}, equations, dg: | |||
end | |||
end | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit
JuliaFormatter
[JuliaFormatter] reported by reviewdog 🐶
@inline function _prolong2mpimortars!(backend::Backend, cache, u, |
[JuliaFormatter] reported by reviewdog 🐶
end # @muladd |
[JuliaFormatter] reported by reviewdog 🐶
mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}}, |
[JuliaFormatter] reported by reviewdog 🐶
_calc_mpi_mortar_flux!(backend, surface_flux_values, mesh, nonconservative_terms, equations, |
[JuliaFormatter] reported by reviewdog 🐶
mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}}, |
[JuliaFormatter] reported by reviewdog 🐶
end # @muladd |
[JuliaFormatter] reported by reviewdog 🐶
mutable struct P4estMPICache{BufferType <: DenseVector, VecInt <: DenseVector{<:Integer}} |
[JuliaFormatter] reported by reviewdog 🐶
mpi_neighbor_interfaces = Adapt.adapt_structure(to, mpi_cache.mpi_neighbor_interfaces) |
[JuliaFormatter] reported by reviewdog 🐶
return P4estMPICache{BufferType, VecInt}(mpi_neighbor_ranks, mpi_neighbor_interfaces, |
[JuliaFormatter] reported by reviewdog 🐶
mesh::Union{StructuredMesh{3}, P4estMesh{3}, T8codeMesh{3}}, |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/test/test_t8code_3d.jl
Line 349 in fd5831e
0.07601887903440395, |
[JuliaFormatter] reported by reviewdog 🐶
Trixi.jl/test/test_t8code_3d.jl
Line 356 in fd5831e
4.848310144356219, |
_contravariant_vectors = Adapt.adapt_structure(to, elements._contravariant_vectors) | ||
_inverse_jacobian = Adapt.adapt_structure(to, elements._inverse_jacobian) | ||
_surface_flux_values = Adapt.adapt_structure(to, elements._surface_flux_values) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
neighbor_ids = unsafe_wrap_or_alloc(to, _neighbor_ids, size(interfaces.neighbor_ids)) | ||
node_indices = unsafe_wrap_or_alloc(to, _node_indices, size(interfaces.node_indices)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
neighbor_ids = unsafe_wrap_or_alloc(to, _neighbor_ids, size(interfaces.neighbor_ids)) | |
node_indices = unsafe_wrap_or_alloc(to, _node_indices, size(interfaces.node_indices)) | |
neighbor_ids = unsafe_wrap_or_alloc(to, _neighbor_ids, | |
size(interfaces.neighbor_ids)) | |
node_indices = unsafe_wrap_or_alloc(to, _node_indices, | |
size(interfaces.node_indices)) |
neighbor_ids = Adapt.adapt_structure(to, boundaries.neighbor_ids) | ||
node_indices = Adapt.adapt_structure(to, boundaries.node_indices) | ||
name = boundaries.name | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
neighbor_ids = unsafe_wrap_or_alloc(to, _neighbor_ids, size(mortars.neighbor_ids)) | ||
node_indices = unsafe_wrap_or_alloc(to, _node_indices, size(mortars.node_indices)) | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
@@ -94,28 +94,28 @@ end | |||
|
|||
if faces[side] == 0 | |||
# Index face in negative x-direction | |||
interfaces.node_indices[side, interface_id] = (:begin, surface_index1, | |||
interfaces.node_indices[side, interface_id] = (Indexing.first, surface_index1, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
interfaces.node_indices[side, interface_id] = (Indexing.first, surface_index1, | |
interfaces.node_indices[side, interface_id] = (Indexing.first, | |
surface_index1, |
surface_flux_values[v, l, m, 2, | ||
element] * | ||
boundary_interp_factor_2) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
surface_flux_values[v, l, m, 2, | |
element] * | |
boundary_interp_factor_2) | |
surface_flux_values[v, l, m, 2, | |
element] * | |
boundary_interp_factor_2) |
surface_flux_values[v, l, m, 3, element] * | ||
boundary_interp_factor_1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
surface_flux_values[v, l, m, 3, element] * | |
boundary_interp_factor_1) | |
surface_flux_values[v, l, m, 3, element] * | |
boundary_interp_factor_1) |
surface_flux_values[v, l, m, 4, | ||
element] * | ||
boundary_interp_factor_2) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
surface_flux_values[v, l, m, 4, | |
element] * | |
boundary_interp_factor_2) | |
surface_flux_values[v, l, m, 4, | |
element] * | |
boundary_interp_factor_2) |
surface_flux_values[v, l, m, 5, element] * | ||
boundary_interp_factor_1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
surface_flux_values[v, l, m, 5, element] * | |
boundary_interp_factor_1) | |
surface_flux_values[v, l, m, 5, element] * | |
boundary_interp_factor_1) |
surface_flux_values[v, l, m, 6, | ||
element] * | ||
boundary_interp_factor_2) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
surface_flux_values[v, l, m, 6, | |
element] * | |
boundary_interp_factor_2) | |
surface_flux_values[v, l, m, 6, | |
element] * | |
boundary_interp_factor_2) |
_calc_surface_integral!(backend, du, u, mesh, equations, surface_integral, dg, cache) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
_calc_surface_integral!(backend, du, u, mesh, equations, surface_integral, dg, cache) | |
_calc_surface_integral!(backend, du, u, mesh, equations, surface_integral, dg, | |
cache) |
end | ||
end | ||
|
||
@inline function _prolong2mpimortars!(backend::Backend, cache, u, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
@inline function _prolong2mpimortars!(backend::Backend, cache, u, | |
@inline function _prolong2mpimortars!(backend::Backend, cache, u, |
surface_integral, dg::DG, cache) | ||
return nothing | ||
end | ||
end # @muladd |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
end # @muladd | |
end # @muladd |
end | ||
|
||
@inline function _prolong2mpimortars!(backend::Nothing, cache, u, | ||
mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
mesh::Union{ParallelP4estMesh{3}, ParallelT8codeMesh{3}}, | |
mesh::Union{ParallelP4estMesh{3}, | |
ParallelT8codeMesh{3}}, |
@@ -416,6 +448,16 @@ | |||
nonconservative_terms, equations, | |||
mortar_l2::LobattoLegendreMortarL2, | |||
surface_integral, dg::DG, cache) | |||
backend = backend_or_nothing(cache.mpi_mortars) | |||
_calc_mpi_mortar_flux!(backend, surface_flux_values, mesh, nonconservative_terms, equations, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
_calc_mpi_mortar_flux!(backend, surface_flux_values, mesh, nonconservative_terms, equations, | |
_calc_mpi_mortar_flux!(backend, surface_flux_values, mesh, nonconservative_terms, | |
equations, |
|
||
function Adapt.adapt_structure(to, mpi_cache::P4estMPICache) | ||
mpi_neighbor_ranks = mpi_cache.mpi_neighbor_ranks | ||
mpi_neighbor_interfaces = Adapt.adapt_structure(to, mpi_cache.mpi_neighbor_interfaces) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
mpi_neighbor_interfaces = Adapt.adapt_structure(to, mpi_cache.mpi_neighbor_interfaces) | |
mpi_neighbor_interfaces = Adapt.adapt_structure(to, | |
mpi_cache.mpi_neighbor_interfaces) |
@assert eltype(mpi_send_buffers) == eltype(mpi_recv_buffers) | ||
BufferType = eltype(mpi_send_buffers) | ||
VecInt = eltype(mpi_neighbor_interfaces) | ||
return P4estMPICache{BufferType, VecInt}(mpi_neighbor_ranks, mpi_neighbor_interfaces, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
return P4estMPICache{BufferType, VecInt}(mpi_neighbor_ranks, mpi_neighbor_interfaces, | |
return P4estMPICache{BufferType, VecInt}(mpi_neighbor_ranks, | |
mpi_neighbor_interfaces, |
end | ||
|
||
@inline function _apply_jacobian!(::Nothing, du, | ||
mesh::Union{StructuredMesh{3}, P4estMesh{3}, T8codeMesh{3}}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
mesh::Union{StructuredMesh{3}, P4estMesh{3}, T8codeMesh{3}}, | |
mesh::Union{StructuredMesh{3}, P4estMesh{3}, | |
T8codeMesh{3}}, |
0.00021710076010951073, | ||
0.0004386796338203878, | ||
0.00020836270267103122, | ||
0.07601887903440395, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
0.07601887903440395, | |
0.07601887903440395 |
0.02980358831035801, | ||
0.048476331898047564, | ||
0.02200137344113612, | ||
4.848310144356219, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
4.848310144356219, | |
4.848310144356219 |
|
||
# start simulation with tiny final time to trigger precompilation | ||
duration_precompile = @elapsed trixi_include(elixir_path, | ||
tspan=(0.0, 1e-14)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
tspan=(0.0, 1e-14)) | |
tspan = (0.0, 1e-14)) |
end | ||
|
||
# start the real simulation | ||
duration_elixir = @elapsed trixi_include(elixir_path, maxiters=maxiters) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
duration_elixir = @elapsed trixi_include(elixir_path, maxiters=maxiters) | |
duration_elixir = @elapsed trixi_include(elixir_path, maxiters = maxiters) |
# reduce metrics per rank | ||
open("metrics.out", "w") do io | ||
for (key, _) in gathered_metrics[1] | ||
println(io, key, ": ", mapreduce(x->x[key], min, gathered_metrics)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[JuliaFormatter] reported by reviewdog 🐶
println(io, key, ": ", mapreduce(x->x[key], min, gathered_metrics)) | |
println(io, key, ": ", mapreduce(x -> x[key], min, gathered_metrics)) |
Very rough summary:
Features
P4estMesh
, except AMR andnonconservative_terms::True
Important changes
Adapt.jl
,adapt_to
for several typesAbstractHeterogeneousContainer
:Trixi.jl/src/auxiliary/containers.jl
Lines 318 to 329 in 33903d9
VecOfArray
Trixi.jl/src/auxiliary/vector_of_arrays.jl
Lines 8 to 13 in 33903d9
Trixi.jl/src/solvers/dgsem_p4est/dg_parallel.jl
Lines 8 to 19 in 33903d9
Trixi.Indexing
module instead of symbolsTrixi.jl/src/solvers/dg.jl
Lines 755 to 778 in 33903d9
Todos
Trixi.jl/src/solvers/dgsem_p4est/containers_parallel.jl
Line 230 in 33903d9
Trixi.jl/src/solvers/dgsem_p4est/dg_parallel.jl
Line 319 in 33903d9