Recommended executioner and preconditioner settings for conjugate heat transfer

[mattfalcone1997]

Hi,

I am running simulations of doubly periodic channel flow with solid walls, using nekRS for the fluid and MOOSE for the solid. I am running on a cluster with 128 cores per node and 4 GPUs per node, with (hopefully) the CPUs for the solid and GPUs for nekRS. However, I have noticed poor performance for the solid, with it taking ~100 times longer than the fluid (0.1s vs 10s). The solid mesh is 4.75 million elements and the fluid mesh has 22 million GLL points. I also find that sub-cycling even with relatively low ratios of time steps leads to divergence.

Firstly, I ought to check that the command I am running is correct
mpirun -np 4 cardinal-opt -i solid.i --nekrs-backend CUDA --n-threads=32 --distributed-mesh

I am wondering if there are some recommended settings for the transient heat conduction solve to help reduce the times between fluid and solid solves. For example:

• Should we use the AD kernels and the Newton scheme
• What PETSc options are recommended.

Thanks in advance!

[aprilnovak]

Hi @mattfalcone1997, is it possible to share your input file you are using for solid.i? I'd like to see what settings you have in the Executioner block.

Another user has also reported issues with rapid divergence unless using lock-step time advancement, so I have it on my to-do list to add some additional heat flux/temperature data transfer options which might help stabilize things. But in either case, 100x longer for the heat conduction sounds very long and should be something we can get faster.

[mattfalcone1997]

Hi @aprilnovak, thanks for responding! I have slightly reduced the solid mesh resolution and rerun the case but ratio is still around 50 with this input script (Total run-time around 20 minutes and elapsed fluid time 24 seconds). My solid.i is shown below.

G = 1.
Re = 150
Pr = 1

lx = 25.6
lz = 8.52
nx = 320
ny = 20
nz = 110
ly = ${fparse 1 + 20/150}
[Mesh]
  [top_wall]
    type = GeneratedMeshGenerator
    dim=3
    nx=${nx}
    ny=${ny}
    nz=${nz}
    xmax=${lx}
    zmax=${lz}
    ymax=${ly}
    ymin=1
  []
  [top_wall_new_ids]
    type=RenameBoundaryGenerator
    input = top_wall
    old_boundary = '0 1 2 3 4 5'
    new_boundary = '6 7 8 9 10 11'
  []
  [top_wall_new_name]
    type=RenameBoundaryGenerator
    input = top_wall_new_ids
    old_boundary = 'left right bottom top front back'
    new_boundary = 'left_top right_top bottom_top top_top front_top back_top'
  []
  [bottom_wall]
    type = GeneratedMeshGenerator
    dim=3
    nx=${nx}
    ny=${ny}
    nz=${nz}
    xmax=${lx}
    zmax=${lz}
    ymax=-1
    ymin=-${ly}
  []
  [bottom_wall_new_name]
    type=RenameBoundaryGenerator
    input = bottom_wall
    old_boundary = 'left right bottom top front back'
    new_boundary = 'left_bottom right_bottom bottom_bottom top_bottom front_bottom back_bottom'
  []

  [walls]
    type = CombinerGenerator
    inputs = 'top_wall_new_name bottom_wall_new_name'
  []
[]

[Variables]
  [temp]
  []
[]

[Kernels]
  [hc]
    type = ADHeatConduction
    variable = temp
  []
  [transient]
    type = ADHeatConductionTimeDerivative
    variable=temp
  []
[]

[ICs]
  [T_ic]
    type=FunctionIC
    variable = temp
    function='-${Re}*${Pr}*(abs(y)-1)'
  []
[]

[BCs]
  [neumann_top]
    type = ADNeumannBC
    variable = temp
    boundary = 'top_top'
    value = -1
  []
  [neumann_bottom]
    type = ADNeumannBC
    variable = temp
    boundary = 'bottom_bottom'
    value = -1
  []
  [match_nek]
    type = ADMatchedValueBC
    variable = temp
    boundary = 'top_bottom bottom_top'
    v=nek_temp
  []
  [./Periodic]
    [./top_x]
      variable = temp
      primary=4
      secondary=2
      translation='${lx} 0 0'
    [../]
    [./bottom_x]
      variable = temp
      primary=10
      secondary=8
      translation='${lx} 0 0'
    [../]
    [./top_z]
      variable = temp
      primary=0
      secondary=5
      translation='0 0 ${lz}'
    [../]
    [./bottom_z]
      variable = temp
      primary=6
      secondary=11
      translation='0 0 ${lz}'
    [../]
  [../]
[]

[Materials]
  [hc]
    type = ADGenericConstantMaterial
    prop_values = '${fparse 1/(G*Re*Pr)} 1 1'
    prop_names = 'thermal_conductivity density specific_heat'
  []
[]

[Executioner]
  type = Transient
  
  solve_type=NEWTON
  petsc_options_iname = '-pc_type -pc_hypre_type'
  petsc_options_value = 'hypre boomeramg'
  reuse_preconditioner=true
  dt = 0.0025
  l_tol = 1e-8

[]

[Outputs]
  csv = true
  hide = 'flux_integral'
  [out]
  type = Exodus
  min_simulation_time_interval=5
  []
[]

[MultiApps]
  [nek]
    type = TransientMultiApp
    input_files = 'nek.i'
    sub_cycling = false
  []
[]

[Transfers]
  [nek_temp]
    type = MultiAppGeneralFieldShapeEvaluationTransfer
    source_variable = temp
    from_multi_app = nek
    variable = nek_temp
  []
  [flux]
    type = MultiAppGeneralFieldShapeEvaluationTransfer
    source_variable = flux
    to_multi_app = nek
    variable = avg_flux
  []
  [flux_integral_to_nek]
    type = MultiAppPostprocessorTransfer
    to_postprocessor = flux_integral
    from_postprocessor = flux_integral
    to_multi_app = nek
  []
[]

[AuxVariables]
  [nek_temp]
  []
  [flux]
    family = MONOMIAL
    order = CONSTANT
  []
[]

[AuxKernels]
  [flux]
    type = DiffusionFluxAux
    diffusion_variable = temp
    component = normal
    diffusivity = thermal_conductivity
    variable = flux
    boundary = '3 7'
  []
[]

[Postprocessors]
  [wall_temp]
    type = SideAverageValue
    variable = temp
    boundary = '3 7'
  []
  [flux_integral]
    type = ADSideDiffusiveFluxIntegral
    diffusivity = thermal_conductivity
    variable = temp
    boundary = '3 7'
  []
[]

[aprilnovak]

Hi @mattfalcone1997, would you please try adding this to each of the sub-blocks in your [Transfers] block?

search_value_conflicts = false

If this speeds up your simulation a lot, then I'll explain what's going on. For now, it's a hunch.

[mattfalcone1997]

Hi @aprilnovak

This didn't seem to speed the solution up that much. I've managed to use some limited subcycling which has helped (1 solid step for every 2-3 fluid steps). I will have a look into profiling the solver to see where it is spending most of the execution time.