Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implicit Field Solve Preconditioner based on Curl-Curl Operator #5286

Open
wants to merge 12 commits into
base: development
Choose a base branch
from

Conversation

debog
Copy link
Contributor

@debog debog commented Sep 18, 2024

Implemented a preconditioner for the implicit E-field solve using the AMReX curl-curl operator and the MLMG solver.

  • Introduced a Preconditioner base class that defines the action of a preconditioner for the JFNK algorithm.
  • Implemented the CurlCurlMLMGPC that uses the multigrid solution for the curl-curl operator (implemented in AMReX) to precondition the E-field JFNK solve.

Other changes needed for this:

  • Partially implemented a mapping between WarpX field boundary types and AMReX's linear operator boundary types.
  • Added some functionalities to ImplicitSolver class that allows preconditioners to access WarpX info (like Geometry, boundaries, etc).

Some premilinary wall times:

Test: inputs_vandb_2d
  Grid: 160 X 160
  dt: 0.125/wpe
  Time iterations: 20
  NPPC = 32

Solver parameters:
  newton.max_iterations = 10
  newton.relative_tolerance = 1.0e-12
  newton.absolute_tolerance = 0.0
  gmres.max_iterations = 1000
  gmres.relative_tolerance = 1.0e-8
  gmres.absolute_tolerance = 0.0

Avg GMRES iterations: ~3 (wPC), ~27 (noPC)

Lassen (MPI + CUDA)
-------------------
  Box  GPU   Walltime (s)
             wPC       noPC
   1    1    2324.7    15004.1
   4    1    2306.8    14356.8
   4    4     758.9     3647.3

Dane (MPI + OMP)
----------------
  Box  CPU  Threads   Walltime (s)
                      wPC      noPC
   1    1      1      6709.3   43200.0*
   1    1      2      3279.1   22296.1
   1    1      4      1696.3   11613.2
   1    1      8      1085.0    6911.4
   1    1     16       724.3    4729.0
   4    1      1      5525.9   33288.8
  16    1      1      4419.4   28467.8
   4    4      1      1324.4    9121.1
  16   16      1       524.9    3658.8

* 43200.0 seconds is 12 hours (max job duration on Dane);
the simulation was almost done (started the 20th step).

@debog debog added enhancement New feature or request component: implicit solvers Anything related to implicit solvers labels Sep 18, 2024
@debog debog marked this pull request as ready for review September 19, 2024 19:05
…X is not yet implemented in 1D, added precompiler guardrails to not use it in 1D
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: implicit solvers Anything related to implicit solvers enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants