01_introduction.tex

\chapter{Introduction}

The software package SPECFEM3D\_GLOBE simulates three-dimensional global and regional seismic wave propagation
and performs full waveform imaging (FWI) or adjoint tomography based upon the spectral-element method (SEM).
The SEM is a continuous Galerkin technique \citep{TrKoLi08,PeKoLuMaLeCaLeMaLiBlNiBaTr11},
which can easily be made discontinuous \citep{BeMaPa94,Ch00,KoWoHu02,ChCaVi03,LaWaBe05,Kop06,WiStBuGh10,AcKo11};
it is then close to a particular case of the discontinuous Galerkin technique \citep{ReHi73,LeRa74,Arn82,JoPi86,BoMaHe91,FaRi99,HuHuRa99,CoKaSh00,GiHeWa02,RiWh03,MoRi05,GrScSc06,AiMoMu06,BeLaPi06,DuKa06,DeSeWh08,PuAmKa09,WiStBuGh10,DeSe10,EtChViGl10}, with optimized efficiency because of its tensorized basis functions \citep{WiStBuGh10,AcKo11}.
In particular, it can accurately handle very distorted mesh elements \citep{OlSe11}.\newline


It has very good accuracy and convergence properties \citep{MaPa89,SePr94,DeFiMu02,Coh02,DeSe07,SeOl08,AiWa09,AiWa10,MeStTh12}.
The spectral element approach admits spectral rates of convergence and allows exploiting $hp$-convergence schemes.
It is also very well suited to parallel implementation on very large supercomputers \citep{KoTsChTr03,TsKoChTr03,KoLaMi08a,CaKoLaTiMiLeSnTr08,KoViCh10} as well as on clusters of GPU accelerating graphics cards \citep{Kom11,MiKo10,KoMiEr09,KoErGoMi10}. Tensor products inside each element can be optimized to reach very high efficiency \citep{DeFiMu02}, and mesh point and element numbering can be optimized to reduce processor cache misses and improve cache reuse \citep{KoLaMi08a}. The SEM can also handle triangular (in 2D) or tetrahedral (in 3D) elements \citep{WinBoyd96,TaWi00,KoMaTrTaWi01,Coh02,MeViSa06} as well as mixed meshes, although with increased cost and reduced accuracy in these elements, as in the discontinuous Galerkin method.\newline


Note that in many geological models in the context of seismic wave propagation studies
(except for instance for fault dynamic rupture studies, in which very high frequencies or supershear rupture need to be modeled near the fault, see e.g. \cite{BeGlCrViPi07,BeGlCrVi09,PuAmKa09,TaCrEtViBeSa10})
a continuous formulation is sufficient because material property contrasts are not drastic and thus
conforming mesh doubling bricks can efficiently handle mesh size variations \citep{KoTr02a,KoLiTrSuStSh04,LeChLiKoHuTr08,LeChKoHuTr09,LeKoHuTr09}.
This is particularly true at the scale of the full Earth.\newline


For a detailed introduction to the SEM as applied to
global and regional seismic wave propagation, please consult \citet{TrKoLi08,PeKoLuMaLeCaLeMaLiBlNiBaTr11,KoVi98,KoTr99,Ch00,KoTr02a,KoTr02b,KoRiTr02,ChCaVi03,CaChViMo03,ChVa04,ChKoViCaVaFe07}.
A detailed theoretical analysis of the dispersion
and stability properties of the SEM is available in \citet{Coh02}, \citet{DeSe07}, \citet{SeOl07}, \citet{SeOl08} and \citet{MeStTh12}.\newline


Effects due to lateral variations in compressional-wave
speed, shear-wave speed, density, a 3D crustal model, ellipticity,
topography and bathymetry, the oceans, rotation, and self-gravitation are included.
The package can accommodate full 21-parameter anisotropy \citep{ChTr07}
as well as lateral variations in attenuation \citep{SaKoTr10}. Adjoint
capabilities and finite-frequency kernel simulations are also included
\citep{TrKoLi08,PeKoLuMaLeCaLeMaLiBlNiBaTr11,LiTr06,LiTr08,FiIgBuKe09,ViOp09}.\newline


The SEM was originally developed in computational fluid dynamics \citep{Pat84,MaPa89}
and has been successfully adapted to address problems in seismic wave propagation.
Early seismic wave propagation applications of the SEM, utilizing Legendre basis functions and a
perfectly diagonal mass matrix, include \cite{CoJoTo93}, \cite{Kom97},
\cite{FaMaPaQu97}, \cite{CaGa97}, \cite{KoVi98} and \cite{KoTr99},
whereas applications involving Chebyshev basis functions and a nondiagonal mass matrix
include \cite{SePr94}, \cite{PrCaSe94} and \cite{SePrPr95}.
In the Legendre version that we use in SPECFEM the mass matrix is purposely slightly inexact but diagonal (but can be made exact if needed, see \cite{Teu15}),
while in the Chebyshev version it is exact but non diagonal.\newline


Beware that, in a spectral-element method, some spurious modes (that have some similarities with classical so-called "Hourglass modes" in finite-element techniques,
although in the SEM they are not zero-energy modes) can appear in some (but not all) cases in the spectral element in which the source is located.
Fortunately, they do not propagate away from the source element.
However, this means that if you put a receiver in the same spectral element as a source, the recorded signals may in some cases be wrong, typically exhibiting some spurious
oscillations, which are often even non causal.
If that is the case, an easy option is to slightly change the mesh in the source region in order to get rid of these Hourglass-like spurious modes,
as explained in \cite{DuLiScGa14}, in which this phenomenon is described in details, and in which practical solutions to avoid it are suggested.\newline


SPECFEM3D\_GLOBE can now perform gravity field calculations in addition (or instead of) seismic wave propagation only.
See flag \texttt{GRAVITY\_INTEGRALS} in file \texttt{setup/constants.h.in}. Please also refer to \citet{Martin2017}.
And yes, that is the reason why there is a gravity observation satellite on the cover of the manual :-)\newline


All SPECFEM3D\_GLOBE software is written in Fortran2003 with full portability
in mind, and conforms strictly to the Fortran2003 standard. It uses
no obsolete or obsolescent features of Fortran. The package uses
parallel programming based upon the Message Passing Interface (MPI)
\citep{GrLuSk94,Pac97}.\newline


SPECFEM3D\_GLOBE won the Gordon Bell award for best performance at the SuperComputing~2003
conference in Phoenix, Arizona (USA) (see \cite{KoTsChTr03}).
It was a finalist again in 2008 for a run at 0.16 petaflops (sustained) on 149,784 processors of the `Jaguar' Cray XT5 system at Oak Ridge National Laboratories (USA) \citep{CaKoLaTiMiLeSnTr08}.
It also won the BULL Joseph Fourier supercomputing award in 2010.\newline


It reached the sustained one petaflop performance level for the first time in February 2013
on the Blue Waters Cray supercomputer at the National Center for Supercomputing Applications (NCSA), located at the University of Illinois at Urbana-Champaign (USA).\newline


The package includes support for GPU graphics card acceleration \citep{Kom11,MiKo10,KoMiEr09,KoErGoMi10} and also supports OpenCL.\newline

%% future plans:
%% better see https://github.com/SPECFEM/specfem3d_globe/wiki/Development-plan

%The next release of the code will include support for
%Convolutional or Auxiliary Differential Equation Perfectly Matched absorbing Layers (C-PML or ADE-PML)
%\citep{MaKoEz08,MaKoGe08,MaKo09,MaKoGeBr10,KoMa07} for the case of single-chunk simulations in regional models.

\section{Citation}

You can find all the references below in \BibTeX format in file \texttt{doc/USER\_MANUAL/bibliography.bib}.\newline


If you use SPECFEM3D\_GLOBE for your own research, please cite at least one
of the following articles: \cite{KoXiBoPeSaLiTr16,TrKoLi08,PeKoLuMaLeCaLeMaLiBlNiBaTr11,VaCaSaKoVi99,LeChLiKoHuTr08,LeChKoHuTr09,LeKoHuTr09,KoMiEr09,KoErGoMi10,WiKoScTr04,KoLiTrSuStSh04,ChKoViCaVaFe07,MaKoDi09,KoViCh10,CaKoLaTiMiLeSnTr08,TrKoHjLiZhPeBoMcFrTrHu10,KoRiTr02,KoTr02a,KoTr02b,KoTr99} or \cite{KoVi98}.\newline


If you use the C-PML absorbing layer capabilities of the code, please cite at least one article
written by the developers of the package, for instance:
%
\begin{itemize}
\item \cite{XiKoMaMa14},
\item \cite{XiMaCrKoMa16}.
\end{itemize}
%
If you use the UNDO\_ATTENUATION option of the code in order to produce full anelastic/viscoelastic sensitivity kernels, please cite at least one article
written by the developers of the package, for instance (and in particular):
%
\begin{itemize}
\item \cite{KoXiBoPeSaLiTr16}.
\end{itemize}
%
More generally, if you use the attenuation (anelastic/viscoelastic) capabilities of the code, please cite at least one article
written by the developers of the package, for instance:
%
\begin{itemize}
\item \cite{KoXiBoPeSaLiTr16},
\item \cite{BlKoChLoXi16}.
\end{itemize}
%
If you use the kernel capabilities of the code, please cite at least one article
written by the developers of the package, for instance:
%
\begin{itemize}
\item \cite{TrKoLi08},
\item \cite{PeKoLuMaLeCaLeMaLiBlNiBaTr11},
\item \cite{LiTr06},
\item \cite{MoLuTr09}.
\end{itemize}


If you use this new version, which has non blocking MPI for much better performance for medium or large runs,
please cite at least one of these five articles,
in which results of 3D non blocking MPI runs are presented: \cite{KoErGoMi10,KoViCh10,Kom11,PeKoLuMaLeCaLeMaLiBlNiBaTr11,CaKoLaTiMiLeSnTr08}.\newline


If you use GPU graphics card acceleration please cite e.g. \cite{Kom11}, \cite{MiKo10}, \cite{KoMiEr09}, and/or \cite{KoErGoMi10}.\newline


If you work on geophysical applications, you may be interested in citing some of these application articles as well, among others:
\cite{WiKoScTr04,JiTsKoTr05,KrJiKoTr06a,KrJiKoTr06b,LeChLiKoHuTr08,LeChKoHuTr09,LeKoHuTr09,ChFaKo04,FaChKo04,RiRiKoTrHe02,GoAmTaCaSmSaMaKo09,TrKo00,SaKoTr10}.
If you use 3D mantle model S20RTS, please cite \citet{RiVaWo99}.\newline


Domain decomposition is explained in detail in \cite{MaKoBlLe08}, and excellent scaling up to 150,000 processor cores in shown for instance in
\cite{CaKoLaTiMiLeSnTr08,KoLaMi08a,MaKoBlLe08,KoErGoMi10,Kom11}.\newline


The corresponding Bib\TeX{} entries may be found
in file \texttt{doc/USER\_MANUAL/bibliography.bib}.

\section{Support}

This material is based upon work supported by the U.S. National Science
Foundation under Grants No. EAR-0406751 and EAR-0711177, by the French
CNRS, French INRIA Sud-Ouest MAGIQUE-3D, French ANR NUMASIS under
Grant No. ANR-05-CIGC-002, and European FP6 Marie Curie International
Reintegration Grant No. MIRG-CT-2005-017461.
Any opinions, findings, and conclusions or recommendations expressed in this material are
those of the authors and do not necessarily reflect the views of the
U.S. National Science Foundation, CNRS, INRIA, ANR or the European
Marie Curie program.