Complete initial implementation for treating internal packages as external found with find_package() (#63)

[bartlettroscoe]

Description

This PR completes the basic implementation for treating internal packages as pre-installed external packages. This completes:

• Use Case 3: Configure/build pointing to a subset of already installed TriBITS packages in same repo

in #63.

Most of the changes are related to dependency-related logic for figuring out what packages needs to be treated as external based on what packages are flagged as external packages with TPL_ENABLE_<Package>=ON. (There are some tricky use cases related to subpackages that had to be addressed.)

This includes some examples to show that it works.

Instructions for reviewers

• As this PR renames some files and moves code around, it likely better to review this PR one commit at at time
• Please read the commit messages and make sure they makes sense
• Please focus most on the updated documentation in the *.cmake source files and in the *.rst files. In particular, please review the GitHub rendered sections:
  • Building against pre-installed packages
  • TriBITS-Compliant Packages
    • TriBITS-Compliant Internal Packages
    • TriBITS-Compliant External Packages
  • Software Engineering Packaging Principles
  • <tplDefsDir>/FindTPL<tplName>.cmake
  • ${TPL_NAME}_FINDMOD
  • How to add a new TriBITS external package/TPL
  • Requirements for FindTPL.cmake modules
• Please look over the new tests and try to understand what they are showing

Tasks

• Stop the install of TribitsExProjConfig.cmake in the upstream package installs
• Provide more examples on pre-installing packages like MixedLang
• Get the WrapExternal package to work (likely by reading the lib name from the target using generator expressions)
• Figure out how should we treat TPLs that are enabled in the pre-installed packages in the downstream TriBITS project? (We should not be finding them again but should just use them as found by the upstream TriBITS-compatible package installed.)
• Update all documentation
• Test against Kokkos, KokkosKernels, Trilinos incremental build/install driving case (see Tweaks to Trilinos to get working with pre-installed Kokkos and KokkosKernels (with TriBITS updates) (#11545) trilinos/Trilinos#11546).

[bartlettroscoe]

CC: @jwillenbring, @KyleFromKitware

Following up on the task above:

• Figure out how should we treat TPLs that are enabled in the pre-installed packages in the downstream TriBITS project? (We should not be finding them again but should just use them as found by the upstream TriBITS-compatible package installed.)

This is more complex than it would first seem. For example, with TribitsExampleProject, when we enable SimpleTpl and configure, build and install SimpleCxx, we should not be trying to find HeaderOnlyTpl and SimpleTpl again when we configure the rest of TribitsExampleProject when we pull in SimpleCxx as a pre-installed external package/TPL, we should not be trying to find HeaderOnlyTpl or SimpleTpl again (i.e. like we currently are as shown here in TEST_7 Configure rest of TribitsExampleProject against pre-installed SimpleCxx). Likewise, when we build and install Kokkos and then build the rest of Trilinos against Kokkos, we should not be trying to find the TPLs that Kokkos depends on either. The TriBITS TPLs being used by Kokkos are Pthread, CUDA, HWLOC, CUSPARSE and DLib as shown by:

$ cd Trilinos/

$ find packages/kokkos -name Dependencies.cmake -exec echo \; -exec echo {} \; -exec echo \; -exec grep _TPLS {} \; 

packages/kokkos/algorithms/cmake/Dependencies.cmake

  LIB_OPTIONAL_TPLS Pthread CUDA HWLOC
  TEST_OPTIONAL_TPLS CUSPARSE

packages/kokkos/cmake/Dependencies.cmake


packages/kokkos/containers/cmake/Dependencies.cmake

  LIB_OPTIONAL_TPLS Pthread CUDA HWLOC
  TEST_OPTIONAL_TPLS CUSPARSE

packages/kokkos/core/cmake/Dependencies.cmake

  LIB_OPTIONAL_TPLS Pthread CUDA HWLOC DLlib
  TEST_OPTIONAL_TPLS CUSPARSE

packages/kokkos/simd/cmake/Dependencies.cmake

  LIB_OPTIONAL_TPLS Pthread CUDA HWLOC
  TEST_OPTIONAL_TPLS CUSPARSE

We should not be trying finding these TPL again when we pull in pre-installed Kokkos when building downstream packages but should instead use what was found in the the upstream Kokkos package configure.

Likewise, what about a case with Trilinos where you might be installing Tpetra and all of its upstream internal package (e.g. Epetra, Teuchos, KokkosKernels, Kokkos) and external packages (e.g. CUDA, CUBLAS, CUSOLVER, BLAS, LAPACK, Pthread, etc.). So in that case, none of the these upstream external packages/TPLs should be directly processed but should instead just process the external package/TPL Tpetra in a downstsream TriBITS CMake project build.

But then what about situations where there are external packages/TPLs with dependencies like TriBITS TPLs CUBLAS and CUSPARSE depend on the TriBITS TPL CUDA? For example, Kokkos depends on the TPLs CUDA and CUSPARSE but not CUBLAS. When we build and install Kokkkos with CUDA and CUSPARSE and then try to to configure Tpetra (which needs CUBLAS) in a downstream CMake project against that external Kokkos. So when the FindTPLCUBLAS.cmake file is processed in the downstream CMake project that builds Tpetra, the CUDA::all_libs target must have already been defined. But that target will not be defined until find_package(Kokkos) is called later in the external package/TPL loop.

So that means that we can't loop over the external packages/TPLs in one loop. Instead, we need to do two loops:

1. a first loop is over the external packages/TPLs that involve full TriBITS compliant <Package>Config.cmake files
2. a second loop for the remaining standard TriBITS TPLs, like CUBLAS

So in the first loop, we only want to call find_package() for the most downstream TriBITS-complient external packages/tpls (i.e. that have no downstream TriBITS-complient external packages/tpls). Calling find_package() on just those terminal TriBITS-complient external packages/tpls will result in find_dependency() being called on all of the upstream TriBITS external packages. So by the time we are done calling find_package(), we should have all of the <Package>::all_libs targets defined in that subgraph of external packages/TPLs. So, in our Trilinos/Kokkos example above, the targets CUDA::all_libs and CUSPARSE::all_libs will be defined when find_package(Kokkos) is called.

Then on the second loop, any upstream packages processed in the first loop will will already have their <Package>::all_libs targets defined so we can generate their <Package>Config.cmake files dependencies on these upstream packages, we can call target_link_libraries() successfully. As for putting in links to upstream TPLs, we will need to use the set(<UpstreamTpl>_DIR ...) then find_dependency(<UpstreamTpl>) approach since we can't assume where the<UpstreamTpl>Config.cmake file actually is contained. This is where things start getting tricky.

[bartlettroscoe]

NOTE:

[bartlettroscoe]

@KyleFromKitware and @jwillenbring, see the updated section Instructions for reviewers for the links to the rendered documentation added and updated related to this PR.

[bartlettroscoe]

@KyleFromKitware, as we just discussed, please finish and post your review of this PR or post whatever review you have complete (and note what commit you got to) before Monday, March 27, 2023. If you can't complete your review before then, we will address whatever review comments you have, merge the PR, and set up the PR for a post-merge review of the final commits.

[rabartlett1972]

With all of the review comments in #560 (review) addressed (or deferred to new Issues), this PR is ready to merge.

Comments were addressed or deferred to new Issues

[rabartlett1972]

Reapproving after merge and then forced push.