Skip to content

Latest commit

 

History

History
2241 lines (1682 loc) · 101 KB

README.adoc

File metadata and controls

2241 lines (1682 loc) · 101 KB

Modern C++ Component

Introduction

Context-lite

C++ is an ISO standard based on the Abstract Machine, it keeps far from the practical and gory details addressing how to build it, even less how to distribute it.

Building and distribution are the realm of 3rd party actors and tools -distinct from the committee- (some people from this realm are sitting on the committee, but this fact does not change the ISO standard scope). As a result, there is little to no standard-prescribed way to go about steps once the code is written.

Motivations

The aim of this document is to gather in one place the steps allowing to create native software with the following properties:

  • Cross-platform and reasonably cross-toolset

  • Usable and re-usable (consumer perspective)

    • By other developers in the same or unrelated organisations

    • By automated processes (e.g. CI)

  • Easy to share (provider/maintainer perspective)

  • Modular

  • Uniform: Both the process and the resulting package have to work great for naively small projects as well as massive code bases.

In the current era of Modern C++, the language itself leads a potential change in the mindset of C++ developers. Those changes are pervading the C++ ecosystem and tooling. This document does not explore the language itself, but how to actually achieve and take advantage of the properties in the above list, by relying on modern tools.

Equally important is hopefully being able to convey the appropriate mindset, so those properties are both understood and desired.

There is no standard, no undisputed approach, and no silver-bullet. But there are solutions with varying levels of traction and growth, and active projects refining themselves. By providing a write-up of an end-to-end (holistic) approach, another goal is to try and attract collaborators with different specialisations, eventually leveraging expert-knowledge regarding specific cogs in the system.
In addition to support enhancing each step, this document also serves as a repository of the different friction-points in the interactions of parts, recording potential future enhancements.

Why adhering to a pre-defined process? Theoretically, the properties above could be reached via innumerable different ways.
In practical situations, each process implements a different trade-off. Reaching a satisfying yet adaptable infrastructure requires a lot of research, prototyping, and validation. The later the build infrastructure of a codebase is addressed, the costlier its deployment tends to be.

Adopting a common and predefined plan greatly reduce the upfrong cost, with other advantages:

  • Having good defaults saves on initial setup and discussions (i.e. avoid bike-shedding). The value added part is the software, not its building and packaging logic.

  • Having a common base gives better chance at interoperability of discrete software packages.

  • Sharing tools and practices gives more chance to find native (and better tested) support with service providers (CI/CD, package hosting, static analysis, etc.).

  • Someone already documented the process for you, with changes appearing in separate versions. This allows to point to an unambigous reference, and eases onboarding new developpers, without your organization having to produce and maintain an internal document.

C++ special case

Many high level languages are able to provide sociable components quite naturally, without requiring to first agree on some external process. Or they have one (a few) de-facto standard.

In addition to the fact that the C++ standard does not make any prescription, C++ (as other native languages) is a special case. It is not trivial to distribute a project that builds easily on all environments, while it is close to impossible to distribute pre-built objects for all environments due to the combinatorial explosion of build parameters:

  • Project versions

  • Static / Shared libraries

  • Compilers, and compilers' versions

  • Standard library

  • Language version

  • The compilation flags galore, which are also compiler dependent

    • Debug, Release, MinSize, and a few other build types

    • Optimisation level

    • …​

  • Calling conventions

  • The upstream dependency-diamond (two separate components might rely on the same upstream library)

  • Code instrumentation

  • …​

When developing sociable code, this is a strong motivation to ensure the component can easily build on consumers' site. Ideally supporting as much as the usual parameters as feasible.

Additionally, once the build process is made reliable, repeatable and consistent, releasing feels safer and tends to happen more often.
In parallel, growing confidence in the ability for downstreams to conduct correct and identifiable builds means less pressure to host and distribute a growing number of combinations with each iteration.

Philosophy

Producing sociable components might require some change in the usual approach: it becomes the responsibility of the component itself to take extra measures and precautions in order to be easily buildable, usable and distributable.

This might contrast with a traditional approach where the burden is on downstream(s) to adapt to the libraries it uses, which might encourage organisations to adhere to an isolation mental model, i.e. to stay away from external dependencies as much as possible.

The best process in the world?

... actually does not sound anything like this doc.

— Tenacious C

Yet the Platonic ideal sets the cap.

This mythical best system would allow every user to create sociable code by default, without wasted effort (Of course, there is intrinsic effort required).
It would allow every user to re-use sociable code through well-defined and concise steps, without imposing any limitation on usage contexts.

There are software quality principles and metrics to evaluate where a process stands. Let’s establish them as goals for the process:

  • Practicality

  • Simplicity and straightforwardness

  • Open-Closed

  • DRY

  • Separation of concerns

The actual system will not be able to strictly enforce all goals at once, but they remain excellent parameters to consider along the way, in order to make informed engineering decisions and trade-offs.

— Mathieu Ropert
The state of package management in C++ (ACCU 2019)

Audience

TODO

The process

This section describe an end-to-end approach to deliver modern C++ components : {Sonat}

TODO

Find a good short name for the process: Sonat will do for now.

— Mateusz Pusz
Git, CMake, Conan - How to ship and reuse our C++ projects (CppCon 2018)

The tools recommendation is the same as in Mateusz Pusz presentation above (there is hope for a status quo):

VCS

git

Build system management

CMake

Package management

Conan

Structuring content

Repositories

The first practical decision when starting a new project from scratch will be the granularity of the repository. The monorepo, the multirepo (repo-per-component), and the reality in between.

One of monorepo’s advantages is facility to setup and use with most toolsets, avoiding different complications to locate dependent components.

One of multirepo’s advantages is about automation:
The easily detectable "atomic unit of change" is the VCS commit (or push). Where there is only one component in the repo, there is no question as to which component processes should be triggered when change is detected.

Generally our tooling works at repo level

As a general rule of thumb, smaller granularity gives better control and flexibility.

Implementing Conan recipes for 3rd party software

An organisation relying on Conan has dependencies overs software not offering Conan package. To adress this situation, the organisation writes Conan recipes for these package. Ideally, each time a recipe code is pushed back to the central repo, the organisation’s CI would pick it and publish the updated recipe. If a single repositories host tens of recipes, the process will either be naive and wasteful, or will require additional logic to rebuild only the edited recipe(s). If each recipe is hosted in a separate repository, it will be trivial to only trigger a build for the changed recipe.

Updating compiler

Another illustration is how monorepo makes it harder for a single team to change compiler in isolation, even in the context of a stable ABI. Since the new compiler might be more strict regarding C++ standard, it could raise new errors and warnings in the codebase. The compiler change is applied to an entire repository at once:

  • In a multirepo, the team will be able to adapt its own component in isolation.

  • In a monorepo, the compiler change has to be synchronized across all teams.

In practice
  • Pure monorepo is not scalable (i.e. in the context of sociable code). The axiom being that "upstream cannot and should not know all downstreams".

  • On the other hand, strictly one repo per component is not practical in the absence of good tool support [see note below]. The idea of manually having to clone and separately build a handful of independent repos for even medium-sized applications should trigger the maintainability alarm.

Different approaches and tools exist to manage multi-repos. Git submodule is an easily accessible tool, since it is integrated with core Git installations. Yet, a recurrent criticism is submodules do not scale well as they are unpractical to use. In particular, the more correlated the submodules/module, the more this can become a problem.

ℹ️
Correlation measure

Likeliness that changes in entity B would entail changes in entity A.

The proposed system recognises the existence of both mono and multi repo, placing them as extrema on a line along which organisations are allowed to move as the development progresses.

Organically growing codebase

Application uno can start as a library libalpha and its frontend uno. Seeing how they are lock-stepped, it makes sense to host both in the same repo (monorepo). Then, identified generic functionalities can be moved out of libalpha in libcommon. libcommon can start its existence in the same repo, and later on move to a separate repo to be offered to other internal projects and/or 3rd parties. There is value in adaptability.

In a nutshell

The actual system should be able to accommodate monorepos and multi-repos, as well as the reality in between: let’s call it anyrepo. It does not allow for circular dependencies.
The formalisation is that repositories can contain 1..N components, and can depend on components in 0..M other repositories. Repositories dependencies are a DAG.

Filesystem organisation

Once defined which component(s) will be held inside a repository, the repository must be organised in a files and folders hierarchy.

{Sonat} proposed structure
CMakeLists.txt (cmr)
README.{xy}
cmake/
toolOne/
toolTwo/
...
src/
    CMakeLists.txt (cmp)
    apps/
        gamma/
            gamma/
                CMakeLists.txt (cmc-C)
                main.cpp
                appclass.h
                appclass.cpp
                ...
        ...
    libs/
        alpha/
            alpha/
                CMakeLists.txt (cmc-A)
                accumulate.h
                accumulate.cpp
                sum.h
                sum.cpp
                ...
                subcomponent/
                    ...
        beta/
            beta/
                CMakeLists.txt (cmc-B)
                multiply.h
                ...
        ...
    ...
resources/
ℹ️
{Sonat} is intended to be extensible and adaptable.
This is notably the case with the filesystem structure. Additional tool-specific files can be added in the tools folder. Other type of components can be added, for example plugins folder could exist alongside, or replace, libs.
README

The README, even a few lines, makes the most difference when a human encounters a repository for the first time. It is the informal social introduction.

Like the rest of the code, it should be treated as an evolving piece of information.

An potential README outline
  1. The first paragraph describes the functionality of the project / components. As well as the intended audience.

  2. Optional examples.

  3. Usage section, with sub-sections for relevant situations. Classically:

    1. building

    2. installing

    3. using

  4. Pointers to the documentation(s).

  5. Section explaining the contribution model, issue reporting, question asking, or explicitly stating they are not welcome.

Building code

Portability considerations

Standard C++ is a cross platform language, with an ever growing ecosystem of tools. Yet the limiting factor for portability often turns out to be the build system.

Achieving a cross-platform and cross-toolset (code editors, compilers and analysers) build system, while keeping it DRY, is a notable challenge.

DON’T: Many project files and component configurations in the repo

Committing a "project file" per combination would violate DRYness, making it very likely to introduce errors for the system that are not in use when transformations are applied. Moreover, it becomes a burden to add other build systems as soon as the project reaches a moderate size.

CMake is a free and open-source build management system. It places itself one level of abstraction above the makefile/IDE project files: it can be seen (at first) as a project file generator for different toolsets.

TODO

Provide CMake usage statistics and evolution

Building with {Sonat}

When it comes to building, the process requires those essential features:

  • Cross-platform and cross toolset

  • Ability to satisfy upstream dependencies

  • Out of source builds

  • Versionable build process

  • Component level granularity for builds

  • Uniform interface to trigger build of selected components, configurations, and combinations of both

CMake is able to address these different points. It relies on CMakeLists.txt files, with optional xxx.cmake accompanying scripts. Those are plain text files, thus manageable as any other source file by the versioning system.

Conceptually, {Sonat} relies on three categories of CMakeLists.txt files:

  • The root file (cmr), located at the root of the repository.

  • The per-component CMakeLists.txt (cmc-x), at the root of each individual component folder

  • The plumbing CMakeLists.txt (cmp)

Root CMakeLists

It is responsible for initialising CMake and expressing what is common to all, or most, components.

Base:

CMakeLists.txt
# CMake initialisation
cmake_minimum_required(VERSION 3.15)

# Setting the VERSION on root project() will populate CMAKE_PROJECT_VERSION
project(MyRepository
        VERSION "${BUILD_VERSION}")

# Common build settings
if(NOT CMAKE_CXX_STANDARD)
  set(CMAKE_CXX_STANDARD 14)
endif()

# Include components
add_subdirectory(src)

With the add_subdirectory(src) directive, CMake executes the named CMakeLists.txt in the src/ subdirectory (cmp).

This top-level file sets the default (likely minimal requirement) C++ standard, unless a value was already provided for CMAKE_CXX_STANDARD variable.

ℹ️
Making CMAKE_CXX_STANDARD a cache variable would allow to remove the if. Yet it is not known of which nature the variable could already be. (e.g. Conan basic_conan_setup() sets it as non-cache)
ℹ️
{Sonat} recommends that the root project() name starts with an uppercase letter.
TODO

Find a way to control warning level and enable warning as errors for all / some targets, without making it a build requirement. Consumers should be able to build a project even if it generates warning on their newer compilers. Warning should only be treated as errors during development/testing, when the workflow dictates so.

Plumbing CMakeLists

This file will add the individual components. It can use basic logic to conditionally add some components (e.g. Making the tests application optional).

src/CMakeLists.txt
add_subdirectory(libs/alpha/alpha)
add_subdirectory(libs/beta/beta)

add_subdirectory(apps/gamma/gamma)

option(BUILD_tests)
if (BUILD_tests)
 add_subdirectory(apps/tests/tests)
endif()
Per-component CMakeLists

One leaf CMakeLists is present in each component, included by (cmp). It is responsible for actually describing how the component is built.

The process relies on the nested project name as the component’s name, and additionally defines several variable for internal use. This is to ensure a DRY solution, in particular when it comes to lists.

src/libs/alpha/alpha/CMakeLists.txt (component without upstream dependencies)
project(alpha VERSION "${CMAKE_PROJECT_VERSION}")

set(${PROJECT_NAME}_HEADERS
    accumulate.h
    sum.h
)

set(${PROJECT_NAME}_SOURCES
    accumulate.cpp
    sum.cpp
)

# Creates the library target
add_library(${PROJECT_NAME}
            ${${PROJECT_NAME}_HEADERS}
            ${${PROJECT_NAME}_SOURCES})

add_library(myrepo::${PROJECT_NAME} ALIAS ${PROJECT_NAME})

# Defines target requirements
target_include_directories(${PROJECT_NAME}
    PUBLIC
        "$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/../>"
    INTERFACE
        "$<INSTALL_INTERFACE:include/${PROJECT_NAME}>")

# Defines target properties
set_target_properties(${PROJECT_NAME}
    PROPERTIES
        VERSION "${${PROJECT_NAME}_VERSION}")
Modern CMake

CMake was initially holding all the properties and requirements (include path, upstream libraries paths, build flags, etc.) in variables and manually setting them at each folder level.

Some years ago, CMake changed toward what is known as Modern CMake: CMake targets represent the individual software components, encapsulating their requirements and propagating these requirements to downstream projects.
Daniel Pfeifer offers a great presentation of this topic in the video Effective CMake (C++now 2017).

The base snippet above does a few things, and is hopefully direct about each:

project(alpha VERSION "${CMAKE_PROJECT_VERSION}")

Implicitly defines the variables:

  • PROJECT_NAME initialised to "alpha"

  • ${PROJECT_NAME}_VERSION initialised to the version provided to the root project() call

ℹ️
{Sonat} recommends that each CMake target (and associated leaf project()) name starts with a lowercase letter.
set(${PROJECT_NAME}_HEADERS ...)

set(${PROJECT_NAME}_SOURCES ...)

Keeps separate list of headers and sources for the current component.

See list command for advanced operations.

add_library(${PROJECT_NAME}
            ${${PROJECT_NAME}_HEADERS}
            ${${PROJECT_NAME}_SOURCES})

Defines a target named alpha for this component with add_library. It would build fine without listing the headers, yet doing so ensures they show up in IDEs.

add_library(myrepo::${PROJECT_NAME} ALIAS ${PROJECT_NAME})

Defines an alias myrepo::alpha for the target, so alpha is accessible to sibling components under namespace myrepo. It avoids to wonder "should the namespace be prepended in this situation?", while making it easier to relocate components independently.

target_include_directories(${PROJECT_NAME}
 PUBLIC
     "$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/../>"
 INTERFACE
     "$<INSTALL_INTERFACE:include/${PROJECT_NAME}>")

Define a build requirement: the include path.

ℹ️
Without this directive, this component could already include its own headers via relative path (e.g. #include "sum.h").

This directive ensures uniformity, permitting both the component source themselves and its downstream users to include the component headers via compiler’s include path (e.g. #include <sum.h>).
For downstream, this is a requirement, while it is added as a convenience for the current component (most useful when including headers in other directories).

🔥

The parent folder is added to the BUILD_INTERFACE include directories. If the parent folder was directly containing all siblings components, this would break component isolation: it would be possible to include files from any sibling components, without stating an explicit dependency on them.

This is the reason for the duplicated library name folder: this way the current component is the only component available in the added include directory.

Requirements are usually set on 1 out of 3 scopes:

  • PRIVATE will be used when building the component itself, i.e. build specification

  • INTERFACE will be used when building downstreams users of the component, i.e. usage requirements

  • PUBLIC is a shortcut which means both PRIVATE and INTERFACE

🔥
This describe the high level semantic from CMake user perspective.
In practice, PRIVATE requirement might still be propagated (in whole or in parts) to downstreams when the implementation dictates so. For example this is mandatory when linking to a static library target alpha, itself privately linking to another static library target beta. Even though downstream code is not aware of beta, linking downstream to alpha will also require linking downstream to symbols in beta. See https://cmake.org/pipermail/cmake/2016-May/063400.html.
set_target_properties(${PROJECT_NAME}
    PROPERTIES
        VERSION "${${PROJECT_NAME}_VERSION}")

Defines a target property: the target version.

Many properties are available for targets. Some properties are actually requirements that can either be set with set_target_properties or with a dedicated CMake function.

🔥
Explicitly listing files

Since the dawn of CMake and to the day of this writing, the official doc advises against GLOBing to collect all sources files automatically instead of listing them explicitly. The argument stating that CMake needs the file to be touched anyway to regenerate might be seen as weak (if the files are listed explicitly, the file is touched too). The second argument has deeper implications, plus:

  • Explicit is better than implicit

  • It makes it possible to add files conditionally depending on the target system, build parameters, etc., and any combination of those (which would be trickier with GLOB)

This is a domain were tooling could alleviate the pain, for example having a script to create new files and add them to the CMakeLists.

Friction point: duplication of library folders

The duplication of library folders is a pragmatic approach to ensure component isolation, yet it makes for an unusual folder hierarchy.

Executable CMake target

Applications are created via add_executable. When making a native GUI application WIN_32 and/or MACOSX_BUNDLE should be added after the application name.

Header only CMake target

Header only libraries are called Interface Libraries in CMake. Since header only components are not built themselves, they do not have PRIVATE requirement but only INTERFACE, hence the name.
They are added via add_library(${PROJECT_NAME} INTERFACE), and cannot list the headers as source files.

💡

CMake generated IDE projects show compiled targets' sources in the IDE UI, yet none are shown for interface (non-compiled) libraries. A workaround is to create a dummy custom target, whose sole purpose it to show up in the IDE.

add_custom_target(${PROJECT_NAME}_ide
                  SOURCES ${${PROJECT_NAME}_HEADERS})

Using upstream dependencies: CMake

The previous entry describes the process to build a component without upstream dependencies. This section adds some upstream dependencies, showing how to build a component which might re-use something not provided by the standard library.

Finding the dependencies
🔥
The direct approach described here is only used to introduce the necessary notions. The actual approach prescribed by the process, which should be used, is described later.
Since the actual approach might appear less direct due to limitations in the tools, this intermediate step is intended as a gradual explanation.

CMake find upstream dependencies through invocation of find_package command. It is a central command in CMake, with extensive documentation containing important information for project maintainers (strictly following {Sonat} should nevertheless make it work "out of the box").

Modern CMake

This command has two modes

Module

is relying on some external "Find" file (several are distributed with CMake), which traditionally populate variables. It can nonetheless create IMPORTED targets, as is the case with FindBoost (as distributed with CMake).

Config

should be the preferred approach when available, but requires supports from the upstream component.

All components created following {Sonat} are located via the more modern config mode.

To find an upstream dependency, invocations of find_package() are added in the per-component CMakeLists.txt (cmc-). One invocation per upstream dependency, of the form:

find_package(UpstreamName [version [EXACT]] [REQUIRED])
REQUIRED

should appear most of the time. That is, unless the current component can actually build without this dependency (the less probable situation). It allows the overall process to fail early: at CMake configuration time, instead of build time.

version

can be specified to add a lower requirement on the version number of the dependency. EXACT additional keyword makes it that only the exact version is accepted.

A second type of package can be distinguished, which propose multiple components to be included separately. In this case, the components to find are listed after COMPONENTS keyword (or OPTIONAL_COMPONENTS for non-required components). The syntax becomes:

find_package(UpstreamName [version [EXACT]] [REQUIRED] [COMPONENTS component1 [component2]])
ℹ️
Locating upstream dependencies in the root CMakeLists.txt

Some componentised projects locate the dependencies in (cmr), potentially removing repeated invocations of find_package for requirements common to multiple components under the same repository.
{Sonat} instead makes each component responsible to locate its own dependencies.

The finer granularity ease potential relocation of components in other repositories, and allows each component to behave more independently. This will also enables a better contained packaging process.

TODO

Understand why Mateusz Pusz proposes that each component can be built in isolation, without necessarily relying on the root CMakeLists.txt.

Consuming the dependencies

Once CMake found the packages, they must be explicitly marked as dependencies for the downstream target(s). We will consider the modern case, where packages are found as IMPORTED targets. (Reminder: {Sonat} components are found as IMPORTED targets)

Stating the direct dependency relation is done via the CMake function target_link_libraries.

target_link_libraries(${PROJECT_NAME}
                      <PRIVATE|PUBLIC|INTERFACE> [ns::]UpstreamTarget [ns::]OtherUpstream [...]
                      [...])

Even though its name might seem narrow compared to its actual function, this command actually provides all usage requirements for the upstream targets, in addition to the linked-to binary:

  • Include folders

  • Compilation flags and definitions

  • …​

  • Propagation of usage requirements for upstream’s upstreams, recursively

The scope of the linkage has the usual requirement scope meaning.

💡
Even though a syntax without specifying the scope is available, always explicitly provide the scope for easier maintainability.
Putting it together

The updated leaf CMakeLists.txt for a component using dependencies would look something like:

src/libs/alpha/alpha/CMakeLists.txt
project(alpha VERSION "${CMAKE_PROJECT_VERSION}")

set(${PROJECT_NAME}_HEADERS
    accumulate.h
    sum.h
)

set(${PROJECT_NAME}_SOURCES
    accumulate.cpp
    sum.cpp
)

find_package(UpstreamOne REQUIRED)
find_package(UpstreamTwo 1.0 REQUIRED COMPONENTS compA compB)
find_package(UpstreamThree 3.2.5 EXACT REQUIRED)

# Creates the library target
add_library(${PROJECT_NAME}
            ${${PROJECT_NAME}_HEADERS}
            ${${PROJECT_NAME}_SOURCES})

add_library(myrepo::${PROJECT_NAME} ALIAS ${PROJECT_NAME})

# Defines target requirements
target_include_directories(${PROJECT_NAME}
    PUBLIC
        "$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/../>"
    INTERFACE
        "$<INSTALL_INTERFACE:include/${PROJECT_NAME}>")

target_link_libraries(${PROJECT_NAME}
    PUBLIC
        nsOne::UpstreamOne
        nsTwo::compA
        nsTwo::compB
    PRIVATE
        nsThree::UpstreamThree
    INTERFACE
        myrepo::beta)

# Defines target properties
set_target_properties(${PROJECT_NAME}
    PROPERTIES
        VERSION "${${PROJECT_NAME}_VERSION}")
ℹ️
It is also possible to specify normal (non-imported) targets defined by other components in the same repository, as is the case in this example with myrepo::beta. For uniformity, we are using the ALIAS`ed target for `beta (following the rationale.)

Packaging components (becoming an upstream dependency): CMake

The previous section describes how a component can depend on others: this is the consumer side of the DAG connection.
To complete the loop, this section describes how to make a component that can be used following the steps above: the provider side of the DAG connection.

Installing the component files

The CMake infrastructure as described up to this point covers the basic needs of a project to build in the build tree, i.e. under a build directory which is defined when invoking CMake.
There is an additional notion of install tree, a folder where the components is deployed when invoking the install build target implicitly created by CMake.

ℹ️
CMAKE_INSTALL_PREFIX CMake variable controls the base folder (prefix) where the installation takes place. It is important to explicitly define it to avoid the default behaviour of installing system-wide.

The different signatures for install command provide control about which files are deployed when install target is built.

In particular, most of the times installing a component will mean deploying the following files:

built binaries

install(TARGETS ${PROJECT_NAME})

header files
install(FILES ${${PROJECT_NAME}_HEADERS}
        DESTINATION include/${PROJECT_NAME}/${PROJECT_NAME})
ℹ️
Installing header files occurs under a duplicated ${PROJECT_NAME} folder. The rationale is similar than for the duplication of component folders.
Modern(er) CMake

Until CMake 3.14, it was mandatory to specify a DESTINATION when installing any TARGET type. CMake now takes a default location from GNUInstallDirs for the most usual types.

Preparing a CMake package

Installation deploys all the essential files constituting a component into a given folder, as seen above. The component now has to be made into a CMake config-file package. This will allow to find it and use it from the CMakeLists.txt of its consumers.

The package provided by {Sonat} will be usable both from the build-tree (for developers working directly on the component as well as its downstream(s)), and from the install-tree (covering local build-and-installation, as well as package manager distribution of the component).

The process relies on CMake export-sets.

An export for the current target is created by editing the first install invocation as follows:

install(TARGETS ${PROJECT_NAME} EXPORT ${PROJECT_NAME}Targets)

This export-set is then used to generate cmake files in both build and install trees:

# build tree
export(EXPORT ${PROJECT_NAME}Targets
       FILE ${CMAKE_BINARY_DIR}/${PROJECT_NAME}Targets.cmake
       NAMESPACE myrepo::)

# install tree
install(EXPORT ${PROJECT_NAME}Targets
        FILE ${PROJECT_NAME}Targets.cmake
        DESTINATION lib/cmake/${PROJECT_NAME}
        NAMESPACE myrepo::)

Calls to find_package() in downstream will search "for a file called <PackageName>Config.cmake or <lower-case-package-name>-config.cmake". The code creates a file name ${PROJECT_NAME}Target.cmake. A file named ${PROJECT_NAME}Config.cmake, which includes the ${PROJECT_NAME}Target.cmake file, is created via a call to configure_file.

While doing that, it is possible to add basic version checks using a file generated by the write_basic_package_version_file command from CMakePackageConfigHelpers module.

Here is the resulting code:

# Generate config file in the build tree
configure_file(${CMAKE_SOURCE_DIR}/cmake/PackageConfig.cmake.in
               ${CMAKE_BINARY_DIR}/${PROJECT_NAME}Config.cmake
               @ONLY)

# Generate the version file in the build tree
if(PROJECT_VERSION)
    include(CMakePackageConfigHelpers)
    set(_version_file ${CMAKE_BINARY_DIR}/${CMAKE_PROJECT_NAME}ConfigVersion.cmake)
    write_basic_package_version_file(${_version_file}
        VERSION ${PROJECT_VERSION}
        COMPATIBILITY AnyNewerVersion)
endif()

# Install the config and version files over to the install tree
install(FILES ${CMAKE_BINARY_DIR}/${PROJECT_NAME}Config.cmake
              ${_version_file}
        DESTINATION lib/cmake/${PROJECT_NAME})

The first command requires the following template file to be added in the cmake folder at the root of the repository:

cmake/PackageConfig.cmake.in
include("${CMAKE_CURRENT_LIST_DIR}/@[email protected]")
ℹ️
It currently seems this file introduces an extra indirection for no reason, yet this template will grow larger with further steps.
ℹ️
AnyNewerVersion can be replaced by any valid value for the COMPATIBILITY argument.
For targets where the produced artifact will be the same for all supported systems, i.e. header only CMake targets, the extra argument ARCH_INDEPENDENT should be given to write_basic_package_version_file.
Failing to do so, when distributing the same package for all systems, said package will break on architectures that do not match the architecture where the unique package was produced.

Putting it together

For a repository containing a single component, an updated leaf CMakeLists.txt able to produce a CMake package would look something like:

src/libs/alpha/alpha/CMakeLists.txt
project(alpha VERSION "${CMAKE_PROJECT_VERSION}")

set(${PROJECT_NAME}_HEADERS
    accumulate.h
    sum.h
)

set(${PROJECT_NAME}_SOURCES
    accumulate.cpp
    sum.cpp
)

find_package(UpstreamOne REQUIRED)
find_package(UpstreamTwo 1.0 REQUIRED COMPONENTS compA compB)
find_package(UpstreamThree 3.2.5 EXACT REQUIRED)

# Creates the library target
add_library(${PROJECT_NAME}
            ${${PROJECT_NAME}_HEADERS}
            ${${PROJECT_NAME}_SOURCES})

add_library(myrepo::${PROJECT_NAME} ALIAS ${PROJECT_NAME})

# Defines target requirements
target_include_directories(${PROJECT_NAME}
    PUBLIC
        "$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/../>"
    INTERFACE
        "$<INSTALL_INTERFACE:include/${PROJECT_NAME}>")

target_link_libraries(${PROJECT_NAME}
    PUBLIC
        nsOne::UpstreamOne
        nsTwo::compA
        nsTwo::compB
    PRIVATE
        nsThree::UpstreamThree
    INTERFACE
        myrepo::beta)

# Defines target properties
set_target_properties(${PROJECT_NAME}
    PROPERTIES
        VERSION "${${PROJECT_NAME}_VERSION}")

install(TARGETS ${PROJECT_NAME} EXPORT ${PROJECT_NAME}Targets)
install(FILES ${${PROJECT_NAME}_HEADERS}
        DESTINATION include/${PROJECT_NAME}/${PROJECT_NAME})

# build tree
export(EXPORT ${PROJECT_NAME}Targets
       FILE ${CMAKE_BINARY_DIR}/${PROJECT_NAME}Targets.cmake
       NAMESPACE myrepo::)
configure_file(${CMAKE_SOURCE_DIR}/cmake/PackageConfig.cmake.in
               ${CMAKE_BINARY_DIR}/${PROJECT_NAME}Config.cmake
               @ONLY)
if(PROJECT_VERSION)
    include(CMakePackageConfigHelpers)
    set(_version_file ${CMAKE_BINARY_DIR}/${CMAKE_PROJECT_NAME}ConfigVersion.cmake)
    write_basic_package_version_file(${_version_file}
        VERSION ${PROJECT_VERSION}
        COMPATIBILITY AnyNewerVersion)
endif()

# install tree
install(EXPORT ${PROJECT_NAME}Targets
        FILE ${PROJECT_NAME}Targets.cmake
        DESTINATION lib/cmake/${PROJECT_NAME}
        NAMESPACE myrepo::)
install(FILES ${CMAKE_BINARY_DIR}/${PROJECT_NAME}Config.cmake
              ${_version_file}
        DESTINATION lib/cmake/${PROJECT_NAME})
Friction point

This task appears to be generic, yet requires to add many line of codes, repeated in each leaf CMakeLists.txt. This boilerplate will grow even larger as package handle their direct dependencies.
For the moment, it is recommended to factorise this logic in a custom CMake function, yet it should ideally be discussed with CMake experts and maintainers to see if this situation can be streamlined.

Multiple components in a single CMake package

The approach described above will produce a CMake package with the name of the leaf project (alpha, in this specific case). This is satisfying for single component repositories, yet a complication arises in the case of multiple components per repo.

When applied in a repository containing many components, this produces as many packages as there are components. This means downstream would issue a distinct find_package() to find each required component, each being a separate CMake package.
Yet, CMake would still install all components from the repository under the common path prefix CMAKE_INSTALL_PREFIX. Due to the find_package() search procedure, this would imply providing CMake with one distinct hint for each component, in each upstream repository.

Instead, {Sonat} relies on the ability of find_package() to locate several components under a common top-level package name:
This fits naturally with the anyrepo model, as each leaf CMakeLists.txt will map to a component, and the top level project() name (the repository) will map to the package name.
It will notably allow to locate all components in all repositories by providing a single CMake hint. (leveraging the <prefix>/<name>*/(lib/<arch>|lib*|share)/cmake/<name>*/ search entry).

To implement this multiple components approach, an additional CMake config file is issued, named after the top level project. This step naturally fits the top-level CMake file:

CMakeLists.txt
# CMake initialisation
cmake_minimum_required(VERSION 3.15)

# Setting the VERSION on root project() will populate CMAKE_PROJECT_VERSION
project(MyRepository
        VERSION "${BUILD_VERSION}")

# Common build settings
if(NOT CMAKE_CXX_STANDARD)
  set(CMAKE_CXX_STANDARD 14)
endif()

# Include components
add_subdirectory(src)

# Multi-component package
# Generate the root config and version check in the build tree
configure_file(${CMAKE_SOURCE_DIR}/cmake/ComponentPackageRootConfig.cmake.in
               ${CMAKE_BINARY_DIR}/${CMAKE_PROJECT_NAME}Config.cmake
               @ONLY)
if(PROJECT_VERSION)
    include(CMakePackageConfigHelpers)
    set(_version_file ${CMAKE_BINARY_DIR}/${CMAKE_PROJECT_NAME}ConfigVersion.cmake)
    write_basic_package_version_file(${_version_file}
        VERSION ${CMAKE_PROJECT_VERSION}
        COMPATIBILITY AnyNewerVersion)
endif()

# Install the root config file over to the install tree
install(FILES ${CMAKE_BINARY_DIR}/${CMAKE_PROJECT_NAME}Config.cmake
              ${_version_file}
        DESTINATION lib/cmake/${CMAKE_PROJECT_NAME})
🔥
This uses the root project() name as the package name. Matching this name with the repository’s name is a convenient solution.

The added code relies on additional template file ComponentPackageRootConfig.cmake.in to exist in cmake folder:

cmake/ComponentPackageRootConfig.cmake.in
if (NOT ${CMAKE_FIND_PACKAGE_NAME}_FIND_COMPONENTS)
    set(${CMAKE_FIND_PACKAGE_NAME}_NOT_FOUND_MESSAGE "The '${CMAKE_FIND_PACKAGE_NAME}' package requires at least one component")
    set(${CMAKE_FIND_PACKAGE_NAME}_FOUND False)
    return()
endif()

include(CMakeFindDependencyMacro)
foreach(module ${${CMAKE_FIND_PACKAGE_NAME}_FIND_COMPONENTS})
    set (_config_location "${CMAKE_CURRENT_LIST_DIR}/${module}")


    # Error when a component has exactly the same identifier as the package_name
    # (would first find the current Config.cmake, because xxx_DIR variable is already set)
    if(module STREQUAL ${CMAKE_FIND_PACKAGE_NAME})
        set(${CMAKE_FIND_PACKAGE_NAME}_NOT_FOUND_MESSAGE
            "The '${CMAKE_FIND_PACKAGE_NAME}' package cannot list a component with identical name.\
             Use at least a distinct capitalization \
             (advice: package names start with capital, components do not).")
        set(${CMAKE_FIND_PACKAGE_NAME}_FOUND False)
        return()
    endif()

    # find_dependency should forward the QUIET and REQUIRED arguments
    find_dependency(${module} CONFIG
                    PATHS "${_config_location}"
                    NO_DEFAULT_PATH)
    if (NOT ${module}_FOUND)
        if (${CMAKE_FIND_PACKAGE_NAME}_FIND_REQUIRED_${module})
            string(CONCAT _${CMAKE_FIND_PACKAGE_NAME}_NOTFOUND_MESSAGE
                   "Failed to find ${CMAKE_FIND_PACKAGE_NAME} component \"${module}\" "
                   "config file at \"${_config_location}\"\n")
        elseif(NOT ${CMAKE_FIND_PACKAGE_NAME}_FIND_QUIETLY)
            message(WARNING "Failed to find ${CMAKE_FIND_PACKAGE_NAME} component \"${module}\" "
                             "config file at \"${_config_location}\"")
        endif()
    endif()

    unset(_config_location)
endforeach()

if (_${CMAKE_FIND_PACKAGE_NAME}_NOTFOUND_MESSAGE)
    set(${CMAKE_FIND_PACKAGE_NAME}_NOT_FOUND_MESSAGE "${_${CMAKE_FIND_PACKAGE_NAME}_NOTFOUND_MESSAGE}")
    set(${CMAKE_FIND_PACKAGE_NAME}_FOUND False)
endif()
ℹ️
Execution of this Config script might be recursive via the find_dependency call, in cases where components of a given CMake package depend on other components inside the same package. Since the different recursive invocations occur in the same "variable scope", the unset(_config_location) occurring in a nested call before returning to its caller would also erase this value for said caller. For this reason, re-set _config_location variable at each iteration of the foreach loop (in case a nested call in a previous iteration of the loop has unset _config_location).

This template leverages the config files still produced and installed by each individual component in order to locate them, via the call to find_dependency().

ℹ️
The template is looking for the individual components config files in a subfolder with the component name, by calling find_dependency with a PATHS value of ${CMAKE_CURRENT_LIST_DIR}/${module}.
This extra folder is added in case one of the components has the same name than the root CMake project, with only differences in capitalization. In such situtation, the the root project XxxConfig.cmake file and the component xxxConfig.cmake file would collide on case-insensitive file systems if they were both placed in the same folder.

This multi-component transformation also induces three changes in the leaf CMakeLists.txt compared to what was presented above:

  • The version file is already generated at the top level, no need to version components individually.

  • The config files must be placed in a subfolder with the component name (see note above).

  • The install destination must be adapted to match the root project name and component subfolder.

src/libs/alpha/alpha/CMakeLists.txt
project(alpha VERSION "${CMAKE_PROJECT_VERSION}")

set(${PROJECT_NAME}_HEADERS
    accumulate.h
    sum.h
)

set(${PROJECT_NAME}_SOURCES
    accumulate.cpp
    sum.cpp
)

find_package(UpstreamOne REQUIRED)
find_package(UpstreamTwo 1.0 REQUIRED COMPONENTS compA compB)
find_package(UpstreamThree 3.2.5 EXACT REQUIRED)

# Creates the library target
add_library(${PROJECT_NAME}
            ${${PROJECT_NAME}_HEADERS}
            ${${PROJECT_NAME}_SOURCES})

add_library(myrepo::${PROJECT_NAME} ALIAS ${PROJECT_NAME})

# Defines target requirements
target_include_directories(${PROJECT_NAME}
    PUBLIC
        "$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/../>"
    INTERFACE
        "$<INSTALL_INTERFACE:include/${PROJECT_NAME}>")

target_link_libraries(${PROJECT_NAME}
    PUBLIC
        nsOne::UpstreamOne
        nsTwo::compA
        nsTwo::compB
    PRIVATE
        nsThree::UpstreamThree
    INTERFACE
        myrepo::beta)

# Defines target properties
set_target_properties(${PROJECT_NAME}
    PROPERTIES
        VERSION "${${PROJECT_NAME}_VERSION}")

install(TARGETS ${PROJECT_NAME} EXPORT ${PROJECT_NAME}Targets)
install(FILES ${${PROJECT_NAME}_HEADERS}
        DESTINATION include/${PROJECT_NAME}/${PROJECT_NAME})

# build tree
export(EXPORT ${PROJECT_NAME}Targets
       FILE ${CMAKE_BINARY_DIR}/${PROJECT_NAME}/${PROJECT_NAME}Targets.cmake
       NAMESPACE myrepo::)
configure_file(${CMAKE_SOURCE_DIR}/cmake/PackageConfig.cmake.in
               ${CMAKE_BINARY_DIR}/${PROJECT_NAME}/${PROJECT_NAME}Config.cmake
               @ONLY)

# Removed lines

# install tree
install(EXPORT ${PROJECT_NAME}Targets
        FILE ${PROJECT_NAME}Targets.cmake
        DESTINATION lib/cmake/${CMAKE_PROJECT_NAME}/${PROJECT_NAME}
        NAMESPACE myrepo::)
install(FILES ${CMAKE_BINARY_DIR}/${PROJECT_NAME}/${PROJECT_NAME}Config.cmake
              # Removed line
        DESTINATION lib/cmake/${CMAKE_PROJECT_NAME}/${PROJECT_NAME})

Finding upstream dependencies from a CMake package

The current CMake code allows downstreams to find requested components in a package, each component forwarding its direct requirements. Those direct requirements in turn forward their own requirements: the requirements are transitively forwarded by a recursive traversal of the upstream dependencies graph.

Yet, for this exhaustive process to take place, each upstream must be found (in order for its corresponding IMPORTED target to exist in the current CMake context) before it is expressed as a direct dependency on a target (via target_link_libraries for dependencies found as IMPORTED targets).

When implementing a component following {Sonat}, its direct dependencies are all found in the component’s leaf CMakeLists.txt: this takes care of the first level of dependency. Yet, those direct dependencies might have their own dependencies, which are no directly found in the initial component’s CMakeLists.txt.

🔥
To sum-up: the xxxTarget.cmake file generated by CMake for the direct dependencies does not find its direct dependencies.

To be properly self-contained, a CMake package must thus find its direct dependencies. Issuing the necessary find_ commands is a responsibility left to the package developer. The official CMake documentation recommends to find the dependencies for the packaged component directly in its xxxConfig.cmake file. Yet, explicitly writing the find_ calls in both the leaf CMakeLists.txt and its generated xxxConfig.cmake would be a major violation of DRY.

{Sonat} improvises a solution to keep a single occurrence of the dependencies list, using only CMake facilities. The calls to find_package are moved away from the leaf CMakeLists.txt to a custom template file CMakeFinds.cmake.in, where the following tokens are wrapped in @ pairs:

  • find_package

  • REQUIRED

  • QUIET

src/libs/alpha/alpha/CMakeFinds.cmake.in
@find_package@(UpstreamOne @REQUIRED@)
@find_package@(UpstreamTwo 1.0 @REQUIRED@ COMPONENTS compA compB)
@find_package@(UpstreamThree 3.2.5 EXACT @REQUIRED@ @QUIET@)
ℹ️
Resulting CMakeFinds.cmake is not a standard CMake file.
🔥
CMake documentation also implies that only PUBLIC dependencies must be found for downstreams. Yet, as seen earlier, this might also be the case for some PRIVATE dependencies, for example static libraries.

In CMakeLists.txt, the different find_package() calls are replaced with a single configuration of the above and execution of the result:

function(local_find)
    set(REQUIRED "REQUIRED")
    set(QUIET "QUIET")
    set(find_package "find_package")
    configure_file(CMakeFinds.cmake.in CMakeFinds.cmake @ONLY)
    include(${CMAKE_CURRENT_BINARY_DIR}/CMakeFinds.cmake)
endfunction()
local_find()
ℹ️
The sole purpose of defining a function here instead of inlining its content is to scope the defined variable to a restricted block. In production code, this function should likely be factorised outside of any leaf CMakeLists.txt, and reused.

In substance, this generates a file with a content strictly equal to what was removed from the leaf CMakeLists.txt, and includes it: functionally equivalent. Yet, it will now be possible to reuse this information from the alphaConfig.cmake file after configuring it with different substitutions.

Yet, this does not address the case of internal dependencies: in the current example alpha having a requirement for myrepo::beta is an internal dependency.
Since those targets are already defined under the same repository / same root CMakeLists.txt, they are not found via calls to find_package in their sibling components: the actual target exists in the current CMake context. On the other hand, when exporting a xxxConfig.cmake file, those sibling targets are not defined anymore. The package developer must then once again take measures to make sure they are explicitly found in the install tree.

{Sonat} finds the internal dependencies alongside the other dependencies, but it defines a separate substitution for internal components: they are using @find_internal_package@ instead of @find_package@.

src/libs/alpha/alpha/CMakeFinds.cmake.in
@find_package@(UpstreamOne @REQUIRED@)
@find_package@(UpstreamTwo 1.0 @REQUIRED@ COMPONENTS compA compB)
@find_package@(UpstreamThree 3.2.5 EXACT @REQUIRED@ @QUIET@)
@find_internal_package@(MyRepository @REQUIRED@ COMPONENTS beta CONFIG)

In the leaf CMakeLists.txt, the local_find() function defined above is extended with a value for @find_internal_package@. It will simply comment-out the instruction when the project itself is configured.

function(local_find)
    set(REQUIRED "REQUIRED")
    set(QUIET "QUIET")
    set(find_package "find_package")
    set(find_internal_package "#Internal component: find_package")
    configure_file(CMakeFinds.cmake.in CMakeFinds.cmake @ONLY)
    include(${CMAKE_CURRENT_BINARY_DIR}/CMakeFinds.cmake)
endfunction()
local_find()

This also achieves functional equivalence to the previous solution, with the added ability to use a different substitution when producing the find file for the package config.

Now, the dependencies information has to be made available and consumed by the package alphaConfig.cmake file.

Making dependency information available

Following recommendations from the official documentation, the package will find its upstream dependencies via the find_dependency() macro instead of the find_package() function. This macro notably forwards QUIET and REQUIRED arguments, so they should not be written explicitly.

This is achieved by configuring the CMakeFinds.cmake.in template a second time from the leaf CMakeLists.txt. This time with different substitutions, in particular no substitution for @REQUIRED@ nor @QUIET@:

function(config_find)
    set(find_package "find_dependency")
    set(find_internal_package "find_dependency")
    # Configure in build tree
    configure_file(CMakeFinds.cmake.in
                   ${CMAKE_BINARY_DIR}/${PROJECT_NAME}/${PROJECT_NAME}FindUpstream.cmake
                   @ONLY)
endfunction()
config_find()
ℹ️
The resulting configured file is placed relative to the root of the binary directory, instead of in the current binary directory as was the case with local_find()

This new file has to be deployed to the install tree:

    install(FILES ${CMAKE_BINARY_DIR}/${PROJECT_NAME}/${PROJECT_NAME}Config.cmake
            # Optional version file if single component repository
            ${CMAKE_BINARY_DIR}/${PROJECT_NAME}/${PROJECT_NAME}FindUpstream.cmake
            DESTINATION lib/cmake/${CMAKE_PROJECT_NAME}/${PROJECT_NAME})

The root template PackageConfig.cmake.in has to be edited to include this file:

cmake/PackageConfig.cmake.in
include(CMakeFindDependencyMacro) # Provides find_dependency() macro
include("${CMAKE_CURRENT_LIST_DIR}/@[email protected]" OPTIONAL)

include("${CMAKE_CURRENT_LIST_DIR}/@[email protected]")

Putting it together

The install and packaging logic proposed by {Sonat} is now complete, which gives the following final leaf CMakeLists.txt for a multi-components repository:

src/libs/alpha/alpha/CMakeLists.txt
project(alpha VERSION "${CMAKE_PROJECT_VERSION}")

set(${PROJECT_NAME}_HEADERS
    accumulate.h
    sum.h
)

set(${PROJECT_NAME}_SOURCES
    accumulate.cpp
    sum.cpp
)

function(local_find)
    set(REQUIRED "REQUIRED")
    set(QUIET "QUIET")
    set(find_package "find_package")
    set(find_internal_package "#Internal component: find_package")
    configure_file(CMakeFinds.cmake.in CMakeFinds.cmake @ONLY)
    include(${CMAKE_CURRENT_BINARY_DIR}/CMakeFinds.cmake)
endfunction()
local_find()

function(config_find)
    set(find_package "find_dependency")
    set(find_internal_package "find_dependency")
    configure_file(CMakeFinds.cmake.in
                   ${CMAKE_BINARY_DIR}/${PROJECT_NAME}/${PROJECT_NAME}FindUpstream.cmake
                   @ONLY)
endfunction()
config_find()

# Creates the library target
add_library(${PROJECT_NAME}
            ${${PROJECT_NAME}_HEADERS}
            ${${PROJECT_NAME}_SOURCES})

add_library(myrepo::${PROJECT_NAME} ALIAS ${PROJECT_NAME})

# Defines target requirements
target_include_directories(${PROJECT_NAME}
    PUBLIC
        "$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/../>"
    INTERFACE
        "$<INSTALL_INTERFACE:include/${PROJECT_NAME}>")

target_link_libraries(${PROJECT_NAME}
    PUBLIC
        nsOne::UpstreamOne
        nsTwo::compA
        nsTwo::compB
    PRIVATE
        nsThree::UpstreamThree
    INTERFACE
        myrepo::beta)

# Defines target properties
set_target_properties(${PROJECT_NAME}
    PROPERTIES
        VERSION "${${PROJECT_NAME}_VERSION}")

install(TARGETS ${PROJECT_NAME} EXPORT ${PROJECT_NAME}Targets)
install(FILES ${${PROJECT_NAME}_HEADERS}
        DESTINATION include/${PROJECT_NAME}/${PROJECT_NAME})

# build tree
export(EXPORT ${PROJECT_NAME}Targets
       FILE ${CMAKE_BINARY_DIR}/${PROJECT_NAME}/${PROJECT_NAME}Targets.cmake
       NAMESPACE myrepo::)
configure_file(${CMAKE_SOURCE_DIR}/cmake/PackageConfig.cmake.in
               ${CMAKE_BINARY_DIR}/${PROJECT_NAME}/${PROJECT_NAME}Config.cmake
               @ONLY)

# install tree
install(EXPORT ${PROJECT_NAME}Targets
        FILE ${PROJECT_NAME}Targets.cmake
        DESTINATION lib/cmake/${CMAKE_PROJECT_NAME}/${PROJECT_NAME}
        NAMESPACE myrepo::)
install(FILES ${CMAKE_BINARY_DIR}/${PROJECT_NAME}/${PROJECT_NAME}Config.cmake
        ${CMAKE_BINARY_DIR}/${PROJECT_NAME}/${PROJECT_NAME}FindUpstream.cmake
        DESTINATION lib/cmake/${CMAKE_PROJECT_NAME}/${PROJECT_NAME})
Friction point: Lengthy boilerplate and hackish workarounds

As already evoked, the leaf CMakeLists.txt now contains even more generic boilerplate, which should at least be factorised away in a function.

Is there a canonical way to reduce this? Would there be interest in turning the repetitive code into an official CMake macro? Even the explicit code is able to adapt to many more different situations, it feels like this case might be a sane default starting point for modern C++ libraries.

Additionally, the current solution to keep the list of external and internal dependencies DRY is a hack, which might be wasteful (all the dependencies will be "found" by the package consumers, even the PRIVATE dependencies that are actually not forwarded):

What is the rationale for not making the automatic xxxTarget.cmake code generation handle the find_ directly? Could CMake provide the actual list of internal and external dependencies which actually need to be found by consumers for the packaged target?

Friction point: find_dependency may have contradictory documentation, and might not behave as expected

In short, find_dependency(beta) indeed forwards REQUIRED from the calling find_package(alpha), which makes the call fails in beta, without the promised diagnostic mentioning that "alpha cannot be used without beta".

A more "natural" approach might actually be not to forward it, since REQUIRED actually only applies to the calling find_package, which might have independently REQUIRED and optional dependencies.

Friction point: Usage of custom CMake variables

{Sonat} current leaf CMakeLists.txt rely on defining several custom variables.
Yet, different talks regarding modern CMake discourage the use of custom variables (see Daniel Pfeifer example). Nevertheless, in the absence of a specialised handling of headers and internal target dependencies, as well as a more integrated handling of package upstream dependencies, this use of variables seems like the lesser evil when compared to DRY violations.

Distribution: Conan

Once the software exists as a self contained package, making it easily available for its entire intended audience is the next goal.

ℹ️

Here, audience is to be taken broadly:

  • Collaborators (Developers, Testers, etc.)

  • Clients

  • Automated processes (CI, CD, etc.)

Distributing the package itself is one step, yet the bigger picture is also concerned with its upstream dependencies.

Motivations

For code to actually become sociable, it must scale all the way from only using a handful of dependencies, to being one component in the middle of a many-thousands dependencies graph.

In some organisations, collaborators locally deploy each dependency manually (via compilation or package manager invocations). This approach is manageable only for shallow dependency graphs, or when most direct dependencies are already well behaved sociable components.

There are CMake facilities intended to ease such steps, with the ability to automatically retrieve / build dependencies. Yet, those automation facilities are usually limited, in the sense that they give only local visibility of the direct dependencies, not the whole-picture dependency graph.

Facing the new challenge of distributing components in varying dependency graphs, separation of concerns is an important consideration:

{Sonat} relies on CMake to do one thing well

describe the local build process for the repository’s component(s) in a portable, tool and platform agnostic textual format.

Dependencies management is a separate goal

retrieving all the artifacts for the dependencies, while handling recursion through the upstream graph (addressing different complications, such as reconciliation of diamond sub-graphs)

When it comes to handling dependencies, a scalable and well-accepted solution is to use a package manager.
In the context of {Sonat}, a package manager should offer those essential features:

  • Cross-platform and cross-toolset

  • Versionable with the code

  • Testable

  • Handle dependencies versioning

  • Ability to generate a complete (recursive) dependency graph, and handle reconciliation of duplicated dependencies in different versions

  • Usable by developers, automated processes, and end-users.

  • Good defaults, with advanced customisation

  • Artifacts caching and sharing (for time-consuming builds and space-consuming resulting binaries)

  • First-class support for the specificity of C++ (native) code, see C++ special case

{Sonat} relies on Conan, self-dubbed the C / C++ Package Manager for Developers.

ℹ️
Conan is cross-toolset in two ways: it offers integrated support for many major tools, while also allowing to easily issue system commands to handle specific situations and non-modern code repositories.
It notably offers excellent first-class support for CMake with different generators, making it a good choice to distribute CMake based repositories.
ℹ️
Conan Getting Started offers a good first-time walkthrough.
ℹ️
Daniel Pfeifer’s requirements for a package manager can be satisfied via Conan.

Adding Conan recipe

Conan relies on recipes, either simple declarative conanfile.txt, or both declarative and imperative (greatly customisable) conanfile.py. Conan follow recipes to produce packages (i.e. the resulting artifact) that can be cached, distributed, and directly retrieved by consumers to satisfy dependencies (alleviating the need to build locally).

Recipes are fully contained, notably providing:

{Sonat} implements a single recipe by repository, independently of its number of components. It can be placed at the root of the repository, yet storing it in a separate conan folder allows to group all Conan related functionalities in one place (e.g. testing). This conan folder is a concrete example of the generic tool folders discussed in filesystem organization.

conan/conanfile.py
from conans import ConanFile, CMake, tools


class MyRepositoryConan(ConanFile):
    # Recipe meta-information
    name = "myrepository"
    license = "MIT"
    url = "..."
    description = "A Conan recipe for {Sonat} sample repository"
    topics = ("demonstration")

    # Which generators are run when obtaining the code dependencies, before build()
    generators = "cmake_paths", "cmake"

    # The default "hash" mode would result in different recipe revisions for Linux and Windows
    # because of difference in line endings
    revision_mode = "scm"

    # (overridable) defaults for consumers
    build_policy = "missing"

    # Package variability:
    # Changing those values will result in distinct packages for the same recipe
    settings = "os", "compiler", "build_type", "arch"
    options = {
        "shared": [True, False],
        "build_tests": [True, False],
    }
    default_options = {
        "shared": False,
        "build_tests": False,
    }

    # Code dependencies
    requires = ("upstreamone/1.0@one/stable",
               "upstreamtwo/[>1.0]@two/stable",
               "upstreamthree/[~=3.2.5]@three/stable")

    # Build dependencies
    #   CMake will not need to be installed to build the project
    #   And if it was installed in a non-compatible version, this will take precedence anyway
    build_requires = "cmake_installer/3.15.7"


    # Build procedure: code retrieval
    #   Git's repository origin remote and its current revision are captured by recipe export
    scm = {
        "type": "git",
        "url": "auto",
        "revision": "auto",
        "submodule": "recursive",
    }


    # shared CMake configuration
    def _configure_cmake(self):
        cmake = CMake(self)
        cmake.definitions["BUILD_tests"] = self.options.build_tests
        cmake.configure()
        return cmake


    # Build procedure: actual build
    def build(self):
        cmake = self._configure_cmake()
        cmake.build()


    # Packaging procedure
    def package(self):
        cmake = self._configure_cmake()
        cmake.install()


    # Package-consumer instructions
    def package_info(self):
        self.cpp_info.libs = tools.collect_libs(self)

This recipe has several sections, each of low complexity. In particular, the build and packaging procedures are short, thanks to first class integration of CMake in Conan:

  1. In each case, a CMake Python object is instantiated, its attributes defined from the provided settings and options, then it is configured.

  2. build() or install() method is invoked according to the current step. Packaging leverages the installation logic provided by CMake through the install target.

🔥
The recipe revision mode is explicitly set to revision_mode = scm, instead of the default hash mode. As its value indicates, the default mode computes the recipe revision by hashing the recipe file.
Since hashing notably takes line endings into account, this might result in different revisions being computed depending on the host system (CRLF vs CR vs LF line endings) and git’s configuration.
Having different revisions for what is actually the exact same recipe would be conceptually wrong, and could also break the actual distribution via Conan repositories: if prebuilt packages for all systems are expected to live under a single recipe revision in the central repository (as is intended), then a for systems with a non-matching line ending, the package might not be found under the correct revision.
ℹ️
The shared option and build_type setting are common in recipes, thus Conan implicitly forwards the corresponding definitions to the CMake object. On the other hand, the custom build_tests option is manually forwarded. This explicit approach allows complete customisation of the CMake variables. The documentation provides the list of automatic variables.
ℹ️
Until v0.7, {Sonat} introduced a cloned_repo subfolder to clone into. The rationale was that: when invoking conan install, Conan will copy the content of its source folder directly at the root of the build folder. With this cloned_repo subfolder, the different files at the root of the repository would not be copied directly at the root of the build folder, reducing the risk of filename collision with build files.

This cloned_repo subfolder has been deprecated, because it breaks Conan commands where the source folder is explicitly provided via the -sf command line arguments. (Usually pointing it to a development folder, not containing this articifial cloned_repo subfolder).

Taking a step back

As Conan package manager was introduced, now is a good time to take a look at the overall picture.

The repository contains a project composed of one or several components. The project needs to be built in order to produce usable artifacts.

While CMake manages the details of the build system for an isolated repository, two essentials issues remain:

  • The code is unique, but there is a lot of variability in the produced artifacts. A first source of variability is the target environment: C++ model is write once, build everywhere (i.e. many times). There is also variability in how a project is built even for a single defined environment (Debug/Release, compiler flags, C++ standard version, optional components, etc.)

  • Building might require to satisfy an arbitrarily complicated dependency graph.

Conan tool is addressing these two issues: it resolves the dependency graphs, and it models the variability of environments and projects via options and settings.

Between the build system and the Conan tool sits the recipe: It lists the different dependencies, as well as the options and settings. One of its crucial responsibility is to abstract the build system behind the recipe’s build() method, while making sure each option and setting is properly translated/forwarded.

From Conan options and settings to CMake variables

Conan’s CMake build helper

One of {Sonat} goal is to minimise coupling. Conan lives in a higher layer than CMake: it ensues CMakeLists.txt scripts should ideally not be aware of Conan, or at the very least should not require it. {Sonat} intends to accommodate a variety of workflows, and it is reasonable for some workflows to build the project directly from CMake generated build files, out of Conan. (Such use-cases optionally could rely on Conan to provide some, or all, of the upstream dependencies. Flexibility is a virtue).

🔥
If a recipe introduces custom options and settings, it must do all the work to provide the values to the build system and make sure the build system is configured according to those values.

Among the different CMake variables defined by the build helper, some are mapped to native CMake variables (usually variables prefixed with CMAKE_). CMake directly takes these native variable into account, as such no further steps are required. The build helper notably defines:

  • CMAKE_BUILD_TYPE (from options.build_type)

  • CMAKE_OSX_ARCHITECTURES (from a combination of settings.os and settings.arch or settings.arch_target)

🔥
This works reliably only if the project’s CMake scripts do not override the values assigned to those variables. Is should be considered a bad practice for a CMake script to discard user-provided values for such variables.

Yet, the majority of CMake variables defined by the helper are custom Conan variables (aptly prefixed with CONAN_).
CMake is unaware of such variables, thus those variables would be ignored by default. It results that further explicit steps must be introduced, otherwise the recipe would not fulfil its contract to properly forward the variability to the build system.

Conan’s CMake generator

As stated above, some extra logic must be introduced to accommodate the CONAN_ CMake variables: Conan generated files to the rescue. One of the early generator proposed by Conan is the cmake generator. It generates a conanbuildinfo.cmake file essentially offering three things:

  • Define package-namespaced variable, providing values for each upstream package independently

  • Define amalgamation variables, encompassing all the upstream packages

  • Propose a set of user-invokable macros, notably the conan_basic_setup() aggregation of other macros.

Some of these macros are handling the CONAN_ prefixed variables, to actually apply them to the build system:

  • check_compiler_version()

  • conan_set_std()

  • conan_set_libcxx()

  • conan_set_vs_runtime()

ℹ️

Those varied features allow this generator to easily fit within a vast variety of the pre-existing CMake based projects in circulation: be it an modern Conan-aware CMake infrastructure leveraging conan_define_targets() to provide its needed targets, or an old(deprecated)-style CMake project entirely relying on setting folder-level property via the loosely-grouped variables. And there are a great many combinations in between.

This might be a reason why it was introduced early in Conan releases, and why it is the advertised generator in the Getting started for package creation. When coupled with conan_basic_setup() invocation, it works reliably in the diverse landscape of CMake based projects, over which Conan developers have little control.

Friction point: Inconsistency in the CMake build helper implicit variables

As illustrated, the CMake build helper directly sets some variables as native CMake variables, while other variables require CMake scripts logic in order to be taken into account. This difference likely exists because the helper directly sets all the native variables it can, yet some values can only be translated during the CMake configuration process:

Enforcing the compiler and its version

Currently implemented as a check comparing the Conan provided value to which compiler CMake actually picked. Is about to change: to be addressed pro-actively via a CMAKE_TOOLCHAIN_FILE.

cppstd and gnuextensions

requires knowing the CMake version, to address versions of CMake before the introduction of native variables CMAKE_CXX_STANDARD and CMAKE_CXX_EXTENSIONS (in {Sonat} specific case, this version is actually known in advance, since CMake is a build_dependency)

stdlib

potentially requires extending the CMAKE_CXX_FLAGS

This inconsistent situation might lead to confusion, and problematic recipes.

conan_basic_setup alternative

While the cmake generator just works in a variety of different situations, {Sonat} projects have well known and precise characteristics. They are written against modern target-based CMake, keeping away from requirements provided as independent variables.

In {Sonat} specific situation, it might appear that the widely encompassing approach taken by the cmake generator brings a few drawbacks:

  • Variable pollution, with a vast majority of globally defined variable remaining unused by the build management

  • Opinionated new defaults, introducing incompatibilities between default Conan builds and default CMake builds. (e.g. conan_basic_setup() disable RPATH by default, which is not CMake’s default)

  • Usage of the generated CMake script is invasive, requiring dedicated code in the root CMakeLists.txt

{Sonat} aims to write canonical CMake scripts. In his presentation "Effective CMake", Daniel Pfeifer presents the canonical way to use an external library. When using an external {Sonat} component, this syntax is the natural solution, and it only requires the (concise) output of cmake_paths generator.

cmake_paths generated script only populates 2 variables, and it does not define any logic. It can be included non-intrusively either as a toolchain, or indirectly at CMake project() invocation.
{Sonat} advocates the second solution. The generated conan_paths.cmake script is included by MyRepository project when CMake (or Conan’s CMake build helper) is configured, by defining the following variable beforehand:

CMAKE_PROJECT_MyRepository_INCLUDE=.../conan_paths.cmake

This inclusion allows the canonical invocations of find_package() to correctly find any Conan-retrieved packages.

Yet, further steps are still needed to actually translate the CONAN_ prefixed variables into variables understood by CMake. As discussed above, the plain cmake generator is outputting a file already providing the necessary logic (among other things).

{Sonat} follows a pragmatic approach, invoking both Conan generators as seen in the recipe, and introducing an additional CMake script to glue them together:

conan/customconan.cmake
# Cannot be a function: some invoked macro modify global variables
macro(conan_handle_compiler_settings)
    include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)

    if(CONAN_EXPORTED)
        conan_message(STATUS "Conan: called by CMake conan helper")
    endif()

    if(CONAN_IN_LOCAL_CACHE)
        conan_message(STATUS "Conan: called inside local cache")
    endif()

    check_compiler_version()
    conan_set_std()
    conan_set_libcxx()
    conan_set_vs_runtime()
endmacro()

include(${CMAKE_BINARY_DIR}/conan_paths.cmake)
conan_handle_compiler_settings()

The Conan recipe itself must also be edited in order to include this file. It is achieved by pointing the CMake variable CMAKE_PROJECT_<name>_INCLUDE to the file above:

conan/conanfile.py
from conans import ConanFile, CMake, tools

from os import path

    ...

    def _configure_cmake(self):
        cmake = CMake(self)
        cmake.definitions["CMAKE_PROJECT_MyRepository_INCLUDE"] = \
            path.join(self.source_folder, "conan", "customconan.cmake")
        cmake.definitions["BUILD_tests"] = self.options.build_tests
        cmake.configure()
        return cmake

    ...
Friction point: Waiting for Conan generation of toolchain files

Working around the presented drawbacks only grew the infrastructure code in each repository even larger.
There is currently an issue tracking Conan Build toolchain POC. It could be beneficial to consider whether the subset of conan_basic_setup() invoked in this customconan.cmake could fit in such a toolchain file, alleviating the need for this custom file.

Friction point: Upstream dependencies duplication

The current approach uses distinct CMake and Conan tools. This separation offers many benefits, yet there is an important overlap when it comes to upstream dependencies:

  • CMake scripts (the leaves CMakeLists.txt) find and explicitly register dependencies in the build specification for each component.

  • Conan retrieves and reconciliates dependencies in a complete dependency graph, based on an explicit list of all dependencies for the repository.

There is a form of repetition here, bringing the potential problem usually associated with duplication. For example, if the only component using a given upstream dependencies gets rid of this dependency, the repository as a whole does not depend on this upstream anymore. Yet, there is a risk to forget to remove this same dependency from the Conan recipe.

Friction point: Consuming {Sonat} packages from other Conan generators.

{Sonat} relies on the convenient CMake system of exported targets to ensure propagation of usage requirements. This works consistently without any extra effort, as longs as all downstream(s) are consuming CMake targets. This is the recommended approach (otherwise, read below)

Conan also offers a mechanism to specify a package usage requirements, via cpp_info to be populated in package_info(). When this attribute is correctly configured, the package can be consumed via other Conan generators. For the repository in this guide, it would at least require to list individual include paths for the library components (include/alpha and include/beta), since {Sonat} duplicates the component folder exactly for this reason (not being able to access separate components from a common include path). There might also be compiler flags, etc.

Unfortunately, this would raise two problems:

Duplication of information

{Sonat} already defines all usage requirements at CMake level, having to maintain a second source of truth might lead to discrepancies and maintenance complications.

Granularity mismatch for multi-component projects

CMake is defining usage requirements per-target, which usually means per-component in {Sonat}. Yet, the cpp_info configuration is unaware of such component granularity, and {Sonat} single recipe approach defines the Conan requirements globally, at the repository level.

Testing the recipe

A Conan recipe is yet another piece of code versioned in your repository, and it should be treated as such. It should notably be tested.

ℹ️
The scope of this test is not to validate the fitness of the business code provided by the repository, but to validate that a Conan recipe produces conformant and usable packages.

Conan tools provide facilities to run such test on recipe, usually via a test_package folder living next to the actual conanfile.py recipe. {Sonat} follows this convention, and could even rely on the default test_folder generated by conan new -t.

Such folder usually consists of 3 files:

  • conan/test_package/example.cpp

  • conan/test_package/CMakeLists.txt

  • conan/test_package/conanfile.py

test_package C++ consumer
conan/test_package/example.cpp
#include <alpha/accumulate.h>
#include <alpha/sum.h>

#include <cstdlib>

int main()
{
    myns::accumulate(3);
    myns::sum(myns::sum(3, 2), 1);
    return EXIT_SUCCESS;
}

Includes headers from component(s) provided by the repository, and use some symbols they define. This ensure the include paths are correctly set, and the ability to link the symbols.

test_package CMake project
conan/test_package/CMakeLists.txt
cmake_minimum_required(VERSION 2.8.12)
project(PackageTest CXX)

set(CMAKE_RUNTIME_OUTPUT_DIRECTORY $<1:${CMAKE_CURRENT_BINARY_DIR}>)

find_package(MyRepository REQUIRED COMPONENTS alpha)

add_executable(example example.cpp)
target_link_libraries(example myrepo::alpha)

Simple CMake project, defining the example target to compile the above example.cpp file. It also finds the component(s) used by the code, and mark them as upstream dependencies for the target.

ℹ️
CMAKE_RUNTIME_OUTPUT_DIRECTORY is re-defined to its default value, but via a dummy generator expression. This way, multi-configurations generators do not append a per-configuration subdirectory.
test_package recipe
conan/test_package/conanfile.py
import os

from conans import ConanFile, CMake, tools


class MyRepositoryTestConan(ConanFile):
    settings = "os", "compiler", "build_type", "arch"
    generators = "cmake_paths", "cmake"

    build_requires = "cmake_installer/3.15.7"

    def build(self):
        cmake = CMake(self)
        cmake.definitions["CMAKE_PROJECT_PackageTest_INCLUDE"] = "../customconan.cmake"
        cmake.configure()
        cmake.build()

    def imports(self):
        self.copy("*.dll", dst="bin", src="bin")
        self.copy("*.dylib*", dst="bin", src="lib")
        self.copy('*.so*', dst='bin', src='lib')

    def test(self):
        if not tools.cross_building(self.settings):
            self.run(".%sexample" % os.sep)

Simple recipe, it builds the test folder via the above CMakesLists.txt file. No need to explicitly state requires of the tested package, Conan will automatically inject it.

🔥
The testing recipe could use the base cmake generator only, as long as this friction point is handled by the tested recipe. Sticking to {Sonat}, this testing recipe prefers to use conan/customconan.cmake in a similar manner to the tested recipe.

Usage

Affinity with different workflows

{Sonat} is a process addressing the infrastructure aspects of code projects. Its ability to cater for different workflows is thus an important aspect of it applicability.
Getting more general than cross-tool and cross-platform qualities, it is possible to establish an coarse spectrum of workflows based on usages:

  • Development

  • Command line user

  • General public/end-user (customer, in the marketing sense)

This is a coarse outline, hopefully introducing the most usual situations. The following usage recommendations should easily adapt to situations in-between.

Prerequisites

A recipe is self-contained via its code and build dependencies.
Thanks to that, using {Sonat} projects might only require:

ℹ️
CMake is a provided by Conan as a build requirement, but an explicit installation of CMake will be required in situations where a {Sonat} project is built out of Conan.

Development

During development, the build responsibility is taken out of Conan, to place more control into the hand of the developers. It additionally requires an appropriate version of CMake to be available on the system.

Development in isolation

This scenario refers to developing the component in isolation: all upstream dependencies are available as installed Conan packages (by opposition to being accessed from their CMake install or build tree).

This scenario is naturally addressed by the system described above, and illustrates well how Conan can complement development environments (making it a one line call to satisfy all dependencies) without staying in the way (once the dependencies are retrieved, Conan might not be needed anymore).

The canonical steps for a beneficialproject repository:
# Clone the project and move to the folder for out-of-source build
git clone --recurse-submodules ${repository}/beneficialproject.git
mkdir beneficialproject/build
cd beneficialproject/build

# Install all dependencies in the dependency graph
# Will also generate files for generators listed in the recipe
conan install ../conan

# Generate project files
cmake -DCMAKE_PROJECT_BeneficialProject_INCLUDE=conan/customconan.cmake \
      -DCMAKE_INSTALL_PREFIX=${local_sdk_folder}/beneficialproject \
      ..

From here on, it is possible to forget about Conan. The developer can concentrate on their usual "edit-build-debug" cycle (e.g. via their familiar IDE, if IDE project files were generated by CMake).

Alternatively, to keep it build-system agnostic, CMake might also be invoked to drive the builds. This might notably be useful for automation:

cmake --build . [--target ...]

Publishing the recipe

Publishing the recipe is a simple step greatly increasing a project sociability. It makes it trivial for downstream to consume the recipe’s component(s).

To publish a with identifier beneficialproject/1.2.0@company/stable:

# From the repository root
conan create ./conan 1.2.0@company/stable
conan upload [-r my_remote] beneficialproject/1.2.0@company/stable
ℹ️
conan-center is a free public repository, picked by default in the absence of -r option.

Development in multiple-repositories

The isolated scenario can be generalised to illustrate development inside an organisation.

Unless this organisation strictly adhere to the monorepo approach, there might be situations where the development process would imply working on more than one repository at once (Each repository is a different {Sonat} project).

It extends the above section: several independent projects will be cloned and built. The initial motivation to work on several projects is their dependency relationship.
It implies to provide hints to CMake regarding the location of downstream projects which are manually cloned and built. This is required so their find_package calls can locate the local upstream(s), which are out of Conan’s cache.

Assuming a development task implying to work in parallel on the sources of both:

  • beneficialproject as handled above

  • profitableapp, a downstream dependency of beneficialproject

The steps for a profitableapp repository
git clone --recurse-submodules ${repository}/profitableapp.git
mkdir profitableapp/build
cd profitableapp/build

# Install a restricted set of dependencies
conan install ../conan/conanfile-dev.txt

cmake .. -DPROJECT_ProfitableApp_INCLUDE=conan/customconan.cmake \
         -DCMAKE_INSTALL_PREFIX=${local_sdk_folder}/profitableapp
         -DCMAKE_PREFIX_PATH=${local_sdk_folder}

There are two changes:

  1. Conan does not install the full conan/conanfile.py, but a subset file (named conan/conanfile-dev.txt in this example). This file only needs to specify the cmake and cmake_paths generators, and to list requirements excluding all requirements that are manually provided out of Conan (i.e. excluding beneficialproject in this example).

  2. CMake variable CMAKE_PREFIX_PATH points to the local installation folder, where the beneficialproject 's install target would deploy beneficialproject CMake package. Thanks to the adopted folder structure, this single hint is enough to find any {Sonat} conformant package installed under this prefix.

This logic can be extended to explicitly build an arbitrary number of dependencies instead of relying on Conan to provide them.

🔥
install tree vs. build tree

By setting CMAKE_PREFIX_PATH to the install folder, the manually built upstream dependencies are found in their install tree. This means that, before any edition applied to them becomes available to downstream(s), they must first invoke their install target (e.g. cmake --build . --target install)

In certain situations, it might be preferable to find the manually built upstream dependencies in their build tree. In this case, the single value provided to CMAKE_PREFIX_PATH should be replaced with distinct definitions for each upstream. In the current example, it would be replaced with
-DBeneficialProject_DIR=${build_dir}. Unless a dependency found in its build tree is a header-only library, it should still be rebuilt (but not necessarily installed anymore) in order for changes to propagate downstream.

Friction point: Distinct dependency graphs

One feature of a package manager is to reconciliate the dependencies version when several paths in the DAG specify the same recipe in but in different versions. With this approach, several independent DAGs are generated (one per manually built repository), losing this important feature (because it does not apply across independent graphs boundaries).

Friction point: The quest for repository component-fluidity

There is a bigger picture here, a potential to ease the developers experience with a more natural continuity between isolated and modularised developments.

It is a stated goal to allow for the anyrepo situation. As such, allowing any level of repo/module granularity - without getting in the way of developers - brings specific challenges.
In particular {Sonat} should not limit which projects are provided by Conan, and which projects are built locally. The section above propose a pragmatic approach to achieve some level of freedom with the current solution, yet it is far from ideal:

  • It breaks dependency resolution

  • A distinct conanfile-xxx.txt file needs to be created for each combination of locally built upstream dependencies

  • Finding upstream dependencies in their build tree require a distinct explicit CMake-hint per upstream repository

Properly addressing those issues are challenging, and might require additional tool support. Yet the potential gains for developers might be equally important.


From a functional stand-point, an ideal solution might look something like:

  1. A list of components to work on (edit) is specified

  2. A process is invoked to:

    • map the components' repositories to their corresponding recipe identifiers in requirements lists

    • solve the DAG for non-local dependencies and retrieve them in Conan local cache

    • ensure local availability of listed repositories (for local edition and build)

    • [optional] setup out-of-source builds for those local projects

There might still be limitations regarding which repositories are allowed to be built locally (maybe preventing any upstream dependency of a non-local dependency to be built locally).

Command-line user

This workflow cover the case of consuming a {Sonat} project in a situation where a shell interface is convenient. It covers different use cases, for example:

  • Automated processes, such as Continuous Integration

  • Situations where all software must be built internally (security policy, compilation options, etc.)

  • Consuming the project when there are no prebuilt binaries for the target environment (or not at all)

  • B2B customer for middleware and general libraries

Assuming a project recipe was made available on an accessible Conan remote, with the identifier beneficialproject/1.2.0@provider/stable, consuming it is as simple as:

conan install beneficialproject/1.2.0@provider/stable

This command will either:

  • retrieve a package for this recipe when one is available with the current combination of settings and options

  • or build it if this combination is not available (thanks to build_policy = "missing" in the project recipe).

Doing it recursively, consuming all the dependencies along the upstream graph

Generators

The command above ensures that once it successfully completed, a package for beneficialproject is available in the local Conan cache. The remaining question is how the consumer expects to use the project, which is notably constrained by the nature of the project.

Library

If the user expects to build upon a library provided by beneficialproject, an appropriate generator from the many supported build environments must be chosen.

For example, to generate a .props file to be imported in a Visual Studio project:

conan install beneficialproject/1.2.0@provider/stable -g visual_studio
ℹ️
Of course, such use-case would benefit from applying {Sonat} to the downstream project relying on beneficialproject. Nevertheless, this example illustrates the potential extension of {Sonat} projects toward downstreams in un-controlled environments. Be aware that it would require to populate the upstream’s Conan recipe cpp_info, which {Sonat} currently does not.
Tool

If the user is interested in a tool (executable) provided by beneficialproject, then other generators, such a virtual environments, will address this situation.

For example, to obtain and invoke the usefulgenerator tool in beneficialproject on a GNU/Linux flavoured environment:

conan install beneficialproject/1.2.0@provider/stable -g virtualrunenv
source activate_run.sh
usefulgenerator --version
source deactivate_run.sh
ℹ️
No global system change are taking place. To install a tool user or system wide would more closely match the end-user scenario.
Friction point

The activation and deactivation steps are system specific here. It might be beneficial to have an abstraction (in the same vein as the abstraction over build invocation offered by cmake --build …​), notably for matrix based CI processes.

No published recipe

In a situation where the recipe is not available in a known remote, it is still possible to use Conan to drive the build process from the project sources. It is a matter of exporting the recipe from the sources to the local Conan cache, then using it to assemble the package.

Given access to the repository content, building the project amounts to:

# From the repository root
conan create ./conan ${VERSION}@user/stable
ℹ️
Such situation is not ideal, notably making it a consumer responsibility to assign recipe identifiers, such as version, user and channel above.
ℹ️
In the command above, the package name is deduced from the recipe name. If an explicit version is specified in the recipe meta-information, it is also be possible to omit it on the conan create command line.

General public

{Sonat} is not addressed at end-users directly: the process relies on qualified tools which are not expected to be available on general public environments. Yet, adopting such a process which facilitates CI/CD makes it easier to cater for customers. It offers a good infrastructure on top of which packaging logic can be implemented, then mechanically invoked.

TODO

Might be illustrated with adding custom CMake targets to produce Windows msi installers, or macOS app bundles.

Annexes

Naming conventions

Naming and syntax convetions are not the intended scope of {Sonat}.
Yet, it strongly encourages to follow existing convention when they exist:

  • Conan package reference are entirely lower-case (e.g. beneficialproject/1.2.0@company/stable)

CMake package with multiple components

Additionnally, there is a situation where using a predefined naming convention can make development more flexible. When a repository is a multi-component CMake package, one of the components might be named like the repository (e.g. if there is one central component, with some peripheral helper components).

To allow for this scenario without introducing collisions at the CMake find_package() level, {Sonat} makes the following recommendations:

  • the root project() name always start with an uppercase letter.

  • each CMake target (which will map to components in the package) always start with a lowercase letter.

This results in find_package() calls looking like:

find_package(Math COMPONENTS bignumber math)

And the CMake path variable used to find a dependency in its build tree has the form:

Math_DIR=${path_to_math_buildtree}

Automated QA

TODO