Scope Brainstorming

[bretbrownjr]

It occurs to me that the initial challenge with this project will be defining scope. There are a wide range of needs and technologies in packaging that can and eventually should be discussed. We'll need to provide ways to organize all that information and carve out a reasonably sized initial subset of all that to create a minimum viable product for packaging standards.

I'm open to better ideas here, but perhaps we should start with fairly simple mechanisms and refine our work habits and culture from there.

To provide focus, I think we should start by describing relevant use cases and requirements in text.

What would I like to see? An ideal description would be clear enough to be objectively achieved or not by a given tool or technology. Merits aside, I like this as a requirement because it's clearly achieved or not:

Metadata files should have a specific file extension to make it trivial to discover them in a package or installed on a filesystem.

In contrast, I think this needs more refinement:

Metadata files need to be trivially discoverable.

Where would I like to see these descriptions? Replies in this thread should work fine. We'll treat this as a thread for brainstorming. If certain ideas need extra work, we can move them into a separate discussion or issue.

Eventually I would like to see these requirements captured in design documents and even tests in this repository. Given we don't have either yet, I'd be happy just having a place to capture thoughts initially. Once we have a certain amount of mass here, I think converting the requirements into some sort of structured design document will be more straightforward. I could see incremental issues and pull requests carrying the work forward from there.

[bretbrownjr]

Banging out some requirements:

To clarify how to properly consume prebuilt libraries, libraries should be described with metadata files colocated with those libraries.

Note that I'm not bringing up packages as such. We want this to work in contexts where a package manager is involved, but also in contexts where the origin of files on disk is not obvious (i.e., libraries installed under the user home directory).

EDIT: Wording

[grafikrobot]

I think this is the key minimal requirement we need. This specific case came up in the context of Boost and B2 early today. Today Boost provides a level of Cmake comparability when building by generating cmake config files for the libraries it builds. Thy are generated from custom B2 code (written by @pdimov). While that's great for Cmake it doesn't help anyone else (particularly package managers). And it means shoving code to satisfy another build system. In an ideal world B2 would produce the equivalent information, and more, in a representation that either Cmake can consume, or have a different tool use the B2 output and it generates the cmake config files. That is this kind of pipeline:

B2 ==> libX.so+libX.info ==> info2cmake ==> libX.so+XConfig.cmake ==> Cmake

Depending on the package manager the information generated with that use case could likely be enough for them also.

[bretbrownjr]

Agreed. At C++Now 2022 @druoso and I more or sketched out an analogous transition plan that allows to generate to or from pkg-config metadata while the new metadata format gains traction. It is covered around this portion of this video: https://youtu.be/VCrLAmJWZFQ?t=3108

Ideally the utility of pkg-config, CMake config files, etc., would decrease as more tools support this metadata file format.

[bretbrownjr]

Following on the above:

The location of a prebuilt library should be described in its metadata file.

I suppose this would allow for complete paths or a combination of a name and a search path (that is, -L and -l on GNU-like toolchains).

[grafikrobot]

Such a requirement would need to walk the challenging minefield of library loading, path resolution, RPATH, and the varied static and dynamic linking variables in OSes and linkers. I've had nightmares thinking on dealing with this int he past.

[bretbrownjr]

Note that I'm focusing on the build-time installation of a library in this case. Probably the requirement could be rephrased to make that more clear. I'm open to suggestions on how to clarify that.

But agreed. I think it's still reasonable, for package management use cases if nothing else, for metadata files to specifically point out the binary or binaries that they are describing.

We do need to figure out how to support static and shared libraries, especially when both are installed at the same time. That probably deserves it own discussion. And we do need to figure out what to do about runtime search path requirements, especially as they're something many existing approaches do not support well.

[bretbrownjr]

So that metadata files are relocatable, locations in metadata files should be described in relative paths.

I'm flexible on whether the paths are relative to the metadata file like $ORIGIN/../../include or if they're relative to an installation prefix like $prefix/include. If there's interest, we can kick out a separate discussion on the options there. Possibly both forms could be supported. Possibly both forms must be supported.

But I expect rewriting /some/long/prefix in these files at install time depending on the workflow and system will be more brittle than we'd be happy with.

[grafikrobot]

I would stay away from "above the root" relative files. And hence stick to relative to some install prefix. I would argue for some form of remap-able prefix. That is have whatever prefix name the generator wants to use. And include a map to from name to install dir. That way as the library gets moved around only the map gets edited and not a bunch of file paths. Having such a map also communicates what different directories are part of the whole library.

[bretbrownjr]

Works for me. Everyone note that some packaging systems provide different installation prefixes for each dependency, so approaches that assume all installs are colocated wouldn't work given, at least not without some redesigns and migrations for existing ecosystems.

[bretbrownjr]

A library metadata file will include ordered paths to zero or more directories to search for headers provided by that library.

My preference would be to require explicitly that zero include directories be enumerated in the case of a library with no header files, just to give the affirmative demonstration that that is the case.

[grafikrobot]

I don't see this. If we want to follow consistency as a goal we will end up with the data enumerating the maximal set of fields always. Which will present a problem when the data format changes and new fields are added. One would need to go back and fill in empty values everywhere.

[bretbrownjr]

I was shooting for clarity and not consistency. I'd rather it be clear that a project ships no headers by design than to guess whether a missing include_dirs field is intentional or a bug.

[GabrielDosReis]

I was shooting for clarity and not consistency.

I agree. I know it is fashionable to be "consistently wrong", but I would rather for us drive for clarity first, then deal with the possible area of consistency second. Sometimes, not being consistent is a good way of drawing attention to the odd balls.

[bretbrownjr]

A library metadata file will include an unordered list of zero or more libraries it depends on.

Similar to include directories, I would like a set of zero dependencies when that is the case.

[grafikrobot]

That would be consistent. And shows the future problem of the zero-data rules :-)

[bretbrownjr]

A library metadata file will have an extension-free basename that uniquely identifies the library. This name will be used by other metadata files to declare a dependency on the library in question. It may also be used by other systems such as packaging systems and build systems to likewise identify the library.

This one has a lot of implications, but we'll need some sort of requirements about identity to be fleshed out. This is a starting point for discussion. Someone feel free to branch of a dedicated discussion about what to do about identities.

I'm OK with "just use the basename of the file" as an approach because, in my experiences at least, the 'name' of a library (i.e., what would be passed to -l in a GNU-like toolchain) implies a coherent namespace of libraries already. There are some edge cases, of course, like zlib and the blas libraries. But it seems like a really good starting point for discussion.

[bretbrownjr]

A library metadata file will be encoded in the TOML file format.

Choosing this requirement based on a straw poll we took in the hallway at CppCon 2022. The goal here is to provide a starting point for a serious proof-of-concept. It's possible the file format will change or more options for file formats will need to be added as we gain more experience.

[andrey-zherikov]

Sounds about right. I'll add that I'd like ecosystems that only provide a C compiler to be able to reasonably able to support this file format if they so desired, so picking either an existing popular format (JSON and TOML work) or an especially trivial-to-parse language with a published grammar would be helpful.

Have you considered YAML?

[DarkArc]

Sounds about right. I'll add that I'd like ecosystems that only provide a C compiler to be able to reasonably able to support this file format if they so desired, so picking either an existing popular format (JSON and TOML work) or an especially trivial-to-parse language with a published grammar would be helpful.

Have you considered YAML?

I'd advocate against YAML due to the complexity of its spec.

I think JSON is fine (it can be pretty printed and become fairly readable). There are issues with using JSON as a configuration due to the lack of comments. If we anticipate a need for comments, I think TOML is worth the "two formats", assuming we don't, I'd just stick with JSON as @grafikrobot suggested.

[DarkArc]

Even if a file is program generated, it doesn't mean that the generated file can't contain clarifying comments. These comments are also useful in the description of the file format in documentation (this allows simple copy-pasting and testing), for example:

- dependencies:
  # any version of package1
  - package1
  # restrict version of package2
  - package2:
    - version:
      - ">= 2.0"

I do think the presence of comments should be driven by how much of this we anticipate a user driving vs a program. For instance, if a workflow occurs where a machine generates a TOML file, a user adds some comments, and then runs the tool to "update" the TOML file, it could become a significant concern (and scope bloat) for the tool doing the update to "try" to preserve comments (doing a quick pass, this is far from a "solved" problem toml-lang/toml#284).

The "description" strategy mentioned by @grafikrobot dodges some of those problems (provided "description" is present as an optional field in the right places). Still, (from a user perspective -- at least IMO) it's significantly harder to work with a "description" string than a series of simple # denoted comment lines.

[alexreinking]

I'll add a 👍 to the pro-TOML, anti-YAML stance... I hate editing CMakePresets.json files. It's tedious and horrible in large part because JSON is only slightly more fun to edit than XML, but also because CMake provides close to zero feedback when the input is in any way malformed.

At the same time, I never feel totally confident anticipating what YAML will do with my whitespace or how its weird pointers feature works. The spec is highly complex, and several implementations are known to be buggy and have security vulnerabilities.

TOML is the best widely used alternative, imho.

[GabrielDosReis]

Even if a file is program generated, it doesn't mean that the generated file can't contain clarifying comments.

Ability to add comments to these MANIFEST is fundamental - I don't see a path forward without them.

[bretbrownjr]

A library metadata file will enumerate an unordered set of preprocessor definitions that must be set by consuming projects.

Of course, generally speaking, moving #define statements to relevant header files will be more portable when possible. We should consider elaborating on "this exists, but only use it when you don't have other options" in an FAQ or some notes on the document.

[grafikrobot]

A library metadata file will include the ABI it's built for and that must be matched by consuming projects.

Yes, I know, I mentioned the elephant in the room. But this is a key aspect for building Boost. As it generates a handful of ABIs in one build. The big question is... Is this something for now or for down the road? Can we live without this initially?

[grafikrobot]

A relevant proposal on this topic is D1864R0 (cplusplus/papers#813).

[bretbrownjr]

I don't know what is meant by "ABI" here. Unless you mean something specific, we probably need to break that down into more specific properties.

And I'll note that one approach, at least for an initial proof-of-concept release, is to clarify that this project targets a coherent ecosystem to be provided via the dependency manager (some combination of engineers, package manager, build system, monorepo, integration build, etc.). That's not to say we don't care about ABI! I think the metadata file format would make the job of detecting and fixing ABI nonsense actually possible in cases where it's just an exercise in grit, bravery, and obsessiveness now.

Also, I think adding ABI details and/or identifiers in future releases seems doable as we develop experience.

[bretbrownjr]

To facilitate requirement introspection and package validation, a library metadata file will inventory headers and module interfaces that are provided with the library.

This is actually a new one in most (all?) contexts. But I was just trying to design a link-what-you-include tool and I'm coming up against the fact that there's no canonical way to look at a #include and know what library is missing from the link line (or CMake link libraries in my case).

It should also prove handy when sorting out what headers go with what in an omnibus include directory, especially when you want higher order tools like IDEs describing libraries as such. See above about link-what-you-include.

Even in the case of header-only libraries, having the headers clearly associated with the library name will allow systems to identify needed build flags, dependencies, etc.