Implemented `build.build-dir` config option #15104

ranger-ross · 2025-01-26T08:21:37Z

(still a work in progress)

What does this PR try to resolve?

This PR adds a new build.build-dir configuration option that was proposed in #14125 (comment)

This new config option allows the user to specify a directory where intermediate build artifacts should be stored.
I have shortened it to just build-dir from target-build-dir, although naming is still subject to change.

What is a final artifact vs an intermediate build artifact

Final artifacts

These are the files that end users will typically want to access directly or indirectly via a third party tool.

binary executable for bin crates
binary executable for examples
.crate files output from cargo package
depinfo files (.d files) for third party build-system integrations (see https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/fingerprint/mod.rs#L194)
cargo doc output (html/css/js/etc)

Intermediate build artifact

These are files that are used internally by Cargo/Rustc during the build process

other depinfo files (generated by rustc, fingerprint, etc. See https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/fingerprint/mod.rs#L164)
rlibs and debug info from dependencies
build script OUT_DIR
output from proc macros (previously stored in target/build)
incremental build output from rustc
fingerprint files used by Cargo for rebuild detection
"pre-uplifted" binary executables. (ie. bins for examples that contain the hash in the name)
CARGO_TARGET_TMPDIR files (see rational for this here)

How should we test and review this PR?

This is probably best reviewed commit by commit. I documented each commit.
I tied to follow the atomic commits recommendation in the Cargo contributors guide, but I split out some commits for ease of review. (Otherwise I think this would have ended up being 1 or 2 large commits 😅)

Testing Strategy

We have the existing Cargo testsuite to be sure we do not introduce regressions.
- I have also run the testsuite locally with the cli flag remove to verify all tests pass with the default build dir (which falls back to the target dir)
For testing thus far, I have been using small hello world project with a few dependencies like rand to verify files are being output to the correct directory.
When this PR is closer to merging, I plan to test with some larger projects with more dependencies, build scripts, ect.
Other testing recommendations are welcome 🙇

Questions

What is the expected behavior of cargo clean?
- At the time of writing it only cleans target and does not impact the build-dir but this is easily changable.
- Answer: See Implemented build.build-dir config option #15104 (comment)
When using cargo package are was expecting just the .crate file to be in target while all other output be stored in build.build-dir? Not sure if we consider things like Cargo.toml, Cargo.toml.orig, .cargo_vcs_info.json part of the user facing interface.
- Current consensus is that only .crate is considered a final artifact
Where should cargo doc output go? HTML/JS for many crates can be pretty large. Moving to the build-dir would help reduce duplication if we find the that acceptable. For cargo doc --open this is not a problem but may be problematic for other use cases?
- Answer: Implemented build.build-dir config option #15104 (comment)
Are bins generated from benches considered final artifacts?
- Since bins from examples are considered final artifacts, it seems natural that benches should also be considered final artifacts. However, unlike examples the benches bins are stored in target/{profile}/deps instead of a dedicated directory (like target/{profile}/examples). We could move them into a dedicated directory (target/{profile}/benches) but that mean would also be changing the structure of the target directory which feels out of scope for this change. If we decide that benches are final artifacts, it would probably be better to consider that changes as part of --artifact-dir (nee --out-dir) Tracking Issue #6790

TODO

rustbot · 2025-01-26T08:21:42Z

r? @ehuss

rustbot has assigned @ehuss.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

weihanglo

Just want to make sure I didn't miss something. From what I can tell these directories/files have been removed right?

target/<profile>/.metabuild
target/<profile>/.fingerprint
target/<profile>/deps
target/<profile>/incremental
target/<profile>/build
target/.cargo-lock
target/tmp
target/.rustc_info.json

weihanglo · 2025-01-26T17:14:46Z

tests/testsuite/build_dir.rs

We usually add some tests verifying a nightly flag is really gated.

weihanglo · 2025-01-26T17:31:14Z

src/cargo/core/compiler/build_runner/mod.rs

@@ -290,7 +290,9 @@ impl<'a, 'gctx> BuildRunner<'a, 'gctx> {
                });
            }

-            super::output_depinfo(&mut self, unit)?;
+            if !self.bcx.gctx.cli_unstable().build_dir {
+                super::output_depinfo(&mut self, unit)?;


Why do we stop generating this when build-dir is enabled?

Sorry, actually I believe this is a mistake on my part 😅

My original assumption was that the /target/debug/{project-name}.d file was the same file as /target/debug/deps/{project-name}-{hash}.d, so I was trying to avoid copying it into target.
But now that I compared the files they appear to be different so it looks like that was a bad assumption.

I believe this should probably be considered an intermediate build artifact so it should probably be moved to the build-dir location.

Modifying this file in particular was a bit tricky as it was not just replacing .target_dir() with .build_dir().
I think I will probably need to go to where the OutputFile is created and modify the hardlink to be the new build-dir path.

Given this comment I am going to revert this change

ranger-ross · 2025-01-27T12:07:57Z

Just want to make sure I didn't miss something. From what I can tell these directories/files have been removed right?

target/<profile>/.metabuild

target/<profile>/.fingerprint

target/<profile>/deps

target/<profile>/incremental

target/<profile>/build

target/.cargo-lock

target/tmp

target/.rustc_info.json

Yes, that with the exception of target/.cargo-lock.
I think we will still want this cargo lock for backwards compatibility with previous versions of cargo.

Ideally in the longer term it can be removed in favor of fine grain locking like #4282

So a typical target directory will be something like

target
├── CACHEDIR.TAG
└── debug
    ├── .cargo-lock
    ├── examples
    └── hello_world // (the binary)

epage · 2025-01-27T16:47:53Z

src/doc/src/reference/unstable.md

+## build-dir
+* Original Issue: [#14125](https://github.com/rust-lang/cargo/issues/14125)
+* Tracking Issue: [#14125](https://github.com/rust-lang/cargo/issues/14125)
+
+This feature allows you to specify the directory where intermediate build artifacts will be stored.
+The final artifacts will also be contained in this directory but will be hardlinked into the artifact directory (usually `target`)
+
+This can be specified in `.cargo/config.toml` files using the `build.build-dir`.
+
+Example:
+
+```toml
+[build]
+build-dir = "out"
+```


Could you include documentation in this that mirrors what would be in https://doc.rust-lang.org/cargo/reference/config.html

epage · 2025-01-27T16:49:29Z

src/cargo/core/compiler/layout.rs

+    /// Same as `_lock` but for the build directory.
+    ///
+    /// NOTE: `_lock` and `_build_lock` can eventually be merged once #14125 is stablized.
+    /// Will be `None` when the build-directory and target-directory are the same path as we cannot
+    /// lock the same path twice.


How can we merge these locks?

In the short term, we'll have

target-dir

build-dir

which both need locking

In the long term, we'll have

artifact-dir

build-dir

which will both need locking

Sorry, you are right. Not quiet sure what I was thinking when I wrote this comment.

epage · 2025-01-27T16:54:58Z

tests/testsuite/build_dir.rs

Also, we generally add (passing) tests before the change so the affect of the change is obvious.

This is more straightforward when fixing a bug. In a case like this, a common option is to write the tests without the new feature or using something similar ( like target-dir) and then in the commit where the feature is added, you switch over the tests.

epage · 2025-01-27T16:55:49Z

tests/testsuite/build_dir.rs

+    assert!(p.root().join("target/debug/foo").is_file());
+}
+
+fn assert_build_dir(path: PathBuf, profile: &str, is_build_dir: bool) {


#[track_caller]?

epage · 2025-01-27T16:59:24Z

What is the expected behavior of cargo clean?

It should clean the build dir

epage · 2025-01-27T17:00:06Z

When using cargo package are was expecting just the .crate file to be in target while all other output be stored in build.build-dir? Not sure if we consider things like Cargo.toml, Cargo.toml.orig, .cargo_vcs_info.json part of the user facing interface.

imo The artifact for cargo package is the .crate. Everything else is part of the "build" process.

epage · 2025-01-27T17:01:18Z

Can we call out explicitly what our testing strategy is?

We likely should also explicitly document in the PR what is considered an artifact and what is a build output and make sure we have tests for these.

ranger-ross · 2025-01-28T14:50:22Z

One other question that came to my mind was the output of cargo doc. HTML/JS for many crates can be pretty large. Moving to the build-dir would help reduce duplication if we find the that acceptable. For cargo doc --open this is not a problem but may be problematic for other use cases?

Perhaps symlinking the index.html from the build dir into target could be an option if we care about keeping an entry point.

ranger-ross · 2025-01-28T14:53:27Z

Can we call out explicitly what our testing strategy is?

We likely should also explicitly document in the PR what is considered an artifact and what is a build output and make sure we have tests for these.

@epage sure, I updated the PR description but let me know if I missed anything.

epage · 2025-01-28T18:27:34Z

One other question that came to my mind was the output of cargo doc. HTML/JS for many crates can be pretty large. Moving to the build-dir would help reduce duplication if we find the that acceptable. For cargo doc --open this is not a problem but may be problematic for other use cases?

imo cargo docs output is an artifact that people will want access to. I suspect it'd be a breaking change to move it out of target-dir.

epage · 2025-01-28T18:31:26Z

depinfo files (.d files)

There are multiple types of depinfo files. I suspect the ones next to final artifacts are also considered final artifacts, see https://doc.rust-lang.org/cargo/reference/build-cache.html#dep-info-files

epage · 2025-01-28T18:35:51Z

FYI I added to the PR description a couple more intermediate artifacts

rlibs and debug info from dependencies

build script OUT_DIR

When are workspace member rlibs considered final artifacts? We're putting them in target/<profile> at times, so I take it that has already been answered.

epage · 2025-01-28T18:37:23Z

Can we call out explicitly what our testing strategy is?
We likely should also explicitly document in the PR what is considered an artifact and what is a build output and make sure we have tests for these.

@epage sure, I updated the PR description but let me know if I missed anything.

Sorry if I wasn't clear but my discussion of testing strategy was in the context of tracking the categorization of artifacts. I'm not saying we have to exhaustively test it but we should think about it and document the choice made and why.

These ares are in preparation to split target-dir into artifact-dir and build-dir

This is in preparation for splitting the intermediate build artifacts from the `target` directory.

This commit adds a `build_dir` option to the `build` table in `config.toml` and adds the equivalent field to `Workspace` and `GlobalContext`.

This commits implements the seperation of the intermidate artifact directory (called "build directory") from the target directory. (see rust-lang#14125)

This commit add the ability to use predefined variables when specifying the `build.build-dir` in the Cargo configuration. The currently supported template variables are: * `{workspace-root}` which will expand to the Cargo workspace root * `{cargo-cache}` which will expand to `CARGO_HOME` but will likely change in the future. * `{workspace-manifest-path-hash}` which will expand to a short hash of the workspace manifest's path.

ranger-ross · 2025-02-02T10:10:08Z

I spent some more time this weekend making some progress on this PR.

Here are my updates

Fixed some issues to make cargo clean & cargo package work with build.build-dir
Added templating support for build.build-dir
Cleanup the commits a bit and tried to follow the request in this comment regarding tests

I updated the PR description with more details as well. notably:

Added another question I would like input form the Cargo team on.
- Are bins generated from benches considered final artifacts? (see PR desc for more details)
Updated the TODO list with a more detailed list of tasks before marking this PR as ready for review.
Cleaned up the "What is a final artifact vs an intermediate build artifact" section
- Also added the following file types to the list of intermediate build artifacts
  - "pre-uplifted" binary executables. (ie. bins for examples that contain the hash in the name)
  - CARGO_TARGET_TMPDIR files (see rational for this here)

rustbot assigned ehuss Jan 26, 2025

ranger-ross changed the title ~~Added build-directory unstable feature flag~~ Implemented build.build-dir config option Jan 26, 2025

weihanglo reviewed Jan 26, 2025

View reviewed changes

epage reviewed Jan 27, 2025

View reviewed changes

epage mentioned this pull request Jan 27, 2025

Redefine CARGO_TARGET_DIR to be only an artifacts directory #14125

Open

4 tasks

ranger-ross force-pushed the target-build-dir branch from 6dec044 to 9ee678b Compare February 1, 2025 03:51

rustbot added the Command-clean label Feb 1, 2025

ranger-ross force-pushed the target-build-dir branch from 9ee678b to dd06427 Compare February 1, 2025 17:49

ranger-ross added 6 commits February 2, 2025 17:03

Added target-dir tests

4bc2fe8

These ares are in preparation to split target-dir into artifact-dir and build-dir

Added build-dir unstable feature flag

f6d0c3e

Derive PartialEq and Eq for Filesystem

4186230

This is in preparation for splitting the intermediate build artifacts from the `target` directory.

Added build.build_dir configuration option

df6c978

This commit adds a `build_dir` option to the `build` table in `config.toml` and adds the equivalent field to `Workspace` and `GlobalContext`.

Seperated build directory from target directory

6eb89af

This commits implements the seperation of the intermidate artifact directory (called "build directory") from the target directory. (see rust-lang#14125)

ranger-ross force-pushed the target-build-dir branch from dd06427 to f53007f Compare February 2, 2025 08:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implemented `build.build-dir` config option #15104

Implemented `build.build-dir` config option #15104

ranger-ross commented Jan 26, 2025 •

edited

Loading

rustbot commented Jan 26, 2025

weihanglo left a comment

weihanglo Jan 26, 2025

weihanglo Jan 26, 2025

ranger-ross Jan 27, 2025

ranger-ross Feb 1, 2025

ranger-ross commented Jan 27, 2025

epage Jan 27, 2025

epage Jan 27, 2025

ranger-ross Jan 28, 2025

epage Jan 27, 2025

epage Jan 27, 2025

epage commented Jan 27, 2025

epage commented Jan 27, 2025

epage commented Jan 27, 2025

ranger-ross commented Jan 28, 2025

ranger-ross commented Jan 28, 2025

epage commented Jan 28, 2025

epage commented Jan 28, 2025

epage commented Jan 28, 2025

epage commented Jan 28, 2025

ranger-ross commented Feb 2, 2025

Implemented build.build-dir config option #15104

Are you sure you want to change the base?

Implemented build.build-dir config option #15104

Conversation

ranger-ross commented Jan 26, 2025 • edited Loading

What does this PR try to resolve?

What is a final artifact vs an intermediate build artifact

Final artifacts

Intermediate build artifact

How should we test and review this PR?

Testing Strategy

Questions

TODO

rustbot commented Jan 26, 2025

weihanglo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ranger-ross commented Jan 27, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

epage commented Jan 27, 2025

epage commented Jan 27, 2025

epage commented Jan 27, 2025

ranger-ross commented Jan 28, 2025

ranger-ross commented Jan 28, 2025

epage commented Jan 28, 2025

epage commented Jan 28, 2025

epage commented Jan 28, 2025

epage commented Jan 28, 2025

ranger-ross commented Feb 2, 2025

Implemented `build.build-dir` config option #15104

Implemented `build.build-dir` config option #15104

ranger-ross commented Jan 26, 2025 •

edited

Loading