EXPERIMENT: build jujutsu with buck2

[thoughtpolice]

This is an experimental and fully un-tested branch, which attempts to add build rules for building jj with Buck2, AKA BUCK files. The goal of such a technology would be to have a fully unified and integrated system for development of Jujutsu — no matter the build platform or client of the build system — even as the project expands and includes more and more features, more developers, and possibly languages and tools.

Please note that I don't think this is a branch that should in its current state be merged (or ever will? Assuming all goes well?) While I was partially motivated to unify things in workspace.dependencies recently because it would help here, that wasn't really the main goal. Part of it was to see if it could even remotely work at all; I am happily using buck2 for my own (private) Rust projects, so it was also an attempt to see if I could bring some of that magic over. I'm happy to say it seems promising but I've run into a (big) issue. In the meantime, I am publishing an interm report and PR on it.

If something like this is truly needed one day, I assume that's because Jujutsu will be wildly successful beyond our dreams, to the point that this sort of tech is a requirement, for reasons I'll go over. But maybe there's more to it than just that? This is above all an experiment, but I'll elaborate.

Because this kind of wide-reaching impact this kind of thing can have (on project direction, on developers, on support, et cetera) the remainder of this description is a very long novel about why you might be interested in this (or maybe why you would hate it.)

Note: This patch series shouldn't be taken as any kind of endorsement or commitment to its maintenance. If you're not interested, don't worry, it's not going anywhere. It may break, get rewritten, or otherwise behave incorrectly. Do not taunt Happy Fun Ball.

NOTE: Jujutsu has many Google employees who work with or on it, or are otherwise in touch with the development; they'll be familiar with much of what I'm about to discuss thanks to the use of Blaze/Bazel for their own needs, but for the purposes of pedagogy I'll assume the reader isn't familiar at all. Perhaps Bazel users will find something interesting or even learn something new, too. Bazel specifically will be discussed later.

Prologue

In the beginning — literally, since the dawn of time — there was make. make is actually not one program but many, but all variants of make have a similar idea: to represent the compilation of software, or software artifacts, as a directed acyclic graph — where each node represents an "action" to perform that results in outputs, and where most actions (non .PHONY ones) can be cached in the filesystem.

make is a good tool and I have a fond spot for it. However, in practice, it is very difficult to write correct Makefiles that work, at scale, for large software projects, without exceedingly austere requirements on the layout of the project, its language, and its specific design points — and while having something that is portable and easy to understand.

For many years many programming languages, including high level ones, used Make (including my beloved Haskell) to drive software compilation; though at some point around the turn of the millenium, it seems engineers began to move away from this model when dealing with high level languages (C/C++ users continue to wallow in misery to this day). There are some decent reasons for this that motivated it, I believe, when you back up a bit:

• Again, make isn't one thing, but a family of things. There is gmake, bmake, there used to be imake, and there is also the venerable mk from plan 9 (which unlike the others actually had the fortitude to fix one of makes worst sins — its interaction with shell escaping in the body of a rule.)
• Even then, make is not just a "build engine", but it's also a DSL and an executable tool. Integrating it is awkward in some ways and often requires contortions like generating fragments of makefiles based on information about the environment. The principles are actually simpler (toposort a dag, check its edges are valid, cache output files by mtime) to reimplement than to piggyback on tool with an impedence mismatch.
• On that note, Make has awful, awful syntax, and this is one of its bigger sins (though other people might disagree), especially its interaction with the shell and how escaping works inside rule bodies.
  • Fun fact: mk solves this by simply never doing any substitution or interpolation in the body of a rule. It writes the body of a rule, literally, to a temporary file, and then executes it while setting environment variables that are set by the mkfile. This was figured out in 1991 or something, but we've suffered with the alternative every day since.
• Make has poor abstraction capabilities, and not only due to its bad syntax; the macro layer for make is typically nothing more than a nightmarish hell of literal in-place expansions, that must respect syntax rules, and which use dynamic scope, not lexical scope.
  • Imagine if the C preprocessor (dynamically scoped, substitute-in-place) was put into a language where you can't even declare a variable ahead of time. That's going to go really well over time, I'm sure!
• Make is simultaneously too low level and too high level to capture many interesting build patterns.
  • For example: make is too low level because it is way too easy to write incorrect rules, and rules leak all kinds of details about them throughout the codebase (such as the need to set global variables based on the body of the rule)
  • It's also too high level because it can't capture important patterns like "dynamic dependencies"; Make often wants to build the entire build graph immediately and start executing, but at scale this is impractical — some things need to be discovered on demand. Make doesn't offer the proper tools to do this. For example, you might generate a .c file from another file, but then you need to scan the .c file for .h files it includes, in order to get a correct build graph. Because make can't dynamically discover this information, you have to repeat the logic of the generator program into the build rules, which is tedious and error prone; you can easily mis-synchronize the generator program and the rules. There is no reuse possible.
  • These aren't theoretical; my former colleague Duncan wrote about the problems with make almost exactly 15 years ago to date, and it's just as relevant now as it was then.
• These things all dovetail to make it really awkward to handle advanced programming language features through make alone, among others. For example, good luck handling C++20 modules. Even for simpler cases, capturing module dependencies is often a bunch of work in Make; for Haskell you need to make sure the compile emits dependency information that is .include'd by Make, similar to gcc -MMD in C or C++.

This is ignoring the packaging and distribution situation for any given language, which are often handled by the build tool too, muddling their responsibilities. But ultimately, I think so many language-specific dependency managers were designed for the above reasons, among many many others — Make just doesn't fit well into many of the patterns modern languages enjoy, and so most decided to go their own way and reimplement the good bits — parallel, toposorted, caching DAG — on their own.

What's the modern standard for language-based build systems?

In my mind, a lot of the "modern" build tools we enjoy sprouted from the seeds originally planted by solutions like pip for Python and rbenv/bundler for ruby. Those tools had a simple insight: every project should have their own list of dependencies and needed tools, and they should be provisioned for that project correctly in an isolated manner.

This was a big step up for language specific tools that make also couldn't match. In practice, the development velocities of these communities became discoupled from traditional package managers in the FOSS world where making sure every dependency X worked with every other dependency Y became difficult. There are other cultural reasons for this, but a big motivation for the "isolation" concept came from that.

Actually, among the modern alternatives, I think rustup and cargo for Rust, as well as ghcup and cabal in Haskell, are about the best examples for modern standard, and I give the edge to cabal somewhat there. (It's not surprising, by the way, that one of the creators of Bundler, Yehuda Katz, was also one of the creators of Cargo.)

But the language-specific-ness remains. Every tool must implement its own build engine, must integrate with language specific features, and has their own distinct UX and lexicon. And those tools also tend to miss out on the greater picture or lack advanced features...

OK, so what's the sales pitch for buck2?

From its own README.md (which was written by yours truly):

• Fast. It doesn't matter whether a single build command takes 60 seconds to complete, or 0.1 seconds: when you have to build things, Buck2 doesn't waste time — it calculates the critical path and gets out of the way, with minimal overhead. It's not just the core design, but also careful attention to detail that makes Buck2 so snappy. Buck2 is up to 2x faster than Buck1 in practice. So you spend more time iterating, and less time waiting.
• Hermetic. When using Remote Execution, Buck2 becomes hermetic: it is required for a build rule to correctly declare all of its inputs; if they aren't specified correctly (e.g. a .c file neeads a .h file that isn't correctly specified), the build will fail. This enforced correctness helps avoids entire classes of errors that most build systems allow, and helps ensure builds work everywhere for all users. And Buck2 correctly tracks dependencies with far better accuracy than Buck1, in more languages, across more scenarios. That means "it compiles on my machine" can become a thing of the past.
• Multi-language. Many teams have to deal with multiple programming languages that have complex inter-dependencies, and struggle to express that. Most people settle with make and tie together dune to pip and cargo. But then how do you run test suites, code coverage, or query code databases? Buck2 is designed to support multiple languages from the start, with abstractions for interoperation. And because it's completely scriptable, and users can implement language support — it's incredibly flexible. Now your Python library can depend on an OCaml library, and your OCaml library can depend on a Rust crate — and with a single build tool, you have a consistent UX to build and test and integrate all of these components.

Why use a large-scale integrated build system like this? What becomes possible? What becomes easy?

Assuming this all worked out, and everyone was on board, here's an example of a few things this could offer us.

• A unified build cache for all developers. One of the best features of these build systems that is immediately visible is that they support caching intermediate build artifacts, often over the network. Buck2 takes this to the biggest extreme it can; nearly all rules it implements can have intermediates cached, it encourages developers to contribute to the cache (not just CI), and it can use pre-existing Bazel-compatible caching services.
  • Example: Any developer can write patches and push a branch into the main jj repo. Any other developer can run jj git fetch and then check out those changes. They run buck2 build and their build completes instantly, thanks to cache hits.
  • Because this works for every commit and every branch, you can say goodbye to ever needing to compile https://crates.io dependencies!
• Remote build execution. Another extreme example of what these tools can do is that, with correctly written BUCK files, all commands can be executed remotely on remote systems in a correct way. This makes many distinctions between e.g. platform-specific development much simpler, because alternative platforms fail in your local development loop just like they would in CI.
  • Example: Any Jujutsu developer who works on Linux can instantly build and run tests on macOS or Windows, all from the same unified build UX (assuming those machines are available to them).
  • Nit: Remote execution does allow nearly infinite build scale when done correctly, so it's really a scale/performance improvement — but for us, I think remote platform builds would actually be the bigger benefit in practice.
• Unification of CI and local development. The previous two points (caching, remote execution) make many of the distinctions between CI systems and local development nearly obsolete. The CI and CD systems can use exactly the same resources developers do.
• A powerful, unified build language. This is actually the most unspoken benefit of such a design: a single build language and UX for building anything. Building docs, building Rust code, building C code, building web pages — they all exist behind buck2 build, using BUCK files that can be programmatically queried and introspected.
• A separation between implementation and interface. In Buck, a rule is just a function call, where the body of the function call is abstract. That means you can change the implementation to your hearts desire, as long as the contract it exposes is preserved.
  • In contrast, systems like Make tie the definition of the rule and the body of the rule together. Due to previous deficiencies, this often means that rule dependencies are dependent on the implementation of rules. Want to refactor a rule? Get ready to change all the dependents.

A good example of this is that in large scale projects, multi-language support becomes essentially necessary, and by then abstraction capabilities are really useful.

For example, the above generality allows much deeper levels of code generation than most tools possibly allow. We can have Rust code depend on a code generator written in Python that depends on C++ that depend on Rust libraries, all with a unified UX. For example, we could automatically generate web pages, man pages, and documentation indicies from the source code itself with techniques like this.

A more concrete example? A theoretical jj web that I keep talking about on Discord. We might have Typescript code that uses React that depends on HTML and hell, maybe all that TypeScript also depends on Rust compiled to WebAssembly. The sky is literally the limit in the webdev world. And we need the Rust code to depend on it, as a distributable. We need to run linters, formatters, and run test suites on that code just like we do with Rust, today to the same standards and quality. And all that can be achieved with a unified UX that behaves the same way. How do you build the HTML frontend? buck2 build. Testing? buck2 test. Running the server with every change? buck2 run.

Interlude: Cargo woes

One of my biggest constant issues with Cargo is that it does not do truly global content-addressable caching. What this means is that you can get into a scenario where things are spuriously recompiled. For example:

• You have a dependency on x=1.2.3
• You build x, resulting in an rlib
• You enable feature foobar on x=1.2.3 and build again.
• The new feature requires recompilation of x, as expected.
• You now disable feature foobar, and build x again
• Cargo will recompile x without foobar, even though it already did that once before

In other words, even though the content of the final build step above is the same as a previous result that was built, Cargo didn't know that and rebuilt it instead. This in turn means that sharing Cargo directories is not possible between projects, because one project may require serde_json with another one requires serde_json with another feature. At a very high level, Cargo doesn't identify things by content, but by name; therefore there can only be one x at a time, regardless of features.

In contrast, Buck2 correctly handles the above scenario, so the second recompilation of x without foobar will be a no-op and complete instantly. Actually, Cargo seems to really like spuriously recompiling things for many reasons beyond my comprehension; I assume this is because it is being safe and conservative to ensure the build succeeds, likely for reasons like this.

You might say "this seems theoretical", but it isn't! It's a specific case of a general problem, and it happens all the time. Anytime you jj edit any new commit, you may have a completely new set of dependencies and versions that can come from a downstream changing. For example, switching between two branches where one has a new feature will cause spurious dependency recompilation, like above.

NOTE: Earlier, I said between the modern alternatives of cargo (Rust) and cabal (Haskell), I gave the edge to cabal — and this exact example is why. Modern cabal based projects use a single globally available build cache that works across all projects, and avoids spurious recompilation like the above. For compilation-heavy languages like Haskell (and Rust), this results in huge disk, I/O, and build-time savings.

... And what becomes hard? (OpenSSL edition)

Dependencies, mostly. One of the drawbacks of the approach tools like Buck, Bazel, and even Nix make is that they truly want the most complete view of the build graph as is possible, and this means they want to understand all inputs and outputs to the system in a deterministic way. Tools operating under this assumption are few and far between e.g. too many programs write files or talk to the network or do any number of other things under the hood in order to be more useful. A simple corollary to keep in mind: small conveniences like these sleights-of-hand actually cost a lot in the long run.

In this version of the design, what this means is that Cargo crates are compiled by Buck, and not really by Cargo. These are automatically generated by a tool called reindeer, which translates Cargo.toml files into BUCK files — effectively translating the static build graph from Cargo to Buck.

This is fantastic in general, but there's a small nit — Rust build.rs scripts, which can range from doing things like "Compile 1 million lines of C++ code on demand" to "Write the current HEAD revision to a file." In general, build.rs scripts are simultaneously:

• Way, way too powerful, as they can do nearly anything and depend on any ambient effect or input,
• Extremely opaque, because their details are not exposed to the user of the build system,
• Really really useful for a wide array of things.

Because of that, and because of the impedence mismatch between Cargo and Buck2 — this patch series contains a commit called buck: add initial reindeer scaffold for rust compilation which adds approximately ~90-100 lines of YAML files in order to work around these sorts of problems; the YAML files are there to inform reindeer about what kind of build.rs scripts the crates in question are using, so it can properly emulate some of their features.

And, right now, this Buck2 build actually is broken right now, and guess why? Because openssl-sys has a build.rs script that leans into the former category — of trying to build and vendor openssl itself and compile a ton of C++ "out of band". I haven't worked a way around this yet, but it is a hard dependency due to the need of openssl by libgit2. Therefore, if this cannot be worked around, this kind of dependency would effectively prevent Buck from ever being used for Jujutsu, until we did something like swith to Gitoxide (as the pure-Rust nature alleviates many of these issues and makes many build scripts simpler; C/C++ compilation is a number-1 cause of many woes like this.)

This is something we take for granted as modern developers; dependencies are easier than ever to add and rely on, but in some sense, we also do not perceive these things as a kind of power that also restricts us and binds us. This is a form of path dependence. This is one of the ways that it can come back to stop promising experiments or alternative avenues dead in their tracks. It's worth keeping in mind.

For the most part, I genuinely find the buck2 workflow to be pretty smooth and far less corruption/recompilation prone than the equivalent Cargo workflows, ignoring bits like IDE integration.

NOTE: I feel it is important to mention that dependencies are something you are responsible for because ultimately, you ship them to your users. I feel Jujutsu should have a very high standard of quality for the dependencies it chooses to rely on; a version control system is one of the most important and vital tools in the entire stack. And personally, I think we do have good standards; the dependency list is relatively small, controlled, and well tested. openssl I think is the only real problematic thing, it's only problematic here in this PR, and there are discussions about gitoxide in the future. The Rust community has very high quality in general, I will add.

So in some ways, the inherently higher demands something like Buck2 places on dependencies might not only be viewed as a limitation, but also a feature and strength; like all things, there is tension in engineering something for and by humans.

Why not Bazel? Could we use it?

Actually, I think Bazel is pretty cool. But I have little experience with it; I'm using Buck2 because I think it's a good project, written by talented people, and I trust the pedigree it came from, and I think many of its design choices mean it actually might be able to give Bazel a true run for its money — whereas many alternative solutions either seem to be catching up or do not match its functionality, Buck2 seems to be hitting all the right high notes and offers many things Bazel itself hasn't or can't.

• Rules in userspace. Bazel rules for a language are often written by third parties, except for one set of libraries: the C/C++/ObjC, and Java rules, all are written in a special variant of Starlark and built into Bazel. These rules cannot be modified or rewritten.

  In contrast, all build rules for all languages in buck2 are written in Starlark in "userspace"; the buck2 binary knows nothing about any programming language. Morally, it's a binary that only knows how to build a graph of commands, and run them (possibly remotely.) This makes things like atomic upgrades between versions, rollbacks, and many other long-tail scenarios like patching the build rules very easy in practice.

• Buck Extension Language, or "BXL". An entire extension framework built on top of the build engine, which can be driven by Starlark, allowing code-driven introspection, automation, and querying. This makes features like LSPs, code search indicies, and other "related" tooling significantly easier to develop and support e.g. you can feasibly develop your own bxl-based LSP integration for rust-analyze or TypeScript.

• Virtual filesystem support. While this is minor and tangential, virtualized jj repositories have been a recurring topic; Buck2 includes direct support for them by way of watchman and integration with Sapling's EdenFS; I don't believe there's any reason we couldn't extend that to Jujutsu as well. I think that makes this a great dogfood opportunity, and a chance for more "vertical" integration between them. I think this could be super exciting if executed correctly.

• Native binary. Again, minor — I don't think it's a huge deal in our case, and I don't care too much about things like GC, the fast startup times from Buck2 being a native executable written in Rust is really wonderful.

As for the could? Well, I suppose. But I don't know how to juggle it, so I can't write it or maintain it.

Should this happen later, instead of now?

Initially it might seem like this is jumping the gun — if we wanted all those great features, we could just wait later for them, right? What's the issue with doing it years down the line when Jujutsu has 20 paid developers and +1 Jujillion installations worldwide?

But there's the rub: by the time you need a thing like this, you are presumably hitting the walls of technical debt that make moving forward on difficult obstacles nearly infeasible, and where existing human-scale constraints make "lateral movement" to new tools like this extremely difficult. It is actually easiest to adopt a tool like this early on, to the most aggressive degree possible, and thus avoid the conundrum entirely, if you believe that you will need it.

A good metaphor is trekking through an unknown jungle. If you walk 10 miles and realize you took the wrong turn 8 miles back, you can of course just begin the long trek back, and take the better path. But it cost you a lot of time. And surely wasn't an easy choice to make — unless the alternative was another 150 miles.

Luckily, the actual code needed to support the build right now is actually pretty darn small if you ignore the prelude. Keep reading for more details on that.

And what about Cargo?

I actually don't think Cargo should be deprecated, if possible, at least not for a very long time — because especially its IDE support through rust-analyzer is fantastic to have for now. Assuming that can be handled (not easy, but also not supremely hard), I think many other problems could be solved; for example, we could invert the whole process, and auto-generate Cargo.toml files from BUCK files instead of the other way around; then use those in crates published on https://crates.io

Or not. I think we could actually get pretty far and a lot of nice bonuses while supporting both. Ideally we would settle on one eventually, I guess.

Should this happen at all?

Maybe not. There are good arguments that Jujutsu might never outgrow the need for something like Cargo, and that the costs of adopting Buck would be wasted — for example, it's just OK to have some manual scripts or repeating yourself a few times, where Buck would allow abstractions. On the other hand, in such a case where you never need its power, Buck would likely never impose many burdens to maintain, so removal would be easy.

A good example of this is again the jj web example before. How do we drive typescript, react, all those tools? What if we wanted those code generators to build e.g. option or command-line-flag indicies for a website? We'd probably have to write a bunch of .bzl code ourselves to do it, in a robust way. So, it's a bunch more work for us, though ostensibly we could make it more widely available, too. But that's a lot more work than a Makefile that's 30 lines and mostly works (except when it breaks.)

Ultimately though it does hinge on whether or not those advantages seem big enough in the grand scale of things. If you're not going to lean into a tool like Buck, and it only ever plays second fiddle to a tool like Cargo, you'll never truly reap its advantages. Doing so requires real commitment, and it is not clear to me if this commitment is the right choice.

So what's the catch? And the risks?

Plenty, right now. Major drawbacks:

• Limited to users of nix develop exclusively
• No testing support
• No simple profile configuration support (e.g. release vs debug)
• No rust-analyzer support (though, it's a high priority for the upstream project, since it's also written in Rust)
• Windows is definitely MIA here, and would need a few questions answered to get it running. (Windows support is definitely a big priority for Buck2 in general, though.)
• Probably many other nits

Warts include:

• Committing about 90,000 LOC from buck2-prelude into the repository is, uh, a choice.
  • The reality is much rosier beyond that, though; there's only about ~200 lines of "real" code added here, and half of that is duplicating Cargo.toml, so it's really only about 100 lines:

austin@GANON:~/src/jj$ wc -l \
    BUCK lib/BUCK cli/BUCK \
    buck/third-party/rust/Cargo.toml \
    buck/.gitattributes buck/update-prelude.sh.in \
    .buckconfig buck/toolchains/BUCK 
  10 BUCK
  39 lib/BUCK
  10 cli/BUCK
  83 buck/third-party/rust/Cargo.toml
   2 buck/.gitattributes
  15 buck/update-prelude.sh.in
  26 .buckconfig
  25 buck/toolchains/BUCK
 210 total

• Duplicating Cargo.toml for Reindeer, even after I went through Herculean *cough* efforts to get rid of that this week.

Risks include:

• Impacted development velocity.
  • The need to more carefully evaluate things like dependencies and build architecture means that development is slowed, as these considerations are more paramount and cannot be ignored.
  • This then switches many operations like adding dependencies from an "optimistic strategy" to a "pessimistic strategy", to lift and abuse a term from the world of concurrency control.
  • Many developers do not care about these things, and may see the inability to e.g. add any dependency as an impediment.
  • The lack of these things tends to result in some NIH syndrome. Sometimes that's good, and sometimes it's bad.
• Continuous maintenance and sunk costs.
  • If all the savings you have are blown on the time you spend hand-holding the tool — multiply that by N developers, and you're deep in the red pretty quickly. (This means the final product needs to have high quality across the board.)
• Higher complexity.
  • It is undeniable that Buck2 is a more complex tool to handle than Cargo, and there is nearly no argument about this in my mind, even if Buck2 takes a stance that much of that complexity is inherent to the problem space.
  • This is also a felt more acutely by those downstream of the project, e.g. those people who package jujutsu.
• People will complain about you on internet forums and internet posts because they don't like that you have gone off the paved road.
  • I actually think this isn't without merit, in a broad sense (some specific complaints may be bogus),
  • But on the other hand we might have some of this already; e.g. jj developers are, you know, expected to use jj to contribute. Maybe not for typofixes, but I also wouldn't expect most typofix patches to need Buck, either.
  • I think providing things like a Cargo build helps ameliorate those concerns.

Can I use this branch right now?

No. The build is currently broken, because I do not have the ability to work around the openssl-sys crate being broken. This is just a WIP snapshot so it does not get lost. But one day, you'll be able to build and run jj-cli like so, perhaps:

$ buck2 build //:jj-cli
$ buck2 run //:jj-cli -- ...

What's the goal of publishing this?

Because I don't want to lose my work, and I would be interested in what people think about all this. I suspect there are a lot of opinions from "sounds awesome" to "OK" to "I'm not convinced".

Will you keep working on it?

Maybe. Actually, the current total time on getting this branch as far as I did took less time than writing up this PR description. So it's not like I've put unbelievable effort into it.

And, here's the thing: I'm committed to Buck2 for the long run for my projects because I do believe it solves many chronic issues I face, to a much greater and more refined degree than its predecessors. Similarly, after a lot playing for the past few weeks, I firmly believe I'm also committed to Jujutsu for the long run, for the same reasons: because it works like how I think a tool of its class should, and does so far better than the competition — it's 10x better for me, not 1x or 2x. You might think of them like peanut butter and jelly, in my opinion. So I'm happy to see them go together, too.

So I might continue this, returning to it occasionally when it's promising, and if it's considered intruiging by the fellow developers who might also be miffed by some of the above issues. It can't be merged in the current state, I think, at least until full builds work. And assuming it went well I can see myself maintaining it for a good long time and documenting it and helping out any/all issues with it. But if nobody is interested, or it's only a resounding "meh", I'll probably shelve it for a good while until its turnkey or I've established my own changes across the codebase more, or something.

[PhilipMetzger]

Thanks for bringing the write up from https://gist.github.com/thoughtpolice/6fcc0b102e10ac968a22b420b540f607, to here. I also fell into the build systems rabbit hole a few years ago and haven't recovered.

From a technical standpoint I like the idea, but I'm pretty sure that @martinvonz and other Google developers use Copybara to import the changes from Github to the Google Monorepo, where they have a Blaze/Bazel BUILD file available.

The drawbacks section speaks for itself and makes it sadly a immediate no-go from an infrastructure standpoint, as no windows and no-nix linux builds make it hard to present a good case for it. If we had something like a developer policy, like LLVM, it could be here to stay as a developer supported or in rustc terms as a Tier-3 or Tier-4 target.

I want to share your enthusiasm about buck2 and also understand the pains of cargo, but this is a non-trivial ask and probably would need a RFC process if it existed.

Sidenote: Once again thanks for a amazing write up in this pr.

[thoughtpolice]

From a technical standpoint I like the idea, but I'm pretty sure that @martinvonz and other Google developers use Copybara to import the changes from Github to the Google Monorepo, where they have a Blaze/Bazel BUILD file available.

Out of curiosity, I wonder: is it possible to have it export usable ones for public consumption here? Could we have those? I'd love to actually give a whirl at Bazel vs Buck2 on a real project (there are a few others in public that have BUCK and BUILD rules) and would be very interested in seeing how the BUILD rules work for jj. Though maybe it's just exactly like my BUCK files here...

The drawbacks section speaks for itself and makes it sadly a immediate no-go from an infrastructure standpoint, as no windows and no-nix linux builds make it hard to present a good case for it.

Actually, Windows is pretty well supported by Buck2; it's a pretty high priority internally with multiple Windows devs assigned directly to it, and I know personally that Rust works with MSVC in this setup (I'm not sure what Bazel's Windows support looks like, but my understanding was that Windows was a long tail project that is now pretty well supported.) I probably overstated the problem in my original description. The real thing is just that the scripts like update-buck2-prelude are Linux specific, and that's due to some upstream bugs that I hope to help fix eventually. In theory this might mostly work on Windows and macOS right now, I just haven't tested it.

Frankly it probably makes sense to even replace that script with a Rust binary.

The Nix thing is, some minor nits aside, mostly because it just magically gives me a buck2 binary in $PATH more than anything. Assuming we had some other way for most of us to get buck2 binaries, that's really all we need.

I do agree though that unless this has working multi-platform builds that at least work in debug mode, OOTB — it's far too experimental to merge into the main repo. But, it's not bad as a motivating case study to get those issues polished/fixed, though (my own projects are typically Linux exclusive, which helps a lot.)

If we had something like a developer policy, like LLVM, it could be here to stay as a developer supported or in rustc terms as a Tier-3 or Tier-4 target.

Yeah, I think that's good probably a lot later on, so for now just keeping this in a branch is OK with me. Honestly, I don't foresee us getting any major complications anytime soon that would make rebasing this hard.

Sidenote: Once again thanks for a amazing write up in this pr.

You got it!

[thoughtpolice]

I do agree though that unless this has working multi-platform builds that at least work in debug mode, OOTB — it's far too experimental to merge into the main repo.

Actually, I'd say my minimum requirement for merging this would be even higher — a unified build cache for all 3 platforms. That's not only a high bar but also something that immediately is useful for everyone.

[PhilipMetzger]

Out of curiosity, I wonder: is it possible to have it export usable ones for public consumption here? Could we have those? I'd love to actually give a whirl at Bazel vs Buck2 on a real project (there are a few others in public that have BUCK and BUILD rules) and would be very interested in seeing how the BUILD rules work for jj. Though maybe it's just exactly like my BUCK files here...

I have no answer to that, as I don't work for/at Google. But I think LLVM went a similar way, partial Google opensourcing and now its community maintained.

Actually, Windows is pretty well supported by Buck2; it's a pretty high priority internally with multiple Windows devs assigned directly to it, and I know personally that Rust works with MSVC in this setup (I'm not sure what Bazel's Windows support looks like, but my understanding was that Windows was a long tail project that is now pretty well supported.) I probably overstated the problem in my original description. The real thing is just that the scripts like update-buck2-prelude are Linux specific, and that's due to some upstream bugs that I hope to help fix eventually. In theory this might mostly work on Windows and macOS right now, I just haven't tested it.

Frankly it probably makes sense to even replace that script with a Rust binary.

I think both Buck2 and Bazel have nowadays good Windows support, but I'm the wrong person to adress it here, as I've had no production experience with them yet. Notable things missing from the Bazel Windows version are I think the builtin sandbox, which makes it attractive for local sandboxed builds.

The whole prelude import is also blocked on glens submodule work, as a job updating the prelude is not available yet.

Bootstrapping is less of problem, if it's easily doable with a single script. So I agree pretty much to everything.

I do agree though that unless this has working multi-platform builds that at least work in debug mode, OOTB — it's far too experimental to merge into the main repo. But, it's not bad as a motivating case study to get those issues polished/fixed, though (my own projects are typically Linux exclusive, which helps a lot.)

Major 👍

The Nix thing is, some minor nits aside, mostly because it just magically gives me a buck2 binary in $PATH more than anything. Assuming we had some other way for most of us to get buck2 binaries, that's really all we need.

Time for buck2go instead of bazelisk?

Actually, I'd say my minimum requirement for merging this would be even higher — a unified build cache for all 3 platforms. That's not only a high bar but also something that immediately is useful for everyone.

That part is actually covered by newer Github runners, I think the cache runner has a bazel cache builtin. I spotted it on the workerd repo.

All in all, I'd wait for the real stakeholders to show up.

[martinvonz]

By the way, I just merged #2115, which adds a way for custom binaries to bake extra configs into the binary. I would have liked to have a test for that, but since Cargo can't yet depend on binaries from another crate, I just had to skip tests. I have never used Buck2, but I assume it would have handled that without a problem.

[PhilipMetzger]

Smallish update on this:

Time for buck2go instead of bazelisk?

We could use ahornby's hashbang to distribute Buck2 for jj developers.

[thoughtpolice]

As an update, I randomly decided to pick this back up and made an important milestone: I can build a working copy of jj with buck2! It also includes DotSlash files to distribute a working copy of buck2 to everyone (this tool is what inspired hashbang as Philip mentioned above.)

First, install dotslash. It is a pretty nifty tool: https://github.com/facebook/dotslash/releases or cargo install dotslash. Then, assuming you have dotslash in your `$PATH afterwords:

$ cd path/to/jj
$ nix develop                                 # grab some extra tools we need
$ export PATH=$(jj root)/tools/bin:$PATH      # put dotslash tools in $PATH

This will put 3 tools in your path from tools/bin:

• reindeer, for converting Cargo files to Buck files
• buck2, the build system
• diffedit3, because I like it and to show we could use this to distribute more tools for development workflows. :)

$ buck2 run :jj -- version

Assuming that the stars align, this should work.

And your output will look like this; a clean build for me took 1m30s:

...
Build ID: 7436fa50-3f1c-495c-a5f4-fade775d2d3a
Network: Up: 0B  Down: 59MiB
Jobs completed: 4328. Time elapsed: 1:31.4s.
Cache hits: 0%. Commands: 1526 (cached: 0, remote: 0, local: 1526)
BUILD SUCCEEDED
jj 0.17.0

---

I had to do a surprisingly small amount of work to get here. Some notes.

• Thanks to some new features in buck2, we don't have to commit 90,000 lines of code from the upstream Buck2 prelude into this repo. So that's nice.
• This build always vendors all versions of C code we depend on. These dependencies are:
  • zstd, which we in theory use for the native backend but we could jettison it for now, since there's nobody using it probably.
  • libgit2, obvious
  • libssh2, by way of libgit2 and perhaps can be dropped in the future with OpenSSH support
  • libz, needed by libgit2 as well
  • boringssl! This is needed by libssh2 and libgit2, because obviously they need cryptography. I had to port Google's BUILD files from Bazel to Buck and adapt them to Buck's cxx_library. But this was actually really easy though, because they completely rearchitected the build system to mostly be "a bunch of C and C++ and assembly files." I then had to build the Rust crate bssl-sys from this, but it was also surprisingly easy because rust_library just produces normal crates.
  • Overall, I'm quite pleased with this amount of necessary C code. It's pretty easy to handle. We could drop or vendor some of these ourselves further, and ideally in the long run after the gitoxide migration, we might only need, say, zstd and maybe boringssl if we can't migrate to rustls.
• The tool reindeer is quite easy to use, and converts Cargo.toml files to BUCK files for us. It also has tons of functionality for the workarounds we need; for example it lets us remove openssl-sys from the dep list of libgit2 and silently replace it with bssl-sys, as well as enabling the boringssl feature.
• This could in theory solve problems like Windows builds OOTB, cf git: Always use OpenSSL on win32 #3554. It turns out OpenSSL's openssl-vendored build basically requires you to have perl/make installed because it's required for the OpenSSL build, so it's hard to build jj fresh from scratch on a brand new machine with no extra tools; you need to use some supplementary thing like vcpkg or
  • Windows certainly does not work yet though
  • We don't have proper dotslash files for Windows either (windows_shim, I can elaborate more on this later)
• macOS also does not work. Probably easier to fix than Windows.
• The build is slower than Cargo right now, and incremental rebuilds are about 3x slower (echo >> lib/src/lib.rs causes a approx 4s rebuild for Cargo but 12s on Buck2.)
  • Thanks to @davidbarsky, a lot of this turned out to be a lack of incremental builds in rustc, and is easily fixable by tweaking the rust_binary/rust_library flags. Now a no-op rebuild starting from jj_lib is about ~5.5s, which is comparable to Cargo and tolerable.
• No support for release or optimization settings
• Efforts for LSP/rust-analyzer support is unknown 🤷🏻‍♂️
• No RBE support; I probably have to fix a dozen bugs at least.
  • Caching might be workable, I'm not sure.
• No clue how tests should look.
• It has a hard Nix dependency right now
  • e.g. I need a copy of bindgen for boringssl, and don't know where else to get it
  • This all needs to be replaced with a hermetic C/C++ and Rust toolchain
  • This might be difficult. Maybe we can rely on the system toolchains for a while. I haven't thought hard about it. But we have to get rid of the Nix dependency IMO.
• Technically, it has one feature that the upstream build system doesn't have! Yet.
  • It always regenerates .proto files into Rust files for every build via gen-protos, and forces them into the build, rather than relying on committing new files to the repo.
  • This is less error prone and easier to understand for everyone.
• There are a few #[cfg(buck_build)] tweaks needed in lib, cli, and gen_protos. The fixes total up roughly to about +/- 20 LOC.
  • 90% of these come down to the #[grammar] macro from pest_derive. When using cargo and buck2, rustc gets executed from different working directories.
  • The gen_protos fix has similar logic: the $CWD of rustc is different under Buck.
  • I think these are very low maintenance and easy to understand, so that's nice I guess?

So this is still a huge experiment. But I consider a reliable build to be a big milestone! And the binary does seem to mostly work fine.

[davidbarsky]

Efforts for LSP/rust-analyzer support is unknown 🤷🏻‍♂️

It works today with a bit of configuration, but once I land this branch into rust-analyzer, the overall experience should be much more seamless and Cargo-like.