From e53edc9d55728bb16afb41be4a1074c56e8fcd91 Mon Sep 17 00:00:00 2001 From: Romain Jobredeaux Date: Mon, 8 Apr 2024 11:11:03 -0400 Subject: [PATCH 1/2] Document some typical usecases for remotetool. --- go/README.md | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git a/go/README.md b/go/README.md index d529b9aa1..092023368 100644 --- a/go/README.md +++ b/go/README.md @@ -1,9 +1,62 @@ # Go SDK ## Usage - TODO(jsharpe): Write usage instructions. +### remotetool + +`remotetool` is a CLI tool to interact with remote build execution. In particular, blobs and input trees can be downloaded and uploaded from CAS, actions and their inputs can be downloaded to be either re-run locally or remotely. + +#### Common flags +Flags that describe the RBE instance used and the authentication method are common to most `remotetool` invocations. The `--instance`, `--service`, and credential flags (`--use_application_default_credentials`, `--use_gce_credentials`, `--credential_file=FILE`, one of which must be set), in particular, are further documented under [pkg/flags/flags.go](pkg/flags/flags.go). + +`--alsologtostderr` and `-v VERBOSITY_LEVEL` can be used to tune the verbosity and location of the output, e.g. `--alsologtosrderr -v 1`. + +#### Downloading an action's inputs and metadata +``` +bazelisk run //go/cmd/remotetool -- \ + --operation=download_action \ + COMMON_FLAGS \ + --digest=DIGEST \ + --path=PATH +``` +- For `COMMON_FLAGS`, see the [above section](#common-flags) +- The `--digest` flag is formatted as `"{hash}/{size}"` and can typically be obtained from logs of the tool that originally ran the action. +- The `--path` flag is the destination where the action inputs will be downloaded (under `PATH/input`), as well as: + - metadata relating to the action (e.g. command hash, input tree hash, individual input file hashes...) + - `run_command.sh`, a script that contains the command line of the action to be re-run + - `run_locally.sh`, a script that contains the docker invocation to re-run the action locally + +The action can then be re-run locally by running `./run_locally.sh`, or remotely using the instructions below. When running the action locally, local modifications made to the `input` folder will be picked up automatically, but not if re-running the action remotely, read on for more on this. + +#### Re-running a downloaded action remotely + +``` +bazelisk run //go/cmd/remotetool -- \ + --operation=execute_action \ + COMMON_FLAGS + [--action_root=PATH|--digest=DIGEST] \ + --path OUTPUT_PATH +``` +- For `COMMON_FLAGS`, see the [above section](#common-flags). +- An action digest formatted as `"{hash}/{size}"` can be provided directly using the `--digest` flag, or a previously downloaded action can be used with `--action_root`. The `--action_root` path should point to the `--path` of a previous `download_action` invocation. Specifically, the `--action_root` folder must contain a `cmd.textproto` and `ac.textproto` files. +- `--path` is the destination where the outputs of the action will be downloaded + +#### Running a modified version of a downloaded action + +1. Download an action to a given `PATH` following the [instructions for downloading an action](#downloading-an-actions-inputs-and-metadata), +1. modify the input under `PATH/input` as needed, +1. upload the newly formed input directory using `remotetool`: + ``` + bazelisk run //go/cmd/remotetool -- \ + --operation=upload_dir \ + COMMON_FLAGS \ + --path=PATH/input + ``` + `remotetool` will log the digest of the newly uploaded input folder, `NEW_DIGEST`. +1. In the `ac.textproto` file under `PATH`, update the `input_root_digest` fields with the `NEW_DIGEST` information. +1. Run the action using the [instructions for re-running a downloaded action](#re-running-a-downloaded-action-remotely) + ## Development Update `BUILD.bazel` files with (with `fix` to be more aggressive): From e22304e970c228ed2dc0b04e3919552ef41c8ac8 Mon Sep 17 00:00:00 2001 From: "Anas H. Sulaiman" Date: Tue, 9 Apr 2024 14:13:25 -0400 Subject: [PATCH 2/2] Update README.md --- go/README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/go/README.md b/go/README.md index 092023368..472d867dd 100644 --- a/go/README.md +++ b/go/README.md @@ -46,6 +46,9 @@ bazelisk run //go/cmd/remotetool -- \ 1. Download an action to a given `PATH` following the [instructions for downloading an action](#downloading-an-actions-inputs-and-metadata), 1. modify the input under `PATH/input` as needed, +1. run the action and be sure to include the `--action_root=PATH` argument. + +Alternatively, you could also: 1. upload the newly formed input directory using `remotetool`: ``` bazelisk run //go/cmd/remotetool -- \