Skip to content

Commit

Permalink
[go] migrate from dep to go mod (chef#1837)
Browse files Browse the repository at this point in the history
Go introduced module mode in v1.11 and it became the default in v1.13.
This change replaces `dep` with `go mod` using a vendor directory.

* You no longer need to have the Chef Automate git repository inside of a `GOPATH`.
  Instead, all `go` utilities will recognize that they are in module mode and will
  handle dependency loading for you. You're now free to keep the Chef Automate
  source code in any directory you wish. You can even unset `GOPATH`. 🎉

* The `go` command supports using a module vendor directory but it has to be
  explicitly enabled with the `-mod=vendor` go flag. If you prefer not specifying
  it each time you can set the `GOFLAGS=-mod=vendor` environment variable. We
  have done this in the studio so all go commands should utilize the vendored
  dependencies, but if you run go commands outside of the studio you'll need to
  specify the flag or set the environment variable to utilize the vendored
  dependencies. If you forget, commands should still execute but they will download
  the modules into your go installations modules directory instead of using the
  vendored packages.

* Inside the studio the `GOBIN` has been set to `/src/bin`, which means `/bin`
  relative to the Chef Automate source code directory outside of the studio. This
  has the effect that installed binaries that are compiled in the studio are
  also available outside of the studio. This provides the benefit that go
  development tools that have compiled inside the studio don't have to be
  re-compiled each time you start a new studio container. It also means that
  protobuf compilers and extensions are also cached, therefore you might have to
  manually remove them if changes are made. If you notice strange behavior when
  compiling protobufs, remove the compiled binaries in `/bin` and try again.

* `enter_go_workspace`, `setup_go_workspace`, and other various studio functions
  that are required when working with go in `GOPATH` mode are no longer required.
  I tried to update every studio function but it's likely that something slipped
  though the cracks. If you notice that your studio function is behaving incorrectly
  please accept my apologies now.

* Some, but not all, of the protobuf compilers and extensions that we use in
  Chef Automate are not compatible in module mode. All of the `grpc.sh` protobuf
  scripts have been updated to utilize relative source path outputs where supported,
  otherwise measures have been put in place to compile them and move them into
  the correct directory afterwards.

* Vendoring is no longer done with `dep`. Adding a new dependency to Chef Automate
  is now as easy as importing the package into your go application and running `make
  revendor` from the root of the Chef Automate source code repo or in the studio. If you
  require go packages for development that are not included in the Chef Automate
  repository,  add them to the `vendor.go` as ignored dependencies and revendor.

* Many packages were updated as part of this migration. Everything has been
  recompiled with the new vendor directory and the single known regression has
  been fixed. If a dep change has caused you problems we can work to pin it.

* We previously relied on 3 different layers of scaffolding: `core/scaffolding-go`,
  `chef/scaffolding-go` and `chef/automate-scaffolding-go`. That's two layers of
  scaffolding too many. Because we're now in module mode our scaffolding that
  handled much of the `GOPATH` wonkiness is no longer required, thus all packages
  that require go have been migrated to `chef/automate-scaffolding-go`, which has
  been updated for the go mod with vendor directory workflow. If you add  a new go
  component  you should use `chef/automate-scaffolding-go`. If you need to
  opt out of the platform scaffolding for a go package you can set the new
  `scaffolding_no_platform` variable to true in the plan.

* Default version linker flags are now set in the scaffolding. You don't need to
  manually set them in your plans.

* Platform TLS CA and certs are handled by the scaffolding, you don't need to
  specify them in your plans.

Sorry, we did our best to make this transition as seamless as possible. Reach
out to #a2-deployment and we'll do our best to help out.

[go] replace dep with go mod vendoring
[automate-scaffolding-go] add go mod support to automate-scaffolding-go
[automate-scaffolding-go] remove dependency on core and chef go scaffolding
[hab-plans] migrate all plans to automate-scaffolding-go
[hab-plans] remove unused platform TLS config
[hab-plans] use scaffolding version linker flags everywhere
[gprc] update all grpc.sh scripts to use relative source paths where possible
[studio] remove GOPATH requirements from all studio functions
[lib/cereal] fix a regression caused by `lib/pq` library breaking changes
[bldr] refactor bldr generator tool for module mode
[repo-health] update repo health checks for module mode

Signed-off-by: Ryan Cragun <[email protected]>
  • Loading branch information
ryancragun authored Oct 24, 2019
1 parent bae811c commit c989f96
Show file tree
Hide file tree
Showing 972 changed files with 184,740 additions and 115,131 deletions.
10 changes: 2 additions & 8 deletions .expeditor/verify.pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,7 @@ steps:
#
- label: "[codegen] A2 protobufs"
command:
- |
hab studio run "source .studiorc &&
compile_all_protobuf_components &&
git add . &&
git diff --staged --exit-code --ignore-submodules=all"
cd api
make lint unit
- hab studio run "source .studiorc && verify_all_protobuf_components"
timeout_in_minutes: 10
retry:
automatic:
Expand All @@ -36,7 +30,7 @@ steps:
#
- label: repo health
command:
- hab studio run "source .studiorc && enter_go_workspace && scripts/repo_health.sh"
- hab studio run "source .studiorc && scripts/repo_health.sh"
timeout_in_minutes: 10
retry:
automatic:
Expand Down
4 changes: 1 addition & 3 deletions .studio/applications-service
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,6 @@ function applications_benchmarks() {
check_if_deployinate_started || return 1

(
enter_go_workspace
install_if_missing core/go go
install_if_missing core/git git

local tmp_txt=/tmp/new.txt
Expand All @@ -94,7 +92,7 @@ function applications_benchmarks() {
log_line "Running applications-service benchmark. (temporarily stored at $(yellow $tmp_txt))"
log_line "$(yellow "This might take a while...")"
cd components/applications-service || return 1
A2_SVC_NAME="applications-service" A2_SVC_PATH="/hab/svc/applications-service" go test \
A2_SVC_NAME="applications-service" A2_SVC_PATH="/hab/svc/applications-service" go_test \
./... \
-bench=Benchmark \
-benchmem \
Expand Down
67 changes: 40 additions & 27 deletions .studio/common
Original file line number Diff line number Diff line change
Expand Up @@ -33,16 +33,17 @@ function compile_go_protobuf_component() {
# Note: the actual generation is triggered by a call to protoc with the
# parameter '--policy_out=...'. Only if the service's methods has the proper
# annotations will the generated code not be a NO-OP.
setup_go_workspace
( cd "${scaffolding_go_pkg_path}" || return 1
local proto_go_tools=(
github.com/chef/automate/components/automate-grpc/protoc-gen-policy
github.com/chef/automate/components/automate-grpc/protoc-gen-grpc-mock # generate mock grpc server implementations
github.com/chef/automate/components/automate-grpc/protoc-gen-a2-config
)
install_go_tool "${proto_go_tools[@]}"
GO_TOOL_VENDOR=true install_go_tool github.com/envoyproxy/protoc-gen-validate
local proto_go_tools=(
github.com/chef/automate/components/automate-grpc/protoc-gen-policy
github.com/chef/automate/components/automate-grpc/protoc-gen-grpc-mock # generate mock grpc server implementations
github.com/chef/automate/components/automate-grpc/protoc-gen-a2-config
github.com/envoyproxy/protoc-gen-validate
github.com/ckaznocha/protoc-gen-lint
)
install_go_tool "${proto_go_tools[@]}"

# For handling faux GOPATH copying for non-module aware protoc extensions
install_if_missing core/rsync rsync

# "api" doesn't compile the config directory so let's not delete the config directory in that case
if [ "$path" == "api" ]; then
Expand Down Expand Up @@ -80,16 +81,16 @@ complete -F _component_auto_complete compile_go_protobuf_component

function compile_client_mocks() {
install_gomock
pushd "$scaffolding_go_pkg_path/api/interservice" >/dev/null
mockgen -source event/event.pb.go -destination event/event.pb.client_mock.go -package event
mockgen -source authz/v2/project.pb.go -destination authz/v2/project.pb.client_mock.go -package v2
mockgen -source authz/authz.pb.go -destination authz/authz.pb.client_mock.go -package authz
mockgen -source authn/authenticate.pb.go -destination authn/authenticate.pb.client_mock.go -package authn
mockgen -source cfgmgmt/service/cfgmgmt.pb.go -destination cfgmgmt/service/cfgmgmt.pb.client_mock.go -package service
mockgen -source ingest/chef.pb.go -destination ingest/chef.pb.client_mock.go -package ingest
mockgen -source event_feed/event_feed.pb.go -destination event_feed/event_feed.pb.client_mock.go -package event_feed
mockgen -source ../../components/nodemanager-service/api/manager/manager.pb.go -destination ../../components/nodemanager-service/api/manager/manager.pb.client_mock.go -package manager
popd >/dev/null
pushd "/src/api/interservice" > /dev/null || exit 1
mockgen -source event/event.pb.go -destination event/event.pb.client_mock.go -package event -self_package github.com/chef/automate/api/interservice/event
mockgen -source authz/v2/project.pb.go -destination authz/v2/project.pb.client_mock.go -package v2 -self_package github.com/chef/automate/api/interservice/authz/v2
mockgen -source authz/authz.pb.go -destination authz/authz.pb.client_mock.go -package authz -self_package github.com/chef/automate/api/interservice/authz
mockgen -source authn/authenticate.pb.go -destination authn/authenticate.pb.client_mock.go -package authn -self_package github.com/chef/automate/api/interservice/authn
mockgen -source cfgmgmt/service/cfgmgmt.pb.go -destination cfgmgmt/service/cfgmgmt.pb.client_mock.go -package service -self_package github.com/chef/automate/api/interservice/cfgmgmt/service
mockgen -source ingest/chef.pb.go -destination ingest/chef.pb.client_mock.go -package ingest -self_package github.com/chef/automate/api/interservice/ingest
mockgen -source event_feed/event_feed.pb.go -destination event_feed/event_feed.pb.client_mock.go -package event_feed -self_package github.com/chef/automate/api/interservice/event_feed
mockgen -source ../../components/nodemanager-service/api/manager/manager.pb.go -destination ../../components/nodemanager-service/api/manager/manager.pb.client_mock.go -package manager -self_package github.com/chef/automate/components/nodemanager-service/api/manager
popd > /dev/null || exit 1
}


Expand All @@ -112,6 +113,24 @@ function compile_all_protobuf_components() {
done
}

document "verify_all_protobuf_components" <<DOC
Verify that all protobuf are up to date
DOC
function verify_all_protobuf_components() {
install_if_missing core/git git

if ! compile_all_protobuf_components; then
return $?
fi

git add .
if ! git diff --staged --exit-code --ignore-submodules=all; then
return $?
fi

go_component_make api lint unit
}

document "link_component_bin" <<DOC
Link binaries from COMPONENT/cmd/* to /hab/bin
DOC
Expand Down Expand Up @@ -264,10 +283,7 @@ the dev environment. Private keys are checked into git.
DOC
function generate_dev_root_ca() {
install_if_missing core/certstrap certstrap
(
enter_go_workspace
go run ./tools/dev-cert-gen --regen-ca
)
go run /src/tools/dev-cert-gen/main.go --regen-ca
}

document "generate_dev_service_cert" <<DOC
Expand All @@ -285,10 +301,7 @@ generate_dev_service_cert foo-service
DOC
function generate_dev_service_cert() {
install_if_missing core/certstrap certstrap
(
enter_go_workspace
go run ./tools/dev-cert-gen "$@"
)
go run /src/tools/dev-cert-gen/main.go "$@"
}

_component_auto_complete()
Expand Down
15 changes: 2 additions & 13 deletions .studio/compliance-service
Original file line number Diff line number Diff line change
Expand Up @@ -24,17 +24,8 @@ document "compliance_unit_tests" <<DOC
'examples' folder.
DOC
function compliance_unit_tests() {
setup_go_workspace
# Run the tests
pushd $scaffolding_go_pkg_path >/dev/null
local GO_PACKAGES
GO_PACKAGES=$(go list ./components/compliance-service/... | grep -v '/examples/')
log_line "Executing Go test"
eval go test -v $GO_PACKAGES -cover
local EXIT_CODE=$?
popd >/dev/null

return $EXIT_CODE
mapfile -t packages < <(go list -f '{{ .Dir }}' /src/components/compliance-service/... | grep -v examples)
go_test "${packages[@]}"
}

document "compliance_integration" <<DOC
Expand Down Expand Up @@ -65,8 +56,6 @@ DOC
function auto_go_update_compliance_service() {
local component="compliance-service"

setup_go_workspace

install_if_missing core/inotify-tools inotifywait
log_line "Starting file watch for $component"
while inotifywait -r /src/components/$component; do
Expand Down
2 changes: 1 addition & 1 deletion .studio/debugging
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ function debug_go_service() {
return 1
fi

GO_TOOL_METHOD="get" install_go_tool github.com/derekparker/delve/cmd/dlv
install_go_tool github.com/go-delve/delve/cmd/dlv

dlv attach --headless -l ":${GO_DEBUG_PORT:-2345}" "$pid" --log=true --api-version=2
}
4 changes: 0 additions & 4 deletions .studio/deployment-service
Original file line number Diff line number Diff line change
Expand Up @@ -205,8 +205,6 @@ function deployment_generate() {
install_if_missing core/go go
install_if_missing core/go gofmt

setup_go_workspace

pushd /src/components/automate-deployment >/dev/null;
make generate;
popd >/dev/null;
Expand All @@ -220,8 +218,6 @@ function deployment_update_bindings() {
install_if_missing core/go go
install_if_missing core/go gofmt

setup_go_workspace

pushd /src/components/automate-deployment >/dev/null;
make update-bindings;
popd >/dev/null;
Expand Down
59 changes: 0 additions & 59 deletions .studio/end-to-end-tests

This file was deleted.

Loading

0 comments on commit c989f96

Please sign in to comment.