Vendoring FAQ

Why do we need to vendor?

Ciao has dependencies on 3rd party packages that are not included in the Go standard library. When you do a

go get github.com/01org/ciao/...

go get calculates all of ciao's dependencies and recursively gets (git clones and go installs) them as well. It knows which packages to pull by looking and the import statements in the ciao code. The problem is that these import statements don't have any version information, so go get pulls down and installs the latest version of these dependencies's master branches. There are two big problems here.

1. Changes to these 3rd party dependencies could break our build.
2. It's possible for two engineers building the same version of ciao with go get -u github.com/01org/ciao/... to get different ciao binaries. This makes it hard to reproduce issues among the development and QA teams, and our end users, as everyone is potentially testing different builds, even though the ciao code used to create each build is the same.

How does vendoring work in Go?

When go resolves a dependency when building a package it first looks in a special directory called vendor, located in the root directory of the repo that contains that package. If it finds the source for the package in this folder, go will build your package against this vendored package. Otherwise it will look in $GOPATH/src for the package. For example, when ciao-launcher imports "gopkg.in/yaml.v2", go looks in $GOPATH/src/github.com/01org/ciao/vendor/gopkg.in/yaml.v2 first. If there's nothing here, go then looks in $GOPATH/src/gopkg.in/yaml.v2.

We can take advantage of this by storing the versions of the dependencies that we want to use in the ciao/vendor folder.

Do we really need to check the vendored code into ciao?

Basically, yes. If we want our users to be able to work with ciao using wildcards, e.g.,

go get -u github.com/01org/ciao/...

go clean -i ./...

go install ./...

go test ./...

we have to do this.

Originally, we were hoping to be able to create git submodules in the ciao/vendor directory that pointed to the versions of the dependencies that ciao actually uses. Unfortunately, this doesn't work. The problem is that go packages are often stored in repositories that also contain other packages. Often ciao only uses 1 package from a repository that contains many packages. If we used submodules to populate our vendor tree, the tree would include a whole pile of code that ciao doesn't actually use, and this is a big problem for two reasons.

1. Go wildcards (./...) operate on the vendor directory. This means a go install ./... builds all the packages in the vendor directory, even those ciao does not need. This is a problem because it slows the build and it also requires us to vendor the dependencies of all these unused dependencies as well, to avoid breaking the build.
2. Some of these repositories have their own vendor directory, which introduces the problems of nested vendoring into ciao, which is a very good way of introducing incomprehensible build breakages (https://github.com/mattfarina/golang-broken-vendor)

How can we get the wildcards to work with vendoring?

We need to copy only the parts of the dependencies we use into the vendor sub folder. For example, look at the $GOPATH/src/github.com/docker/distribution repo. This repo contains multiple packages spanning 232 directories and 868 files. We only want the github.com/docker/distribution/uuid package. If you look at what is currently checked into ciao/vendor/github.com/docker/distribution you should see this:

# tree vendor/github.com/docker/distribution
vendor/github.com/docker/distribution/
├── AUTHORS
├── LICENSE
├── MAINTAINERS
├── README.md
└── uuid
    └── uuid.go

i.e., five files.

How do we vendor?

It's too tricky and error prone to do this by hand so we need to use a tool. Currently, we're using a custom tool to do the vendoring, ciao-vendor/ciao-vendor.go. If we find it too awkward to use or too painful to maintain we could move to a 3rd party tool (govendor seems to be closest to ciao-vendor). Right now I like having the flexibility of a custom tool as we learn about vendoring and our dependencies.

How do I get a list of ciao's dependencies?

cd github.com/01org/ciao

go run ciao-vendor/ciao-vendor.go deps

This will give you a list suitable for the IP plan, e.g.,

Package Root                Repo                        Version                License
github.com/Sirupsen/logrus        https://github.com/Sirupsen/logrus.git        v0.9.0                MIT
github.com/boltdb/bolt            https://github.com/boltdb/bolt.git        144418e                MIT
...

To see all the individual packages type

go run ciao-vendor/ciao-vendor.go packages

The differences between deps and packages is that deps shows you information about the git repos in which our dependencies are stored where as packages shows information about the individual packages. Packages will typically have more results as we use multiple packages from some repos.

What is this package root thing?

Many of the ciao-vendor commands require a package root. For simple packages that are stored in the top level of their repository, the package root is the same as the package name, i.e, the name you use in the import statement in go code. However, some repos contain multiple packages. In this case the package root of these packages is the name of their repos' top level package, even if the top level package doesn't actually exist, i.e., doesn't contain any go code.

For example, the package root of

github.com/rackspace/gophercloud/openstack/identity/v2/tenants

and

github.com/rackspace/gophercloud/openstack/identity/v2/tokens

is

github.com/rackspace/gophercloud

The package root is always unvendored. So always use github.com/rackspace/gophercloud and not github.com/01org/ciao/vendor/github.com/rackspace/gophercloud when passing the package root to ciao-vendor commands.

You can see the package root of the currently vendored packages using the

go run ciao-vendor/ciao-vendor.go packages

The package root is listed in the third column, currently erroneously called repo.

How do I add a new dependency?

1. Add the dependency to your code as normal. Presumably you'll go get the dependency into your $GOPATH.

2. Check to see whether the repo that contains your package is already synced in your local $GOPATH. If it is, ensure that you have made no local modifications to this dependency.

3. When you have your code working you can vendor your package. First you need to know four pieces of information:

4. The name of the root package that contains the package you wish to vendor. This may or may not be the same as the name of the package you are vendoring.

5. The version (the branch, the tag or the commit id) of the repo that contains the package you wish to use.

6. The git URL of the repo, use https were possible

7. The repos's license.

8. When you have all this information you simply need to type

   cd $GOPATH/src/github.com/01org/ciao

   go run ciao-vendor/ciao-vendor.go vendornew rootpackagename version license URL

   For example, to vendor the github.com/agtorre/gocolorize package we might type

   go run ciao-vendor/ciao-vendor.go vendornew github.com/agtorre/gocolorize ece46eb MIT https://github.com/agtorre/gocolorize

   This might take some time as it will download the latest version of gocolorize.

9. Check everything is vendored. This might not be the case if gocolorize depends on other non-vendored dependencies. You would then need to vendor them, i.e., go to step 2.

   go run ciao-vendor/ciao-vendor.go check.

10. Check in the changes to ciao-vendor/packages.json and the vendor folder.

How do I vendor a package that is stored in the same repo as a package that has already been vendored?

When ciao-vendor vendors a new package it only copies the files it needs from the repo that contains that package. If at a later stage you update ciao to use a second package from the same repo, you will need to run the vendor tool again to vendor the new package. The good news is that ciao-vendor already has all the information it needs to vendor the new package. All you need to do is to:

1. Check to see whether the repo that contains your package is already synced in your local $GOPATH. If it is, ensure that you have made no local modifications to this dependency.

2. Modify your ciao code to import the new package.

3. Run the vendor tool

   cd $GOPATH/src/github.com/01org/ciao

   go run ciao-vendor/ciao-vendor.go vendor

4. Check everything is vendored.

   go run ciao-vendor/ciao-vendor.go check.

5. Check in the changes to the vendor folder.

How do I update a dependency?

1. Check to see whether the dependency is already synced in your local $GOPATH. If it is, ensure that you have made no local modifications to this dependency.

2. Run the vendor tool

   cd $GOPATH/src/github.com/01org/ciao

   go run ciao-vendor/ciao-vendor.go revendor rootpackagename version

3. go run ciao-vendor/ciao-vendor.go check

4. Commit the changes to ciao-vendor/packages.json and the vendor directory

Note that the ciao release process requires you to ensure that all the unit tests for the new version of your vendored package pass. This can be done easily using ciao vendor. See 'How can I test a dependency?' below.

Also, revendoring updates all the vendored packages that shared the same repo.

How do I figure out who is using a given package?

It can be useful to know which of the ciao and vendored packages depend on a given package. For example, you might wish to remove a 3rd party dependency from ciao to reduce the maintenance overhead of the vendored packages. You can only really do this if you know that the package in question is used by ciao components. If it's also needed by a vendored package, you there's no point in removing it unless you also submit a PR to the upstream vendored package. ciao-vendor includes a command that prints the clients of a given package.

For example,

go run ciao-vendor/ciao-vendor.go uses github.com/01org/ciao/vendor/github.com/Sirupsen/logrus

outputs

github.com/01org/ciao/ciao-launcher
github.com/01org/ciao/vendor/github.com/docker/engine-api/client
github.com/01org/ciao/vendor/github.com/docker/go-connections/tlsconfig

The packages that are outputted by this command use github.com/01org/ciao/vendor/github.com/Sirupsen/logrus either directly or indirectly.

To see only the packages that are directly dependant on the specified package, use the -d switch.

How do I figure out which vendored dependencies are out of date?

Run

go run ciao-vendor/ciao-vendor.go updates

This should print a table indicating which vendored dependencies trail master, e.g.

Package                                 Status				
github.com/Sirupsen/logrus              46 commits behind HEAD		
github.com/boltdb/bolt                  16 commits behind HEAD		
github.com/coreos/go-iptables           Up to date			
github.com/docker/distribution          166 commits behind HEAD		
github.com/docker/docker                3611 commits behind HEAD	
github.com/docker/engine-api            190 commits behind HEAD		
github.com/docker/go-connections        4 commits behind HEAD		
github.com/docker/go-units              13 commits behind HEAD		
github.com/docker/libnetwork            174 commits behind HEAD		
github.com/golang/glog                  Up to date			
github.com/gorilla/context              4 commits behind HEAD		
github.com/gorilla/mux                  13 commits behind HEAD		
github.com/mattn/go-sqlite3             7 commits behind HEAD		
github.com/mitchellh/mapstructure       Up to date			
github.com/opencontainers/runc          233 commits behind HEAD		
github.com/rackspace/gophercloud        11 commits behind HEAD		
github.com/tylerb/graceful              13 commits behind HEAD		
github.com/vishvananda/netlink          3 commits behind HEAD		
github.com/vishvananda/netns            Up to date			
golang.org/x/net                        78 commits behind HEAD		
gopkg.in/yaml.v2                        Up to date		

Note this command can take some time to execute as it needs to first update the dependencies in your $GOPATH. Some of the dependencies, like docker, can take a long time to sync.

How can I test a dependency?

ciao-vendor contains a helper command called test to run the unit tests of a specific version of a given package. It can be used as follows

go run ciao-vendor/ciao-vendor.go test [-s] package version [go test flags]

The -s switch can be used to run the unit tests via sudo. This is necessary for some of our networking dependencies. Package is the normal package name, not the vendored name, e.g., github.com/vishvananda/netns. Version is the branch or commit to test. The version number can be proceeded by a number of arguments, e.g., -v, -race. These arguments are passed directly to go test.

go test can take a little bit of time to run as it first needs to go get the package, and all its dependencies including test dependencies, before it can run the unit tests.