Vendoring FAQ

Why do we need to vendor?

Ciao has dependencies on 3rd party packages that are not included in the Go standard library. When you do a

go get github.com/01org/ciao/...

go get calculates all of ciao's dependencies and recursively gets (git clones and go installs) them as well. It knows which packages to pull by looking and the import statements in the ciao code. The problem is that these import statements don't have any version information, so go get pulls down and installs the latest version of these dependencies's master branches. There are two big problems here.

1. Changes to these 3rd party dependencies could break our build.
2. It's possible for two engineers building the same version of ciao with go get -u github.com/01org/ciao/... to get different ciao binaries. This makes it hard to reproduce issues among the development and QA teams, and our end users, as everyone is potentially testing different builds, even though the ciao code used to create each build is the same.

How does vendoring work in Go?

When go resolves a dependency when building a package it first looks in a special directory called vendor, located in the root directory of the repo that contains that package. If it finds the source for the package in this folder, go will build your package against this vendored package. Otherwise it will look in $GOPATH/src for the package. For example, when ciao-launcher imports "gopkg.in/yaml.v2", go looks in $GOPATH/src/github.com/01org/ciao/vendor/gopkg.in/yaml.v2 first. If there's nothing here, go then looks in $GOPATH/src/gopkg.in/yaml.v2.

We can take advantage of this by storing the versions of the dependencies that we want to use in the ciao/vendor folder.

Do we really need to check the vendored code into ciao?

Basically, yes. If we want our users to be able to work with ciao using wildcards, e.g.,

go get -u github.com/01org/ciao/...

go clean -i ./...

go install ./...

go test ./...

we have to do this.

Originally, we were hoping to be able to create git submodules in the ciao/vendor directory that pointed to the versions of the dependencies that ciao actually uses. Unfortunately, this doesn't work. The problem is that go packages are often stored in repositories that also contain other packages. Often ciao only uses 1 package from a repository that contains many packages. If we used submodules to populate our vendor tree, the tree would include a whole pile of code that ciao doesn't actually use, and this is a big problem for two reasons.

1. Go wildcards (./...) operate on the vendor directory. This means a go install ./... builds all the packages in the vendor directory, even those ciao does not need. This is a problem because it slows the build and it also requires us to vendor the dependencies of all these unused dependencies as well, to avoid breaking the build.
2. Some of these repositories have their own vendor directory, which introduces the problems of nested vendoring into ciao, which is a very good way of introducing incomprehensible build breakages (https://github.com/mattfarina/golang-broken-vendor)

How can we get the wildcards to work with vendoring?

We need to copy only the parts of the dependencies we use into the vendor sub folder. For example, look at the $GOPATH/src/github.com/docker/distribution repo. This repo contains multiple packages spanning 232 directories and 868 files. We only want the github.com/docker/distribution/uuid package. If you look at what is currently checked into ciao/vendor/github.com/docker/distribution you should see this:

# tree vendor/github.com/docker/distribution
vendor/github.com/docker/distribution/
├── AUTHORS
├── LICENSE
├── MAINTAINERS
├── README.md
└── uuid
    └── uuid.go

i.e., five files.

How do we vendor?

It's too tricky and error prone to do this by hand so we need to use a tool. Currently, we're using a custom tool to do the vendoring, ciao-vendor/ciao-vendor.go. If we find it too awkward to use or too painful to maintain we could move to a 3rd party tool (govendor seems to be closest to ciao-vendor). Right now I like having the flexibility of a custom tool as we learn about vendoring and our dependencies.

How do I get a list of ciao's dependencies?

cd github.com/01org/ciao

go run ciao-vendor/ciao-vendor.go deps

This will give you a list suitable for the IP plan, e.g.,

Package Root                Repo                        Version                License
github.com/Sirupsen/logrus        https://github.com/Sirupsen/logrus.git        v0.9.0                MIT
github.com/boltdb/bolt            https://github.com/boltdb/bolt.git        144418e                MIT
...

To see all the individual packages type

go run ciao-vendor/ciao-vendor.go packages

The differences between deps and packages is that deps shows you information about the git repos in which our dependencies are stored where are packages shows information about the individual packages. Packages will typically have more results as we use multiple packages from some repos.

How do I add a new dependency?

This is a bit clunky at the moment. Here's what you need to do.

1. Add the dependency to your code as normal. Presumably you'll go get the dependency into your $GOPATH.

2. When you have your code working you need to decide which version of the dependency to use, and then edit the repos map at the top of ciao-vendor.go. We'll update the tool soon to do this for you but for now you need to edit the file. Let's say we wanted to vendor github.com/agtorre/gocolorize. You'd do this by adding the following entry to the repos map

   "github.com/agtorre/gocolorize":     {"https://github.com/agtorre/gocolorize", "ece46eb", "MIT"},
   

   • "github.com/agtorre/gocolorize" is the location of the repository that contains the package in $GOPATH/src.
   • "https://github.com/agtorre/gocolorize" is the URL of the repository. I'll remove this soon I think
   • "ece46eb" is the version of the repository to check out
   • "MIT" is the license. We'll see if there is a way to detect this automatically. For now you need to enter it by hand.

3. Run the ciao-vendor tool

   cd $GOPATH/src/github.com/01org/ciao

   go run ciao-vendor/ciao-vendor.go vendor

   This might take some time as it will download the latest version of gocolorize.

4. Check everything is vendored. This might not be the case if gocolorize depends on other non-vendored dependencies. You would then need to vendor them, i.e., go to step 2. go run ciao-vendor/ciao-vendor.go check.

5. Check in the changes to ciao-vendor.go and the vendor folder.

How do I update a dependency?

Again this is a little clunky right now. We'll add a special command for doing this but right now.

1. Check to see whether the dependency is already synced in your local $GOPATH. If it is, ensure that you have made no local modifications to this dependency.
2. Update the version in the repos map in ciao-vendor.go.
3. cd $GOPATH/src/github.com/01org/ciao
4. Delete the existing repo under the vendor directory. So if we were going to update github.com/docker/distribution we'd do rm -rf vendor/github.com/docker/distribution
5. go run ciao-vendor/ciao-vendor.go vendor
6. go run ciao-vendor/ciao-vendor.go check
7. Commit the changes to ciao-vendor.go and the vendor directory

How do I figure out who is using a given package?

It can be useful to know which of the ciao and vendored packages depend on a given package. For example, you might wish to remove a 3rd party dependency from ciao to reduce the maintenance overhead of the vendored packages. You can only really do this if you know that the package in question is used by ciao components. If it's also needed by a vendored package, you there's no point in removing it unless you also submit a PR to the upstream vendored package. ciao-vendor includes a command that prints the clients of a given package.

For example,

go run ciao-vendor/ciao-vendor.go uses github.com/01org/ciao/vendor/github.com/Sirupsen/logrus

outputs

github.com/01org/ciao/ciao-launcher
github.com/01org/ciao/vendor/github.com/docker/engine-api/client
github.com/01org/ciao/vendor/github.com/docker/go-connections/tlsconfig

The packages that are outputted by this command use github.com/01org/ciao/vendor/github.com/Sirupsen/logrus either directly or indirectly.

To see only the packages that are directly dependant on the specified package, use the -d switch.

How do I figure out which vendored dependencies are out of date?

Run

go run ciao-vendor/ciao-vendor.go updates

This should print a table indicating which vendored dependencies trail master, e.g.

Package                                 Status				
github.com/Sirupsen/logrus              46 commits behind HEAD		
github.com/boltdb/bolt                  16 commits behind HEAD		
github.com/coreos/go-iptables           Up to date			
github.com/docker/distribution          166 commits behind HEAD		
github.com/docker/docker                3611 commits behind HEAD	
github.com/docker/engine-api            190 commits behind HEAD		
github.com/docker/go-connections        4 commits behind HEAD		
github.com/docker/go-units              13 commits behind HEAD		
github.com/docker/libnetwork            174 commits behind HEAD		
github.com/golang/glog                  Up to date			
github.com/gorilla/context              4 commits behind HEAD		
github.com/gorilla/mux                  13 commits behind HEAD		
github.com/mattn/go-sqlite3             7 commits behind HEAD		
github.com/mitchellh/mapstructure       Up to date			
github.com/opencontainers/runc          233 commits behind HEAD		
github.com/rackspace/gophercloud	        11 commits behind HEAD		
github.com/tylerb/graceful              13 commits behind HEAD		
github.com/vishvananda/netlink          3 commits behind HEAD		
github.com/vishvananda/netns            Up to date			
golang.org/x/net	                        78 commits behind HEAD		
gopkg.in/yaml.v2	                        Up to date		

Note this command can take some time to execute as it needs to first update the dependencies in your $GOPATH. Some of the dependencies, like docker, can take a long time to sync.