adding kitchen tests to validate basic driver functionality

[mwrock]

This adds Test-Kitchen for validating basic driver functionality. It adds a synced folder for the parent gem directory so that tests are run using the "developed" gem and not what's on rubygems. I created this for my other PR #56 but think its best to submit this separately. This PR tests the following:

• Creation of a container
• Creation of a container via from_image
• Destroying a container
• Convergence and port listening

My PR #56 will add testing of action :stop and the machine_execute resource.

I think this is also helpful because it creates a working docker environment and therefore can be run on a more vanilla environment (even Windows) that does not have docker or its prerequisites installed.

This uses the hashicorp/precise64 vagrant box instead of the chef ubuntu box because the hashicorp image supports more providers, notably Hyper-V which I use for personal projects.

[mwrock]

There is a considerable amount of what I think is "test cruft" in this project. This includes what was previously in /test and perhaps x.rb I left those in tact, but unless they are being actively used I think they should be removed.

[marc-]

Why not chef_gem ? Test-kitchen will install chef omnibus with embedded ruby on VM, so there should be no ruby installed in system. No sure how gem_package handle this, but chef_gem will install it using that embedded ruby.

[mwrock]

I tried that and had issues getting it to load from a path. Seemed like that should have worked but even if it does, I don't believe it can install from raw source (a .gemspec and not .gem)

[marc-]

I was using following:

execute '/opt/chef/embedded/bin/gem build chef-provisioning-docker.gemspec' do
  cwd '/tmp/chef-provisioning-docker'
  action :nothing
end.run_action(:run)

chef_gem 'chef-provisioning-docker' do
  source Dir[ '/tmp/chef-provisioning-docker/*.gem' ].first
  compile_time true if Chef::Resource::ChefGem.instance_methods(false).include?(:compile_time)
  action :install
end

[mwrock]

ahh right I didn't read all of this and had already forgotten that I ended up using gem_package instead of gem install from the batch resource. I cant remember all the details but I had alot of problems with this whole gem install. I cant remember if it was chef_gem but even with doing this at compile time, the gem was not available for the below require however, if I immediately SSH'd to the box or simply ran the script a second time, it worked.

Finally this "just worked" and I left it but am happy to give chef_gem another shot.

[mwrock]

Just added a couple fixes for running tests on centos. However, I am having issues adding a synced folder on centos in VirtualBox. It looks like others are having issues with this as well and are blaming it on either the version of guest tools in the image or a bug in a recent VirtualBox release. I have tried several ways to fix this and the only working solution is running a yum upgrade -y after the initial kitchen create and then a vagrant reload. Then the chef-docker cookbook fails in the docker.socker.erb ultimately resulting in a failure installing docker. Looks like this is fixed in master of that cookbook but not yet in the supermarket.

Unless anyone thinks of a better idea, I'd propose removing centos from the vagrant platforms until these are fixed. I think just having ubuntu support provides enough value but having centos support would be really nice too. The ubuntu image is working for me.

We could also just install docker manually. This test suite just needs a no frills install but we'd still be left with a broken base box.

[marc-]

Didn't have this issue with CentOS 7. Though, didn't update vagrant box for quite some time. Will try to run your patch om my environment.

[mwrock]

Well I found a working centos image that has docker preinstalled: box-cutter/centos65-docker

Still need to tweak something for it since I get this when attempting to use docker:

Connection refused - connect(2) for /var/run/docker.sock (Errno::ECONNREFUSED)

If I recall correctly this has something to do with TLS. I'll look into this tonight but looks like we can get centos to work.

[mwrock]

heh. actually this is just because I'm an idiot :). Need to actually start docker

[marc-]

How about using Chef logging instead?

[randomcamel]

kitchen test default-ubuntu-1204 fails for me on OS X (ChefDK 0.6.0). Let me know if it's meant to run under other conditions.

       Chef Client failed. 12 resources updated in 224.931774463 seconds
       [2015-06-18T19:43:44+00:00] ERROR: machine[ssh1] (docker-tests::create-containers line 1) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
       ---- Begin output of docker run --name ssh1 6d4946999d4fb403f40e151ecbd13cb866da125431eb1df0cdfd4dc72674e3c6 mkdir -p /etc/chef ----
       STDOUT:
       STDERR: Error response from daemon: Cannot start container 1b4bd9cb3a79a77b9649a64c08e3e97398c77beb63a0f00072d11f279a85aaa3: device or resource busy
       ---- End output of docker run --name ssh1 6d4946999d4fb403f40e151ecbd13cb866da125431eb1df0cdfd4dc72674e3c6 mkdir -p /etc/chef ----
       Ran docker run --name ssh1 6d4946999d4fb403f40e151ecbd13cb866da125431eb1df0cdfd4dc72674e3c6 mkdir -p /etc/chef returned 1
       [2015-06-18T19:43:45+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
>>>>>> Converge failed on instance <default-ubuntu-1204>.
>>>>>> Please see .kitchen/logs/default-ubuntu-1204.log for more details
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: SSH exited (1) for command: [sh -c '

sudo -E /opt/chef/bin/chef-client --local-mode --config /tmp/kitchen/client.rb --log_level auto --force-formatter --no-color --json-attributes /tmp/kitchen/dna.json --chef-zero-port 8889
']
>>>>>> ----------------------

[mwrock]

Yeah thuis morning I noticed that ubuntu was having issues on virtualbox on my linux box but working on hyper-v on my windows box. Made some changes (removing the docker cookbook and just installing the packages myself) which got it al working on my linux box/virtualbox but then started to see this exact error on my windows box. So I'm gonna spend some more time stabilizing it to work on both platforms tonight.

[randomcamel]

Does Docker and chef-provisioning-docker behave differently on different platforms? It'd be a lot less work to just run RSpecs on whichever host platform.

[marc-]

That's pretty confusing. Docker works only on GNU/Linux. What you have on Windows/MacOS is VirtualBox VM with GNU/Linux (based on Tiny Core Linux) managed by boot2docker. So, why do we talk about Windows/MacOS at all? Do we have use cases for this? Or we just need ability to run test-kitchen on any system?
upd: I probably need to educate myself on boot2docker.

[mwrock]

Right docker only works on linux, this uses vagrant to create a ubuntu or centos box regardless of the physical host that can act as a docker host. This can get tricky because different vagrant provider images even for the same vagrant box can be slightly different. From what I've seen, these differences are usually unimpactful but I am seeing different behavior here. Differences can also be impacted by hypervisor config. For example VirtualBox uses NAT by default and Hyper-V uses a virtual switch.

My objective in this PR is to be able to have kitchen/vagrant create a docker host where one can reliably test driver changes and validate that the results are what one expects. I think this is possible, there are just some nuances that are a pain.

[randomcamel]

@mwrock I've got #51 near to merging; I don't see much overlap yet, but we should keep an eye on it. The TK stuff looks more about making a safe and reliable dev environment, so I'd like to make sure any specs on the driver functionality itself can also be run without the TK harness, on e.g. Linux or OS X where the dev (or CI) environment is less quirky.

[mwrock]

I have rebased master and squashed my commits. I have also gotten the default test recipe into a state where it reliably creates a docker environment on both my windows and linux laptops. Both the ubuntu and centos platforms are working. (the centos vagrant box has no Hyper-V provider so I have only validated it on linux). I'm confident this would likely work on a mac as well now and would be interested to know if anyone has trouble.

@randomcamel, @marc- I wasn't thinking this would serve as "official" CI. Rather, I was hoping this would make it easier to:

1. Provide a reliable, reproducable and disposable environment that any contributor could use to test their work.
2. Provide a demo environment that makes it easy for people to experiment with docker provisioning.

As I was working on my PR #56, I found it cumbersome to have to cleanup my images and containers manually after each run and to perform container inspections to make sure the changes worked. Also, because the containers converge supermarket cookbooks, TK provides Berkshelf integration so I dont have to muck with vendoring cookbooks. I'm definitely a fan of rspec and have leaned heavily on non-kitchen integration tests developing my chef-provisioning-vsphere driver. Because the docker driver implements a separate transport and machine, it is nice having an extra layer of testing that uses actual recipes and resources.

I started working on #56 because I want to blog about chef orchestration patterns - something I find that chef-provisioning can facilitate well and I think the docker driver serves as an ideal "example" driver because its light weight, requires no hosting expenses, and can be hosted in a VM on any platform. Side note: without 56, I cant demo reconverges that dont shut down a node,or how to tell a fleet of machines to run an arbitrary command.

Someone should ideally be able to simply pull the repo, run kitchen converge and now you skip having to tell them how to install docker or explain berks.

Hope that makes sense but I totally respect the decision of this repo's maintainers if having kitchen tests is not desirable.

[marc-]

Having TK makes perfect sense too me. Use it on daily bases with chef-provisioning-docker myself to test cookbooks. I believe it is good addition to specs.

[mwrock]

Trued up this PR with latest commits.

[randomcamel]

CentOS passes, but the Ubuntu VM still fails for me:

       [2015-07-29T18:29:46+00:00] ERROR: machine[ssh1] (docker-tests::create-containers line 1) had an error: Errno::ENOENT: No such file or directory @ rb_sysopen - proc/14372/root/etc/chef/client.pem
       [2015-07-29T18:29:48+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

I can gist the stacktrace if that's useful. I'm not opposed to just merging this even though it fails for me, if it passes on someone else's OS X host.

[mwrock]

yeah if you could gist the stack trace that would be awesome.

[mwrock]

nevermind. I reproduced on an ubuntu host. Will fix.

[mwrock]

so it looks like hashicorp/precise64 has different kernel versions for the Hyper-V and VirtualBox providers. The VirtualBox provider has the correct precise kernel but it needs a reboot after upgrading. So the kitchen run is running on an older kernel and that messes with the docker file system.

I guess we'll just use the 14.04 bento box to avoid this. Sadly that has no Hyper-V provider but the vast majority of users will be unaffected and I have access to ubuntu hosts. I may contribute to the bento project and add Hyper-V providers to all boxes using this technique.

@randomcamel : If you dont mind giving the ubuntu suite another shot and if it succeeds, I'd consider this good to go. Its worked for me on my ubuntu laotop.

[randomcamel]

The Ubuntu run is still failing for me. I've got TK 1.3.1 + Vagrant 1.7.2 + chef-provisioning-docker 0.7 on OS X--is it possible there's an issue in there somewhere?

@tyler-ball or @fnichol, can one of you try kitchen test on this branch? If it passes on someone's OS X machine, we can chalk it up to environment issues on my end and get this merged finally.

(We could add it to the Travis run, but I don't think that would be easier to debug.)

       Chef Client failed. 19 resources updated in 374.617123574 seconds
       [2015-08-05T15:14:51+00:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out
       [2015-08-05T15:14:51+00:00] ERROR: machine[ssh1] (docker-tests::create-containers line 1) had an error: Docker::Error::TimeoutError: read timeout reached
       [2015-08-05T15:14:52+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

[randomcamel]

Just tried it again on OS X and it passed, so I'll assume I saw transient errors. Thanks for the work, and for your patience in getting it merged.

[mwrock]

Woo hoo! Thanks!!