diff --git a/CHANGE.md b/CHANGE.md new file mode 100644 index 00000000..71caf71d --- /dev/null +++ b/CHANGE.md @@ -0,0 +1,12 @@ +v0.0.18 +* Added ability to filter paths from the set of files copied to the guest + See https://github.com/simonmcc/kitchen-salt/blob/master/provisioner_options.md#salt_copy_filter +* Added is_file_root flag - treat this project as a complete file_root. + See https://github.com/simonmcc/kitchen-salt/blob/master/provisioner_options.md#is_file_root +* Pillar data specified using the pillars-from-files option are no longer + passed through YAML.load/.to_yaml + This was causing subtle data transformations with unexpected results when + the reuslting yaml was consumed by salt +* Added "Data failed to compile" and "No matching sls found for" to + strings we watch salt-call output for signs of failure + diff --git a/README.md b/README.md index 66f31118..06a577d6 100644 --- a/README.md +++ b/README.md @@ -5,20 +5,29 @@ The provider works by generating a salt-minion config, creating pillars based on This provider has been tested against the Ubuntu boxes running in vagrant/virtualbox & vagrant-lxc boxes on Ubuntu. -## Requirements -You'll need a driver box that is supported by both the SaltStack [bootstrap](https://github.com/saltstack/salt-bootstrap) system & the Chef Omnibus installer (the Chef Omnibus installer is only needed to provide busser with a useable ruby environment, you can tell busser to use an alternative ruby if your box has suitable ruby support built in). - - ## Installation & Setup You'll need the test-kitchen & kitchen-salt gem's installed in your system, along with kitchen-vagrant or some ther suitable driver for test-kitchen. Please see the [INTRODUCTION](https://github.com/simonmcc/kitchen-salt/blob/master/INTRODUCTION.md). -# Releasing +## Catching salt failures +Catching salt failures is particularly troublesome, as salt & salt-call don't do a very good job of setting the exit +code to something useful, around ~0.17.5, the `--retcode-passthrough` option was added, but even in 2014.1.0 this is +still flawed, [PR11337](https://github.com/saltstack/salt/pull/11337) should help fix some of those problems. In the +mean time, we scan the salt-call output for signs of failure (essentially `grep -e Result.*False -e Data.failed.to.compile -e +No.matching.sls.found.for`) and check for a +non-zero exit code from salt-call. + + +## Requirements +You'll need a driver box that is supported by both the SaltStack [bootstrap](https://github.com/saltstack/salt-bootstrap) system & the Chef Omnibus installer (the Chef Omnibus installer is only needed to provide busser with a useable ruby environment, you can tell busser to use an alternative ruby if your box has suitable ruby support built in). + +## Releasing # hack. work. test. git add stuff gem bump --no-commit git add lib/kitchen-salt/version.rb git commit -v + git push gem release diff --git a/docs/provisioner.md b/docs/provisioner.md deleted file mode 100644 index 658ff85a..00000000 --- a/docs/provisioner.md +++ /dev/null @@ -1,85 +0,0 @@ -# kitchen-salt provisioner configuration - -The provisioner block is used to select & configure many aspects of the provisioner being used by test-kitchen, the following options are supported by kitchen-salt: - -* name -* formula -* salt_version -* salt_install -* salt_bootstrap_url -* salt_bootstrap_options -* salt_apt_repo -* salt_apt_repo_key -* chef_bootstrap_url -* salt_config -* salt_minion_config -* salt_file_root -* salt_pillar_root -* salt_state_top -* salt_run_highstate -* state_collection -* state_top -* pillars -* pillars-from-files - -### pillars - -define the pillars you want supplied to salt, you must define top.sls so that any subsequent pillars are loaded: - - pillars: - top.sls: - base: - '*': - - beaver - beaver.sls: - beaver: - transport: tcp - -### pillars-from-files - -This options allows you to load a pillar yaml file from disk, instead of it having to be declared in full in .kitchen.yml, this also allows you to re-use any pillar.example files you are distributing with your formula: - - provisioner: - pillars-from-files: - beaver.sls: pillar.example - pillars: - top.sls: - base: - '*': - - beaver -And the contents of pillar.example is a normal pillar file: - - $ cat pillar.example - # defaults are set in map.jinja and can be over-ridden like this - beaver: - transport: stdout - format: json - - -## Global and per-suite configuration -The kitchen-salt provisioner can be configured either globally or on a per suite basis, the same as any other test-kitchen provisioner. - - provisioner: - name: salt_solo - formula: beaver - - suites: - - name: default - provisioner: - pillars: - top.sls: - base: - '*': - - beaver - beaver.sls: - beaver: - transport: tcp - state_top: - base: - '*': - - beaver - - beaver.ppa - - -# Catching salt failures -Catching salt failures is particularly troublesome, as salt & salt-call don't do a very good job of setting the exit code to something useful, around ~0.17.5, the `--retcode-passthrough` option was added, but even in 2014.1.0 this is still flawed, [PR11337](https://github.com/saltstack/salt/pull/11337) should help fix some of those problems. In the mean time, we scan the salt-call output for signs of failure (`grep -e Result.*False` bascially) and check for a non-zero exit code from salt-call. \ No newline at end of file diff --git a/docs/ruby-for-the-tinhat.md b/docs/ruby-for-the-tinhat.md new file mode 100644 index 00000000..6f31c3f8 --- /dev/null +++ b/docs/ruby-for-the-tinhat.md @@ -0,0 +1,16 @@ +RVM give you the willies? + + +Try this, start with Debian Testing/Jessie and add the bits of ruby we need: + + # PREREQ: install Vagrant from http://www.vagrantup.com/ + # install enough for a sensible ruby execution environment + sudo apt-get install ruby bundler git + +Now lets get a sample salt repo that's been prepared for test-kitchen: + + git clone https://github.com/simonmcc/beaver-formula.git + cd beaver-formula/ + + + diff --git a/lib/kitchen-salt/version.rb b/lib/kitchen-salt/version.rb index aed163bc..b1752b6c 100644 --- a/lib/kitchen-salt/version.rb +++ b/lib/kitchen-salt/version.rb @@ -1,5 +1,5 @@ module Kitchen module Salt - VERSION = "0.0.17" + VERSION = "0.0.18" end end diff --git a/lib/kitchen/provisioner/salt_solo.rb b/lib/kitchen/provisioner/salt_solo.rb index be0a2066..23c95050 100644 --- a/lib/kitchen/provisioner/salt_solo.rb +++ b/lib/kitchen/provisioner/salt_solo.rb @@ -53,7 +53,7 @@ class SaltSolo < Base default_config :state_top, {} default_config :state_top_from_file, false default_config :salt_run_highstate, true - default_config :salt_copy_filter, ['.git', '.svn', '.kitchen'] + default_config :salt_copy_filter, [] default_config :is_file_root, false @@ -184,8 +184,7 @@ def run_command end # scan the output for signs of failure, there is a risk of false negatives - fail_grep = 'grep -e Result.*False -e Data.failed.to.compile' - fail_grep = 'grep -e Result.*False' + fail_grep = 'grep -e Result.*False -e Data.failed.to.compile -e No.matching.sls.found.for' # capture any non-zero exit codes from the salt-call | tee pipe cmd = 'set -o pipefail ; ' << cmd # Capture the salt-call output & exit code @@ -408,7 +407,10 @@ def cp_r_with_filter(source_path, target_path, filter=[]) debug("cp_r_with_filter:source = #{source}") debug("cp_r_with_filter:target = #{target}") if File.directory? source - Find.prune if filter.include?(File.basename(source)) + if filter.include?(File.basename(source)) + debug("Found #{source} in #{filter}, pruning it from the Find") + Find.prune + end FileUtils.mkdir target unless File.exists? target else FileUtils.copy source, target diff --git a/provisioner_options.md b/provisioner_options.md index 82535a89..639920e2 100644 --- a/provisioner_options.md +++ b/provisioner_options.md @@ -4,6 +4,7 @@ key | default value | Notes ----|---------------|-------- formula | | name of the formula, used to derive the path we need to copy to the guest +[is_file_root](#is_file_root) | false | Treat this project as a complete file_root, not just a state collection or formula salt_install| "bootstrap" | Method by which to install salt, "bootstrap" or "apt" salt_bootstrap_url | "http://bootstrap.saltstack.org" | location of bootstrap script [salt_bootstrap_options](#salt_bootstrap_options) | | optional options passed to the salt bootstrap script @@ -12,6 +13,7 @@ salt_apt_repo | "http://apt.mccartney.ie"| apt repo salt_apt_repo_key| "http://apt.mccartney.ie/KEY"| apt repo key chef_bootstrap_url| "https://www.getchef.com/chef/install.sh"| the chef bootstrap installer, used to provide Ruby for the serverspec test runner on the guest OS salt_config| "/etc/salt"| +[salt_copy_filter](#salt_copy_filter) | [] | List of filenames to be excluded when copying states, formula & pillar data down to guest instances. salt_minion_config| "/etc/salt/minion"| salt_file_root| "/srv/salt"| salt_pillar_root| "/srv/pillar"| @@ -53,10 +55,70 @@ The provisioner can be configured globally or per suite, global settings act as provisioner: salt_version: 0.16.2 salt_install: apt - + + - name: tcp-output + provisioner: + pillars: + top.sls: + base: + '*': + - beaver + beaver.sls: + beaver: + transport: tcp + in this example, the default suite will install salt via the bootstrap method, meaning that it will get the latest package available for the platform via the [bootstrap shell script](http://bootstrap.saltstack.org). We then define another suite called `default_0162`, this has the provisioner install salt-0.16.2 via apt-get (this defaults to a mini repo of mine, which you can override, my repo only contains 0.16.2) ### [formula](id:formula) +When running in normal mode, this must be set to the name of the formula your testing, this name is used to derive the name of the directory that should copied down to the guest. + +For a project layout like this: + + .kitchen.yml + beaver/init.sls + beaver/foo.sls + +formula should be set to ```beaver``` + + +If you want all files & directories copied down to the host, see the [is_file_root](#is_file_root) option. +### [is_file_root](id:is_file_root) +Setting the ```is_file_root``` flag allows you to work with a directory tree that more closely resembles a built file_root on a salt-master, where you have may have multiple directories of states or formula. The project is recursively copied down to guest instance, excluding any hidden files or directories (i.e. .git is not copied down, this is the standard behaviour of ruby's FileUtil.cp_r method) + +Consider a directory that looks like this: + + top.sls + .kitchen.yml + apache/init.sls + mysql/init.sls + mysql/client.sls + mysql/server.sls + php/init.sls + ... + +With a .kitchen.yml like this you can now test the completed collection: + + --- + driver: + name: vagrant + + provisioner: + name: salt_solo + is_file_root: true + state_top: + base: + '*': + - apache + - mysql.client + + platforms: + - name: ubuntu-12.04 + + suites: + - name: default + +In this example, the apache state could use functionality from the php state etc. You're not just restricted to a single formula. + ### [salt_install](id:salt_install) ### [salt_bootstrap_options](id:salt_bootstrap_options) Options to pass to the salt bootstrap installer. For example, you could choose to install salt from the develop branch like this: @@ -68,6 +130,22 @@ Options to pass to the salt bootstrap installer. For example, you could choose Details on the various options available at the [salt-bootstrap](https://github.com/saltstack/salt-bootstrap/blob/develop/bootstrap-salt.sh#L180) documentation. +### [salt_copy_filter](id:salt_copy_filter) +When kitchen copies states, formula & pillars down to the guests it creates to execute the states & run tests against, you can filter out paths that you don't want copied down. The copy is conducted by ruby's FileUtils.cp method, so all hidden directories are skipped (e.g. ```.git```, ```.kitchen``` etc). + +You can supply a list of paths or files to skip by setting an array in the provisioner block: + + + suites: + - name: copy_filter_example + provisioner: + salt_copy_filter: + - somefilenametoskip + - adirectorythatshouldbeskipped + + + + ### [salt_version](id:salt_version) Version of salt to install, via the git bootstrap method, unless ```salt_install``` is set to ```apt```, in which case the version number is used to generate the package name requested via apt @@ -110,6 +188,7 @@ Instead of rendering ```top.sls``` on the guest from the definition in .kitchen. ### [state_collection](id:state_collection) +Setting the ```state_collection``` flag to true makes kitchen-salt assume that the state files are at the same level as the ```.kitchen.yml```, unlike a formula, where the states are in a directory underneath the directory containing ```.kitchen.yml```. When using ```state_collection:true```, you must also set the [collection_name](#collection_name). ### [collection_name](id:collection_name) When dealing with a collection of states, it's necessary to set the primary collection name, so that when we call salt-call in the guest, the states have been put into directory that matches the name referenced in the state_top, for example, consider this simple logrotate state collection: @@ -143,6 +222,17 @@ In order for salt-call to be able to find the logrotate state and apply init.sls ### [pillars](id:pillars) +define the pillars you want supplied to salt, you must define top.sls so that any subsequent pillars are loaded: + + pillars: + top.sls: + base: + '*': + - beaver + beaver.sls: + beaver: + transport: tcp + ### [pillars-from-files](id:pillars-from-files) The pillars-from-files option allows you to load pillar data from an external file, instead of being embedded in the .kitchen.yml. This allows you to re-use the example files or reduce the clutter in your .kitchen.yml @@ -151,14 +241,23 @@ Consider the following suite definition: - name: tcp-output-external-pillar provisioner: pillars-from-files: - beaver.sls: beaver-example.sls + beaver.sls: pillar.example pillars: top.sls: base: '*': - beaver + + +And the contents of pillar.example is a normal pillar file: + + $ cat pillar.example + # defaults are set in map.jinja and can be over-ridden like this + beaver: + transport: stdout + format: json + -In this example, the beaver pillar is loaded from the example file in the repo, ``beaver-example.sls``, but we can still define the ``top.sls`` inline in the .kitchen.yml file. ### [grains](id:grains) (since v0.0.15)