defaults not superseeded by environment

[EugenMayer]

i seem to simply to grasp you way of overlaying. It kinda drives me mad every time i try to do anything with defaults and consul or environment
having this order of sources data_sources: [ "defaults", "file", "environment"]
lets call it that way http://pastebin.com/E8P2BpWt .. now i set an env ware export db_password=test and run tiller with environment=production and getting

 Warning, merging duplicate data values.
test_db_password => 'test' being replaced by : '' from DefaultsDataSource

So once again, the default superseeds the environment value, and that does simply not make sense to me. Defaults are the static values you want to ensure having set, if nbobody does anything. Environemt vars are specific values you want to have propagated, since they are concrete, dynamic

I think this has a lot to do with #26 (comment) and while i understand what you explain about template values, you should really evaluate, if you want to tree template values defined in DEFAULTS to superseed ENV

To be honest, the question is, whether the environement vars should be ever overriden when defined as above, since they are on the right - superseed.

So yes, i can solve the issue by moving anything in defaults->template->config to ->defaults->globals but thats simply not the solution, since you can have several templates which now will conflict probably. Defining 'config' in defaults for a template is like namespacing them

[markround]

OK, I see your point. For my usage, the current behaviour suits me so, but we can make this a configurable option. How about this solution:

I introduce a new remap_globals option, if this is set to true in common.yaml, then any plugin (like environment) that usually only exposes a global value will get their global values also exposed as template values. This will ensure that any value specified here will always take precedence.

I've got this working in a test branch now - and also added a warn_on_merge flag, as otherwise you'll get a lot of noise when using the environment plugin. So, here's what common.yaml looks like in my test:

---
exec: [ 'cat' , 'test.txt' ]
data_sources: [ "defaults" , "file" , "environment" ]
template_sources: [ "file" ]
remap_globals: true
warn_on_merge: false
defaults:
  test.erb:
    env_var: "From defaults module"

environments:
  development:
    test.erb:
      target: test.txt

And when I run var='From environment' ./bin/tiller -b ./test -v, I get:

tiller v0.8.0 (https://github.com/markround/tiller) <[email protected]>
Using common.yaml v2 format configuration file
Using configuration from ./test
Using plugins from /usr/local/lib/tiller
Using environment development
Template sources loaded [FileTemplateSource]
Data sources loaded [DefaultsDataSource, FileDataSource, EnvironmentDataSource]
Available templates : ["test.erb"]
Building template test.erb
Not running as root, so not setting ownership/permissions on test.txt
Template generation completed
Executing ["cat", "test.txt"]...
Child process forked with PID 82174.
This is the test variable : From environment
Child process exited with status 0
Child process finished, Tiller is stopping.

And when leaving out the remap_globals option, we revert to the previous behaviour:

tiller v0.8.0 (https://github.com/markround/tiller) <[email protected]>
...
Warning, merging duplicate data values.
env_var => 'From environment' being replaced by : 'From defaults module' from DefaultsDataSource
...
Executing ["cat", "test.txt"]...
Child process forked with PID 82506.
This is the test variable : From defaults module
...

[EugenMayer]

Hey @markround thanks for looking at this.

First off, having this as an opt-in is perfectly fine fore me.

Though, i would still prefer to talk about this topic a bit more general. When you try to define the groups of data-sources into semantic groups, you might end up having kind of this definition:

static sources (when deployed, unchangeable - same for all tiller instances):

1. template config (environment speciifc) (common.yml)
2. defaults (globals or template) (common.yml)

dynamic sources (can change values during runtime/start)

1. Environment variables
2. KV Sources (etcd/consul/zookeeper)
3. Files ( yml files or similar )

While there can be discussion, how overriding should work in the inner of the groups ( globals always get overriden by template config, obviously, e.g.) there is a natural order on tht dynamic values should always superseed static values - without any exception.

The reason for that is, that tillers common.yml and template can be seen as a base, deployed to any kind of image/container/barebone. Then, the specific configuration for the environment of this container/barebone happens, and this always happens by the dynamic values.

The static values try to ensure a common ground or a fallback, if nothing else defined, while the dynamic sources do adjust to the actual runtime state and use-case.

This cannot, ever, be the other way arround.

Saying this, consul/etcd should always override anything default defines, same goes for environment values.

consul special case
Since consul can define its own globals/defaults, this need to be defined. The same rules apply, while consul defaults override static defaults, consul defaults get overriden by any environment specific defintion / template values .

There can be a lot of arguing, whether env variables should override consul/etcd, because it will mostly be going down to the scenario. In development, you want to easily recondigure the stack using the compose file, while in production, you will never ever try to mix up consul and env variables, you will always go with either of those. This said, there might be no obvious rule here, so it should be a configuration variable.

I hope you understand my effort as "it should not fit my specific use case" but rather "how should tiller work in general"

[markround]

Released 0.9.0 which fixes this behaviour, as discussed in the Gitter channel.