Skip to content

Commit

Permalink
Support recursive config includes; some small bugfixes
Browse files Browse the repository at this point in the history
  • Loading branch information
Ryan Jung committed Feb 5, 2021
1 parent 7b619e8 commit 0fd36f6
Show file tree
Hide file tree
Showing 5 changed files with 76 additions and 17 deletions.
12 changes: 11 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,15 @@ Retemplate is configured with a YAML file consisting of three main sections:
* `templates` (which files get worked over)

### Global Settings
Global settings come under the `retemplate` section. Currently, the only globally adjustable setting is `logging`, wherein you can supply options to pass into the Python logger library's [basicConfig function](https://docs.python.org/3/library/logging.html#logging.basicConfig). [config.yml.example](config.yml.example) shows a few simple options.
Global settings come under the `retemplate` section.

#### logging
In the `logging` section, you can supply options to pass into the Python logger library's [basicConfig function](https://docs.python.org/3/library/logging.html#logging.basicConfig). [config.yml.example](config.yml.example) shows a few simple options.

#### include
This is a list of other YAML config files to include. Regardless of where in your config file this option appears, Retemplate will always interpret them in the order listed, **after** completing the interpretation of the config file the directive is found in. Options found in those config files will override conflicting options discovered beforehand. In other words, configs read by include directives will take precedence.

You can use regular expressions to indicate inclusion of all files matching the string. For example, to include all YAML files in a directory, you could use `/etc/retemplate/conf.d/*.yml`. Any value supported by Python's [glob.glob function](https://docs.python.org/3/library/glob.html#glob.glob) is valid here.

### Data Stores
The `stores` section defines your data stores, which are services or functions that retrieve data for templating purposes. There are currently five types of data stores, each with their own configuration options. In the configuration file, these are defined by a dictionary entry where the key is the name of the data store as it will be referenced later in templates and the value is a dictionary of configuration options to be passed into that data store. Although specific configurations may vary between data stores, they all have a `type`, defined below.
Expand Down Expand Up @@ -169,6 +177,8 @@ Here, `templates` is the root-level option underneath which all of your template
* **frequency**: *The number of seconds to wait between template renders.*
* **random_offset_max**: *When present, this causes the rendering process to wait an additional amount of time - up to this number of seconds - before it gets underway. This is designed to prevent the alignment of jobs such that they all make API calls or disk access requests simultaneously. If not set, there is no additional time offset.*

The `owner`, `group`, and `chmod` options are technically optional. If you do not supply them, the OS will leave these at the defaults for the user that Retemplate is running as. Because this software often runs as `root`, that could leave your files unreadable by the programs that need to use them. It is recommended that you explicitly set these options.

## Writing Templates
The template rendering process is a three-stage one:

Expand Down
2 changes: 2 additions & 0 deletions config.yml.example
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ retemplate:
level: INFO
filename: retemplate.log
format: '%(levelname)s,%(asctime)s,%(funcName)s,%(message)s'
include:
- /etc/retemplate/config.d/*.yml
stores:
ec2-metadata:
type: aws-local-meta
Expand Down
20 changes: 14 additions & 6 deletions retemplate/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -220,12 +220,14 @@ def __init__(self, name, *args, **kwargs):
def get_value(self, key, **kwargs):
try:
subp_args = [ self.command ]
subp_args.extend(kwargs['arg'])
if 'arg' in kwargs:
subp_args.extend(kwargs['arg'])

# Clean these args up a bit
for i in range(0, len(subp_args)):
subp_args[i] = subp_args[i].strip()

logging.debug('Running command: {}'.format(' '.join(subp_args)))
proc = subprocess.run(subp_args, capture_output=True)
output = proc.stdout.decode('utf-8').strip()
return output
Expand Down Expand Up @@ -256,7 +258,7 @@ def __init__(self, target, template, stores, display_values, **kwargs):
self.settings = {
'owner': None,
'group': None,
'chmod': 600,
'chmod': None,
'onchange': None,
'frequency': 60,
'random_offset_max': None
Expand Down Expand Up @@ -426,12 +428,18 @@ def write_file(self, content):
'''

try:
logging.info("Writing {}; setting ownership to {}:{} and mode to {}".format(
self.target, self.settings['owner'], self.settings['group'], self.settings['chmod']))
logging.info('Writing {}'.format(self.target))
with open(self.target, 'w') as fh:
fh.write(content)
shutil.chown(self.target, user=self.settings['owner'], group=self.settings['group'])
subprocess.run([ 'chmod', self.settings['chmod'], self.target ])
owner = self.settings['owner'] if 'owner' in self.settings else None
group = self.settings['group'] if 'group' in self.settings else None
chmod = self.settings['chmod'] if 'chmod' in self.settings else None
if owner or group:
logging.info('Setting ownership of {} to {}:{}'.format(self.target, owner, group))
shutil.chown(self.target, user=owner, group=group)
if chmod:
logging.info('Setting mode of {} to {}'.format(self.target, chmod))
subprocess.run([ 'chmod', self.settings['chmod'], self.target ])
return True
except IOError:
logging.error('Cannot write target file {}'.format(self.target))
Expand Down
57 changes: 48 additions & 9 deletions rtpl
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#!/bin/env python3

import glob
import logging
import json
import retemplate
Expand All @@ -14,6 +15,8 @@ This file serves as an entry point to the module's features, and is the recommen
executing templates.
'''

all_configs = list()

def parseargs():
'''
Interpets command line arguments
Expand All @@ -32,6 +35,43 @@ def parseargs():
action='store_true')
return parser.parse_args()

def read_config(path):
try:
with open(path, 'r') as fh:
return yaml.load(fh.read(), Loader=yaml.FullLoader)
except IOError:
logging.error('Could not read config file: {}'.format(args.config))
sys.exit(1)


def get_configs(includes):
if len(includes) > 0:
for include in includes:
files = glob.glob(include)
for file in files:
config = read_config(file)
all_configs.append(config)
if 'retemplate' in config and 'include' in config['retemplate']:
get_configs(config['retemplate']['include'])

def deep_merge(d, u):
for k, v in u.items():
if isinstance(v, dict):
d[k] = deep_merge(d.get(k, {}), v)
else:
d[k] = v
return d

def merge_configs(base_config):
# logging.info('Base config: {}'.format(json.dumps(base_config, indent=2)))
for config in all_configs:
# logging.info('Merging in config: {}'.format(json.dumps(config, indent=2)))
base_config = deep_merge(base_config, config)
# logging.info('Base config after merge: {}'.format(json.dumps(base_config, indent=2)))
if 'retemplate' in base_config and 'include' in base_config['retemplate']:
del(base_config['retemplate']['include'])
return base_config

def main():
'''
Main entry point to the rtpl utility. This handles configuration of the application before
Expand All @@ -54,26 +94,25 @@ def main():
logging.basicConfig(**logcfg)

logging.info('Starting up Retemplate')
config = dict()
stores = dict()
templates = list()
threads = list()

# Parse the config file
logging.info('Parsing config file at {}'.format(args.config))
try:
with open(args.config, 'r') as fh:
config = yaml.load(fh.read(), Loader=yaml.FullLoader)
except IOError:
logging.error('Could not read config file: {}'.format(args.config))
sys.exit(1)
# Parse config files
base_config = read_config(args.config)
if 'retemplate' in base_config and 'include' in base_config['retemplate']:
get_configs(base_config['retemplate']['include'])
config = merge_configs(base_config)

logging.info('Retemplate loaded with this configuration:\n{}'.format(json.dumps(config, indent=2)))

# Update logging config based on user configs if Python allows it
if 'retemplate' in config and 'logging' in config['retemplate'] and sys.version_info.minor >= 8:
logcfg.update(config['retemplate']['logging'])
logging.basicConfig(**logcfg)
logging.debug('Logging configured')


# Configure DataStores
for store in config['stores']:
if config['stores'][store]['type'] == 'aws-local-meta':
Expand Down
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

setup(
name='retemplate',
version='0.0.11',
version='0.0.12',
description="A module to execute a Jinja template on a schedule, supporting several backends for value storage",
url='https://github.com/ryanjjung/retemplate',
author='Ryan Jung',
Expand Down

0 comments on commit 0fd36f6

Please sign in to comment.