Extension for basic Preprocessor class with useful methods and functions.
To use classes and functions from this module, install the utils package it with command
pip3 install foliantcontrib.utilsUsually when we create preprocessors we inherit from BasePreprocessor:
class Preprocessor(BasePreprocessor):
...To use all features of preprocessor_ext we should inherit from BasePreprocessorExt instead:
class Preprocessor(BasePreprocessorExt):
...Usually to process tags in Markdown-files we write something like this:
class Preprocessor(BasePreprocessor):
...
def _process_tags(content):
def sub(match):
# process the tag
return processed_string
self.pattern.sub(sub, content)
def apply(self):
self.logger.info('Applying preprocessor')
for markdown_file_path in self.working_dir.rglob('*.md'):
with open(markdown_file_path, encoding='utf8') as markdown_file:
content = markdown_file.read()
processed_content = self._process_tags(content)
if processed_content:
with open(markdown_file_path, 'w') as markdown_file:
markdown_file.write(processed_content)
self.logger.info('Preprocessor applied')BasePreprocessorExt saves us from writing these many lines of the code which appear unchanged in most preprocessors.
So now instead of all that code we will write:
from foliant.preprocessors.utils.preprocessor_ext import BasePreprocessorExt
class Preprocessor(BasePreprocessorExt):
...
def _process_tag(match):
# process the tag
return processed_string
def apply(self):
self._process_tags_for_all_files(func=self._process_tag, buffer=False)
self.logger.info('Preprocessor applied')Note the buffer=False parameter (which in this case is excessive because it is False by default). If buffer=True, markdown files processing will be buffered, e.g. they won't be updated until all of them are processed.
In your _process_tag function you have access to self.current_filepath (Path of currently processing file), self.current_pos (offset of current processed tag in the document) and self.current_filename (the chapter name, as it is would be stated in chapters foliant.yml section).
When using _process_tags_for_all_files method to process tags we can also take advantage of _warning method.
What this method does:
- Prints warning message to user and adds the md-file name to this message.
- Logs this message.
- May also add to logged message context of the tag where the problem occured.
- May also add to logged message the error traceback.
- If debug=true, #3 and #4 are also added to the message which is printed to user.
Simple example:
class Preprocessor(BasePreprocessorExt):
...
def _process_tag(match):
try:
config = open(self.options.get('config'))
except FileNotFoundError as e:
self._warning('Config file not found! Using default', error=e)
...Here if the exception gets triggered, user will see something like this in console:
Parsing config
Done
Applying preprocessor my_preprocessor
WARNING: [index.md] Config file not found! Using default
Done
Applying preprocessor _unescape
Done
────────────────────
Result: slug.pre
As we see, we've only supplied the message, but the preprocessor also added the WARNING: prefix and current file name to console. If we'd run the make command in debug mode (with -d --debug flag), we would also see full traceback of the error. In any case, traceback is stored in log.
Sometimes we want to show the context of the tag, where we met some problems. By context I mean some words before the tag, some contents of the tag body and some words after the tag. It's really useful for debugging large md-files, with context you usually can identify the place in the document which causes errors.
For this purpose BasePreprocessorExt class has a handy method called get_tag_context. Give it the match object you are currently working with and it will return you the string with tag context.
For example:
class Preprocessor(BasePreprocessorExt):
...
def _process_tag(match):
try:
config = open(self.options.get('config'))
except FileNotFoundError as e:
context = self.get_tag_context(match)
print(f'Config not found, check the tag:\n{context}')
...This will print:
Parsing config
Done
Applying preprocessor my_preprocessor
Config not found, check the tag:
...amet, consectetur adipisicing elit. Dolores ipsum non nisi voluptatum alias.
<my_tag param="value" config="wrong/path">
Tag body consectetur adipisicing elit. Voluptatem.
</template>
End of document.Now user can easily understand where's the problem in his document.
get_tag_context function accepts two parameters:
limit (default: 100) — number of characters included in context from before the tag, after the tag, and of the tag body;
full_tag (default: False) — if this is True, the tag body is copied into context without cropping (useful for relatively small expected tag bodies).
One last thing to use the full power of BasePreprocessorExt warnings:
self._warning accepts the context parameter. You can send there the context string. The context is only shown to user in the console if the debug mode is on, but it is always saved in the log file.
Example:
class Preprocessor(BasePreprocessorExt):
...
def _process_tag(match):
try:
config = open(self.options.get('config'))
except FileNotFoundError as e:
self._warning('Config file not found! Using default',
context=self.get_context(match),
error=e)
...Now if we catch this exception in normal mode, we will only get the md-filename and the message in the console. But if we run it in debug mode, we will get a full python traceback and the context of the tag. And a happy user.
Often we don't want the whole preprocessor to crash if there are some problems in just one tag of the document. We can easily achieve this using the allow_fail decorator, which is included in the preprocessor_ext module. Decorate your function, which is then sent to _process_tags_for_all_files method:
from foliant.preprocessors.utils.preprocessor_ext import (BasePreprocessorExt,
allow_fail)
class Preprocessor(BasePreprocessorExt):
...
@allow_fail()
def _process_tag(match):
# process the tag
return processed_string
def apply(self):
self._process_tags_for_all_files(func=self._process_tag)
self.logger.info('Preprocessor applied')Now in case any error occurs in the _process_tag function, preprocessor will issue warning, show it to user, save it into log and skip the tag.
The allow_fail decorator accepts one argument, the error message, which will be shown to user in case of exception. It defaults to: Failed to process tag. Skipping.
If your preprocessor doesn't have tags, you're probably doing somehing like this:
class Preprocessor(BasePreprocessor):
...
def _process_file(content):
processed = content
# do something with the content
return processed
def apply(self):
self.logger.info('Applying preprocessor')
for markdown_file_path in self.working_dir.rglob('*.md'):
with open(markdown_file_path, encoding='utf8') as markdown_file:
content = markdown_file.read()
processed_content = self._process_tags(content)
if processed_content:
with open(markdown_file_path, 'w') as markdown_file:
markdown_file.write(processed_content)
self.logger.info('Preprocessor applied')BasePreprocessorExt saves us from writing these many lines of the code which appear unchanged in most preprocessors.
So now instead of all that code we will write:
class Preprocessor(BasePreprocessorExt):
...
def _process_file(self, content):
processed = content
# do something with the content
return processed
def apply(self):
self._process_all_files(func=self._process_file, buffer=False)
self.logger.info('Preprocessor applied')Note the buffer=False parameter (which in this case is excessive because it is False by default). If buffer=True, markdown files processing will be buffered, e.g. they won't be updated until all of them are processed.
In your _process_file function you have access to self.current_filepath (Path of currently processing file) and self.current_filename (the chapter name, as it is would be stated in chapters foliant.yml section).