Develop (#11)

* Updated README.md

* Base extractor - Initial cut (#1)

* Initial commit of base for extractors

* Fixed dockerfile

* Added README

* Changed extractor to transformer in readme

* Updated README

* Fleshing out functionality

* Updates for file list support

* Variable name clarification

* Bug fixes

* Optimizing Dockerfile

* Fixed typo

* Merging into Develop (#2)

* Initial commit of base for extractors

* Fixed dockerfile

* Added README

* Changed extractor to transformer in readme

* Updated README

* Fleshing out functionality

* Updates for file list support

* Variable name clarification

* Bug fixes

* Optimizing Dockerfile

* Fixed typo

* Debugging

* Removed TERRA REF left-over commented out code

* Adding command line parameter storage

* Added text abour return codes

* Fix up transformer declared types (#4)

* Changed list to tuple

* type declaration correction and additional commenting

* Pylint changes

* Test development (#7)

* Adding test files

* Was missing the 'test_transformer.py' file and made some changes to 'test_transformer_class.py'

* adding the initial travis yml

* Filled out the yml file

* Changed the tests from base unittest to pytest

* Changed to pytest over basic unittest

* Moved tests to own folder

* Removed duplicate files

* Moved the yml file out of base-image

* modified travis yml to move directories for testing

* Added -v to pytest run for more clarity

* Ignoring a test for testing purposes

* Reverted git ignore changes and temporarily removed a test

* Placed test_entrypoint.py back into folder

* Pushing from new local

* Changed file naming conventions to be more conformative

* removing duplicates

* Fixed a test

* Added pylint installation to travis yml

* Added test doc as a link in main readme.

* fixed a formatting issue

* Initial test folder readme draft

* Clarified testing process

* added links to helpful documentation

* Clarified type of testing being done

* Adding pylint file

* renamed pylintrc file

* changes to pylint file

* fixed pylint call

* trying to include all files from test folder

* moved pylint file

* testing pylint

* Fixed yml file

* fixing yml

* I think I figured out the problem

* I'll seperate them and make to calls

* Made seperate calls

* misspelled directory name

* made some pylint recommended adjustments

* now pulling pylintrc from repo

* adding organization repo

* Hopefully this works

* Updated readme

*  Added some new lines

* Trying again

* hopefully this is it

* please

* done

* Minor readme changes

* Added pylint protocol link in readmes

* Pylint changes

* made some pylint recomended changes

* Helping pylint along

* Directory issues

* Resetting python path

* adding another line

* adding another line

* adding another line

* added some ignore lines

* adding another line

* Moved files around

* Finding pylint rc

* Missed a quotation mark

* fixed pylint call

* pylint diables

* Update base-image/test-files/README.md

Co-Authored-By: Chris Schnaufer <[email protected]>

* Update base-image/test-files/README.md

Co-Authored-By: Chris Schnaufer <[email protected]>

* Update base-image/test-files/README.md

Co-Authored-By: Chris Schnaufer <[email protected]>

* Update base-image/test-files/README.md

Co-Authored-By: Chris Schnaufer <[email protected]>

* Update base-image/test-files/README.md

Co-Authored-By: Chris Schnaufer <[email protected]>

* Incorporated Chris' suggestions

* Fixed a mistake

* Made changes recommended by Chris

* missed a change

* Loading multiple metadata files & pylint changes (#6)

* Update .gitignore, allowing multiple --metadata arguments, spelling & pylint changes

* Logging what metadatafiles are loading

* Added ability to download files after transformer gives the go-ahead (#8)

* Update .gitignore, allowing multiple --metadata arguments, spelling & pylint changes

* Logging what metadatafiles are loading

* Adding call to download files

* Yaml metadata support, optional metadata flag variable, updated to Python3.7 (#10)

* Initial support

* Updating to Python3.7

* Added link to 'python' command

Co-authored-by: Jorge Barrios <[email protected]>

base-image/Dockerfile

@@ -11,8 +11,11 @@ RUN chown -R extractor /home/extractor \
 RUN apt-get update && \
     apt-get upgrade -y && \
     apt-get install -y --no-install-recommends \
-        python3 \
+        python3.7 \
         python3-pip && \
+    ln -sfn /usr/bin/python3.7 /usr/bin/python && \
+    ln -sfn /usr/bin/python3.7 /usr/bin/python3 && \
+    ln -sfn /usr/bin/python3.7m /usr/bin/python3m && \
     apt-get autoremove -y && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*

base-image/configuration.py

@@ -6,3 +6,7 @@
 
 # The transformer description
 TRANSFORMER_DESCRIPTION = ""
+
+# Override flag for disabling the metadata file requirement.
+# Uncomment and set to False to override default behavior
+#METADATA_NEEDED = False

base-image/entrypoint.py

@@ -7,6 +7,7 @@
 import json
 import logging
 from typing import Optional
+import yaml
 
 import transformer_class
 
@@ -55,18 +56,24 @@ def load_metadata(metadata_path: str) -> dict:
             won't contain metadata but will contain an error message under an 'error' key.
         """
         try:
+            if os.path.splitext(metadata_path)[1] in ('.yml', '.yaml'):
+                load_func = yaml.safe_load
+            else:
+                load_func = json.load
             with open(metadata_path, 'r') as in_file:
-                md_load = json.load(in_file)
+                md_load = load_func(in_file)
                 if md_load is not None:
                     md_return = {'metadata': md_load}
                 else:
-                    msg = 'Invalid JSON specified in metadata file "%s"' % metadata_path
+                    msg = 'Invalid JSON/YAML specified in metadata file "%s"' % metadata_path
                     logging.error(msg)
                     md_return = {'error': msg}
         except Exception as ex:
-            msg = 'Unable to load metadata file "%s"' % metadata_path
+            msg = "Unable to load metadata file '%s'" % metadata_path
             logging.error(msg)
             logging.error('Exception caught: %s', str(ex))
+            if logging.getLogger().level == logging.DEBUG:
+                logging.exception(msg)
             md_return = {'error': msg}
 
         return md_return
@@ -123,6 +130,24 @@ def check_retrieve_results_error(transformer_retrieve: tuple) -> Optional[dict]:
 
         return None
 
+    @staticmethod
+    def check_metadata_needed() -> bool:
+        """Checks if metadata is required
+        Return:
+            Returns True if metadata is required (the default is that it's required), or False if not
+        """
+        # Disable the following check since it's not a valid test here (METADATA_NEEDED is an optional variable)
+        # pylint: disable=no-member
+
+        # If we have a variable defined, check the many ways of determining False
+        if hasattr(configuration, "METADATA_NEEDED"):
+            if not configuration.METADATA_NEEDED:
+                return False
+            if isinstance(configuration.METADATA_NEEDED, str):
+                if configuration.METADATA_NEEDED.lower().strip() == 'false':
+                    return False
+        return True
+
     @staticmethod
     def load_metadata_files(metadata_files: list) -> dict:
         """Loads the specified metadata files
@@ -358,7 +383,7 @@ def do_work(parser: argparse.ArgumentParser, **kwargs) -> dict:
     logging.getLogger().setLevel(args.debug if args.debug == logging.DEBUG else args.info)
 
     # Check that we have mandatory metadata
-    if not args.metadata:
+    if not args.metadata and __internal__.check_metadata_needed():
         result = __internal__.handle_error(-1, "No metadata paths were specified.")
     else:
         md_results = __internal__.load_metadata_files(args.metadata)