This repository has been archived by the owner on Dec 18, 2019. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 335
Provide a setuptools-based build infrastructure #93
Open
languitar
wants to merge
3
commits into
etsy:master
Choose a base branch
from
languitar:setuptools
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This rather longish commit provides a complete build infrastructure based on setuptools that allows to install skyline like any other python application. For setuptools to work correctly, a few fundamental changes had to be made apart from minor fixes. The most fundamental changes are: * Inside the src directory a dedicated skyline folder / package has been created to ensure that the skyline classes do not interfere with other python packages. * The python scripts called by the bash scripts that encapsulate the real functionality of skyline are now generated by setuptools. For this to work the scripts had to be converted to real modules, which means that a dash in the name was not acceptable. In case anyone has used these scripts directly, the name has now changed. It might be possible to restore the behavior by adding symlinks or small bash wrapper scripts inside the src tree, but this isn't done so far. The general bash scripts in bin reflect the changed names. * The settings.py.example has been renamed to settings.py so that setuptools can directly execute the unit tests without requiring user interaction. In the future, providing a settings.py with defaults that do not require any user interaction, might make skyline completely usable out of the box. In detail, the following changes have been made: * .gitignore: added stuff generated by setuptools, removed settings.py, which is now part of the distribution so that unit tests can execute without user interaction * .travis.yml: updated to new build system. Unit tests now executed through setup.py. Calling nose is still posible, though * MANIFEST.in: added a setuptools manifest to package the static files for the webapp * bin/*.d: updated to new script names for in-source use. Moreover, an if-else branch was added to decided whether the scripts should call the python files directly when called inside the source tree, or the generated setuptools wrapper scripts in case of a real installation * setup.cfg: contains settings for running the nose tests from within setuptools by calling python setup.py test * setup.py: The real build system logic. Some values are quite empty still, since I do not know the correct contents. Please updated accordingly. * src: moved all files to skyline package, made every folder a python package by adding an __ini__.py file to be discoverable by setuptools. * src/*/*-agent.py: renamed to agent.py to have a python module name so that setuptools can use these modules as entry point for console scripts. Moreover, executed script contents were moved to a run function which is then callable as a setuptools entry point. Module main functionality has been preserved by calling the run function inside the main block. Several smaller content changes had to be made to preserve the functionality after these changes. * src/*.py, tests/*.py: Manual changes to the pythonpath have been removed since with a real package structure, everything is now done automatically by python and for testing by the setuptools nose integration. * webapp.py: added a route to server anomalies.json from any configured location to /anomalies.json. This is necessary, since users should not modify the contents of the installed python egg just to push the updated anomaly detections to a path that is served by the webserver. * skyline.js: Fetch anomalies from /anomalies.json as described above. This enables the user to specify any location he wants for the generated json file. * horizon/listen.py: Explicitly added the StringIO import. No idea how this could work before. * utils/*.py: Updated imports to reflect new package structure
Wow, thanks for all this! Will go through it, hopefully everything checks out. |
Oops, seems I missed this change for the initial commit.
I just noticed I missed a hunk. It is added now to the pull request. |
earthgecko
added a commit
to earthgecko/skyline
that referenced
this pull request
Jul 17, 2016
- Added a variant of @astanway Crucible into the Skyline - Added a Crucible app so that ad-hoc timeseries can be feed to it and analyzed - Restructured the Skyline layout to be more inline with a Python package and setuptools with a merge of @languitar changes for setuptools and more pythonic structure that was submitted as per: etsy/skyline#93 etsy/skyline#91 Provide a setuptools-based build infrastructure #93 - etsy#91 - Not totally setuptools compliant yet - Added sphinx docs and documentation build pattern - More documentation - Serve sphinx documentation via webapp /static/docs/ - Handle pandas versions changes to - PANDAS_VERSION - Attempted to handle logging without log overwrites, not pretty but works - Performance profiled - pprofile, RunSnakeRun, cProfile, vmprof, snakeviz - deroomba - kill any lingering vacuum processes - ROOMBA_TIMEOUT - Self monitor Analyzer spin_process threads and terminate if any spin_process has run for longer than 180 seconds - MAX_ANALYZER_PROCESS_RUNTIME - Optimizations to Analyzer workflow logic - RUN_OPTIMIZED_WORKFLOW - Some general code optimizations based on profiling results - Addition of algorithm_breakdown metrics - ENABLE_ALGORITHM_RUN_METRICS and SKYLINE_TMP_DIR, tmpfs over multiprocessing Value - Updated current requirements - Patterned and tested in Python virtualenv for ease of python version, package management and dependencies management - More documentation - More documentation - More documentation - More documentation, docstrings, docstrings, docstrings - Panorama modules files and related Webapp UI changes - rebrow (added https://github.com/marians/rebrow) - Upgraded Webapp UI jquery, dygraph, bootstrap - Tested all Skyline components on Python 2.7.11 and 2.7.12 Added: CHANGES.md bin/analyzer_dev.d bin/crucible.d bin/panorama.d bin/skyline.d docs/Makefile docs/_build/html/.buildinfo docs/_build/html/_images/mirage-1.png docs/_build/html/_images/nupic.radar.predicted.14.month.requests.png docs/_build/html/_images/nupic.radar.real.predicted.difference.14.month.requests.overlayed.png docs/_build/html/_images/nupic.radar.real.predicted.difference.14.month.requests.png docs/_build/html/_images/panorama.closest.approximation.aggregrated.png docs/_build/html/_images/radar.real.14.month.requests.png docs/_build/html/_modules/algorithm_exceptions.html docs/_build/html/_modules/analyzer/agent.html docs/_build/html/_modules/analyzer/alerters.html docs/_build/html/_modules/analyzer/algorithms.html docs/_build/html/_modules/analyzer/analyzer.html docs/_build/html/_modules/analyzer_dev/agent.html docs/_build/html/_modules/analyzer_dev/alerters.html docs/_build/html/_modules/analyzer_dev/algorithms_dev.html docs/_build/html/_modules/analyzer_dev/analyzer_dev.html docs/_build/html/_modules/boundary/agent.html docs/_build/html/_modules/boundary/boundary.html docs/_build/html/_modules/boundary/boundary_alerters.html docs/_build/html/_modules/boundary/boundary_algorithms.html docs/_build/html/_modules/crucible/agent.html docs/_build/html/_modules/crucible/crucible.html docs/_build/html/_modules/crucible/crucible_algorithms.html docs/_build/html/_modules/horizon/agent.html docs/_build/html/_modules/horizon/listen.html docs/_build/html/_modules/horizon/roomba.html docs/_build/html/_modules/horizon/worker.html docs/_build/html/_modules/index.html docs/_build/html/_modules/logging.html docs/_build/html/_modules/mirage/agent.html docs/_build/html/_modules/mirage/mirage.html docs/_build/html/_modules/mirage/mirage_alerters.html docs/_build/html/_modules/mirage/mirage_algorithms.html docs/_build/html/_modules/mirage/negaters.html docs/_build/html/_modules/panorama/agent.html docs/_build/html/_modules/panorama/panorama.html docs/_build/html/_modules/skyline_functions.html docs/_build/html/_modules/webapp/backend.html docs/_build/html/_modules/webapp/webapp.html docs/_build/html/_sources/alert-testing.txt docs/_build/html/_sources/analyzer-optimizations.txt docs/_build/html/_sources/analyzer.txt docs/_build/html/_sources/boundary.txt docs/_build/html/_sources/building-documentation.txt docs/_build/html/_sources/crucible.txt docs/_build/html/_sources/debian-and-vagrant-installation-tips.txt docs/_build/html/_sources/development/index.txt docs/_build/html/_sources/development/webapp.txt docs/_build/html/_sources/getting-data-into-skyline.txt docs/_build/html/_sources/getting-started.txt docs/_build/html/_sources/horizon.txt docs/_build/html/_sources/index.txt docs/_build/html/_sources/installation.txt docs/_build/html/_sources/logging.txt docs/_build/html/_sources/mirage.txt docs/_build/html/_sources/modules.txt docs/_build/html/_sources/monitoring-skyline.txt docs/_build/html/_sources/overview.txt docs/_build/html/_sources/panorama.txt docs/_build/html/_sources/redis-integration.txt docs/_build/html/_sources/releases.txt docs/_build/html/_sources/releases/1_0_0.txt docs/_build/html/_sources/requirements.txt docs/_build/html/_sources/roadmap.txt docs/_build/html/_sources/running-in-python-virtualenv.txt docs/_build/html/_sources/skyline-and-friends.txt docs/_build/html/_sources/skyline.analyzer.txt docs/_build/html/_sources/skyline.analyzer_dev.txt docs/_build/html/_sources/skyline.boundary.txt docs/_build/html/_sources/skyline.crucible.txt docs/_build/html/_sources/skyline.horizon.txt docs/_build/html/_sources/skyline.mirage.txt docs/_build/html/_sources/skyline.panorama.txt docs/_build/html/_sources/skyline.txt docs/_build/html/_sources/skyline.webapp.txt docs/_build/html/_sources/tuning-tips.txt docs/_build/html/_sources/upgrading.txt docs/_build/html/_sources/webapp.txt docs/_build/html/_sources/whats-new.txt docs/_build/html/_static/ajax-loader.gif docs/_build/html/_static/basic.css docs/_build/html/_static/comment-bright.png docs/_build/html/_static/comment-close.png docs/_build/html/_static/comment.png docs/_build/html/_static/css/badge_only.css docs/_build/html/_static/css/theme.css docs/_build/html/_static/doctools.js docs/_build/html/_static/down-pressed.png docs/_build/html/_static/down.png docs/_build/html/_static/file.png docs/_build/html/_static/fonts/Inconsolata-Bold.ttf docs/_build/html/_static/fonts/Inconsolata-Regular.ttf docs/_build/html/_static/fonts/Lato-Bold.ttf docs/_build/html/_static/fonts/Lato-Regular.ttf docs/_build/html/_static/fonts/RobotoSlab-Bold.ttf docs/_build/html/_static/fonts/RobotoSlab-Regular.ttf docs/_build/html/_static/fonts/fontawesome-webfont.eot docs/_build/html/_static/fonts/fontawesome-webfont.svg docs/_build/html/_static/fonts/fontawesome-webfont.ttf docs/_build/html/_static/fonts/fontawesome-webfont.woff docs/_build/html/_static/jquery-1.11.1.js docs/_build/html/_static/jquery.js docs/_build/html/_static/js/modernizr.min.js docs/_build/html/_static/js/theme.js docs/_build/html/_static/minus.png docs/_build/html/_static/plus.png docs/_build/html/_static/pygments.css docs/_build/html/_static/searchtools.js docs/_build/html/_static/skyline.styles.css docs/_build/html/_static/underscore-1.3.1.js docs/_build/html/_static/underscore.js docs/_build/html/_static/up-pressed.png docs/_build/html/_static/up.png docs/_build/html/_static/websupport.js docs/_build/html/alert-testing.html docs/_build/html/analyzer-optimizations.html docs/_build/html/analyzer.html docs/_build/html/boundary.html docs/_build/html/building-documentation.html docs/_build/html/crucible.html docs/_build/html/debian-and-vagrant-installation-tips.html docs/_build/html/development/index.html docs/_build/html/development/webapp.html docs/_build/html/genindex.html docs/_build/html/getting-data-into-skyline.html docs/_build/html/getting-started.html docs/_build/html/horizon.html docs/_build/html/index.html docs/_build/html/installation.html docs/_build/html/logging.html docs/_build/html/mirage-1.hires.png docs/_build/html/mirage-1.pdf docs/_build/html/mirage-1.png docs/_build/html/mirage-1.py docs/_build/html/mirage.html docs/_build/html/modules.html docs/_build/html/monitoring-skyline.html docs/_build/html/objects.inv docs/_build/html/overview.html docs/_build/html/panorama.html docs/_build/html/py-modindex.html docs/_build/html/redis-integration.html docs/_build/html/releases.html docs/_build/html/releases/1_0_0.html docs/_build/html/requirements.html docs/_build/html/roadmap.html docs/_build/html/running-in-python-virtualenv.html docs/_build/html/search.html docs/_build/html/searchindex.js docs/_build/html/skyline-and-friends.html docs/_build/html/skyline.analyzer.html docs/_build/html/skyline.analyzer_dev.html docs/_build/html/skyline.boundary.html docs/_build/html/skyline.crucible.html docs/_build/html/skyline.horizon.html docs/_build/html/skyline.html docs/_build/html/skyline.mirage.html docs/_build/html/skyline.panorama.html docs/_build/html/skyline.webapp.html docs/_build/html/tuning-tips.html docs/_build/html/upgrading.html docs/_build/html/webapp.html docs/_build/html/whats-new.html docs/_build/plot_directive/mirage-1.hires.png docs/_build/plot_directive/mirage-1.pdf docs/_build/plot_directive/mirage-1.png docs/_static/skyline.styles.css docs/alert-testing.rst docs/analyzer-optimizations.rst docs/analyzer.rst docs/boundary.rst docs/building-documentation.md docs/conf.py docs/crucible.rst docs/debian-and-vagrant-installation-tips.md docs/development/index.rst docs/development/webapp.rst docs/getting-data-into-skyline.rst docs/getting-started.rst docs/horizon.md docs/images/nupic.radar.predicted.14.month.requests.png docs/images/nupic.radar.real.predicted.difference.14.month.requests.overlayed.png docs/images/nupic.radar.real.predicted.difference.14.month.requests.png docs/images/panorama.closest.approximation.aggregrated.png docs/images/radar.real.14.month.requests.png docs/index.rst docs/installation.rst docs/logging.md docs/mirage.rst docs/modules.rst docs/monitoring-skyline.md docs/overview.rst docs/panorama.rst docs/redis-integration.md docs/releases.rst docs/releases/1_0_0.rst docs/requirements.rst docs/roadmap.rst docs/running-in-python-virtualenv.rst docs/skyline-and-friends.md docs/skyline.analyzer.rst docs/skyline.analyzer_dev.rst docs/skyline.boundary.rst docs/skyline.crucible.rst docs/skyline.horizon.rst docs/skyline.mirage.rst docs/skyline.panorama.rst docs/skyline.rst docs/skyline.webapp.rst docs/tuning-tips.md docs/upgrading.rst docs/webapp.rst docs/whats-new.md etc/skyline.conf etc/skyline.httpd.conf.d.example examples/data/Real_time_energy_data_October.csv skyline.png skyline/__init__.py skyline/algorithm_exceptions.py skyline/analyzer/__init__.py skyline/analyzer/agent.py skyline/analyzer/alerters.py skyline/analyzer/algorithms.py skyline/analyzer/analyzer.py skyline/analyzer_dev/__init__.py skyline/analyzer_dev/agent.py skyline/analyzer_dev/alerters.py skyline/analyzer_dev/algorithms_dev.py skyline/analyzer_dev/analyzer_dev.py skyline/boundary/LICENSE.md skyline/boundary/__init__.py skyline/boundary/agent.py skyline/boundary/boundary.py skyline/boundary/boundary_alerters.py skyline/boundary/boundary_algorithms.py skyline/crucible/LICENSE.md skyline/crucible/__init__.py skyline/crucible/agent.py skyline/crucible/crucible.py skyline/crucible/crucible_algorithms.py skyline/horizon/__init__.py skyline/horizon/agent.py skyline/horizon/listen.py skyline/horizon/roomba.py skyline/horizon/worker.py skyline/mirage/LICENSE.md skyline/mirage/__init__.py skyline/mirage/agent.py skyline/mirage/mirage.py skyline/mirage/mirage_alerters.py skyline/mirage/mirage_algorithms.py skyline/mirage/negaters.py skyline/panorama/LICENSE.md skyline/panorama/__init__.py skyline/panorama/agent.py skyline/panorama/panorama.py skyline/settings.py skyline/skyline.sql skyline/skyline_functions.py skyline/skyline_version.py skyline/webapp/__init__.py skyline/webapp/backend.py skyline/webapp/gunicorn.py skyline/webapp/rebrow/LICENSE skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap-theme.css skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap-theme.css.map skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap-theme.min.css skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap-theme.min.css.map skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap.css skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap.css.map skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap.min.css skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap.min.css.map skyline/webapp/static/bootstrap-3.3.6-dist/fonts/glyphicons-halflings-regular.eot skyline/webapp/static/bootstrap-3.3.6-dist/fonts/glyphicons-halflings-regular.svg skyline/webapp/static/bootstrap-3.3.6-dist/fonts/glyphicons-halflings-regular.ttf skyline/webapp/static/bootstrap-3.3.6-dist/fonts/glyphicons-halflings-regular.woff skyline/webapp/static/bootstrap-3.3.6-dist/fonts/glyphicons-halflings-regular.woff2 skyline/webapp/static/bootstrap-3.3.6-dist/js/bootstrap.js skyline/webapp/static/bootstrap-3.3.6-dist/js/bootstrap.min.js skyline/webapp/static/bootstrap-3.3.6-dist/js/npm.js skyline/webapp/static/css/skyline.styles.css skyline/webapp/static/docs skyline/webapp/static/dump/.gitignore skyline/webapp/static/dygraph-1.1.1/dygraph-combined.js skyline/webapp/static/fontawesome-4.6.3/css/font-awesome.css skyline/webapp/static/fontawesome-4.6.3/css/font-awesome.css.map skyline/webapp/static/fontawesome-4.6.3/css/font-awesome.min.css skyline/webapp/static/fontawesome-4.6.3/fonts/FontAwesome.otf skyline/webapp/static/fontawesome-4.6.3/fonts/fontawesome-webfont.eot skyline/webapp/static/fontawesome-4.6.3/fonts/fontawesome-webfont.svg skyline/webapp/static/fontawesome-4.6.3/fonts/fontawesome-webfont.ttf skyline/webapp/static/fontawesome-4.6.3/fonts/fontawesome-webfont.woff skyline/webapp/static/fontawesome-4.6.3/fonts/fontawesome-webfont.woff2 skyline/webapp/static/fonts/montserrat-bold.woff skyline/webapp/static/fonts/montserrat-regular.woff skyline/webapp/static/images/favicon.ico skyline/webapp/static/jquery-2.2.4/dist/jquery.js skyline/webapp/static/jquery-2.2.4/dist/jquery.min.js skyline/webapp/static/jquery-2.2.4/dist/jquery.min.map skyline/webapp/static/js/cubism.v1.min.js skyline/webapp/static/js/mousetrap.min.js skyline/webapp/static/js/panorama.js skyline/webapp/static/js/skyline.js skyline/webapp/static/strftime-0.9.2/strftime-min.js skyline/webapp/templates/docs.html skyline/webapp/templates/layout.html skyline/webapp/templates/now.html skyline/webapp/templates/panorama.html skyline/webapp/templates/rebrow_key.html skyline/webapp/templates/rebrow_keys.html skyline/webapp/templates/rebrow_login.html skyline/webapp/templates/rebrow_server_db.html skyline/webapp/templates/uh_oh.html skyline/webapp/webapp.py tests/test_crucible_algorithms.py tests/test_imports.py Modified: bin/analyzer.d bin/boundary.d bin/horizon.d bin/mirage.d bin/webapp.d readme.md requirements.txt tests/algorithms_test.py utils/continuity.py utils/seed_data.py utils/verify_alerts.py .gitignore
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Included in this pull request is a complete build system based on setuptools, as I requested in #91. I suspect there will be a lot of questions or doubts about such a big change, but I am happy to answer them. So I'd really be glad if this would be accepted eventually.
commit message
This rather longish commit provides a complete build infrastructure
based on setuptools that allows to install skyline like any other python
application.
For setuptools to work correctly, a few fundamental changes had to be
made apart from minor fixes. The most fundamental changes are:
created to ensure that the skyline classes do not interfere with other
python packages.
real functionality of skyline are now generated by setuptools. For
this to work the scripts had to be converted to real modules, which
means that a dash in the name was not acceptable. In case anyone has used
these scripts directly, the name has now changed. It might be possible
to restore the behavior by adding symlinks or small bash wrapper
scripts inside the src tree, but this isn't done so far. The general
bash scripts in bin reflect the changed names.
setuptools can directly execute the unit tests without requiring user
interaction. In the future, providing a settings.py with defaults that do
not require any user interaction, might make skyline completely usable
out of the box.
In detail, the following changes have been made:
which is now part of the distribution so that unit tests can execute
without user interaction
through setup.py. Calling nose is still posible, though
for the webapp
if-else branch was added to decided whether the scripts should call
the python files directly when called inside the source tree, or the
generated setuptools wrapper scripts in case of a real installation
setuptools by calling python setup.py test
still, since I do not know the correct contents. Please updated
accordingly.
package by adding an ini.py file to be discoverable by setuptools.
that setuptools can use these modules as entry point for console
scripts. Moreover, executed script contents were moved to a run
function which is then callable as a setuptools entry point. Module
main functionality has been preserved by calling the run function inside
the main block. Several smaller content changes had to be made to
preserve the functionality after these changes.
with a real package structure, everything is now done automatically by
python and for testing by the setuptools nose integration.
location to /anomalies.json. This is necessary, since users should not
modify the contents of the installed python egg just to push the
updated anomaly detections to a path that is served by the webserver.
This enables the user to specify any location he wants for the
generated json file.
this could work before.