Skip to content

Latest commit

 

History

History
49 lines (33 loc) · 1.82 KB

README.md

File metadata and controls

49 lines (33 loc) · 1.82 KB
Raw

IceProd2 resource requirements harvesting

This is a collection of tools for extracting resource usage summaries from the IceProd2 database.

REST API (>2.6)

  • fetch_iceprod_stats.py: extract stats from production database using the REST API. Requires login to IceCube SSO.

Usage

  • Set up a Python environment: 
    python3 -m virtualenv env
. env/bin/activate
pip install -r requirements.txt
  • Get resource usage: python fetch_iceprod_stats.py logs resource_usage.iceprod25.v0.hdf5
  • Get configs: python fetch_iceprod_stats.py configs configs.iceprod25.v0.json
  • Upload to share directory (optional): ./cloudsend.sh resource_usage.iceprod25.v2.hdf5 https://desycloud.desy.de/index.php/s/...

REST API (2.3 < v < 2.6)

These versions of IceProd2 requred a special API token.

  • fetch_iceprod_stats_v2.3.py: extract stats from production database using the REST API. Requires an API token.

Usage

  • Get an API token from https://iceprod2.icecube.wisc.edu/profile and refresh it daily
  • Set the API token as an environment variable: echo ICEPROD_TOKEN=ICEPROD_TOKEN=eyJ0eXAiO...
  • Get resource usage: python fetch_iceprod_stats_v2.3.py logs resource_usage.iceprod25.v0.hdf5
  • Get configs: python fetch_iceprod_stats_v2.3.py configs configs.iceprod25.v0.json
  • Upload to share directory (optional): ./cloudsend.sh resource_usage.iceprod25.v2.hdf5 https://desycloud.desy.de/index.php/s/...

sqlite dump (<=v2.3)

Older versions of IceProd2 were backed by a relational DB with no outside API access. This was analyzed by migrating the sqlite database into postgres and using its json functions to filter out task resource summaries.

  • schema.postgres.sql: postgres-ified schema from the sqlite database
  • migrate.sh: migrate from sqlite to postgres
  • queries.sql: horrible master query