-2. Set Environment variables.
- Library uses DVC for artifact versioning and git for code versioning. It requires a set of environment variables to operate DVC and git. The list of environment variables needed can be found in example-get-started/sample_env.
- Copy sample_env from example-get-started directory to local directory. - Modify sample_env and run 'source sample_env' to set the environment variables.
-3. Copy initialize.sh from example-get-started folder to your local directory.
- Run command - sh initialize.sh
- Before running the script, please ensure that required environment variables are set.
- This configures DVC and git with the provided variables in Step 1.
- - -## Logging metadata with CMF -#### Import of the library to record the metadata -```python -from cmflib import cmf - -``` - -### Create the metadata writer -The metadata writer is responsible to manage the backend to record the metadata. -It also creates a pipeline abstraction, which helps to group individual stages and execution. -```python -cmf = cmf.Cmf(filename="mlmd", - pipeline_name="Test-env") - -``` -### Create the stage in pipeline. -An ML pipeline can have multiple stages. This context abstraction tracks the stage and its metadata. -A dictionary can be passed to hold the user given metadata. The custom properties is an optional argument - ```python -context = cmf.create_context(pipeline_stage="Prepare", - custom_properties={"user-metadata1":"metadata_value"}) -``` - -#### Create the execution -A stage in ML pipeline can have multiple executions. Every run is marked as an execution. -This API helps to track the metadata associated with the execution -```python -execution = cmf.create_execution(execution_type="Prepare-1", custom_properties = {"user-metadata1":"metadata_value"}) -``` -#### Log artifacts -An Execution could have multiple artifacts associated with it as Input or Output. The path of the artifact provided should be path from root of the repo. -The metadata associated with the artifact could be logged as an optional argument which takes in a dictionary -```python -cmf.log_dataset(input, "input", custom_properties={"user-metadata1":"metadata_value"}) -cmf.log_dataset(output_train, "output", custom_properties={"user-metadata1":"metadata_value"}) -cmf.log_dataset(output_test, "output", custom_properties={"user-metadata1":"metadata_value"}) -``` -#### Log model -A model developed as part of training step or used in a evaluation or inference step can be logged. It can be input or output -The metadata associated with the artifact could be logged as an optional argument which takes in a dictionary -```python -cmf.log_model(path="model.pkl", event="output", model_framework="SKlearn", model_type="RandomForestClassifier", model_name="RandomForestClassifier:default" ) -cmf.log_model(path="model.pkl", event="input", model_framework="SKlearn", model_type="RandomForestClassifier", model_name="RandomForestClassifier:default" ) -``` -#### Log metrics -Metrics of each step can be logged -The metadata associated with the artifact could be logged as argument which takes in a dictionary -```python -#Can be called at every epoch or every step in the training. This is logged to a parquet file and commited at the commit stage. -while True: #Inside training loop - cmf.log_metric("training_metrics", {"loss":loss}) -cmf.commit_metrics("training_metrics") -``` -#### Log Stage metrics -Metrics for each stage. -```python -cmf.log_execution_metrics("metrics", {"avg_prec":avg_prec, "roc_auc":roc_auc}) -``` -#### Creating the dataslices -This helps to track a subset of the data. For eg- Accuracy of the model for a slice of data(gender, ethnicity etc) -```python -dataslice = cmf.create_dataslice("slice-a") -for i in range(1,20,1): - j = random.randrange(100) - dataslice.add_data("data/raw_data/"+str(j)+".xml") -dataslice.commit() -``` -### To use Graph Layer -CMF library has an optional Graph layer which stores the relationships in a Graph Database(NEO4J) -To use the graph layer, the "graph" parameter in the library init call should be set to true. This is set as false by default. -The library reads the configuration parameters of the Database from the environment variables. -The variables "NEO4J_URI", "NEO4J_USER_NAME", "NEO4J_PASSWD" should be either set as environment variables. - -``` - export NEO4J_URI="bolt://10.93.244.219:7687" - export NEO4J_USER_NAME=neo4j - export NEO4J_PASSWD=neo4j +# cmf +## Common Metadata Framework +[Getting Started](docs/index.md)
+[Detailed documentation of the API's](https://hewlettpackard.github.io/cmf/api/public/cmf)
+ +Interactions in data pipelines can be complex. The Different stages in the pipeline, (which may not be next to each other) may have to interact to produce or transform artifacts. As the artifacts navigates and undergo transformations through this pipeline, it can take a complicated path, which might also involve bidirectional movement across these stages. Also there could be dependencies between the multiple stages, where the metrics produced by a stage could influence the metrics at a subsequent stage. It is important to track the metadata across a pipeline to provide features like, lineage tracking, provenance and reproducibility. + +The tracking of metadata through these complex pipelines have multiple challenges, some of them being, + +- Each stage in the pipeline could be executed in a different datacenter or an edge site having intermittent connection to the core datacenter. + +- Each stage in the pipeline could be possibly managed by different teams. + +- The artifacts (input or output) needs to be uniquely identified across different sites and across multiple pipelines. + + +Common metadata framework (CMF) addresses the problems associated with tracking of pipeline metadata from distributed sites and tracks code, data and metadata together for end-to-end traceability. + +The framework automatically tracks the code version as one of the metadata for an execution. Additionally the data artifacts are also versioned automatically using a data versioning framework (like DVC) and the metadata regarding the data version is stored along with the code. The framework stores the Git commit id of the metadata file associated with the artifact and content hash of the artifact as metadata. The framework provides API’s to track the hyper parameters and other metadata of pipelines. Therefore from the metadata stored, users can zero in on the hyper parameters, code version and the artifact version used for the experiment. + +Identifying the artifacts by content hash allows the framework, to uniquely identify an artifact anywhere in the distributed sites. This enables the metadata from the distributed sites to be precisely merged to a central repository, thereby providing a single global metadata from the distributed sites. + +On this backbone, we build the Git like experience for metadata, enabling users to push their local metadata to the remote repository, where it is merged to create the global metadata and pull metadata from the global metadata to the local, to create a local view, which would contain only the metadata of interest. + +The framework can be used to track various types of pipelines such as data pipelines or AI pipelines. +
+ 
+
+ 
+
+
+
+## Metadata Library
+The API’s and the abstractions provided by the library enables tracking of pipeline metadata. It tracks the stages in
+the pipeline, the input and output artifacts at each stage and metrics. The framework allows metrics to be tracked both
+at coarse and fine-grained intervals. It could be a stage metrics, which could be captured at the end of a stage or
+fine-grained metrics which is tracked per step (epoch) or at regular intervals during the execution of the stage.
+
+The metadata logged through the APIs are written to a backend relational database. The library also provides API’s to
+query the metadata stored in the relational database for the users to inspect pipelines.
+
+In addition to explicit tracking through the API’s library also provides, implicit tracking. The implicit tracking
+automatically tracks the software version used in the pipelines. The function arguments and function return values can
+be automatically tracked by adding metadata tracker class decorators on the functions.
+
+Before writing the metadata to relational database, the metadata operations are journaled in the metadata journal log.
+This enables the framework to transfer the local metadata to the central server.
+
+All artifacts are versioned with a data versioning framework (for e.g., DVC). The content hash of the artifacts are
+generated and stored along with the user provided metadata. A special artifact metadata file called a “.dvc” file is
+created for every artifact (file / folder) which is added to data version management system. The .dvc file contains the
+content hash of the artifact.
+
+For every new execution, the metadata tracker creates a new branch to track the code. The special metadata file created
+for artifacts, the “.dvc” file is also committed to GIT and its commit id is tracked as a metadata information.
+The artifacts are versioned through the versioning of its metadata file. Whenever there is a change in the artifact,
+the metadata file is modified to reflect its current content hash, and the file is tracked as a new version of the
+metadata file.
+
+The metadata tracker automatically tracks the start commit when the library was initialized and creates separate commit
+for each change in the artifact along the experiment. This helps to track the transformations on the artifacts along the
+different stages in the pipeline.
+
+## Local Client
+The metadata client interacts with the metadata server. It communicates with the server, for synchronization of metadata.
+
+After the experiment is completed, the user invokes the “Cmf push” command to push the collected metadata to the remote.
+This transfers the existing metadata journal to the server.
+
+The metadata from the central repository can be pulled to the local repository, either using the artifacts or using the
+project as the identifier or both.
+
+When artifact is used as the identifier, all metadata associated with the artifacts currently present in the branch of
+the cloned Git repository is pulled from the central repository to the local repository. The pulled metadata consist of
+not only the immediate metadata associated with the artifacts, it contains the metadata of all the artifacts in its
+chain of lineage.
+
+When project is used as the identifier, all the metadata associated with the current branch of the pipeline code that
+is checked out is pulled to the local repository.
+
+## Central Server
+The central server, exposes REST API’s that can be called from the remote clients. This can help in situations where the
+connectivity between the core datacenter and the remote client is robust. The remote client calls the API’s exposed by
+the central server to log the metadata directly to the central metadata repository.
+
+Where the connectivity with the central server is intermittent, the remote clients log the metadata to the local
+repository. The journaled metadata is pushed by the remote client to the central server. The central server, will
+replay the journal and merge the incoming metadata with the metadata already existing in the central repository. The
+ability to accurately identify the artifacts anywhere using their content hash, makes this merge robust.
+
+## Central Repositories
+The common metadata framework consist of three central repositories for the code, data and metadata.
+
+#### Central Metadata repository
+Central metadata repository holds the metadata pushed from the distributed sites. It holds metadata about all the
+different pipelines that was tracked using the common metadata tracker. The consolidated view of the metadata stored
+in the central repository, helps the users to learn across various stages in the pipeline executed at different
+locations. Using the query layer that is pointed to the central repository, the users gets the global view of the
+metadata which provides them with a deeper understanding of the pipelines and its metadata. The metadata helps to
+understand nonobvious results like performance of a dataset with respect to other datasets, Performance of a particular
+pipeline with respect to other pipelines etc.
+
+#### Central Artifact storage repository
+Central Artifact storage repository stores all the artifacts related to experiment. The data versioning framework (DVC)
+stores the artifacts in a content addressable layout. The artifacts are stored inside the folder with name as the first
+two characters of the content hash and the name of the artifact as the remaining part of the content hash. This helps
+in efficient retrieval of the artifacts.
+
+#### Git Repository
+Git repository is used to track the code. Along with the code, the metadata file of the artifacts which contain the
+content hash of the artifacts are also stored in GIT. The Data versioning framework (dvc) would use these files to
+retrieve the artifacts from the artifact storage repository.
diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md
new file mode 100644
index 00000000..a51b0017
--- /dev/null
+++ b/docs/architecture/overview.md
@@ -0,0 +1,37 @@
+# Architecture Overview
+
+Interactions in data pipelines can be complex. The Different stages in the pipeline, (which may not be next to each
+other) may have to interact to produce or transform artifacts. As the artifacts navigates and undergo transformations
+through this pipeline, it can take a complicated path, which might also involve bidirectional movement across these
+stages.
+Also, there could be dependencies between the multiple stages, where the metrics produced by a stage could influence the
+metrics at a subsequent stage. It is important to track the metadata across a pipeline to provide features like,
+lineage tracking, provenance and reproducibility.
+
+The tracking of metadata through these complex pipelines have multiple challenges, some of them being,
+
+- Each stage in the pipeline could be executed in a different datacenter or an edge site having intermittent connection
+ to the core datacenter.
+- Each stage in the pipeline could be possibly managed by different teams.
+- The artifacts (input or output) needs to be uniquely identified across different sites and across multiple pipelines.
+
+Common metadata framework (CMF) addresses the problems associated with tracking of pipeline metadata from distributed
+sites and tracks code, data and metadata together for end-to-end traceability.
+
+The framework automatically tracks the code version as one of the metadata for an execution. Additionally, the data
+artifacts are also versioned automatically using a data versioning framework (like DVC) and the metadata regarding the
+data version is stored along with the code. The framework stores the Git commit id of the metadata file associated with
+the artifact and content hash of the artifact as metadata. The framework provides API’s to track the hyperparameters and
+other metadata of pipelines. Therefore, from the metadata stored, users can zero in on the hyperparameters, code
+version and the artifact version used for the experiment.
+
+Identifying the artifacts by content hash allows the framework, to uniquely identify an artifact anywhere in the
+distributed sites. This enables the metadata from the distributed sites to be precisely merged to a central repository,
+thereby providing a single global metadata from the distributed sites.
+
+On this backbone, we build the Git like experience for metadata, enabling users to push their local metadata to the
+remote repository, where it is merged to create the global metadata and pull metadata from the global metadata to the
+local, to create a local view, which would contain only the metadata of interest.
+
+The framework can be used to track various types of pipelines such as data pipelines or AI pipelines.
+
diff --git a/docs/assets/Metadata_stored.png b/docs/assets/Metadata_stored.png
new file mode 100644
index 00000000..d1b60dd1
Binary files /dev/null and b/docs/assets/Metadata_stored.png differ
diff --git a/docs/assets/Python_kernel.png b/docs/assets/Python_kernel.png
new file mode 100644
index 00000000..5616b1d9
Binary files /dev/null and b/docs/assets/Python_kernel.png differ
diff --git a/docs/assets/distributed_architecture.png b/docs/assets/distributed_architecture.png
new file mode 100644
index 00000000..38cf8932
Binary files /dev/null and b/docs/assets/distributed_architecture.png differ
diff --git a/docs/assets/framework.png b/docs/assets/framework.png
new file mode 100644
index 00000000..639c5fb2
Binary files /dev/null and b/docs/assets/framework.png differ
diff --git a/docs/assets/jupyter.png b/docs/assets/jupyter.png
new file mode 100644
index 00000000..02638b18
Binary files /dev/null and b/docs/assets/jupyter.png differ
diff --git a/docs/assets/ml_pipeline_def.png b/docs/assets/ml_pipeline_def.png
new file mode 100644
index 00000000..1465784f
Binary files /dev/null and b/docs/assets/ml_pipeline_def.png differ
diff --git a/docs/assets/ml_pipeline_stage_execution.png b/docs/assets/ml_pipeline_stage_execution.png
new file mode 100644
index 00000000..c28b46be
Binary files /dev/null and b/docs/assets/ml_pipeline_stage_execution.png differ
diff --git a/docs/assets/neo4j_output.PNG b/docs/assets/neo4j_output.PNG
new file mode 100644
index 00000000..81494a97
Binary files /dev/null and b/docs/assets/neo4j_output.PNG differ
diff --git a/docs/assets/neo4j_server.png b/docs/assets/neo4j_server.png
new file mode 100644
index 00000000..e544614d
Binary files /dev/null and b/docs/assets/neo4j_server.png differ
diff --git a/docs/assets/python_kernel_broader.png b/docs/assets/python_kernel_broader.png
new file mode 100644
index 00000000..ce2328bc
Binary files /dev/null and b/docs/assets/python_kernel_broader.png differ
diff --git a/docs/assets/slack_logo.png b/docs/assets/slack_logo.png
new file mode 100644
index 00000000..fb5d71a4
Binary files /dev/null and b/docs/assets/slack_logo.png differ
diff --git a/docs/examples/getting_started.md b/docs/examples/getting_started.md
new file mode 100644
index 00000000..4682ab29
--- /dev/null
+++ b/docs/examples/getting_started.md
@@ -0,0 +1,105 @@
+# Getting Started
+
+> This example depends on the following packages: `git`. We also recommend installing
+> [anaconda](https://docs.anaconda.com/anaconda/install/linux/) to manage python virtual environments.
+> This example was tested in the following environments:
+>
+> - `Ubuntu-22.04 with python-3.8.15`
+
+This example demonstrates how CMF tracks a metadata associated with executions of various machine learning (ML)
+pipelines. ML pipelines differ from other pipelines (e.g., data Extract-Transform-Load pipelines) by the presence of
+ML steps, such as training and testing ML models. More comprehensive ML pipelines may include steps such as deploying a
+trained model and tracking its inference parameters (such as response latency, memory consumption etc.). This example,
+located [here](https://github.com/HewlettPackard/cmf/tree/master/examples/example-get-started) implements a simple
+pipeline consisting of four steps:
+
+- The [parse](https://github.com/HewlettPackard/cmf/blob/master/examples/example-get-started/src/parse.py) step splits
+ the [raw data](https://github.com/HewlettPackard/cmf/tree/master/examples/example-get-started/artifacts) into
+ `train` and `test` raw datasets for training and testing a machine learning model. This step registers one
+ input artifact (raw `dataset`) and two output artifacts (train and test `datasets`).
+- The [featurize](https://github.com/HewlettPackard/cmf/blob/master/examples/example-get-started/src/featurize.py)
+ step creates two machine learning splits - train and test splits - that will be used by an ML training algorithm to
+ train ML models. This step registers two input artifacts (raw train and test datasets) and two output artifacts (
+ train and test ML datasets).
+- The next [train](https://github.com/HewlettPackard/cmf/blob/master/examples/example-get-started/src/train.py) step
+ trains an ML model (random forest classifier). It registers one input artifact (train ML dataset) and one
+ output artifact (trained ML model).
+- The fourth [test](https://github.com/HewlettPackard/cmf/blob/master/examples/example-get-started/src/test.py) step
+ tests the ML model trained in the third `train` step. This step registers two input artifacts (ML model and test
+ dataset) and one output artifact (performance metrics).
+- The last [query](https://github.com/HewlettPackard/cmf/blob/master/examples/example-get-started/src/query.py) step
+ is a demonstration that shows how pipeline metadata can be retrieved from CMF. It will print metadata associated with
+ all executions of the above steps. This means that if you rerun the pipeline again, the output will include not only
+ metadata associated with the last run, but also metadata associated with all previous runs.
+
+
+## Pre-requisites
+
+We start by creating (1) a workspace directory that will contain all files for this example and (2) a python virtual
+environment. Then we will clone the CMF project that contains this example project.
+```shell
+# Create workspace directory
+mkdir cmf_getting_started_example
+cd cmf_getting_started_example
+
+# Create and activate Python virtual environment (the Python version may need to be adjusted depending on your system)
+conda create -n cmf_getting_started_example python=3.8
+conda activate cmf_getting_started_example
+
+# Clone the CMF project from GitHub and install CMF
+git clone https://github.com/HewlettPackard/cmf
+pip install ./cmf
+```
+
+## Project initialization
+We need to copy the source tree of the example in its own directory (that must be outside the CMF source tree), and
+initialize `git` and `dvc` for this project.
+
+```shell
+# Create a separate copy of the example project
+cp -r ./cmf/examples/example-get-started/ ./example-get-started
+cd ./example-get-started
+```
+
+Review the content of the
+[sample_env](https://github.com/HewlettPackard/cmf/blob/master/examples/example-get-started/sample_env) file which is
+located in the root directory of the example. For the demonstration purposes, you can leave all fields as is. Once this
+file is reviewed, source that file and run
+[initialize.sh](https://github.com/HewlettPackard/cmf/blob/master/examples/example-get-started/initialize.sh)
+to initialize `git` and `dvc` repositories.
+```shell
+# Export environmental variables
+source ./sample_env
+# Initialize the example project
+sh ./initialize.sh
+```
+
+## Project execution
+The `initialize.sh` script executed above has printed some details about the project. To execute the example
+pipeline, run the
+[test_script.sh](https://github.com/HewlettPackard/cmf/blob/master/examples/example-get-started/test_script.sh)
+file (before that, study the contents of that file). Basically, that script will run a sequence of steps
+common for a typical machine learning project - getting raw data, converting it into machine learning train/test splits,
+training and testing a model. The execution of these steps (and parent pipeline) will be recorded by the CMF.
+```shell
+# Run the example pipeline
+sh ./test_script.sh
+```
+
+This script will run the pipeline and will store its metadata in a sqlite file named mlmd. Verify that all stages are
+done using `git log` command. You should see commits corresponding to the artifacts that were created.
+
+Under normal conditions, the next steps would be to: (1) execute the `dvc push` command to push the artifacts to dvc
+remote and (2) execute the `git push origin` command to track the metadata of the generated artifacts.
+
+
+## Query
+The stored metadata can be explored using the query layer. Example Jupyter notebook
+[Query_Tester-base_mlmd.ipynb](https://github.com/HewlettPackard/cmf/blob/master/examples/example-get-started/Query_Tester-base_mlmd.ipynb)
+can be found in this directory.
+
+## Clean Up
+Metadata is stored in sqlite file named "mlmd". To clean up, delete the "mlmd" file.
+
+## Steps to test dataslice
+Run the following command: `python test-data-slice.py`.
diff --git a/docs/extra.css b/docs/extra.css
new file mode 100644
index 00000000..dc646e02
--- /dev/null
+++ b/docs/extra.css
@@ -0,0 +1,3 @@
+.md-typeset__table {
+ min-width: 100%;
+}
diff --git a/docs/index.md b/docs/index.md
new file mode 100644
index 00000000..3dac5a26
--- /dev/null
+++ b/docs/index.md
@@ -0,0 +1,339 @@
+# CMF in a nutshell
+
+CMF (Common Metadata Framework) collects and stores information associated with Machine Learning (ML) pipelines. It
+also implements APIs to query this metadata. The CMF adopts a data-first approach: all artifacts (such as datasets, ML
+models and performance metrics) recorded by the framework are versioned and identified by their content hash.
+
+## Installation
+CMF requires 3.7 >= Python <= 3.9. Create python virtual environment:
+
+=== "Conda"
+ ```shell
+ conda create -n cmf python=3.8
+ conda activate cmf
+ ```
+
+=== "VirtualEnv"
+ ```shell
+ virtualenv --python=3.8 .cmf
+ source .cmf/bin/activate
+ ```
+
+Install CMF
+
+=== "Latest version form GitHub"
+ ```shell
+ pip install https://github.com/HewlettPackard/cmf
+ ```
+
+=== "Stable version form PyPI"
+ ```shell
+ # Work in progress: not available yet.
+ # pip install cmflib
+ ```
+### [Jupyter Lab docker container with CMF pre-installed](#docker-section)
+## Introduction
+Complex ML projects rely on `ML pipelines` to train and test ML models. An ML pipeline is a sequence of stages where
+each stage performs a particular task, such as data loading and pre-processing, ML model training and testing. Stages
+
+- consume `inputs` and produce `outputs`.
+- are parametrized by parameters that guide the process of producing outputs.
+
+
+
+CMF uses abstractions defined in [ML Metadata](https://www.tensorflow.org/tfx/guide/mlmd) (MLMD) library to represent
+CMF concepts. Each pipeline has a name. Users provide it when they initialize the CMF. Each stage is characterized by
+metadata represented as MLMD's `Context` object. When users actually run a stage, this is recorded by the MLMD's
+`Execution` object. Inputs and outputs of stages are represented as MLMD's `Artifact` object, while parameters of stages
+are recorded as properties of executions.
+
+| + +=== "1 Init" + Start tracking the pipeline metadata by initializing the CMF runtime. The metadata will be associated with the + pipeline named `test_pipeline`. + ```python + from cmflib.cmf import Cmf + from ml_metadata.proto import metadata_store_pb2 as mlpb + + cmf = Cmf( + filename="mlmd", + pipeline_name="test_pipeline", + ) + ``` + +=== "2 Stage type" + Before we can start tracking metadata, we need to let CMF know about stage type. This is not yet associated with + this particular execution. + ```python + context: mlmd.proto.Context = cmf.create_context( + pipeline_stage="train" + ) + ``` + +=== "3 New execution" + Now we can create a new stage execution associated with the `train` stage. The CMF always creates a new execution, + and will adjust its name, so it's unique. This is also the place where we log stage `parameters`. + ```python + execution: mlmd.proto.Execution = cmf.create_execution( + execution_type="train", + custom_properties = {"num_epochs": 100, "learning_rate": 0.01} + ) + ``` + +=== "4 Log Artifacts" + Finally, we can log an input (train dataset), and once trained, an output (ML model) artifacts. + ```python + cmf.log_dataset( + 'artifacts/test_dataset.csv', # Dataset path + "input" # This is INPUT artifact + ) + cmf.log_model( + "artifacts/model.pkl", # Model path + event="output" # This is OUTPUT artifact + ) + ``` + + + | +
+
+ |
+
+ `create .env file in current folder using env-example as a template. Modify the .env file for the following variables +USER,UID,GROUP,GID,GIT_USER_NAME,GIT_USER_EMAIL,GIT_REMOTE_URL #These are used by docker-compose.yml`
+#### Step 2.
+**Update `docker-compose.yml` as needed.**
+ your .ssh folder is mounted inside the docker conatiner to enable you to push and pull code from git
+ **To-Do**
+ Create these directories in your home folder
+``` +mkdir $HOME/workspace +mkdir $HOME/dvc_remote +``` +workspace - workspace will be mounted inside the cmf pre-installed docker conatiner (can be your code directory)
+dvc_remote - remote data store for dvc
+ +***or***
+Change the below lines in docker-compose to reflect the appropriate directories
+``` + If your workspace is named "experiment" change the below line +$HOME/workspace:/home/jovyan/workspace to +$HOME/experiment:/home/jovyan/wokspace +``` +``` +If your remote is /extmount/data change the line +$HOME/dvc_remote:/home/jovyan/dvc_remote to +/extmount/data:/home/jovyan/dvc_remote +``` +***Start the docker*** +``` +docker-compose up --build -d +``` +***Access the jupyter notebook*** +http://[HOST.IP.AD.DR]:8888 (default token: `docker`) + +Click the terminal icon
+
+***Quick Start*** +``` +cd example-get-started +sh initialize.sh +sh test_script.sh +dvc push +``` +The above steps will run a pre coded example pipeline and the metadata is stored in a file named "mlmd".
+The artifacts created will be pushed to configured dvc remote (default: /home/dvc_remote)
+The stored metadata is displayed as + + +Metadata lineage can be accessed in neo4j.
+Open http://host:7475/browser/ +Connect to server with default password neo4j123 (To change this modify .env file)
+
+Run the query
+``` +MATCH (a:Execution)-[r]-(b) WHERE (b:Dataset or b:Model or b:Metrics) RETURN a,r, b +``` +Expected output
+
+ +***Jupyter Lab Notebook***
+Select the kernel as Python[conda env:python37]
+
+ +***Shutdown/remove (Remove volumes as well)*** +``` +docker-compose down -v +``` + +## License +CMF is an open source project hosted on [GitHub](https://github.com/HewlettPackard/cmf) and distributed according to +the Apache 2.0 [licence](https://github.com/HewlettPackard/cmf/blob/master/LICENSE). We are welcome user contributions - +send us a message on the Slack [channel](https://commonmetadata.slack.com/) or open a GitHub +[issue](https://github.com/HewlettPackard/cmf/issues) or a [pull request](https://github.com/HewlettPackard/cmf/pulls) +on GitHub. + +## Citation +```bibtex +@mist{foltin2022cmf, + title={Self-Learning Data Foundation for Scientific AI}, + author={Martin Foltin, Annmary Justine, Sergey Serebryakov, Cong Xu, Aalap Tripathy, Suparna Bhattacharya, + Paolo Faraboschi}, + year={2022}, + note = {Presented at the "Monterey Data Conference"}, + URL={https://drive.google.com/file/d/1Oqs0AN0RsAjt_y9ZjzYOmBxI8H0yqSpB/view}, +} +``` + +## Community +[
](https://commonmetadata.slack.com/)
+
+!!! help
+
+ Common Metadata Framework and its documentation are in active stage of development and are very new. If there is
+ anything unclear, missing or there's a typo, please, open an issue or pull request
+ on [GitHub](https://github.com/HewlettPackard/cmf).
diff --git a/docs/requirements.txt b/docs/requirements.txt
new file mode 100644
index 00000000..b2ed12cc
--- /dev/null
+++ b/docs/requirements.txt
@@ -0,0 +1,4 @@
+mkdocs>=1.0.4
+mkdocs-material>=4.4.0
+pymdown-extensions>=7.1
+mkdocstrings[python]
\ No newline at end of file
diff --git a/env-example b/env-example
index 0f8ae362..de5cd5c3 100644
--- a/env-example
+++ b/env-example
@@ -1,16 +1,14 @@
-USER=tripataa
-UID=121956566
+USER=frist
+UID=123456789
GROUP=users
GID=100
-AWS_ACCESS_KEY_ID=blabla
-AWS_SECRET_ACCESS_KEY=blabla
-DVC_REMOTE_URL=/tmp/myremote
-GIT_USER_NAME=aalap-tripathy
-GIT_USER_EMAIL=aalap.tripathy@hpe.com
-GIT_REMOTE_URL=git@github.hpe.com:aalap-tripathy/cmd-test.git
-GIT_BRANCH=tripataa_experiments
-MYSQL_ROOT_PASSWORD=password
-NEO4J_USER_NAME=neo4j
-NEO4J_PASSWD=neo4j123
+DVC_REMOTE_URL=/home/jovyan/dvc_remote
+GIT_USER_NAME=first.second
+GIT_USER_EMAIL=first.second@hpe.com
+GIT_REMOTE_URL=git@github.hpe.com:first-second/cmf-demo.git
+GIT_BRANCH=first_checkout
+NEO4J_USER_NAME=USER
+NEO4J=TRUE
+NEO4J_PASSWD=XXXXX
HTTP_PROXY=http://proxy.houston.hpecorp.net:8080
-HTTPS_PROXY=http://proxy.houston.hpecorp.net:8080
\ No newline at end of file
+HTTPS_PROXY=http://proxy.houston.hpecorp.net:8080
diff --git a/examples/al_object_detection/dvc_cmf_ingest.py b/examples/active_learning/dvc_cmf_ingest.py
similarity index 97%
rename from examples/al_object_detection/dvc_cmf_ingest.py
rename to examples/active_learning/dvc_cmf_ingest.py
index 215cd282..aa3baf8c 100644
--- a/examples/al_object_detection/dvc_cmf_ingest.py
+++ b/examples/active_learning/dvc_cmf_ingest.py
@@ -46,14 +46,14 @@ def get_cmf_hierarchy(execution_lineage:str):
execution_exist : True if it exeist, False otherwise
metawrite: cmf object
"""
-def ingest_metadata(execution_lineage:str, metadata:dict, execution_exist:bool, metawriter:cmf.Cmf, command:str = "") :
+def ingest_metadata(execution_lineage:str, metadata:dict, execution_exist:bool, metawriter:cmf.Cmf) :
pipeline_name, context_name, execution = get_cmf_hierarchy(execution_lineage)
_ = metawriter.create_context(pipeline_stage=context_name)
if execution_exist:
_ = metawriter.update_execution(int(execution))
else :
- _ = metawriter.create_execution(execution, {}, command)
+ _ = metawriter.create_execution(execution)
for k, v in metadata.items():
if k == "deps":
@@ -123,7 +123,7 @@ def ingest_metadata(execution_lineage:str, metadata:dict, execution_exist:bool,
pipeline_name = "Pipeline"+"-"+str(uuid.uuid4()) if not pipeline_name else pipeline_name
-metawriter = cmf.Cmf(filename="mlmd", pipeline_name=pipeline_name, graph=False)
+metawriter = cmf.Cmf(filename="mlmd", pipeline_name=pipeline_name, graph=True)
"""
Parse the dvc.lock dictionary and get the command section
@@ -146,6 +146,8 @@ def ingest_metadata(execution_lineage:str, metadata:dict, execution_exist:bool,
"""
vvv.pop(0)
cmd = cmd_exe.get(str(vvv), None)
+ print(cmd_exe)
+ print(str(vvv))
if cmd is not None:
"""
cmd(lineage) - eg - '1,eval,active_learning '
@@ -160,6 +162,6 @@ def ingest_metadata(execution_lineage:str, metadata:dict, execution_exist:bool,
context_name = k
execution_name = vvv[0]
lineage = execution_name+","+context_name+","+ pipeline_name
- ingest_metadata(lineage, vv, False, metawriter, str(vvv))
+ ingest_metadata(lineage, vv, False, metawriter)
metawriter.log_dvc_lock("dvc.lock")
diff --git a/examples/active_learning/sample_env b/examples/active_learning/sample_env
index a58e9d43..7892bbdb 100644
--- a/examples/active_learning/sample_env
+++ b/examples/active_learning/sample_env
@@ -1,7 +1,7 @@
export DVC_REMOTE_URL=/tmp/myremote
-export GIT_USER_NAME=annmary-roy@hpe.com
-export GIT_USER_EMAIL=annmary.roy@hpe.com
-export GIT_REMOTE_URL=git@github.hpe.com:annmary-roy/crop-dataset.git
-export NEO4J_USER_NAME=neo4j
-export NEO4J_PASSWD=test
-export NEO4J_URI="bolt://ai-datafoundation.labs.hpecorp.net:7687"
+export GIT_USER_NAME=first-second@hpe.com
+export GIT_USER_EMAIL=first.second@hpe.com
+export GIT_REMOTE_URL=git@github.hpe.com:first-second/crop-dataset.git
+export NEO4J_USER_NAME=user
+export NEO4J_PASSWD=XXXX
+export NEO4J_URI="bolt://xxxx:xxx"
diff --git a/examples/al_object_detection/.gitignore b/examples/al_object_detection/.gitignore
index a6d44692..9d4bce4f 100644
--- a/examples/al_object_detection/.gitignore
+++ b/examples/al_object_detection/.gitignore
@@ -1,3 +1,4 @@
**/__pyccache__
/work_dirs
*.pyc
+/al_cycle_next.json
diff --git a/examples/al_object_detection/al_cycle.json b/examples/al_object_detection/al_cycle.json
new file mode 100644
index 00000000..50184808
--- /dev/null
+++ b/examples/al_object_detection/al_cycle.json
@@ -0,0 +1,5 @@
+{
+ "al_cycle": 1,
+ "next_cycle": 2,
+ "al_seed": 123
+}
diff --git a/examples/al_object_detection/cmf_dvc_ingest.py b/examples/al_object_detection/cmf_dvc_ingest.py
new file mode 100644
index 00000000..459c84ee
--- /dev/null
+++ b/examples/al_object_detection/cmf_dvc_ingest.py
@@ -0,0 +1,187 @@
+###
+# Copyright (2024) Hewlett Packard Enterprise Development LP
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# You may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+###
+
+import argparse
+import yaml
+import pandas as pd
+import typing as t
+import uuid
+from ml_metadata.metadata_store import metadata_store
+from ml_metadata.proto import metadata_store_pb2 as mlpb
+from cmflib import cmfquery
+from cmflib import cmf
+
+
+"""
+Ingest the metadata in dvc.lock file into CMF.
+If cmf mlmd file exist with metadata from metrics logging or logging for other
+metadata not captured in dvc.lock file, pass that mlmd file as the input file.
+This code queries the file for existing pipelines, stages and executions and stores
+as a dictionary. If the execution cmd in the stored dict, matches the execution command
+in the dvc.lock file, that execution is updated with additional metadata.
+If there is no prior execution captured, a new execution is created
+"""
+uuid_ = str(uuid.uuid4())
+
+pipeline_name = ""
+
+parser = argparse.ArgumentParser()
+"""
+File name of the cmf metadata file . Pass the file name if the pipeline has been recorded
+with CMF explicit log statements , to record metadata not part of dvc.lock file
+"""
+parser.add_argument('--cmf_filename', type=str, default="mlmd", help="cmf filename")
+args = parser.parse_args()
+
+"""
+Parses the string and returns, pipeline name, context name and execution name
+"""
+def get_cmf_hierarchy(execution_lineage:str):
+ cmf_levels = execution_lineage.split(',')
+ return cmf_levels[-1], cmf_levels[1], cmf_levels[0]
+
+"""
+Ingest the metadata into cmf
+args
+ execution_lineage: format- execution id(if present)/execution file name/context/pipeline
+ eg - demo_train.py,active_learning_training@2,active_learning -- without existing execution
+ - 3,eval,active_learning -- with existing execution in cmf metadata file
+ metadata: The parsed dictionary from dvc.lock file
+ execution_exist : True if it exeist, False otherwise
+ metawrite: cmf object
+"""
+def ingest_metadata(execution_lineage:str, metadata:dict, metawriter:cmf.Cmf, command:str = "") :
+ pipeline_name, context_name, execution = get_cmf_hierarchy(execution_lineage)
+
+ _ = metawriter.create_execution(
+ str(context_name) + '_' + str(execution),
+ {},
+ cmd = str(command),
+ create_new_execution=False
+ )
+
+ for k, v in metadata.items():
+ if k == "deps":
+ for dep in v:
+ metawriter.log_dataset_with_version(dep["path"], dep["md5"], "input")
+ if k == "outs":
+ for out in v:
+ metawriter.log_dataset_with_version(out["path"], out["md5"], "output")
+
+def find_location(string, elements):
+ for index, element in enumerate(elements):
+ if string == element:
+ return index
+ return None
+
+#Query mlmd to get all the executions and its commands
+cmd_exe = {}
+cmf_query = cmfquery.CmfQuery(args.cmf_filename)
+pipelines: t.List[str] = cmf_query.get_pipeline_names()
+for pipeline in pipelines:
+ pipeline_name = pipeline
+ stages: t.List[str] = cmf_query.get_pipeline_stages(pipeline)
+ for stage in stages:
+ exe_df: pd.DataFrame = cmf_query.get_all_executions_in_stage(stage)
+ """
+ Parse all the executions in a stage
+ eg- exe_step = ['demo_eval.py', '--trained_model', 'data/model-1', '--enable_df', 'True', '--round', '1']
+ """
+ for index, row in exe_df.iterrows():
+ exe_step = row['Execution']
+ '''
+ if already same execution command has been captured previously use the latest
+ execution id to associate the new metadata
+ '''
+ if None is cmd_exe.get(exe_step, None):
+ cmd_exe[exe_step] = str(row['id']) + "," + stage + "," + pipeline
+ else:
+ if row['id'] > int(cmd_exe.get(exe_step, None).split(',')[0]):
+ cmd_exe[exe_step] = str(row['id']) + "," + stage + "," + pipeline
+
+"""
+Parse the dvc.lock file.
+"""
+pipeline_dict = {}
+with open("dvc.lock", 'r') as f:
+ valuesYaml = yaml.load(f, Loader=yaml.FullLoader)
+
+for k in valuesYaml['stages']:
+ pipeline_dict[k] = {}
+ commands=[]
+ deps = []
+ outs = []
+ k_dict = {}
+ i = 0
+
+ for kk in valuesYaml['stages'][k]:
+ if kk == 'cmd':
+ cmd_list = valuesYaml['stages'][k][kk].split()
+ commands.append(cmd_list)
+ k_dict['cmd'] = cmd_list
+ i = i + 1
+ if kk == 'deps':
+ deps = valuesYaml['stages'][k][kk]
+ k_dict['deps'] = deps
+ if kk == 'outs':
+ outs = valuesYaml['stages'][k][kk]
+ k_dict['outs'] = outs
+
+ pipeline_dict[k][str(i)] = k_dict
+
+"""
+Create a unique Pipeline name if there is no mlmd file
+"""
+
+
+pipeline_name = "Pipeline"+"-"+str(uuid_) if not pipeline_name else pipeline_name
+metawriter = cmf.Cmf(filename="mlmd", pipeline_name=pipeline_name, graph=True)
+
+"""
+Parse the dvc.lock dictionary and get the command section
+"""
+for k, v in pipeline_dict.items():
+ for kk, vv in v.items():
+ for kkk, vvv in vv.items():
+ if kkk == 'cmd':
+ """
+ Key eg - cmd
+ Value eg - ['python3', 'demo.py', '--enable_df', 'True']
+ cmd_exe eg - {"['demo_eval.py', '--trained_model', 'data/model-3', '--enable_df', 'True', '--round', '3']":
+ '3,eval,active_learning',
+ "['demo_eval.py', '--trained_model', 'data/model-2', '--enable_df', 'True', '--round', '2']": '2,eval,active_learning',
+ "['demo_eval.py', '--trained_model', 'data/model-1', '--enable_df', 'True', '--round', '1']": '1,eval,active_learning'}
+ In the next line pop out the python
+ if the pipeline_dict command is already there in the cmd_exe dict got from parsing the mlmd pop that cmd out
+ and use the stored lineage from the mlmd
+ """
+ vvv.pop(0)
+ pos = find_location('--execution_name', vvv)
+ if pos:
+ execution_name = vvv[pos+1]
+ else:
+ execution_name = uuid_
+
+ context_name = k
+ lineage = execution_name+","+context_name+","+ pipeline_name
+
+ cmd = cmd_exe.get(str(' '.join(vvv)), None)
+ _ = metawriter.create_context(pipeline_stage=context_name)
+
+ ingest_metadata(lineage, vv, metawriter, str(' '.join(vvv)))
+
+
+metawriter.log_dvc_lock("dvc.lock")
diff --git a/examples/al_object_detection/cmf_logger_mmcv/__init__.py b/examples/al_object_detection/cmf_logger_mmcv/__init__.py
new file mode 100644
index 00000000..53c1122e
--- /dev/null
+++ b/examples/al_object_detection/cmf_logger_mmcv/__init__.py
@@ -0,0 +1,72 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+from .base_module import BaseModule, ModuleDict, ModuleList, Sequential
+from .base_runner import BaseRunner
+from .builder import RUNNERS, build_runner
+from .checkpoint import (CheckpointLoader, _load_checkpoint,
+ _load_checkpoint_with_prefix, load_checkpoint,
+ load_state_dict, save_checkpoint, weights_to_cpu)
+from .default_constructor import DefaultRunnerConstructor
+from .dist_utils import (allreduce_grads, allreduce_params, get_dist_info,
+ init_dist, master_only)
+from .epoch_based_runner import EpochBasedRunner, Runner
+from .fp16_utils import LossScaler, auto_fp16, force_fp32, wrap_fp16_model
+from .hooks import (HOOKS, CheckpointHook, ClosureHook, DistEvalHook,
+ DistSamplerSeedHook, DvcliveLoggerHook, EMAHook, EvalHook,
+ Fp16OptimizerHook, GradientCumulativeFp16OptimizerHook,
+ GradientCumulativeOptimizerHook, Hook, IterTimerHook,
+ LoggerHook, MlflowLoggerHook, NeptuneLoggerHook,
+ OptimizerHook, PaviLoggerHook, SegmindLoggerHook,
+ SyncBuffersHook, TensorboardLoggerHook, TextLoggerHook,
+ WandbLoggerHook, CmfLoggerHook)
+from .hooks.lr_updater import StepLrUpdaterHook # noqa
+from .hooks.lr_updater import (CosineAnnealingLrUpdaterHook,
+ CosineRestartLrUpdaterHook, CyclicLrUpdaterHook,
+ ExpLrUpdaterHook, FixedLrUpdaterHook,
+ FlatCosineAnnealingLrUpdaterHook,
+ InvLrUpdaterHook, LinearAnnealingLrUpdaterHook,
+ LrUpdaterHook, OneCycleLrUpdaterHook,
+ PolyLrUpdaterHook)
+from .hooks.momentum_updater import (CosineAnnealingMomentumUpdaterHook,
+ CyclicMomentumUpdaterHook,
+ LinearAnnealingMomentumUpdaterHook,
+ MomentumUpdaterHook,
+ OneCycleMomentumUpdaterHook,
+ StepMomentumUpdaterHook)
+from .iter_based_runner import IterBasedRunner, IterLoader
+from .log_buffer import LogBuffer
+from .optimizer import (OPTIMIZER_BUILDERS, OPTIMIZERS,
+ DefaultOptimizerConstructor, build_optimizer,
+ build_optimizer_constructor)
+from .priority import Priority, get_priority
+from .utils import get_host_info, get_time_str, obj_from_dict, set_random_seed
+
+# initialize ipu to registor ipu runner to RUNNERS
+from mmcv.device import ipu # isort:skip # noqa
+
+__all__ = [
+ 'BaseRunner', 'Runner', 'EpochBasedRunner', 'IterBasedRunner', 'LogBuffer',
+ 'HOOKS', 'Hook', 'CheckpointHook', 'ClosureHook', 'LrUpdaterHook',
+ 'FixedLrUpdaterHook', 'StepLrUpdaterHook', 'ExpLrUpdaterHook',
+ 'PolyLrUpdaterHook', 'InvLrUpdaterHook', 'CosineAnnealingLrUpdaterHook',
+ 'FlatCosineAnnealingLrUpdaterHook', 'CosineRestartLrUpdaterHook',
+ 'CyclicLrUpdaterHook', 'OneCycleLrUpdaterHook', 'MomentumUpdaterHook',
+ 'StepMomentumUpdaterHook', 'CosineAnnealingMomentumUpdaterHook',
+ 'CyclicMomentumUpdaterHook', 'OneCycleMomentumUpdaterHook',
+ 'OptimizerHook', 'IterTimerHook', 'DistSamplerSeedHook', 'LoggerHook',
+ 'PaviLoggerHook', 'TextLoggerHook', 'TensorboardLoggerHook',
+ 'NeptuneLoggerHook', 'WandbLoggerHook', 'MlflowLoggerHook',
+ 'DvcliveLoggerHook', '_load_checkpoint', 'load_state_dict',
+ 'load_checkpoint', 'weights_to_cpu', 'save_checkpoint', 'Priority',
+ 'get_priority', 'get_host_info', 'get_time_str', 'obj_from_dict',
+ 'init_dist', 'get_dist_info', 'master_only', 'OPTIMIZER_BUILDERS',
+ 'OPTIMIZERS', 'DefaultOptimizerConstructor', 'build_optimizer',
+ 'build_optimizer_constructor', 'IterLoader', 'set_random_seed',
+ 'auto_fp16', 'force_fp32', 'wrap_fp16_model', 'Fp16OptimizerHook',
+ 'SyncBuffersHook', 'EMAHook', 'build_runner', 'RUNNERS', 'allreduce_grads',
+ 'allreduce_params', 'LossScaler', 'CheckpointLoader', 'BaseModule',
+ '_load_checkpoint_with_prefix', 'EvalHook', 'DistEvalHook', 'Sequential',
+ 'ModuleDict', 'ModuleList', 'GradientCumulativeOptimizerHook',
+ 'GradientCumulativeFp16OptimizerHook', 'DefaultRunnerConstructor',
+ 'SegmindLoggerHook', 'LinearAnnealingMomentumUpdaterHook',
+ 'LinearAnnealingLrUpdaterHook', 'CmfLoggerHook'
+]
diff --git a/examples/al_object_detection/cmf_logger_mmcv/hooks/__init__.py b/examples/al_object_detection/cmf_logger_mmcv/hooks/__init__.py
new file mode 100644
index 00000000..501562f8
--- /dev/null
+++ b/examples/al_object_detection/cmf_logger_mmcv/hooks/__init__.py
@@ -0,0 +1,47 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+from .checkpoint import CheckpointHook
+from .closure import ClosureHook
+from .ema import EMAHook
+from .evaluation import DistEvalHook, EvalHook
+from .hook import HOOKS, Hook
+from .iter_timer import IterTimerHook
+from .logger import (DvcliveLoggerHook, LoggerHook, MlflowLoggerHook,
+ NeptuneLoggerHook, PaviLoggerHook, SegmindLoggerHook,
+ TensorboardLoggerHook, TextLoggerHook, WandbLoggerHook, CmfLoggerHook)
+from .lr_updater import (CosineAnnealingLrUpdaterHook,
+ CosineRestartLrUpdaterHook, CyclicLrUpdaterHook,
+ ExpLrUpdaterHook, FixedLrUpdaterHook,
+ FlatCosineAnnealingLrUpdaterHook, InvLrUpdaterHook,
+ LinearAnnealingLrUpdaterHook, LrUpdaterHook,
+ OneCycleLrUpdaterHook, PolyLrUpdaterHook,
+ StepLrUpdaterHook)
+from .memory import EmptyCacheHook
+from .momentum_updater import (CosineAnnealingMomentumUpdaterHook,
+ CyclicMomentumUpdaterHook,
+ LinearAnnealingMomentumUpdaterHook,
+ MomentumUpdaterHook,
+ OneCycleMomentumUpdaterHook,
+ StepMomentumUpdaterHook)
+from .optimizer import (Fp16OptimizerHook, GradientCumulativeFp16OptimizerHook,
+ GradientCumulativeOptimizerHook, OptimizerHook)
+from .profiler import ProfilerHook
+from .sampler_seed import DistSamplerSeedHook
+from .sync_buffer import SyncBuffersHook
+
+__all__ = [
+ 'HOOKS', 'Hook', 'CheckpointHook', 'ClosureHook', 'LrUpdaterHook',
+ 'FixedLrUpdaterHook', 'StepLrUpdaterHook', 'ExpLrUpdaterHook',
+ 'PolyLrUpdaterHook', 'InvLrUpdaterHook', 'CosineAnnealingLrUpdaterHook',
+ 'FlatCosineAnnealingLrUpdaterHook', 'CosineRestartLrUpdaterHook',
+ 'CyclicLrUpdaterHook', 'OneCycleLrUpdaterHook', 'OptimizerHook',
+ 'Fp16OptimizerHook', 'IterTimerHook', 'DistSamplerSeedHook',
+ 'EmptyCacheHook', 'LoggerHook', 'MlflowLoggerHook', 'PaviLoggerHook',
+ 'TextLoggerHook', 'TensorboardLoggerHook', 'NeptuneLoggerHook',
+ 'WandbLoggerHook', 'DvcliveLoggerHook', 'MomentumUpdaterHook',
+ 'StepMomentumUpdaterHook', 'CosineAnnealingMomentumUpdaterHook',
+ 'CyclicMomentumUpdaterHook', 'OneCycleMomentumUpdaterHook',
+ 'SyncBuffersHook', 'EMAHook', 'EvalHook', 'DistEvalHook', 'ProfilerHook',
+ 'GradientCumulativeOptimizerHook', 'GradientCumulativeFp16OptimizerHook',
+ 'SegmindLoggerHook', 'LinearAnnealingLrUpdaterHook',
+ 'LinearAnnealingMomentumUpdaterHook', 'CmfLoggerHook'
+]
diff --git a/examples/al_object_detection/cmf_logger_mmcv/hooks/logger/__init__.py b/examples/al_object_detection/cmf_logger_mmcv/hooks/logger/__init__.py
new file mode 100644
index 00000000..da3dedd1
--- /dev/null
+++ b/examples/al_object_detection/cmf_logger_mmcv/hooks/logger/__init__.py
@@ -0,0 +1,17 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+from .base import LoggerHook
+from .dvclive import DvcliveLoggerHook
+from .mlflow import MlflowLoggerHook
+from .neptune import NeptuneLoggerHook
+from .pavi import PaviLoggerHook
+from .segmind import SegmindLoggerHook
+from .tensorboard import TensorboardLoggerHook
+from .text import TextLoggerHook
+from .wandb import WandbLoggerHook
+from .cmflogger import CmfLoggerHook
+
+__all__ = [
+ 'LoggerHook', 'CmfLoggerHook', 'MlflowLoggerHook', 'PaviLoggerHook',
+ 'TensorboardLoggerHook', 'TextLoggerHook', 'WandbLoggerHook',
+ 'NeptuneLoggerHook', 'DvcliveLoggerHook', 'SegmindLoggerHook'
+]
diff --git a/examples/al_object_detection/cmf_logger_mmcv/hooks/logger/cmflogger.py b/examples/al_object_detection/cmf_logger_mmcv/hooks/logger/cmflogger.py
new file mode 100644
index 00000000..1c3e0cff
--- /dev/null
+++ b/examples/al_object_detection/cmf_logger_mmcv/hooks/logger/cmflogger.py
@@ -0,0 +1,105 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+
+###
+# Copyright (2024) Hewlett Packard Enterprise Development LP
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# You may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+###
+
+from mmcv.utils import TORCH_VERSION
+from ...dist_utils import master_only
+from ..hook import HOOKS
+from .base import LoggerHook
+import uuid
+import os
+import sys
+
+
+@HOOKS.register_module()
+class CmfLoggerHook(LoggerHook):
+ """Class to log metrics and (optionally) a trained model to CMF.
+
+ It requires `cmflib`_ to be installed.
+
+ Args:
+ mlmd_file_path (str): Path to mlmd file.
+ exp_name (str, optional): Name of the experiment to be used.
+ Default None. If not None, set the active experiment.
+ If experiment_name does not exist, an experiment_name
+ with be created using uuid.
+ tags (Dict[str], optional): Tags for the current run.
+ Default None. If not None, set tags for the current run.
+ interval (int): Logging interval (every k iterations). Default: 10.
+ ignore_last (bool): Ignore the log of last iterations in each epoch
+ if less than `interval`. Default: True.
+ reset_flag (bool): Whether to clear the output buffer after logging.
+ Default: False.
+ by_epoch (bool): Whether EpochBasedRunner is used. Default: True.
+ """
+ Training_metric_count = 1
+ Validation_metric_count = 1
+
+ @master_only
+ def __init__(self,
+ mlmd_file_path = os.path.join('/'+os.environ['DVC_ROOT']) + '/mlmd',
+ exp_name=None,
+ tags=None,
+ interval=10, #needed by super class
+ ignore_last=True, #needed by super class
+ reset_flag=False, #needed by super class
+ by_epoch=True): #needed by super class
+ super(CmfLoggerHook, self).__init__(interval, ignore_last,
+ reset_flag, by_epoch)
+ self.mlmdFilePath = mlmd_file_path
+ self.exp_name = exp_name if exp_name else str(uuid.uuid4())
+ self.tags = tags
+
+ try:
+ from cmflib import cmf
+ from cmflib import cmfquery
+ except ImportError:
+ raise ImportError(
+ 'Please run "pip install cmflib" to install cmflib')
+ self.cmf_logger = cmf.Cmf(filename=self.mlmdFilePath, pipeline_name=self.exp_name , graph=True)
+ self.context = self.cmf_logger.create_context(os.environ['stage_name'])
+
+ cmd = str(' '.join(sys.argv))
+ self.execution = self.cmf_logger.create_execution(
+ str(os.environ['stage_name'])+'_'+str(os.environ['execution_name']),
+ {},
+ cmd = str(cmd),
+ create_new_execution=False
+ )
+
+ self.prefix = 'Training_Metrics_' + str(os.environ['stage_name'])+'_'+str(os.environ['execution_name'])
+
+ @master_only
+ def log(self, runner):
+ tags = self.get_loggable_tags(runner)
+ mode = self.get_mode(runner)
+ self.mode = mode
+ if tags:
+ if mode == 'train':
+ self.commit_name = self.prefix + '_' + str(CmfLoggerHook.Training_metric_count)
+ self.cmf_logger.log_metric(self.commit_name, tags)
+
+ else:
+ prefix = 'Validation_Metrics_' + str(os.environ['stage_name'])+'_'+str(os.environ['execution_name'])
+ commit_name = prefix + '_' + str(CmfLoggerHook.Validation_metric_count)
+ self.cmf_logger.log_execution_metrics(commit_name, tags)
+ CmfLoggerHook.Validation_metric_count+=1
+
+ @master_only
+ def after_run(self, runner):
+ self.cmf_logger.commit_metrics(self.commit_name)
+ CmfLoggerHook.Training_metric_count+=1
diff --git a/examples/al_object_detection/configs/MIAOD-GRAY.py b/examples/al_object_detection/configs/MIAOD-GRAY.py
index 4ca6556f..efa9faa0 100644
--- a/examples/al_object_detection/configs/MIAOD-GRAY.py
+++ b/examples/al_object_detection/configs/MIAOD-GRAY.py
@@ -1,5 +1,6 @@
# Please change the dataset directory to your actual directory
-data_root = '/mnt/beegfs/PAMS/data/tomography_data/tiled_annotations/'
+#data_root = '/lustre/data/hdcdatasets/'
+data_root = '/mnt/beegfs/HDC/data/tomography_data/tiled_annotations'
_base_ = [
'./_base_/retinanet_r50_fpn.py', './_base_/hdc.py',
@@ -9,16 +10,17 @@
data = dict(
test=dict(
ann_file=[
- data_root + 'train.txt'#TRAINING SET INDEXES?
+ data_root + '/train.txt',
],
- img_prefix=[data_root])
+ img_prefix=[data_root ])
)
-model = dict(bbox_head=dict(C=2))
+model = dict(bbox_head=dict(C=1))
# The initial learning rate, momentum, weight decay can be changed here.
-optimizer = dict(type='SGD', lr=1e-6, momentum=0.9, weight_decay=0.0001)#changed lr from 1e-3
-#optimizer = dict(type='Adam', lr=1e-6, momentum=0.9, weight_decay=0.0001)#changed lr from 1e-3
+optimizer = dict(type='Adam', lr=12e-6, weight_decay=0.0001)
+
+optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))
+#optimizer_config = dict(grad_clip=None)
-optimizer_config = dict(grad_clip=None)
# The moment when the learning rate drops can be changed here.
lr_config = dict(policy='step', step=[2])
# The frequency of saving models can be changed here.
@@ -31,7 +33,7 @@
# The frequency of evaluating the model can be changed here.
evaluation = dict(interval=epoch_ratio[0], metric='mAP')
# The number of outer loops (i.e., all 3 training steps except the first Label Set Training step) can be changed here.
-epoch = 1
+epoch = 7
# The repeat time for the labeled sets and unlabeled sets can be changed here.
# The number of repeat times can be equivalent to the number of actual training epochs.
X_L_repeat = 2
@@ -41,10 +43,16 @@
k = 10000
# The size of the initial labeled set and the newly selected sets after each cycle can be set here.
# Note that there are 16551 images in the PASCAL VOC 2007+2012 trainval sets.
-X_S_size = 12150#9964//40 #should be percentage of all available images (set this to max)
-X_L_0_size = 12150#9964//20#(set to zero)?
+def make_even(x):
+ return (x//2)*2
+X_S_size = 1000#make_even(12150//40)#9964//40
+X_L_0_size = make_even(12150)#make_even(12150//10)#9964//20
# The active learning cycles can be changed here.
cycles = [0, 1, 2, 3, 4, 5, 6]
# The work directory for saving logs and files can be changed here. Please refer to README.md for more information.
work_directory = './work_dirs/MI-AOD'
-
+# Output bounding boxes for most uncertain image regions to help guide labeling
+plot_nuboxes = 5
+nubox_colors = [(0,0,255), (0,255,255), (0,255,0), (255,255,0), (255,0,0)]
+check_nuboxes = 1000
+guide_image_dir = data_root + 'hdc/GuideImages'
diff --git a/examples/al_object_detection/configs/MIAOD.py b/examples/al_object_detection/configs/MIAOD.py
index f53cb78b..57198af9 100644
--- a/examples/al_object_detection/configs/MIAOD.py
+++ b/examples/al_object_detection/configs/MIAOD.py
@@ -1,6 +1,6 @@
# Please change the dataset directory to your actual directory
#data_root = '/home/mfoltin2/AI/al/object_detection/data/VOCdevkit/'
-data_root = '/mnt/beegfs/PAMS/data/tomography_data/tiled_annotations/'
+data_root = '/mnt/beegfs/HDC/data/tomography_data/tiled_annotations/'
_base_ = [
'./_base_/retinanet_r50_fpn.py', './_base_/voc0712.py',
diff --git a/examples/al_object_detection/configs/_base_/.hdc.py.swp b/examples/al_object_detection/configs/_base_/.hdc.py.swp
new file mode 100644
index 00000000..fd87f2ff
Binary files /dev/null and b/examples/al_object_detection/configs/_base_/.hdc.py.swp differ
diff --git a/examples/al_object_detection/configs/_base_/default_runtime.py b/examples/al_object_detection/configs/_base_/default_runtime.py
index 2908c54a..eb1d4a0b 100644
--- a/examples/al_object_detection/configs/_base_/default_runtime.py
+++ b/examples/al_object_detection/configs/_base_/default_runtime.py
@@ -3,8 +3,8 @@
log_config = dict(
interval=1,
hooks=[
- dict(type='TextLoggerHook'),
- # dict(type='TensorboardLoggerHook')
+ dict(type='TensorboardLoggerHook'),
+ dict(type = 'CmfLoggerHook', exp_name = 'ExperimentActiveLearning')
])
# yapf:enable
dist_params = dict(backend='nccl')
diff --git a/examples/al_object_detection/configs/_base_/hdc.py b/examples/al_object_detection/configs/_base_/hdc.py
index 68414141..b63ff182 100644
--- a/examples/al_object_detection/configs/_base_/hdc.py
+++ b/examples/al_object_detection/configs/_base_/hdc.py
@@ -1,5 +1,6 @@
# Please change the dataset directory to your actual directory
-data_root = '/mnt/beegfs/PAMS/data/tomography_data/tiled_annotations/'
+#data_root = '/lustre/data/hdcdatasets/'
+data_root = '/mnt/beegfs/HDC/data/tomography_data/tiled_annotations'
# dataset settings
dataset_type = 'HDCDataset'
@@ -10,7 +11,9 @@
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(type='Resize', img_scale=(960, 960), keep_ratio=True),
- dict(type='RandomFlip', flip_ratio=0.66),#,direction=['horizontal','vertical','diagonal']),
+ dict(type='RandomFlip', flip_ratio=0.5),#,direction=['horizontal','vertical','diagonal']),
+ dict(type='RandomFlip', flip_ratio=0.5,direction='vertical'),
+
dict(type='Normalize', **img_norm_cfg),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
@@ -26,7 +29,8 @@
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
- dict(type='RandomFlip', flip_ratio=0.66),#, direction=['horizontal','vertical','diagonal']),
+ dict(type='RandomFlip', flip_ratio=0.5),#,direction=['horizontal','vertical','diagonal']),
+ dict(type='RandomFlip', flip_ratio=0.5,direction='vertical'),
dict(type='Normalize', **img_norm_cfg),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
@@ -34,26 +38,26 @@
])
]
data = dict(
- samples_per_gpu=2,
- workers_per_gpu=2,
+ samples_per_gpu=8,
+ workers_per_gpu=8,
train=dict(
type='RepeatDataset',
- times=3,
+ times=3,#was 3
dataset=dict(
type=dataset_type,
ann_file=[
- data_root + 'train.txt'#training set indexes
+ data_root + '/train.txt'#training set indexes
],
img_prefix=[data_root ],
pipeline=train_pipeline)),
val=dict(
type=dataset_type,
- ann_file=data_root + 'train_val.txt',#test set indexes
+ ann_file=data_root + '/train_val.txt',#test set indexes
img_prefix=data_root,
pipeline=test_pipeline),
test=dict(
type=dataset_type,
- ann_file=data_root + 'train.txt',#should be same as training
+ ann_file=data_root + '/train.txt',#should be same as training
img_prefix=data_root ,
pipeline=test_pipeline))
evaluation = dict(interval=1, metric='mAP')
diff --git a/examples/al_object_detection/configs/_base_/hdc_copy.py b/examples/al_object_detection/configs/_base_/hdc_copy.py
new file mode 100644
index 00000000..af7344d6
--- /dev/null
+++ b/examples/al_object_detection/configs/_base_/hdc_copy.py
@@ -0,0 +1,86 @@
+# Please change the dataset directory to your actual directory
+#data_root = '/lustre/data/hdcdatasets/'
+data_root = '/mnt/beegfs/HDC/data/tomography_data/tiled_annotations/'
+
+# dataset settings
+dataset_type = 'HDCDataset'
+img_norm_cfg = dict(
+ mean=[123.675, 123.675, 123.675], std=[58.395, 58.395, 58.395], to_rgb=True)
+train_pipeline = [
+ #dict(type='LoadImageFromFile', color_type='grayscale'),
+ dict(type='LoadImageFromFile'),
+ dict(type='LoadAnnotations', with_bbox=True),
+ dict(type='Resize', img_scale=(960, 960), keep_ratio=True),
+ dict(type='RandomFlip', flip_ratio=0.5),#,direction=['horizontal','vertical','diagonal']),
+ dict(type='RandomFlip', flip_ratio=0.5,direction='vertical'),
+
+ dict(type='Normalize', **img_norm_cfg),
+ dict(type='Pad', size_divisor=32),
+ '''dict(type='Albu',
+ transforms=[dict(
+ type='ShiftScaleRotate',
+ shift_limit=0.0,
+ scale_limit=0.0,
+ rotate_limit=[90,90],
+ interpolation=1,
+ p=0.5),
+ ],
+ bbox_params=dict(
+ type='BboxParams',
+ format='pascal_voc',
+ label_fields=['gt_labels'],
+ min_visibility=0.0,
+ filter_lost_elements=True),
+ keymap={
+ 'img': 'image',
+ 'gt_masks': 'masks',
+ 'gt_bboxes': 'bboxes'
+ },
+ update_pad_shape=False,
+ skip_img_without_anno=False
+ ),'''
+ dict(type='DefaultFormatBundle'),
+ dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']),
+]
+test_pipeline = [
+
+ #dict(type='LoadImageFromFile', color_type='grayscale'),
+ dict(type='LoadImageFromFile'),
+ dict(
+ type='MultiScaleFlipAug',
+ img_scale=(960, 960),
+ flip=False,
+ transforms=[
+ dict(type='Resize', keep_ratio=True),
+ dict(type='RandomFlip', flip_ratio=0.5),#,direction=['horizontal','vertical','diagonal']),
+ dict(type='RandomFlip', flip_ratio=0.5,direction='vertical'),
+ dict(type='Normalize', **img_norm_cfg),
+ dict(type='Pad', size_divisor=32),
+ dict(type='ImageToTensor', keys=['img']),
+ dict(type='Collect', keys=['img']),
+ ])
+]
+data = dict(
+ samples_per_gpu=8,
+ workers_per_gpu=8,
+ train=dict(
+ type='RepeatDataset',
+ times=3,#was 3
+ dataset=dict(
+ type=dataset_type,
+ ann_file=[
+ data_root + 'train.txt'#training set indexes
+ ],
+ img_prefix=[data_root ],
+ pipeline=train_pipeline)),
+ val=dict(
+ type=dataset_type,
+ ann_file=data_root + 'train_val.txt',#test set indexes
+ img_prefix=data_root,
+ pipeline=test_pipeline),
+ test=dict(
+ type=dataset_type,
+ ann_file=data_root + 'train.txt',#should be same as training
+ img_prefix=data_root ,
+ pipeline=test_pipeline))
+evaluation = dict(interval=1, metric='mAP')
diff --git a/examples/al_object_detection/configs/model.py b/examples/al_object_detection/configs/model.py
new file mode 100644
index 00000000..d66aa47d
--- /dev/null
+++ b/examples/al_object_detection/configs/model.py
@@ -0,0 +1,371 @@
+import torch.nn as nn
+import torch
+import math
+import torch.utils.model_zoo as model_zoo
+from torchvision.ops import nms
+from retinanet.utils import BasicBlock, Bottleneck, BBoxTransform, ClipBoxes
+from retinanet.anchors import Anchors
+from retinanet import losses
+
+model_urls = {
+ 'resnet18': 'https://download.pytorch.org/models/resnet18-5c106cde.pth',
+ 'resnet34': 'https://download.pytorch.org/models/resnet34-333f7ec4.pth',
+ 'resnet50': 'https://download.pytorch.org/models/resnet50-19c8e357.pth',
+ 'resnet101': 'https://download.pytorch.org/models/resnet101-5d3b4d8f.pth',
+ 'resnet152': 'https://download.pytorch.org/models/resnet152-b121ed2d.pth',
+}
+
+
+class PyramidFeatures(nn.Module):
+ def __init__(self, C3_size, C4_size, C5_size, feature_size=256):
+ super(PyramidFeatures, self).__init__()
+
+ # upsample C5 to get P5 from the FPN paper
+ self.P5_1 = nn.Conv2d(C5_size, feature_size, kernel_size=1, stride=1, padding=0)
+ self.P5_upsampled = nn.Upsample(scale_factor=2, mode='nearest')
+ self.P5_2 = nn.Conv2d(feature_size, feature_size, kernel_size=3, stride=1, padding=1)
+
+ # add P5 elementwise to C4
+ self.P4_1 = nn.Conv2d(C4_size, feature_size, kernel_size=1, stride=1, padding=0)
+ self.P4_upsampled = nn.Upsample(scale_factor=2, mode='nearest')
+ self.P4_2 = nn.Conv2d(feature_size, feature_size, kernel_size=3, stride=1, padding=1)
+
+ # add P4 elementwise to C3
+ self.P3_1 = nn.Conv2d(C3_size, feature_size, kernel_size=1, stride=1, padding=0)
+ self.P3_2 = nn.Conv2d(feature_size, feature_size, kernel_size=3, stride=1, padding=1)
+
+ # "P6 is obtained via a 3x3 stride-2 conv on C5"
+ self.P6 = nn.Conv2d(C5_size, feature_size, kernel_size=3, stride=2, padding=1)
+
+ # "P7 is computed by applying ReLU followed by a 3x3 stride-2 conv on P6"
+ self.P7_1 = nn.ReLU()
+ self.P7_2 = nn.Conv2d(feature_size, feature_size, kernel_size=3, stride=2, padding=1)
+
+ def forward(self, inputs):
+ C3, C4, C5 = inputs
+
+ P5_x = self.P5_1(C5)
+ P5_upsampled_x = self.P5_upsampled(P5_x)
+ P5_x = self.P5_2(P5_x)
+
+ P4_x = self.P4_1(C4)
+ P4_x = P5_upsampled_x + P4_x
+ P4_upsampled_x = self.P4_upsampled(P4_x)
+ P4_x = self.P4_2(P4_x)
+
+ P3_x = self.P3_1(C3)
+ P3_x = P3_x + P4_upsampled_x
+ P3_x = self.P3_2(P3_x)
+
+ P6_x = self.P6(C5)
+
+ P7_x = self.P7_1(P6_x)
+ P7_x = self.P7_2(P7_x)
+
+ return [P3_x, P4_x, P5_x, P6_x, P7_x]
+
+
+class RegressionModel(nn.Module):
+ def __init__(self, num_features_in, num_anchors=9, feature_size=256):
+ super(RegressionModel, self).__init__()
+
+ self.conv1 = nn.Conv2d(num_features_in, feature_size, kernel_size=3, padding=1)
+ self.act1 = nn.ReLU()
+
+ self.conv2 = nn.Conv2d(feature_size, feature_size, kernel_size=3, padding=1)
+ self.act2 = nn.ReLU()
+
+ self.conv3 = nn.Conv2d(feature_size, feature_size, kernel_size=3, padding=1)
+ self.act3 = nn.ReLU()
+
+ self.conv4 = nn.Conv2d(feature_size, feature_size, kernel_size=3, padding=1)
+ self.act4 = nn.ReLU()
+
+ self.output = nn.Conv2d(feature_size, num_anchors * 4, kernel_size=3, padding=1)
+
+ def forward(self, x):
+ out = self.conv1(x)
+ out = self.act1(out)
+
+ out = self.conv2(out)
+ out = self.act2(out)
+
+ out = self.conv3(out)
+ out = self.act3(out)
+
+ out = self.conv4(out)
+ out = self.act4(out)
+
+ out = self.output(out)
+
+ # out is B x C x W x H, with C = 4*num_anchors
+ out = out.permute(0, 2, 3, 1)
+
+ return out.contiguous().view(out.shape[0], -1, 4)
+
+
+class ClassificationModel(nn.Module):
+ def __init__(self, num_features_in, num_anchors=9, num_classes=80, prior=0.01, feature_size=256):
+ super(ClassificationModel, self).__init__()
+
+ self.num_classes = num_classes
+ self.num_anchors = num_anchors
+
+ self.conv1 = nn.Conv2d(num_features_in, feature_size, kernel_size=3, padding=1)
+ self.act1 = nn.ReLU()
+
+ self.conv2 = nn.Conv2d(feature_size, feature_size, kernel_size=3, padding=1)
+ self.act2 = nn.ReLU()
+
+ self.conv3 = nn.Conv2d(feature_size, feature_size, kernel_size=3, padding=1)
+ self.act3 = nn.ReLU()
+
+ self.conv4 = nn.Conv2d(feature_size, feature_size, kernel_size=3, padding=1)
+ self.act4 = nn.ReLU()
+
+ self.output = nn.Conv2d(feature_size, num_anchors * num_classes, kernel_size=3, padding=1)
+ self.output_act = nn.Sigmoid()
+
+ def forward(self, x):
+ out = self.conv1(x)
+ out = self.act1(out)
+
+ out = self.conv2(out)
+ out = self.act2(out)
+
+ out = self.conv3(out)
+ out = self.act3(out)
+
+ out = self.conv4(out)
+ out = self.act4(out)
+
+ out = self.output(out)
+ out = self.output_act(out)
+
+ # out is B x C x W x H, with C = n_classes + n_anchors
+ out1 = out.permute(0, 2, 3, 1)
+
+ batch_size, width, height, channels = out1.shape
+
+ out2 = out1.view(batch_size, width, height, self.num_anchors, self.num_classes)
+
+ return out2.contiguous().view(x.shape[0], -1, self.num_classes)
+
+
+class ResNet(nn.Module):
+
+ def __init__(self, num_classes, block, layers):
+ self.inplanes = 64
+ super(ResNet, self).__init__()
+ self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False)
+ self.bn1 = nn.BatchNorm2d(64)
+ self.relu = nn.ReLU(inplace=True)
+ self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
+ self.layer1 = self._make_layer(block, 64, layers[0])
+ self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
+ self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
+ self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
+
+ if block == BasicBlock:
+ fpn_sizes = [self.layer2[layers[1] - 1].conv2.out_channels, self.layer3[layers[2] - 1].conv2.out_channels,
+ self.layer4[layers[3] - 1].conv2.out_channels]
+ elif block == Bottleneck:
+ fpn_sizes = [self.layer2[layers[1] - 1].conv3.out_channels, self.layer3[layers[2] - 1].conv3.out_channels,
+ self.layer4[layers[3] - 1].conv3.out_channels]
+ else:
+ raise ValueError(f"Block type {block} not understood")
+
+ self.fpn = PyramidFeatures(fpn_sizes[0], fpn_sizes[1], fpn_sizes[2])
+
+ self.regressionModel = RegressionModel(256)
+ self.classificationModel = ClassificationModel(256, num_classes=num_classes)
+
+ self.anchors = Anchors()
+
+ self.regressBoxes = BBoxTransform()
+
+ self.clipBoxes = ClipBoxes()
+
+ self.focalLoss = losses.FocalLoss()
+
+ for m in self.modules():
+ if isinstance(m, nn.Conv2d):
+ n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
+ m.weight.data.normal_(0, math.sqrt(2. / n))
+ elif isinstance(m, nn.BatchNorm2d):
+ m.weight.data.fill_(1)
+ m.bias.data.zero_()
+
+ prior = 0.01
+
+ self.classificationModel.output.weight.data.fill_(0)
+ self.classificationModel.output.bias.data.fill_(-math.log((1.0 - prior) / prior))
+
+ self.regressionModel.output.weight.data.fill_(0)
+ self.regressionModel.output.bias.data.fill_(0)
+
+ self.freeze_bn()
+
+ def _make_layer(self, block, planes, blocks, stride=1):
+ downsample = None
+ if stride != 1 or self.inplanes != planes * block.expansion:
+ downsample = nn.Sequential(
+ nn.Conv2d(self.inplanes, planes * block.expansion,
+ kernel_size=1, stride=stride, bias=False),
+ nn.BatchNorm2d(planes * block.expansion),
+ )
+
+ layers = [block(self.inplanes, planes, stride, downsample)]
+ self.inplanes = planes * block.expansion
+ for i in range(1, blocks):
+ layers.append(block(self.inplanes, planes))
+
+ return nn.Sequential(*layers)
+
+ def freeze_bn(self):
+ '''Freeze BatchNorm layers.'''
+ for layer in self.modules():
+ if isinstance(layer, nn.BatchNorm2d):
+ layer.eval()
+
+ def forward(self, *inputs, **kw):
+ if 'device' in kw:
+ device = kw['device']
+ else:
+ device = None
+ if 'gamma' in kw:
+ gamma = kw['gamma']
+ if 'alpha' in kw:
+ alpha = kw['alpha']
+
+ img_batch = inputs[0]#if testing then we have no annotations
+ x = self.conv1(img_batch)
+ x = self.bn1(x)
+ x = self.relu(x)
+ x = self.maxpool(x)
+
+ x1 = self.layer1(x)
+ x2 = self.layer2(x1)
+ x3 = self.layer3(x2)
+ x4 = self.layer4(x3)
+
+ features = self.fpn([x2, x3, x4])
+
+ regression = torch.cat([self.regressionModel(feature) for feature in features], dim=1)
+
+ classification = torch.cat([self.classificationModel(feature) for feature in features], dim=1)
+ #classification is (batch, NUM_FEATURES, NUM_CLASSES)
+ anchors = self.anchors(img_batch)
+ if self.training:
+ return classification, regression, anchors
+ #return self.focalLoss(classification, regression, anchors, annotations)#, alpha, gamma)
+ else:
+ transformed_anchors = self.regressBoxes(anchors, regression)
+ transformed_anchors = self.clipBoxes(transformed_anchors, img_batch)
+ BATCH = classification.shape[0]
+ NUM_CLASSES = classification.shape[2]
+ allScores = []
+ allClasses = []
+ allAnchors = []
+ for batch_ind in range(BATCH):
+ finalResult = [[], [], []]
+
+ finalScores = torch.Tensor([])
+ finalAnchorBoxesIndexes = torch.Tensor([]).long()
+ finalAnchorBoxesCoordinates = torch.Tensor([])
+ if device is not None:
+ finalScores = finalScores.to(device)
+ finalAnchorBoxesIndexes = finalAnchorBoxesIndexes.to(device)
+ finalAnchorBoxesCoordinates = finalAnchorBoxesCoordinates.to(device)
+ elif torch.cuda.is_available():
+ finalScores = finalScores.cuda()
+ finalAnchorBoxesIndexes = finalAnchorBoxesIndexes.cuda()
+ finalAnchorBoxesCoordinates = finalAnchorBoxesCoordinates.cuda()
+
+ for i in range(NUM_CLASSES):#for every class
+ #print('classif',classification.shape)
+ scores = torch.squeeze(classification[batch_ind, :, i])#classification[:, :, i]
+ scores_over_thresh = (scores > 0.05)
+ if scores_over_thresh.sum() == 0:
+ # no boxes to NMS, just continue
+ continue
+
+ scores = scores[scores_over_thresh]
+ #print('tbox',transformed_anchors.shape)
+ anchorBoxes = torch.squeeze(transformed_anchors[batch_ind,:,:])
+ #print('abox',anchorBoxes.shape)
+ anchorBoxes = anchorBoxes[scores_over_thresh]
+ #print('abox after',anchorBoxes.shape)
+ anchors_nms_idx = nms(anchorBoxes, scores, 0.5)#check if confidence over 0.5
+ finalResult[0].extend(scores[anchors_nms_idx])
+ finalResult[1].extend(torch.tensor([i] * anchors_nms_idx.shape[0]))
+ finalResult[2].extend(anchorBoxes[anchors_nms_idx])
+
+ finalScores = torch.cat((finalScores, scores[anchors_nms_idx]))
+ finalAnchorBoxesIndexesValue = torch.tensor([i] * anchors_nms_idx.shape[0])
+ if torch.cuda.is_available():
+ finalAnchorBoxesIndexesValue = finalAnchorBoxesIndexesValue.cuda()
+
+ finalAnchorBoxesIndexes = torch.cat((finalAnchorBoxesIndexes, finalAnchorBoxesIndexesValue))
+ finalAnchorBoxesCoordinates = torch.cat((finalAnchorBoxesCoordinates, anchorBoxes[anchors_nms_idx]))
+
+ #results += [finalScores, finalAnchorBoxesIndexes, finalAnchorBoxesCoordinates]
+ allScores.append(finalScores)
+ allClasses.append(finalAnchorBoxesIndexes)
+ allAnchors.append(finalAnchorBoxesCoordinates)
+ return allScores, allClasses , allAnchors
+
+
+def resnet18(num_classes, pretrained=False, **kwargs):
+ """Constructs a ResNet-18 model.
+ Args:
+ pretrained (bool): If True, returns a model pre-trained on ImageNet
+ """
+ model = ResNet(num_classes, BasicBlock, [2, 2, 2, 2], **kwargs)
+ if pretrained:
+ model.load_state_dict(model_zoo.load_url(model_urls['resnet18'], model_dir='./Resnet'), strict=False)
+ return model
+
+
+def resnet34(num_classes, pretrained=False, **kwargs):
+ """Constructs a ResNet-34 model.
+ Args:
+ pretrained (bool): If True, returns a model pre-trained on ImageNet
+ """
+ model = ResNet(num_classes, BasicBlock, [3, 4, 6, 3], **kwargs)
+ if pretrained:
+ model.load_state_dict(model_zoo.load_url(model_urls['resnet34'], model_dir='./ResNet'), strict=False)
+ return model
+
+
+def resnet50(num_classes, pretrained=False, **kwargs):
+ """Constructs a ResNet-50 model.
+ Args:
+ pretrained (bool): If True, returns a model pre-trained on ImageNet
+ """
+ model = ResNet(num_classes, Bottleneck, [3, 4, 6, 3], **kwargs)
+ if pretrained:
+ model.load_state_dict(model_zoo.load_url(model_urls['resnet50'], model_dir='./ResNet'), strict=False)
+ return model
+
+
+def resnet101(num_classes, pretrained=False, **kwargs):
+ """Constructs a ResNet-101 model.
+ Args:
+ pretrained (bool): If True, returns a model pre-trained on ImageNet
+ """
+ model = ResNet(num_classes, Bottleneck, [3, 4, 23, 3], **kwargs)
+ if pretrained:
+ model.load_state_dict(model_zoo.load_url(model_urls['resnet101'], model_dir='./ResNet'), strict=False)
+ return model
+
+
+def resnet152(num_classes, pretrained=False, **kwargs):
+ """Constructs a ResNet-152 model.
+ Args:
+ pretrained (bool): If True, returns a model pre-trained on ImageNet
+ """
+ model = ResNet(num_classes, Bottleneck, [3, 8, 36, 3], **kwargs)
+ if pretrained:
+ model.load_state_dict(model_zoo.load_url(model_urls['resnet152'], model_dir='./ResNet'), strict=False)
+ return model
diff --git a/examples/al_object_detection/cycle.yaml b/examples/al_object_detection/cycle.yaml
new file mode 100644
index 00000000..03a7f564
--- /dev/null
+++ b/examples/al_object_detection/cycle.yaml
@@ -0,0 +1,76 @@
+vars:
+ - configs/MIAOD-GRAY.py:data_root
+ - al_cycle.json:al_cycle,next_cycle,al_seed
+
+stages:
+ input_sample_lists:
+ cmd:
+ python input_lists.py configs/MIAOD-GRAY.py --work_directory work_dirs/test --labeled ${data_root}/training.txt --unlabeled ${data_root}/unlabeled.txt --seed ${al_seed} --train ${data_root}/train.txt --selected work_dirs/test/X_L_${al_cycle}.npy --unselected work_dirs/test/X_U_${al_cycle}.npy
+ deps:
+ - ${data_root}/training.txt
+ - ${data_root}/unlabeled.txt
+ outs:
+ - ${data_root}/train.txt
+ - work_dirs/test/X_L_${al_cycle}.npy
+ - work_dirs/test/X_U_${al_cycle}.npy
+ params:
+ - configs/MIAOD-GRAY.py:
+ - data_root
+ - al_cycle.json:
+ - al_cycle
+ - al_seed
+
+ active_learning_training:
+ cmd:
+ #single gpu
+ python cycle_train.py configs/MIAOD-GRAY.py --work_directory work_dirs/test --cycle ${al_cycle} --labeled work_dirs/test/X_L_${al_cycle}.npy --unselected work_dirs/test/X_U_${al_cycle}.npy --seed ${al_seed} --model work_dirs/test/cycle.pth
+ #distributed mode
+ #python -m torch.distributed.launch --nproc_per_node=6 --master_port=39025 cycle_train.py configs/MIAOD-GRAY.py --work_directory work_dirs/test --cycle ${al_cycle} --labeled work_dirs/test/X_L_${al_cycle}.npy --unselected work_dirs/test/X_U_${al_cycle}.npy --seed ${al_seed} --model work_dirs/test/cycle.pth --gpus 6 --launcher pytorch
+ deps:
+ - work_dirs/test/X_L_${al_cycle}.npy
+ - work_dirs/test/X_U_${al_cycle}.npy
+ outs:
+ - work_dirs/test/cycle.pth
+ params:
+ - al_cycle.json:
+ - al_cycle
+ - al_seed
+
+ active_learning_selection:
+ cmd:
+ #single gpu
+ python cycle_select.py configs/MIAOD-GRAY.py --work_directory work_dirs/test --cycle ${al_cycle} --model work_dirs/test/cycle.pth --labeled work_dirs/test/X_L_${al_cycle}.npy --seed ${al_seed} --labeled_next work_dirs/test/X_L_${next_cycle}.npy --unselected work_dirs/test/X_U_${next_cycle}.npy --bbox_output work_dirs/test/labeling_hints_${next_cycle}.txt
+ #distributed mode
+ #python -m torch.distributed.launch --nproc_per_node=4 --master_port=39016 cycle_select.py configs/MIAOD-GRAY.py --work_directory work_dirs/test --cycle ${al_cycle} --model work_dirs/test/cycle.pth --labeled work_dirs/test/X_L_${al_cycle}.npy --seed ${al_seed} --labeled_next work_dirs/test/X_L_${next_cycle}.npy --unselected work_dirs/test/X_U_${next_cycle}.npy --bbox_output work_dirs/test/labeling_hints_${next_cycle}.txt --gpus 4 --launcher pytorch
+ deps:
+ - work_dirs/test/cycle.pth
+ - work_dirs/test/X_L_${al_cycle}.npy
+ outs:
+ - work_dirs/test/X_L_${next_cycle}.npy
+ - work_dirs/test/X_U_${next_cycle}.npy
+ params:
+ - al_cycle.json:
+ - al_cycle
+ - next_cycle
+ - al_seed
+
+ output_sample_lists:
+ cmd:
+ python output_lists.py configs/MIAOD-GRAY.py --work_directory work_dirs/test --selected work_dirs/test/X_L_${al_cycle}.npy --selected_next work_dirs/test/X_L_${next_cycle}.npy --train ${data_root}/train.txt --map ${data_root}/map.txt --seed ${al_seed} --label_next ${data_root}/label_next.txt --labeled ${data_root}/training_next.txt --unlabeled ${data_root}/unlabeled_next.txt --cycle_config al_cycle.json --cycle_config_next al_cycle_next.json
+ deps:
+ - work_dirs/test/X_L_${al_cycle}.npy
+ - work_dirs/test/X_L_${next_cycle}.npy
+ - ${data_root}/train.txt
+ - ${data_root}/map.txt
+ outs:
+ - ${data_root}/label_next.txt
+ - ${data_root}/training_next.txt
+ - ${data_root}/unlabeled_next.txt
+ - al_cycle_next.json
+ params:
+ - configs/MIAOD-GRAY.py:
+ - data_root
+ - al_cycle.json:
+ - al_cycle
+ - next_cycle
+ - al_seed
diff --git a/examples/al_object_detection/cycle_select.py b/examples/al_object_detection/cycle_select.py
index 1c758f2f..a962a85c 100644
--- a/examples/al_object_detection/cycle_select.py
+++ b/examples/al_object_detection/cycle_select.py
@@ -2,6 +2,7 @@
import os
import os.path as osp
import time
+import json
import mmcv
import torch
@@ -20,6 +21,7 @@
from mmdet.utils import collect_env, get_root_logger
from mmdet.utils.active_datasets import *
+import cv2
def parse_args():
parser = argparse.ArgumentParser(description='Select informative images')
@@ -30,6 +32,7 @@ def parse_args():
parser.add_argument('--labeled_next',
help='next cycle labeled samples list file')
parser.add_argument('--unselected', help='unselected samples list file')
+ parser.add_argument('--bbox_output', help='labeling guidance file')
parser.add_argument('--work_directory',
help='the dir to save logs and model checkpoints')
parser.add_argument(
@@ -51,6 +54,11 @@ def parse_args():
choices=['none', 'pytorch', 'slurm', 'mpi'],
default='none', help='job launcher')
parser.add_argument('--local_rank', type=int, default=0)
+
+ parser.add_argument('--strategy', choices=['active_learning', 'random'],
+ default='active_learning')
+ parser.add_argument('--stage_name', help='Name for current execution')
+ parser.add_argument('--execution_name', help='Name for current execution')
args = parser.parse_args()
if 'LOCAL_RANK' not in os.environ:
os.environ['LOCAL_RANK'] = str(args.local_rank)
@@ -71,6 +79,13 @@ def main():
if args.options is not None:
cfg.merge_from_dict(args.options)
+ # if cfg.plot_nuboxes is set, check that bbox_output file name is given
+ assert (not cfg.get('plot_nuboxes') or args.bbox_output), \
+ ('When output of labeling guides is turned on in the config file '
+ '(cfg.plot_nuboxes > 0), command line argument --bbox_output + Create a file inside the folder mmdet/datasets.
+ we can take voc.py in the same folder as a template and make necessary modifications.
+ Newly added file hdc.py was created for the custom dataset.
+ Please modify the class names in this file appropriately.
+ + 3. Prepare config file for the new dataset in the folder config/_base_
+ hdc.py was created for the custom dataset.
+ Please modify it appropriately.
+ + 4. Config changes + 1. Dataset path
+ - Modify *configs/MIAOD-GRAY* to reflect the dataset path
+ sections - data_root, data
+ - Modify *configs/_base_/hdc.py* to reflect the dataset path
+ sections - data_root, data
+ + 2. Number of classes
+ Modify the section model in MIAOD-GRAY.
+ C=2, for two class dataset.
+ + 3. Number of images to train.
+ Modify the section X_S_size AND X_L_0_size in *config/MIAOD-GRAY*
+ + # MI-AOD Language: [简体中文](README_cn.md) | English diff --git a/examples/al_object_detection/mmdet.egg-info/SOURCES.txt b/examples/al_object_detection/mmdet.egg-info/SOURCES.txt index fdc59982..d715ea51 100644 --- a/examples/al_object_detection/mmdet.egg-info/SOURCES.txt +++ b/examples/al_object_detection/mmdet.egg-info/SOURCES.txt @@ -78,6 +78,8 @@ mmdet/datasets/coco.py mmdet/datasets/custom.py mmdet/datasets/dataset_wrappers.py mmdet/datasets/deepfashion.py +mmdet/datasets/gray.py +mmdet/datasets/hdc.py mmdet/datasets/lvis.py mmdet/datasets/voc.py mmdet/datasets/wider_face.py @@ -213,4 +215,13 @@ mmdet/utils/collect_env.py mmdet/utils/contextmanagers.py mmdet/utils/logger.py mmdet/utils/profiling.py -mmdet/utils/util_mixins.py \ No newline at end of file +mmdet/utils/util_mixins.py +mmdet/utils/hdc/__init__.py +mmdet/utils/hdc/active_datasets.py +tests/test_anchor.py +tests/test_assigner.py +tests/test_async.py +tests/test_config.py +tests/test_fp16.py +tests/test_masks.py +tests/test_version.py \ No newline at end of file diff --git a/examples/al_object_detection/mmdet/apis/test.py b/examples/al_object_detection/mmdet/apis/test.py index b710c5de..a9307e5f 100644 --- a/examples/al_object_detection/mmdet/apis/test.py +++ b/examples/al_object_detection/mmdet/apis/test.py @@ -10,19 +10,30 @@ from mmcv.runner import get_dist_info from mmdet.core import encode_mask_results, tensor2imgs from mmdet.models.detectors.base import * +from mmcv.ops.nms import batched_nms -def calculate_uncertainty(cfg, model, data_loader, return_box=False): +def calculate_uncertainty(cfg, model, data_loader, plot_nuboxes=0, return_box=False): model.eval() model.cuda() dataset = data_loader.dataset print('>>> Computing Instance Uncertainty...') uncertainty = torch.zeros(len(dataset)).cuda(torch.cuda.current_device()) + if plot_nuboxes > 0: + udets = torch.zeros(len(dataset), plot_nuboxes, 5).cuda(torch.cuda.current_device()) + if cfg.get('check_nuboxes') and (cfg.check_nuboxes > 0): + check_nuboxes = cfg.check_nuboxes + else: + check_nuboxes = cfg.k + zero_offsets = torch.zeros(check_nuboxes).cuda(torch.cuda.current_device()) for i, data in enumerate(data_loader): with torch.no_grad(): data['img'][0] = data['img'][0].cuda() data.update({'x': data.pop('img')}) - y_head_f_1, y_head_f_2, y_head_cls = model(return_loss=False, rescale=True, return_box=return_box, **data) + if plot_nuboxes > 0: + y_head_f_1, y_head_f_2, y_f_r, y_head_cls = model(return_loss=False, rescale=True, return_box=return_box, uncertain_box=True, **data) + else: + y_head_f_1, y_head_f_2, y_head_cls = model(return_loss=False, rescale=True, return_box=return_box, uncertain_box=False, **data) y_head_f_1 = torch.cat(y_head_f_1, 0) y_head_f_2 = torch.cat(y_head_f_2, 0) y_head_f_1 = nn.Sigmoid()(y_head_f_1) @@ -30,11 +41,23 @@ def calculate_uncertainty(cfg, model, data_loader, return_box=False): loss_l2_p = (y_head_f_1 - y_head_f_2).pow(2) uncertainty_all_N = loss_l2_p.mean(dim=1) arg = uncertainty_all_N.argsort() - uncertainty_single = uncertainty_all_N[arg[-cfg.k:]].mean() - uncertainty[i] = uncertainty_single + if plot_nuboxes > 0: + img_shape = data['img_metas'][0].data[0][0]['img_shape'] + scale_factor = data['img_metas'][0].data[0][0]['scale_factor'] + ubboxes_single = model.bbox_head.get_uncertain_bboxes(arg, + y_f_r, img_shape, scale_factor, check_nuboxes, cfg, + rescale=True) + uncertainty_single = uncertainty_all_N[arg[-check_nuboxes:]] + udets_single, _ = batched_nms(ubboxes_single, + uncertainty_single, zero_offsets, cfg.test_cfg.nms) + udets[i] = udets_single[:plot_nuboxes] + uncertainty[i] = uncertainty_all_N[arg[-cfg.k:]].mean() if i % 1000 == 0: print('>>> ', i, '/', len(dataset)) - return uncertainty.cpu() + if plot_nuboxes > 0: + return uncertainty.cpu(), udets.cpu() + else: + return uncertainty.cpu() def single_gpu_test(model, data_loader, show=False): diff --git a/examples/al_object_detection/mmdet/core/evaluation/bbox_overlaps.py b/examples/al_object_detection/mmdet/core/evaluation/bbox_overlaps.py index 93559ea0..cbacc61c 100644 --- a/examples/al_object_detection/mmdet/core/evaluation/bbox_overlaps.py +++ b/examples/al_object_detection/mmdet/core/evaluation/bbox_overlaps.py @@ -1,20 +1,20 @@ import numpy as np -def bbox_overlaps(bboxes1, bboxes2, mode='iou', eps=1e-6): +def bbox_overlaps(bboxes1, bboxes2, mode='iog', eps=1e-6): """Calculate the ious between each bbox of bboxes1 and bboxes2. Args: bboxes1(ndarray): shape (n, 4) bboxes2(ndarray): shape (k, 4) mode(str): iou (intersection over union) or iof (intersection - over foreground) + over foreground) or iog (intersection over ground truth) Returns: ious(ndarray): shape (n, k) """ - assert mode in ['iou', 'iof'] + assert mode in ['iou', 'iof', 'iog'] bboxes1 = bboxes1.astype(np.float32) bboxes2 = bboxes2.astype(np.float32) @@ -39,8 +39,10 @@ def bbox_overlaps(bboxes1, bboxes2, mode='iou', eps=1e-6): y_end - y_start, 0) if mode == 'iou': union = area1[i] + area2 - overlap - else: + elif mode == 'iof': union = area1[i] if not exchange else area2 + else: + union = area1[i] if exchange else area2 union = np.maximum(union, eps) ious[i, :] = overlap / union if exchange: diff --git a/examples/al_object_detection/mmdet/core/evaluation/mean_ap.py b/examples/al_object_detection/mmdet/core/evaluation/mean_ap.py index 9611c814..d45870f5 100644 --- a/examples/al_object_detection/mmdet/core/evaluation/mean_ap.py +++ b/examples/al_object_detection/mmdet/core/evaluation/mean_ap.py @@ -80,8 +80,8 @@ def tpfp_imagenet(det_bboxes, """ # an indicator of ignored gts gt_ignore_inds = np.concatenate( - (np.zeros(gt_bboxes.shape[0], dtype=np.bool), - np.ones(gt_bboxes_ignore.shape[0], dtype=np.bool))) + (np.zeros(gt_bboxes.shape[0], dtype=bool), + np.ones(gt_bboxes_ignore.shape[0], dtype=bool))) # stack gt_bboxes and gt_bboxes_ignore for convenience gt_bboxes = np.vstack((gt_bboxes, gt_bboxes_ignore)) @@ -173,8 +173,8 @@ def tpfp_default(det_bboxes, """ # an indicator of ignored gts gt_ignore_inds = np.concatenate( - (np.zeros(gt_bboxes.shape[0], dtype=np.bool), - np.ones(gt_bboxes_ignore.shape[0], dtype=np.bool))) + (np.zeros(gt_bboxes.shape[0], dtype=bool), + np.ones(gt_bboxes_ignore.shape[0], dtype=bool))) # stack gt_bboxes and gt_bboxes_ignore for convenience gt_bboxes = np.vstack((gt_bboxes, gt_bboxes_ignore)) @@ -248,7 +248,6 @@ def get_cls_results(det_results, annotations, class_id): Returns: tuple[list[np.ndarray]]: detected bboxes, gt bboxes, ignored gt bboxes """ - cls_dets = [img_res[class_id] for img_res in det_results] cls_gts = [] cls_gts_ignore = [] @@ -261,7 +260,6 @@ def get_cls_results(det_results, annotations, class_id): cls_gts_ignore.append(ann['bboxes_ignore'][ignore_inds, :]) else: cls_gts_ignore.append(np.empty((0, 4), dtype=np.float32)) - return cls_dets, cls_gts, cls_gts_ignore @@ -316,6 +314,7 @@ def eval_map(det_results, # get gt and det bboxes of this class cls_dets, cls_gts, cls_gts_ignore = get_cls_results( det_results, annotations, i) + # choose proper function according to datasets to compute tp and fp if dataset in ['det', 'vid']: tpfp_func = tpfp_imagenet @@ -341,7 +340,9 @@ def eval_map(det_results, num_gts[k] += np.sum((gt_areas >= min_area) & (gt_areas < max_area)) # sort all det bboxes by score, also sort tp and fp + cls_dets = np.vstack(cls_dets) + num_dets = cls_dets.shape[0] sort_inds = np.argsort(-cls_dets[:, -1]) tp = np.hstack(tp)[:, sort_inds] @@ -357,7 +358,7 @@ def eval_map(det_results, recalls = recalls[0, :] precisions = precisions[0, :] num_gts = num_gts.item() - mode = 'area' if dataset != 'voc07' or 'hdc' else '11points' + mode = 'area' if dataset != 'voc07' or 'HDCDataset' or 'hdc' else '11points' ap = average_precision(recalls, precisions, mode) eval_results.append({ 'num_gts': num_gts, @@ -420,6 +421,7 @@ def print_map_summary(mean_ap, if scale_ranges is not None: assert len(scale_ranges) == num_scales + num_classes = len(results) recalls = np.zeros((num_scales, num_classes), dtype=np.float32) aps = np.zeros((num_scales, num_classes), dtype=np.float32) diff --git a/examples/al_object_detection/mmdet/core/mask/structures.py b/examples/al_object_detection/mmdet/core/mask/structures.py index 8bbbfe47..093fc4b4 100644 --- a/examples/al_object_detection/mmdet/core/mask/structures.py +++ b/examples/al_object_detection/mmdet/core/mask/structures.py @@ -571,5 +571,5 @@ def polygon_to_bitmap(polygons, height, width): """ rles = maskUtils.frPyObjects(polygons, height, width) rle = maskUtils.merge(rles) - bitmap_mask = maskUtils.decode(rle).astype(np.bool) + bitmap_mask = maskUtils.decode(rle).astype(bool) return bitmap_mask diff --git a/examples/al_object_detection/mmdet/datasets/hdc.py b/examples/al_object_detection/mmdet/datasets/hdc.py index ebf2a3e9..233407a0 100644 --- a/examples/al_object_detection/mmdet/datasets/hdc.py +++ b/examples/al_object_detection/mmdet/datasets/hdc.py @@ -6,7 +6,7 @@ @DATASETS.register_module() class HDCDataset(XMLDataset): - CLASSES = ['Defect', 'Background'] + CLASSES = ['Defect'] def __init__(self, **kwargs): #print("came in VOCDataset") diff --git a/examples/al_object_detection/mmdet/datasets/xml_style.py b/examples/al_object_detection/mmdet/datasets/xml_style.py index e5b7cae6..c92c64ea 100644 --- a/examples/al_object_detection/mmdet/datasets/xml_style.py +++ b/examples/al_object_detection/mmdet/datasets/xml_style.py @@ -8,6 +8,7 @@ from .builder import DATASETS from .custom import CustomDataset +ANNOTATIONS_FOLDER = 'Annotations_no_dummy' @DATASETS.register_module() class XMLDataset(CustomDataset): @@ -38,11 +39,13 @@ def load_annotations(self, ann_file): img_ids = mmcv.list_from_file(ann_file) for img_id in img_ids: filename = f'JPEGImages/{img_id}.jpg' - xml_path = osp.join(self.img_prefix, 'Annotations', + xml_path = osp.join(self.img_prefix, ANNOTATIONS_FOLDER, f'{img_id}.xml') - tree = ET.parse(xml_path) - root = tree.getroot() - size = root.find('size') + size = None + if osp.exists(xml_path): + tree = ET.parse(xml_path) + root = tree.getroot() + size = root.find('size') width = 0 height = 0 if size is not None: @@ -63,15 +66,16 @@ def get_subset_by_classes(self): subset_data_infos = [] for data_info in self.data_infos: img_id = data_info['id'] - xml_path = osp.join(self.img_prefix, 'Annotations', + xml_path = osp.join(self.img_prefix, ANNOTATIONS_FOLDER, f'{img_id}.xml') - tree = ET.parse(xml_path) - root = tree.getroot() - for obj in root.findall('object'): - name = obj.find('name').text - if name in self.CLASSES: - subset_data_infos.append(data_info) - break + if osp.exists(xml_path): + tree = ET.parse(xml_path) + root = tree.getroot() + for obj in root.findall('object'): + name = obj.find('name').text + if name in self.CLASSES: + subset_data_infos.append(data_info) + break return subset_data_infos @@ -86,50 +90,55 @@ def get_ann_info(self, idx): """ img_id = self.data_infos[idx]['id'] - xml_path = osp.join(self.img_prefix, 'Annotations', f'{img_id}.xml') - tree = ET.parse(xml_path) - root = tree.getroot() + xml_path = osp.join(self.img_prefix, ANNOTATIONS_FOLDER, f'{img_id}.xml') bboxes = [] labels = [] bboxes_ignore = [] labels_ignore = [] - for obj in root.findall('object'): - name = obj.find('name').text - if name not in self.CLASSES: - continue - label = self.cat2label[name] - difficult = int(obj.find('difficult').text) - bnd_box = obj.find('bndbox') - # TODO: check whether it is necessary to use int - # Coordinates may be float type - bbox = [ - int(float(bnd_box.find('xmin').text)), - int(float(bnd_box.find('ymin').text)), - int(float(bnd_box.find('xmax').text)), - int(float(bnd_box.find('ymax').text)) - ] - ignore = False - if self.min_size: - assert not self.test_mode - w = bbox[2] - bbox[0] - h = bbox[3] - bbox[1] - if w < self.min_size or h < self.min_size: - ignore = True - if difficult or ignore: - bboxes_ignore.append(bbox) - labels_ignore.append(label) - else: - bboxes.append(bbox) - labels.append(label) + if osp.exists(xml_path): + tree = ET.parse(xml_path) + root = tree.getroot() + for obj in root.findall('object'): + name = obj.find('name').text + if name not in self.CLASSES: + continue + label = self.cat2label[name] + difficult = int(obj.find('difficult').text) + bnd_box = obj.find('bndbox') + # TODO: check whether it is necessary to use int + # Coordinates may be float type + bbox = [ + int(float(bnd_box.find('xmin').text)), + int(float(bnd_box.find('ymin').text)), + int(float(bnd_box.find('xmax').text)), + int(float(bnd_box.find('ymax').text)) + ] + ignore = False + if self.min_size: + assert not self.test_mode + w = bbox[2] - bbox[0] + h = bbox[3] - bbox[1] + if w < self.min_size or h < self.min_size: + ignore = True + if difficult or ignore: + bboxes_ignore.append(bbox) + labels_ignore.append(label) + else: + bboxes.append(bbox) + labels.append(label) if not bboxes: - bboxes = np.zeros((0, 4)) - labels = np.zeros((0, )) + #bboxes = np.zeros((0, 4)) + #labels = np.zeros((0, )) + bboxes = np.array([[0, 0, 10, 10]], ndmin=2) -1 + labels = np.zeros((1, )) else: bboxes = np.array(bboxes, ndmin=2) - 1 labels = np.array(labels) if not bboxes_ignore: - bboxes_ignore = np.zeros((0, 4)) - labels_ignore = np.zeros((0, )) + #bboxes_ignore = np.zeros((0, 4)) + #labels_ignore = np.zeros((0, )) + bboxes_ignore = np.array([[0, 0, 10, 10]], ndmin=2) -1 + labels_ignore = np.zeros((1, )) else: bboxes_ignore = np.array(bboxes_ignore, ndmin=2) - 1 labels_ignore = np.array(labels_ignore) @@ -152,14 +161,15 @@ def get_cat_ids(self, idx): cat_ids = [] img_id = self.data_infos[idx]['id'] - xml_path = osp.join(self.img_prefix, 'Annotations', f'{img_id}.xml') - tree = ET.parse(xml_path) - root = tree.getroot() - for obj in root.findall('object'): - name = obj.find('name').text - if name not in self.CLASSES: - continue - label = self.cat2label[name] - cat_ids.append(label) + xml_path = osp.join(self.img_prefix, ANNOTATIONS_FOLDER, f'{img_id}.xml') + if osp.exists(xml_path): + tree = ET.parse(xml_path) + root = tree.getroot() + for obj in root.findall('object'): + name = obj.find('name').text + if name not in self.CLASSES: + continue + label = self.cat2label[name] + cat_ids.append(label) return cat_ids diff --git a/examples/al_object_detection/mmdet/models/dense_heads/MIAOD_head.py b/examples/al_object_detection/mmdet/models/dense_heads/MIAOD_head.py index 7c86dedf..8a494fab 100644 --- a/examples/al_object_detection/mmdet/models/dense_heads/MIAOD_head.py +++ b/examples/al_object_detection/mmdet/models/dense_heads/MIAOD_head.py @@ -369,6 +369,9 @@ def l_det(self, y_head_f_single, y_head_f_r_single, x_i_single, y_cls_single, la x_i_single = x_i_single.reshape(-1, 4) y_head_f_r_single = self.bbox_coder.decode(x_i_single, y_head_f_r_single) l_det_loc = self.SmoothL1(y_head_f_r_single, y_loc_single, bbox_weights, avg_factor=num_total_samples) + + l_det_loc = l_det_loc.clamp(None, 10) + return l_det_cls, l_det_loc # Label Set Training @@ -477,7 +480,7 @@ def L_wave_min(self, y_f, y_f_r, y_head_cls, y_loc_img, y_cls_img, img_metas, y_ l_det_cls2, l_det_loc2 = multi_apply(self.l_det, y_f[1], y_f_r, all_x_i, y_cls, label_weights_list, y_loc, bbox_weights_list, num_total_samples=num_total_samples) - if y_loc_img[0][0][0] < 0: + if img_metas[0]['is_unlabeled'] : l_det_cls = list(map(lambda m, n: (m + n) * 0, l_det_cls1, l_det_cls2)) l_det_loc = list(map(lambda m, n: (m + n) * 0, l_det_loc1, l_det_loc2)) for (i, value) in enumerate(l_det_loc): @@ -562,8 +565,8 @@ def L_wave_max(self, y_f, y_f_r, y_head_cls, y_loc_img, y_cls_img, img_metas, y_ num_total_samples=num_total_samples) l_det_cls2, l_det_loc2 = multi_apply(self.l_det, y_f[1], y_f_r, all_x_i, y_cls, label_weights_list, y_loc, bbox_weights_list, - num_total_samples=num_total_samples) - if y_loc_img[0][0][0] < 0: + num_total_samples=num_total_samples) + if img_metas[0]['is_unlabeled']: l_det_cls = list(map(lambda m, n: (m + n) * 0, l_det_cls1, l_det_cls2)) l_det_loc = list(map(lambda m, n: (m + n) * 0, l_det_loc1, l_det_loc2)) for (i, value) in enumerate(l_det_loc): @@ -712,3 +715,25 @@ def _get_bboxes_single(self, y_head_f_single_list, y_head_f_r_single_list, def loss(self, **kwargs): # This function is to avoid the TypeError caused by the abstract method defined in "base_dense_head.py". return + + @force_fp32(apply_to=('y_f_r', )) + def get_uncertain_bboxes(self, arg, y_f_r, img_shape, scale_factor, nuboxes, cfg=None, rescale=False): + cfg = self.test_cfg if cfg is None else cfg + num_levels = len(y_f_r) + assert y_f_r[0].size(0) == 1 + device = y_f_r[0].device + featmap_sizes = [y_f_r[i].shape[-2:] for i in range(num_levels)] + mlvl_anchors = self.anchor_generator.grid_anchors(featmap_sizes, device= +device) + y_head_f_r = [] + for y_head_f_r_single in y_f_r: + y_head_f_r.append(y_head_f_r_single.permute(0,2,3,1).reshape(-1, 4)) + y_head_f_r = torch.cat(y_head_f_r, 0) + mlvl_anchors = torch.cat(mlvl_anchors, 0) + assert y_head_f_r.size(0) == mlvl_anchors.size(0) + y_head_f_r = y_head_f_r[arg[-nuboxes:]] + mlvl_anchors = mlvl_anchors[arg[-nuboxes:]] + bboxes = self.bbox_coder.decode(mlvl_anchors, y_head_f_r, max_shape=img_shape) + if rescale: + bboxes /= bboxes.new_tensor(scale_factor) + return bboxes diff --git a/examples/al_object_detection/mmdet/models/detectors/single_stage.py b/examples/al_object_detection/mmdet/models/detectors/single_stage.py index b52c2edd..a541c680 100644 --- a/examples/al_object_detection/mmdet/models/detectors/single_stage.py +++ b/examples/al_object_detection/mmdet/models/detectors/single_stage.py @@ -83,7 +83,7 @@ def forward_train(self, x, img_metas, y_loc_img, y_cls_img, y_loc_img_ignore=Non losses = self.bbox_head.forward_train(x, img_metas, y_loc_img, y_cls_img, y_loc_img_ignore) return losses - def simple_test(self, x, img_metas, return_box=True, rescale=False): + def simple_test(self, x, img_metas, return_box=True, rescale=False, uncertain_box=False): """Test function without test time augmentation. Args: @@ -104,7 +104,10 @@ def simple_test(self, x, img_metas, return_box=True, rescale=False): y_head_f_1_1level.append(y_head_f_i_single.permute(0,2,3,1).reshape(-1, self.bbox_head.cls_out_channels)) for y_head_f_i_single in y_head_f_2: y_head_f_2_1level.append(y_head_f_i_single.permute(0,2,3,1).reshape(-1, self.bbox_head.cls_out_channels)) - return y_head_f_1_1level, y_head_f_2_1level, y_head_cls + if uncertain_box: + return y_head_f_1_1level, y_head_f_2_1level, y_head_f_r, y_head_cls + else: + return y_head_f_1_1level, y_head_f_2_1level, y_head_cls outs = (y_head_f_1, y_head_f_r) y_head_loc_cls = self.bbox_head.get_bboxes(*outs, img_metas, rescale=rescale) # skip post-processing when exporting to ONNX diff --git a/examples/al_object_detection/mmdet/utils/active_datasets.py b/examples/al_object_detection/mmdet/utils/active_datasets.py index db5dbf3e..d56587c5 100644 --- a/examples/al_object_detection/mmdet/utils/active_datasets.py +++ b/examples/al_object_detection/mmdet/utils/active_datasets.py @@ -57,7 +57,7 @@ def load_ann_list(paths): return anns -def update_X_L(uncertainty, X_all, X_L, X_S_size): +def update_X_L(uncertainty, X_all, X_L, X_S_size, return_X_S=False): uncertainty = uncertainty.cpu().numpy() all_X_U = np.array(list(set(X_all) - set(X_L))) uncertainty_X_U = uncertainty[all_X_U] @@ -72,4 +72,25 @@ def update_X_L(uncertainty, X_all, X_L, X_S_size): X_U_next = np.concatenate((X_U_next, X_L_next[:X_L_next.shape[0] - X_U_next.shape[0]])) X_L_next.sort() X_U_next.sort() - return X_L_next, X_U_next + if return_X_S: + return X_L_next, X_U_next, X_S + else: + return X_L_next, X_U_next + +def update_X_L_random(X_all, X_L, X_S_size, return_X_S=False): + all_X_U = np.array(list(set(X_all) - set(X_L))) + np.random.shuffle(all_X_U) + X_S = all_X_U[:X_S_size] + X_L_next = np.concatenate((X_L, X_S)) + all_X_U_next = np.array(list(set(X_all) - set(X_L_next))) + np.random.shuffle(all_X_U_next) + X_U_next = all_X_U_next[:X_L_next.shape[0]] + if X_L_next.shape[0] > X_U_next.shape[0]: + np.random.shuffle(X_L_next) + X_U_next = np.concatenate((X_U_next, X_L_next[:X_L_next.shape[0] - X_U_next.shape[0]])) + X_L_next.sort() + X_U_next.sort() + if return_X_S: + return X_L_next, X_U_next, X_S + else: + return X_L_next, X_U_next diff --git a/examples/al_object_detection/mmdet/utils/hdc/active_datasets.py b/examples/al_object_detection/mmdet/utils/hdc/active_datasets.py index baa5d010..4135656f 100644 --- a/examples/al_object_detection/mmdet/utils/hdc/active_datasets.py +++ b/examples/al_object_detection/mmdet/utils/hdc/active_datasets.py @@ -58,7 +58,7 @@ def load_ann_list(paths): return anns -def update_X_L(uncertainty, X_all, X_L, X_S_size): +def update_X_L(uncertainty, X_all, X_L, X_S_size, return_X_S=False): uncertainty = uncertainty.cpu().numpy() all_X_U = np.array(list(set(X_all) - set(X_L))) uncertainty_X_U = uncertainty[all_X_U] @@ -73,4 +73,7 @@ def update_X_L(uncertainty, X_all, X_L, X_S_size): X_U_next = np.concatenate((X_U_next, X_L_next[:X_L_next.shape[0] - X_U_next.shape[0]])) X_L_next.sort() X_U_next.sort() - return X_L_next, X_U_next + if return_X_S: + return X_L_next, X_U_next, X_S + else: + return X_L_next, X_U_next diff --git a/examples/al_object_detection/output_lists.py b/examples/al_object_detection/output_lists.py new file mode 100644 index 00000000..24b24d05 --- /dev/null +++ b/examples/al_object_detection/output_lists.py @@ -0,0 +1,144 @@ +import argparse +import os +import os.path as osp +import json + +from mmcv import Config + +import mmcv +import numpy as np +from mmdet.apis import set_random_seed +from mmdet.utils import get_root_logger + +def parse_args(): + parser = argparse.ArgumentParser( + description='Output list of images to be labeled') + parser.add_argument('config', help='train config file path') + parser.add_argument('--selected', + help='numpy file with indexes of samples labeled before this cycle') + parser.add_argument('--selected_next', + help='numpy file with indexes of samples to be labeled in this cycle') + parser.add_argument('--train', + help='file with list of all training samples') + parser.add_argument('--map', + help='file with mapping between image number names and full names') + parser.add_argument('--label_next', + help='text file with list of samples to be labeled in this cycle') + parser.add_argument('--labeled', + help='text file with list of all samples labeled after this cycle') + parser.add_argument('--unlabeled', + help='text file with list of all samples unlabeled after this cycle') + parser.add_argument('--cycle_config', + help='config file with active learning cycle and seed') + parser.add_argument('--cycle_config_next', + help='config file with next active learning cycle and seed') + parser.add_argument('--work_directory', + help='the dir to save logs and model checkpoints') + parser.add_argument('--seed', type=int, default=666, help='random seed') + parser.add_argument('--deterministic', action='store_true', + help='whether to set deterministic options for CUDNN backend.') + parser.add_argument('--stage_name', help='Name for current execution') + parser.add_argument('--execution_name', help='Name for current execution') + args = parser.parse_args() + return args + +def main(): + args = parse_args() + + assert (args.selected and args.selected_next and args.train and + args.map and args.label_next and args.labeled and + args.unlabeled and args.cycle_config and args.cycle_config_next), \ + ('Plase specify file names of numpy lists of images labeled before ' + 'this cycle and in this cycle with arguments "--selected" and ' + '"--selected_next". Also specify the file name with the list of all ' + 'images that the lists of selected and unselected samples point to, ' + 'with argument "--train". Then specify file name for the map between ' + 'image number names and full image names with argument "--map". ' + 'Finally, specify names for text files output by this code with lists ' + 'of images to be labeled in this cycle, all labeled after this cycle ' + 'and remaining unlabeled, with arguments "--label_next", "--labeled" ' + 'and "--unlabeled". Also specify config file with active learning ' + 'cycle number and random number generator seed with argument ' + '"--cycle_config" and the name of the next cycle config file output ' + 'by this code with argument "--cycle_config_next"') + + cfg = Config.fromfile(args.config) + + # work_directory is determined in this priority: CLI > config > default + if args.work_directory is not None: + # update work_directory from CLI args if args.work_directory is not None + cfg.work_directory = args.work_directory + elif cfg.get('work_directory', None) is None: + # derive work_directory from config name if cfg.work_directory is None + cfg.work_directory = osp.join('./work_dirs', + osp.splitext(osp.basename(args.config))[0]) + + # create work_directory + mmcv.mkdir_or_exist(osp.abspath(cfg.work_directory)) + + stage_name = args.stage_name + os.environ['stage_name'] = stage_name + + execution_name = args.execution_name + os.environ['execution_name'] = execution_name + + # init the logger before other steps + log_file = osp.join(cfg.work_directory, 'input_sample_lists.log') + logger = get_root_logger(log_file=log_file, log_level=cfg.log_level) + + # set random seed + if args.seed is not None: + logger.info( + f'Set random seed to {args.seed}, deterministic: {args.deterministic}') + set_random_seed(args.seed, deterministic=args.deterministic) + cfg.seed = args.seed + + # load the map of image number to image name + map = {} + with open(args.map) as f: + for line in f: + tokens = line.split(",") + assert (len(tokens) == 2), \ + (f'Invalid format of file {args.map} : Expected 2 tokens ' + 'per line, received {len(tokens)} : {line}') + map[int(tokens[0])] = tokens[1].strip() + + # get the lists of images to label, all labeled and unlabeled + labeled_indexes = np.load(args.selected) + labeled_next_indexes = np.load(args.selected_next) + to_label_indexes = np.array(list(set(labeled_next_indexes) - + set(labeled_indexes))) + with open(args.train) as f: + line = f.readline().strip() + num_digits = len(line) + all = np.loadtxt(args.train, dtype=np.uintc) + to_label = all[to_label_indexes] + to_label.sort() + labeled = all[labeled_next_indexes] + labeled.sort() + unlabeled = np.delete(all, labeled_next_indexes) + unlabeled.sort() + + # save the lists after converting from file numbers to file names in + # the list of images to label + with open(args.label_next, 'w') as f: + for image_number in to_label: + image_name = map[image_number] + f.write(image_name + "\n") + np.savetxt(args.labeled, labeled, fmt='%0'+str(num_digits)+'u') + np.savetxt(args.unlabeled, unlabeled, fmt='%0'+str(num_digits)+'u') + + # Update active learning config file with new cycle and random number + # generator seed + cycle_config = {} + with open(args.cycle_config) as f: + cycle_config = json.load(f) + cycle_config['al_cycle'] += 1 + cycle_config['next_cycle'] += 1 + rng = np.random.default_rng(args.seed) + cycle_config['al_seed'] = int(rng.integers(1000)) + with open(args.cycle_config_next, "w") as f: + json.dump(cycle_config, f, indent = 4) + +if __name__ == '__main__': + main() diff --git a/examples/al_object_detection/test_hints.sh b/examples/al_object_detection/test_hints.sh new file mode 100755 index 00000000..84e6a145 --- /dev/null +++ b/examples/al_object_detection/test_hints.sh @@ -0,0 +1 @@ +python cycle_select.py configs/MIAOD-GRAY.py --work_directory work_dirs/test --cycle 0 --model work_dirs/test/cycle.pth --labeled work_dirs/test/X_L_0.npy --seed 666 --labeled_next work_dirs/test/X_L_1.npy --unselected work_dirs/test/X_U_1.npy --bbox_output work_dirs/test/labeling_hints_1.txt diff --git a/examples/al_object_detection/train.py b/examples/al_object_detection/train.py new file mode 100644 index 00000000..e4aefecb --- /dev/null +++ b/examples/al_object_detection/train.py @@ -0,0 +1,216 @@ +import argparse +import copy +import os +import os.path as osp +import time + +import mmcv +import torch +import random +import numpy as np +from mmcv import Config, DictAction +from mmcv.parallel import MMDataParallel, MMDistributedDataParallel +from mmcv.runner import init_dist, load_checkpoint, save_checkpoint +from mmcv.utils import get_git_hash + +from mmdet import __version__ +from mmdet.apis import set_random_seed, train_detector +from mmdet.apis import single_gpu_test, calculate_uncertainty +from mmdet.datasets import build_dataloader, build_dataset +from mmdet.models import build_detector +from mmdet.utils import collect_env, get_root_logger +#from mmdet.utils.active_datasets import * +from mmdet.utils.hdc.active_datasets import * +from tools.utils import losstype +from torch import distributed as dist + + +def parse_args(): + parser = argparse.ArgumentParser(description='Train a detector') + parser.add_argument('config', help='train config file path') + parser.add_argument('--cycle', help='active learning iteration >= 0') + parser.add_argument('--labeled', help='labeled samples list file') + parser.add_argument('--unselected', help='unselected samples list file') + parser.add_argument('--model_prev', + help='previous cycle model checkpoint file') + parser.add_argument('--model', help='model checkpoint file') + parser.add_argument('--work_directory', + help='the dir to save logs and model checkpoints') + parser.add_argument('--no-validate', action='store_false', + help='whether not to evaluate the checkpoint during training') + group_gpus = parser.add_mutually_exclusive_group() + group_gpus.add_argument('--gpus', type=int, + help='number of gpus to use (only applicable to non-distributed run)') + group_gpus.add_argument('--gpu_ids', type=int, nargs='+', + help='ids of gpus to use (only applicable to non-distributed run)') + parser.add_argument('--seed', type=int, default=666, help='random seed') + parser.add_argument('--deterministic', action='store_true', + help='whether to set deterministic options for CUDNN backend.') + parser.add_argument('--options', nargs='+', action=DictAction, + help='arguments in dict') + parser.add_argument('--launcher', + choices=['none', 'pytorch', 'slurm', 'mpi'], + default='none', help='job launcher') + parser.add_argument('--local_rank', type=int, default=0) + args = parser.parse_args() + if 'LOCAL_RANK' not in os.environ: + os.environ['LOCAL_RANK'] = str(args.local_rank) + return args + + +def main(): + args = parse_args() + + assert (args.cycle and args.labeled and args.unselected and args.model), \ + ('Please specify active learning cycle, file names of labeled ' + 'and unselected image list files, and model checkpoint file ' + 'with arguments "--cycle", "--labeled", "--unselected" and "model"') + + cfg = Config.fromfile(args.config) + if args.options is not None: + cfg.merge_from_dict(args.options) + + # set cudnn_benchmark + if cfg.get('cudnn_benchmark', False): + torch.backends.cudnn.benchmark = True + + # work_directory is determined in this priority: CLI > config > defasult + if args.work_directory is not None: + # update work_directory from CLI args if args.work_directory is not None + cfg.work_directory = args.work_directory + elif cfg.get('work_directory', None) is None: + # derive work_directory from config name if cfg.work_directory is None + cfg.work_directory = osp.join('./work_dirs', + osp.splitext(osp.basename(args.config))[0]) + + # TO DO: placeholder for distributed processing. Will require code changes + if args.gpu_ids is not None: + cfg.gpu_ids = args.gpu_ids + else: + cfg.gpu_ids = range(1) if args.gpus is None else range(args.gpus) + + # init distributed env first, since logger depends on the dist info. + if args.launcher == 'none': + distributed = False + else: + distributed = True + init_dist(args.launcher, **cfg.dist_params) + + # create work_directory + mmcv.mkdir_or_exist(osp.abspath(cfg.work_directory)) + + # dump config + cfg.dump(osp.join(cfg.work_directory, + f'cycle_train{args.cycle}_' + osp.basename(args.config))) + + # init the logger before other steps + timestamp = time.strftime('%Y%m%d_%H%M%S', time.localtime()) + log_file = osp.join(cfg.work_directory, f'cycle_train{args.cycle}.log') + logger = get_root_logger(log_file=log_file, log_level=cfg.log_level) + + # init the meta dict to record some important information + # such as environment info and seed, which will be logged + meta = dict() + + # log env info + env_info_dict = collect_env() + env_info = '\n'.join([f'{k}: {v}' for k, v in env_info_dict.items()]) + dash_line = '-' * 60 + '\n' + logger.info('Environment info:\n' + dash_line + env_info + '\n' + dash_line) + meta['env_info'] = env_info + + # log some basic info + logger.info(f'Distributed training: {distributed}') + logger.info(f'Config:\n{cfg.pretty_text}') + + # set random seeds + if args.seed is not None: + logger.info( + f'Set random seed to {args.seed}, deterministic: {args.deterministic}') + set_random_seed(args.seed, deterministic=args.deterministic) + cfg.seed = args.seed + meta['seed'] = args.seed + + # load lists of all, labeled and unselected images + all_anns = load_ann_list(cfg.data.train.dataset.ann_file) + + if len(all_anns[0]) == 1: + X_all = np.arange(len(all_anns)) + else: + j = 0 + for i in range(len(all_anns)): + j += len(all_anns[i]) + X_all = np.arange(j) + X_L = np.load(args.labeled) + X_U = np.load(args.unselected) + cycle = args.cycle + cfg.cycles = list((args.cycle)) + initial_step = cfg.lr_config.step + + # get the config of the labeled dataset + cfg = create_X_L_file(cfg, X_L, all_anns, cycle) + + # build the model and load checkpoint, if specified + if args.model_prev is not None: + cfg.model.pretrained = None + if cfg.model.get('neck'): + if cfg.model.neck.get('rfp_backbone'): + if cfg.model.neck.rfp_backbone.get('pretrained'): + cfg.model.neck.rfp_backbone.pretrained = None + model = build_detector(cfg.model, train_cfg=cfg.train_cfg, + test_cfg=cfg.test_cfg) + if args.model_prev is not None: + checkpoint = load_checkpoint(model, args.model_prev) + + # load dataset + datasets = [build_dataset(cfg.data.train)] + if len(cfg.workflow) == 2: + + val_dataset = copy.deepcopy(cfg.data.val) + val_dataset.pipeline = cfg.data.train.dataset.pipeline + datasets.append(build_dataset(val_dataset)) + model.CLASSES = datasets[0].CLASSES + + # save mmdet version, config file content and class names in + # checkpoints as meta data + if cfg.checkpoint_config is None: + cfg.checkpoint_config = dict() + cfg.checkpoint_config.meta = dict( + mmdet_version=__version__ + get_git_hash()[:7], + config=cfg.pretty_text, CLASSES=datasets[0].CLASSES) + + for epoch in range(cfg.epoch): + # Only in the last 3 epoch does the learning rate need to be reduced + # and the model needs to be evaluated. + if epoch == cfg.epoch - 1: + cfg.lr_config.step = initial_step + cfg.evaluation.interval = cfg.epoch_ratio[0] + else: + cfg.lr_config.step = [1000] + #cfg.evaluation.interval = 100 + + + + cfg = create_X_L_file(cfg, X_L, all_anns, cycle) + + if dist.is_initialized(): + torch.distributed.barrier() + datasets = [build_dataset(cfg.data.train)] + losstype.update_vars(0) + cfg.total_epochs = cfg.epoch_ratio[0] + cfg_bak = cfg.deepcopy() + for name, value in model.named_parameters(): + value.requires_grad = True + time.sleep(2) + train_detector(model, datasets, cfg, + distributed=distributed, + validate=args.no_validate, + timestamp=timestamp, meta=meta) + cfg = cfg_bak + + # save the model to checkpoint + save_checkpoint(model, args.model, meta=cfg.checkpoint_config.meta) + + +if __name__ == '__main__': + main() diff --git a/examples/al_object_detection/uuid.json b/examples/al_object_detection/uuid.json new file mode 100644 index 00000000..5b05a272 --- /dev/null +++ b/examples/al_object_detection/uuid.json @@ -0,0 +1 @@ +{"uuid_var": "b4d14eb9-2c07-48fa-bcd6-63be0c8df8f4"} \ No newline at end of file diff --git a/examples/example-get-started/Query_Tester-base_mlmd.ipynb b/examples/example-get-started/Query_Tester-base_mlmd.ipynb index a8588ea0..69ec3afa 100644 --- a/examples/example-get-started/Query_Tester-base_mlmd.ipynb +++ b/examples/example-get-started/Query_Tester-base_mlmd.ipynb @@ -10,7 +10,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 40, "id": "d8cecabd", "metadata": {}, "outputs": [], @@ -24,25 +24,36 @@ "id": "9ae54008", "metadata": {}, "source": [ - "##### Initialize the library and get all the stages in the pipeline\n", + "### Initialize the library and get all the stages in the pipeline\n", "Point the library to the metadata file.
\n", "The `get_pipeline_stages` call point to the different stages in the pipeline." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 53, "id": "8b735117", "metadata": { "pycharm": { "is_executing": true } }, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "['Prepare', 'Featurize', 'Train', 'Evaluate']" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], "source": [ "query = cmfquery.CmfQuery(\"./mlmd\")\n", - "stages = query.get_pipeline_stages(\"Test-env\")\n", - "print(stages)" + "pipelines = query.get_pipeline_names()\n", + "stages = query.get_pipeline_stages(pipelines[0])\n", + "display(stages)" ] }, { @@ -50,95 +61,340 @@ "id": "5ac1591f", "metadata": {}, "source": [ - "##### Query the Executions in each stage" + "### Query the Executions in each stage" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 54, "id": "da0ee66d", "metadata": { "pycharm": { "is_executing": true } }, - "outputs": [], - "source": [ - "executions = query.get_all_executions_in_stage('Prepare')\n", - "print(executions)" - ] - }, - { - "cell_type": "markdown", - "id": "cbc05fd9", - "metadata": {}, - "source": [ - "##### Query the Executions in each stage\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "cd81efe0", - "metadata": { - "pycharm": { - "is_executing": true - } - }, - "outputs": [], - "source": [ - "executions = query.get_all_executions_in_stage('Featurize')\n", - "print(executions)" - ] - }, - { - "cell_type": "markdown", - "id": "dd5c7d70", - "metadata": { - "pycharm": { - "name": "#%% md\n" - } - }, - "source": [ - "##### Query the Executions in each stage" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "32c8fd0c", - "metadata": { - "pycharm": { - "is_executing": true - } - }, - "outputs": [], - "source": [ - "executions = query.get_all_executions_in_stage('Train')\n", - "print(executions)" - ] - }, - { - "cell_type": "markdown", - "id": "0342196a", - "metadata": {}, - "source": [ - "##### Query the Executions in each stage" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "e1bcfcd0", - "metadata": { - "pycharm": { - "is_executing": true + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Displaying execution for stage Prepare\n" + ] + }, + { + "data": { + "text/html": [ + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Context_ID Context_Type \\\n",
+ "0 2 Prepare \n",
+ "\n",
+ " Execution \\\n",
+ "0 ['src/parse.py', 'artifacts/data.xml.gz', 'artifacts/parsed'] \n",
+ "\n",
+ " Git_End_Commit Git_Repo \\\n",
+ "0 /tmp/cmf/example_get_started/git_remote \n",
+ "\n",
+ " Git_Start_Commit Pipeline_Type Pipeline_id id \\\n",
+ "0 8158283953c04affb8fe5ea6710656564ede7d3a Test-env 1 1 \n",
+ "\n",
+ " seed split \n",
+ "0 20170428 0.2 "
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Displaying execution for stage Featurize\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | Context_ID | \n", + "Context_Type | \n", + "Execution | \n", + "Git_End_Commit | \n", + "Git_Repo | \n", + "Git_Start_Commit | \n", + "Pipeline_Type | \n", + "Pipeline_id | \n", + "id | \n", + "seed | \n", + "split | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "2 | \n", + "Prepare | \n", + "['src/parse.py', 'artifacts/data.xml.gz', 'artifacts/parsed'] | \n", + "\n", + " | /tmp/cmf/example_get_started/git_remote | \n", + "8158283953c04affb8fe5ea6710656564ede7d3a | \n", + "Test-env | \n", + "1 | \n", + "1 | \n", + "20170428 | \n", + "0.2 | \n", + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Context_ID Context_Type \\\n",
+ "0 3 Featurize-execution \n",
+ "\n",
+ " Execution \\\n",
+ "0 ['src/featurize.py', 'artifacts/parsed', 'artifacts/features'] \n",
+ "\n",
+ " Git_End_Commit Git_Repo \\\n",
+ "0 /tmp/cmf/example_get_started/git_remote \n",
+ "\n",
+ " Git_Start_Commit Pipeline_Type Pipeline_id id \\\n",
+ "0 8158283953c04affb8fe5ea6710656564ede7d3a Test-env 1 2 \n",
+ "\n",
+ " max_features ngrams \n",
+ "0 3000 2 "
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Displaying execution for stage Train\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | Context_ID | \n", + "Context_Type | \n", + "Execution | \n", + "Git_End_Commit | \n", + "Git_Repo | \n", + "Git_Start_Commit | \n", + "Pipeline_Type | \n", + "Pipeline_id | \n", + "id | \n", + "max_features | \n", + "ngrams | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "3 | \n", + "Featurize-execution | \n", + "['src/featurize.py', 'artifacts/parsed', 'artifacts/features'] | \n", + "\n", + " | /tmp/cmf/example_get_started/git_remote | \n", + "8158283953c04affb8fe5ea6710656564ede7d3a | \n", + "Test-env | \n", + "1 | \n", + "2 | \n", + "3000 | \n", + "2 | \n", + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Context_ID Context_Type \\\n",
+ "0 4 Train-execution \n",
+ "\n",
+ " Execution Git_End_Commit \\\n",
+ "0 ['src/train.py', 'artifacts/features', 'artifacts/model'] \n",
+ "\n",
+ " Git_Repo \\\n",
+ "0 /tmp/cmf/example_get_started/git_remote \n",
+ "\n",
+ " Git_Start_Commit Pipeline_Type Pipeline_id id \\\n",
+ "0 8158283953c04affb8fe5ea6710656564ede7d3a Test-env 1 3 \n",
+ "\n",
+ " min_split n_est seed \n",
+ "0 64 100 20170428 "
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Displaying execution for stage Evaluate\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | Context_ID | \n", + "Context_Type | \n", + "Execution | \n", + "Git_End_Commit | \n", + "Git_Repo | \n", + "Git_Start_Commit | \n", + "Pipeline_Type | \n", + "Pipeline_id | \n", + "id | \n", + "min_split | \n", + "n_est | \n", + "seed | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "4 | \n", + "Train-execution | \n", + "['src/train.py', 'artifacts/features', 'artifacts/model'] | \n", + "\n", + " | /tmp/cmf/example_get_started/git_remote | \n", + "8158283953c04affb8fe5ea6710656564ede7d3a | \n", + "Test-env | \n", + "1 | \n", + "3 | \n", + "64 | \n", + "100 | \n", + "20170428 | \n", + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Context_ID Context_Type \\\n",
+ "0 5 Evaluate-execution \n",
+ "\n",
+ " Execution \\\n",
+ "0 ['src/test.py', 'artifacts/model', 'artifacts/features', 'artifacts/tes... \n",
+ "\n",
+ " Git_End_Commit Git_Repo \\\n",
+ "0 /tmp/cmf/example_get_started/git_remote \n",
+ "\n",
+ " Git_Start_Commit Pipeline_Type Pipeline_id id \n",
+ "0 8158283953c04affb8fe5ea6710656564ede7d3a Test-env 1 4 "
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
}
- },
- "outputs": [],
+ ],
"source": [
- "executions = query.get_all_executions_in_stage('Evaluate')\n",
- "print(executions)"
+ "for stage in stages:\n",
+ " executions = query.get_all_executions_in_stage(stage)\n",
+ " print(f\"Displaying execution for stage {stage}\")\n",
+ " display(executions)"
]
},
{
@@ -146,44 +402,146 @@
"id": "5a3599af",
"metadata": {},
"source": [
- "##### Get all the artifacts of execution. \n",
+ "### Get all artifacts of an execution. \n",
"input parameter - execution_id| \n", + " | Context_ID | \n", + "Context_Type | \n", + "Execution | \n", + "Git_End_Commit | \n", + "Git_Repo | \n", + "Git_Start_Commit | \n", + "Pipeline_Type | \n", + "Pipeline_id | \n", + "id | \n", + "
|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "5 | \n", + "Evaluate-execution | \n", + "['src/test.py', 'artifacts/model', 'artifacts/features', 'artifacts/tes... | \n", + "\n", + " | /tmp/cmf/example_get_started/git_remote | \n", + "8158283953c04affb8fe5ea6710656564ede7d3a | \n", + "Test-env | \n", + "1 | \n", + "4 | \n", + "
\n", "output parameter - artifacts
\n" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 56, "id": "6fa93876", "metadata": { "pycharm": { "is_executing": true } }, - "outputs": [], - "source": [ - "artifacts = query.get_all_artifacts_for_execution(1)\n", - "print(artifacts)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "2e82cf28", - "metadata": { - "pycharm": { - "is_executing": true + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Displaying the artifacts for execution with id 1 belonging to Prepare\n" + ] + }, + { + "data": { + "text/html": [ + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Commit create_time_since_epoch \\\n",
+ "0 commit 03c25dfdb6c188b7b04f7e675dec072de192b851 1667255770792 \n",
+ "1 commit 4fba7197919fb85dd1a0899d2cf5c5c690ee607c 1667255774532 \n",
+ "2 commit 5dfd3ac63c950f6394e5b7cebd55343402c7fdb6 1667255776391 \n",
+ "\n",
+ " event git_repo id \\\n",
+ "0 INPUT /tmp/cmf/example_get_started/git_remote 1 \n",
+ "1 OUTPUT /tmp/cmf/example_get_started/git_remote 2 \n",
+ "2 OUTPUT /tmp/cmf/example_get_started/git_remote 3 \n",
+ "\n",
+ " last_update_time_since_epoch \\\n",
+ "0 1667255778222 \n",
+ "1 1667255774532 \n",
+ "2 1667255776391 \n",
+ "\n",
+ " name type \\\n",
+ "0 artifacts/data.xml.gz:236d9502e0283d91f689d7038b8508a2 Dataset \n",
+ "1 artifacts/parsed/train.tsv:22ec7737f442cfc81e8c701fb58d1007 Dataset \n",
+ "2 artifacts/parsed/test.tsv:03e3627bda150c8cf51a55ef96ab3ede Dataset \n",
+ "\n",
+ " uri user-metadata1 user-metadata2 \n",
+ "0 236d9502e0283d91f689d7038b8508a2 metadata_value metadata_value \n",
+ "1 22ec7737f442cfc81e8c701fb58d1007 NaN NaN \n",
+ "2 03e3627bda150c8cf51a55ef96ab3ede NaN NaN "
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
}
- },
- "outputs": [],
+ ],
"source": [
- "# print(pd.options.display.max_colwidth)\n",
- "pd.options.display.max_colwidth = 75\n",
- "artifacts = query.get_all_artifacts_for_execution(2)\n",
"\n",
- "# print(artifacts)\n",
- "print(artifacts[\"name\"])\n",
- "print(artifacts[\"event\"])"
+ "executions = query.get_all_executions_in_stage(stages[0])\n",
+ "print(f\"Displaying the artifacts for execution with id {executions.iloc[0]['id']} belonging to {stages[0]}\")\n",
+ "artifacts = query.get_all_artifacts_for_execution(executions.iloc[0][\"id\"])\n",
+ "display(artifacts)"
]
},
{
@@ -191,22 +549,78 @@
"id": "d15b7386",
"metadata": {},
"source": [
- "#### get all executions for an artifact(pass the artifact full name as the input parameter)"
+ "### Get all executions for an artifact (pass the artifact full name as the input parameter)"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 57,
"id": "f1632d60",
"metadata": {
"pycharm": {
"is_executing": true
}
},
- "outputs": [],
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | Commit | \n", + "create_time_since_epoch | \n", + "event | \n", + "git_repo | \n", + "id | \n", + "last_update_time_since_epoch | \n", + "name | \n", + "type | \n", + "uri | \n", + "user-metadata1 | \n", + "user-metadata2 | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "commit 03c25dfdb6c188b7b04f7e675dec072de192b851 | \n", + "1667255770792 | \n", + "INPUT | \n", + "/tmp/cmf/example_get_started/git_remote | \n", + "1 | \n", + "1667255778222 | \n", + "artifacts/data.xml.gz:236d9502e0283d91f689d7038b8508a2 | \n", + "Dataset | \n", + "236d9502e0283d91f689d7038b8508a2 | \n", + "metadata_value | \n", + "metadata_value | \n", + "
| 1 | \n", + "commit 4fba7197919fb85dd1a0899d2cf5c5c690ee607c | \n", + "1667255774532 | \n", + "OUTPUT | \n", + "/tmp/cmf/example_get_started/git_remote | \n", + "2 | \n", + "1667255774532 | \n", + "artifacts/parsed/train.tsv:22ec7737f442cfc81e8c701fb58d1007 | \n", + "Dataset | \n", + "22ec7737f442cfc81e8c701fb58d1007 | \n", + "NaN | \n", + "NaN | \n", + "
| 2 | \n", + "commit 5dfd3ac63c950f6394e5b7cebd55343402c7fdb6 | \n", + "1667255776391 | \n", + "OUTPUT | \n", + "/tmp/cmf/example_get_started/git_remote | \n", + "3 | \n", + "1667255776391 | \n", + "artifacts/parsed/test.tsv:03e3627bda150c8cf51a55ef96ab3ede | \n", + "Dataset | \n", + "03e3627bda150c8cf51a55ef96ab3ede | \n", + "NaN | \n", + "NaN | \n", + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Type execution_id execution_name pipeline stage\n",
+ "0 INPUT 1 Test-env Prepare"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
"source": [
- "linked = query.get_all_executions_for_artifact(\"artifacts/data.xml.gz:a304afb96060aad90176268345e10355\")\n",
- "print(linked)"
+ "#Provide the artifact in name:hash format\n",
+ "artifacts = query.get_all_artifacts_for_execution(executions.iloc[0]['id'])\n",
+ "for index, art in artifacts.iterrows():\n",
+ " if art[\"event\"] == \"INPUT\":\n",
+ " artifact_name = art[\"name\"]\n",
+ " break\n",
+ "linked = query.get_all_executions_for_artifact(artifact_name)\n",
+ "display(linked)"
]
},
{
@@ -214,22 +628,138 @@
"id": "7ad864e3",
"metadata": {},
"source": [
- "#### Get all the parent artifacts of an artifact. Provides the artifact lineage chain"
+ "### Get all the parent artifacts of an artifact. Provides the artifact lineage chain"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 58,
"id": "09652709",
"metadata": {
"pycharm": {
"is_executing": true
}
},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Parent artifact of artifacts/features/train.pkl:5de5e987eadb4b86fc47604b59cb3725\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | Type | \n", + "execution_id | \n", + "execution_name | \n", + "pipeline | \n", + "stage | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "INPUT | \n", + "1 | \n", + "\n", + " | Test-env | \n", + "Prepare | \n", + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Commit create_time_since_epoch \\\n",
+ "0 commit 4fba7197919fb85dd1a0899d2cf5c5c690ee607c 1667255774532 \n",
+ "1 commit 5dfd3ac63c950f6394e5b7cebd55343402c7fdb6 1667255776391 \n",
+ "2 commit 03c25dfdb6c188b7b04f7e675dec072de192b851 1667255770792 \n",
+ "\n",
+ " git_repo id last_update_time_since_epoch \\\n",
+ "0 /tmp/cmf/example_get_started/git_remote 2 1667255774532 \n",
+ "1 /tmp/cmf/example_get_started/git_remote 3 1667255776391 \n",
+ "2 /tmp/cmf/example_get_started/git_remote 1 1667255778222 \n",
+ "\n",
+ " name type \\\n",
+ "0 artifacts/parsed/train.tsv:22ec7737f442cfc81e8c701fb58d1007 Dataset \n",
+ "1 artifacts/parsed/test.tsv:03e3627bda150c8cf51a55ef96ab3ede Dataset \n",
+ "2 artifacts/data.xml.gz:236d9502e0283d91f689d7038b8508a2 Dataset \n",
+ "\n",
+ " uri user-metadata1 user-metadata2 \n",
+ "0 22ec7737f442cfc81e8c701fb58d1007 NaN NaN \n",
+ "1 03e3627bda150c8cf51a55ef96ab3ede NaN NaN \n",
+ "2 236d9502e0283d91f689d7038b8508a2 metadata_value metadata_value "
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
"source": [
- "linked = query.get_all_parent_artifacts(\"artifacts/features/test.pkl\")\n",
- "print(linked)"
+ "artifacts = query.get_all_artifacts_for_execution(2)\n",
+ "for index, art in artifacts.iterrows():\n",
+ " if art[\"event\"] == \"OUTPUT\":\n",
+ " artifact_name = art[\"name\"]\n",
+ " break\n",
+ "print(f\"Parent artifact of {artifact_name}\")\n",
+ "linked = query.get_all_parent_artifacts(artifact_name)\n",
+ "display(linked)"
]
},
{
@@ -237,25 +767,133 @@
"id": "63b615f1",
"metadata": {},
"source": [
- "#### Get all the child artifacts of an artifact. Provides the lineage chain in the downstream direction"
+ "### Get all child artifacts of an artifact. It provides the lineage chain of successors"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 46,
"id": "57b85ea6",
"metadata": {
"pycharm": {
"is_executing": true
}
},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Child artifact of artifacts/features/train.pkl:5de5e987eadb4b86fc47604b59cb3725\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | Commit | \n", + "create_time_since_epoch | \n", + "git_repo | \n", + "id | \n", + "last_update_time_since_epoch | \n", + "name | \n", + "type | \n", + "uri | \n", + "user-metadata1 | \n", + "user-metadata2 | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "commit 4fba7197919fb85dd1a0899d2cf5c5c690ee607c | \n", + "1667255774532 | \n", + "/tmp/cmf/example_get_started/git_remote | \n", + "2 | \n", + "1667255774532 | \n", + "artifacts/parsed/train.tsv:22ec7737f442cfc81e8c701fb58d1007 | \n", + "Dataset | \n", + "22ec7737f442cfc81e8c701fb58d1007 | \n", + "NaN | \n", + "NaN | \n", + "
| 1 | \n", + "commit 5dfd3ac63c950f6394e5b7cebd55343402c7fdb6 | \n", + "1667255776391 | \n", + "/tmp/cmf/example_get_started/git_remote | \n", + "3 | \n", + "1667255776391 | \n", + "artifacts/parsed/test.tsv:03e3627bda150c8cf51a55ef96ab3ede | \n", + "Dataset | \n", + "03e3627bda150c8cf51a55ef96ab3ede | \n", + "NaN | \n", + "NaN | \n", + "
| 2 | \n", + "commit 03c25dfdb6c188b7b04f7e675dec072de192b851 | \n", + "1667255770792 | \n", + "/tmp/cmf/example_get_started/git_remote | \n", + "1 | \n", + "1667255778222 | \n", + "artifacts/data.xml.gz:236d9502e0283d91f689d7038b8508a2 | \n", + "Dataset | \n", + "236d9502e0283d91f689d7038b8508a2 | \n", + "metadata_value | \n", + "metadata_value | \n", + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Commit avg_prec \\\n",
+ "0 commit 8158283953c04affb8fe5ea6710656564ede7d3a NaN \n",
+ "1 NaN 0.526754 \n",
+ "\n",
+ " create_time_since_epoch id last_update_time_since_epoch \\\n",
+ "0 1667255811813 6 1667255811813 \n",
+ "1 1667255818388 7 1667255818388 \n",
+ "\n",
+ " metrics_name model_framework \\\n",
+ "0 NaN SKlearn \n",
+ "1 metrics:878d492e-596c-11ed-99a3-b47af137252e:4 NaN \n",
+ "\n",
+ " model_name model_type \\\n",
+ "0 RandomForestClassifier:default RandomForestClassifier \n",
+ "1 NaN NaN \n",
+ "\n",
+ " name roc_auc \\\n",
+ "0 artifacts/model/model.pkl:5f6e4aa57cce9e3a0b2f12e5766d19be:3 NaN \n",
+ "1 metrics:878d492e-596c-11ed-99a3-b47af137252e:4 0.959238 \n",
+ "\n",
+ " type uri \n",
+ "0 Model 5f6e4aa57cce9e3a0b2f12e5766d19be \n",
+ "1 Metrics 878d492e-596c-11ed-99a3-b47af137252e "
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
"source": [
- "linked = query.get_all_child_artifacts(\"artifacts/features/train.pkl\")\n",
- "\n",
- "print(\"Name : \" + linked[\"name\"].to_string(index=False, header=False))\n",
- "print(\"Type : \" + linked[\"type\"].to_string(index=False, header=False))\n",
- "print(\"URI : \" + linked[\"uri\"].to_string(index=False, header=False))"
+ "print(f\"Child artifact of {artifact_name}\")\n",
+ "linked = query.get_all_child_artifacts(artifact_name)\n",
+ "display(linked)\n"
]
},
{
@@ -267,64 +905,313 @@
}
},
"source": [
- "#### Get all the parent artifacts of an artifact. Provides the artifact lineage chain"
+ "### Get all the parent artifacts of an artifact. Provides the artifact lineage chain of predecessors"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 59,
"id": "493bd571",
"metadata": {
"pycharm": {
"is_executing": true
}
},
- "outputs": [],
- "source": [
- "linked = query.get_all_parent_artifacts(\"artifacts/model/model.pkl\")\n",
- "print(\"NAME\")\n",
- "print(linked[\"name\"].to_string(index=False, header=False))\n",
- "print(\"TYPE\")\n",
- "print(linked[\"type\"].to_string(index=False, header=False))\n",
- "print(\"URI\")\n",
- "print(linked[\"uri\"].to_string(index=False, header=False))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "a4c338be",
- "metadata": {
- "pycharm": {
- "is_executing": true
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | Commit | \n", + "avg_prec | \n", + "create_time_since_epoch | \n", + "id | \n", + "last_update_time_since_epoch | \n", + "metrics_name | \n", + "model_framework | \n", + "model_name | \n", + "model_type | \n", + "name | \n", + "roc_auc | \n", + "type | \n", + "uri | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "commit 8158283953c04affb8fe5ea6710656564ede7d3a | \n", + "NaN | \n", + "1667255811813 | \n", + "6 | \n", + "1667255811813 | \n", + "NaN | \n", + "SKlearn | \n", + "RandomForestClassifier:default | \n", + "RandomForestClassifier | \n", + "artifacts/model/model.pkl:5f6e4aa57cce9e3a0b2f12e5766d19be:3 | \n", + "NaN | \n", + "Model | \n", + "5f6e4aa57cce9e3a0b2f12e5766d19be | \n", + "
| 1 | \n", + "NaN | \n", + "0.526754 | \n", + "1667255818388 | \n", + "7 | \n", + "1667255818388 | \n", + "metrics:878d492e-596c-11ed-99a3-b47af137252e:4 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "metrics:878d492e-596c-11ed-99a3-b47af137252e:4 | \n", + "0.959238 | \n", + "Metrics | \n", + "878d492e-596c-11ed-99a3-b47af137252e | \n", + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Commit create_time_since_epoch \\\n",
+ "0 commit 03c25dfdb6c188b7b04f7e675dec072de192b851 1667255770792 \n",
+ "\n",
+ " git_repo id last_update_time_since_epoch \\\n",
+ "0 /tmp/cmf/example_get_started/git_remote 1 1667255778222 \n",
+ "\n",
+ " name type \\\n",
+ "0 artifacts/data.xml.gz:236d9502e0283d91f689d7038b8508a2 Dataset \n",
+ "\n",
+ " uri user-metadata1 user-metadata2 \n",
+ "0 236d9502e0283d91f689d7038b8508a2 metadata_value metadata_value "
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
}
- },
- "outputs": [],
+ ],
"source": [
- "linked = query.get_all_parent_artifacts(\"artifacts/parsed/test.tsv\")\n",
- "print(\"Name : \" + linked[\"name\"].to_string(index=False, header=False))\n",
- "print(\"Type : \" + linked[\"type\"].to_string(index=False, header=False))\n",
- "print(\"URI : \" + linked[\"uri\"].to_string(index=False, header=False))"
+ "\n",
+ "linked = query.get_all_parent_artifacts(linked.iloc[0][\"name\"])\n",
+ "display(linked)"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 60,
"id": "4eb35ec5",
"metadata": {
"pycharm": {
"is_executing": true
}
},
- "outputs": [],
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | Commit | \n", + "create_time_since_epoch | \n", + "git_repo | \n", + "id | \n", + "last_update_time_since_epoch | \n", + "name | \n", + "type | \n", + "uri | \n", + "user-metadata1 | \n", + "user-metadata2 | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "commit 03c25dfdb6c188b7b04f7e675dec072de192b851 | \n", + "1667255770792 | \n", + "/tmp/cmf/example_get_started/git_remote | \n", + "1 | \n", + "1667255778222 | \n", + "artifacts/data.xml.gz:236d9502e0283d91f689d7038b8508a2 | \n", + "Dataset | \n", + "236d9502e0283d91f689d7038b8508a2 | \n", + "metadata_value | \n", + "metadata_value | \n", + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Commit avg_prec \\\n",
+ "0 commit 4fba7197919fb85dd1a0899d2cf5c5c690ee607c NaN \n",
+ "1 commit 5dfd3ac63c950f6394e5b7cebd55343402c7fdb6 NaN \n",
+ "2 commit 4546b0679bcae18bd85893c69581db91da40495c NaN \n",
+ "3 commit d67bedaa20e64e45fe9f553935d9ff0726f19b59 NaN \n",
+ "4 commit 8158283953c04affb8fe5ea6710656564ede7d3a NaN \n",
+ "5 NaN 0.526754 \n",
+ "\n",
+ " create_time_since_epoch git_repo id \\\n",
+ "0 1667255774532 /tmp/cmf/example_get_started/git_remote 2 \n",
+ "1 1667255776391 /tmp/cmf/example_get_started/git_remote 3 \n",
+ "2 1667255800206 /tmp/cmf/example_get_started/git_remote 4 \n",
+ "3 1667255802382 /tmp/cmf/example_get_started/git_remote 5 \n",
+ "4 1667255811813 NaN 6 \n",
+ "5 1667255818388 NaN 7 \n",
+ "\n",
+ " last_update_time_since_epoch \\\n",
+ "0 1667255774532 \n",
+ "1 1667255776391 \n",
+ "2 1667255800206 \n",
+ "3 1667255802382 \n",
+ "4 1667255811813 \n",
+ "5 1667255818388 \n",
+ "\n",
+ " metrics_name model_framework \\\n",
+ "0 NaN NaN \n",
+ "1 NaN NaN \n",
+ "2 NaN NaN \n",
+ "3 NaN NaN \n",
+ "4 NaN SKlearn \n",
+ "5 metrics:878d492e-596c-11ed-99a3-b47af137252e:4 NaN \n",
+ "\n",
+ " model_name model_type \\\n",
+ "0 NaN NaN \n",
+ "1 NaN NaN \n",
+ "2 NaN NaN \n",
+ "3 NaN NaN \n",
+ "4 RandomForestClassifier:default RandomForestClassifier \n",
+ "5 NaN NaN \n",
+ "\n",
+ " name roc_auc \\\n",
+ "0 artifacts/parsed/train.tsv:22ec7737f442cfc81e8c701fb58d1007 NaN \n",
+ "1 artifacts/parsed/test.tsv:03e3627bda150c8cf51a55ef96ab3ede NaN \n",
+ "2 artifacts/features/train.pkl:5de5e987eadb4b86fc47604b59cb3725 NaN \n",
+ "3 artifacts/features/test.pkl:b1f98b4ebd09a0bdc72f1a8c102065dd NaN \n",
+ "4 artifacts/model/model.pkl:5f6e4aa57cce9e3a0b2f12e5766d19be:3 NaN \n",
+ "5 metrics:878d492e-596c-11ed-99a3-b47af137252e:4 0.959238 \n",
+ "\n",
+ " type uri \n",
+ "0 Dataset 22ec7737f442cfc81e8c701fb58d1007 \n",
+ "1 Dataset 03e3627bda150c8cf51a55ef96ab3ede \n",
+ "2 Dataset 5de5e987eadb4b86fc47604b59cb3725 \n",
+ "3 Dataset b1f98b4ebd09a0bdc72f1a8c102065dd \n",
+ "4 Model 5f6e4aa57cce9e3a0b2f12e5766d19be \n",
+ "5 Metrics 878d492e-596c-11ed-99a3-b47af137252e "
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
"source": [
- "linked = query.get_all_child_artifacts(\"artifacts/parsed/test.tsv\")\n",
- "print(\"NAME\")\n",
- "print(linked[\"name\"].to_string(index=False, header=False))\n",
- "print(\"TYPE\")\n",
- "print(linked[\"type\"].to_string(index=False, header=False))\n",
- "print(\"URI\")\n",
- "print(linked[\"uri\"].to_string(index=False, header=False))"
+ "linked = query.get_all_child_artifacts(linked.iloc[0][\"name\"])\n",
+ "display(linked)"
]
},
{
@@ -332,53 +1219,225 @@
"id": "29060ba9",
"metadata": {},
"source": [
- "#### Get immediate child artifacts of an artifact. "
+ "### Get immediate child artifacts of an artifact. "
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 61,
"id": "93bd401d",
"metadata": {
"pycharm": {
"is_executing": true
}
},
- "outputs": [],
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | Commit | \n", + "avg_prec | \n", + "create_time_since_epoch | \n", + "git_repo | \n", + "id | \n", + "last_update_time_since_epoch | \n", + "metrics_name | \n", + "model_framework | \n", + "model_name | \n", + "model_type | \n", + "name | \n", + "roc_auc | \n", + "type | \n", + "uri | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "commit 4fba7197919fb85dd1a0899d2cf5c5c690ee607c | \n", + "NaN | \n", + "1667255774532 | \n", + "/tmp/cmf/example_get_started/git_remote | \n", + "2 | \n", + "1667255774532 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "artifacts/parsed/train.tsv:22ec7737f442cfc81e8c701fb58d1007 | \n", + "NaN | \n", + "Dataset | \n", + "22ec7737f442cfc81e8c701fb58d1007 | \n", + "
| 1 | \n", + "commit 5dfd3ac63c950f6394e5b7cebd55343402c7fdb6 | \n", + "NaN | \n", + "1667255776391 | \n", + "/tmp/cmf/example_get_started/git_remote | \n", + "3 | \n", + "1667255776391 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "artifacts/parsed/test.tsv:03e3627bda150c8cf51a55ef96ab3ede | \n", + "NaN | \n", + "Dataset | \n", + "03e3627bda150c8cf51a55ef96ab3ede | \n", + "
| 2 | \n", + "commit 4546b0679bcae18bd85893c69581db91da40495c | \n", + "NaN | \n", + "1667255800206 | \n", + "/tmp/cmf/example_get_started/git_remote | \n", + "4 | \n", + "1667255800206 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "artifacts/features/train.pkl:5de5e987eadb4b86fc47604b59cb3725 | \n", + "NaN | \n", + "Dataset | \n", + "5de5e987eadb4b86fc47604b59cb3725 | \n", + "
| 3 | \n", + "commit d67bedaa20e64e45fe9f553935d9ff0726f19b59 | \n", + "NaN | \n", + "1667255802382 | \n", + "/tmp/cmf/example_get_started/git_remote | \n", + "5 | \n", + "1667255802382 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "artifacts/features/test.pkl:b1f98b4ebd09a0bdc72f1a8c102065dd | \n", + "NaN | \n", + "Dataset | \n", + "b1f98b4ebd09a0bdc72f1a8c102065dd | \n", + "
| 4 | \n", + "commit 8158283953c04affb8fe5ea6710656564ede7d3a | \n", + "NaN | \n", + "1667255811813 | \n", + "NaN | \n", + "6 | \n", + "1667255811813 | \n", + "NaN | \n", + "SKlearn | \n", + "RandomForestClassifier:default | \n", + "RandomForestClassifier | \n", + "artifacts/model/model.pkl:5f6e4aa57cce9e3a0b2f12e5766d19be:3 | \n", + "NaN | \n", + "Model | \n", + "5f6e4aa57cce9e3a0b2f12e5766d19be | \n", + "
| 5 | \n", + "NaN | \n", + "0.526754 | \n", + "1667255818388 | \n", + "NaN | \n", + "7 | \n", + "1667255818388 | \n", + "metrics:878d492e-596c-11ed-99a3-b47af137252e:4 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "metrics:878d492e-596c-11ed-99a3-b47af137252e:4 | \n", + "0.959238 | \n", + "Metrics | \n", + "878d492e-596c-11ed-99a3-b47af137252e | \n", + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Commit create_time_since_epoch \\\n",
+ "0 commit 8158283953c04affb8fe5ea6710656564ede7d3a 1667255811813 \n",
+ "\n",
+ " id last_update_time_since_epoch model_framework \\\n",
+ "0 6 1667255811813 SKlearn \n",
+ "\n",
+ " model_name model_type \\\n",
+ "0 RandomForestClassifier:default RandomForestClassifier \n",
+ "\n",
+ " name type \\\n",
+ "0 artifacts/model/model.pkl:5f6e4aa57cce9e3a0b2f12e5766d19be:3 Model \n",
+ "\n",
+ " uri \n",
+ "0 5f6e4aa57cce9e3a0b2f12e5766d19be "
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "linked = query.get_one_hop_child_artifacts(artifact_name)\n",
+ "display(linked)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "529817a7",
+ "metadata": {},
"source": [
- "linked = query.get_one_hop_child_artifacts(\"artifacts/data.xml.gz\")\n",
- "print(\"NAME\")\n",
- "print(linked[\"name\"].to_string(index=False, header=False))\n",
- "print(\"TYPE\")\n",
- "print(linked[\"type\"].to_string(index=False, header=False))\n",
- "print(\"URI\")\n",
- "print(linked[\"uri\"].to_string(index=False, header=False))"
+ "### Get all child artifacts "
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 62,
"id": "496ee2bc",
"metadata": {
"pycharm": {
"is_executing": true
}
},
- "outputs": [],
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | Commit | \n", + "create_time_since_epoch | \n", + "id | \n", + "last_update_time_since_epoch | \n", + "model_framework | \n", + "model_name | \n", + "model_type | \n", + "name | \n", + "type | \n", + "uri | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "commit 8158283953c04affb8fe5ea6710656564ede7d3a | \n", + "1667255811813 | \n", + "6 | \n", + "1667255811813 | \n", + "SKlearn | \n", + "RandomForestClassifier:default | \n", + "RandomForestClassifier | \n", + "artifacts/model/model.pkl:5f6e4aa57cce9e3a0b2f12e5766d19be:3 | \n", + "Model | \n", + "5f6e4aa57cce9e3a0b2f12e5766d19be | \n", + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Commit avg_prec \\\n",
+ "0 commit 8158283953c04affb8fe5ea6710656564ede7d3a NaN \n",
+ "1 NaN 0.526754 \n",
+ "\n",
+ " create_time_since_epoch id last_update_time_since_epoch \\\n",
+ "0 1667255811813 6 1667255811813 \n",
+ "1 1667255818388 7 1667255818388 \n",
+ "\n",
+ " metrics_name model_framework \\\n",
+ "0 NaN SKlearn \n",
+ "1 metrics:878d492e-596c-11ed-99a3-b47af137252e:4 NaN \n",
+ "\n",
+ " model_name model_type \\\n",
+ "0 RandomForestClassifier:default RandomForestClassifier \n",
+ "1 NaN NaN \n",
+ "\n",
+ " name roc_auc \\\n",
+ "0 artifacts/model/model.pkl:5f6e4aa57cce9e3a0b2f12e5766d19be:3 NaN \n",
+ "1 metrics:878d492e-596c-11ed-99a3-b47af137252e:4 0.959238 \n",
+ "\n",
+ " type uri \n",
+ "0 Model 5f6e4aa57cce9e3a0b2f12e5766d19be \n",
+ "1 Metrics 878d492e-596c-11ed-99a3-b47af137252e "
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
"source": [
- "linked = query.get_all_child_artifacts(\"artifacts/data.xml.gz\")\n",
- "#print(linked.sort_values('create_time_since_epoch', ascending=True))\n",
- "print(\"NAME\")\n",
- "print(linked[\"name\"].to_string(index=False, header=False))\n",
- "print(\"TYPE\")\n",
- "print(linked[\"type\"].to_string(index=False, header=False))\n",
- "print(\"URI\")\n",
- "print(linked[\"uri\"].to_string(index=False, header=False))"
+ "linked = query.get_all_child_artifacts(artifact_name)\n",
+ "display(linked)"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 63,
"id": "1ae1d868",
"metadata": {
"pycharm": {
@@ -387,13 +1446,12 @@
},
"outputs": [],
"source": [
+ "# Provide Execution id corresponding to a Evaluate stage\n",
"linked = query.get_all_artifacts_for_execution(4)\n",
- "print(\"NAME\")\n",
- "print(linked[\"name\"].to_string(index=False, header=False))\n",
- "print(\"TYPE\")\n",
- "print(linked[\"type\"].to_string(index=False, header=False))\n",
- "print(\"URI\")\n",
- "print(linked[\"uri\"].to_string(index=False, header=False))"
+ "for index, row in linked.iterrows():\n",
+ " if row[\"type\"] == \"Metrics\":\n",
+ " break\n",
+ "\n"
]
},
{
@@ -401,44 +1459,95 @@
"id": "ae3109b7",
"metadata": {},
"source": [
- "### Change the metrics name in the get_artifact call with the metrics name from output of the previous cell"
+ "### Get artifact "
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 64,
"id": "8bf8a41d",
"metadata": {
"pycharm": {
"is_executing": true
}
},
- "outputs": [],
- "source": [
- "artifacts = query.get_artifact(\"metrics:aaae534e-915d-11ec-b106-89841b9859cd:4\")\n",
- "print(artifacts)\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "ca26387d",
- "metadata": {
- "pycharm": {
- "name": "#%%\n",
- "is_executing": true
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | Commit | \n", + "avg_prec | \n", + "create_time_since_epoch | \n", + "id | \n", + "last_update_time_since_epoch | \n", + "metrics_name | \n", + "model_framework | \n", + "model_name | \n", + "model_type | \n", + "name | \n", + "roc_auc | \n", + "type | \n", + "uri | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "commit 8158283953c04affb8fe5ea6710656564ede7d3a | \n", + "NaN | \n", + "1667255811813 | \n", + "6 | \n", + "1667255811813 | \n", + "NaN | \n", + "SKlearn | \n", + "RandomForestClassifier:default | \n", + "RandomForestClassifier | \n", + "artifacts/model/model.pkl:5f6e4aa57cce9e3a0b2f12e5766d19be:3 | \n", + "NaN | \n", + "Model | \n", + "5f6e4aa57cce9e3a0b2f12e5766d19be | \n", + "
| 1 | \n", + "NaN | \n", + "0.526754 | \n", + "1667255818388 | \n", + "7 | \n", + "1667255818388 | \n", + "metrics:878d492e-596c-11ed-99a3-b47af137252e:4 | \n", + "NaN | \n", + "NaN | \n", + "NaN | \n", + "metrics:878d492e-596c-11ed-99a3-b47af137252e:4 | \n", + "0.959238 | \n", + "Metrics | \n", + "878d492e-596c-11ed-99a3-b47af137252e | \n", + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " id type uri \\\n",
+ "0 7 Metrics 878d492e-596c-11ed-99a3-b47af137252e \n",
+ "\n",
+ " name create_time_since_epoch \\\n",
+ "0 metrics:878d492e-596c-11ed-99a3-b47af137252e:4 1667255818388 \n",
+ "\n",
+ " last_update_time_since_epoch \\\n",
+ "0 1667255818388 \n",
+ "\n",
+ " metrics_name avg_prec roc_auc \n",
+ "0 metrics:878d492e-596c-11ed-99a3-b47af137252e:4 0.526754 0.959238 "
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
}
- },
- "outputs": [],
+ ],
"source": [
- "new_parquet_df = pd.read_parquet(\"./slice-a\")\n",
- "print(new_parquet_df)"
+ "artifact = query.get_artifact(row[\"name\"])\n",
+ "display(artifact)\n"
]
}
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3 (ipykernel)",
+ "display_name": "Python 3",
"language": "python",
"name": "python3"
},
@@ -452,9 +1561,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.8.10"
+ "version": "3.6.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
-}
\ No newline at end of file
+}
diff --git a/examples/example-get-started/README.md b/examples/example-get-started/README.md
index 71607c01..088fb07c 100644
--- a/examples/example-get-started/README.md
+++ b/examples/example-get-started/README.md
@@ -4,7 +4,7 @@
1. Copy contents of `example-get-started` directory to a separate directory outside this repository.
-2. Create python virtual environment (version >= 3.6), install git, install python dependencies
+2. Create python virtual environment (version >= 3.6 and < 3.9), install git, install python dependencies
3. Modify the [sample_env](./sample_env) file with appropriate values for the exports.
diff --git a/examples/example-get-started/initialize.sh b/examples/example-get-started/initialize.sh
index 36910017..1435cdaf 100644
--- a/examples/example-get-started/initialize.sh
+++ b/examples/example-get-started/initialize.sh
@@ -8,18 +8,18 @@
echo "[1/5] [GIT/DVC INIT ] executing git init and dvc init."
git init -q
dvc init -q
-git config --global user.name "${GIT_USER_NAME}"
-git config --global user.email "${GIT_USER_EMAIL}"
+git config --global user.name "${GIT_USER_NAME:-first second}"
+git config --global user.email "${GIT_USER_EMAIL:-first.second@corp.com}"
echo "[2/5] [INITIAL COMMIT] performing initial blank commit into main."
git checkout -b master
git commit --allow-empty -n -m "Initial code commit"
echo "[3/5] [GIT REMOTE ] setting git remote to ${GIT_REMOTE_URL}"
-git remote add origin "${GIT_REMOTE_URL}"
+git remote add origin "${GIT_REMOTE_URL:-/tmp/gitremote/url}"
echo "[4/5] [DVC REMOTE ] setting dvc remote to ${DVC_REMOTE_URL}"
-dvc remote add myremote "${DVC_REMOTE_URL}"
+dvc remote add myremote -f "${DVC_REMOTE_URL:-/tmp/dvcremote}"
dvc remote default myremote
echo "[5/5] [NEXT STEPS ]"
diff --git a/examples/example-get-started/sample_env b/examples/example-get-started/sample_env
index 789891d6..e1db4518 100644
--- a/examples/example-get-started/sample_env
+++ b/examples/example-get-started/sample_env
@@ -2,6 +2,6 @@ export DVC_REMOTE_URL=/tmp/cmf/example_get_started/dvc_remote
export GIT_USER_NAME="First Second"
export GIT_USER_EMAIL=first.second@corp.org
export GIT_REMOTE_URL=git@github.com:first-second/experiment-repo.git
-export NEO4J_USER_NAME=neo4j
-export NEO4J_PASSWD=neo4j
+export NEO4J_USER_NAME=user
+export NEO4J_PASSWD=XXXXXX
export NEO4J_URI="bolt://IP:PORT"
diff --git a/examples/example-get-started/src/featurize.py b/examples/example-get-started/src/featurize.py
index 31df2833..0cb99973 100644
--- a/examples/example-get-started/src/featurize.py
+++ b/examples/example-get-started/src/featurize.py
@@ -72,8 +72,9 @@ def featurize(input_dir: str, output_dir: str) -> None:
os.makedirs(output_dir, exist_ok=True)
output_ds = Dataset(train=os.path.join(output_dir, "train.pkl"), test=os.path.join(output_dir, "test.pkl"))
-
- metawriter = cmf.Cmf(filename="mlmd", pipeline_name="Test-env")
+ graph_env = os.getenv("NEO4J","False")
+ graph = True if graph_env == "True" or graph_env == "TRUE" else False
+ metawriter = cmf.Cmf(filename="mlmd", pipeline_name="Test-env", graph=graph)
_ = metawriter.create_context(pipeline_stage="Featurize")
_ = metawriter.create_execution(execution_type="Featurize-execution", custom_properties=params)
diff --git a/examples/example-get-started/src/parse.py b/examples/example-get-started/src/parse.py
index 2eaea35b..23729d46 100644
--- a/examples/example-get-started/src/parse.py
+++ b/examples/example-get-started/src/parse.py
@@ -59,8 +59,9 @@ def parse(input_file: str, output_dir: str) -> None:
"""
params = yaml.safe_load(open("params.yaml"))["parse"]
random.seed(params["seed"])
-
- metawriter = cmf.Cmf(filename="mlmd", pipeline_name="Test-env")
+ graph_env = os.getenv("NEO4J","False")
+ graph = True if graph_env == "True" or graph_env == "TRUE" else False
+ metawriter = cmf.Cmf(filename="mlmd", pipeline_name="Test-env", graph=graph)
_ = metawriter.create_context(pipeline_stage="Prepare", custom_properties={"user-metadata1": "metadata_value"})
_ = metawriter.create_execution(execution_type="Prepare", custom_properties=params)
_ = metawriter.log_dataset(input_file, "input", custom_properties={"user-metadata1": "metadata_value"})
diff --git a/examples/example-get-started/src/test.py b/examples/example-get-started/src/test.py
index 065e9843..27e54bbc 100644
--- a/examples/example-get-started/src/test.py
+++ b/examples/example-get-started/src/test.py
@@ -46,8 +46,9 @@ def test(model_dir: str, dataset_dir: str, output_dir: str) -> None:
prc=os.path.join(output_dir, 'prc.json'),
roc=os.path.join(output_dir, 'roc.json')
)
-
- metawriter = cmf.Cmf(filename="mlmd", pipeline_name="Test-env")
+ graph_env = os.getenv("NEO4J","False")
+ graph = True if graph_env == "True" or graph_env == "TRUE" else False
+ metawriter = cmf.Cmf(filename="mlmd", pipeline_name="Test-env", graph=graph)
_ = metawriter.create_context(pipeline_stage="Evaluate")
_ = metawriter.create_execution(execution_type="Evaluate-execution")
diff --git a/examples/example-get-started/src/train.py b/examples/example-get-started/src/train.py
index 863b9eb1..b7e1124f 100644
--- a/examples/example-get-started/src/train.py
+++ b/examples/example-get-started/src/train.py
@@ -37,8 +37,9 @@ def train(input_dir: str, output_dir: str) -> None:
Output: ${output_dir}/model.pkl
"""
params = yaml.safe_load(open("params.yaml"))["train"]
-
- metawriter = cmf.Cmf(filename="mlmd", pipeline_name="Test-env")
+ graph_env = os.getenv("NEO4J","False")
+ graph = True if graph_env == "True" or graph_env == "TRUE" else False
+ metawriter = cmf.Cmf(filename="mlmd", pipeline_name="Test-env", graph=graph)
_ = metawriter.create_context(pipeline_stage="Train")
_ = metawriter.create_execution(execution_type="Train-execution", custom_properties=params)
diff --git a/examples/example-get-started/test-data-slice.py b/examples/example-get-started/test-data-slice.py
index 0442851f..c4f95c08 100644
--- a/examples/example-get-started/test-data-slice.py
+++ b/examples/example-get-started/test-data-slice.py
@@ -59,11 +59,11 @@ def generate_dataset():
# Creating the data slice - today we have only path and hash.
# Would need to expand to take in more metadata.
for i in range(1, 3, 1):
- dataslice: cmf.Cmf.dataslice = metawriter.create_dataslice(name="slice-" + str(i))
+ dataslice: cmf.Cmf.DataSlice = metawriter.create_dataslice(name="slice-" + str(i))
for _ in range(1, 20, 1):
j = random.randrange(1, 100)
print(folder_path + "/" + str(j) + ".txt")
- dataslice.add_data(path=folder_path + "/" + str(j) + ".txt", custom_props={"key1": "value1", "key2": "value2"})
+ dataslice.add_data(path=folder_path + "/" + str(j) + ".txt", custom_properties={"key1": "value1", "key2": "value2"})
dataslice.commit()
# Reading the files in the slice.
@@ -73,13 +73,13 @@ def generate_dataset():
for label, content in df.iterrows():
record = label
row_content = content
-
+print("Updating the value from `value1` to `1` and from `value2` to `2`")
print("Before update")
print(record)
print(row_content)
# Update the metadata for a record in the slice.
-metawriter.update_dataslice(name="slice-1", record=record, custom_props={"key1": "1", "key2": "2"})
+metawriter.update_dataslice(name="slice-1", record=record, custom_properties={"key1": "1", "key2": "2"})
df = metawriter.read_dataslice(name="slice-1")
print("After update")
diff --git a/mkdocs.yaml b/mkdocs.yaml
new file mode 100644
index 00000000..041b2aeb
--- /dev/null
+++ b/mkdocs.yaml
@@ -0,0 +1,88 @@
+###
+# Copyright (2022) Hewlett Packard Enterprise Development LP
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# You may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+###
+
+
+site_name: CMF
+site_description: CMF - Common Metadata Framework
+site_author: AI Research & Development Lab (Hewlett-Packard Labs)
+
+
+repo_name: HewlettPackard/cmf
+repo_url: https://github.com/HewlettPackard/cmf
+edit_uri: ""
+docs_dir: docs/
+
+
+theme:
+ name: material
+ language: en
+ features:
+ - navigation.sections # Sections are included in the navigation on the left.
+ - toc.integrate # Table of contents is integrated on the left; does not appear separately on the right.
+ - header.autohide # header disappears as you scroll
+ palette:
+ primary: 'blue'
+ accent: 'blue'
+ font:
+ text: 'Circular'
+ code: 'Inconsolata'
+
+
+markdown_extensions:
+ - admonition
+ - pymdownx.arithmatex: # Render LaTeX via MathJax
+ generic: true
+ - pymdownx.tasklist
+ - pymdownx.details # Allowing hidden expandable regions denoted by ???
+ - pymdownx.superfences # Seems to enable syntax highlighting when used with the Material theme.
+ - pymdownx.tabbed:
+ alternate_style: true
+ - toc:
+ permalink: True
+ - attr_list
+ - md_in_html # Parse markdown inside HTML tags (default is not to parse).
+
+extra_css:
+ - extra.css # This defines custom properties, in particular, enforces custom HTML tables to be 100% width.
+
+
+plugins:
+ - autorefs # Need to reference mkdocstrings-generated content from documentation files ([][] thing).
+ - search
+ - mkdocstrings: # Automatically generate API documentation.
+ default_handler: python
+
+extra:
+ social:
+ # Buttons at the bottom of every page.
+ - icon: fontawesome/brands/github
+ link: https://github.com/HewlettPackard/cmf
+ - icon: fontawesome/brands/slack
+ link: https://commonmetadata.slack.com/
+
+
+# This defines the structure of the CMF documentation portal (all must be in `docs/` directory).
+nav:
+ - index.md
+ - Examples:
+ - Getting Started: examples/getting_started.md
+ - Architecture:
+ - Overview: architecture/overview.md
+ - Components: architecture/components.md
+ - Advantages: architecture/advantages.md
+ - Public API:
+ - CMF: api/public/cmf.md
+ - DataSlice: api/public/dataslice.md
diff --git a/presentations/LFAI_v2.pdf b/presentations/LFAI_v2.pdf
new file mode 100644
index 00000000..65c61fcb
Binary files /dev/null and b/presentations/LFAI_v2.pdf differ
diff --git a/presentations/LFAI_v2.pptx b/presentations/LFAI_v2.pptx
new file mode 100644
index 00000000..0337734f
Binary files /dev/null and b/presentations/LFAI_v2.pptx differ
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 00000000..d509b501
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,17 @@
+[project]
+name = "cmflib"
+version = "0.0.2"
+authors = [
+ { name="Hewlett Packard Enterprise"},
+]
+description = "Track metadata for AI pipeline"
+readme = "README.md"
+requires-python = ">=3.7,<3.10"
+classifiers = [
+ "Programming Language :: Python :: 3",
+ "Operating System :: POSIX :: Linux",
+]
+
+[project.urls]
+"Homepage" = "https://github.com/HewlettPackard/cmf"
+"Bug Tracker" = "https://github.com/HewlettPackard/cmf/issues"
diff --git a/setup.cfg b/setup.cfg
new file mode 100644
index 00000000..76b73e13
--- /dev/null
+++ b/setup.cfg
@@ -0,0 +1,3 @@
+[metadata]
+description-file=README.md
+license_files=LICENSE
diff --git a/setup.py b/setup.py
index e5d6071d..0116090a 100644
--- a/setup.py
+++ b/setup.py
@@ -1,6 +1,6 @@
from setuptools import setup, find_packages
-VERSION = '0.0.1'
+VERSION = '0.0.2'
DESCRIPTION = 'Metadata Python Package'
LONG_DESCRIPTION = 'Metadata framework storing AI metadata into MLMD'
@@ -9,13 +9,13 @@
# the name must match the folder name 'verysimplemodule'
name="cmflib",
version=VERSION,
- author="Annmary Justine",
- author_email="annmary.roy@hpe.com",
+ author="Hewlett Packard Enterprise",
description=DESCRIPTION,
long_description=LONG_DESCRIPTION,
packages=find_packages(),
- install_requires=["ml-metadata==1.3.0",
- "dvc","pandas","retrying", "pyarrow", "neo4j", "scikit-learn", "tabulate", "click"], # add any additional packages that
+ install_requires=["ml-metadata==1.11.0",
+ "dvc", "pandas", "retrying", "pyarrow", "neo4j", \
+ "scikit-learn", "tabulate", "click"], # add any additional packages that
# needs to be installed along with your package. Eg: 'caer'
keywords=['python', 'first package'],
@@ -23,6 +23,6 @@
"Development Status :: 3 - Alpha",
"Intended Audience :: Education",
"Programming Language :: Python :: 3",
- "Operating System :: Linux :: Ubuntu",
+ "Operating System :: POSIX :: Linux",
]
)
| \n", + " | id | \n", + "type | \n", + "uri | \n", + "name | \n", + "create_time_since_epoch | \n", + "last_update_time_since_epoch | \n", + "metrics_name | \n", + "avg_prec | \n", + "roc_auc | \n", + "
|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "7 | \n", + "Metrics | \n", + "878d492e-596c-11ed-99a3-b47af137252e | \n", + "metrics:878d492e-596c-11ed-99a3-b47af137252e:4 | \n", + "1667255818388 | \n", + "1667255818388 | \n", + "metrics:878d492e-596c-11ed-99a3-b47af137252e:4 | \n", + "0.526754 | \n", + "0.959238 | \n", + "