Skip to content

Latest commit

 

History

History
148 lines (101 loc) · 4.29 KB

README.md

File metadata and controls

148 lines (101 loc) · 4.29 KB
Raw

Mockingbird: Generate mock documents for data classification

About

Mockingbird is a Python library for generating mock documents in various formats. It accepts user-defined data, and embeds it into documents generated in many different formats. Developers can use Mockingbird to quickly generate datasets, with particular use for validating the efficacy of a data classification software.

Installation

The easiest way to install Mockingbird is by using pip:

pip install mockingbird

For local development, clone the repository and run pip install .

Getting Started

Mockingbird can run as a functional Python library or as a CLI.

CLI Usage

Once installed with pip, unix-like systems can use the command mockingbird_cli --h to access Mockingbird's command line interface. Some sample CLI calls are:

mockingbird_cli --type dry -o ./output/dry_test/
mockingbird_cli --type csv -i ./samples/csv_sample.csv -o ./output/csv/
mockingbird_cli --type csv_curl -i <curl'able URL> -o ./output/csv_curl/
mockingbird_cli --type mockaroo -i ./samples/sample_schema.json --mockaroo_api <mockaroo API> -o ./output/mockaroo

As a Python Library

Starting from Code

Mockingbird functions as a fully functional Python library. A basic example generating documents using mock-data is demonstrated below. In this example, key-value pairs are inserted as strings mapping to a list of strings.

from mockingbird import Mockingbird

# Spawn a new Mockingbird session
fab = Mockingbird()

# Set which file extensions to output
fab.set_file_extensions(["html", "docx", "yaml", "xlsx", "odt"])

# Input the data we want to test / inject into the documents
fab.add_sensitive_data(keyword="ssn", entries=["000-000-0000", "999-999-9999"])
fab.add_sensitive_data(keyword="dob", entries=["01/01/1991", "02/02/1992"])

# Generate and save the fabricated documents
fab.save(save_path="./output_basic/")
fab.dump_meta_data(output_file="./output_basic/meta_data.json")

Starting from CSV

Mockingbird can be started using a CSV file, treating the column headers as keywords, and the remaining rows as entries.

The CSV's are expected to be structured as the following,

FILE: mockingbird_data.csv

ssn, dob
000-000-000, 01/01/1991
999-999-999, 02/02/1992

from mockingbird.mb_wrappers import MockingbirdFromCSV


# This effectively loads files from the csv and generates a session using each column
fab = MockingbirdFromCSV("csv_sample.csv")
fab.set_all_extensions()

fab.save(save_path="./output_csv/")
fab.dump_meta_data(output_file="./output_csv/meta_data.json")

Optionally, multiple keywords can be defined in the CSV header file, which Mockingbird will split up into separate keywords. For example, rather than just testing the keyword ssn, we can test ssn and social security number. Multiple keywords can be defined in the CSV file by using ; as a delimiter.

For example,

FILE: mockingbird_data.csv

ssn;social security number,dob;date of birth;birth
000-000-000, 01/01/1991
999-999-999, 02/02/1992

This will generate documents for each keyword in each column header.

Starting Using Mockaroo

Using a Mockaroo API key, we can request mocked data using json requests from Mockaroo's servers. Currently, the request has to be saved to a json file on disk, and loaded during runtime. More documentation can be found at Mockaroo's Website, but below is a json-example.

FILE: mockaroo_request.json

[
  {
    "name": "ssn;social security;social",
    "type": "SSN"
  },
  {
    "name": "cc;credit card",
    "type": "Credit Card #"
  },
  {
    "name": "phone;phone-number;number",
    "type": "Phone"
  },
  {
    "name": "name;fullname;full name",
    "type": "Full Name"
  }
]

In code, Mockingbird can use this request as a json-payload,

import json
from mockingbird.mb_wrappers import MockingbirdFromMockaroo

with open("mockaroo_request.json") as json_file:
    schema_request = json.load(json_file)

fab = MockingbirdFromMockaroo(api_key="MOCKAROO_API_KEY", schema_request=schema_request)
fab.set_all_extensions()
fab.save(save_path="./output_mockaroo/")
fab.dump_meta_data(output_file="./output_mockaroo/meta_data.json")

License

Licensed under the Apache License, Version 2.0. See LICENSE for the full license text.