tutorials/test_data_quality_at_scale.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test data quality at scale with PyDeequ\n",
    "\n",
    "Authors: Calvin Wang (calviwan@), Chris Ghyzel (cghyzel@), Joan Aoanan (jaoanan@), Veronika Megler (meglerv@) "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You generally write unit tests for your code, but do you also test your data? Incoming data quality can make or break your machine learning application. Incorrect, missing or malformed data can have a large impact on production systems. Examples of data quality issues are:\n",
    "\n",
    "* Missing values can lead to failures in production system that require non-null values (NullPointerException).\n",
    "* Changes in the distribution of data can lead to unexpected outputs of machine learning models.\n",
    "* Aggregations of incorrect data can lead to wrong business decisions.\n",
    "\n",
    "In this blog post, we introduce PyDeequ, an open source Python wrapper over [Deequ](https://aws.amazon.com/blogs/big-data/test-data-quality-at-scale-with-deequ/) (an open source tool developed and used at Amazon).  While Deequ is written in Scala, PyDeequ allows you to use its  data quality and testing capabilities from Python and PySpark, the language of choice of many data scientists. PyDeequ democratizes and extends the power of Deequ by allowing you to use it alongside the many data science libraries that are available in that language. Furthermore, PyDeequ allows for fluid interface with [Pandas](https://pandas.pydata.org/) DataFrame as opposed to restricting within Spark DataFrames. \n",
    "\n",
    "Deequ allows you to calculate data quality metrics on your dataset, define and verify data quality constraints, and be informed about changes in the data distribution. Instead of implementing checks and verification algorithms on your own, you can focus on describing how your data should look. Deequ supports you by suggesting checks for you. Deequ is implemented on top of [Apache Spark](https://spark.apache.org/) and is designed to scale with large datasets (think billions of rows) that typically live in a distributed filesystem or a data warehouse. PyDeequ gives you access to this capability, but also allows you to use it from the familiar environment of your Python Jupyter notebook.\n",
    "\n",
    "## Deequ at Amazon \n",
    "\n",
    "Deequ is being used internally at Amazon for verifying the quality of many large production datasets. Dataset producers can add and edit data quality constraints. The system computes data quality metrics on a regular basis (with every new version of a dataset), verifies constraints defined by dataset producers, and publishes datasets to consumers in case of success. In error cases, dataset publication can be stopped, and producers are notified to take action. Data quality issues do not propagate to consumer data pipelines, reducing their blast radius. \n",
    "\n",
    "Deequ is also used within [Amazon SageMaker Model Monitor](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html#model-monitor-how-it-works). Now with the availability of PyDeequ, it is finding its way into a broader set of environments - SageMaker Notebooks, AWS Glue, and more.\n",
    "\n",
    "## Overview of PyDeequ\n",
    "\n",
    "Let’s look at PyDeequ’s main components, and how they relate to Deequ (shown in Figure 1). \n",
    "\n",
    "* Metrics Computation — Deequ computes data quality metrics, that is, statistics such as completeness, maximum, or correlation. Deequ uses Spark to read from sources such as Amazon S3, and to compute metrics through an optimized set of aggregation queries. You have direct access to the raw metrics computed on the data.\n",
    "* Constraint Verification — As a user, you focus on defining a set of data quality constraints to be verified. Deequ takes care of deriving the required set of metrics to be computed on the data. Deequ generates a data quality report, which contains the result of the constraint verification.\n",
    "* Constraint Suggestion — You can choose to define your own custom data quality constraints, or use the automated constraint suggestion methods that profile the data to infer useful constraints.\n",
    "* Python wrappers — You can call each of the Deequ functions using Python syntax. The wrappers translate the commands to the underlying Deequ calls, and return their response.\n",
    "\n",
    "![image.png](../imgs/pydeequ_architecture.png)\n",
    "\n",
    "Figure 1. Overview of PyDeequ components. \n",
    "\n",
    "## Example \n",
    "\n",
    "As a running example, we use [a customer review dataset provided by Amazon](https://s3.amazonaws.com/amazon-reviews-pds/readme.html) on Amazon S3. We have intentionally followed the example in the [Deequ blog](https://aws.amazon.com/blogs/big-data/test-data-quality-at-scale-with-deequ/), to show the similarity in functionality and execution. We begin the way many data science projects do: with initial data exploration and assessment in a Jupyter notebook. \n",
    "\n",
    "During the data exploration phase, you’d like to easily answer some basic questions about the data: \n",
    "\n",
    "* Are the fields that are supposed to contain unique values, really unique? Are there fields that are missing values? \n",
    "* How many distinct categories are there in the categorical fields?\n",
    "* Are there correlations between some key features?\n",
    "* If there are two supposedly similar datasets (different categories, or different time periods, say), are they really similar?\n",
    "\n",
    "Then, we’ll show you how to scale this approach to large-scale datasets, using the same code on an EMR cluster. This is how you’d likely do your ML training, and later as you move into a production setting.\n",
    "\n",
    "### Setup: Start a PySpark Session in a SageMaker Notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash \n",
    "\n",
    "# install PyDeequ via pip \n",
    "pip install pydeequ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession, Row, DataFrame\n",
    "import json\n",
    "import pandas as pd\n",
    "import sagemaker_pyspark\n",
    "\n",
    "import pydeequ\n",
    "\n",
    "classpath = \":\".join(sagemaker_pyspark.classpath_jars())\n",
    "\n",
    "spark = (SparkSession\n",
    "    .builder\n",
    "    .config(\"spark.driver.extraClassPath\", classpath)\n",
    "    .config(\"spark.jars.packages\", pydeequ.deequ_maven_coord)\n",
    "    .config(\"spark.jars.excludes\", pydeequ.f2j_maven_coord)\n",
    "    .getOrCreate())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We will be using the Amazon Product Reviews dataset -- specifically the Electronics subset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- marketplace: string (nullable = true)\n",
      " |-- customer_id: string (nullable = true)\n",
      " |-- review_id: string (nullable = true)\n",
      " |-- product_id: string (nullable = true)\n",
      " |-- product_parent: string (nullable = true)\n",
      " |-- product_title: string (nullable = true)\n",
      " |-- star_rating: integer (nullable = true)\n",
      " |-- helpful_votes: integer (nullable = true)\n",
      " |-- total_votes: integer (nullable = true)\n",
      " |-- vine: string (nullable = true)\n",
      " |-- verified_purchase: string (nullable = true)\n",
      " |-- review_headline: string (nullable = true)\n",
      " |-- review_body: string (nullable = true)\n",
      " |-- review_date: date (nullable = true)\n",
      " |-- year: integer (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df = spark.read.parquet(\"s3a://amazon-reviews-pds/parquet/product_category=Electronics/\")\n",
    "\n",
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Analysis \n",
    "\n",
    "Before we define checks on the data, we want to calculate some statistics on the dataset; we call them metrics. As with Deequ, PyDeequ supports a rich set of metrics (they are described in this blog (https://aws.amazon.com/blogs/big-data/test-data-quality-at-scale-with-deequ/) and in this Deequ package (https://github.com/awslabs/deequ/tree/master/src/main/scala/com/amazon/deequ/analyzers)). In the following example, we show how to use the _AnalysisRunner (https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/analyzers/runners/AnalysisRunner.scala)_ to capture the metrics you are interested in. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------+--------------------+-------------------+--------------------+\n",
      "|     entity|            instance|               name|               value|\n",
      "+-----------+--------------------+-------------------+--------------------+\n",
      "|     Column|           review_id|       Completeness|                 1.0|\n",
      "|     Column|           review_id|ApproxCountDistinct|           3010972.0|\n",
      "|Mutlicolumn|total_votes,star_...|        Correlation|-0.03451097996538765|\n",
      "|    Dataset|                   *|               Size|           3120938.0|\n",
      "|     Column|         star_rating|               Mean|   4.036143941340712|\n",
      "|     Column|     top star_rating|         Compliance|  0.7494070692849394|\n",
      "|Mutlicolumn|total_votes,helpf...|        Correlation|  0.9936463809903863|\n",
      "+-----------+--------------------+-------------------+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pydeequ.analyzers import *\n",
    "\n",
    "analysisResult = AnalysisRunner(spark) \\\n",
    "                    .onData(df) \\\n",
    "                    .addAnalyzer(Size()) \\\n",
    "                    .addAnalyzer(Completeness(\"review_id\")) \\\n",
    "                    .addAnalyzer(ApproxCountDistinct(\"review_id\")) \\\n",
    "                    .addAnalyzer(Mean(\"star_rating\")) \\\n",
    "                    .addAnalyzer(Compliance(\"top star_rating\", \"star_rating >= 4.0\")) \\\n",
    "                    .addAnalyzer(Correlation(\"total_votes\", \"star_rating\")) \\\n",
    "                    .addAnalyzer(Correlation(\"total_votes\", \"helpful_votes\")) \\\n",
    "                    .run()\n",
    "                    \n",
    "analysisResult_df = AnalyzerContext.successMetricsAsDataFrame(spark, analysisResult)\n",
    "analysisResult_df.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### You can also get that result in a Pandas Dataframe!\n",
    "\n",
    "Passing `pandas=True` in any call for getting metrics as DataFrames will return the dataframe in Pandas form! We'll see more of it down the line! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>entity</th>\n",
       "      <th>instance</th>\n",
       "      <th>name</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Column</td>\n",
       "      <td>review_id</td>\n",
       "      <td>Completeness</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Column</td>\n",
       "      <td>review_id</td>\n",
       "      <td>ApproxCountDistinct</td>\n",
       "      <td>3.010972e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Mutlicolumn</td>\n",
       "      <td>total_votes,star_rating</td>\n",
       "      <td>Correlation</td>\n",
       "      <td>-3.451098e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Dataset</td>\n",
       "      <td>*</td>\n",
       "      <td>Size</td>\n",
       "      <td>3.120938e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Column</td>\n",
       "      <td>star_rating</td>\n",
       "      <td>Mean</td>\n",
       "      <td>4.036144e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Column</td>\n",
       "      <td>top star_rating</td>\n",
       "      <td>Compliance</td>\n",
       "      <td>7.494071e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Mutlicolumn</td>\n",
       "      <td>total_votes,helpful_votes</td>\n",
       "      <td>Correlation</td>\n",
       "      <td>9.936464e-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        entity                   instance                 name         value\n",
       "0       Column                  review_id         Completeness  1.000000e+00\n",
       "1       Column                  review_id  ApproxCountDistinct  3.010972e+06\n",
       "2  Mutlicolumn    total_votes,star_rating          Correlation -3.451098e-02\n",
       "3      Dataset                          *                 Size  3.120938e+06\n",
       "4       Column                star_rating                 Mean  4.036144e+00\n",
       "5       Column            top star_rating           Compliance  7.494071e-01\n",
       "6  Mutlicolumn  total_votes,helpful_votes          Correlation  9.936464e-01"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analysisResult_pd_df = AnalyzerContext.successMetricsAsDataFrame(spark, analysisResult, pandas=True)\n",
    "analysisResult_pd_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this, we learn that: \n",
    "\n",
    "* review_id has no missing values and approximately 3,010,972 unique values. \n",
    "* 74.9% of reviews have a star_rating of 4 or higher \n",
    "* total_votes and star_rating are not correlated. \n",
    "* helpful_votes and total_votes are strongly correlated \n",
    "* the average star_rating is 4.0 \n",
    "* The dataset contains 3,120,938 reviews. \n",
    "\n",
    "## Define and Run Tests for Data\n",
    "\n",
    "After analyzing and understanding the data, we want to verify that the properties we have derived also hold for new versions of the dataset. By defining assertions on the data distribution as part of a data pipeline, we can ensure that every processed dataset is of high quality, and that any application consuming the data can rely on it.\n",
    "\n",
    "For writing tests on data, we start with the _VerificationSuite (https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/VerificationSuite.scala)_ and add _Checks (https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/checks/Check.scala)_ on attributes of the data. In this example, we test for the following properties of our data:\n",
    "\n",
    "* There are at least 3 million rows in total. \n",
    "* review_id is never NULL.\n",
    "* review_id is unique. \n",
    "* star_rating has a minimum of 1.0 and maximum of 5.0. \n",
    "* marketplace only contains “US”, “UK”, “DE”, “JP”, or “FR”.\n",
    "* year does not contain negative values. \n",
    "\n",
    "This is the code that reflects the previous statements. For information about all available checks, see _this GitHub repository (https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/checks/Check.scala)_. You can run this directly in the Spark shell as previously explained:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python Callback server started!\n",
      "Verification Run Status: Warning\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>check</th>\n",
       "      <th>check_level</th>\n",
       "      <th>check_status</th>\n",
       "      <th>constraint</th>\n",
       "      <th>constraint_status</th>\n",
       "      <th>constraint_message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Amazon Electronic Products Reviews</td>\n",
       "      <td>Warning</td>\n",
       "      <td>Warning</td>\n",
       "      <td>SizeConstraint(Size(None))</td>\n",
       "      <td>Success</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Amazon Electronic Products Reviews</td>\n",
       "      <td>Warning</td>\n",
       "      <td>Warning</td>\n",
       "      <td>MinimumConstraint(Minimum(star_rating,None))</td>\n",
       "      <td>Success</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Amazon Electronic Products Reviews</td>\n",
       "      <td>Warning</td>\n",
       "      <td>Warning</td>\n",
       "      <td>MaximumConstraint(Maximum(star_rating,None))</td>\n",
       "      <td>Success</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Amazon Electronic Products Reviews</td>\n",
       "      <td>Warning</td>\n",
       "      <td>Warning</td>\n",
       "      <td>CompletenessConstraint(Completeness(review_id,...</td>\n",
       "      <td>Success</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Amazon Electronic Products Reviews</td>\n",
       "      <td>Warning</td>\n",
       "      <td>Warning</td>\n",
       "      <td>UniquenessConstraint(Uniqueness(List(review_id...</td>\n",
       "      <td>Failure</td>\n",
       "      <td>Value: 0.9926566948782706 does not meet the co...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Amazon Electronic Products Reviews</td>\n",
       "      <td>Warning</td>\n",
       "      <td>Warning</td>\n",
       "      <td>CompletenessConstraint(Completeness(marketplac...</td>\n",
       "      <td>Success</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Amazon Electronic Products Reviews</td>\n",
       "      <td>Warning</td>\n",
       "      <td>Warning</td>\n",
       "      <td>ComplianceConstraint(Compliance(marketplace co...</td>\n",
       "      <td>Success</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Amazon Electronic Products Reviews</td>\n",
       "      <td>Warning</td>\n",
       "      <td>Warning</td>\n",
       "      <td>ComplianceConstraint(Compliance(year is non-ne...</td>\n",
       "      <td>Success</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                check check_level check_status  \\\n",
       "0  Amazon Electronic Products Reviews     Warning      Warning   \n",
       "1  Amazon Electronic Products Reviews     Warning      Warning   \n",
       "2  Amazon Electronic Products Reviews     Warning      Warning   \n",
       "3  Amazon Electronic Products Reviews     Warning      Warning   \n",
       "4  Amazon Electronic Products Reviews     Warning      Warning   \n",
       "5  Amazon Electronic Products Reviews     Warning      Warning   \n",
       "6  Amazon Electronic Products Reviews     Warning      Warning   \n",
       "7  Amazon Electronic Products Reviews     Warning      Warning   \n",
       "\n",
       "                                          constraint constraint_status  \\\n",
       "0                         SizeConstraint(Size(None))           Success   \n",
       "1       MinimumConstraint(Minimum(star_rating,None))           Success   \n",
       "2       MaximumConstraint(Maximum(star_rating,None))           Success   \n",
       "3  CompletenessConstraint(Completeness(review_id,...           Success   \n",
       "4  UniquenessConstraint(Uniqueness(List(review_id...           Failure   \n",
       "5  CompletenessConstraint(Completeness(marketplac...           Success   \n",
       "6  ComplianceConstraint(Compliance(marketplace co...           Success   \n",
       "7  ComplianceConstraint(Compliance(year is non-ne...           Success   \n",
       "\n",
       "                                  constraint_message  \n",
       "0                                                     \n",
       "1                                                     \n",
       "2                                                     \n",
       "3                                                     \n",
       "4  Value: 0.9926566948782706 does not meet the co...  \n",
       "5                                                     \n",
       "6                                                     \n",
       "7                                                     "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pydeequ.checks import *\n",
    "from pydeequ.verification import *\n",
    "\n",
    "check = Check(spark, CheckLevel.Warning, \"Amazon Electronic Products Reviews\")\n",
    "\n",
    "checkResult = VerificationSuite(spark) \\\n",
    "    .onData(df) \\\n",
    "    .addCheck(\n",
    "        check.hasSize(lambda x: x >= 3000000) \\\n",
    "        .hasMin(\"star_rating\", lambda x: x == 1.0) \\\n",
    "        .hasMax(\"star_rating\", lambda x: x == 5.0)  \\\n",
    "        .isComplete(\"review_id\")  \\\n",
    "        .isUnique(\"review_id\")  \\\n",
    "        .isComplete(\"marketplace\")  \\\n",
    "        .isContainedIn(\"marketplace\", [\"US\", \"UK\", \"DE\", \"JP\", \"FR\"]) \\\n",
    "        .isNonNegative(\"year\")) \\\n",
    "    .run()\n",
    "\n",
    "print(f\"Verification Run Status: {checkResult.status}\")\n",
    "checkResult_df = VerificationResult.checkResultsAsDataFrame(spark, checkResult, pandas=True)\n",
    "checkResult_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After calling run(), PyDeequ translates your test description into Deequ, which in its turn translates it into a series of Spark jobs which are executed to compute metrics on the data. Afterwards, it invokes your assertion functions (e.g., lambda x: x == 1.0 for the minimum star-rating check) on these metrics to see if the constraints hold on the data. \n",
    "\n",
    "Interestingly, the review_id column is not unique, which resulted in a failure of the check on uniqueness. We can also look at all the metrics that Deequ computed for this check by running: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>entity</th>\n",
       "      <th>instance</th>\n",
       "      <th>name</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Column</td>\n",
       "      <td>review_id</td>\n",
       "      <td>Completeness</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Column</td>\n",
       "      <td>review_id</td>\n",
       "      <td>Uniqueness</td>\n",
       "      <td>9.926567e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Dataset</td>\n",
       "      <td>*</td>\n",
       "      <td>Size</td>\n",
       "      <td>3.120938e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Column</td>\n",
       "      <td>star_rating</td>\n",
       "      <td>Maximum</td>\n",
       "      <td>5.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Column</td>\n",
       "      <td>star_rating</td>\n",
       "      <td>Minimum</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Column</td>\n",
       "      <td>year is non-negative</td>\n",
       "      <td>Compliance</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Column</td>\n",
       "      <td>marketplace contained in US,UK,DE,JP,FR</td>\n",
       "      <td>Compliance</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Column</td>\n",
       "      <td>marketplace</td>\n",
       "      <td>Completeness</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    entity                                 instance          name  \\\n",
       "0   Column                                review_id  Completeness   \n",
       "1   Column                                review_id    Uniqueness   \n",
       "2  Dataset                                        *          Size   \n",
       "3   Column                              star_rating       Maximum   \n",
       "4   Column                              star_rating       Minimum   \n",
       "5   Column                     year is non-negative    Compliance   \n",
       "6   Column  marketplace contained in US,UK,DE,JP,FR    Compliance   \n",
       "7   Column                              marketplace  Completeness   \n",
       "\n",
       "          value  \n",
       "0  1.000000e+00  \n",
       "1  9.926567e-01  \n",
       "2  3.120938e+06  \n",
       "3  5.000000e+00  \n",
       "4  1.000000e+00  \n",
       "5  1.000000e+00  \n",
       "6  1.000000e+00  \n",
       "7  1.000000e+00  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "checkResult_df = VerificationResult.successMetricsAsDataFrame(spark, checkResult, pandas=True)\n",
    "checkResult_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated Constraint Suggestion \n",
    "\n",
    "If you own a large number of datasets or if your dataset has many columns, it may be challenging for you to manually define appropriate constraints. Deequ can automatically suggest useful constraints based on the data distribution. Deequ first runs a data profiling method and then applies a set of rules on the result. For more information about how to run a data profiling method, see _this GitHub repository. (https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/examples/data_profiling_example.md)_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"constraint_suggestions\": [\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(review_id,None))\",\n",
      "      \"column_name\": \"review_id\",\n",
      "      \"current_value\": \"Completeness: 1.0\",\n",
      "      \"description\": \"'review_id' is not null\",\n",
      "      \"suggesting_rule\": \"CompleteIfCompleteRule()\",\n",
      "      \"rule_description\": \"If a column is complete in the sample, we suggest a NOT NULL constraint\",\n",
      "      \"code_for_constraint\": \".isComplete(\\\"review_id\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"UniquenessConstraint(Uniqueness(List(review_id),None))\",\n",
      "      \"column_name\": \"review_id\",\n",
      "      \"current_value\": \"ApproxDistinctness: 0.9647650802419017\",\n",
      "      \"description\": \"'review_id' is unique\",\n",
      "      \"suggesting_rule\": \"UniqueIfApproximatelyUniqueRule()\",\n",
      "      \"rule_description\": \"If the ratio of approximate num distinct values in a column is close to the number of records (within the error of the HLL sketch), we suggest a UNIQUE constraint\",\n",
      "      \"code_for_constraint\": \".isUnique(\\\"review_id\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(customer_id,None))\",\n",
      "      \"column_name\": \"customer_id\",\n",
      "      \"current_value\": \"Completeness: 1.0\",\n",
      "      \"description\": \"'customer_id' is not null\",\n",
      "      \"suggesting_rule\": \"CompleteIfCompleteRule()\",\n",
      "      \"rule_description\": \"If a column is complete in the sample, we suggest a NOT NULL constraint\",\n",
      "      \"code_for_constraint\": \".isComplete(\\\"customer_id\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"ComplianceConstraint(Compliance('customer_id' has no negative values,customer_id >= 0,None))\",\n",
      "      \"column_name\": \"customer_id\",\n",
      "      \"current_value\": \"Minimum: 10005.0\",\n",
      "      \"description\": \"'customer_id' has no negative values\",\n",
      "      \"suggesting_rule\": \"NonNegativeNumbersRule()\",\n",
      "      \"rule_description\": \"If we see only non-negative numbers in a column, we suggest a corresponding constraint\",\n",
      "      \"code_for_constraint\": \".isNonNegative(\\\"customer_id\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"AnalysisBasedConstraint(DataType(customer_id,None),<function1>,Some(<function1>),None)\",\n",
      "      \"column_name\": \"customer_id\",\n",
      "      \"current_value\": \"DataType: Integral\",\n",
      "      \"description\": \"'customer_id' has type Integral\",\n",
      "      \"suggesting_rule\": \"RetainTypeRule()\",\n",
      "      \"rule_description\": \"If we detect a non-string type, we suggest a type constraint\",\n",
      "      \"code_for_constraint\": \".hasDataType(\\\"customer_id\\\", ConstrainableDataTypes.Integral)\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(review_date,None))\",\n",
      "      \"column_name\": \"review_date\",\n",
      "      \"current_value\": \"Completeness: 1.0\",\n",
      "      \"description\": \"'review_date' is not null\",\n",
      "      \"suggesting_rule\": \"CompleteIfCompleteRule()\",\n",
      "      \"rule_description\": \"If a column is complete in the sample, we suggest a NOT NULL constraint\",\n",
      "      \"code_for_constraint\": \".isComplete(\\\"review_date\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(helpful_votes,None))\",\n",
      "      \"column_name\": \"helpful_votes\",\n",
      "      \"current_value\": \"Completeness: 1.0\",\n",
      "      \"description\": \"'helpful_votes' is not null\",\n",
      "      \"suggesting_rule\": \"CompleteIfCompleteRule()\",\n",
      "      \"rule_description\": \"If a column is complete in the sample, we suggest a NOT NULL constraint\",\n",
      "      \"code_for_constraint\": \".isComplete(\\\"helpful_votes\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"ComplianceConstraint(Compliance('helpful_votes' has no negative values,helpful_votes >= 0,None))\",\n",
      "      \"column_name\": \"helpful_votes\",\n",
      "      \"current_value\": \"Minimum: 0.0\",\n",
      "      \"description\": \"'helpful_votes' has no negative values\",\n",
      "      \"suggesting_rule\": \"NonNegativeNumbersRule()\",\n",
      "      \"rule_description\": \"If we see only non-negative numbers in a column, we suggest a corresponding constraint\",\n",
      "      \"code_for_constraint\": \".isNonNegative(\\\"helpful_votes\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(star_rating,None))\",\n",
      "      \"column_name\": \"star_rating\",\n",
      "      \"current_value\": \"Completeness: 1.0\",\n",
      "      \"description\": \"'star_rating' is not null\",\n",
      "      \"suggesting_rule\": \"CompleteIfCompleteRule()\",\n",
      "      \"rule_description\": \"If a column is complete in the sample, we suggest a NOT NULL constraint\",\n",
      "      \"code_for_constraint\": \".isComplete(\\\"star_rating\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"ComplianceConstraint(Compliance('star_rating' has no negative values,star_rating >= 0,None))\",\n",
      "      \"column_name\": \"star_rating\",\n",
      "      \"current_value\": \"Minimum: 1.0\",\n",
      "      \"description\": \"'star_rating' has no negative values\",\n",
      "      \"suggesting_rule\": \"NonNegativeNumbersRule()\",\n",
      "      \"rule_description\": \"If we see only non-negative numbers in a column, we suggest a corresponding constraint\",\n",
      "      \"code_for_constraint\": \".isNonNegative(\\\"star_rating\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(year,None))\",\n",
      "      \"column_name\": \"year\",\n",
      "      \"current_value\": \"Completeness: 1.0\",\n",
      "      \"description\": \"'year' is not null\",\n",
      "      \"suggesting_rule\": \"CompleteIfCompleteRule()\",\n",
      "      \"rule_description\": \"If a column is complete in the sample, we suggest a NOT NULL constraint\",\n",
      "      \"code_for_constraint\": \".isComplete(\\\"year\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"ComplianceConstraint(Compliance('year' has no negative values,year >= 0,None))\",\n",
      "      \"column_name\": \"year\",\n",
      "      \"current_value\": \"Minimum: 1999.0\",\n",
      "      \"description\": \"'year' has no negative values\",\n",
      "      \"suggesting_rule\": \"NonNegativeNumbersRule()\",\n",
      "      \"rule_description\": \"If we see only non-negative numbers in a column, we suggest a corresponding constraint\",\n",
      "      \"code_for_constraint\": \".isNonNegative(\\\"year\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(product_title,None))\",\n",
      "      \"column_name\": \"product_title\",\n",
      "      \"current_value\": \"Completeness: 1.0\",\n",
      "      \"description\": \"'product_title' is not null\",\n",
      "      \"suggesting_rule\": \"CompleteIfCompleteRule()\",\n",
      "      \"rule_description\": \"If a column is complete in the sample, we suggest a NOT NULL constraint\",\n",
      "      \"code_for_constraint\": \".isComplete(\\\"product_title\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(review_headline,None))\",\n",
      "      \"column_name\": \"review_headline\",\n",
      "      \"current_value\": \"Completeness: 0.9999987183340393\",\n",
      "      \"description\": \"'review_headline' has less than 1% missing values\",\n",
      "      \"suggesting_rule\": \"RetainCompletenessRule()\",\n",
      "      \"rule_description\": \"If a column is incomplete in the sample, we model its completeness as a binomial variable, estimate a confidence interval and use this to define a lower bound for the completeness\",\n",
      "      \"code_for_constraint\": \".hasCompleteness(\\\"review_headline\\\", lambda x: x >= 0.99, \\\"It should be above 0.99!\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(product_id,None))\",\n",
      "      \"column_name\": \"product_id\",\n",
      "      \"current_value\": \"Completeness: 1.0\",\n",
      "      \"description\": \"'product_id' is not null\",\n",
      "      \"suggesting_rule\": \"CompleteIfCompleteRule()\",\n",
      "      \"rule_description\": \"If a column is complete in the sample, we suggest a NOT NULL constraint\",\n",
      "      \"code_for_constraint\": \".isComplete(\\\"product_id\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(total_votes,None))\",\n",
      "      \"column_name\": \"total_votes\",\n",
      "      \"current_value\": \"Completeness: 1.0\",\n",
      "      \"description\": \"'total_votes' is not null\",\n",
      "      \"suggesting_rule\": \"CompleteIfCompleteRule()\",\n",
      "      \"rule_description\": \"If a column is complete in the sample, we suggest a NOT NULL constraint\",\n",
      "      \"code_for_constraint\": \".isComplete(\\\"total_votes\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"ComplianceConstraint(Compliance('total_votes' has no negative values,total_votes >= 0,None))\",\n",
      "      \"column_name\": \"total_votes\",\n",
      "      \"current_value\": \"Minimum: 0.0\",\n",
      "      \"description\": \"'total_votes' has no negative values\",\n",
      "      \"suggesting_rule\": \"NonNegativeNumbersRule()\",\n",
      "      \"rule_description\": \"If we see only non-negative numbers in a column, we suggest a corresponding constraint\",\n",
      "      \"code_for_constraint\": \".isNonNegative(\\\"total_votes\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(product_parent,None))\",\n",
      "      \"column_name\": \"product_parent\",\n",
      "      \"current_value\": \"Completeness: 1.0\",\n",
      "      \"description\": \"'product_parent' is not null\",\n",
      "      \"suggesting_rule\": \"CompleteIfCompleteRule()\",\n",
      "      \"rule_description\": \"If a column is complete in the sample, we suggest a NOT NULL constraint\",\n",
      "      \"code_for_constraint\": \".isComplete(\\\"product_parent\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"ComplianceConstraint(Compliance('product_parent' has no negative values,product_parent >= 0,None))\",\n",
      "      \"column_name\": \"product_parent\",\n",
      "      \"current_value\": \"Minimum: 6478.0\",\n",
      "      \"description\": \"'product_parent' has no negative values\",\n",
      "      \"suggesting_rule\": \"NonNegativeNumbersRule()\",\n",
      "      \"rule_description\": \"If we see only non-negative numbers in a column, we suggest a corresponding constraint\",\n",
      "      \"code_for_constraint\": \".isNonNegative(\\\"product_parent\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"AnalysisBasedConstraint(DataType(product_parent,None),<function1>,Some(<function1>),None)\",\n",
      "      \"column_name\": \"product_parent\",\n",
      "      \"current_value\": \"DataType: Integral\",\n",
      "      \"description\": \"'product_parent' has type Integral\",\n",
      "      \"suggesting_rule\": \"RetainTypeRule()\",\n",
      "      \"rule_description\": \"If we detect a non-string type, we suggest a type constraint\",\n",
      "      \"code_for_constraint\": \".hasDataType(\\\"product_parent\\\", ConstrainableDataTypes.Integral)\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(review_body,None))\",\n",
      "      \"column_name\": \"review_body\",\n",
      "      \"current_value\": \"Completeness: 0.9999724441818453\",\n",
      "      \"description\": \"'review_body' has less than 1% missing values\",\n",
      "      \"suggesting_rule\": \"RetainCompletenessRule()\",\n",
      "      \"rule_description\": \"If a column is incomplete in the sample, we model its completeness as a binomial variable, estimate a confidence interval and use this to define a lower bound for the completeness\",\n",
      "      \"code_for_constraint\": \".hasCompleteness(\\\"review_body\\\", lambda x: x >= 0.99, \\\"It should be above 0.99!\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"ComplianceConstraint(Compliance('vine' has value range 'N', 'Y',`vine` IN ('N', 'Y'),None))\",\n",
      "      \"column_name\": \"vine\",\n",
      "      \"current_value\": \"Compliance: 1\",\n",
      "      \"description\": \"'vine' has value range 'N', 'Y'\",\n",
      "      \"suggesting_rule\": \"CategoricalRangeRule()\",\n",
      "      \"rule_description\": \"If we see a categorical range for a column, we suggest an IS IN (...) constraint\",\n",
      "      \"code_for_constraint\": \".isContainedIn(\\\"vine\\\", [\\\"N\\\", \\\"Y\\\"])\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(vine,None))\",\n",
      "      \"column_name\": \"vine\",\n",
      "      \"current_value\": \"Completeness: 1.0\",\n",
      "      \"description\": \"'vine' is not null\",\n",
      "      \"suggesting_rule\": \"CompleteIfCompleteRule()\",\n",
      "      \"rule_description\": \"If a column is complete in the sample, we suggest a NOT NULL constraint\",\n",
      "      \"code_for_constraint\": \".isComplete(\\\"vine\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"ComplianceConstraint(Compliance('vine' has value range 'N' for at least 99.0% of values,`vine` IN ('N'),None))\",\n",
      "      \"column_name\": \"vine\",\n",
      "      \"current_value\": \"Compliance: 0.9939271462617969\",\n",
      "      \"description\": \"'vine' has value range 'N' for at least 99.0% of values\",\n",
      "      \"suggesting_rule\": \"FractionalCategoricalRangeRule(0.9)\",\n",
      "      \"rule_description\": \"If we see a categorical range for most values in a column, we suggest an IS IN (...) constraint that should hold for most values\",\n",
      "      \"code_for_constraint\": \".isContainedIn(\\\"vine\\\", [\\\"N\\\"], lambda x: x >= 0.99, \\\"It should be above 0.99!\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"ComplianceConstraint(Compliance('marketplace' has value range 'US', 'UK', 'DE', 'JP', 'FR',`marketplace` IN ('US', 'UK', 'DE', 'JP', 'FR'),None))\",\n",
      "      \"column_name\": \"marketplace\",\n",
      "      \"current_value\": \"Compliance: 1\",\n",
      "      \"description\": \"'marketplace' has value range 'US', 'UK', 'DE', 'JP', 'FR'\",\n",
      "      \"suggesting_rule\": \"CategoricalRangeRule()\",\n",
      "      \"rule_description\": \"If we see a categorical range for a column, we suggest an IS IN (...) constraint\",\n",
      "      \"code_for_constraint\": \".isContainedIn(\\\"marketplace\\\", [\\\"US\\\", \\\"UK\\\", \\\"DE\\\", \\\"JP\\\", \\\"FR\\\"])\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(marketplace,None))\",\n",
      "      \"column_name\": \"marketplace\",\n",
      "      \"current_value\": \"Completeness: 1.0\",\n",
      "      \"description\": \"'marketplace' is not null\",\n",
      "      \"suggesting_rule\": \"CompleteIfCompleteRule()\",\n",
      "      \"rule_description\": \"If a column is complete in the sample, we suggest a NOT NULL constraint\",\n",
      "      \"code_for_constraint\": \".isComplete(\\\"marketplace\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"ComplianceConstraint(Compliance('marketplace' has value range 'US' for at least 99.0% of values,`marketplace` IN ('US'),None))\",\n",
      "      \"column_name\": \"marketplace\",\n",
      "      \"current_value\": \"Compliance: 0.9949982985884372\",\n",
      "      \"description\": \"'marketplace' has value range 'US' for at least 99.0% of values\",\n",
      "      \"suggesting_rule\": \"FractionalCategoricalRangeRule(0.9)\",\n",
      "      \"rule_description\": \"If we see a categorical range for most values in a column, we suggest an IS IN (...) constraint that should hold for most values\",\n",
      "      \"code_for_constraint\": \".isContainedIn(\\\"marketplace\\\", [\\\"US\\\"], lambda x: x >= 0.99, \\\"It should be above 0.99!\\\")\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"ComplianceConstraint(Compliance('verified_purchase' has value range 'Y', 'N',`verified_purchase` IN ('Y', 'N'),None))\",\n",
      "      \"column_name\": \"verified_purchase\",\n",
      "      \"current_value\": \"Compliance: 1\",\n",
      "      \"description\": \"'verified_purchase' has value range 'Y', 'N'\",\n",
      "      \"suggesting_rule\": \"CategoricalRangeRule()\",\n",
      "      \"rule_description\": \"If we see a categorical range for a column, we suggest an IS IN (...) constraint\",\n",
      "      \"code_for_constraint\": \".isContainedIn(\\\"verified_purchase\\\", [\\\"Y\\\", \\\"N\\\"])\"\n",
      "    },\n",
      "    {\n",
      "      \"constraint_name\": \"CompletenessConstraint(Completeness(verified_purchase,None))\",\n",
      "      \"column_name\": \"verified_purchase\",\n",
      "      \"current_value\": \"Completeness: 1.0\",\n",
      "      \"description\": \"'verified_purchase' is not null\",\n",
      "      \"suggesting_rule\": \"CompleteIfCompleteRule()\",\n",
      "      \"rule_description\": \"If a column is complete in the sample, we suggest a NOT NULL constraint\",\n",
      "      \"code_for_constraint\": \".isComplete(\\\"verified_purchase\\\")\"\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "from pydeequ.suggestions import *\n",
    "\n",
    "suggestionResult = ConstraintSuggestionRunner(spark) \\\n",
    "             .onData(df) \\\n",
    "             .addConstraintRule(DEFAULT()) \\\n",
    "             .run()\n",
    "\n",
    "# Constraint Suggestions in JSON format\n",
    "print(json.dumps(suggestionResult, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above result contains a list of constraints with descriptions and Python code, so that you can directly apply it in your data quality checks."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scaling to Production \n",
    "\n",
    "So far we’ve shown you how to use these capabilities in the context of data exploration using a Jupyter notebook running on a SageMaker Notebook instance. As your project matures, you’ll want to use the same capabilities on larger and larger datasets, and in a production environment. With PyDeequ, it’s easy to make that transition.\n",
    "\n",
    "![image.png](../imgs/pydeequ_deployment.png)\n",
    "\n",
    "As seen in the diagram above, you can leverage both an AWS EMR cluster and/or AWS Glue for larger or production purposes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More Examples on GitHub\n",
    "\n",
    "You can find examples of more advanced features at _Deequ’s GitHub page (https://github.com/awslabs/deequ)_:\n",
    "\n",
    "* Deequ not only provides data quality checks with fixed thresholds. Learn how to use _anomaly detection on data quality metrics (https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/examples/anomaly_detection_example.md)_ to apply tests on metrics that change over time.\n",
    "* Deequ offers support for storing and loading metrics. Learn how to use the _MetricsRepository (https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/examples/metrics_repository_example.md)_ for this use case.\n",
    "* If your dataset grows over time or is partitioned, you can use Deequ’s _incremental metrics computation (https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/examples/algebraic_states_example.md)_ capability. For each partition, Deequ stores a state for each computed metric. To compute metrics for the union of partitions, Deequ can use these states to efficiently derive overall metrics without reloading the data.\n",
    "\n",
    "## Additional Resources\n",
    "\n",
    "Learn more about the inner workings of Deequ in the VLDB 2018 paper “_Automating large-scale data quality verification. (http://www.vldb.org/pvldb/vol11/p1781-schelter.pdf)_”\n",
    "\n",
    "## Conclusion\n",
    "\n",
    "This blog post showed you how to use PyDeequ for calculating data quality metrics, verifying data quality metrics, and profiling data to automate the configuration of data quality checks. PyDeequ is available for you now to build your own data quality management pipeline.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}