tutorials/constraint_suggestion_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automatic Suggestion of Constraints\n",
    "\n",
    "In our experience, a major hurdle in data validation is that someone needs to come up with the actual constraints to apply on the data. This can be very difficult for large, real-world datasets, especially if they are very complex and contain information from a lot of different sources. We build so-called constraint suggestion functionality into deequ to assist users in finding reasonable constraints for their data.\n",
    "\n",
    "Our constraint suggestion first [profiles the data](./data_profiling_example.ipynb) and then applies a set of heuristic rules to suggest constraints. In the following, we give a concrete example on how to have constraints suggested for your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession, Row, DataFrame\n",
    "import json\n",
    "import pandas as pd\n",
    "import sagemaker_pyspark\n",
    "\n",
    "import pydeequ\n",
    "\n",
    "classpath = \":\".join(sagemaker_pyspark.classpath_jars())\n",
    "\n",
    "spark = (SparkSession\n",
    "    .builder\n",
    "    .config(\"spark.driver.extraClassPath\", classpath)\n",
    "    .config(\"spark.jars.packages\", pydeequ.deequ_maven_coord)\n",
    "    .config(\"spark.jars.excludes\", pydeequ.f2j_maven_coord)\n",
    "    .getOrCreate())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's first generate some example data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = spark.sparkContext.parallelize([\n",
    "    Row(productName=\"thingA\", totalNumber=\"13.0\", status=\"IN_TRANSIT\", valuable=\"true\"),\n",
    "    Row(productName=\"thingA\", totalNumber=\"5\", status=\"DELAYED\", valuable=\"false\"),\n",
    "    Row(productName=\"thingB\", totalNumber=None, status=\"DELAYED\", valuable=None),\n",
    "    Row(productName=\"thingC\", totalNumber=None, status=\"IN_TRANSIT\", valuable=\"false\"),\n",
    "    Row(productName=\"thingD\", totalNumber=\"1.0\", status=\"DELAYED\", valuable=\"true\"),\n",
    "    Row(productName=\"thingC\", totalNumber=\"7.0\", status=\"UNKNOWN\", valuable=None),\n",
    "    Row(productName=\"thingC\", totalNumber=\"20\", status=\"UNKNOWN\", valuable=None),\n",
    "    Row(productName=\"thingE\", totalNumber=\"20\", status=\"DELAYED\", valuable=\"false\"),\n",
    "    Row(productName=\"thingA\", totalNumber=\"13.0\", status=\"IN_TRANSIT\", valuable=\"true\"),\n",
    "    Row(productName=\"thingA\", totalNumber=\"5\", status=\"DELAYED\", valuable=\"false\"),\n",
    "    Row(productName=\"thingB\", totalNumber=None, status=\"DELAYED\", valuable=None),\n",
    "    Row(productName=\"thingC\", totalNumber=None, status=\"IN_TRANSIT\", valuable=\"false\"),\n",
    "    Row(productName=\"thingD\", totalNumber=\"1.0\", status=\"DELAYED\", valuable=\"true\"),\n",
    "    Row(productName=\"thingC\", totalNumber=\"17.0\", status=\"UNKNOWN\", valuable=None),\n",
    "    Row(productName=\"thingC\", totalNumber=\"22\", status=\"UNKNOWN\", valuable=None),\n",
    "    Row(productName=\"thingE\", totalNumber=\"23\", status=\"DELAYED\", valuable=\"false\")]).toDF()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we ask PyDeequ to compute constraint suggestions for us on the data. It will profile the data and then apply the set of rules specified in `addConstraintRules()` to suggest constraints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydeequ.suggestions import *\n",
    "\n",
    "suggestionResult = ConstraintSuggestionRunner(spark) \\\n",
    "             .onData(df) \\\n",
    "             .addConstraintRule(DEFAULT()) \\\n",
    "             .run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now investigate the constraints that deequ suggested. We get a textual description and the corresponding Python code for each suggested constraint. Note that the constraint suggestion is based on heuristic rules and assumes that the data it is shown is 'static' and correct, which might often not be the case in the real world. Therefore the suggestions should always be manually reviewed before being applied in real deployments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Constraint suggestion for 'valuable': 'valuable' has less than 62% missing values\n",
      "The corresponding Python code is: .hasCompleteness(\"valuable\", lambda x: x >= 0.38, \"It should be above 0.38!\")\n",
      "\n",
      "Constraint suggestion for 'valuable': 'valuable' has type Boolean\n",
      "The corresponding Python code is: .hasDataType(\"valuable\", ConstrainableDataTypes.Boolean)\n",
      "\n",
      "Constraint suggestion for 'totalNumber': 'totalNumber' has no negative values\n",
      "The corresponding Python code is: .isNonNegative(\"totalNumber\")\n",
      "\n",
      "Constraint suggestion for 'totalNumber': 'totalNumber' has less than 47% missing values\n",
      "The corresponding Python code is: .hasCompleteness(\"totalNumber\", lambda x: x >= 0.53, \"It should be above 0.53!\")\n",
      "\n",
      "Constraint suggestion for 'totalNumber': 'totalNumber' has type Fractional\n",
      "The corresponding Python code is: .hasDataType(\"totalNumber\", ConstrainableDataTypes.Fractional)\n",
      "\n",
      "Constraint suggestion for 'productName': 'productName' has value range 'thingC', 'thingA', 'thingB', 'thingE', 'thingD'\n",
      "The corresponding Python code is: .isContainedIn(\"productName\", [\"thingC\", \"thingA\", \"thingB\", \"thingE\", \"thingD\"])\n",
      "\n",
      "Constraint suggestion for 'productName': 'productName' is not null\n",
      "The corresponding Python code is: .isComplete(\"productName\")\n",
      "\n",
      "Constraint suggestion for 'status': 'status' has value range 'DELAYED', 'UNKNOWN', 'IN_TRANSIT'\n",
      "The corresponding Python code is: .isContainedIn(\"status\", [\"DELAYED\", \"UNKNOWN\", \"IN_TRANSIT\"])\n",
      "\n",
      "Constraint suggestion for 'status': 'status' is not null\n",
      "The corresponding Python code is: .isComplete(\"status\")\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for sugg in suggestionResult['constraint_suggestions']:\n",
    "    print(f\"Constraint suggestion for \\'{sugg['column_name']}\\': {sugg['description']}\")\n",
    "    print(f\"The corresponding Python code is: {sugg['code_for_constraint']}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first suggestions we get are for the `valuable` column. **PyDeequ** correctly identified that this column is actually a `boolean` column 'disguised' as string column and therefore suggests a constraint on the `boolean` datatype. Furthermore, it saw that this column contains some missing values and suggests a constraint that checks that the ratio of missing values should not increase in the future.\n",
    "\n",
    "```\n",
    "Constraint suggestion for 'valuable': 'valuable' has less than 62% missing values\n",
    "The corresponding Python code is: .hasCompleteness(\"valuable\", lambda x: x >= 0.38, \"It should be above 0.38!\")\n",
    "\n",
    "Constraint suggestion for 'valuable': 'valuable' has type Boolean\n",
    "The corresponding Python code is: .hasDataType(\"valuable\", ConstrainableDataTypes.Boolean)\n",
    "```\n",
    "\n",
    "Next we look at the `totalNumber` column. PyDeequ identified that this column is actually a numeric column 'disguised' as string column and therefore suggests a constraint on a fractional datatype (such as `float` or `double`). Furthermore, it saw that this column contains some missing values and suggests a constraint that checks that the ratio of missing values should not increase in the future. Additionally, it suggests that values in this column should always be positive (as it did not see any negative values in the example data), which probably makes a lot of sense for this count-like data.\n",
    "\n",
    "```\n",
    "Constraint suggestion for 'totalNumber': 'totalNumber' has no negative values\n",
    "The corresponding Python code is: .isNonNegative(\"totalNumber\")\n",
    "\n",
    "Constraint suggestion for 'totalNumber': 'totalNumber' has less than 47% missing values\n",
    "The corresponding Python code is: .hasCompleteness(\"totalNumber\", lambda x: x >= 0.53, \"It should be above 0.53!\")\n",
    "\n",
    "Constraint suggestion for 'totalNumber': 'totalNumber' has type Fractional\n",
    "The corresponding Python code is: .hasDataType(\"totalNumber\", ConstrainableDataTypes.Fractional)\n",
    "```\n",
    "\n",
    "Finally, we look at the suggestions for the `productName` and `status` columns. Both of them did not have a single missing value in the example data, so an `isComplete` constraint is suggested for them. Furthermore, both of them only have a small set of possible values, therefore an `isContainedIn` constraint is suggested, which would check that future values are also contained in the range of observed values.\n",
    "\n",
    "```\n",
    "Constraint suggestion for 'productName': 'productName' has value range 'thingC', 'thingA', 'thingB', 'thingE', 'thingD'\n",
    "The corresponding Python code is: .isContainedIn(\"productName\", [\"thingC\", \"thingA\", \"thingB\", \"thingE\", \"thingD\"])\n",
    "\n",
    "Constraint suggestion for 'productName': 'productName' is not null\n",
    "The corresponding Python code is: .isComplete(\"productName\")\n",
    "\n",
    "Constraint suggestion for 'status': 'status' has value range 'DELAYED', 'UNKNOWN', 'IN_TRANSIT'\n",
    "The corresponding Python code is: .isContainedIn(\"status\", [\"DELAYED\", \"UNKNOWN\", \"IN_TRANSIT\"])\n",
    "\n",
    "Constraint suggestion for 'status': 'status' is not null\n",
    "The corresponding Python code is: .isComplete(\"status\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently, we leave it up to the user to decide whether they want to apply the suggested constraints or not, and provide the corresponding Scala code for convenience. For larger datasets, it makes sense to evaluate the suggested constraints on some held-out portion of the data to see whether they hold or not. You can test this by adding an invocation of .useTrainTestSplitWithTestsetRatio(0.1) to the ConstraintSuggestionRunner. With this configuration, it would compute constraint suggestions on 90% of the data and evaluate the suggested constraints on the remaining 10%.\n",
    "\n",
    "Finally, we would also like to note that the constraint suggestion code provides access to the underlying [column profiles](./data_profiling_example.ipynb) that it computed via `suggestionResult.columnProfiles`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}