cwpk-29-extracting-object-data-properties.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#29: Extracting Object and Data Properties\n",
    "=======================================\n",
    "\n",
    "We Continue the Theme of Structural Extraction\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "In this installment of the [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) series, we continue the theme of extracting the structural backbone to [KBpedia](https://kbpedia.org/). Our attention shifts now from classes to properties, the predicates found in the middle of a *subject - predicate - object* [semantic triple](https://en.wikipedia.org/wiki/Semantic_triple). A *s-p-o* triple is the basic assertion in the [RDF](https://en.wikipedia.org/wiki/Resource_Description_Framework) and [OWL](https://en.wikipedia.org/wiki/Web_Ontology_Language) languages.\n",
    "\n",
    "There are three types of predicate properties in OWL. [Object properties](https://www.w3.org/TR/2012/REC-owl2-primer-20121211/) relate a subject to another named entity, one which may be found at an [IRI](https://en.wikipedia.org/wiki/Internationalized_Resource_Identifier) address, local or on the Web. [Data properties](https://www.w3.org/TR/2012/REC-owl2-primer-20121211/) are a value characterization of the subject, and may be represented by strings (labels or text) or date, time, location, or numeric values. Data properties are represented by [datatypes](https://en.wikipedia.org/wiki/Data_type), not IRIs. [Annotation properties](https://www.w3.org/TR/2012/REC-owl2-primer-20121211/) are pointers or descriptors to the subject and may be either a datatype or an IRI, but there is no reasoning over annotations across subjects. All property types can be represented in hierarchies using a <code>subPropertyOf</code> predicate similar to <code>subClassOf</code> for classes.\n",
    "\n",
    "In addition, KBpedia uses a triadic split of predicates based on the [universal categories](V) of [Charles Sanders Peirce](https://en.wikipedia.org/wiki/Charles_Sanders_Peirce). These map fairly closely with the OWL splits, but with some minor differences (not important to our current processing tasks). *Representations* are pointers, indicators, or descriptors to the thing at hand, the subject. These map closely to the OWL annotation properties. *Attributes* are the characterizations of the subject, intensional in nature, and are predominantly data properties (though it is not a violation to assign object properties where a value is one of an enumerated list). *Direct relations* are extensional relations between two entities, where the object in *s-p-o* must be an object property, represented by an IRI.\n",
    "\n",
    "In most of today's practice <code>subPropertyOf</code> is little used, though KBpedia is becoming active in exploring this area. In terms of semantic inheritance, properties are classes, though with important distinctions. Object and data properties may have functional roles, restrictions as to the size and nature of their sets, and specifications as to what types of subject they may represent (<code>domain</code>) or what type of object they may connect (<code>range</code>).\n",
    "\n",
    "Though supporting these restrictions, [Owlready2](http://www.lesfleursdunormal.fr/static/informatique/owlready/index_en.html) has less robust support for properties than classes. In the last installment's work on the class backbone we saw the advantage of the <code>.descendant()</code> method for collecting children or grandchildren throughout the subsumption tree of class descent. Owlready2 does not document or expose this method for properties, but with properties a sub-class of class in the Owlready2 code, I found I could use many of the class methods. Woohoo!\n",
    "\n",
    "What I outline below is a parallel structure extraction to what we saw in the last installment regarding classes. In the next installment we will transition from structure extraction to annotation extraction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Starting and Load\n",
    "We begin with our standard opening routine:\n",
    "\n",
    "<div style=\"background-color:#eee; border:1px dotted #aaa; vertical-align:middle; margin:15px 60px; padding:8px;\"><strong>Which environment?</strong> The specific load routine you should choose below depends on whether you are using the online MyBinder service (the 'raw' version) or local files. The example below is based on using local files (though replace with your own local directory specification). If loading from MyBinder, replace with the lines that are commented (<code>#</code>) out.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "main = 'C:/1-PythonProjects/kbpedia/sandbox/kbpedia_reference_concepts.owl'\n",
    "# main = 'https://raw.githubusercontent.com/Cognonto/CWPK/master/sandbox/builds/ontologies/kbpedia_reference_concepts.owl'\n",
    "skos_file = 'http://www.w3.org/2004/02/skos/core' \n",
    "kko_file = 'C:/1-PythonProjects/kbpedia/sandbox/kko.owl'\n",
    "# kko_file = 'https://raw.githubusercontent.com/Cognonto/CWPK/master/sandbox/builds/ontologies/kko.owl'\n",
    "\n",
    "from owlready2 import *\n",
    "world = World()\n",
    "kb = world.get_ontology(main).load()\n",
    "rc = kb.get_namespace('http://kbpedia.org/kko/rc/')               \n",
    "\n",
    "skos = world.get_ontology(skos_file).load()\n",
    "kb.imported_ontologies.append(skos)\n",
    "core = world.get_namespace('http://www.w3.org/2004/02/skos/core#')\n",
    "\n",
    "kko = world.get_ontology(kko_file).load()\n",
    "kb.imported_ontologies.append(kko)\n",
    "kko = kb.get_namespace('http://kbpedia.org/ontologies/kko#')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we execute each cell as we progress down this notebook page by pressing <code>shift+enter</code> for the highlighted cell or by choosing Run from the notebook menu.\n",
    "\n",
    "Let's first begin by inspecting the populated lists of our three types of properties, beginning with object (prefix <code>po_</code>), and then data (prefix <code>pd_</code>) and annotation (prefix <code>pa_</code>) properties, checking the length for the number of records as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "po_set = list(world.object_properties())\n",
    "list(po_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1309"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(po_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd_set = list(world.data_properties())\n",
    "list(pd_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(pd_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pa_set = list(world.annotation_properties())\n",
    "list(pa_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(pa_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may want to Cell &rarr; All Output &rarr; Clear to remove these long listings from your notebook.\n",
    "\n",
    "### Getting the Subsets Right\n",
    "When we inspect these lists, however, we see that many of the predicates are 'standard' ones that we have in our core KBpedia Knowledge Ontology (see the [KKO image](https://kbpedia.org/knowledge-graph/)). Recall that our design has us nucleating our knowledge graph build efforts with a starting ontology. In KBpedia's case that is KKO.\n",
    "\n",
    "Now we could just build all of the properties each time from scratch. But, similar to our typology design for a modular class structure, we very much like our more direct mapping to predicates to Peirce's universal categories. \n",
    "\n",
    "So, we test whether we can use the same <code>.descendants()</code> approach we used in the prior installment, only now applied to properties. In the case of the annotations property, that corresponds to our <code>kko.representations</code> predicate. So, we test this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "root = kko.representations\n",
    "pa_set=root.descendants()\n",
    "\n",
    "len(pa_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that we dropped 11 predicates that were in our first approach.\n",
    "\n",
    "We can list the set and verify that nearly all of our descendant properties are indeed in the reference concept (rc) namespace (we will address the minor exceptions in some installments to come), so we have successfully separated our additions from the core KKO starting point:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(pa_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since I like keeping the core ontology design idea, I will continue to use this more specific way to set the roots for KBpedia properties for these extraction routines. It adds a few more files to process down the road, but it can all be automated and I am better able to keep the distnction between KKO and the specific classes and properties that populate it for the domain at hand. It does mean that all new properties introduced to the system must be made a <code>rdfs:subPropertyOf</code> of one of the tie-in roots, but that also enforces the explicit treatment of new properties in relation to the Peircean universal categories.\n",
    "\n",
    "Under this approach, the root for annotation properties is <code>kko.representations</code> as noted. For object properties, the root is <code>kko.predicateProperties</code>. (The other two main branches are <code>kko.mappingProeperties</code> and <code>skos.skosProperties</code>, which we consider central to KKO.) For data properties, the root is <code>kko.predicateDataProperties</code>. The other data properties are also built in to KKO.\n",
    "\n",
    "If one wanted to adopt the code base in this **CWPK** series for other purposes, perhaps with a different core or bootstrap, other design choices could be made. But this approach feels correct for the design and architecture of KBpedia."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Iterating Sub Properties\n",
    "\n",
    "Now that we have decided this scope question, let's try the final code block from the last installment (also based on <code>.descendants()</code> and <code>is_a</code>) so see if and how it works in the property context. We make two changes to the last installment routine in that we now specify the <code>rdfs:subPropertyOf</code> property and replace our iterated set with <code>pa_set</code>: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "o_frag = set()\n",
    "s_frag = set()\n",
    "p_item = 'rdfs:subPropertyOf'\n",
    "for s_item in pa_set:\n",
    "  o_set = s_item.is_a\n",
    "  for o_item in o_set:\n",
    "     print(s_item,',',p_item,',',o_item,'.','\\n', sep='', end='')\n",
    "     o_frag.add(o_item)\n",
    "  s_frag.add(s_item) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, again! Our prior logic is directly transferable. The nice thing about this code applied to properties is that we also get the specifications for creating a new property, useful when [roundtripping](https://en.wikipedia.org/wiki/Round-trip_engineering) the information for build routines.\n",
    "\n",
    "So, we clear out the currently active cell and are ready to move on. But first, we also made some nice discoveries in working out today's installment, so I will end today's installment with a couple of tips."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus Tip\n",
    "While doing the research for this installment, I came across a nifty method within owlready2 for controlling how these extraction retrievals display, with full IRIs, namespaces, or not. First, run the original script for listing the <code>pa_set</code> above. Then, for contrast, try these two options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def render_using_label(entity):\n",
    "    return entity.label.first() or entity.name\n",
    "\n",
    "set_render_func(render_using_label)\n",
    "list(pa_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def render_using_iri(entity):\n",
    "    return entity.iri\n",
    "\n",
    "set_render_func(render_using_iri)\n",
    "list(pa_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These two suggestions came from the owlready2 [documentation](https://owlready2.readthedocs.io/en/latest/annotations.html#custom-rendering-of-entities). But, after trying them, I wanted to get back to the original (default) formatting. But the documentation is silent on this question. After poking through the code a bit, I found this initialization method for returning to the default. Again, try it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set_render_func(default_render_func)\n",
    "list(pa_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus Tip #2\n",
    "Here is a nice method for getting a listing of all of the properties applied to a given class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rc.Mammal.get_class_properties()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Documentation\n",
    "\n",
    "- [Owlready2 property documentation](https://owlready2.readthedocs.io/en/latest/properties.html), including for [restrictions](https://owlready2.readthedocs.io/en/latest/restriction.html).\n",
    "\n",
    "\n",
    " <div style=\"background-color:#efefff; border:1px dotted #ceceff; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}