3-choosing-and-collecting-text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Step 1: Choosing and Collecting Text\n",
    "\n",
    "---\n",
    "---\n",
    "\n",
    "## The Text-Mining Pipeline: 5 Steps of Text-Mining\n",
    "There is no set way to do text-mining, but typically a workflow will involve steps like these:\n",
    "1. Choosing and collecting your text\n",
    "2. Cleaning and preparing your text\n",
    "3. Exploring your data\n",
    "4. Analysing your data\n",
    "5. Presenting the results of your analysis\n",
    "\n",
    "![Text-mining pipeline](assets/pipeline.png)\n",
    "\n",
    "You may go through these steps more than once to refine your data and results, and frequently steps may be merged together. The important thing to realise is that steps 1-2 are critical in ensuring your data is capable of actually answering your research questions. You are likely to spend significant time on cleaning and preparing your text.\n",
    "\n",
    "Remember:\n",
    "\n",
    "> **Rubbish in = rubbish out**\n",
    "\n",
    "This notebook covers step 1. The next notebook [4-cleaning-and-exploring-text](4-cleaning-and-exploring-text.ipynb) will show steps 2-3 and notebook [5-analysis-and-visualisation](5-analysis-and-visualisation.ipynb) will show steps 4-5.\n",
    "\n",
    "No matter your research subject, you need to be aware of the many issues of electronic data collection. We cannot cover them all here, but you should ask yourself some questions as you start to collect data, such as:\n",
    "* What sort of data do I need to answer my research questions?\n",
    "* What data is available?\n",
    "* What is the quality of the data?\n",
    "* How can I get the data?\n",
    "* Am I allowed to use it for text-mining?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## A Simple Example: Top Words Used in Homer's *Iliad*\n",
    "\n",
    "Our research question will be:\n",
    "\n",
    "> What are the top 10 words used in Homer's Iliad in English translation?\n",
    "\n",
    "### What sort of data do I need to answer my research questions?\n",
    "\n",
    "I need a copy of Homer's *Iliad* in English translation. In this instance, I am not bothered by which translation.\n",
    "\n",
    "### What data is available?\n",
    "\n",
    "[Project Gutenberg](http://www.gutenberg.org/) is the first provider of free electronic books and has over 58,000. \"You will find the world's great literature here, with focus on older works for which U.S. copyright has expired. Thousands of volunteers digitized and diligently proofread the eBooks, for enjoyment and education.\"\n",
    "\n",
    "Here is Homer's *Iliad*, translated by Samuel Butler in 1898: http://www.gutenberg.org/ebooks/2199\n",
    "\n",
    "### What is the quality of the data?\n",
    "\n",
    "Potentially variable. When some books are digitised by OCR ([Optical Character Recognition](https://en.wikipedia.org/wiki/Optical_character_recognition)) they don't get corrected before being published online. Common OCR errors occur because the text has been 'transcribed' incorrectly and OCR software often makes the same mistakes time and again. We are very fortunate that this text does not suffer from common OCR errors because it has been corrected by volunteers.\n",
    "\n",
    "I won't be covering what to do about OCR errors in this course, but if you are curious you can read more about [ways of correcting predictable errors](https://usesofscale.com/gritty-details/basic-ocr-correction/) and how the British Library has dealt with this in a blog post [Dealing with Optical Character Recognition errors in Victorian newspapers](https://blogs.bl.uk/digital-scholarship/2016/07/dealing-with-optical-character-recognition-errors-in-victorian-newspapers.html).\n",
    "\n",
    "### How can I get the data?\n",
    "\n",
    "Project Gutenberg clearly states on their [Terms of Use](https://www.gutenberg.org/policy/terms_of_use.html) that their website is 'intended for human users only'. If you want to use code to get their data you must use one of their [mirror sites](http://www.gutenberg.org/MIRRORS.ALL) -- you should pick the one that is nearest to your location.\n",
    "\n",
    "We will be using the text file at http://www.mirrorservice.org/sites/ftp.ibiblio.org/pub/docs/books/gutenberg/2/1/9/2199/2199-0.txt\n",
    "\n",
    "### Am I allowed to use it for text-mining?\n",
    "\n",
    "Project Gutenberg says in their [Permission: How To](https://www.gutenberg.org/policy/permission.html) that \"The vast majority of Project Gutenberg eBooks are in the public domain in the US.\" However, since UK copyright is different from US copyright, we still have to check for ourselves. This is a complicated area, but broadly we can say that UK copyright expires 70 years after the death of the author. Since [Samuel Butler](https://en.wikipedia.org/wiki/Samuel_Butler_(novelist)) died in 1902, we are probably ok to use his work.\n",
    "\n",
    "UK law now allows for using copyright material for computation analysis. The onus is on the individual researcher to ensure that their use is covered under the exception, so make sure that you check your plans before you start to collect your source materials. Cambridge University Library provides a [Text & Data Mining: Law on TDM](http://libguides.cam.ac.uk/tdm/law) guide and enquiry support for doing this.\n",
    "\n",
    "---\n",
    "---\n",
    "## Getting a Copy of the Homer's *Iliad* Text\n",
    "We saw in the last notebook that we can use a Python library called `requests` to get Web pages. We can therefore get a copy of the text file like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "response = requests.get('http://www.mirrorservice.org/sites/ftp.ibiblio.org/pub/docs/books/gutenberg/2/1/9/2199/2199-0.txt')\n",
    "iliad = response.text\n",
    "iliad[23664:23990]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can find out how many characters the file has by using the `len()` method (function)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(iliad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can search for a particular string in the file. The function `find()` returns the index of the _first_ matching string it finds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word = 'shield'\n",
    "iliad.find(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write code to get the immediate context of this word 'shield'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Inspecting the Text\n",
    "The first thing to do is inspect the text to check it is what we expected, and to give us an idea of the sort of cleaning we'll need to do in the next step. \n",
    "\n",
    "First, download a copy to your computer by going to the main URL http://www.gutenberg.org/ebooks/2199 and clicking on the link 'Plain Text UTF-8'. You can either view it in your browser or save it locally and open it with the text editor of your choice.\n",
    "\n",
    "Looking again at the text by eye we can see that the book starts with a load of front matter we don't want.\n",
    "\n",
    "The book actually starts after the text \"`*** START OF THIS PROJECT GUTENBERG EBOOK THE ILIAD ***`\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iliad[708:854]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is also unwanted matter at the end after \"`***END OF THE PROJECT GUTENBERG EBOOK THE ILIAD OF HOMER***`\" that we should get rid of too.\n",
    "\n",
    "Why does the text have all these `\\r` and `\\n` in them? The character sequence `\\r\\n` is used to signify the start of a new line. In a typical text editor, the software reads this sequence and displays it as a newline, supressing the actual characters in the text file and removing them from view. The output in a Jupyter notebook is less sophisticated than this and shows all the characters as they appear in the file.\n",
    "\n",
    "---\n",
    "### Editing Text Files\n",
    "Text files are files that should be *opened* as plain text and nothing else. They often have file extensions such as `.txt`, `.html`, `.xml`, `.csv`. Microsoft Word documents (`.doc` and `.docx`) are not plain text and you should never edit your `.txt` files in Microsoft Word or WordPad. You need a proper *text editor*.\n",
    "\n",
    "Recommended text editors:\n",
    "* All platforms: [Atom](https://atom.io/) or [Sublime Text](https://www.sublimetext.com/)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Creating and Preparing a Local Copy\n",
    "\n",
    "It is not very efficient (nor allowed) to keep making Web requests to Project Gutenberg, especially with a very large corpus. I have therefore downloaded a copy for us (`ORIGINAL-2199-0.txt`) and placed it in the project under the `data` folder. \n",
    "\n",
    "I have also taken some steps to prepare a usable version of the file on your behalf (`PREPPED-2199-0.txt`). We will use this local copy instead from now on. In the spirit of full transparency and documentation here is what I have done:\n",
    "\n",
    "* Removed the unwanted Gutenberg-related matter at the front and back of the book.\n",
    "* Made sure that the character encoding is 'UTF-8'.\n",
    "\n",
    "You don't need to worry about the details of _character encoding_ for this course. You only need to know that Python works most easily with UTF-8 files and so we must have the file in that encoding to avoid problems.\n",
    "\n",
    "---\n",
    "### Character Encoding\n",
    "Character encoding is a very important topic, but it is not an easy one. If you end up dealing with a lot of text files in building up your corpus you will have to be aware that dealing with files that have different, or unknown, character encodings can get very messy. If you don't know, or wrongly assume the character encoding of a file you can end up with this sort of thing: � �ࡻࢅ�࢖\n",
    "\n",
    "This is especially important if your corpus is written in a non-English language, because the accents or non-Latin alphabet characters of the text may get mangled. The short answer to the problem is to always make sure your files are encoded with 'UTF-8' if you can.\n",
    "\n",
    "> **'UTF-8' is often not the default encoding on Windows machines. This means that you can quickly end up in a mess when you edit and save your 'UTF-8' text files on Windows. The encoding may be automatically changed to 'ISO-8859-1' or 'latin-1'. You should find out how to save files as 'UTF-8' in your text editor.**\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Summary\n",
    "\n",
    "In this notebook:\n",
    "\n",
    "* List the 5 typical steps of a **text-mining pipeline**.\n",
    "* Consider **issues of electronic data collection** such as choice, availability, quality, access and copyright.\n",
    "* Getting and saving a **local copy**.\n",
    "* **Inspecting** the text by eye for issues."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}