Objective1_d.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Playground: Refine `_base` function to fit more jobs.\n",
    "Also, start learning to simplify what is needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import relevant modules\n",
    "import os           # interface with the underlying OS\n",
    "import numpy as np  # linear algebra\n",
    "import pandas as pd # dataframe\n",
    "import re           # regular expression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define path to look at\n",
    "path = 'CityofLA/Job Bulletins/'\n",
    "\n",
    "# Get a list of all txt files in this path\n",
    "all_txt_files = os.listdir(path)\n",
    "all_txt_files.sort() # WARNING: this mutates the list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "683\n",
      "683\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Do some checks here\n",
    "print(len(all_txt_files))      # length should be 683 as manually verified\n",
    "print(len(set(all_txt_files))) # see if each file is unique. hopefully 683 as well!\n",
    "print('SYSTEMS ANALYST 1596 102717.txt' in all_txt_files) # should be True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "609\n"
     ]
    }
   ],
   "source": [
    "# Find `SYSTEMS ANALYST 1596 102717.txt` (which exists).\n",
    "print(all_txt_files.index('SYSTEMS ANALYST 1596 102717.txt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import file as text\n",
    "`sa` stands for systems analyst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'SYSTEMS ANALYST\\n\\nClass Code: 1596\\nOpen Date: 10-27-17\\n(Exam Open to All, including Current City Employees)\\n\\nANNUAL SALARY\\n\\n$68,611 to $100,307\\nThe salary in the Department of Water and Power is $70,908 to $88,092 and $83,770 to $104,065.\\n\\nNOTES:\\n\\n1. Candidates from the eligible list are normally appointed to vacancies in the lower pay grade positions.  \\n2. Annual salary is at the start of the pay range. The current salary range is subject to change. Please confirm the starting salary with the hiring department before accepting a job offer.\\n3. For information regarding reciprocity between the City of Los Angeles departments and LADWP, go to http://per.lacity.org/Reciprocity_CityDepts_and_DWP.pdf.\\n\\n\\nDUTIES\\n\\nA Systems Analyst analyzes procedures, methods and operations of computer-based information systems; designs, implements, and recommends information systems to improve the efficiency and economy of City operations; performs cost benefit and feasibility analyses related to the modification of existing computer-based information systems, the maintenance and support of information systems, or the implementation of new computer-based systems; accesses and analyzes information from automated files using high-level retrieval languages.\\n\\nREQUIREMENTS/MINIMUM QUALIFICATIONS\\n\\n1. Graduation from an accredited four-year college or university with a major in Computer Science, Information Systems, or Geographical Information Systems; or\\n2. Graduation from an accredited four-year college or university and two years of full-time paid experience in a class at the level of Management Assistant which provides experience in:\\na. the development, analysis, implementation or major modification of new or existing computer-based information systems or relational databases; or\\nb. performing cost benefit, feasibility and requirements analysis for a large-scale computer-based information system; or\\nc. performing system implementation and support activities including software and hardware acquisition, installation, modifications to system configuration, system and application upgrade installation; or\\n3. Two years of full-time paid experience as a Systems Aide with the City of Los Angeles; and\\na. Satisfactory completion of four courses, of at least three semester or four quarter units each, in Information Systems, Systems Analysis, or a closely related degree program, professional designation, or certificate program from an accredited college or university.\\nb. At least three of the courses must be from the core courses required in the program, and one course may be from either the required core courses or the prescribed elective courses of the program.  A course in systems analysis and design is especially desired, but not required. \\n\\nPROCESS NOTES\\n\\n1. Some positions may require a valid California driver\\'s license. Candidates may not be eligible for appointment to these positions if their record within the last 36 months reflects three or more moving violations and/or at-fault accidents, or a conviction of a major moving violation (such as DUI).\\n2. Please note that qualifying education must be from a college or university accredited by a City of Los Angeles recognized agency. A list of approved accrediting agencies can be found at http://per.lacity.org/Accredited%20Institutions%2008-21-08.pdf.\\n3. Applicants qualifying under Requirement #2 above must identify in the Supplemental Questions section of the application the specific projects that provide qualifying experience. The minor application, modification or use of existing systems is not considered to be qualifying experience.\\n4. Four years of full-time paid experience in any of the areas noted in Requirement #2 above may be substituted for the required degree only. The two years\\' experience in the areas noted is still required to fully meet Requirement #2.\\n5. Applicants qualifying under Requirement #3 above must list in the Supplemental Question section of the application the name of the school attended, course titles, number of units, completion dates, and grades. General education courses will not meet the education requirement outlined in Requirement #3. A copy of the offering institution\\'s catalog program description and course list must be attached to the on-line application and submitted to the Personnel Department at the time of filing. Failure to submit the requested documents may result in processing delays and/or disqualification.\\n6. Candidates completing the exam process will be contacted by the Personnel Department to provide required proof of qualifying four-year degree.\\n\\n\\nSELECTIVE CERTIFICATION\\n\\nIn accordance with Charter Section 1010 (b) and Civil Service Rule 5.31, selective certification will be used for some positions that require special skills and/or training. For such positions, only applicants possessing the specific professional certification(s) at the time of filing may be considered. Some positions will require a professional industry certification(s) in one or more of the following:\\n\\n1. Adobe Dreamweaver CS6 ACE\\n2. Adobe Photoshop CS5 or CS6 ACE\\n3. ArcGIS Desktop Developer Associate\\n4. BMC Certified Administrator\\n5. Cisco Certified Entry Networking Technician (CCENT) \\n6. Cisco Certified Network Associate (CCNA)\\n7. Citrix Certified Associate - Virtualization (CCA-V)\\n8. CompTIA A+ \\n9. Delphi Developer Certification\\n10. Enterprise Administration Associate \\n11. Enterprise System Design Associate\\n12. Enterprise Geodata Management Associate\\n13. Enterprise Geodata Management Professional\\n14. HP Service Manager \\n15. IBM Certified Solutions Expert \\n16. IBM Certified Operator\\n17. IBM Certified Systems Administrator\\n18. JAVA SE Programmer Certification\\n19. Linux Foundation Certified System Administrator  (LFCS) \\n20. Linux Foundation Certified Engineer (LFCE)\\n21. MCTS: Administering and Deploying System Center 2012 Configuration Manager\\n1. \\n22. Microsoft Certified Solutions Associate (MCSA)\\n23. Microsoft Certified Solutions Developer (MCSD) \\n24. Microsoft Certified Solutions Expert (MCSE)\\n25. Microsoft Certified Systems Administrator (MCSA) \\n26. Microsoft Certified Systems Engineer (MCSE)\\n27. Microsoft Technology Associate (MTA)\\n28. Microsoft Technology Associate: IT Infrastructure\\n29. Microsoft: Core Solutions of Microsoft SharePoint Server 2013\\n30. Microsoft: Programming in HTML5 with JavaScript and CSS3\\n31. Microsoft: Programming in C#\\n32. Microsoft: Developing ASP.NET MVC Web Applications\\n33. Microsoft: Windows Server 2008 Active Directory, Configuring \\n34. Microsoft: Windows Server 2008, Server Administrator\\n35. Microsoft: Windows 7, Configuring\\n36. Microsoft: Windows 7, Enterprise Desktop Support Technician\\n37. Microsoft: Installing and Configuring Windows 10\\n38. Microsoft: Installation, Storage, and Compute with Windows Server 2016\\n39. Microsoft: Database Fundamentals\\n40. Certified Novell Administrator (CNA)\\n41. Oracle E-Business Suite R12: E-Business Essentials\\n42. Oracle Database SQL Certified Associate \\n43. Red Hat Certified System Administrator (RHCSA)\\n44. Unix System Administration\\n45. VMware Certified Associate 6 - Data Center Virtualization\\n46. VMware Certified Professional 6.5 - Data Center Virtualization\\n\\t\\t\\nApplicants must attach a copy of the certificate(s) to the on-line application at the time of filing. The certificate(s) must be attached to the on-line application BEFORE the application is submitted. If the certificate(s) is attached AFTER the on-line application has been submitted, it will not be available to the City and applicants WILL NOT be credited with having submitted the certificate(s). Applicants who fail to submit the certificate(s) at the time of filing by not attaching it to the on-line application will not be considered for this part of selective certification.\\n\\nWHERE TO APPLY\\n\\nApplications will only be accepted on-line. When you are viewing the on-line job bulletin of your choice, simply scroll to the top of the page and select the \"Apply\" icon. On-line job bulletins are also available at https://www.governmentjobs.com/careers/lacity for Open Competitive Examinations and at https://www.governmentjobs.com/careers/lacity/promotionaljobs for Promotional Examinations.\\n\\nNOTE:\\n\\nApplicants are urged to apply early to ensure you have time to resolve any technical issues you may encounter.\\n\\nAPPLICATION DEADLINE\\n\\nThe examination may close without prior notice at any time after a sufficient number of applications have been received. For administrative purposes, filing may close periodically and reopen periodically.\\n\\nIn accordance with Civil Service Rule 4.2, all applicants who apply may not be tested in this examination. To meet anticipated hiring needs, only a limited number of qualified applicants will be invited to participate in the weighted multiple-choice test in the following order: 1) Los Angeles City Promotional applicants who meet the minimum requirements. You must have received a regular appointment to a City position or be on a reserve list to apply for this examination as a promotional candidate; 2) Applicants currently employed by the City of Los Angeles on a part-time or exempt basis who meet the minimum requirements; 3) Remaining applicants who meet the minimum requirements in sufficient numbers to meet hiring needs in the order that applications were received. Applications submitted during the filing period will be kept on file for two years from the date of filing in the event that additional applicants need to be tested to meet hiring needs.\\nSELECTION PROCESS\\n\\nAfter meeting minimum qualifications, and in accordance with Rule 4.2 as stated above, candidates will be scheduled for the following:\\n\\nExamination Weight: Multiple-Choice Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .100%\\n\\t\\t\\nThe examination will consist entirely of a weighted multiple-choice test administered and proctored online. In the online multiple-choice test, the following competencies may be evaluated: Analytical Ability, Interpersonal Skills, Teamwork, Written Communication, and Technology Application, including: office productivity tools and/or software; local area network (LAN) configurations and communication protocols; computer hardware and their use, networking devices, and software tools; various operating systems including their use, interoperability, capabilities, and limitations; basic principles and concepts of database design; systems development life cycle (SDLC) methodology including the associated steps and phases; cloud-based computing services and their use; disaster recovery systems, procedures, and/or best practices; Information Technology Service Management (ITSM) frameworks and methodologies; and other necessary skills, knowledge, and abilities.\\n\\nAdditional information can be obtained by going to http://per.lacity.org/index.cfm?content=jobanalyses and clicking on Competencies under Systems Analyst.\\n\\nThe multiple-choice test will be proctored and administered on-line during a single session. Candidates invited to participate in the online multiple-choice test will be able to take the test from any location using a computer with a webcam and a reliable internet connection. Candidates will receive an email from the City of Los Angeles outlining the dates and specific steps on how to take the multiple-choice test on-line. It is anticipated that candidates invited to participate in the first administration of the weighted multiple-choice test will be able to take the multiple-choice test online starting SATURDAY, JANUARY 6, 2018 THROUGH SATURDAY, JANUARY 13, 2018. Candidates who do not complete and submit the multiple-choice test on-line by SATURDAY, JANUARY 13, 2018 at 11:59 p.m., Pacific Standard Time, will not be considered further in this examination. Additional test dates may be added as needed.\\n\\n\\nNOTES:\\t\\n\\n1. This examination is based on a validation study, and as provided by Civil Service Commission Rule 4.20, the multiple-choice test will not be subject to candidate inspection.\\n2. As a covered entity under the Fair Employment and Housing Act and Title II of the Americans with Disabilities Act, the City of Los Angeles does not discriminate on the basis of disability and upon request, will provide reasonable accommodations to ensure equal access to its programs, services, and activities.  To request a disability accommodation, please complete the Disability Accommodation Form within 14 calendar days of the submittal of the City application.  The Disability Accommodation Form can be obtained at http://per.lacity.org/exams/verify_disability.pdf.\\n3. A final average score of 70% or higher is required to be placed on the eligible list.\\n4. Applications are accepted subject to review to ensure that minimum qualifications are met. Candidates may be disqualified at any time if is determined that they do not possess the minimum qualifications stated on this bulletin.\\n5. In accordance with Civil Service Rule, Sec. 4.24, review periods may be combined. Candidates in the examination process may file protests as provided in Sec. 4.20, 4.22 and 4.23 as applicable and within the required time frame; however, the Personnel Department may respond to and resolve protests prior to the establishment of the eligible list.\\n6. In conjunction with Civil Service Rules, applicants who have received a regular appointment to a City position or are on a reserve list will be considered Promotional candidates while all other applicants will be considered Open candidates.\\n7. The promotional list will ordinarily be used ahead of the open competitive list. However, if open competitive candidates receive a higher score, without military credits, than the highest available promotional candidate, after adding 0.25 of a point seniority credit for each year of continuous classified City service, the Civil Service Commission, upon request of the appointing authority, may approve certification of such open competitive candidates ahead of the promotional candidates.\\n8. You may take this examination only once every 365 days. Your name may be removed from the eligible list after six (6) months.\\n9. Your rank on the employment list may change as scores of candidates from other administrations of this examination are merged onto one list.\\n\\n\\nNOTICE: \\n\\nIf you receive and accept an offer of employment to a regular position with the City of Los Angeles, your employee benefit coverage (including health and dental coverage as well as life insurance) will commence approximately six weeks after your original regular appointment. Not all positions in the City receive benefit coverage; you should inquire regarding the availability of employee benefits prior to accepting a position.\\nTHIS EXAMINATION IS TO BE GIVEN BOTH ON AN\\nINTERDEPARTMENTAL PROMOTIONAL AND AN OPEN COMPETITIVE BASIS\\n\\nThe City of Los Angeles does not discriminate on the basis of race, religion, national origin, sex, age, marital status, sexual orientation, gender identity, gender expression, disability, creed, color, ancestry, medical condition (cancer), or Acquired Immune Deficiency Syndrome.\\nClass Code 1596 CONT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-27-17\\n\\n\\nAN EQUAL EMPLOYMENT OPPORTUNITY EMPLOYER\\nFor additional information and FAQs regarding the City\\'s hiring process, please go to: http://per.lacity.org/index.cfm?content=employmenttestingprocess\\n\\n'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sa_path = path + all_txt_files[609]\n",
    "sa = open(sa_path, 'rt').read()\n",
    "sa # display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First, rewrite `_base` function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'> Need to use regex for a better information location. </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'REQUIREMENTS/MINIMUM QUALIFICATIONS\\n\\n1. Graduation from an accredited four-year college or university with a major in Computer Science, Information Systems, or Geographical Information Systems; or\\n2. Graduation from an accredited four-year college or university and two years of full-time paid experience in a class at the level of Management Assistant which provides experience in:\\na. the development, analysis, implementation or major modification of new or existing computer-based information systems or relational databases; or\\nb. performing cost benefit, feasibility and requirements analysis for a large-scale computer-based information system; or\\nc. performing system implementation and support activities including software and hardware acquisition, installation, modifications to system configuration, system and application upgrade installation; or\\n3. Two years of full-time paid experience as a Systems Aide with the City of Los Angeles; and\\na. Satisfactory completion of four courses, of at least three semester or four quarter units each, in Information Systems, Systems Analysis, or a closely related degree program, professional designation, or certificate program from an accredited college or university.\\nb. At least three of the courses must be from the core courses required in the program, and one course may be from either the required core courses or the prescribed elective courses of the program.  A course in systems analysis and design is especially desired, but not required. \\n\\n'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Locate the information\n",
    "temp = sa[sa.find('REQUIREMENTS/MINIMUM QUALIFICATIONS'):sa.find('PROCESS NOTES')]\n",
    "temp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1', '2', '3']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the digits that mark one part from the other\n",
    "digits_as_str = [sentence[0] for sentence in temp.split() if sentence[0].isdigit()]\n",
    "digits_as_str"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I avoid try/except as in `Object1_c` because sometimes they will skip over important data if there is a bug in the try clause. The downside is, of course, code may get more complicated. However, since we've already familiarize ourselves with the tasks and code flow, it shouldn't be too hard to understand what's written."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1. Graduation from an accredited four-year college or university with a major in Computer Science, Information Systems, or Geographical Information Systems; or\\n',\n",
       " '2. Graduation from an accredited four-year college or university and two years of full-time paid experience in a class at the level of Management Assistant which provides experience in:\\na. the development, analysis, implementation or major modification of new or existing computer-based information systems or relational databases; or\\nb. performing cost benefit, feasibility and requirements analysis for a large-scale computer-based information system; or\\nc. performing system implementation and support activities including software and hardware acquisition, installation, modifications to system configuration, system and application upgrade installation; or\\n',\n",
       " '3. Two years of full-time paid experience as a Systems Aide with the City of Los Angeles; and\\na. Satisfactory completion of four courses, of at least three semester or four quarter units each, in Information Systems, Systems Analysis, or a closely related degree program, professional designation, or certificate program from an accredited college or university.\\nb. At least three of the courses must be from the core courses required in the program, and one course may be from either the required core courses or the prescribed elective courses of the program.  A course in systems analysis and design is especially desired, but not required. \\n\\n']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Separate requirements one from the other.\n",
    "## Initialization\n",
    "line_item = [] # borrow from accounting term. means items in a list\n",
    "## Loop through idx of digits_as_str to get the text from one digit to the other\n",
    "for idx in range(len(digits_as_str)-1):\n",
    "    start = digits_as_str[idx]  ; pos1 = temp.index(start)\n",
    "    end   = digits_as_str[idx+1]; pos2 = temp.index(end)\n",
    "    item  = temp[pos1:pos2]     ; line_item.append(item)\n",
    "## Put the last part into the list of all possible requirements\n",
    "final_dig    = digits_as_str[-1]; final_pos = temp.index(final_dig)\n",
    "final_item = temp[final_pos:] ; line_item.append(final_item)\n",
    "## Display results\n",
    "line_item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['1. Graduation from an accredited four-year college or university with a major in Computer Science, Information Systems, or Geographical Information Systems; or'],\n",
       " ['2. Graduation from an accredited four-year college or university and two years of full-time paid experience in a class at the level of Management Assistant which provides experience in:',\n",
       "  'a. the development, analysis, implementation or major modification of new or existing computer-based information systems or relational databases; or',\n",
       "  'b. performing cost benefit, feasibility and requirements analysis for a large-scale computer-based information system; or',\n",
       "  'c. performing system implementation and support activities including software and hardware acquisition, installation, modifications to system configuration, system and application upgrade installation; or'],\n",
       " ['3. Two years of full-time paid experience as a Systems Aide with the City of Los Angeles; and',\n",
       "  'a. Satisfactory completion of four courses, of at least three semester or four quarter units each, in Information Systems, Systems Analysis, or a closely related degree program, professional designation, or certificate program from an accredited college or university.',\n",
       "  'b. At least three of the courses must be from the core courses required in the program, and one course may be from either the required core courses or the prescribed elective courses of the program.  A course in systems analysis and design is especially desired, but not required. ']]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Split further by newline characters\n",
    "line_item = [[line for line in item.split('\\n') if len(line) > 1] for item in line_item]\n",
    "line_item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('1',\n",
       "  ' Graduation from an accredited four-year college or university with a major in Computer Science, Information Systems, or Geographical Information Systems; or',\n",
       "  'A',\n",
       "  None,\n",
       "  None,\n",
       "  None),\n",
       " ('2',\n",
       "  ' Graduation from an accredited four-year college or university and two years of full-time paid experience in a class at the level of Management Assistant which provides experience in:',\n",
       "  'A',\n",
       "  ' the development, analysis, implementation or major modification of new or existing computer-based information systems or relational databases; or',\n",
       "  None,\n",
       "  None),\n",
       " ('2',\n",
       "  ' Graduation from an accredited four-year college or university and two years of full-time paid experience in a class at the level of Management Assistant which provides experience in:',\n",
       "  'B',\n",
       "  ' performing cost benefit, feasibility and requirements analysis for a large-scale computer-based information system; or',\n",
       "  None,\n",
       "  None),\n",
       " ('2',\n",
       "  ' Graduation from an accredited four-year college or university and two years of full-time paid experience in a class at the level of Management Assistant which provides experience in:',\n",
       "  'C',\n",
       "  ' performing system implementation and support activities including software and hardware acquisition, installation, modifications to system configuration, system and application upgrade installation; or',\n",
       "  None,\n",
       "  None),\n",
       " ('3',\n",
       "  ' Two years of full-time paid experience as a Systems Aide with the City of Los Angeles; and',\n",
       "  'A',\n",
       "  None,\n",
       "  ' Satisfactory completion of four courses, of at least three semester or four quarter units each, in Information Systems, Systems Analysis, or a closely related degree program, professional designation, or certificate program from an accredited college or university.',\n",
       "  ' At least three of the courses must be from the core courses required in the program, and one course may be from either the required core courses or the prescribed elective courses of the program.  A course in systems analysis and design is especially desired, but not required. ')]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "requirement_list = []\n",
    "for aList in line_item:\n",
    "    # First item in the list aList may be a requirement\n",
    "    # requirement_id is the first element in the string\n",
    "    # requirement_txt is anything after, for example, 1., 2., 3., etc. (2 characters, that's why 2:)\n",
    "    main_information = aList[0]\n",
    "    requirement_id   = main_information[0]; requirement_txt = main_information[2:]\n",
    "    # Step 1\n",
    "    if len(aList) == 1:\n",
    "        ## Step 1a: Handle case \"1. and 2.\", \"1. or 2.\"\n",
    "        if main_information.endswith('and'):\n",
    "            ### Find next item's requirement id and the text associated with it\n",
    "            next_list_idx   = line_item.index(aList)+1; next_list = line_item[next_list_idx]; next_req = next_list[0]\n",
    "            next_digit      = next_req[0]             ; next_req_txt = next_req[2:]\n",
    "            ### Form a nested requirement and put into requirement list\n",
    "            nested_reqmnt   = (requirement_id, requirement_txt, next_digit, next_req_txt)\n",
    "            requirement_list.append(nested_reqmnt)\n",
    "        ## Step 1b: Handle case \"1. or 2.\"\n",
    "        else:\n",
    "            requirement = (requirement_id, requirement_txt, 'A')\n",
    "            requirement_list.append(requirement)\n",
    "\n",
    "    # Step 2\n",
    "    elif len(aList) > 1:\n",
    "        # Step 2a: Handle case \"1. a. or b.\"\n",
    "        if all([sub_item.endswith('or') for sub_item in aList[1:-1]]):\n",
    "            all_sub_requirements = aList[1:]\n",
    "            for sub_requirement in all_sub_requirements:\n",
    "                requirement_subset_id  = sub_requirement[0].upper(); requirement_subset_txt = sub_requirement[2:]\n",
    "                nested_reqmnt = (requirement_id, requirement_txt, requirement_subset_id, requirement_subset_txt)\n",
    "                requirement_list.append(nested_reqmnt)\n",
    "        # Step 2b: Handle case \"3. a. and b.\"\n",
    "        else:\n",
    "            nested_reqmnt = (requirement_id, requirement_txt)\n",
    "            for possibly_a_sub_requirement in aList[1:]:\n",
    "                ## Query the previous item to see if it ends with 'and'. For example, 3. ... and\n",
    "                previous_item_idx = aList.index(possibly_a_sub_requirement)-1\n",
    "                previous_item_txt = aList[previous_item_idx]\n",
    "                if previous_item_txt.endswith('and'):\n",
    "                    sub_requirement_id  = possibly_a_sub_requirement[0].upper()\n",
    "                    sub_requirement_txt = possibly_a_sub_requirement[2:]\n",
    "                    nested_reqmnt      += (sub_requirement_id, None, sub_requirement_txt)\n",
    "                else:\n",
    "                    miscellaneous_txt   = possibly_a_sub_requirement[2:]\n",
    "                    nested_reqmnt      += (miscellaneous_txt, )\n",
    "            requirement_list.append(nested_reqmnt)\n",
    "\n",
    "# Normalize each element in requirement list to length 6\n",
    "requirement_list = [tup+(None, )*(6-len(tup)) for tup in requirement_list]\n",
    "\n",
    "# Display result\n",
    "requirement_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Graduation from an accredited four-year colle...</td>\n",
       "      <td>A</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Graduation from an accredited four-year colle...</td>\n",
       "      <td>A</td>\n",
       "      <td>the development, analysis, implementation or ...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Graduation from an accredited four-year colle...</td>\n",
       "      <td>B</td>\n",
       "      <td>performing cost benefit, feasibility and requ...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>Graduation from an accredited four-year colle...</td>\n",
       "      <td>C</td>\n",
       "      <td>performing system implementation and support ...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>Two years of full-time paid experience as a S...</td>\n",
       "      <td>A</td>\n",
       "      <td>None</td>\n",
       "      <td>Satisfactory completion of four courses, of a...</td>\n",
       "      <td>At least three of the courses must be from th...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0                                                  1  2  \\\n",
       "0  1   Graduation from an accredited four-year colle...  A   \n",
       "1  2   Graduation from an accredited four-year colle...  A   \n",
       "2  2   Graduation from an accredited four-year colle...  B   \n",
       "3  2   Graduation from an accredited four-year colle...  C   \n",
       "4  3   Two years of full-time paid experience as a S...  A   \n",
       "\n",
       "                                                   3  \\\n",
       "0                                               None   \n",
       "1   the development, analysis, implementation or ...   \n",
       "2   performing cost benefit, feasibility and requ...   \n",
       "3   performing system implementation and support ...   \n",
       "4                                               None   \n",
       "\n",
       "                                                   4  \\\n",
       "0                                               None   \n",
       "1                                               None   \n",
       "2                                               None   \n",
       "3                                               None   \n",
       "4   Satisfactory completion of four courses, of a...   \n",
       "\n",
       "                                                   5  \n",
       "0                                               None  \n",
       "1                                               None  \n",
       "2                                               None  \n",
       "3                                               None  \n",
       "4   At least three of the courses must be from th...  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(requirement_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5, 6)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>REQUIREMENT_SET_ID</th>\n",
       "      <th>MINE_1</th>\n",
       "      <th>REQUIREMENT_SUBSET_ID</th>\n",
       "      <th>EXP_JOB_CLASS_FUNCTION</th>\n",
       "      <th>MINE_2</th>\n",
       "      <th>MISC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Graduation from an accredited four-year colle...</td>\n",
       "      <td>A</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Graduation from an accredited four-year colle...</td>\n",
       "      <td>A</td>\n",
       "      <td>the development, analysis, implementation or ...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Graduation from an accredited four-year colle...</td>\n",
       "      <td>B</td>\n",
       "      <td>performing cost benefit, feasibility and requ...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>Graduation from an accredited four-year colle...</td>\n",
       "      <td>C</td>\n",
       "      <td>performing system implementation and support ...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>Two years of full-time paid experience as a S...</td>\n",
       "      <td>A</td>\n",
       "      <td>None</td>\n",
       "      <td>Satisfactory completion of four courses, of a...</td>\n",
       "      <td>At least three of the courses must be from th...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  REQUIREMENT_SET_ID                                             MINE_1  \\\n",
       "0                  1   Graduation from an accredited four-year colle...   \n",
       "1                  2   Graduation from an accredited four-year colle...   \n",
       "2                  2   Graduation from an accredited four-year colle...   \n",
       "3                  2   Graduation from an accredited four-year colle...   \n",
       "4                  3   Two years of full-time paid experience as a S...   \n",
       "\n",
       "  REQUIREMENT_SUBSET_ID                             EXP_JOB_CLASS_FUNCTION  \\\n",
       "0                     A                                               None   \n",
       "1                     A   the development, analysis, implementation or ...   \n",
       "2                     B   performing cost benefit, feasibility and requ...   \n",
       "3                     C   performing system implementation and support ...   \n",
       "4                     A                                               None   \n",
       "\n",
       "                                              MINE_2  \\\n",
       "0                                               None   \n",
       "1                                               None   \n",
       "2                                               None   \n",
       "3                                               None   \n",
       "4   Satisfactory completion of four courses, of a...   \n",
       "\n",
       "                                                MISC  \n",
       "0                                               None  \n",
       "1                                               None  \n",
       "2                                               None  \n",
       "3                                               None  \n",
       "4   At least three of the courses must be from th...  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define a helper function\n",
    "def _base(job, use_df=True):\n",
    "    '''\n",
    "    _BASE attempts to parse the structure in REQUIREMENTS/MINIMUM QUALIFICATIONS. \n",
    "    It takes in a job and returns a dataframe of information, which supports other functions.\n",
    "    \n",
    "    It is intended to use inside other functions (hence started with an underscore), not for stand-alone use.\n",
    "    '''\n",
    "    # Each hash # below is the code in one of the cell above. For output/logic, see above cells.\n",
    "    \n",
    "    # From REQUIREMENTS/MINIMUM QUALIFICATIONS to PROCESS NOTES is where the information located\n",
    "    temp = job[job.find('REQUIREMENTS/MINIMUM QUALIFICATIONS'):job.find('PROCESS NOTES')]\n",
    "    \n",
    "    # Get the digits that mark one part from the other\n",
    "    digits_as_str = [sentence[0] for sentence in temp.split() if sentence[0].isdigit()]\n",
    "    \n",
    "    # Separate requirements one from the other.\n",
    "    line_item = [] # borrow from accounting term. means items in a list\n",
    "    ## Loop through idx of digits_as_str to get the text from one digit to the other\n",
    "    for idx in range(len(digits_as_str)-1):\n",
    "        start = digits_as_str[idx]  ; pos1 = temp.index(start)\n",
    "        end   = digits_as_str[idx+1]; pos2 = temp.index(end)\n",
    "        item  = temp[pos1:pos2]     ; line_item.append(item)\n",
    "    ## Put the last part into the list of all possible requirements\n",
    "    final_dig    = digits_as_str[-1]; final_pos = temp.index(final_dig)\n",
    "    final_item = temp[final_pos:] ; line_item.append(final_item)\n",
    "\n",
    "    # Split further by newline characters\n",
    "    line_item = [[line for line in item.split('\\n') if len(line) > 1] for item in line_item]\n",
    "    \n",
    "    # Implementation of the algorithm\n",
    "    requirement_list = []\n",
    "    for aList in line_item:\n",
    "        ## First item in the list aList may be a requirement\n",
    "        ## requirement_id is the first element in the string\n",
    "        ## requirement_txt is anything after, for example, 1., 2., 3., etc. (2 characters, that's why 2:)\n",
    "        main_information = aList[0]\n",
    "        requirement_id   = main_information[0]; requirement_txt = main_information[2:]\n",
    "        ## Step 1\n",
    "        if len(aList) == 1:\n",
    "            ### Step 1a: Handle case \"1. and 2.\", \"1. or 2.\"\n",
    "            if main_information.endswith('and'):\n",
    "                #### Find next item's requirement id and the text associated with it\n",
    "                next_list_idx   = line_item.index(aList)+1; next_list = line_item[next_list_idx]; next_req = next_list[0]\n",
    "                next_digit      = next_req[0]             ; next_req_txt = next_req[2:]\n",
    "                #### Form a nested requirement and put into requirement list\n",
    "                nested_reqmnt   = (requirement_id, requirement_txt, next_digit, next_req_txt)\n",
    "                requirement_list.append(nested_reqmnt)\n",
    "            ### Step 1b: Handle case \"1. or 2.\"\n",
    "            else:\n",
    "                requirement = (requirement_id, requirement_txt, 'A')\n",
    "                requirement_list.append(requirement)\n",
    "        ## Step 2\n",
    "        elif len(aList) > 1:\n",
    "            ## Step 2a: Handle case \"1. a. or b.\"\n",
    "            if all([sub_item.endswith('or') for sub_item in aList[1:-1]]):\n",
    "                all_sub_requirements = aList[1:]\n",
    "                for sub_requirement in all_sub_requirements:\n",
    "                    requirement_subset_id  = sub_requirement[0].upper(); requirement_subset_txt = sub_requirement[2:]\n",
    "                    nested_reqmnt = (requirement_id, requirement_txt, requirement_subset_id, requirement_subset_txt)\n",
    "                    requirement_list.append(nested_reqmnt)\n",
    "            ## Step 2b: Handle case \"3. a. and b.\"\n",
    "            else:\n",
    "                nested_reqmnt = (requirement_id, requirement_txt)\n",
    "                for possibly_a_sub_requirement in aList[1:]:\n",
    "                    ### Query the previous item to see if it ends with 'and'. For example, 3. ... and\n",
    "                    previous_item_idx = aList.index(possibly_a_sub_requirement)-1\n",
    "                    previous_item_txt = aList[previous_item_idx]\n",
    "                    if previous_item_txt.endswith('and'):\n",
    "                        sub_requirement_id  = possibly_a_sub_requirement[0].upper()\n",
    "                        sub_requirement_txt = possibly_a_sub_requirement[2:]\n",
    "                        nested_reqmnt      += (sub_requirement_id, None, sub_requirement_txt)\n",
    "                    else:\n",
    "                        miscellaneous_txt   = possibly_a_sub_requirement[2:]\n",
    "                        nested_reqmnt      += (miscellaneous_txt, )\n",
    "                requirement_list.append(nested_reqmnt)\n",
    "    ## Normalize each element in requirement list to length 6\n",
    "    requirement_list = [tup+(None, )*(6-len(tup)) for tup in requirement_list]\n",
    "\n",
    "    # Returns\n",
    "    df = pd.DataFrame(data=requirement_list, columns=['REQUIREMENT_SET_ID',\n",
    "                                                      'MINE_1',\n",
    "                                                      'REQUIREMENT_SUBSET_ID',\n",
    "                                                      'EXP_JOB_CLASS_FUNCTION',\n",
    "                                                      'MINE_2',\n",
    "                                                      'MISC'])\n",
    "    return df\n",
    "\n",
    "# Test\n",
    "print(_base(job=sa).shape)\n",
    "_base(job=sa)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now, let's test it on some other jobs\n",
    "* `CARPENTER 3344 011218.txt`\n",
    "* `TREE SURGEON 3114 081916 REV 080818.txt`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CARPENTER 3344 011218.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "94\n"
     ]
    }
   ],
   "source": [
    "# Find ``CARPENTER 3344 011218.txt`\n",
    "print(all_txt_files.index('CARPENTER 3344 011218.txt'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'CARPENTER\\n\\nClass Code: 3344\\nOpen Date: 01-12-18\\n\\t                                 (Exam Open to All, including Current City Employees)\\n\\nANNUAL SALARY\\n\\n$86,312 (flat-rated)\\nThe salary in the Department of Water and Power is $93,542 (flat-rated)\\n\\nNOTES:\\n\\n1. For information regarding reciprocity between the City of Los Angeles departments and LADWP, go to: http://per.lacity.org/Reciprocity_CityDepts_and_DWP.pdf.\\n2. Annual salary is at the start of the pay range. The current salary range is subject to change. Please confirm the starting salary with the hiring department before accepting a job offer.\\n\\nDUTIES\\n\\nA Carpenter does skilled rough and finish carpentry in building and repairing wood structures and articles; work may include forming for the placement of concrete.\\n\\nREQUIREMENTS/MINIMUM QUALIFICATIONS\\n\\n1. Completion of an apprenticeship as a carpenter or cabinetmaker; or\\n2. Six years of full-time paid experience performing carpenter or cabinetmaker work.\\n\\nCompletion of a minimum 18-month or longer program in carpentry or cabinetmaking and millwork from an accredited institution may be substituted for each year of experience lacking up to a maximum of two years.\\n\\nPROCESS NOTES\\n\\n1. Applicants who lack six months or less of the required experience may file for this examination. However, they cannot be appointed until the full experience requirement is met.\\n2. Applicants filing under Requirement #1 must submit a copy of their apprenticeship certificate. Applicants must attach a copy of the Apprenticeship Certificate to the on-line application at the time of filing. The certificate must be attached to the on-line application BEFORE the application is submitted. If the certificate is attached AFTER the on-line application has been submitted, it will not be available to the City and applicants WILL NOT be credited with having submitted the certificate. Applicants who fail to submit the Apprenticeship Certificate at the time of filing by not attaching it to the on-line application will not be considered further in this examination. \\n3. Applicants substituting an 18-month or longer carpentry or cabinetmaking and millwork program for each year of experience lacking up to a maximum of two years must attach a copy of their certificate of completion of the program to the on-line application at the time of filing. The certificate must be attached to the on-line application BEFORE the application is submitted. If the certificate is attached AFTER the on-line application has been submitted, it will not be available to the City and applicants WILL NOT be credited with having submitted the certificate. Applicants who fail to submit the applicable certificate at the time of filing by not attaching it to the on-line application will not be considered further in this examination. \\n4. Some positions may require a valid California driver\\'s license. Candidates may not be eligible for appointment to these positions if their record within the last 36 months reflects three or more moving violations and/or at-fault accidents, or a conviction of a major moving violation (such as DUI). \\n\\nWHERE TO APPLY\\n\\nApplications will only be accepted on-line.  When you are viewing the on-line job bulletin of your choice, simply scroll to the top of the page and select the \"Apply\" icon. On-line job bulletins are also available at https://www.governmentjobs.com/careers/lacity for Open Competitive Examinations and at https://www.governmentjobs.com/careers/lacity/promotionaljobs for Promotional Examinations.\\n\\nNOTE:\\n\\nApplicants are urged to apply early to ensure you have time to resolve any technical issues you may encounter.\\nAPPLICATION DEADLINE\\n\\nApplications must be received by THURSDAY, JANUARY 25, 2018.\\n\\nIn accordance with Civil Service Rule 4.2, all applicants who apply may not be tested in this examination. To meet anticipated hiring needs, only a limited number of qualified applicants will be invited to participate in the weighted multiple-choice test in the following order: 1) Los Angeles City Promotional applicants who meet the minimum requirements. You must have received a regular appointment to a City position or be on a reserve list to apply for this examination as a promotional candidate; 2) Applicants currently employed by the City of Los Angeles on a part-time or exempt basis who meet the minimum requirements; 3) Remaining applicants who meet the minimum requirements in sufficient numbers to meet hiring needs in the order that applications were received. Applications submitted during the filing period will be kept on file for two years from January 12, 2018 in the event that additional applicants need to be tested to meet hiring needs.\\n\\nSELECTION PROCESS\\n\\nAfter meeting minimum qualifications, and in accordance with Rule 4.2 as stated above, candidates will be scheduled for the following:\\n\\nExamination Weight: Multiple-Choice Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100%\\n\\nThe examination will consist entirely of a weighted multiple-choice test administered and proctored on-line. In the on-line multiple-choice test, candidates may be examined for knowledge of: the proper use of hand tools used in the field of carpentry; the component parts of wood and chain link fences and gates; symbols and terminology commonly used on carpentry blueprints and sketches; joints, hardware, and parts as used in conjunction with doors, windows, sashes, frames and other items made of wood; types of wood used in the field of cabinetry; tools used in constructing scaffolds and spans; shoring for trenching; how to use a builder\\'s level; adhesives and techniques used to layout various types of floors and ceilings; various types of suspended ceiling components; connectors and fasteners used for heavy construction; terminology used in roofing, including roofing materials and roof framing; how to construct footings; various types of glass, putties and fasteners; how to install steel stud walls; state safety requirements pertaining to stairways and footings; Cal/OSHA safety codes and personal protective equipment; math including algebra and geometry as it relates to the field of carpentry; and the ability to: read, analyze, and work from blueprints and sketches; and other necessary skills, knowledge, and abilities.\\n\\nThe multiple-choice test will be proctored and administered on-line during a single session. Candidates invited to participate in the on-line multiple-choice test will be able to take the test from any location using a computer with a webcam and a reliable internet connection. Candidates will receive an email from the City of Los Angeles outlining the dates and specific steps on how to take the multiple-choice test on-line. It is anticipated that the remote proctored multiple-choice test will be held in APRIL 2018. Candidates who do not complete and submit the remote proctored multiple-choice test on-line using a computer by the specified date and time will not be considered further in this examination.\\n\\nNOTES:\\n\\n1. This examination is based on a validation study, and as provided by Civil Service Commission Rule 4.20, the multiple-choice test will not be subject to candidate inspection.\\n2. As a covered entity under the Fair Employment and Housing Act and Title II of the Americans with Disabilities Act, the City of Los Angeles does not discriminate on the basis of disability and upon request, will provide reasonable accommodations to ensure equal access to its programs, services, and activities. To request a disability accommodation, please complete the Disability Accommodation Form within 14 calendar days of the submittal of the City application. The Disability Accommodation Form can be obtained at http://per.lacity.org/exams/verify_disability.pdf.\\n3. The incumbents of some positions in this class may be required to pass an annual or periodic work fitness evaluation to determine their eligibility for continued employment in this class.\\n4. Based on the Federal Omnibus Transportation Employee Testing Act of 1994, you may be required to undergo mandatory drug and alcohol testing prior to and during employment in this class.\\n5. In conjunction with Civil Service Rules, applicants who have received a regular appointment to a City position or are on a reserve list will be considered Promotional candidates while all other applicants will be considered Open candidates.\\n6. The promotional list will ordinarily be used ahead of the open competitive list. However, if open competitive candidates receive a higher score, without military credits, than the highest available promotional candidate, after adding 0.25 of a point seniority credit for each year of continuous classified City service, the Civil Service Commission, upon request of the appointing authority, may approve certification of such open competitive candidates ahead of the promotional candidates.\\n7. Applications are accepted subject to review to ensure that minimum qualifications are met. Candidates may be disqualified at any time if it is determined that they do not possess the minimum qualifications stated on this bulletin.\\n8. A final average score of 70% or higher is required to be placed on the eligible list.\\n9. In accordance with Civil Service Rule, Sec. 4.24, review periods may be combined. Candidates in the examination process may file protests as provided in Sec. 4.20, 4.22, and 4.23 as applicable and within the required time frame; however, the Personnel Department may respond to and resolve protests prior to the establishment of the eligible list. \\n\\nNOTICE:\\n\\nIf you receive and accept an offer of employment to a regular position with the City of Los Angeles, your employee benefit coverage (including health and dental coverage as well as life insurance) will commence approximately six weeks after your original regular appointment. Not all positions in the City receive benefit coverage; you should inquire regarding the availability of employee benefits prior to accepting a position.\\n\\nTHIS EXAMINATION IS TO BE GIVEN BOTH ON AN\\nINTERDEPARTMENTAL PROMOTIONAL AND AN OPEN COMPETITIVE BASIS\\n\\nThe City of Los Angeles does not discriminate on the basis of race, religion, national origin, sex, age, marital status, sexual orientation, gender identity, gender expression, disability, creed, color, ancestry, medical condition (cancer), or Acquired Immune Deficiency Syndrome.\\nClass Code 3344. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 01-12-18\\n\\nAN EQUAL EMPLOYMENT OPPORTUNITY EMPLOYER\\n\\nAN EQUAL EMPLOYMENT OPPORTUNITY EMPLOYER\\nFor additional information and FAQs regarding the City\\'s hiring process, please go to: http://per.lacity.org/index.cfm?content=employmenttestingprocess\\n\\n\\n\\n\\n\\n'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import file as text\n",
    "carpenter_path = path + all_txt_files[94]\n",
    "carpenter = open(carpenter_path, 'rt').read()\n",
    "carpenter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "list index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-17-75610aef4fa9>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Test\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0m_base\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjob\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcarpenter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-12-21a14eafe168>\u001b[0m in \u001b[0;36m_base\u001b[0;34m(job, use_df)\u001b[0m\n\u001b[1;32m     35\u001b[0m         \u001b[0;31m## requirement_id is the first element in the string\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     36\u001b[0m         \u001b[0;31m## requirement_txt is anything after, for example, 1., 2., 3., etc. (2 characters, that's why 2:)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 37\u001b[0;31m         \u001b[0mmain_information\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0maList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     38\u001b[0m         \u001b[0mrequirement_id\u001b[0m   \u001b[0;34m=\u001b[0m \u001b[0mmain_information\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m;\u001b[0m \u001b[0mrequirement_txt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmain_information\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     39\u001b[0m         \u001b[0;31m## Step 1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mIndexError\u001b[0m: list index out of range"
     ]
    }
   ],
   "source": [
    "# Test\n",
    "_base(job=carpenter)\n",
    "\n",
    "# Oops!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'REQUIREMENTS/MINIMUM QUALIFICATIONS\\n\\n1. Completion of an apprenticeship as a carpenter or cabinetmaker; or\\n2. Six years of full-time paid experience performing carpenter or cabinetmaker work.\\n\\nCompletion of a minimum 18-month or longer program in carpentry or cabinetmaking and millwork from an accredited institution may be substituted for each year of experience lacking up to a maximum of two years.\\n\\n'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Locate the information\n",
    "temp = carpenter[carpenter.find('REQUIREMENTS/MINIMUM QUALIFICATIONS'):carpenter.find('PROCESS NOTES')]\n",
    "temp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1', '2', '1']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the digits that mark one part from the other\n",
    "digits_as_str = [sentence[0] for sentence in temp.split() if sentence[0].isdigit()]\n",
    "digits_as_str"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1', '2']"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fix the above problem\n",
    "digits_as_str = [sentence[-1] for sentence in temp.split('.') if sentence[-1].isdigit()]\n",
    "digits_as_str"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1. Completion of an apprenticeship as a carpenter or cabinetmaker; or\\n',\n",
       " '2. Six years of full-time paid experience performing carpenter or cabinetmaker work.\\n\\nCompletion of a minimum 18-month or longer program in carpentry or cabinetmaking and millwork from an accredited institution may be substituted for each year of experience lacking up to a maximum of two years.\\n\\n']"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Separate requirements one from the other.\n",
    "## Initialization\n",
    "line_item = [] # borrow from accounting term. means items in a list\n",
    "## Loop through idx of digits_as_str to get the text from one digit to the other\n",
    "for idx in range(len(digits_as_str)-1):\n",
    "    start = digits_as_str[idx]  ; pos1 = temp.index(start)\n",
    "    end   = digits_as_str[idx+1]; pos2 = temp.index(end)\n",
    "    item  = temp[pos1:pos2]     ; line_item.append(item)\n",
    "## Put the last part into the list of all possible requirements\n",
    "final_dig    = digits_as_str[-1]; final_pos = temp.index(final_dig)\n",
    "final_item = temp[final_pos:] ; line_item.append(final_item)\n",
    "## Display results\n",
    "line_item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['1. Completion of an apprenticeship as a carpenter or cabinetmaker; or'],\n",
       " ['2. Six years of full-time paid experience performing carpenter or cabinetmaker work.',\n",
       "  'Completion of a minimum 18-month or longer program in carpentry or cabinetmaking and millwork from an accredited institution may be substituted for each year of experience lacking up to a maximum of two years.']]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Split further by newline characters\n",
    "line_item = [[line for line in item.split('\\n') if len(line) > 1] for item in line_item]\n",
    "line_item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('1',\n",
       "  ' Completion of an apprenticeship as a carpenter or cabinetmaker; or',\n",
       "  'A',\n",
       "  None,\n",
       "  None,\n",
       "  None),\n",
       " ('2',\n",
       "  ' Six years of full-time paid experience performing carpenter or cabinetmaker work.',\n",
       "  'C',\n",
       "  'mpletion of a minimum 18-month or longer program in carpentry or cabinetmaking and millwork from an accredited institution may be substituted for each year of experience lacking up to a maximum of two years.',\n",
       "  None,\n",
       "  None)]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "requirement_list = []\n",
    "for aList in line_item:\n",
    "    # First item in the list aList may be a requirement\n",
    "    # requirement_id is the first element in the string\n",
    "    # requirement_txt is anything after, for example, 1., 2., 3., etc. (2 characters, that's why 2:)\n",
    "    main_information = aList[0]\n",
    "    requirement_id   = main_information[0]; requirement_txt = main_information[2:]\n",
    "    # Step 1\n",
    "    if len(aList) == 1:\n",
    "        ## Step 1a: Handle case \"1. and 2.\", \"1. or 2.\"\n",
    "        if main_information.endswith('and'):\n",
    "            ### Find next item's requirement id and the text associated with it\n",
    "            next_list_idx   = line_item.index(aList)+1; next_list = line_item[next_list_idx]; next_req = next_list[0]\n",
    "            next_digit      = next_req[0]             ; next_req_txt = next_req[2:]\n",
    "            ### Form a nested requirement and put into requirement list\n",
    "            nested_reqmnt   = (requirement_id, requirement_txt, next_digit, next_req_txt)\n",
    "            requirement_list.append(nested_reqmnt)\n",
    "        ## Step 1b: Handle case \"1. or 2.\"\n",
    "        else:\n",
    "            requirement = (requirement_id, requirement_txt, 'A')\n",
    "            requirement_list.append(requirement)\n",
    "\n",
    "    # Step 2\n",
    "    elif len(aList) > 1:\n",
    "        # Step 2a: Handle case \"1. a. or b.\"\n",
    "        if all([sub_item.endswith('or') for sub_item in aList[1:-1]]):\n",
    "            all_sub_requirements = aList[1:]\n",
    "            for sub_requirement in all_sub_requirements:\n",
    "                requirement_subset_id  = sub_requirement[0].upper(); requirement_subset_txt = sub_requirement[2:]\n",
    "                nested_reqmnt = (requirement_id, requirement_txt, requirement_subset_id, requirement_subset_txt)\n",
    "                requirement_list.append(nested_reqmnt)\n",
    "        # Step 2b: Handle case \"3. a. and b.\"\n",
    "        else:\n",
    "            nested_reqmnt = (requirement_id, requirement_txt)\n",
    "            for possibly_a_sub_requirement in aList[1:]:\n",
    "                ## Query the previous item to see if it ends with 'and'. For example, 3. ... and\n",
    "                previous_item_idx = aList.index(possibly_a_sub_requirement)-1\n",
    "                previous_item_txt = aList[previous_item_idx]\n",
    "                if previous_item_txt.endswith('and'):\n",
    "                    sub_requirement_id  = possibly_a_sub_requirement[0].upper()\n",
    "                    sub_requirement_txt = possibly_a_sub_requirement[2:]\n",
    "                    nested_reqmnt      += (sub_requirement_id, None, sub_requirement_txt)\n",
    "                else:\n",
    "                    miscellaneous_txt   = possibly_a_sub_requirement[2:]\n",
    "                    nested_reqmnt      += (miscellaneous_txt, )\n",
    "            requirement_list.append(nested_reqmnt)\n",
    "\n",
    "# Normalize each element in requirement list to length 6\n",
    "requirement_list = [tup+(None, )*(6-len(tup)) for tup in requirement_list]\n",
    "\n",
    "# Display result\n",
    "requirement_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2, 6)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>REQUIREMENT_SET_ID</th>\n",
       "      <th>MINE_1</th>\n",
       "      <th>REQUIREMENT_SUBSET_ID</th>\n",
       "      <th>EXP_JOB_CLASS_FUNCTION</th>\n",
       "      <th>MINE_2</th>\n",
       "      <th>MISC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Completion of an apprenticeship as a carpente...</td>\n",
       "      <td>A</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Six years of full-time paid experience perfor...</td>\n",
       "      <td>C</td>\n",
       "      <td>mpletion of a minimum 18-month or longer progr...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  REQUIREMENT_SET_ID                                             MINE_1  \\\n",
       "0                  1   Completion of an apprenticeship as a carpente...   \n",
       "1                  2   Six years of full-time paid experience perfor...   \n",
       "\n",
       "  REQUIREMENT_SUBSET_ID                             EXP_JOB_CLASS_FUNCTION  \\\n",
       "0                     A                                               None   \n",
       "1                     C  mpletion of a minimum 18-month or longer progr...   \n",
       "\n",
       "  MINE_2  MISC  \n",
       "0   None  None  \n",
       "1   None  None  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Re-define a helper function. In particular, fix the part at digits_as_str\n",
    "def _base(job, use_df=True):\n",
    "    '''\n",
    "    _BASE attempts to parse the structure in REQUIREMENTS/MINIMUM QUALIFICATIONS. \n",
    "    It takes in a job and returns a dataframe of information, which supports other functions.\n",
    "    \n",
    "    It is intended to use inside other functions (hence started with an underscore), not for stand-alone use.\n",
    "    '''\n",
    "    # Each hash # below is the code in one of the cell above. For output/logic, see above cells.\n",
    "    \n",
    "    # From REQUIREMENTS/MINIMUM QUALIFICATIONS to PROCESS NOTES is where the information located\n",
    "    temp = job[job.find('REQUIREMENTS/MINIMUM QUALIFICATIONS'):job.find('PROCESS NOTES')]\n",
    "    \n",
    "    # Get the digits that mark one part from the other\n",
    "    digits_as_str = [sentence[-1] for sentence in temp.split('.') if sentence[-1].isdigit()]\n",
    "    \n",
    "    # Separate requirements one from the other.\n",
    "    line_item = [] # borrow from accounting term. means items in a list\n",
    "    ## Loop through idx of digits_as_str to get the text from one digit to the other\n",
    "    for idx in range(len(digits_as_str)-1):\n",
    "        start = digits_as_str[idx]  ; pos1 = temp.index(start)\n",
    "        end   = digits_as_str[idx+1]; pos2 = temp.index(end)\n",
    "        item  = temp[pos1:pos2]     ; line_item.append(item)\n",
    "    ## Put the last part into the list of all possible requirements\n",
    "    final_dig    = digits_as_str[-1]; final_pos = temp.index(final_dig)\n",
    "    final_item = temp[final_pos:] ; line_item.append(final_item)\n",
    "\n",
    "    # Split further by newline characters\n",
    "    line_item = [[line for line in item.split('\\n') if len(line) > 1] for item in line_item]\n",
    "    \n",
    "    # Implementation of the algorithm\n",
    "    requirement_list = []\n",
    "    for aList in line_item:\n",
    "        ## First item in the list aList may be a requirement\n",
    "        ## requirement_id is the first element in the string\n",
    "        ## requirement_txt is anything after, for example, 1., 2., 3., etc. (2 characters, that's why 2:)\n",
    "        main_information = aList[0]\n",
    "        requirement_id   = main_information[0]; requirement_txt = main_information[2:]\n",
    "        ## Step 1\n",
    "        if len(aList) == 1:\n",
    "            ### Step 1a: Handle case \"1. and 2.\", \"1. or 2.\"\n",
    "            if main_information.endswith('and'):\n",
    "                #### Find next item's requirement id and the text associated with it\n",
    "                next_list_idx = line_item.index(aList)+1; next_list = line_item[next_list_idx]; next_req=next_list[0]\n",
    "                next_digit    = next_req[0]             ; next_req_txt = next_req[2:]\n",
    "                #### Form a nested requirement and put into requirement list\n",
    "                nested_reqmnt   = (requirement_id, requirement_txt, next_digit, next_req_txt)\n",
    "                requirement_list.append(nested_reqmnt)\n",
    "            ### Step 1b: Handle case \"1. or 2.\"\n",
    "            else:\n",
    "                requirement = (requirement_id, requirement_txt, 'A')\n",
    "                requirement_list.append(requirement)\n",
    "        ## Step 2\n",
    "        elif len(aList) > 1:\n",
    "            ## Step 2a: Handle case \"1. a. or b.\"\n",
    "            if all([sub_item.endswith('or') for sub_item in aList[1:-1]]):\n",
    "                all_sub_requirements = aList[1:]\n",
    "                for sub_requirement in all_sub_requirements:\n",
    "                    requirement_subset_id  = sub_requirement[0].upper(); requirement_subset_txt = sub_requirement[2:]\n",
    "                    nested_reqmnt = (requirement_id, requirement_txt, requirement_subset_id, requirement_subset_txt)\n",
    "                    requirement_list.append(nested_reqmnt)\n",
    "            ## Step 2b: Handle case \"3. a. and b.\"\n",
    "            else:\n",
    "                nested_reqmnt = (requirement_id, requirement_txt)\n",
    "                for possibly_a_sub_requirement in aList[1:]:\n",
    "                    ### Query the previous item to see if it ends with 'and'. For example, 3. ... and\n",
    "                    previous_item_idx = aList.index(possibly_a_sub_requirement)-1\n",
    "                    previous_item_txt = aList[previous_item_idx]\n",
    "                    if previous_item_txt.endswith('and'):\n",
    "                        sub_requirement_id  = possibly_a_sub_requirement[0].upper()\n",
    "                        sub_requirement_txt = possibly_a_sub_requirement[2:]\n",
    "                        nested_reqmnt      += (sub_requirement_id, None, sub_requirement_txt)\n",
    "                    else:\n",
    "                        miscellaneous_txt   = possibly_a_sub_requirement[2:]\n",
    "                        nested_reqmnt      += (miscellaneous_txt, )\n",
    "                requirement_list.append(nested_reqmnt)\n",
    "    ## Normalize each element in requirement list to length 6\n",
    "    requirement_list = [tup+(None, )*(6-len(tup)) for tup in requirement_list]\n",
    "\n",
    "    # Returns\n",
    "    df = pd.DataFrame(data=requirement_list, columns=['REQUIREMENT_SET_ID',\n",
    "                                                      'MINE_1',\n",
    "                                                      'REQUIREMENT_SUBSET_ID',\n",
    "                                                      'EXP_JOB_CLASS_FUNCTION',\n",
    "                                                      'MINE_2',\n",
    "                                                      'MISC'])\n",
    "    return df\n",
    "\n",
    "# Test\n",
    "print(_base(job=carpenter).shape)\n",
    "_base(job=carpenter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=8, color='red'> Quite messed up. Requires a new approach! </font>\n",
    "See Notebook `Object1_e`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}