Objective1_a.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get experience job class title\n",
    "This was darn hard to me. For example, how do you know words like, Management Assistant, is there to look up? All of a sudden, I remember that we can scrape the pdf files in City Job Paths using Bob's code! Then, I can build a list of keys for job class title. Finally, all I need to do is to do a look up.\n",
    "\n",
    "This is more like a trial/error process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, import relevant modules\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#from https://www.blog.pythonlibrary.org/2018/05/03/exporting-data-from-pdfs-with-python/\n",
    "\n",
    "#install pdfminer on python 3\n",
    "#!python -m pip install pdfminer.six \n",
    "\n",
    "import io\n",
    "from pdfminer.converter import TextConverter\n",
    "from pdfminer.pdfinterp import PDFPageInterpreter\n",
    "from pdfminer.pdfinterp import PDFResourceManager\n",
    "from pdfminer.pdfpage import PDFPage\n",
    "\n",
    "def extract_text_from_pdf(pdf_path):\n",
    "    resource_manager = PDFResourceManager()\n",
    "    fake_file_handle = io.StringIO()\n",
    "    converter = TextConverter(resource_manager, fake_file_handle)\n",
    "    page_interpreter = PDFPageInterpreter(resource_manager, converter)\n",
    " \n",
    "    with open(pdf_path, 'rb') as fh:\n",
    "        for page in PDFPage.get_pages(fh, \n",
    "                                      caching=True,\n",
    "                                      check_extractable=True):\n",
    "            page_interpreter.process_page(page)\n",
    " \n",
    "        text = fake_file_handle.getvalue()\n",
    " \n",
    "    # close open handles\n",
    "    converter.close()\n",
    "    fake_file_handle.close()\n",
    " \n",
    "    if text:\n",
    "        return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'CITY OF LOS ANGELES PERSONNEL DEPARTMENT CAREER-OPPORTUNITIES FOR ACCOUNTANT The following information is being given to describe potential opportunities as an Accountant.  The career ladders that these titles commonly follow have been illustrated in the diagram below.  With specific types of experience, promotional or lateral movement between these lines is also possible.  You may review the class specifications and some job bulletins through our Personnel’s Department website.  It is encouraged to examine the options available, to be able to promote for what you qualify for.                  Principal Utility Accountant (Water and Power) Chief Internal Auditor Principal Accountant Fiscal System Specialist Senior Utility Accountant (Water and Power) Internal Auditor Senior Auditor Senior Tax Auditor Senior Accountant Auditor Tax Auditor Accountant Departmental Chief Accountant Financial Manager Accounting Aide Accounting Records Supervisor Payroll Supervisor Accounting Clerk \\uf0a7 Open/Entry-Level \\uf0a7 Promotional Utility Accountant (Water and Power)  Principal Tax Auditor (Water and Power) Chief Auditor Controller (Water and Power) \\x0c'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's do an example\n",
    "extract_text_from_pdf('CityofLA/Additional data/City Job Paths/Accountant.pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice! Note that we can ignore:\n",
    "* \"CITY OF LOS ANGELES PERSEONEL... qualify for.\"\n",
    "* \\uf0a7 \n",
    "* Open/Entry-Level\n",
    "* \\x0c\n",
    "\n",
    "Next, we'll use `nltk` to build a list of keys for `EXP_JOB_CLASS_TITLE`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "import string\n",
    "\n",
    "# Build a list of stop words. These includes: (1) English stop words, (2) punctuation, \n",
    "# and (3) the last 3 bullet points above.\n",
    "stop = stopwords.words('english') + list(string.punctuation) + ['\\uf0a7', 'Open/Entry-Level', '\\x0c']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "folder_path = 'CityofLA/Additional data/City Job Paths/'\n",
    "all_pdf_filenames = os.listdir(folder_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['specifications', 'AuditorSenior', 'Pilot', 'Financial', 'Transmission', 'titles', 'Vehicle', 'Bindery', 'Heavy', 'Examiner', 'Park', 'ExaminerIPolygraph', 'Treatment', 'Land', 'Auditor', 'Technical', 'options', 'Buyer', 'Engineering', 'Commission', 'Bureau', 'Air', 'available', 'Zoo', 'Reader', 'Stores', 'City', 'information', 'Toolroom', 'Motor', 'Golf', 'Cleaning', 'Warehouse', 'Operating', 'Body', 'Plant', 'Water', 'Secretary', 'Senior', 'Waste', 'Tax', 'Material', 'Associate', 'Commercial', 'Drafting', 'Officer', 'Refrigeration', 'Analyst', 'Personnel', 'Supply', 'You', 'bulletins', 'Deck', 'Worker', 'Waterworks', 'Zoning', 'InspectorSenior', 'Gardener', '\\uf0a7Open/Entry-Level\\uf0a7PromotionalLocksmithBuilding', 'Technician', 'Metal', 'Cartographer', 'Cleaner', 'given', 'potential', 'Biologist', 'Crew', 'Birds', 'Sales', 'Supervising', 'able', 'Delivery', 'Sanitation', 'examine', 'IVPolygraph', 'IIPolygraph', 'also', 'Fleet', 'Principal', 'job', 'Forensic', 'Executive', 'FIREFIGHTER', 'Port', 'Captain', 'Printing', 'Lighting', 'cid:131', '4', 'Representative', 'Steam', 'Detention', 'Press', 'Cement', 'Title', 'describe', 'Structural', 'IIMaintenance', 'Repairer', 'Service', 'Curator', 'Manager', 'Print', 'Management', 'Line', 'Education', 'Open/Entry‐Level', 'OperatorSenior', 'experience', 'illustrated', 'Sr', 'System', 'Administrator', 'Collection', 'Property', 'Processing', 'Grounds', 'III', 'Light', 'Internal', 'Community', 'Engineer', 'Window', 'Event', 'Custodian', 'Research', 'Mechanic', 'Attendant', 'Typist', 'Street', 'Industrial', 'Project', 'Parking', 'Fireboat', 'Apprentice', 'Geologist', 'Caretaker', 'Party', 'Occupational', 'Sergeant', 'PERSONNEL', 'Chief', 'Wastewater', 'Messenger', 'Vessel', 'Plumbing', 'Craft', 'CAREER‐OPPORTUNITIES', 'types', 'Shovel', 'Public', 'lateral', 'IIIInformation', 'Boat', 'Planner', 'Communications', 'Maker', 'Fire', 'Inspector', 'Office', 'WorkerAirports', 'Veterinary', 'Reptiles', 'Electrician', 'TechnicianComputer', 'Pipefitter', 'Auto', 'Duplicating', 'Events', 'Assistant', 'Sign', 'diagram', 'IV', 'Estate', 'Detective', 'Clerk', 'Open', 'Security', 'Distribution', 'Station', 'Mechanical', 'following', 'opportunities', 'Elevator', 'Animal', 'Promotional', 'Poster', 'Survey', '\\uf0a7Open/Entry-Level\\uf0a7PromotionalSenior', 'Equipment', 'Power', 'movement', 'Operator', 'Graphics', 'Heating', 'Trainee', 'Computer', 'Departmental', 'Mate', 'Care', 'Procurement', 'Lot', 'Storekeeper', 'Architectural', 'Sr.', 'Helper', 'ladders', 'PromotionalUtility', 'Services', 'InspectorChief', 'Load', 'Electric', 'Systems', 'lines', 'Duty', 'Keeper', 'Workers', 'Repair', 'II', 'IIIPainterCement', 'Years', 'Instrument', 'Wharfinger', 'Environmental', 'Cabinet', 'Surveys', 'RepairerI', '\\uf0a7Open\\uf0a7PromotionalLaboratory', 'EngineerTraffic', 'Helicopter', 'Director', 'Carpenter', 'Compliance', 'Maintenance', 'Elevators', 'Photographer', 'Pressure', 'Custodial', 'IISenior', 'Head', 'Building', 'Sweeper', 'V', 'Resources', 'Operations', 'specific', 'Accounting', 'Sheet', 'Vessels', 'Machine', 'District', 'Control', 'Records', 'FOR', 'Surveying', 'Shop', 'Irrigation', 'Truck', 'Commander', 'Designer', 'Compensation', 'Nurse', 'follow', 'S', 'Safety', 'With', 'Programs', 'Marking', 'Health', 'Meter', 'Civil', 'Machinist', 'Masonry', 'Painter', 'Advisor', 'possible', 'Apparatus', 'Starter', 'Relations', 'Materials', 'Claims', 'Accountant', 'Transportation', 'Quality', 'The', 'Labor', 'Aide', 'HelperBuilding', 'qualify', 'Laboratory', 'Housing', 'Traffic', 'Dispatcher', 'Managing', 'Builder', 'Rehabilitation', 'Stenographer', 'Specialist', 'General', 'Supervisor', 'Plumber', 'Recreation', 'Disposal', 'Finisher', 'Pumping', 'Police', 'Administrative', 'DEPARTMENT', 'Automotive', 'Information', 'Tree', 'ANGELES', 'Solid', 'Superintendent', 'Ranger', 'Protection', 'It', 'Planning', 'Firefighter', 'Harbor', 'Laborer', 'Utility', 'career', 'Data', 'Aid', 'Battalion', 'B', 'Librarian', 'Bus', 'promotional', 'Airport', 'Fiscal', 'Controller', 'Conditioning', 'Electrical', 'Hand', 'Coordinator', 'Testing', 'Telecommunications', 'Instructor', 'Division', 'Lieutenant', 'Airports', 'Marketing', 'A', 'may', 'Compressor', 'website', '-or-', 'commonly', 'TechnicianEnvironmental', 'class', 'Programmer', 'Development', 'Construction', 'Deputy', 'Special', 'Surgeon', '\\uf0a7Open/Entry-Level\\uf0a7PromotionalPolygraph', 'Driver', 'Applications', 'Payroll', 'Facility', 'review', 'promote', 'Department', 'Refuse', 'Garage', 'Signal', 'I', 'Field', 'encouraged', 'Real', '’', 'Geographic']\n"
     ]
    }
   ],
   "source": [
    "#%%timeit. Weird, this makes an infinite loop???\n",
    "exp_job_class_title_keywords = []\n",
    "for filename in all_pdf_filenames:\n",
    "    # Get pdf path\n",
    "    pdf_path = folder_path + filename\n",
    "    # Convert pdf to text using helper function\n",
    "    job_path_as_text = extract_text_from_pdf(pdf_path)\n",
    "    # Simplify text per Note above\n",
    "    job_path_as_text_simplified = job_path_as_text[job_path_as_text.find('qualify for')+len('qualify for'):]\n",
    "    # Tokenize text if key is not in the pre-defined stop words\n",
    "    tokens = [key for key in word_tokenize(job_path_as_text_simplified) if key not in stop]\n",
    "    # Append to exp_job_class_title_keywords. Use set to remove dups in list. list again to make it a list.\n",
    "    exp_job_class_title_keywords = list(set(exp_job_class_title_keywords + tokens))\n",
    "    \n",
    "print(exp_job_class_title_keywords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from the result, `extract_text_from_pdf` does not always work. There are two things we can do here: (1) tweak code to simplify the result (to avoid words like 'ANGELES'), and (2) manually adjust the list above to have a cleaner one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "import string\n",
    "\n",
    "# Build a list of stop words. These includes: (1) English stop words, (2) punctuation, \n",
    "# and (3) the last 3 bullet points above.\n",
    "stop = stopwords.words('english') + list(string.punctuation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initializations\n",
    "exp_job_class_title_keywords = []\n",
    "ignored_text1 = 'CITY OF LOS ANGELES PERSONNEL DEPARTMENT CAREER-OPPORTUNITIES FOR'\n",
    "ignored_text2 = 'The following information is being given to describe potential opportunities as'\n",
    "ignored_text3 = 'The career ladders that these titles commonly follow have been illustrated in the diagram below'  \n",
    "ignored_text4 = 'With specific types of experience, promotional or lateral movement between these lines is also possible'\n",
    "ignored_text5 = 'You may review the class specifications and some job bulletins through our Personnel’s Department website.'\n",
    "ignored_text6 = 'It is encouraged to examine the options available, to be able to promote for what you qualify for'\n",
    "ignored_text7 = '\\uf0a7'\n",
    "ignored_text8 = 'Open/'\n",
    "ignored_text9 = 'Entry-Level'\n",
    "ignored_text10 = 'Promotional'\n",
    "ignored_text11 = '\\x0c'\n",
    "\n",
    "ignored_list = [ignored_text1, ignored_text2, ignored_text3, ignored_text4, ignored_text5, \n",
    "                ignored_text6, ignored_text7, ignored_text8, ignored_text9, ignored_text10, ignored_text11]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['steam', 'waste', 'specifications', 'body', 'angeles', 'harbor', 'waterworks', 'titles', 'custodial', 'systems', 'police', 'apparatus', 'polygraph', 'land', 'ivpolygraph', 'deputy', 'tax', 'distribution', 'operations', 'grounds', 'treatment', 'options', 'reader', 'available', 'associate', 'shop', 'crew', 'information', 'claims', 'worker', 'occupational', 'drafting', 'bus', 'maker', 'truck', 'officer', 'coordinator', 'applications', 'firefighter', 'workers', 'sergeant', 'engineer', 'real', 'specialist', 'attendant', 'testing', 'iimaintenance', 'duplicating', 'utilization', 'forensic', 'pilot', 'bulletins', 'poster', 'metal', 'iisenior', 'division', 'elevators', 'driver', 'golf', 'craft', 'director', 'given', 'potential', 'lot', 'managing', 'manager', 'instructor', 'able', 'heating', 'district', 'instrument', 'examine', 'locksmithbuilding', 'also', 'machine', 'job', 'technicianenvironmental', 'auditorsenior', 'messenger', 'cleaning', 'power', 'boat', 'planner', 'administrator', 'laboratory', 'computer', 'cid:131', '4', 'marking', 'finisher', 'describe', 'station', 'marketing', 'caretaker', 'representative', 'vehicle', 'window', 'garage', 'material', 'storekeeper', 'disposal', 'recreation', 'transmission', 'illustrated', 'experience', 'bureau', 'entry‐level', 'accountant', 'departmental', 'commander', 'helicopter', 'zoo', 'property', 'controller', 'research', 'engineertraffic', 'warehouse', 'supervisor', 'los', 'years', 'construction', 'compensation', 'lieutenant', 'openlaboratory', 'utility', 'signal', 'dispatcher', 'engineering', 'graphics', 'system', 'sanitation', 'street', 'birds', 'services', 'formerly', 'types', 'field', 'records', 'transportation', 'payroll', 'aide', 'surgeon', 'lateral', 'printing', 'airports', 'structural', 'helperbuilding', 'reptiles', 'health', 'veterinary', 'auditor', 'analyst', 'sr.', 'processing', 'starter', 'directo', 'stenographer', 'geologist', 'maintenance', 'refrigeration', 'planning', 'nurse', 'senior', 'lighting', 'aid', 'general', 'cleaner', 'diagram', 'relations', 'ii', 'detective', 'labor', 'office', 'following', 'repair', 'opportunities', 'management', 'apprentice', 'shovel', 'biologist', 'sheet', 'print', 'safety', 'movement', 'captain', 'b', 'line', 'battalion', 'fireboat', 'mechanical', 'iipolygraph', 'repaireri', 'builder', 'development', 'gardener', 'traffic', 'quality', 'protection', 'ladders', 'supply', 'wastewater', 'cartographer', 'associates', 'department', 'compressor', 'inspectorsenior', 'internal', 'lines', 'inspectorchief', 'examineripolygraph', 'clerk', 'surveying', 'reprsentative', 'assistant', 'delivery', 'examiner', 'automotive', 'secretary', 'rehabilitation', 'architectural', 'title', 'chief', 'technical', 'machinist', 'community', 'mechanic', 'equipment', 'building', 'heavy', 'curator', 'commission', 'motor', 'specific', 'departmentcareer-opportunities', 'data', 'career‐opportunities', 'bindery', 'cabinet', 'supervising', 'detention', 'fleet', 'iiipaintercement', 'party', 'auditor..', 'operating', 'programs', 'follow', 'auto', 'photographer', 'locksmith', 'park', 'painter', 'plumber', 'event', 'press', 'open', 'telecommunications', 'possible', 'compliance', 'light', 'load', 'custodian', 'carpenter', 'librarian', 'electric', 'helper', 'care', 'parking', 'repairer', 'communications', 'keeper', 'project', 'pumping', 'qualify', 'electrical', 'pressure', 'toolroom', 'duty', 'electrician', 'administrative', 'events', 'vessel', 'plant', 'fire', 'geographic', 'service', 'mate', 'special', 'transporation', 'buyer', 'solid', 'executive', 'port', 'techniciancomputer', 'wharfinger', 'fiscal', 'personnel', 'facility', 'tree', 'pipefitter', 'irrigation', 'water', 'sign', 'designer', 'sales', 'public', 'ranger', 'security', 'financial', 'civil', 'operatorsenior', 'air', 'career', 'accounting', 'materials', 'hand', 'animal', 'survey', 'promotional', 'advisor', 'zoning', 'commercial', 'deck', 'education', 'resources', 'sweeper', 'workerairports', 'masonry', 'environmental', 'typist', 'superintendent', 'industrial', 'may', 'stores', 'meter', 'trainee', 'vessels', 'programmer', 'website', '-or-', 'commonly', 'procurement', 'housing', 'class', 'estate', 'airport', 'operator', 'iv', 'surveys', 'conditioning', 'iii', 'v', 'technician', 'elevator', 'control', 'promote', 'review', 'principal', 'plumbing', 'inspector', 'collection', 'head', 'cement', 'angelespersonnel', 'laborer', 'city', 'encouraged', 'iiiinformation', '’', 'refuse']\n"
     ]
    }
   ],
   "source": [
    "#%%timeit. Weird, this makes an infinite loop???\n",
    "exp_job_class_title_keywords = []\n",
    "for filename in all_pdf_filenames:\n",
    "    # Get pdf path\n",
    "    pdf_path = folder_path + filename\n",
    "    # Convert pdf to text using helper function\n",
    "    job_path_as_text = extract_text_from_pdf(pdf_path)\n",
    "    # Simplify text\n",
    "    for ig in ignored_list:\n",
    "        job_path_as_text = job_path_as_text.replace(ig, '')\n",
    "    # Tokenize text if key is not in the pre-defined stop words\n",
    "    tokens = [key for key in word_tokenize(job_path_as_text.lower()) if key not in stop]\n",
    "    # Append to exp_job_class_title_keywords. Use set to remove dups in list. list again to make it a list.\n",
    "    exp_job_class_title_keywords = list(set(exp_job_class_title_keywords + tokens))\n",
    "    \n",
    "print(exp_job_class_title_keywords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# See Bob, I can do it too!\n",
    "This is not helpful at all, so I'll come back to the first list, copy/paste it to a word file, and manually remove inappropriate words to have a cleaned list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "326\n"
     ]
    }
   ],
   "source": [
    "l = ['Polygraph', 'Grounds', 'Street', 'Starter','Traffic', 'Inspector', 'Plumbing', 'Officer', 'Helper', \n",
    "     'Accountant', 'Claims', 'Records', 'Community', 'Typist', 'Land', 'Resources','Warehouse', 'Designer', \n",
    "     'Irrigation', 'Air', 'Meter', 'Security', 'Technician', 'Marketing', 'Reader', 'Operator', 'Window', \n",
    "     'Analyst', 'Ranger', 'Transportation', 'Cleaner', '4', 'Engineer', 'Technician’, ‘Environmental', \n",
    "     'Auditor’, ’Senior’, ‘Operator’, ‘Senior', 'Electrical', 'Laborer', 'Airports', 'Internal', 'Planner', \n",
    "     'Elevator', 'Recreation', 'Principal', 'Executive', 'Mate', 'Communications', 'Sanitation', 'Attendant', \n",
    "     'Surveying', 'Associate', 'Surveys', 'Commander', 'Auto', 'Animal', 'Forensic', 'Poster', 'Utility', \n",
    "     'Steam', 'Locksmith’, Building', 'Materials', 'Fireboat', 'Helicopter', 'Driver', 'Drafting', 'Delivery', \n",
    "     'Director', 'Birds', 'Heating', 'Structural', 'Line', 'City', 'Industrial', 'Gardener', 'Body', 'Sign', \n",
    "     'Compliance', 'Housing', 'Safety', 'Equipment', 'Quality', 'Conditioning', 'Masonry', 'Geologist', 'Maker', \n",
    "     'Services', 'Lieutenant', 'Civil', 'Deck', 'Computer', 'Apparatus', 'Environmental', 'Dispatcher', \n",
    "     'Worker’, ‘Airports', 'Division', 'Information', 'Sales', 'Chief', 'Estate', 'Aid', 'Telecommunications', \n",
    "     'Automotive', 'Coordinator', 'Departmental', 'Financial','Mechanic', 'Instrument', 'Operating', 'Lighting', \n",
    "     'Superintendent', 'Commercial', 'Assistant', 'Fleet', 'Biologist', 'Programmer', 'Detective', 'Managing', \n",
    "     'Development', 'Sweeper', 'Pilot', 'Workers', 'Personnel', 'Reptiles', 'Inspector', 'Senior', 'Craft', \n",
    "     'Print', 'Inspector', 'Chief', 'Electric', 'Representative', 'Trainee', 'Station', 'Administrative', \n",
    "     'Worker', 'Party', 'Administrator', 'Engineer', 'Traffic', 'Laboratory', 'Technical', 'Office', 'Cabinet', \n",
    "     'Truck', 'Pipefitter', 'Utility', 'Fire', 'IV', 'Polygraph', 'Rehabilitation', 'Helper', 'Building', \n",
    "     'Power', 'Aide', 'Sheet', 'Property','Zoo', 'General', 'Engineering', 'Cartographer', 'Construction', \n",
    "     'Builder', 'Police', 'Technician', 'Computer', 'Facility', 'Programs', 'Bureau', 'Secretary', 'Boat', \n",
    "     'Repair', 'Motor', 'Tax', 'Machinist', 'District', 'Parking', 'Electrician', 'Collection', 'Controller', \n",
    "     'Survey', 'Pressure', 'Lot', 'Hand', 'Deputy', 'Buyer', 'Treatment', 'Pumping', 'Event', 'Machine', 'Sergeant', \n",
    "     'Management', 'Special', 'Protection', 'Golf', 'Manager', 'Crew', 'Head', 'Care', 'III', 'Painter', 'Cement', \n",
    "     'Firefighter', 'Water', 'Plant', 'Education', 'Bindery', 'Surgeon', 'Storekeeper', 'Examiner’ ‘Polygraph', \n",
    "     'Park', 'Cement', 'Keeper', 'Metal', 'Commission', 'Harbor', 'Cleaning', 'Events', 'Veterinary', \n",
    "     'Architectural', 'Custodial', 'Marking', 'Tree', 'Stenographer', 'V', 'Fiscal', 'Refuse', 'Real', \n",
    "     'Senior', 'Garage', 'Supply', 'III', 'Supervisor', 'Title', 'Carpenter', 'Data', \n",
    "     'Battalion', 'Relations', 'IV', 'Custodian', 'Laboratory', 'Elevators', 'Duty', 'Vehicle', \n",
    "     'Finisher', 'I', 'Signal', 'Printing', 'Stores', 'Port', 'IT', 'Distribution', 'Photographer', 'Vessel', \n",
    "     'Clerk', 'Firefighter', 'Processing', 'Advisor', 'Planning', 'Geographic', 'Shovel', 'Toolroom', 'Public', \n",
    "     'Light', 'Systems', 'II', 'Maintenance', 'Supervising', 'Occupational', 'Examiner', 'Transmission', \n",
    "     'Compressor', 'Payroll', 'Vessels', 'lateral', 'Specialist', 'Plumber', 'Field', 'Operations', 'Repairer', \n",
    "     'Accounting', 'Building', 'Waste', 'Research', 'Detention', 'Project', 'Procurement', 'Compensation', \n",
    "     'Refrigeration', 'Bus', 'Graphics', 'Messenger', 'Heavy', 'Load', 'Zoning', 'Auditor', 'Wharfinger', 'Shop', \n",
    "     'Service', 'Applications', 'Labor', 'Apprentice', 'Testing', 'Control', 'Curator', 'Waterworks', 'Material', \n",
    "     'Caretaker', 'Captain', 'Disposal', 'Mechanical', 'System', 'Duplicating', 'Nurse', 'Airport', 'Press', \n",
    "     'Maintenance', 'Wastewater', 'Health', 'III', 'Information', 'II', 'Solid', 'Instructor', 'Repairer', 'I', \n",
    "     'Librarian', 'Painter']\n",
    "\n",
    "print(len(l))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "302\n",
      "['Body', 'Stenographer', 'Advisor', 'Specialist', 'Electrician', 'I', 'Director', 'Occupational', 'Motor', 'Representative', 'Garage', 'Signal', 'Manager', 'Painter', 'Marking', 'Vehicle', 'Auditor', 'Superintendent', 'Management', 'Mate', 'Helicopter', 'Vessel', 'Personnel', 'IT', 'Processing', 'Procurement', 'Control', 'Sales', 'Zoo', 'Coordinator', 'Geologist', 'Attendant', 'Information', 'Examiner’ ‘Polygraph', 'Refuse', 'Laboratory', 'Stores', 'Steam', 'Fleet', 'Fiscal', 'Claims', 'Quality', 'Community', 'Carpenter', 'Duty', 'Applications', 'Press', 'Executive', 'Surveying', 'Sweeper', 'Technician', 'Port', 'Instructor', 'Internal', 'Principal', 'Security', 'Machine', 'Recreation', 'Compensation', 'Operating', 'Deck', 'Zoning', 'Distribution', 'Street', 'Lieutenant', 'Operator', 'Commander', 'Water', 'Officer', 'Special', 'Detention', 'Relations', 'Land', 'Maintenance', 'Auto', 'Trainee', 'Examiner', 'Typist', 'Housing', 'Dispatcher', 'Aid', 'Craft', 'Treatment', 'Aide', 'Utility', 'General', 'Controller', 'Fire', 'Assistant', 'IV', 'Heating', 'Collection', 'Wharfinger', 'Health', 'Automotive', 'Supply', 'Messenger', 'Industrial', 'Departmental', 'Laborer', 'Print', 'Window', 'Instrument', 'Construction', 'Plumbing', 'Commercial', 'Cabinet', 'Accountant', 'Resources', 'Communications', 'Worker', 'Administrator', 'Designer', 'Secretary', 'Keeper', 'Metal', 'V', 'Conditioning', 'Chief', 'Events', 'Cartographer', 'Architectural', 'Waterworks', 'III', 'Field', 'Repairer', 'Project', 'Polygraph', 'Repair', 'Drafting', 'Hand', 'Computer', 'Environmental', 'Electric', 'Inspector', 'Estate', 'Programs', 'Custodian', 'Supervising', 'Technical', 'Pumping', 'Starter', 'Material', '4', 'Services', 'Bindery', 'Surveys', 'Education', 'Research', 'Engineering', 'Technician’, ‘Environmental', 'Driver', 'Apparatus', 'Real', 'Grounds', 'Truck', 'Accounting', 'Solid', 'Firefighter', 'Labor', 'Geographic', 'Bus', 'Biologist', 'Machinist', 'Caretaker', 'Shovel', 'Pipefitter', 'Division', 'Clerk', 'Fireboat', 'Gardener', 'Equipment', 'Development', 'Compliance', 'Helper', 'Lot', 'Apprentice', 'II', 'Wastewater', 'Payroll', 'Traffic', 'Poster', 'Load', 'Nurse', 'Pressure', 'Records', 'Service', 'Locksmith’, Building', 'Sheet', 'Boat', 'Captain', 'Party', 'Planner', 'Workers', 'Park', 'Station', 'Compressor', 'Event', 'Structural', 'Reptiles', 'Sergeant', 'Airport', 'Refrigeration', 'Delivery', 'Facility', 'Storekeeper', 'Public', 'Deputy', 'Engineer', 'Administrative', 'Head', 'Lighting', 'Senior', 'Testing', 'Toolroom', 'Civil', 'Worker’, ‘Airports', 'Rehabilitation', 'Power', 'Harbor', 'Cleaning', 'Sign', 'Data', 'Meter', 'Transmission', 'System', 'Battalion', 'Librarian', 'Warehouse', 'Bureau', 'Forensic', 'Builder', 'Elevators', 'Tax', 'Property', 'Airports', 'Veterinary', 'Photographer', 'Curator', 'Elevator', 'Financial', 'Light', 'Mechanic', 'Heavy', 'Irrigation', 'Telecommunications', 'Buyer', 'Supervisor', 'Graphics', 'Cement', 'Materials', 'Plant', 'Reader', 'City', 'Planning', 'District', 'Associate', 'Air', 'Parking', 'Building', 'Custodial', 'Printing', 'Marketing', 'Transportation', 'Plumber', 'Detective', 'Police', 'Ranger', 'Systems', 'Mechanical', 'Vessels', 'Birds', 'Managing', 'Duplicating', 'Finisher', 'Analyst', 'Survey', 'lateral', 'Waste', 'Protection', 'Shop', 'Commission', 'Cleaner', 'Electrical', 'Safety', 'Maker', 'Office', 'Golf', 'Operations', 'Auditor’, ’Senior’, ‘Operator’, ‘Senior', 'Sanitation', 'Line', 'Crew', 'Surgeon', 'Animal', 'Title', 'Disposal', 'Tree', 'Care', 'Masonry', 'Pilot', 'Programmer']\n"
     ]
    }
   ],
   "source": [
    "exp_job_class_title_keywords = list(set(l))\n",
    "print(len(exp_job_class_title_keywords))\n",
    "print(exp_job_class_title_keywords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}