Day022_HW.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 動態網頁爬蟲 - 使用 Selenium\n",
    "\n",
    "* 了解 Selenium 用於動態網頁爬蟲的原理\n",
    "* 能夠使用 Selenium 撰寫動態網頁爬蟲"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 作業目標\n",
    "\n",
    "* 1. 根據範例使用 selenuim 取出 Hahow 網站中所有課程的資料，並用一個適合的資料結構儲存（Hint: 將所有資料存在一個變數當中取出）\n",
    "* 2. 剛剛的做法只能取出第一頁，試著取出所有頁面的資料。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. 根據範例使用 selenuim 取出 Hahow 網站中所有課程的資料，並用一個適合的資料結構儲存（Hint: 將所有資料存在一個變數當中取出）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.support.ui import Select\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "browser = webdriver.Chrome(executable_path='chromedriver')\n",
    "browser.get(\"https://hahow.in/courses\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 取得資料（等瀏覽器資料出現後才執行）\n",
    "html_source = browser.page_source\n",
    "html_source\n",
    "\n",
    "# 取得資料\n",
    "html_source = browser.page_source\n",
    "soup = BeautifulSoup(html_source, 'html.parser')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "此頁共有 24 門課程\n['職場英文大補帖｜從社交心法到溝通金句', 'Java 工程師必備！Spring Boot 零基礎入門', '輕鬆打造超華麗報表:無痛入門 Data Studio', '紙上御花園－初階水墨花卉探幽', '用戶x商業x數據 | UX‧三刀流產品體驗設計', '自學英語口說：提升英文流暢度的5大步驟', '拍一次就搞定！FB IG YT 通用的行銷影片', '說說立體刺繡花葉動物，交織裡的平衡韻律', '邏輯表達力-周震宇的口語邏輯優化課', '五星級法式經典甜點在家做', '掌握鉤針編織邏輯：五種針法自造生活小物', 'Python 推薦系統入門: 打造令人上癮的產品', '堯中人像攝影創作：掌握光影發掘人像質感', '日新麥克筆魔法：十支筆創造新視界－慕夏篇', '志祺的架構性思考：從資訊整理到觀點表達', ' PS 核心技法 | 調色技法完全講解篇', '҉唐鳳҉數位溝通社：就這樣把你增幅', '四叉貓網美談資社：美少女反串溝通愛心光波', '楊大正台語詞創社：咱就大力來寫著歌', '大支嘻哈研究社：饒舌饒出台灣味', '賴品妤 Cosplay 社：ㄘ我的二次元替身攻擊', '月月鳥昆蟲研究社：跟鄭運鵬變身甲蟲王', '單人也可佈光：有趣的L角佈燈術', '人氣喫茶店 KADOYA 主廚帶你做日式洋菓子']\n"
     ]
    }
   ],
   "source": [
    "courses = []\n",
    "\n",
    "# 此網站有做RWD，因此視窗大小也會造成網頁元素有差異\n",
    "course_blocks = soup.select('h4.title')\n",
    "for course_block in course_blocks:\n",
    "    courses.append(\n",
    "        course_block.text\n",
    "    )\n",
    "\n",
    "print(f'此頁共有 {len(courses)} 門課程')\n",
    "print(courses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "browser.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 剛剛的做法只能取出第一頁，試著取出所有頁面的資料。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.support.ui import Select\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "browser = webdriver.Chrome(executable_path='chromedriver')\n",
    "browser.get(\"https://hahow.in/courses\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 取得資料（等瀏覽器資料出現後才執行）\n",
    "html_source = browser.page_source\n",
    "soup = BeautifulSoup(html_source, 'html.parser')\n"
   ]
  },
  {
   "source": [
    "\n",
    "# 2. 每頁都取出資料放入 courses\n",
    "\n",
    "course_title_selector = 'h4.title'\n",
    "courses = []\n",
    "for i in range(0,25):\n",
    "    url_ = f'https://hahow.in/courses?page={i+1}'\n",
    "    browser.get(url_)\n",
    "    \n",
    "    # 等待元素載入\n",
    "    try:\n",
    "        selector_ = (By.CSS_SELECTOR, course_title_selector)\n",
    "        WebDriverWait(browser, 20, 0.5).until(EC.presence_of_element_located(selector_))\n",
    "    except:\n",
    "        print('網頁載入失敗，跳過')\n",
    "        continue\n",
    "    \n",
    "    html_source = browser.page_source\n",
    "    soup = BeautifulSoup(html_source, 'html.parser')\n",
    "\n",
    "    course_blocks = soup.select(course_title_selector)\n",
    "    for course_block in course_blocks:\n",
    "        courses.append(\n",
    "            course_block.text\n",
    "        )\n",
    "        \n",
    "browser.close()"
   ],
   "cell_type": "code",
   "metadata": {},
   "execution_count": 44,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n",
      "網頁載入失敗，跳過\n"
     ]
    }
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "全部共有 0 門課程\n[]\n"
     ]
    }
   ],
   "source": [
    "# 3. done!\n",
    "\n",
    "print(f'全部共有 {len(courses)} 門課程')\n",
    "print(courses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}