Scraping public datasets

[yiwen-h]

• e.g. Appointments in General Practice
• How to download a webpage
• How to find releases
• How to fight problems
• How to be responsible in what you download
• Could include {rvest} and {polite}, also point to APIs where possible with {httr2} (Matt)

[StatsRhian]

Notes from our chat (2024-11-27)

Overview

• Data scraping can be different levels. Certain things (links in a website) are more complicated to scrape reliably than an RSS feed.
• How do we watch for changes?
• Jonathan shares how he has done this on other projects and what the challenges were?
• What makes scraping easy? What makes it hard?
• What if the website changes it's structure?
• How can we quickly identify key parts of a website?

Being polite

• What is the etiquette when scraping?
• What do we need to bare in mind?
• What tools can help us scrape nicely?

Related topics

These are adjacent topics which are relevant to data scraping but we won't go into detail on

• Job scheduling (windows scheduler, CRON jobs, GitHub Actions, Connect scheduler)
• Data validation (does the data that I downloaded meet the expected structure)
• APIs (how to write a query)
• Scraping structured data (XML, RSS etc,)

Specific Project

• Jonathan & Rhian would like to mock up an example of scraping unstructured data
• For example Appointments in General Practice websites
• Could try and adapt the workflow we use to see if it also works with similar NHS Digital pages (e.g. Workforce statistics)

Future work

• Jonathan is interested in taking this work further with the more challenging board meeting data.
• This could make a nice NLP data set
• C&C is a good way to gauge if anyone in SU would be interested in looking into this