New Scottish Parliament Scraper

[ajparsons]

This adds a new scraper for the Scottish Parliament's new site.

I've made a new sp_2024 folder and pulled across some of the elements needed for the ID parser.

There are three main steps:

• Download: Uses the search page to find all agendas for that day, download the items of business and composite that into one file per committee.
• Parse: Create structured information from the debates, containing the IDs and information being passed by the SP website.
• Convert: Convert this structured information into TWFY format, add debate and person_ids.

(common and resolvenames are lightly re-formatted versions of the modules from the old scraper).

This seems to work as I'd expect for some recent ones - haven't tested actually loading the data.

There's some more special case stuff that could be loaded from the old scraper, but probably makes more sense to bring it across as things break?

[dracos]

Really good start, but think we need to do a bit more before switching to it. Have made a start on some at the download end. To sum up the I think important ones:

• Remembering the Presiding Officer / Deputy from their first use
• And/or using the SP person IDs
• Spotting topical question subheadings both in parse and convert
• Handling unspoken names using or-bill-section-bold
• Spotting timestamps (which include suspend/resume as meta text to include I think)
• Spotting or-italic
• Fixing speech after division

[ajparsons]

That should be all the major issues tidied up - does this also need an adjustment in TWFY to pull from the sp_2024 directory it puts the finished files in?

[dracos]

Yep, once the parser is updated and has pulled in some data, we can update it so it starts loading in from there