CA Ventura Sheriff

[irenecasado]

No description provided.

[stucka]

For the record, I'm trying to finish this scraper. A few things I know need to be worked on:

• Include AB748 page
• Ensure single-document cases are properly scraped from the index pages ... possibly need to split by the H2s to get the case IDs?
• Ensure multi-document cases are properly scraped, as the URLs will look a lot like the generic nav links.
• Ensure subpages are properly scraped, to verify that no index pages are ignored.

Some of this requires the visually identifiable content section to be properly extruded, and codewise it's a mess, e.g., things I thought might be good indicators turn out to be the 18th incarnation of a particular class. This may work on index and subpages but needs to be tested:

pageguts = soup.find("div", attrs={"class": "page-content"})
focusedguts = BeautifulSoup("<h2" + "<h2".join(pageguts.prettify().split("<h2")[1:]))

[stucka]

newsroomdev suggests trying to cure BeautifulSoup vs. mypy problems particularly with lines 214-215. https://github.com/biglocalnews/clean-scraper/blob/ca-51/clean/ca/sacramento_pd.py#L180-L224

[stucka]

@irenecasado , I have ... destroyed all the history here. I am so so very sorry, and so tired. Please make a copy of your local version ...

[stucka]

@newsroomdev @zstumgoren I believe this is ready for review if you want.

[newsroomdev]

Could you walk me through setting it to False here? On the backend side, once these are saved to a local cache, they're copied over to Azure Blob Storage with timestamps. Leaving it as True ensures we're catching changes to the page and its asset_url's byte lengths

[stucka]

Whatever you'd like to do is fine by me. Looks like there were about 58 subpages and I was assuming there was no real reason to rescrape those. Index pages are of course forced to rescrape.

[stucka]

There's no info on file size here that I'm aware of, FYI. So unless you rescrape the asset itself; or alternately grab the HEAD and get accurate, reliable results; rescraping the subpage would only show you any new assets, and I don't know if anything ever gets refreshed.