Add documentation for scrapers #25

hancush · 2024-08-29T21:23:32Z

Description

Hark! A documentation site!

Succeeding build here: https://github.com/Metro-Records/scrapers-lametro/actions/runs/10622927449/job/29448271525

Testing

Review the documentation at https://metro-records.github.io/scrapers-lametro/ and confirm it's a good first draft.
Try to run the Quarto docs locally following the instructions in the README and confirm you can preview the built site and make updates.

hancush · 2024-08-29T21:32:37Z

.github/workflows/publish_docs.yml

+      - name: Set up Quarto
+        if: steps.docs-changed.outputs.any_changed == 'true'
+        uses: quarto-dev/quarto-actions/setup@v2
+
+      - name: Render and Publish
+        if: steps.docs-changed.outputs.any_changed == 'true'
+        uses: quarto-dev/quarto-actions/publish@v2
+        with:
+          path: docs
+          target: gh-pages
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}


These steps are only run if there are changes in the docs/ directory.

hancush · 2024-08-29T21:39:54Z

docker-compose.councilmatic.yml

+version: '2.4'
+
+services:
+  scrapers:
+    image: scrapers-lametro
+    container_name: scrapers-lametro
+    build: .
+    stdin_open: true
+    tty: true
+    volumes:
+      - .:/app
+    environment:
+      # Populate the local Councilmatic database
+      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/lametro
+      DJANGO_SETTINGS_MODULE: pupa.settings
+      OCD_DIVISION_CSV: "/app/lametro/divisions.csv"
+    command: pupa update lametro
+    # Connect the scraper container to the app network
+    networks:
+      - app_net
+
+networks:
+  # Define connection to the app's Docker network
+  app_net:
+    name: la-metro-councilmatic_default
+    external: true


ah this makes things way easier, ty!

hancush · 2024-08-29T21:40:07Z

docs/debugging.qmd

@@ -0,0 +1,224 @@
+---


Migrated from the Councilmatic wiki.

hancush · 2024-08-29T21:40:30Z

docs/local-development.qmd

+docker compose -f docker-compose.councilmatic.yml run --rm scrapers
+```
+
+### Useful pupa commands


Adapted from the Councilmatic wiki.

xmedr

This is really great! And the quarto docs run/update locally. Got some little comments below.

I also wanted to ask: I found that it took some time to figure out how some terms in Legistar mapped to Metro. For example, that "matter" equates to "board report", or a "body" refers to a "legislation" (I might be wrong about that). Would it be useful to have a small note of alternative names that certain items might be called in the individual scrapers sections?

xmedr · 2024-09-03T18:51:36Z

docs/debugging.qmd

+
+#### Legistar contains the wrong data
+
+Data issues can occur when the Legistar API or web interface displays the wrong information. This generally happens with Metro staff enters information that is incorrect.


might be a small typo. was it supposed to be something like:

This generally happens <when> Metro staff enters information that is incorrect.

xmedr · 2024-09-03T19:01:03Z

docs/debugging.qmd

+
+The dashboard lives on GitHub, [here](https://github.com/Metro-Records/la-metro-dashboard).
+
+Apache maintains thorough documentation of [core concepts](http://airflow.apache.org/docs/stable/concepts.html), as well as [navigating the UI](http://airflow.apache.org/docs/stable/ui.html). If you've never used Airflow before, these are great resources to start with!


this is super helpful!

xmedr · 2024-09-03T19:26:55Z

docs/debugging.qmd

+
+![windowed_bill_scraping DAG tree view](https://i.imgur.com/9NDdEpy.png)
+
+Note that the scraping DAGs employ [branch operators](https://airflow.apache.org/docs/stable/concepts.html?highlight=branch#branching) to determine what kind of scraping task to run. Be sure to look closely to verify that the task you're expecting is green (succeeded), not pink (skipped).


this link leads to the core concepts page without highlighting anything. would this link make sense?

https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#concepts-branching

xmedr · 2024-09-03T19:29:29Z

docker-compose.councilmatic.yml

+version: '2.4'
+
+services:
+  scrapers:
+    image: scrapers-lametro
+    container_name: scrapers-lametro
+    build: .
+    stdin_open: true
+    tty: true
+    volumes:
+      - .:/app
+    environment:
+      # Populate the local Councilmatic database
+      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/lametro
+      DJANGO_SETTINGS_MODULE: pupa.settings
+      OCD_DIVISION_CSV: "/app/lametro/divisions.csv"
+    command: pupa update lametro
+    # Connect the scraper container to the app network
+    networks:
+      - app_net
+
+networks:
+  # Define connection to the app's Docker network
+  app_net:
+    name: la-metro-councilmatic_default
+    external: true


ah this makes things way easier, ty!

xmedr · 2024-09-03T19:36:52Z

docs/scrapers/events.qmd

+- Metro streams audio in both English and Spanish. They cannot associated multiple
+broadcast link with one event in Legistar, so they create two nearly identical events that


super small typo. is it supposed to be:

They cannot associate multiple broadcast links with one event in Legistar, ...

antidipyramid

Amazing @hancush! Just one question inline.

antidipyramid · 2024-09-04T20:33:35Z

docs/debugging.qmd

+
+If scrapes are running, by far the most common source of issues is the scraper failing to capture changes to a bill or event. The root issue is that we rely on timestamps that should indicate an update to determine which bills and events to scrape. In reality, these timestamps do not always update when a change is made to an event or bill in Legistar. We have a couple of strategies to get around this:
+
+1. During [the Friday support window](https://github.com/datamade/la-metro-dashboard/blob/ac416e5e03f6a97fb9b0c6112093c679cefb0d1c/dags/constants.py#L41-L59), we scrape all events and bills at the top of every hour.


Should we remove references to the Friday support window?

…rence

hancush · 2024-09-05T15:53:48Z

@xmedr, @antidipyramid Thanks for the review, you two! Addressed your comments and published the changes manually: https://metro-records.github.io/scrapers-lametro/

Xavier, thanks for flagging synonyms! I've added those to each of the scraper pages, and I also added core concepts to the Jurisdiction scraper, since I found that really confusing when I got started. I also added links to the Open Civic Data documentation where relevant. They're really useful for understanding why data is structured the way it is.

Let me know if this looks good to merge!

xmedr

These are hugely helpful, thank you! Looks good to me

hancush added 9 commits August 29, 2024 16:07

Add scraper documentation

76e574e

Don't ignore _site/

ac51ea6

Add workflow to publish docs

7502141

docker-compose -> docker compose

0a6c839

Trigger build from PR

35f0580

Trigger build from PR, redux

5e74acf

Trigger build from PR, redux #2

c2f6318

Trigger build from PR, redux #3

91ba0a3

Remove PR trigger

4ea96fa

hancush commented Aug 29, 2024

View reviewed changes

hancush added 2 commits August 29, 2024 16:38

Update README

2b9a269

One more README update

2a516ce

hancush commented Aug 29, 2024

View reviewed changes

docs/debugging.qmd

@@ -0,0 +1,224 @@

---

Copy link

Collaborator Author

hancush Aug 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Migrated from the Councilmatic wiki.

hancush commented Aug 29, 2024

View reviewed changes

hancush requested review from antidipyramid and xmedr August 29, 2024 21:42

hancush mentioned this pull request Aug 29, 2024

What's up with Quarto? datamade/how-to#334

Open

xmedr requested changes Sep 3, 2024

View reviewed changes

antidipyramid mentioned this pull request Sep 4, 2024

Sync/Scraping Schedule Documentation Metro-Records/la-metro-councilmatic#451

Closed

antidipyramid reviewed Sep 4, 2024

View reviewed changes

Add synonyms and core concepts, fix typos, strike Friday support refe…

2e6f4df

…rence

hancush requested review from antidipyramid and xmedr September 5, 2024 15:54

xmedr approved these changes Sep 5, 2024

View reviewed changes

hancush merged commit a512658 into main Sep 26, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add documentation for scrapers #25

Add documentation for scrapers #25

hancush commented Aug 29, 2024 •

edited

Loading

hancush Aug 29, 2024

hancush Aug 29, 2024

xmedr Sep 3, 2024

hancush Aug 29, 2024

hancush Aug 29, 2024

xmedr left a comment

xmedr Sep 3, 2024 •

edited

Loading

xmedr Sep 3, 2024

xmedr Sep 3, 2024

xmedr Sep 3, 2024

xmedr Sep 3, 2024

antidipyramid left a comment

antidipyramid Sep 4, 2024

hancush Sep 5, 2024

hancush commented Sep 5, 2024

xmedr left a comment


		#### Legistar contains the wrong data

		Data issues can occur when the Legistar API or web interface displays the wrong information. This generally happens with Metro staff enters information that is incorrect.


		The dashboard lives on GitHub, [here](https://github.com/Metro-Records/la-metro-dashboard).

		Apache maintains thorough documentation of [core concepts](http://airflow.apache.org/docs/stable/concepts.html), as well as [navigating the UI](http://airflow.apache.org/docs/stable/ui.html). If you've never used Airflow before, these are great resources to start with!


		![windowed_bill_scraping DAG tree view](https://i.imgur.com/9NDdEpy.png)

		Note that the scraping DAGs employ [branch operators](https://airflow.apache.org/docs/stable/concepts.html?highlight=branch#branching) to determine what kind of scraping task to run. Be sure to look closely to verify that the task you're expecting is green (succeeded), not pink (skipped).

		- Metro streams audio in both English and Spanish. They cannot associated multiple
		broadcast link with one event in Legistar, so they create two nearly identical events that


		If scrapes are running, by far the most common source of issues is the scraper failing to capture changes to a bill or event. The root issue is that we rely on timestamps that should indicate an update to determine which bills and events to scrape. In reality, these timestamps do not always update when a change is made to an event or bill in Legistar. We have a couple of strategies to get around this:

		1. During [the Friday support window](https://github.com/datamade/la-metro-dashboard/blob/ac416e5e03f6a97fb9b0c6112093c679cefb0d1c/dags/constants.py#L41-L59), we scrape all events and bills at the top of every hour.

Add documentation for scrapers #25

Add documentation for scrapers #25

Conversation

hancush commented Aug 29, 2024 • edited Loading

Description

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmedr left a comment

Choose a reason for hiding this comment

xmedr Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antidipyramid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hancush commented Sep 5, 2024

xmedr left a comment

Choose a reason for hiding this comment

hancush commented Aug 29, 2024 •

edited

Loading

xmedr Sep 3, 2024 •

edited

Loading