Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update legislative session automatically #26

Merged
merged 13 commits into from
Sep 24, 2024
1 change: 1 addition & 0 deletions keyrings/live/blackbox-admins.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
datamade@scrapers-us-municipal
[email protected]
[email protected]
[email protected]
Binary file modified keyrings/live/pubring.gpg
Binary file not shown.
26 changes: 18 additions & 8 deletions lametro/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
from .bills import LametroBillScraper
from .people import LametroPersonScraper
from .events import LametroEventScraper
from datetime import datetime


class Lametro(Jurisdiction):
Expand All @@ -16,14 +17,23 @@ class Lametro(Jurisdiction):
"events": LametroEventScraper,
}

legislative_sessions = []
for year in range(2014, 2025):
session = {
"identifier": "{}".format(year),
"start_date": "{}-07-01".format(year),
"end_date": "{}-06-30".format(year + 1),
}
legislative_sessions.append(session)
@staticmethod
def get_legislative_sessions():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want our subclass to have the same signature as the base Jurisdiction class in pupa. How about defining this as a @property instead of a @staticmethod?

today = datetime.now()
this_year = today.year
allowed_years = list(range(2014, this_year))

if (today.month == 6 and today.day >= 23) or today.month >= 7:
allowed_years.append(this_year)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love this. Can you add a short comment explaining the logic here?


for year in allowed_years:
session = {
"identifier": "{}".format(year),
"start_date": "{}-07-01".format(year),
"end_date": "{}-06-30".format(year + 1),
}
yield session
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

legislative_sessions should be a list rather than a generator. Since we control its contents and know it will only grow by one each year, we can be sure that the list will be quite small, so we don't need to worry about the performance penalty of creating a list in memory.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, I'll get the change in! So generators seem to be more useful for medium to larger datasets and lists are fine when it's a small to medium set of values, does that sound right? Or is this more because we want to preserve the return value of the base class's legislative_sessions property?



def get_organizations(self):
org = Organization(name="Board of Directors", classification="legislature")
Expand Down
2 changes: 1 addition & 1 deletion lametro/bills.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ def session(self, action_date) :
localize = pytz.timezone(self.TIMEZONE).localize
fmt = '%Y-%m-%d'

for session in Lametro.legislative_sessions:
for session in Lametro.get_legislative_sessions():
start_datetime = datetime.datetime.strptime(session['start_date'], fmt)
end_datetime = datetime.datetime.strptime(session['end_date'], fmt)

Expand Down
Binary file modified lametro/secrets.py.gpg
Binary file not shown.
1 change: 1 addition & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,6 @@ pytest-mock==1.10.1
requests-mock==1.5.2
requests[security]
sentry-sdk
freezegun==1.5.1
pdfplumber==0.11.3
pytesseract==0.3.10
35 changes: 35 additions & 0 deletions tests/test_jurisdiction.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
from datetime import datetime

import pytest
from freezegun import freeze_time
from lametro import Lametro


@pytest.mark.parametrize("test_date", [
'2024-06-10',
'2024-06-23',
'2024-10-01',
'2025-05-01',
'2025-06-30'
])
def test_legislative_session(test_date):
'''
Test that next fiscal year's legislative sessions are included
only when it's the last week of the current one.
'''
date_format = "%Y-%m-%d"
test_year = test_date.split('-')[0]
last_week_of_year = datetime.strptime(f"{test_year}-06-23", date_format).date()

with freeze_time(test_date):
fake_now = datetime.now()
fake_date = fake_now.date()
next_year = str(fake_now.year + 1)

sessions = list(Lametro.get_legislative_sessions())
latest_session_date = sessions[-1]["end_date"]

if fake_date < last_week_of_year:
assert next_year not in latest_session_date
elif fake_date >= last_week_of_year:
assert next_year in latest_session_date
hancush marked this conversation as resolved.
Show resolved Hide resolved