Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add kingfisher collect log warnings with scrapyloganalyzer #385

Merged
merged 6 commits into from
Jan 23, 2025
Merged

feat: add kingfisher collect log warnings with scrapyloganalyzer #385

merged 6 commits into from
Jan 23, 2025

Conversation

yolile
Copy link
Member

@yolile yolile commented Jan 3, 2025

ref #29

@yolile
Copy link
Member Author

yolile commented Jan 3, 2025

I'm not sure why gunicorn[setproctitle] and yapw[perf] are being removed when running uv pip compile requirements.in -o requirements.txt

@jpmckinney
Copy link
Member

It's okay - they should be removed.

@yolile yolile force-pushed the 29-kingfisher-collect-warnings branch from 6e307a3 to 9079500 Compare January 20, 2025 13:59
@yolile yolile requested a review from jpmckinney January 20, 2025 14:14
jpmckinney
jpmckinney reviewed Jan 23, 2025
View reviewed changes
Copy link
Member

@jpmckinney jpmckinney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a commit. Was {scrapy_log.error_rate} instead of scrapy_log.error_rate a typo or intentional?

data_registry/models.py Outdated
"<dt><code>item_dropped_count</code></dt>"
"<dd>The number of items dropped by the crawl</dd>"
"<dt><code>invalid_json_count</code></dt>"
"<dd>The number of invalid JSON files dropped by the crawl</dd>"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a JSONL file, I think it will drop only the invalid lines. Need to update migration as well.

Suggested change
"<dd>The number of invalid JSON files dropped by the crawl</dd>"
"<dd>The number of invalid JSON items dropped by the crawl</dd>"

@yolile yolile requested a review from jpmckinney January 23, 2025 12:53
@yolile
Copy link
Member Author

yolile commented Jan 23, 2025

a typo or intentional?

A typo!

@jpmckinney
Copy link
Member

This PR now includes commits from #389

@jpmckinney jpmckinney force-pushed the 29-kingfisher-collect-warnings branch from cfdddd0 to 311b62b Compare January 23, 2025 20:56
@jpmckinney
Copy link
Member

I rebased to remove 6ce6c79 and a merge commit.

@jpmckinney jpmckinney merged commit f8d7c03 into main Jan 23, 2025
15 of 17 checks passed
@jpmckinney jpmckinney deleted the 29-kingfisher-collect-warnings branch January 23, 2025 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jpmckinney jpmckinney jpmckinney approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@yolile @jpmckinney