Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Push crawl results from the cronjob to postgres #1184

Closed
Tracked by #1167
michielbdejong opened this issue Sep 26, 2024 · 3 comments
Closed
Tracked by #1167

Push crawl results from the cronjob to postgres #1184

michielbdejong opened this issue Sep 26, 2024 · 3 comments

Comments

@michielbdejong
Copy link
Member

No description provided.

@michielbdejong michielbdejong mentioned this issue Sep 26, 2024
Open
13 tasks
@michielbdejong
Copy link
Member Author

This is an adaptation of milestone 3.2, which reads:

Load data from federated API of Open Terms Archive into ToS;DR

@michielbdejong
Copy link
Member Author

michielbdejong commented Sep 26, 2024 •
edited
Loading

Given the trouble we ran into with the use of very large git repositories, basically trying to use GitHub as a database server, we decided it's better if the crawl engine writes its crawl results directly to the postgres database from which ToS;DR is already loading its data.

michielbdejong added a commit to tosdr/ota-engine that referenced this issue Sep 26, 2024
@michielbdejong
First attempt at tosdr/edit.tosdr.org#1184
5339b34
@michielbdejong
Copy link
Member Author

I'm working on this in tosdr/ota-engine#2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@michielbdejong