Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add workers (xloader and harvesters) #103

Merged
merged 8 commits into from
Mar 8, 2024
Merged

Add workers (xloader and harvesters) #103

merged 8 commits into from
Mar 8, 2024

Conversation

mjanez
Copy link
Owner

@mjanez mjanez commented Mar 8, 2024 •
edited
Loading

Use supervisor to allow fetch and gather consumers in the background: https://github.com/ckan/ckanext-harvest#setting-up-the-harvesters-on-a-production-server

#100

@mjanez
Update ckanext-spatial
7e5e4e8 
Add Spatial - v2.1.1 version with pyproj patch.
@mjanez
Setting up the harvesters (ckan container)
fb158d7 
Use supervisor to allow fetch and gather consumers in the background:
https://github.com/ckan/ckanext-harvest#setting-up-the-harvesters-on-a-production-server
@mjanez
Fix token revocation in 01_setup_xloader.sh
96a4c39
@mjanez
Fix supervisord command in start_ckan_development.sh.override
c2c1ef6
@mjanez
Update ckanext-harvest patch
3b9e94b
@mjanez
Update CKAN user management instructions
0ed33f2
@mjanez
Add harvest & xloader worker configuration
68a9ce9 
- Update Dockerfiles
- Update start_ckan.sh with custom workers
- Update start_ckan_development.sh.override to remove automatic workers
- Add supervisor harvester.conf
- Add supervisor xloader.conf
- Update Scheming DCAT version to 2.1.0
@mjanez
Merge pull request #100 from mjanez/feature/harvester-worker
904530e 
Background worker setup (CKAN container) | ckanext-harvest & ckanext-xloader
@mjanez mjanez added bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request ckan-ogc https://github.com/mjanez/ckan-ogc ckanext-schemingdcat labels Mar 8, 2024
@mjanez mjanez self-assigned this Mar 8, 2024
This was linked to issues Mar 8, 2024
Load data into datastore #10
Closed
Feature - Harvest/Xloader workers in background #102
Closed
@mjanez mjanez merged commit 4c6e6bd into master Mar 8, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@mjanez mjanez

Labels
bug Something isn't working ckan-ogc https://github.com/mjanez/ckan-ogc ckanext-schemingdcat documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Feature - Harvest/Xloader workers in background Load data into datastore
1 participant
@mjanez