Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement/checking deployments and testing #1064

Merged
merged 6 commits into from
Oct 24, 2023

Conversation

caparker
Copy link
Collaborator

A few different things to review here

  • In the lib/measurements.js file I added an error catcher for adapter errors
  • Added a lib/request.js file that holds a flexible client wrapper for got, see the defra adapter for its use. More could be done here, for example, by fine tuning the retries.
  • Tweaked the deployment scheduler to deal with the fact that it would not run a deployment of once source. This was because of the two different filtering methods we were using and so I removed the old one.
  • Cleaned up the testing script (index.js) to allow for better local runs of both the fetcher and the scheduler. I added a deployments argument that forces the testing through the deployments path and a nofetch argument that skips the actual fetching part. This lets us isolate the scheduler a bit better for local testing and development.
  • Changed the default timeout to 15sec instead of 60. Not sure if you want to keep that or not.

  Instead of using the old pipe use(filter) we are going to rely on
  filtering the sources ahead of time and passing all desired sources
  to be run.
@majesticio majesticio merged commit 5c09182 into main Oct 24, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants