Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve 'updateLocations' Task for Database Efficiency #1096

Merged
merged 1 commit into from
Dec 23, 2023

Conversation

ljdelight
Copy link
Contributor

Modified the 'updateLocations' task to enhance database efficiency. The recurring task now uses a transaction per iteration instead of one for the entire task, eliminating manual commits. Cache removals are also now done with each transaction, avoiding extra database lookups.

The main point here is to reduce contention. I don't expect much change to the overall speed of the scheduled task. Validation is to check the 'finished run' time (ofc check for exceptions as well):

  • docker logs maproulette-api | grep "Task 'Updating locations': Finished run"
  • or journalctl -u docker --since "today" CONTAINER_NAME=maproulette-api | grep "Task 'Updating locations': Finished run"

Metrics Before change:

  • Staging is consistent ~45s before the change. Prod ranges to around 50s to 90s.

Modified the 'updateLocations' task to enhance database efficiency. The recurring task now uses a transaction per iteration instead of one for the entire task, eliminating manual commits. Cache removals are also now done with each transaction, avoiding extra database lookups.
Copy link

Quality Gate Passed Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

@ljdelight
Copy link
Contributor Author

This works fine.

13:24:14.271 INFO [][application-akka.actor.default-dispatcher-35][org.maproulette.jobs.SchedulerActor] - Scheduled Task 'Updating locations': Finished run. Time spent: 46665ms
15:24:11.920 INFO [][application-akka.actor.default-dispatcher-22][org.maproulette.jobs.SchedulerActor] - Scheduled Task 'Updating locations': Finished run. Time spent: 44313ms
17:24:13.945 INFO [][application-akka.actor.default-dispatcher-5][org.maproulette.jobs.SchedulerActor] - Scheduled Task 'Updating locations': Finished run. Time spent: 46339ms

@ljdelight ljdelight merged commit 8406fc9 into main Dec 23, 2023
7 checks passed
@ljdelight ljdelight deleted the ljdelight/tweakUpdateLocations branch December 23, 2023 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant