Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(search): timings #305

Merged
merged 2 commits into from
May 11, 2024
Merged

fix(search): timings #305

merged 2 commits into from
May 11, 2024

Conversation

aleksasiriski
Copy link
Member

@aleksasiriski aleksasiriski commented May 11, 2024

  • Refactored config structs into multiple files (0 code logic changes, only renamed a few fields)
  • Fixed bug in engines unit testing, where the buggy colly timeout was being set but the hard timeout was left at default
  • Created a new algo with better comment explanation and naming of config fields
  • BREAKING CHANGE: removed colly timeout for both normal and pages collectors (since it's proven to be buggy)
  • BREAKING CHANGE: removed category info (also worked with !wiki) in favor of quick (also !fast), and added more engines to it
  • BREAKING CHANGE: renamed category surf to broad (also works with !surf)

Algo explanation:

  1. Wait for PreferredTimeoutMin before checking the number of engines
  2. Check if the number of results is more than or equal to the PreferredResultsNumber
  3. If not, wait additional StepTime until either PreferredResultsNumber or PreferredTimeoutMax is reached
  4. Check if the number of results is more than or equal to the MinimumResultsNumber
  5. If not, wait additional StepTime until either MinimumResultsNumber or HardTimeout is reached

If at any time all engines respond, the results will be returned immediately. This now will hopefully return results as quickly as possible but also prefer to return slow but some results instead of not returning anything.

Default options for GENERAL:

  • PreferredTimeoutMin: 1s
  • PreferredTimeoutMax: 2s
  • PreferredResultsNumber: 20
  • StepTime: 50ms
  • MinimumResultsNumber: 10
  • HardTimeout: 3s

Default options for IMAGES:

  • PreferredTimeoutMin: 1s
  • PreferredTimeoutMax: 2s
  • PreferredResultsNumber: 40
  • StepTime: 100ms
  • MinimumResultsNumber: 20
  • HardTimeout: 3s

Default options for QUICK:

  • PreferredTimeoutMin: 500ms
  • PreferredTimeoutMax: 1.5s
  • PreferredResultsNumber: 10
  • StepTime: 25ms
  • MinimumResultsNumber: 5
  • HardTimeout: 3s

@aleksasiriski aleksasiriski requested a review from a team May 11, 2024 20:49
@aleksasiriski aleksasiriski merged commit dd1ed8f into main May 11, 2024
5 of 6 checks passed
@aleksasiriski aleksasiriski deleted the as/fix/timings branch May 11, 2024 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants