Reduce number of concurrent DNS lookups in Machine Tracker

[stveit]

Fixes #2669

Limits concurrent parallel lookups to 100 at a time.

Tested manually that machine tracker no longer crashes from hitting the file descriptor limit with ulimit set as low as 200 on a search with 4096 results. This would previously crash instantly

The amount of parallel lookups should perhaps be configurable but that can be separate pr if we see the need

A practical change I made is that saving the results from a lookup to the dict self.results will now happen during the parallel lookups, instead of at the end. I made this change simply because using Cooperator changes how the DefferedList we use look like. Instead of all the separate lookup tasks being in the list, the list now contains 100 iterators.
I dont know how to add a callback to the list that will actually be able to go through all the results produced instead of just going over the iterators themselves, so I moved the saving step to be part of the smaller work units.

Errors that were previously raised by themselves are now eaten up by the Cooperator. Tried to fix this by saving all errors then raising the first one that occurred. I expect this keeps the interfacing with the asyncdns module intact, but side effect is that it will only raise the exceptions after its done running, so its slower to fail

[codecov]

Codecov Report

Attention: Patch coverage is 95.83333% with 1 line in your changes missing coverage. Please review.

Project coverage is 59.52%. Comparing base (a2be786) to head (93fca87).
Report is 87 commits behind head on 5.10.x.

Files	Patch %	Lines
python/nav/asyncdns.py	95.83%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           5.10.x    #2917      +/-   ##
==========================================
- Coverage   60.21%   59.52%   -0.69%     
==========================================
  Files         601      602       +1     
  Lines       43981    43308     -673     
==========================================
- Hits        26481    25780     -701     
- Misses      17500    17528      +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

Test results

     11 files       11 suites   11m 8s ⏱️
3 325 tests 3 325 ✔️ 0 💤 0 ❌
9 443 runs  9 443 ✔️ 0 💤 0 ❌

Results for commit 93fca87.

♻️ This comment has been updated with latest results.

[lunkwill42]

This feels like putting lipstick on a pig. It's still ugly, but it works - great work 😆

I have a feeling this will be revisited again at some point in the future (hopefully when we replace Twisted with asyncio)

The base branch was changed.

[lunkwill42]

This feels like putting lipstick on a pig. It's still ugly, but it works - great work 😆

I have a feeling this will be revisited again at some point in the future (hopefully when we replace Twisted with asyncio)

I changed the base to the new stable branch, which caused my old review to be dismissed - so posting the same again.