investigate performance

[jku]

The test suite feels a little slow: there should be no need for an individual test to take a perceptible amount of time

My guess is that the pytest integration has lead to infrastructure setup happening for each test: We should really only need one web server (SimulatorServer) as it gives each RepositorySimulator a subdirectory and should be able to serve all of them.

[jku]

see also #17

[jku]

Only the python-tuf tests seem slow so this is unlikely to be related to the http server startup/shutdown.

[jku]

Quick look at cProfile results implies the issue is Python startup (basically module import) for clients/python-tuf: we're starting a new python executable several hundred times (since each init, refresh and download_target is a new execution) which is not optimal for an interpreted language. Each startup seems to take something like 0.1 seconds on my laptop.

Potential improvements:

• keep the client running and give the commands over IPC: this seems very complicated and would be another client API
• parallelize test runs: the infra should be roughly designed correctly for that. pytest-xdist makes the test run ~50% faster based on a quick try
• shave unnecessary repetition from tests
  • e.g. test_multiple_changes_to_target() invokes the client 43 times. I think shaving 80% of those would be easy and would not reduce coverage: the test could do all of the metadata changes it now does but only invoke client a few times instead of constantly (test_file_download.py: Don't make > 40 client calls in a test #165)
  • a common pattern is refresh+download_target. The refresh is not strictly necessary.