500 errors are not logged as request failures

[jgrevich]

Summary

I noticed a number of 500 errors while running a load test however I did not see them reported as request failures in the results summary.

Detail

I started the SP based sign in test using the following configuration:

NUM_USERS=100000 SP_HOST=https://sp-oidc-sinatra.pt.identitysandbox.gov locust --locustfile load_testing/sp_sign_in.locustfile.py --host https://idp.pt.identitysandbox.gov  --users 50 --hatch-rate 10 --run-time 30m --headless

While running I noticed repeating 500 errors:

[2020-10-05 14:36:43,524] j12/ERROR/locust.user.task: 500 Server Error: Internal Server Error for url: https://sp-oidc-sinatra.pt.identitysandbox.gov/auth/result?code=c409a63a-614a-48f9-874f-c488cfea0bea&state=b6426c42524518350dff2ea8a9d368cf
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/locust/user/task.py", line 284, in run
    self.execute_next_task()
  File "/usr/local/lib/python3.8/site-packages/locust/user/task.py", line 309, in execute_next_task
    self.execute_task(self._task_queue.pop(0))
  File "/usr/local/lib/python3.8/site-packages/locust/user/task.py", line 321, in execute_task
    task(self)
  File "/Users/jjg/Documents/sources/login/identity-loadtest/load_testing/sp_sign_in.locustfile.py", line 9, in sp_sign_in_load_test
    flow_sp_sign_in.do_sp_sign_in(self)
  File "/Users/jjg/Documents/sources/login/identity-loadtest/load_testing/common_flows/flow_sp_sign_in.py", line 69, in do_sp_sign_in
    resp = do_request(
  File "/Users/jjg/Documents/sources/login/identity-loadtest/load_testing/common_flows/flow_helper.py", line 26, in do_request
    resp.raise_for_status()
  File "/usr/local/lib/python3.8/site-packages/locust/clients.py", line 22, in raise_for_status
    Response.raise_for_status(self)
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://sp-oidc-sinatra.pt.identitysandbox.gov/auth/result?code=c409a63a-614a-48f9-874f-c488cfea0bea&state=b6426c42524518350dff2ea8a9d368cf

However, no corresponding failures are reported in the summary as expected.

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
 POST /                                                           592     0(0.00%)    1572    1281    2322  |    1500    1.40    0.00
 POST /login/two_factor/sms                                       568     0(0.00%)   18148     622   28144  |   20000    0.10    0.00
 POST /sign_up/completed                                           26     0(0.00%)   20044   10213   26638  |   20000    0.00    0.00
 GET https://idp.pt.identitysandbox.gov/openid_connect/logout     564     0(0.00%)   11728    5115   24145  |    9400    0.20    0.00
 GET https://sp-oidc-sinatra.pt.identitysandbox.gov               608     0(0.00%)    4744     612   13295  |    4200    2.00    0.00
 GET https://sp-oidc-sinatra.pt.identitysandbox.gov/auth/request?aal=&ial=1     596     0(0.00%)    7061     735   13325  |    7400    1.20    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                      2954     0(0.00%)    8621     612   28144  |    6900    4.90    0.00

Steps To Reproduce

See above.

Expected Behavior

500 errors should result in a request failure and logged to the results.

Actual Behavior

500 errors are ignored and I assume the request is retried.

Suggested Actions

We should make sure the 500 errors (and any others) are logged and counted as request failures. Perhaps we should have some way of confirming the request counts by type using a separate system.

Acceptance Criteria

• 500 errors are logged as request failures