identity-loadtest

Load testing scripts and tooling for the Login.gov, currently using locust.

Local setup

Python and Locust

Install python3 and dependencies

brew install python
pip3 install -r requirements.txt

Login.gov IdP

Login IdP must be running with these settings in application.yml

telephony_adapter: 'test'
disable_email_sending: 'true'
enable_load_testing_mode: 'true'
enable_rate_limiting: 'false'
otp_delivery_blocklist_maxretry: 1000000

Note that it's also important to assure that the scrypt cost is configured the same as prod. The values will inherit from the idp-repo unless the values are overridden in the secrets s3 bucket.

Running Locust

You can only run one locustfile at a time, there are many to choose from that end in .locustfile.py.

• users is the total number of concurrent Locust users.
• hatch-rate is the number of users to spawn per second, starting from zero.

Common locust cmd line arguments

--host http://localhost:3000 --users 1 --spawn-rate 1 --run-time 15m --headless

Or omit --headless and open http://localhost:8089 for a UI.

Add --csv=<base-name> to generate CSV output

Adding new tests

Add new *.loucstfile.py files to the project for new test scenarios.

Sign-Up load test

• This will create lots of users in your database

locust --locustfile load_testing/sign_up.locustfile.py --host http://localhost:3000 --users 1 --spawn-rate 1 --run-time 15m --headless

Sign-In load test

• You must run a rake task in the IdP before using this test, something like: rake dev:random_users NUM_USERS=100 (source)
• You also must pass in a matching NUM_USERS=100 to the locust call.

NUM_USERS=100 locust --locustfile load_testing/sign_in.locustfile.py --host http://localhost:3000 --users 1 --spawn-rate 1 --run-time 15m --headless

Sign-In remembered device load test

Tests sign ins simulating a very high (90%) ratio of users who are signing back in using a remembered browser (device).

• You must run a rake task in the IdP before using this test, something like: rake dev:random_users NUM_USERS=100' (source)
• You also must pass in a matching NUM_USERS=100 to the locust call.

NUM_USERS=100 locust --locustfile load_testing/sign_in_remember_me.locustfile.py --host http://localhost:3000 --users 1 --spawn-rate 1 --run-time 15m --headless

Sign up + Sign-In load test

• This test mixes Sign-up and Sign-in together
• You must run the same rake task as above in the IdP before using this test
• You also must pass in a matching NUM_USERS=100 to the locust call.

NUM_USERS=100 locust --locustfile load_testing/sign_up_sign_in.locustfile.py --host http://localhost:3000 --users 1 --spawn-rate 1 --run-time 15m --headless

IAL2 load tests

• Same rules as above, but use ial2_sign_* filenames.
• Uses "desktop proofing" experience, not mobile.
• Requires two images that represent the front and back of a drivers license. By default we have included two files, mock-front.jpeg and mock-back.jpeg which were chosen since they are similar to the average size of the images used on login.gov.

NUM_USERS=100 locust --locustfile load_testing/ial2_sign_in.locustfile.py --host http://localhost:3000 --users 1 --spawn-rate 1 --run-time 15m --headless

NUM_USERS=100 locust --locustfile load_testing/ial2_sign_up.locustfile.py --host http://localhost:3000 --users 1 --spawn-rate 1 --run-time 15m --headless

SP Sign in load test

• This requires that identity-oidc-sinatra be running as an SP
• This requires the NUM_USERS env varible
• This requires the SP_HOST env varible, something like SP_HOST=http://localhost:9292

NUM_USERS=100 SP_HOST=http://localhost:9292 locust --locustfile load_testing/sp_sign_in.locustfile.py --host http://localhost:3000 --users 1 --spawn-rate 1 --run-time 15m --headless

Production Simulator load test

This is a hybrid test with the test mix roughly matching Login.gov's workload. (Subject to change. See test source for details.)

The ratio of remembered devices for sign ins can be adjusted with the REMEMBERED_PERCENT variable. (Default: 60)

For uniformity and simple calculation, test ratios should add up to 10000 (1 == 0.01%) and can be adjusted by setting a corresponding environment variable. The following are available, and defaults can be found at the top of load_testing/production_simulator.locustfile.py:

• RATIO_SIGN_IN: Sign in test using REMEMBERED_PERCENT remember me ratio.
• RATIO_SIGN_UP: Sign up test ratio.
• RATIO_SIGN_IN_AND_PROOF: Sign in followed by IAL2 proofing ratio.
• RATIO_SIGN_UP_AND_PROOF: Sign up followed by IAL2 proofing ratio.
• RATIO_SIGN_IN_USER_NOT_FOUND: Failed sign in with nonexistent user.
• RATIO_SIGN_IN_INCORRECT_PASSWORD: Failed sign in with bad password.
• RATIO_SIGN_IN_INCORRECT_SMS_OTP: Failed sign in with bad SMS OTP.

Test requirements:

• Requires prepopulated users (See Sign-In load test)
• You also must pass in a matching NUM_USERS=100 to the locust call.

Example (including overrides of the sign in and sign up tests)

NUM_USERS=100 RATIO_SIGN_IN=5000 RATIO_SIGN_UP=1010 locust --locustfile load_testing/production_simulator.locustfile.py --host http://localhost:3000 --users 1 --spawn-rate 1 --run-time 15m --headless

Running the test suite

There are tests for these load tests, find them in the tests folder.

# Run the tests
pytest

# Run the tests and show coverage
coverage run -m pytest
coverage report

If you install the CircleCI CLI you can test a CircleCI run in a local Docker container with circleci local execute.

Debugging Locust scripts

The HTTP Library is called Requests: https://requests.readthedocs.io/en/master/

The python debugger should just work. Here are some commands The following will drop you into a debugger:

import pdb; pdb.set_trace()