[timebox: 5d] Catalog load- and stress-testing in cloud.gov

[adborden]

User Story

In order to ensure sufficient capacity and performance for catalog, data.gov wants spend up to 5 days running load- and stress-testing activities on the deployment in cloud.gov.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• WHEN I perform load-testing on catalog
  THEN the sustainable request load is sufficient to handle typical traffic loads we've seen to catalog in the FCS environment
  AND sufficient to handle routine peak traffic.
• WHEN I look at the available quota vs the memory in use
  THEN I see a comfortable margin for spinning up more instances if needed.
• We're satisfied that any other immediate concerns that arose during the 5d are properly triaged.

Background

See notes from the previous load-testing

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]
By right-sizing our app deployments, we'll be less likely to suffer self-imposed DoS conditions when normal peak traffic happens. (Actual attack is a different concern.)

Sketch

• Implement a traffic profile in GSA/datagov-load-testing
• Perform a baseline test with the current catalog.
• Decide how many instances of catalog are sufficient for expected traffic levels
• Run harvests and see that they don't fail
• Run reindexes and see that they don't fail
• Verify datasets with deep pagination are returning OK
• Verify we can add/delete users
• Verify Solr 504 Gateway Time-out #3636 is addressed

[adborden]

Let's consider this as part of the cloud.gov work.

[nickumia-reisys]

Updates:

• Recent access log analysis done
• Minor optimizations to datagov-load-testing repo
• Will perform complete load test once more data is populated in the cloud.gov environment.

[mogul]

Having recently done some tuning on Solr, we're now waiting for a Solr reindex before we conduct more tests.

[nickumia-reisys]

To summarize the state of this issue:

• Solr as a custom brokered cloud foundry service was a bit premature due to complexities between the datagov-ssb, datagov-brokerpak-eks and datagov-brokerpak-solr repositories coupled with the Solr 6/8 integration with CKAN.
• A different process has been created for deploying Solr 6 (which Inventory still relies on as of writing) and Solr 8 (which Catalog has been upgraded to be compatible with)
• This issue was held back by the following issues,
  • Solr service entering unrecoverable state under load #3603
  • Enable 307 redirects for EKS load-balancer (using NLBs instead of ALBs) #3677
  • Random Loss of Credentials apache/solr-operator#399
  • Solr Resiliency Issues #3647
  • Solr 504 Gateway Time-out #3636
  • Terraform driven eks deploy alternative providers datagov-ssb#123
  • Add support for Bind Module Reference GSA-TTS/datagov-brokerpak-eks#79
  • Limit external-dns to exactly one zone GSA-TTS/datagov-brokerpak-eks#82
  • Solr Optimizations for CKAN Integrations GSA-TTS/datagov-brokerpak-solr#32
• Solr has been the hurdle to catalog resiliency. We have fixed all of the issues impeding catalog solr from operating and have faith that solr will pass this round of load testing.
• We've been through multiple iterations of spinning up a full catalog deployment and fixing core functionality bugs.
• Catalog staging is currently undergoing solr reindexing, after which time this issue will be actively completed.

Total ETA for Solr Index: 59 hours
Estimated completed time: Wednesday, March 9th, morning

[nickumia-reisys]

Update: I just realized that the current version of the solr-brokerpak doesn't have persistent storage enabled at the solrcloud level, even though eks-brokerpak supports it now... so we'll have to restart the index once the correct solr is deployed.

[mogul]

Here's a useful reference on optimizing Solr that I just rediscovered.

[nickumia-reisys]

Concerning the most recent load testing, the data.gov team believes that our cloud.gov setup is resilient enough to migrate and then address two uncommon situations as a later time. At the current time, we are achieving upwards of 20 sustained requests per second. With certain network-capable testing environments, we've achieved ~70 requests per second.

The two problems carrying forward related to:

Deep pagination Solr search issues (#3636 and #3642)

Harvest source page issues (#3749)

Summary of Load testing results:

Sustained >30 RPM

Response time percentiles (approximated)
 Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|---------|------|------|------|------|------|------|------|------|------|------|------|
 GET      api-group-list                                                                        130    130    140    150    280    300    310    330    330    330    330     55
 GET      api-organization-list                                                                 130    140    150    160    420    430    450    710    710    710    710     62
 GET      api-package-search                                                                    280    310    330    340    400    470    620    900   1900   5000   5300  23644
 GET      api-package-search-harvest                                                            190    220    230    250    300    310    320    610    610    610    610     63
 GET      api-package-show                                                                      200    220    240    250    300    360    450    610   1300   2100   3000  13043
 GET      dataset                                                                               810    920   1000   1100   1300   1500   1700   1900   2700   5600   5600   4636
 GET      dataset_search                                                                       2600   3100   3400   3600   4100   4700   5400   6000   9000  65000  65000   8271
 GET      datasets-home                                                                        2800   3200   3500   3700   4200   4700   5400   6100  14000  63000  63000   3750
 GET      group                                                                                2100   2500   2900   3100   3700   4200   4900   5500   9800  63000  63000  13953
 GET      groups-home                                                                           650    790    810    900   1300   1900   1900   2100   2100   2100   2100     52
 GET      home                                                                                 3000   3400   3600   3800   4400   4900   5600   6500  12000  64000  64000  16420
 GET      organization                                                                         2000   2500   2900   3100   3800   4500   5300   6000  10000  63000  64000  21376
 GET      organizations-home                                                                   1700   1900   2100   2200   2700   3000   3300   3700   3800   3800   3800    158
 GET      static_assets                                                                         170    210    210    220    240    260    310    350   1200   2700   4200  14950
--------|--------------------------------------------------------------------------------|---------|------|------|------|------|------|------|------|------|------|------|------|
 None     Aggregated                                                                            760   2100   2600   2800   3500   4100   4800   5400   8800  63000  65000 120433

 Name                                                                              # reqs      # fails  |     Avg     Min     Max  Median  |   req/s failures/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------
 GET api-group-list                                                                    55     0(0.00%)  |     146     105     329     130  |    0.02    0.00
 GET api-organization-list                                                             62     0(0.00%)  |     182     113     705     130  |    0.02    0.00
 GET api-package-search                                                             23644     0(0.00%)  |     302     161    5337     280  |    6.57    0.00
 GET api-package-search-harvest                                                        63     0(0.00%)  |     211     149     608     190  |    0.02    0.00
 GET api-package-show                                                               13043     0(0.00%)  |     226     144    2979     200  |    3.62    0.00
 GET dataset                                                                         4636     0(0.00%)  |     885     413    5641     810  |    1.29    0.00
 GET dataset_search                                                                  8271     3(0.04%)  |    2719     186   64881    2600  |    2.30    0.00
 GET datasets-home                                                                   3750     0(0.00%)  |    3078    1609   63476    2800  |    1.04    0.00
 GET group                                                                          13953     0(0.00%)  |    2240     505   62959    2100  |    3.88    0.00
 GET groups-home                                                                       52     0(0.00%)  |     796     490    2062     650  |    0.01    0.00
 GET home                                                                           16420     1(0.01%)  |    3196     586   64159    3000  |    4.56    0.00
 GET organization                                                                   21376    11(0.05%)  |    2090     298   64382    2000  |    5.94    0.00
 GET organizations-home                                                               158     0(0.00%)  |    1787    1043    3813    1700  |    0.04    0.00
 GET static_assets                                                                  14950     0(0.00%)  |     180     100    4215     170  |    4.15    0.00
----------------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                                        120433    15(0.01%)  |    1492     100   64881     760  |   33.46    0.00