Benchmark performance for xAPI over Celery to Ralph

[bmtcril]

Tests for various Celery configurations.

Remaining tests:

• Multiple workers with batching
• 10k row single worker with 100 batch size

[bmtcril]

Test 1 (913dd5) - 1k rows, no batching

Test system configuration:

• Tutor version: tutor, version 17.0.2-nightly
• Aspects version: 0.91.0
• Environment specifications:
  • Tutor local / MacOS / OrbStack
  • 10GiB RAM
  • 5 CPU limit
  • Storage unlimited
  • Tutor plugins: aspects, mfe
• Relevant settings
  • RUN_CLICKHOUSE: true
  • RUN_KAFKA_SERVER: false
  • RUN_RALPH: true
  • RUN_SUPERSET: true
  • RUN_VECTOR: false
  • EVENT_ROUTING_BACKEND_BATCHING_ENABLED: False

Load generation specifications:

• Tool - platform-plugin-aspects load_test_tracking_events management command
• Exact scripts
  • tutor local run lms ./manage.py lms monitor_load_test_tracking --sleep_time 5 --backend celery
  • tutor local run cms ./manage.py cms load_test_tracking_events --num_events 1000 --sleep_time 0 --tags celery 1k local novector nobatch

Data captured for results:

• Length of run
  • Event generation: 0:00:38.091298
  • Monitoring / total run length: 0:07:02.613890
• Sleep time: 0
• Events: 1000
• Raw stats attached: 913dd5_stats.txt

Findings:

• Celery could not keep up, ClickHouse was 376 seconds behind on a 416 second run at the end
• Celery queue grew to 1,340 pending events before the generation stopped and then took 380 seconds to catch up
• Inserted rows per second: 2.2 - 2.6

[bmtcril]

Test 2 (972dc4) - 1k rows, batching 10

Test system configuration:

• Tutor version: tutor, version 17.0.2-nightly
• Aspects version: 0.91.0
• Environment specifications:
  • Tutor local / MacOS / OrbStack
  • 10GiB RAM
  • 5 CPU limit
  • Storage unlimited
  • Tutor plugins: aspects, mfe
• Relevant settings
  • RUN_CLICKHOUSE: true
  • RUN_KAFKA_SERVER: false
  • RUN_RALPH: true
  • RUN_SUPERSET: true
  • RUN_VECTOR: false
  • EVENT_ROUTING_BACKEND_BATCHING_ENABLED: True
  • EVENT_ROUTING_BACKEND_BATCH_SIZE: 10

Load generation specifications:

• Tool - platform-plugin-aspects load_test_tracking_events management command
• Exact scripts
  • tutor local run lms ./manage.py lms monitor_load_test_tracking --sleep_time 5 --backend celery
  • tutor local run cms ./manage.py cms load_test_tracking_events --num_events 1000 --sleep_time 0 --tags celery 1k local novector batch10

Data captured for results:

• Length of run
  • Event generation: 0:00:57.751662
  • Monitoring / total run length: 0:01:09.289254
• Sleep time: 0
• Events: 1000
• No stats captured

Findings:

• Celery could barely keep up, but was only 12 seconds behind by the end of the run
• Celery queue remained even and did not grow over this test
• Inserted rows per second: 23 - 25

[saraburns1]

Test 3 (bcd1b3) - 10k rows, batching 100

Test system configuration:

• Tutor version: tutor, version 18.1.1-nightly
• Aspects version: 1.0.0
• Environment specifications:
  • Tutor local / MacOS / OrbStack
  • 10GiB RAM
  • 5 CPU limit
  • Storage unlimited
  • Tutor plugins: aspects, mfe
• Relevant settings
  • RUN_CLICKHOUSE: true
  • RUN_KAFKA_SERVER: false
  • RUN_RALPH: true
  • RUN_SUPERSET: true
  • RUN_VECTOR: false
  • EVENT_ROUTING_BACKEND_BATCHING_ENABLED: True
  • EVENT_ROUTING_BACKEND_BATCH_SIZE: 100

Load generation specifications:

• Tool - platform-plugin-aspects load_test_tracking_events management command
• Exact scripts
  • tutor local run lms ./manage.py lms monitor_load_test_tracking --sleep_time 5 --backend celery
  • tutor local run cms ./manage.py cms load_test_tracking_events --num_events 10000 --sleep_time 0 --tags celery 10k batch100

Data captured for results:

• Length of run
  • Event generation: 0:04:48
  • Monitoring / total run length: 0:05:10
• Sleep time: 0
• Events: 10000
• No stats captured

Findings:

• Celery kept up, 22 seconds behind by the end of the run
• Celery queue remained even and did not grow over this test
• Inserted rows per second: ~32

[saraburns1]

Test 4 (730cd4) - 100k rows, batching 100

Test system configuration:

• Tutor version: tutor, version 18.1.1-nightly
• Aspects version: 1.0.0
• Environment specifications:
  • Tutor local / MacOS / OrbStack
  • 10GiB RAM
  • 5 CPU limit
  • Storage unlimited
  • Tutor plugins: aspects, mfe
• Relevant settings
  • RUN_CLICKHOUSE: true
  • RUN_KAFKA_SERVER: false
  • RUN_RALPH: true
  • RUN_SUPERSET: true
  • RUN_VECTOR: false
  • EVENT_ROUTING_BACKEND_BATCHING_ENABLED: True
  • EVENT_ROUTING_BACKEND_BATCH_SIZE: 100

Load generation specifications:

• Tool - platform-plugin-aspects load_test_tracking_events management command
• Exact scripts
  • tutor local run lms ./manage.py lms monitor_load_test_tracking --sleep_time 5 --backend celery
  • tutor local run cms ./manage.py cms load_test_tracking_events --num_events 100000 --sleep_time 0 --tags celery 100k batch100

Data captured for results:

• Length of run
  • Event generation: 1:07:07
  • Monitoring / total run length: 1:07:43
• Sleep time: 0
• Events: 100,000
• No stats captured

Findings:

• Celery kept up well, less than 40 seconds behind
• Celery queue remained even and did not grow over this test
• Inserted rows per second: ~24

[saraburns1]

Performance testing findings here:
https://docs.google.com/document/d/1skhbV3V1f85Qr0Bb-QOltKVxOKdcuT9i3HmMfcq-SVo/edit

Summary
Processing with Celery and batch size of 100 is already as performant as it can be. Adding additional parallelism/concurrency makes no difference or decreases performance.

[bmtcril]

Seems like we've managed the goal of making sure insert performance remains good enough as of v1 so I'm going to close these tasks out.