Merge branch 'master' into ap/es6-install

changelog/0084-add-geospatial-queue.yml

@@ -0,0 +1,30 @@
+title: New Geospatial Celery Queue
+key: add-geospatial-queue
+date: 2024-11-06
+optional_per_env: no
+min_commcare_version:
+max_commcare_version: 513700958cf443a85f0480ff681979d207f22db0
+context: |
+  The Microplanning feature requires the location properties on cases to be indexed correctly so
+  that they can be used correctly in this feature. A separate queue is added to handle this indexing
+  process when the Microplanning feature is enabled for a domain. This can potentially be a long-running
+  task, and so having a separate queue ensures that other queues do not get clogged up by this indexing task.
+
+details: |
+  This creates a geospatial_queue in Celery, which will be used for indexing cases when enabling the
+  Microplanning feature.
+
+update_steps: |
+  This update should be performed before updating CommCare to a version more recent than that specified above.
+
+  First, add a queue called `geospatial_queue` to your app_processes.yml,
+  following the examples in commcare-cloud commit [a94636b1d](https://github.com/dimagi/commcare-cloud/commit/a94636b1d).
+
+  Then, to apply this change:
+
+  ```
+  commcare-cloud <env> update-supervisor-confs
+  ```
+
+  Please note that a restart is required for the change to take effect and can be done as part of the above command
+  or with a separately done deploy.

docs/source/changelog/0084-add-geospatial-queue.md

@@ -0,0 +1,38 @@
+<!--THIS FILE IS AUTOGENERATED: DO NOT EDIT-->
+<!--See https://github.com/dimagi/commcare-cloud/blob/master/changelog/README.md for instructions-->
+# 84. New Geospatial Celery Queue
+
+**Date:** 2024-11-06
+
+**Optional per env:** _required on all environments_
+
+
+## CommCare Version Dependency
+CommCare versions beyond the following commit require this change to function correctly:
+[51370095](https://github.com/dimagi/commcare-hq/commit/513700958cf443a85f0480ff681979d207f22db0)
+
+
+## Change Context
+The Microplanning feature requires the location properties on cases to be indexed correctly so
+that they can be used correctly in this feature. A separate queue is added to handle this indexing
+process when the Microplanning feature is enabled for a domain. This can potentially be a long-running
+task, and so having a separate queue ensures that other queues do not get clogged up by this indexing task.
+
+## Details
+This creates a geospatial_queue in Celery, which will be used for indexing cases when enabling the
+Microplanning feature.
+
+## Steps to update
+This update should be performed before updating CommCare to a version more recent than that specified above.
+
+First, add a queue called `geospatial_queue` to your app_processes.yml,
+following the examples in commcare-cloud commit [a94636b1d](https://github.com/dimagi/commcare-cloud/commit/a94636b1d).
+
+Then, to apply this change:
+
+```
+commcare-cloud <env> update-supervisor-confs
+```
+
+Please note that a restart is required for the change to take effect and can be done as part of the above command
+or with a separately done deploy.

docs/source/changelog/index.md

@@ -7,6 +7,14 @@ need to be applied on your environment to keep it up to date.
 
 ### Changelog
 
+#### **2024-11-06** [New Geospatial Celery Queue](0084-add-geospatial-queue.md)
+The Microplanning feature requires the location properties on cases to be indexed correctly so
+that they can be used correctly in this feature. A separate queue is added to handle this indexing
+process when the Microplanning feature is enabled for a domain. This can potentially be a long-running
+task, and so having a separate queue ensures that other queues do not get clogged up by this indexing task.
+
+
+---
 #### **2024-10-25** [Database Backup Retention Change](0083-database_backup_retention_change.md)
 The logic used to retain database backups has been modified to better respect the settings that specify
 how many days to keep backups.

docs/source/reference/1-commcare-cloud/2-configuration.rst

@@ -190,7 +190,8 @@ Each ``<queue-name>`` must be one of the following values:
 ``linked_domain_queue``, ``reminder_case_update_queue``, ``reminder_queue``,
 ``reminder_rule_queue``, ``repeat_record_queue``, ``saved_exports_queue``,
 ``sumologic_logs_queue``, ``send_report_throttled``, ``sms_queue``,
-``submission_reprocessing_queue``, ``ucr_indicator_queue``, ``ucr_queue``.
+``submission_reprocessing_queue``, ``ucr_indicator_queue``, ``ucr_queue``,
+``geospatial_queue``.
 For all features to work, each of these queues must
 appear at least once, and up to once per host.
 

environments/development/app-processes.yml

@@ -9,7 +9,7 @@ celery_processes:
       concurrency: 1
     email_queue,reminder_case_update_queue:
       concurrency: 1
-    linked_domain_queue,reminder_rule_queue,repeat_record_queue,saved_exports_queue:
+    linked_domain_queue,reminder_rule_queue,repeat_record_queue,saved_exports_queue,geospatial_queue:
       concurrency: 1
     ucr_indicator_queue,ucr_queue:
       concurrency: 1

environments/echis/app-processes.yml

@@ -79,6 +79,8 @@ celery_processes:
       prefetch_multiplier: 1
     logistics_reminder_queue:
       concurrency: 2
+    geospatial_queue:
+      concurrency: 1
   echis_server5:
     ucr_indicator_queue:
       concurrency: 15

environments/india/app-processes.yml

@@ -36,6 +36,8 @@ celery_processes:
       concurrency: 1
     background_queue,dashboard_comparison_queue:
       concurrency: 2
+    geospatial_queue:
+      concurrency: 1
 pillows:
   pillow5:
     AppDbChangeFeedPillow:

environments/pna/app-processes.yml

@@ -11,7 +11,7 @@ celery_processes:
       concurrency: 2
     background_queue,export_download_queue,saved_exports_queue,analytics_queue:
       concurrency: 4
-    linked_domain_queue,ucr_queue,async_restore_queue,email_queue,case_rule_queue:
+    linked_domain_queue,ucr_queue,async_restore_queue,email_queue,case_rule_queue,geospatial_queue:
       concurrency: 1
     beat: {}
     celery_periodic:

environments/production/app-processes.yml

@@ -76,6 +76,8 @@ celery_processes:
       prefetch_multiplier: 1
     ush_background_tasks:
       concurrency: 8
+    geospatial_queue:
+      concurrency: 1
 
   celerybeat_a0:
     # not really queues

environments/production/terraform.yml

@@ -67,7 +67,7 @@ servers:
     server_auto_recovery: true
 
   - server_name: "web_m_a2{i}-production"
-    server_instance_type: c6a.8xlarge
+    server_instance_type: m6a.4xlarge
     network_tier: "app-private"
     az: "a"
     volume_size: 40
@@ -77,7 +77,7 @@ servers:
     server_auto_recovery: true
 
   - server_name: "web_m_b2{i}-production"
-    server_instance_type: c6a.8xlarge
+    server_instance_type: m6a.4xlarge
     network_tier: "app-private"
     az: "b"
     volume_size: 40

environments/staging/app-processes.yml

@@ -1,6 +1,5 @@
-formplayer_command_args: ''
 # Use the following to enable datadog tracing
-# formplayer_command_args: '-javaagent:/home/cchq/dd-java-agent.jar -Dsrc.main.java.org.javarosa.enableOpenTracing=true'
+formplayer_command_args: '-javaagent:/home/cchq/dd-java-agent.jar -Dsrc.main.java.org.javarosa.enableOpenTracing=true -Ddd.service=formplayer -Ddd.env=staging'
 
 django_command_prefix: '{{ virtualenv_home }}/bin/ddtrace-run '
 datadog_pythonagent: True
@@ -62,6 +61,8 @@ celery_processes:
       prefetch_multiplier: 1
     ush_background_tasks:
       concurrency: 2
+    geospatial_queue:
+      concurrency: 1
     beat: {}
     flower: {}
 pillows:

environments/swiss/app-processes.yml

@@ -10,7 +10,7 @@ celery_processes:
       concurrency: 1  
     export_download_queue:
       concurrency: 1
-    case_import_queue,reminder_case_update_queue,linked_domain_queue:
+    case_import_queue,reminder_case_update_queue,linked_domain_queue,geospatial_queue:
       concurrency: 1
     background_queue,case_rule_queue:
       concurrency: 1

requirements.txt

@@ -42,7 +42,7 @@ clint==0.5.1
     # via commcare-cloud (setup.py)
 couchdb-cluster-admin==0.7.2
     # via commcare-cloud (setup.py)
-cryptography==42.0.4
+cryptography==43.0.1
     # via
     #   ansible-core
     #   commcare-cloud (setup.py)

setup.py

@@ -11,7 +11,7 @@
     'boto3>=1.9.131',
     'clint',
     'couchdb-cluster-admin',
-    'cryptography~=42.0',
+    'cryptography>=42,<44',
     'datadog>=0.2.0',
     'dimagi-memoized>=1.1.0',
     'dnspython',

src/commcare_cloud/ansible/deploy_webworker.yml

@@ -12,6 +12,31 @@
   roles:
     - webworker
 
+- name: Rotate gunicorn logs
+  hosts: webworkers
+  become: true
+  roles:
+    - role: ansible-logrotate
+      tags:
+      - webworkers
+      - logging
+      logrotate_scripts:
+        - name: "{{ project }}-gunicorn"
+          path: "{{ log_home }}/{{ project }}.gunicorn.log"
+          options:
+            - daily
+            - size 50M
+            - rotate 5
+            - missingok
+            - compress
+            - delaycompress
+            - copytruncate
+            - nocreate
+            - notifempty
+            - sharedscripts
+          scripts:
+            postrotate: "[ -s {{ service_home }}/gunicorn.pid ] && kill -USR1 `cat {{ service_home }}/gunicorn.pid`"
+
 # There used to be some code here for updating workers one by one
 # but it's not functionally useful until we're off fab deploy and makes things
 # look back to commit 94b08e0390bb031dd048804ce5a9c8dfcfeccbc1 if you want it

src/commcare_cloud/ansible/roles/commcarehq/tasks/celery.yml

@@ -37,3 +37,7 @@
         no_proxy: "{% if http_proxy_address is defined %}{{ groups['all'] | join(',') }},{{ app_processes_config.additional_no_proxy_hosts }}{% endif %}"
         PROMETHEUS_PUSHGATEWAY_HOST: "{% if prometheus_celery_pushgateway is defined %}{{ prometheus_celery_pushgateway }}{% endif %}"
         TMPDIR: '{{ encrypted_tmp }}'
+        DD_TRACE_ENABLED: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"
+        DD_TRACE_DJANGO_ENABLED: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"
+        DD_TRACE_DEBUG: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"
+        DD_INSTRUMENTATION_TELEMETRY_ENABLED: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"

src/commcare_cloud/ansible/roles/commcarehq/tasks/management_commands.yml

@@ -11,3 +11,7 @@
     - env_vars:
         TMPDIR: '{{ encrypted_tmp }}'
         PROMETHEUS_MULTIPROC_DIR: "{% if prometheus_monitoring_enabled|default(False) %}{{ metrics_home }}{% endif %}"
+        DD_TRACE_ENABLED: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"
+        DD_TRACE_DJANGO_ENABLED: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"
+        DD_TRACE_DEBUG: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"
+        DD_INSTRUMENTATION_TELEMETRY_ENABLED: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"

src/commcare_cloud/ansible/roles/commcarehq/tasks/pillowtop.yml

@@ -12,3 +12,7 @@
         TMPDIR: '{{ encrypted_tmp }}'
         READ_FROM_PLPROXY_STANDBYS: 1
         PROMETHEUS_MULTIPROC_DIR: "{% if prometheus_monitoring_enabled|default(False) %}{{ metrics_home }}{% endif %}"
+        DD_TRACE_ENABLED: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"
+        DD_TRACE_DJANGO_ENABLED: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"
+        DD_TRACE_DEBUG: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"
+        DD_INSTRUMENTATION_TELEMETRY_ENABLED: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"

src/commcare_cloud/ansible/roles/commcarehq/tasks/webworkers.yml

@@ -19,4 +19,8 @@
         DD_REQUESTS_SERVICE_NAME: "{% if app_processes_config.datadog_pythonagent %}requests{% endif %}"
         DD_REQUESTS_SPLIT_BY_DOMAIN: "{% if app_processes_config.datadog_pythonagent %}True{% endif %}"
         DD_ENV: "{% if app_processes_config.datadog_pythonagent and app_processes_config.django_command_prefix %}{{ env_monitoring_id }}{% endif %}"
+        DD_TRACE_ENABLED: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"
+        DD_TRACE_DJANGO_ENABLED: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"
+        DD_TRACE_DEBUG: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"
+        DD_INSTRUMENTATION_TELEMETRY_ENABLED: "{% if not app_processes_config.datadog_pythonagent %}False{% endif %}"
       gunicorn_workers: "{{ app_processes_config.gunicorn_workers_static_factor + (ansible_processor_vcpus * app_processes_config.gunicorn_workers_factor)|int }}"

src/commcare_cloud/ansible/roles/commcarehq/templates/supervisor_celery_workers.conf.j2

@@ -3,7 +3,7 @@
 
 {% if queue_names not in ("flower", "beat") %}
 [program:{{ project }}-{{ deploy_env }}-celery_{{ queue_names }}_{{worker_num}}]
-environment={% for name, value in item.env_vars.items() %}{% if value %}{{ name }}="{{ value }}",{% endif %}{% endfor %}
+environment={% for name, value in item.env_vars.items() %}{% if value|string|length > 0 %}{{ name }}="{{ value }}",{% endif %}{% endfor %}
 
 command=/bin/bash {{ service_home }}/{{ deploy_env }}_celery_bash_runner.sh
     --events --loglevel=INFO --without-gossip

src/commcare_cloud/ansible/roles/commcarehq/templates/supervisor_django.conf.j2

@@ -1,7 +1,7 @@
 [program:{{ project }}-{{ deploy_env }}-django]
 directory={{ code_home }}/
 ; gunicorn
-environment={% for name, value in item.env_vars.items() %}{% if value %}{{ name }}="{{ value }}",{% endif %}{% endfor %}
+environment={% for name, value in item.env_vars.items() %}{% if value|string|length > 0 %}{{ name }}="{{ value }}",{% endif %}{% endfor %}
 
 command={{ app_processes_config.django_command_prefix }}{{ virtualenv_home }}/bin/gunicorn
     deployment.gunicorn.commcarehq_wsgi:application

src/commcare_cloud/ansible/roles/commcarehq/templates/supervisor_formplayer_spring.conf.j2

@@ -1,5 +1,5 @@
 [program:{{ project }}-{{ deploy_env }}-formsplayer-spring]
-environment={% for name, value in item.env_vars.items() %}{% if value %}{{ name }}="{{ value }}",{% endif %}{% endfor %}
+environment={% for name, value in item.env_vars.items() %}{% if value|string|length > 0 %}{{ name }}="{{ value }}",{% endif %}{% endfor %}
 
 command={{formplayer_java}} {{ app_processes_config.formplayer_command_args }} {% if use_formplayer_debug_options %} -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9998 {% endif %} -Xms{{ app_processes_config.formplayer_memory }} -Xmx{{ app_processes_config.formplayer_memory }} {{ extra_formplayer_args }} -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize={{ app_processes_config.formplayer_g1heapregionsize }} -XX:MaxMetaspaceSize={{ app_processes_config.formplayer_maxmetaspacesize }} -XX:-OmitStackTraceInFastThrow -Xss1024k {% if use_formplayer_debug_options %} -XX:NativeMemoryTracking=summary -XX:+UnlockDiagnosticVMOptions {% endif %} -jar {{ www_home }}/formplayer_build/current/formplayer.jar
 user={{ cchq_user }}

src/commcare_cloud/ansible/roles/commcarehq/templates/supervisor_management_commands.conf.j2

@@ -2,7 +2,7 @@
 [program:{{ project }}-{{ deploy_env }}-{{ command_name }}]
 directory={{ code_home }}
 environment=TMPDIR="{{ encrypted_tmp }}"
-environment={% for name, value in item.env_vars.items() %}{% if value %}{{ name }}="{{ value }}",{% endif %}{% endfor %}
+environment={% for name, value in item.env_vars.items() %}{% if value|string|length > 0 %}{{ name }}="{{ value }}",{% endif %}{% endfor %}
 
 command={{ django_bash_runner if prometheus_monitoring_enabled else django_direct_runner }} {{ command_name }}
 user={{ cchq_user }}

src/commcare_cloud/ansible/roles/commcarehq/templates/supervisor_pillowtop.conf.j2

@@ -2,7 +2,7 @@
 {% for num_process in range(params.get('start_process', 0), params.get('start_process', 0) + params.get('num_processes', 1)) %}
 
 [program:commcare-hq-{{ deploy_env }}-pillowtop-{{ pillow_name }}-{{ num_process }}]
-environment={% for name, value in item.env_vars.items() %}{% if value %}{{ name }}="{{ value }}",{% endif %}{% endfor %}
+environment={% for name, value in item.env_vars.items() %}{% if value|string|length > 0 %}{{ name }}="{{ value }}",{% endif %}{% endfor %}
 
 command={{ django_bash_runner if prometheus_monitoring_enabled else django_direct_runner }} run_ptop
     --pillow-name {{ pillow_name }}

src/commcare_cloud/ansible/roles/common/templates/etc_aliases.j2

@@ -19,6 +19,11 @@ couchdb: root
 rabbitmq: root
 kafka: root
 zookeeper: root
+monit: root
+ansible: root
+{% if cchq_user is defined %}
+{{ cchq_user }}: root
+{% endif %}
 {% if root_email is defined %}
 root: {{ root_email }}
 {% endif %}

src/commcare_cloud/commands/deploy/commcare.py

@@ -248,8 +248,12 @@ def get_deployed_version(environment, from_source=False):
     # hosts. A private release can be converted to a full release
     # using --resume=RELEASE_NAME without --private.
     versions = {host_result.get('stdout') for host_result in res.values() if host_result.get('rc') == 0}
-    if not versions or len(versions) > 1:
-        raise BadAnsibleResult("Unable to get version from hosts")
+    if not versions:
+        raise BadAnsibleResult("Unable to get version from hosts: no versions found. Try again soon.")
+    if len(versions) > 1:
+        raise BadAnsibleResult(
+            f"Unable to get version from hosts. Multiple versions found: {versions}. Try again soon."
+        )
     return list(versions)[0]
 
 

src/commcare_cloud/environment/schemas/app_processes.py

@@ -137,7 +137,8 @@ def __new__(cls, name, required=True, is_queue=True, blockage_threshold=None,
     CeleryProcess("ucr_indicator_queue", required=False, blockage_threshold=60 * 60),
     CeleryProcess("ucr_queue", required=False, blockage_threshold=60 * 60),
     CeleryProcess("user_import_queue", required=False, blockage_threshold=60 * 60),
-    CeleryProcess("ush_background_tasks", required=False, blockage_threshold=3 * 60 * 60)
+    CeleryProcess("ush_background_tasks", required=False, blockage_threshold=3 * 60 * 60),
+    CeleryProcess("geospatial_queue", required=False, blockage_threshold=6 * 60 * 60)
 ]