Merge branch 'master' into sr/connectid

docs/source/installation/2-manual-install.rst

@@ -70,7 +70,7 @@ cluster’s proxy server.
 
 Prepare all machines for automated deploy
 -----------------------------------------
-Do the following on the monolith, or on each machine in the cluster.
+Do the following on the monolith or on each machine in the cluster.
 
 Enable root login via SSH
 ~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -111,6 +111,12 @@ reference.
 
         PasswordAuthentication yes
 
+    To allow keyboard interactive authentication, ensure
+
+    ::
+
+        KbdInteractiveAuthentication yes
+
 4.  Restart SSH:
 
     ::
@@ -126,46 +132,14 @@ To be used in the installation process.
     $ sudo touch /var/log/ansible.log
     $ sudo chmod 666 /var/log/ansible.log
 
-Install system dependencies
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-This only needs to be done on the control machine. In the case of a monolith,
+Prepare control machine for automated deploy
+--------------------------------------------
+The following steps only need to be done on the control machine. In the case of a monolith,
 there is only one machine to manage so that is also the control machine. In
 our example cluster, the control machine is named “control1”.
 
-
-1.  SSH into control1 as the “ansible” user, or the user you created during installation. You can skip this step if you are installing a monolith:
-
-    ::
-
-        $ ssh ansible@control1
-
-    This instruction assumes that the control machine’s name resolves to its IP address.
-    Replace the name with the IP address if necessary.
-
-2.  On the control machine, or the monolith, install required packages:
-
-    ::
-
-        $ sudo apt update
-        $ sudo apt install python3-pip python3-dev python3-distutils python3-venv libffi-dev sshpass net-tools
-
-3.  Check your default Python version for Python 3.x:
-
-    ::
-
-        $ python --version
-
-    If your default version is not 3.x or if the “python” command was
-    not found, make python3 your default by running the command below,
-    otherwise skip it.
-
-    ::
-
-        $ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 10
-
-
 Create a user for yourself
---------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 In general, CommCare environments are managed by a team. Each member of
 the team has their own user account.
@@ -180,22 +154,44 @@ add them to the “sudo” user group. For example, if your username were
    ...
    $ sudo usermod -a -G sudo jbloggs
 
+Switch to this user for the remainder of these setup steps:
+
+::
 
-Install CommCare Cloud
-----------------------
+    $ su - jbloggs
 
-1.  On the control machine or the monolith, install and configure Git:
+
+Install system dependencies
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1.  Install the required packages:
+
+    ::
+
+        $ sudo apt update
+        $ sudo apt install python3-pip python3-dev python3-distutils python3-venv libffi-dev sshpass net-tools git
+
+2.  Configure Git:
 
     ::
 
-        $ sudo apt install git
         $ git config --global user.name "Jay Bloggs"
         $ git config --global user.email "[email protected]"
 
     (Of course, substitute “Jay Bloggs” with your name, and
     “[email protected]” with your email address.)
 
-2.  Clone and initialize CommCare Cloud:
+3.  Make python3 default for python command:
+
+    ::
+
+        $ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 10
+
+
+Install and Configure CommCare Cloud
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1.  Clone and initialize CommCare Cloud:
 
     ::
 
@@ -211,15 +207,23 @@ Install CommCare Cloud
         Do you want to have the CommCare Cloud environment setup on login?
         (y/n): y
 
-3.  Clone the sample CommCare Cloud “environments” folder into your home
+    If the input times out before entering 'y', you can follow the prompt instructions
+    to setup CommCare Cloud on login:
+
+    ::
+
+        $ echo '[ -t 1 ] && source ~/init-ansible' >> ~/.profile
+
+
+2.  Clone the sample CommCare Cloud “environments” folder into your home
     directory.
 
     ::
 
         $ cd ~
         $ git clone https://github.com/dimagi/sample-environment.git environments
 
-4.  Rename your environment. You could name it after your organization
+3.  Rename your environment. You could name it after your organization
     or your project. If you are installing a monolith you could leave
     its name as “monolith”. For this example we will name it “cluster”.
 
@@ -229,36 +233,36 @@ Install CommCare Cloud
         $ git mv monolith cluster
         $ git commit -m "Renamed environment"
 
-5.  Remove the “origin” Git remote. (You will not be pushing your
+4.  Remove the “origin” Git remote. (You will not be pushing your
     changes back to the Dimagi “sample-environment” repository.)
 
     ::
 
         $ git remote remove origin
 
-6.  (Optional) You are encouraged to add a remote for your own Git
+5.  (Optional) You are encouraged to add a remote for your own Git
     repository, so that you can share and track changes to your
     environment’s configuration. For example:
 
     ::
 
         $ git remote add origin [email protected]:your-organization/commcare-environment.git
 
-7.  Configure your CommCare environment.
+6.  Configure your CommCare environment.
 
     See :ref:`configure-env` for more information.
 
-8.  Add your username to the ``present`` section of
+7.  Add your username to the ``present`` section of
     ``~/environments/_users/admins.yml``.
 
     ::
 
        $ nano ~/environments/_users/admins.yml
 
-9.  Copy your **public** key to ``~/environments/_authorized_keys/``.
+8.  Copy your **public** key to ``~/environments/_authorized_keys/``.
     The filename must correspond to your username.
 
-10. Change “monolith.commcarehq.test” to your real domain name,
+9. Change “monolith.commcarehq.test” to your real domain name,
 
     ::
 
@@ -281,8 +285,7 @@ Install CommCare Cloud
        -  ``ALLOWED_HOSTS``
 
 
-
-11. Change default emails
+10. Change default emails
 
     ::
 
@@ -291,7 +294,7 @@ Install CommCare Cloud
     You should find references in ``public.yml``
 
 
-12. Configure ``inventory.ini``
+11. Configure ``inventory.ini``
 
     .. rubric:: For a monolith
        :name: for-a-monolith
@@ -429,7 +432,7 @@ Install CommCare Cloud
        db1
        db2
 
-13. Configure the ``commcare-cloud`` command.
+12. Configure the ``commcare-cloud`` command.
 
     ::
 
@@ -467,13 +470,13 @@ Install CommCare Cloud
 
        $ cp ~/commcare-cloud/src/commcare_cloud/config.example.py ~/commcare-cloud/src/commcare_cloud/config.py
 
-    Update the known hosts file
+    Update the known hosts file (substituting your environment name if necessary)
 
     ::
 
        $ commcare-cloud cluster update-local-known-hosts
 
-14. Generate secured passwords for the vault
+13. Generate secured passwords for the vault
 
     In this step, we’ll generate passwords in the ``vault.yml`` file.
     This file will store all the passwords used in this CommCare
@@ -494,7 +497,7 @@ Install CommCare Cloud
     Find the value of “ansible_sudo_pass” and record it in your password
     manager. We will need this to deploy CommCare HQ.
 
-15. Encrypt the provided vault file, using that “ansible_sudo_pass”. (As
+14. Encrypt the provided vault file using a newly generated password. (As
     usual, substitute “cluster” with the name of your environment.)
 
     ::
@@ -511,26 +514,31 @@ will tell you more about how we use this vault file.
 Deploy CommCare HQ services
 ---------------------------
 
-You will need the SSH agent to have your SSH key for Ansible.
+The first step is to setup the expected user configuration. You will be prompted for
+the vault password from earlier and the SSH password, which is the root user's password.
+After this step, the root user will not be able to log in via SSH.
 
 ::
 
-   $ eval `ssh-agent`
-   $ ssh-add ~/.ssh/id_rsa
+    $ commcare-cloud cluster bootstrap-users
 
-When you run the “commcare-cloud deploy-stack”, you will be prompted for
-the vault password from earlier. You will also be prompted for an SSH
-password. This is the root user’s password. After this step, the root
-user will not be able to log in via SSH.
+
+Once this completes successfully, you will now be able to ssh into this machine from your previously created user (e.g., jbloggs).
+You should exit your current ssh session, and ssh back into the machine using the "-A" option to enable agent forwarding.
+This is necessary to escalate privileges when running commcare-cloud commands, as well as for executing commands on other machines if
+you are setting up a cluster.
 
 ::
 
-   $ commcare-cloud cluster deploy-stack --first-time -e 'CCHQ_IS_FRESH_INSTALL=1'
+    $ exit  # exit until no longer connected to the machine
+    $ ssh -A jbloggs@control1
+
+
+You are now ready to deploy CommCare HQ services.
+
+::
 
-   This command will apply without running the check first. Continue? [y/N]y
-   ansible-playbook /home/jbloggs/commcare-cloud/src/commcare_cloud/ansible/deploy_stack.yml -i /home/jbloggs/environments/cluster/inventory.ini -e @/home/jbloggs/environments/cluster/vault.yml -e @/home/jbloggs/environments/cluster/public.yml -e @/home/jbloggs/environments/cluster/.generated.yml --diff --tags=bootstrap-users -u root --ask-pass --vault-password-file=/home/jbloggs/commcare-cloud/src/commcare_cloud/ansible/echo_vault_password.sh --ask-pass --ssh-common-args -o=UserKnownHostsFile=/home/jbloggs/environments/cluster/known_hosts
-   Vault Password for 'cluster': <ansible_sudo_pass>
-   SSH password: <root user's password>
+   $ commcare-cloud cluster deploy-stack -e 'CCHQ_IS_FRESH_INSTALL=1'
 
 This will run a series of Ansible commands that will take quite a long
 time to run.

environments/india/public.yml

@@ -139,23 +139,23 @@ localsettings:
   ES_CASE_SEARCH_INDEX_NAME: "case_search_2022-10-14"
   ES_MULTIPLEX_TO_VERSION: '6'
   # Index Multiplexer Settings  
-  ES_APPS_INDEX_MULTIPLEXED: True
-  ES_CASE_SEARCH_INDEX_MULTIPLEXED: True
-  ES_CASES_INDEX_MULTIPLEXED: True
-  ES_DOMAINS_INDEX_MULTIPLEXED: True
-  ES_FORMS_INDEX_MULTIPLEXED: True
-  ES_GROUPS_INDEX_MULTIPLEXED: True
-  ES_SMS_INDEX_MULTIPLEXED: True
-  ES_USERS_INDEX_MULTIPLEXED: True
+  ES_APPS_INDEX_MULTIPLEXED: False
+  ES_CASE_SEARCH_INDEX_MULTIPLEXED: False
+  ES_CASES_INDEX_MULTIPLEXED: False
+  ES_DOMAINS_INDEX_MULTIPLEXED: False
+  ES_FORMS_INDEX_MULTIPLEXED: False
+  ES_GROUPS_INDEX_MULTIPLEXED: False
+  ES_SMS_INDEX_MULTIPLEXED: False
+  ES_USERS_INDEX_MULTIPLEXED: False
   # Index Swap Settings  
-  ES_APPS_INDEX_SWAPPED: False
-  ES_CASE_SEARCH_INDEX_SWAPPED: False
-  ES_CASES_INDEX_SWAPPED: False
-  ES_DOMAINS_INDEX_SWAPPED: False
-  ES_FORMS_INDEX_SWAPPED: False
-  ES_GROUPS_INDEX_SWAPPED: False
-  ES_SMS_INDEX_SWAPPED: False
-  ES_USERS_INDEX_SWAPPED: False
+  ES_APPS_INDEX_SWAPPED: True
+  ES_CASE_SEARCH_INDEX_SWAPPED: True
+  ES_CASES_INDEX_SWAPPED: True
+  ES_DOMAINS_INDEX_SWAPPED: True
+  ES_FORMS_INDEX_SWAPPED: True
+  ES_GROUPS_INDEX_SWAPPED: True
+  ES_SMS_INDEX_SWAPPED: True
+  ES_USERS_INDEX_SWAPPED: True
   # Index Settings End
   IS_DIMAGI_ENVIRONMENT: True
   EULA_COMPLIANCE: True

environments/india/terraform.yml

@@ -309,7 +309,7 @@ servers:
 
 proxy_servers:
   - server_name: "proxy_a1-india"
-    server_instance_type: "t3a.medium"
+    server_instance_type: "t3a.large"
     network_tier: "app-private"
     az: "a"
     volume_size: 80
@@ -319,7 +319,7 @@ proxy_servers:
     os: jammy
 
   - server_name: "proxy_b1-india"
-    server_instance_type: "t3a.medium"
+    server_instance_type: "t3a.large"
     network_tier: "app-private"
     az: "b"
     volume_size: 80
@@ -330,7 +330,7 @@ proxy_servers:
 
 rds_instances:
   - identifier: "pgmain0-india"
-    instance_type: "db.t3.medium"
+    instance_type: "db.t4g.medium"
     storage: 500
     storage_type: gp3
     multi_az: true

environments/production/app-processes.yml

@@ -5,7 +5,9 @@ gunicorn_workers_static_factor: 0
 gunicorn_workers_factor: 4
 formplayer_memory: "31g"
 formplayer_g1heapregionsize: "16m"
-formplayer_command_args: '-javaagent:/home/cchq/dd-java-agent.jar -Dsrc.main.java.org.javarosa.enableOpenTracing=true -Ddd.profiling.enabled=true -XX:FlightRecorderOptions=stackdepth=256 -Ddd.service=formplayer -Ddd.env=production -Ddd.trace.sample.rate=0.1'
+formplayer_command_args: ''
+# Use the following to enable datadog tracing
+# -javaagent:/home/cchq/dd-java-agent.jar -Dsrc.main.java.org.javarosa.enableOpenTracing=true -Ddd.profiling.enabled=true -XX:FlightRecorderOptions=stackdepth=256 -Ddd.service=formplayer -Ddd.env=production -Ddd.trace.sample.rate=0.1'
 management_commands:
   celerybeat_a0:
     run_submission_reprocessing_queue:

environments/production/public.yml

@@ -174,7 +174,7 @@ localsettings:
   # Index Multiplexer Settings  
   ES_APPS_INDEX_MULTIPLEXED: True
   ES_CASE_SEARCH_INDEX_MULTIPLEXED: False # Index swapped
-  ES_CASES_INDEX_MULTIPLEXED: True
+  ES_CASES_INDEX_MULTIPLEXED: False # Index swapped
   ES_DOMAINS_INDEX_MULTIPLEXED: True
   ES_FORMS_INDEX_MULTIPLEXED: True
   ES_GROUPS_INDEX_MULTIPLEXED: True
@@ -183,9 +183,9 @@ localsettings:
   # Index Swap settings  
   ES_APPS_INDEX_SWAPPED: False
   ES_CASE_SEARCH_INDEX_SWAPPED: True
-  ES_CASES_INDEX_SWAPPED: False
+  ES_CASES_INDEX_SWAPPED: True
   ES_DOMAINS_INDEX_SWAPPED: False
-  ES_FORMS_INDEX_SWAPPED: False
+  ES_FORMS_INDEX_SWAPPED: True
   ES_GROUPS_INDEX_SWAPPED: False
   ES_SMS_INDEX_SWAPPED: False
   ES_USERS_INDEX_SWAPPED: False