feat: add consul cluster to the HashiStack

kral2 · Oct 13, 2021 · 922df19 · 922df19
1 parent dbea075
commit 922df19
Show file tree

Hide file tree

Showing 32 changed files with 465 additions and 94 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,19 +1,33 @@
-terraform.tfvars
-.terraform
+# Compiled files
 *.tfstate
 *.tfstate.backup
+*.tfstate.lock.info
 *.lock
 
+# Directories
+.terraform
+ignored
+
+# Ignored files
+*gitignore*.tf
+terraform.tfvars
+provision/compute_instance_image_source_ocid.auto.tfvars
+*pkrvars.hcl
+
+# Ignored Ansible files
+hosts.ini
+all
+
 **/ssh_authorized_keys.txt
 
 .DS_Store
 .vscode
 
 packer-manifest.json
 
-*pkrvars.hcl
 
-hosts.ini
-all
 
-ignored
+
+
+
+
diff --git a/.terraform-docs.yml b/.terraform-docs.yml
@@ -20,7 +20,7 @@ content: |-
   {{ .Footer }}
 
 output:
-  file: "../README.md"
+  file: "README.md"
   mode: inject
 
 output-values:

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -10,6 +10,20 @@ Given a version number MAJOR.MINOR.PATCH:
 - MINOR version when adding functionality in a backwards compatible manner,
 - PATCH version when making backwards compatible bug fixes.
 
+## [0.2.0] - 2021-10-13
+
+### Added
+
+- Consul Cluster : the compute instances that run the Nomad cluster now also run a 3-node Consul cluster.
+- Terraform provisioning update: a new VCN Security List for Consul
+- New playbook to restart Consul or Nomad services on running clusters
+
+### Changed
+
+- Nomad and Consul systemd service units are now in state enabled at the build stage.
+- Use [go-sockaddr](https://github.com/hashicorp/go-sockaddr) template on consul_config/server.hcl to determine the node IP address at launch time and not hard coded in the Packer build image.
+- hashistack-installer is now pinned to a tag version for more deterministic output
+
 ## [0.1.0] - 2021-10-11
 
 Initial release. Deploys the infrastructure required to run an HashiStack demo on OCI.

diff --git a/README.md b/README.md
@@ -18,66 +18,14 @@
 
 ## About
 
-This module build an HashiStack demo on OCI. It is a 3-step workflow:
+This project build an "HashiStack" demo on [OCI](https://docs.oracle.com/en-us/iaas/Content/home.htm). It is a 3-step workflow:
 
-1. BUILD, using Packer, Ansible, Bash scripts. The output is a Custom Image ready to deploy Nomad nodes on Linux/aarch
-2. PROVISION using Terraform Registry, Terraform Cloud, OCI. The output is an OCI Networking structure and security rules, together with 3 Computes Instances and a bastion host
-3. RUN : Nomad on a 3-nodes cluster, with some example jobs to come
+1. **BUILD:** using Packer, Ansible and Bash scripts. The output is a Custom Image, ready to deploy Consul+Nomad nodes on Linux/arm64 using [Ampere Altra CPU](https://amperecomputing.com/)
+2. **PROVISION:** using Terraform Registry, Terraform Cloud with OCI. The output is an OCI Networking structure and security rules, together with 3 Computes Instances and a bastion host
+3. **RUN:** with [Consul](https://www.consul.io/) and [Nomad](https://www.nomadproject.io/) on a 3-nodes cluster, with some example Nomad jobs and Consul content to come
 
-![diagram](https://github.com/kral2/hashistack_on_oci/blob/main/_files/3-Steps_workflow.png?raw=true&sanitize=true)
+![diagram](https://github.com/kral2/hashistack_on_oci/blob/main/_files/images/3-Steps_workflow.png?raw=true&sanitize=true)
 
-<!-- BEGIN_TF_DOCS -->
+Details about the provisioned infrastructure can be found in the ["Provision" folder](https://github.com/kral2/hashistack_on_oci/tree/main/provision) of this repo.
 
-## Requirements
-
-| Name | Version |
-|------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0.0 |
-| <a name="requirement_oci"></a> [oci](#requirement\_oci) | >=4.40.0 |
-## Modules
-
-| Name | Source | Version |
-|------|--------|---------|
-| <a name="module_instance_bastion"></a> [instance\_bastion](#module\_instance\_bastion) | oracle-terraform-modules/compute-instance/oci | 2.2.0 |
-| <a name="module_instance_hashistack"></a> [instance\_hashistack](#module\_instance\_hashistack) | oracle-terraform-modules/compute-instance/oci | 2.2.0 |
-| <a name="module_vcn_hashistack"></a> [vcn\_hashistack](#module\_vcn\_hashistack) | oracle-terraform-modules/vcn/oci | 3.0.0 |
-## Resources
-
-| Name | Type |
-|------|------|
-| [oci_core_security_list.nomad_servers](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/core_security_list) | resource |
-| [oci_core_subnet.vcn_hashistack_private](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/core_subnet) | resource |
-| [oci_core_subnet.vcn_hashistack_public](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/core_subnet) | resource |
-## Inputs
-
-| Name | Description | Type | Default | Required |
-|------|-------------|------|---------|:--------:|
-| <a name="input_block_storage_sizes_in_gbs"></a> [block\_storage\_sizes\_in\_gbs](#input\_block\_storage\_sizes\_in\_gbs) | Sizes of volumes to create and attach to each instance. | `list(string)` | `[]` | no |
-| <a name="input_compartment_id"></a> [compartment\_id](#input\_compartment\_id) | compartment ocid where to create all resources | `string` | n/a | yes |
-| <a name="input_defined_tags"></a> [defined\_tags](#input\_defined\_tags) | predefined and scoped to a namespace to tag the resources created using defined tags. | `map(string)` | `null` | no |
-| <a name="input_fingerprint"></a> [fingerprint](#input\_fingerprint) | n/a | `string` | `null` | no |
-| <a name="input_freeform_tags"></a> [freeform\_tags](#input\_freeform\_tags) | simple key-value pairs to tag the resources created using freeform tags. | `map(string)` | `null` | no |
-| <a name="input_instance_ad_number"></a> [instance\_ad\_number](#input\_instance\_ad\_number) | The availability domain number of the instance. If none is provided, it will start with AD-1 and continue in round-robin. | `number` | `1` | no |
-| <a name="input_instance_count"></a> [instance\_count](#input\_instance\_count) | Number of identical instances to launch from a single module. | `number` | `1` | no |
-| <a name="input_instance_display_name"></a> [instance\_display\_name](#input\_instance\_display\_name) | (Updatable) A user-friendly name for the instance. Does not have to be unique, and it's changeable. | `string` | `"instance-module"` | no |
-| <a name="input_instance_flex_memory_in_gbs"></a> [instance\_flex\_memory\_in\_gbs](#input\_instance\_flex\_memory\_in\_gbs) | (Updatable) The total amount of memory available to the instance, in gigabytes. | `number` | `6` | no |
-| <a name="input_instance_flex_ocpus"></a> [instance\_flex\_ocpus](#input\_instance\_flex\_ocpus) | (Updatable) The total number of OCPUs available to the instance. | `number` | `1` | no |
-| <a name="input_private_key"></a> [private\_key](#input\_private\_key) | n/a | `string` | `null` | no |
-| <a name="input_private_key_path"></a> [private\_key\_path](#input\_private\_key\_path) | n/a | `string` | `null` | no |
-| <a name="input_public_ip"></a> [public\_ip](#input\_public\_ip) | Whether to create a Public IP to attach to primary vnic and which lifetime. Valid values are NONE, RESERVED or EPHEMERAL. | `string` | `"RESERVED"` | no |
-| <a name="input_region"></a> [region](#input\_region) | n/a | `string` | `null` | no |
-| <a name="input_shape"></a> [shape](#input\_shape) | The shape of an instance. | `string` | `"VM.Standard.A1.Flex"` | no |
-| <a name="input_source_ocid"></a> [source\_ocid](#input\_source\_ocid) | The OCID of an image or a boot volume to use, depending on the value of source\_type. | `string` | n/a | yes |
-| <a name="input_source_type"></a> [source\_type](#input\_source\_type) | The source type for the instance. | `string` | `"image"` | no |
-| <a name="input_ssh_public_keys"></a> [ssh\_public\_keys](#input\_ssh\_public\_keys) | Public SSH keys to be included in the ~/.ssh/authorized\_keys file for the default user on the instance. To provide multiple keys, see docs/instance\_ssh\_keys.adoc. | `string` | `null` | no |
-| <a name="input_tenancy_ocid"></a> [tenancy\_ocid](#input\_tenancy\_ocid) | n/a | `string` | `null` | no |
-| <a name="input_user_data"></a> [user\_data](#input\_user\_data) | Provide your own base64-encoded data to be used by Cloud-Init to run custom scripts or provide custom Cloud-Init configuration. | `string` | `null` | no |
-| <a name="input_user_ocid"></a> [user\_ocid](#input\_user\_ocid) | n/a | `string` | `null` | no |
-## Outputs
-
-| Name | Description |
-|------|-------------|
-| <a name="output_instance_bastion"></a> [instance\_bastion](#output\_instance\_bastion) | ocid of created instances. |
-| <a name="output_instance_hashistack"></a> [instance\_hashistack](#output\_instance\_hashistack) | ocid of created instances. |
-
-<!-- END_TF_DOCS -->
+The whole architecure deployed by this project can be contained in the [Oracle Cloud Always Free tier](https://www.oracle.com/cloud/free/) allocation, which comes with Compute, Storage, the cloud networking stuff and a hefty 10TB Outbound Data Transfer per month. So it should not cost you a single penny to try :-D
diff --git a/_files/3-Steps_workflow.png b/_files/3-Steps_workflow.png
diff --git a/_files/consul_config/consul.hcl b/_files/consul_config/consul.hcl
@@ -0,0 +1,5 @@
+# Consul config file
+
+datacenter = "fra"
+data_dir = "/opt/consul"
+retry_join = ["172.16.1.11", "172.16.1.12", "172.16.1.13"]
diff --git a/_files/consul_config/consul.service b/_files/consul_config/consul.service
@@ -0,0 +1,20 @@
+[Unit]
+Description="HashiCorp Consul - A service mesh solution"
+Documentation=https://www.consul.io/
+Requires=network-online.target
+After=network-online.target
+ConditionFileNotEmpty=/etc/consul.d/consul.hcl
+
+[Service]
+Type=notify
+User=consul
+Group=consul
+ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/
+ExecReload=/bin/kill --signal HUP $MAINPID
+KillMode=process
+KillSignal=SIGTERM
+Restart=on-failure
+LimitNOFILE=65536
+
+[Install]
+WantedBy=multi-user.target
diff --git a/_files/consul_config/server.hcl b/_files/consul_config/server.hcl
@@ -0,0 +1,9 @@
+# Consul config file
+
+server = true
+bootstrap_expect = 3
+bind_addr = "{{ GetPrivateInterfaces | include \"network\" \"172.16.1.0/24\" | attr \"address\" }}"
+client_addr = "{{ GetPrivateInterfaces | include \"network\" \"172.16.1.0/24\" | attr \"address\" }}"
+ui_config = {
+  enabled = true
+}
diff --git a/_files/download_hashistack.sh b/_files/download_hashistack.sh
@@ -7,6 +7,6 @@
 # If/when https://iac.sh evovles to support the installation of HashiCorp products other than Terraform and Packer,
 # we may switch to it instead of my current installer.
 
-curl -LO https://raw.github.com/kral2/hashistack-installer/main/hashistack-install.sh
+curl -LO https://raw.githubusercontent.com/kral2/hashistack-installer/v0.3.1/hashistack-install.sh
 chmod +x hashistack-install.sh
 ./hashistack-install.sh consul vault nomad
diff --git a/_files/images/3-Steps_workflow.png b/_files/images/3-Steps_workflow.png
diff --git a/_files/images/deployed_infrastructure.png b/_files/images/deployed_infrastructure.png
diff --git a/_files/nomad_config/nomad.hcl b/_files/nomad_config/nomad.hcl
@@ -1,4 +1,5 @@
 # Nomad config file
 
+region = "oci.eu-frankfurt-1"
 datacenter = "fra"
-data_dir = "/opt/nomad"
+data_dir = "/opt/nomad"
diff --git a/_files/nomad_config/server.hcl b/_files/nomad_config/server.hcl
@@ -7,4 +7,4 @@ server {
         retry_join = [ "172.16.1.11", "172.16.1.12", "172.16.1.13" ]
         retry_max = 1
         retry_interval = "15s"  }
-}
+}
diff --git a/build/ansible/ansible.cfg b/build/ansible/ansible.cfg
@@ -0,0 +1,3 @@
+[defaults]
+inventory = hosts.ini
+host_key_checking = False
diff --git a/build/ansible/consul_1_prereq.yaml b/build/ansible/consul_1_prereq.yaml
@@ -0,0 +1,26 @@
+---
+
+- name: 1. Prepare the Operating System | consul environment
+  hosts: all
+  gather_facts: True
+  become: yes
+  become_user: root
+
+  tasks:
+    - name: Ensure base packages are present
+      ansible.builtin.yum:
+        name: "{{ packages }}"
+        state: present
+      vars:
+        packages:
+        - git
+        - tree
+      tags: global
+
+    - name: Ensure service user is present
+      ansible.builtin.user:
+        name: consul
+        comment: 'consul - Ansibled'
+        state: present
+      tags: global, user
+...
diff --git a/build/ansible/consul_2_install.yaml b/build/ansible/consul_2_install.yaml
@@ -0,0 +1,97 @@
+---
+
+- name: 2. Install the software | Consul
+  hosts: all
+  gather_facts: True
+  become: yes
+  become_user: root
+
+  tasks:
+    - name: Ensure Consul directories are present
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+        mode: '0755'
+        owner: consul
+        group: consul
+      with_items:
+      - /opt/consul
+      - /etc/consul.d
+      tags: global
+
+    - name: Ensure Consul service unit file is present
+      ansible.builtin.copy:
+        src: ../../_files/consul_config/consul.service
+        dest: /etc/systemd/system/consul.service
+        owner: root
+        group: root
+        mode: '0664'
+    - name: Ensure Consul service is enabled
+      ansible.builtin.shell: systemctl enable consul
+
+    - name: Ensure Consul Configuration files are present
+      ansible.builtin.copy:
+        src: "{{ item }}"
+        dest: /etc/consul.d/
+        mode: '744'
+      with_items:
+      - ../../_files/consul_config/consul.hcl
+      - ../../_files/consul_config/server.hcl
+
+    - name: Configure firewalld | Consul RPC (Client)
+      firewalld:
+        port: 8300/tcp
+        permanent: yes
+        immediate: yes
+        state: enabled
+      become: yes
+      become_user: root
+    - name: Configure firewalld | Consul GOSSIP /tcp (Serf LAN)
+      firewalld:
+        port: 8301/tcp
+        permanent: yes
+        immediate: yes
+        state: enabled
+      become: yes
+      become_user: root
+    - name: Configure firewalld | Consul GOSSIP /udp (Serf LAN)
+      firewalld:
+        port: 8301/udp
+        permanent: yes
+        immediate: yes
+        state: enabled
+      become: yes
+      become_user: root
+    - name: Configure firewalld | Consul GOSSIP /tcp (Serf WAN)
+      firewalld:
+        port: 8302/tcp
+        permanent: yes
+        immediate: yes
+        state: enabled
+      become: yes
+      become_user: root
+    - name: Configure firewalld | Consul GOSSIP /udp (Serf WAN)
+      firewalld:
+        port: 8302/udp
+        permanent: yes
+        immediate: yes
+        state: enabled
+      become: yes
+      become_user: root
+    - name: Configure firewalld | Consul HTTP API
+      firewalld:
+        port: 8500/tcp
+        permanent: yes
+        immediate: yes
+        state: enabled
+      become: yes
+      become_user: root
+    - name: Configure firewalld | Consul DNS Interface
+      firewalld:
+        port: 8600/tcp
+        permanent: yes
+        immediate: yes
+        state: enabled
+      become: yes
+      become_user: root
+...
diff --git a/build/ansible/consul_3_restart_service.yaml b/build/ansible/consul_3_restart_service.yaml
@@ -0,0 +1,13 @@
+---
+
+- name: 3. Start service | nomad
+  hosts: all
+  gather_facts: True
+  become: yes
+  become_user: root
+
+  tasks:
+    - name: Execute collect_sar_info.sh on remote targets
+      ansible.builtin.shell: systemctl restart consul
+
+...
diff --git a/build/ansible/main.yaml b/build/ansible/main.yaml
@@ -2,9 +2,9 @@
 
 # This playbook install a HashiCorp nomad node
 
-- name: Install nomad Prereqs
-  import_playbook: nomad_1_prereq.yaml
+- name: Run Consul playbooks
+  import_playbook: main_consul.yaml
 
-- name: Install nomad
-  import_playbook: nomad_2_install.yaml
+- name: Run Nomad Prereqs
+  import_playbook: main_nomad.yaml
 ...
diff --git a/build/ansible/main_consul.yaml b/build/ansible/main_consul.yaml
@@ -0,0 +1,10 @@
+---
+
+# This playbook install a HashiCorp consul node
+
+- name: Install consul Prereqs
+  import_playbook: consul_1_prereq.yaml
+
+- name: Install consul
+  import_playbook: consul_2_install.yaml
+...
diff --git a/build/ansible/main_nomad.yaml b/build/ansible/main_nomad.yaml
@@ -0,0 +1,10 @@
+---
+
+# This playbook install a HashiCorp nomad node
+
+- name: Install nomad Prereqs
+  import_playbook: nomad_1_prereq.yaml
+
+- name: Install nomad
+  import_playbook: nomad_2_install.yaml
+...