Skip to content

Latest commit

 

History

History
88 lines (84 loc) · 5.4 KB

README.md

File metadata and controls

88 lines (84 loc) · 5.4 KB
Raw

Hylastix Candidate Test

Simple Architecture

graph  LR
  client([client])-.load balancer .->ingress[Traefik];
  ingress;
  subgraph Nomad
  ingress;
  ingress-->pod1[Keyclaoak];
  pod1-->pod3[Postgres];
  ingress-->pod2[Grafana];
  pod2-->pod1[Keycloak];
  end
  classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000;
  classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff;
  classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5;
  class ingress,service,pod1,pod2 k8s;
  class client plain;
  class cluster cluster;
Loading

Improvments

  • Variable naming to better describe what variable is used for.
  • Using valid domain instead of dynamic dns service.
  • Expanding configuration templates for Consul and Nomad.
  • Improving github actions to have clear separation of jobs, instead having only one job.
  • Expanding terraform code to deploy full cluster instead single node cluster.
  • Implement Azure availability zones with terraform.
  • Enable persistant sotorage for Nomad cluster with managed disk or with CSI.
  • Move Keycloak ansible task into separate role.
  • Dynamically add/remove nsg rule for github actions runner.

Pre-requisites

  • ansible
  • Terraform
  • Azure Service Principal
  • Azure Storage Account for storing the state file

Ansible Role(s) Variables

Name Value Description
consul_server Default = false Set VM to Consul server
data_center_name Default = "dc1" Data Center Name
enable_local_script_checks Default = false Enable script checks in Consul
bootstrap_expect Default = 1 Consul/Nomad how many servers to bootstrap
docker_sock_enabled Default = false Add docker socket readonly volume map in nomad configuration
docker_privileged_mode Default = false Enable docker to run in privileged mode
nomad_ui_label_backgroung_color Default = "purple" Set label background color in Nomad UI
nomad_ui_label_text_color Default = "white" Set label color in Nomad UI
psql_password Default = "changeme" Psql Server password
keycloak_admin_user Default = "admin" Set Keycloak admin user
keycloak_admin_password Default = "changeme" Set Keycloak admin user password
kc_grafana_client_roles Default = ['admin','editor','viewer'] Keycloak client roles for Grafana
kc_realm_name Default = "grafana" Keycloak realm name
kc_client_name Default = "grafana-oauth" Keycloak client name
kc_client_secret Default = "dummy string" Keycloak client secret
kc_url Default = "keycloak.hyl" Keycloak URL
grafana_url Default = "grafana.hyl" Grafana URL
grafana_oauth_tls_skip_verify Default = true Grafana skip cert validation for oauth login
nomad_url Default = "nomad.hyl" Nomad URL
consul_url Default = "consul.hyl" Consul URL
kc_user_grafana_realm Default = kc_user_grafana_realm:
     -name: user1
       password: changeme
       role: viewer		 Keycloak client dummy users with mapped client roles
duck_dns_token Default = "dummy string" Token for Duckdns to update ip of dns records
letsencrypt_email Default = "dummy string" Email to allow traefik to generate let's encrypt certificates
top_domain Default = "ip-ddns.com" Cloudns domain
Note: If you want to override default values, please change values in the inventory file.

Example inventory file

consul_server:
  hosts:
    hylvm:
      ansible_host: xxx.xxx.xxx.xxx
      ansible_user: adminuser
      data_center_name: dev
      consul_server: true

Notes, Tips and Troubleshooting

  • export ANSIBLE_HOST_KEY_CHECKING=False
  • consul_server_ips Ansible variable will look for server group named consul_server in inventory file to fill proper IPs for retry_join stanza in Consul configuration
  • Environment is ephemeral
  • Environment depends on cloudns (Dynamic DNS). Cron job is checking for IP change every 3 minutes.
  • If 404 code emerges use https when typing url(s)
  • Certificates are dynamically created by traefik. If certificate is not valid that means that default traefik certificate is used. Check traefik logs in nomad UI to resolve the problem.
    • ACME protocol rate limit: ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [keycloak.hyl.ip-ddns.com]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many certificates (50) already issued for \"ip-ddns.com\" in the last 168h0m0s, retry after 2024-11-04 13:52:22 UTC
    • DNS can't reslove domain record ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [hyl-nomad.duckdns.org]: error: one or more domains had a problem:\n[hyl-nomad.duckdns.org] acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: SERVFAIL looking up A for hyl-nomad.duckdns.org - the domain's nameservers may be malfunctioning; DNS problem: SERVFAIL looking up AAAA for hyl-nomad.duckdns.org - the domain's nameservers may be malfunctioning\n"
    • Because of problems mentioned above grafana_oauth_tls_skip_verify is set to true by default.