diff --git a/.gitignore b/.gitignore index 4e1d858..71bee1d 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,5 @@ data/logs data/letsencrypt data/maps data/certbot +.vscode +test diff --git a/README.md b/README.md index a5bdbf1..86c459a 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,13 @@ If you actually need to create the stuff e.g. the mbtiles or the graph you bette ## Concept -This repository contains a bunch of service „modules“. Each "module" has a specified job and contains a README. In order to host multiple apps we also introduce the concept of a "city". Each city has a config in `./config` directory and can have a different set of "modules" enabled. For example: city 'Germany-Hamburg' can have the modules `tileserver`, `photon` and `otp` enabled. Another city 'Bolivia-Cochabamba' can have the modules `tileserver` and `otp` enabled. You see here is `photon` missing and that is completely ok. Cities are strictly taken apart from each other. They only share the same nginx instance, the same project name and the same network. +This repository contains a bunch of service „modules“. Each "module" has "services" and contains a README. Each service has a specified job and deals as a module component. + +In order to be able to host multiple instances of a modules we also introduce the concept of a "city". Each city has a config in `./config` directory and can have a different set of "modules" enabled. For example: city 'Germany-Hamburg' can have the modules `tileserver`, `photon` and `otp` enabled. Another city 'Bolivia-Cochabamba' can have the modules `tileserver` and `otp` enabled. You see here is `photon` missing and that is completely ok. Cities are strictly taken apart from each other. + +At last we introduce "chiefs" which are all services specified in the `docker-compose.yml` at project root. They don't share the concept of modules and cities. As they don't share this concept you cannot add/remove them in the classical way with `add_module`/`remove_module`. Instead they will be ruled by `server` which creates/removes/starts/stops them automatically depending on what you do with the modules. + +[Extending Trufi Multi-Instance Server - add/remove chiefs](./docs/extend.md#chiefs) ## Config @@ -33,13 +39,13 @@ Create a new one based on the already existing config files to get an idea of th ### Global -For other but important parameters set using the `init` script which apply globally to all cities we use a global configuration stored in file `./data/instance.conf`. +For other but important parameters which apply globally to all cities, we use a global configuration stored in file `./data/instance.conf`. | Variable | Example | Description | | -------- | -------------- | ------------------------------------------------------------ | -| env | development | Sets the execution environment. The value `development` or `production` are only valid. | -| curmode | virtual domain | The nginx domain structure to use. Accepted values are `virtual domain` (all cities run under the same domain) and `real domains` (each city has its own domain) | -| intraweb | yes | Toggles the Intraweb feature on or off. Valid values are `yes` or `no`. This feature will be turned on automatically if you told the `init` script about your plan to run this structure in production environment. And automatically turned off when told about running in a development environment. If you change the `env` variable without consulting `init` script then the value of the variable `intraweb` will not get changed.

The script `add_module` will read this setting to know wherever to create the configuration necessary for Intraweb. | +| ssl | yes | Tells `add_module` to configure SSL for the particular city. Valid values are `yes` or `no`. | +| curmode | virtual domain | The nginx domain structure to use. Accepted values are `virtual domain` (all cities run under the same domain) and `real domains` (each city has its own domain). Tells `add_module` how to configure nginx. | +| intraweb | yes | Toggles the Intraweb feature on or off. Valid values are `yes` or `no`.

The script `add_module` will read this setting to know wherever to create the configuration necessary for Intraweb.
[Documentation of Intraweb](./docs/intraweb.md) | @@ -47,32 +53,7 @@ For other but important parameters set using the `init` script which apply globa You can find all in the [modules](./modules) folder. Each module has a README file with more detailed info. -- **[otp](./modules/otp)** - This is [OpenTripPlanner](https://opentripplanner.org) used to calculate the best route for the user of the app. *This service is mandatory for the app to work.* -- **[photon](./modules/photon)** - This is [Photon by Komoot](https://photon.komoot.io) used to provide online search results inside the app when the user searches for a POI to navigate from or to using public transportation. *This service is mandatory for the search feature of the app to work.* -- **[static_maps](./modules/static_maps)** - Use this service to serve pre-generated background map tiles. *This use of the service is optional but we recommend it if you have a server which is less in resources.* -- **[tileserver](./modules/tileserver)** - Use this service to serve the data needed to display the background map shown in the app. This does not include the styling (e.g. a highways are yellow lines and the water blue). The styling is done on the client side. This allows you/us to make modifications to the stylings without the need to rerender all pngs of the background map for your city. It generates the png background map tiles on the fly for clients which do not support dynamic map tiles. *This use of the service is optional and cause much CPU usage when it needs to generate background maps on the fly (this is a wrong usage of this service). Our app currently does not support client side rendering of background maps so we only recommend using this service on a server with much CPU resources.* - -Concerning background map tiles: Decide wherever you want to use the module *static_maps* or *tileserver*. Using both in **one** city is useless. - -## Intraweb - -In a [production environment](#production_environmemt) this backend wires up a complete Intraweb for sysadmins to debug/check (misbehaving) modules related to a city. A sysadmin can enjoy the beauty of the webportals of some modules without the need to expose them to the public (our provided nginx configurations for each module limit access extremely). The Intraweb is accessible at `localhost:8090` on your server. So do the well known [SSH port forwarding](https://phoenixnap.com/kb/ssh-port-forwarding) magic to access it through your webbrowser on your own machine. - -```sh -ssh @ -L 8090:127.0.0.1:8090 -``` - -e.g. - -```sh -ssh foo@example.com -L 8090:127.0.0.1:8090 -``` - -And we can type in a url following the scheme `http://_.localhost:8090/` e.g. to reach the module `tileserver` of city `Germany-Hamburg` type `http://tileserver_Germany-Hamburg.localhost:8090/` +In order to add more modules compatible to [trufi-server-modules](https://github.com/trufi-association/trufi-server-modules) like [tsm-locaco](https://github.com/trufi-association/tsm-locaco) you just have to do a `git clone` inside the `module` folder, to execute `modifyComposes.py` and to work with it as you did with the other modules pre-installed. ## Setup & Maintenance @@ -83,18 +64,22 @@ Definitely you used our tools in the [Trufi Server Resources](https://github.com To copy over the files to their appropriate location here you just need to do the following: In your very own copy of [Trufi Server Resources](https://github.com/trufi-server-resources) go to its `data` directory and copy the content of the `` which holds your data. Paste them into the `modules` folder of this one and accept the merge with already existing data. Or to put it into the words/world of sysadmins just execute ```bash -cp -a ./trufi-server-resources/data//* ./trufi-server/modules --verbose +cp -a ./trufi-server-resources/data//* ./trufi-server-cities/modules --verbose ``` +Then rename all `data` folders inside the modules to `data_Country-City` e.g. `data_Germany-Hamburg`. + **TO DO:** Introduce `autosetup` to ease setting up a new city or update an existing one (script needs to be developed still) -### Production environment +### Encrypting connections to this backend -We hide all (optional) services behind a nginx proxy which also handles the encryption for them. To make encrypted connections to that nginx proxy possible we need to issue a HTTPS certificate. Luckily there is *Let’s Encrypt* issuing them to anyone without the need to pay. Initialization is to be done using `init.prod` and affects all cities but it does not do the certification for you. Instead SSL/TLS support relies on the certificate management infrastructure of the docker host (your linux system). There are multiple ways connecting the dots. +We hide all (optional) services behind a nginx proxy which **can** handle the encryption for them. To make encrypted connections to that nginx proxy possible we need to issue a HTTPS certificate. Luckily there is *Let’s Encrypt* issuing them to anyone without the need to pay. This backend relies on the SSL/TLS certificate management infrastructure of the docker host (your linux system). There are multiple ways connecting the dots. -Our docker `nginx` by default listens on the host port `8290` for HTTP and on `8300` for HTTPS requests. +Our docker `cief-nginx` by default listens on the host port `8290` for HTTP and on `8300` for HTTPS requests. -#### Docker `nginx` has its own HTTPS certificate store and runs on a different port +#### Docker `chief-nginx` has its own HTTPS certificate store and runs on a different port + +This requires the `ssl` variable in `./data/instance.conf` set to `yes` before any execution of `add_module`. If you accidentally run `add_module` before doing that then use `remove_module` to remove the configuration files as `remove_module` is the opposite of `add_module`. The docker container `nginx` in our provided configuration expects to find the necessary files needed to answer https requests in the host path `./data/certbot/conf` which is in its schematics equal to the well known [/etc/letsencrypt](https://eff-certbot.readthedocs.io/en/stable/using.html#where-certs). Of course this behaviour can be changed that the nginx docker container sees `/etc/letsencrypt` of your host directly although not recommended. This just adds one more security risk to manage. If you care about security you just put the files in `./data/certbot/conf` you need for running this docker infrastructure. To put it in other words: If you have the following domains `example.com` , `example.uk` and `example.org` . The first domain `example.com` is used by your web server running on the host linux system and the others two are used by the `nginx` running in this docker infrastructure then just put the necessary files for HTTPS on `example.org` and `example.com` into `./data/certbot/conf` . This way you have `/etc/letsencrypt` on your host for the HTTPS server running on your host system. The other one `./data/certbot/conf` for the HTTPS server running for this infrastructure. @@ -102,7 +87,9 @@ In that case you need to run `openssl dhparam -out ./data/nginx/inc/dhparam.pem #### Central HTTPS server on your host system -This is useful if you are already providing other services in need of HTTPS to your customers e.g. you are already hosting a website. Now you want to have your server provide different services for different cities. Also this is your only option if your server is behind a firewall just letting port 80 and 443 pass through. Go setup nginx on your host which then does all the HTTPS stuff and takes care to redirect to the appropriate sub webservers based on specified parameters you have defined. This allows you to use this structure in development mode without any HTTPS configuration. Is that is the case you surely don’t want to miss the [Intraweb](#intraweb) feature. By default this feature will be turned off by the `init` script when telling it that you run this in a development environment. To enable it you need to change the value of the `intraweb` variable manually in the [global config](#global). +This requires the `ssl` variable in `./data/instance.conf` set to `no` before any execution of `add_module`. If you accidentally run `add_module` before doing that then use `remove_module` to remove the configuration files as `remove_module` is the opposite of `add_module`. + +This is useful if you are already providing other services in need of HTTPS to your customers e.g. you are already hosting a website. Now you want to have your server provide different services for different cities. Also this is your only option if your server is behind a firewall just letting port 80 and 443 pass through. Go setup nginx on your host which then does all the HTTPS stuff and takes care to redirect to the appropriate sub webservers based on specified parameters you have defined. This allows you to use this structure without any HTTPS configuration ( `ssl="no"` ) because encryption is handled by your HTTPS server on your host. #### Get HTTPS certificate (independent from your infrastructure) @@ -116,102 +103,8 @@ We created a highly flexible script to ease HTTPS certificate creation and it ac **Example:** `certify Germany-Hamburg /srv/trufi/nginx/www` -### Commands - -After executing one of the init scripts you can work with the modules. See [how to run scripts on linux](https://www.cyberciti.biz/faq/howto-run-a-script-in-linux/). To make things easier source the `workon` script which changes your current Bash session so you don't have to fill in `` each time. Also it adds [command aliases](https://www.tutorialspoint.com/unix_commands/alias.htm) to each script name to make it smoother to type. Source it by executing +[Read more about certify](./docs/commands/certify.md) -```bash -. ./workon -``` - -e.g. - -```bash -. ./workon "Bolivia-Cochabamba" -``` - -#### Module management - -- **To add an module to the run configuration (does not start it automatically)** - - This command accepts a list of modules to add like `./add_module "Bolivia-Cochabamba" otp tileserver` or `add otp tileserver` - - - Command: `add_module []` - - Example: `add_module "Bolivia-Cochabamba" otp` - - Example (using `workon` script): `add otp` - -- **To remove an module from the run configuration (removes container but will not remove any files)** - - This command accepts a list of modules to remove like `./remove_module "Bolivia-Cochabamba" otp tileserver` or `remove otp tileserver` - - - Command: `remove_module []` - - Example: `remove_module "Bolivia-Cochabamba" otp` - - Example (using `workon` script): `remove otp` - -- **To (re)start an module (just use when the module hangs or other unusual things happened)** - - *This is deprecated and will be removed soon!* - - - Command: `restart_module ` - - Example: `restart_module "Bolivia-Cochabamba" otp` - - Example (using `workon` script): `restart_module otp` - -After adding or removing a module we should advertise the change to the web server nginx. We do so by executing `./server nginx reload` which causes nginx to reload its configuration without restarting - -#### Server management (all active cities and their enabled modules + web server) - -```sh -./server [] -``` - -The `server` script behaves differently based on how it is run. If you use it in a Bash modified by the `workon` script then the action is relative to the current city. Of course you don't have to provide the argument `` anymore as in the previous section. This is the 'city' scope. If you use it in the standard Bash and you don't provide the argument `` then the action will be applied to all added modules in all cities. This is the 'global' scope. - -Some examples: - -```sh -. ./workon Germany-Hamburg -server up # will bring all added modules in city 'Germany-Hamburg' up (scope 'city') -``` - -```sh -./server "Germany-Hamburg" up # will bring all added modules in city 'Germany-Hamburg' up (scope 'city') -``` - -```sh -./server up # will bring all added modules in all cities up (scope 'global') -``` - -- **To start the server for the first time ((re)creating docker containers etc.)** - - Command: `server run` or `server up` -- **To start the server** - - Command: `server start` -- **To (re)start the server** - - Command: `server restart` -- **To stop the server** - - Command: `server stop` -- **To perform actions on the nginx (web server) only** - - Command: `server nginx ` - - Reload nginx after its configuration has changed: `server nginx reload` -- **To view a list of running modules** - - Command: `server ls` (filtered Trufi optimised result for `docker container ps` command) - - Command: `server ps` (native docker-compose command for every single module) -- **To refresh services after updates to their `yml` files** - - Command: `server refresh` (city scope) - - Command: `server refresh` (global scope) - -- **(production only) View logs for a module from current day** - - Command: `viewlog ` - - Example: `viewlog "Bolivia-Cochabamba" otp` - - Example (using `workon` script): `viewlog "Bolivia-Cochabamba" otp` -- **(production only) View logs for a module with custom commands to the underlying `journalctl` command** - - Command: `viewlog []` - -**Not all available commands are listed here e.g. most commands for `server` are documented in its source code or indirectly by `docker-compose`** - -#### After upgrade - -When we changed something on the repo and you pulled the new changes do the following: +### Commands -```bash -./server refresh -``` +For a list of available commands see the [command documentation](./docs/commands/README.md) diff --git a/add_module b/add_module index f035e4d..9c17f59 100755 --- a/add_module +++ b/add_module @@ -1,5 +1,7 @@ #!/bin/bash +projectname=`basename "$PWD"` + source ./lib/colorful source ./lib/env @@ -17,12 +19,15 @@ addModule() { if ! [ -f "$ymlFileInactive" ] && ! [ -f "$ymlFile" ]; then orangeecho " building & adding in directory '$moduleDir' ) ..." export city_normalize="${city,,}" - (export `cat ./$cityfile | xargs` + (export `cat ./$cityfile | xargs`; export projectname="$projectname" envsubst < $moduleDir/docker-compose.yml > "$ymlFile" ) if ! [ -f "$ymlFile" ]; then redecho " Creating & adding operation failed! Do I have write access to '$moduleDir'?">&2 exit 1 fi + if [ -d "$moduleDir/data_template" ]; then + cp -R "$moduleDir/data_template" "$moduleDir/data_$city" + fi elif [ -f "$ymlFileInactive" ]; then orangeecho " adding ..." mv "$ymlFileInactive" $ymlFile --verbose @@ -44,28 +49,16 @@ addModule() { nginxcityconf="$nginxcityfolder.conf" # e.g. './data/nginx/interweb/Germany-Hamburg.conf' # This file to be created will contain the altered nginx reverse proxy configuration for the module '$modulename' e.g. 'tileserver' nginxcitymoduleconf="$nginxcityfolder/$modulename.conf" # e.g. './data/nginx/interweb/Germany-Hamburg/tileserver.conf' - # This file should better exist for the safety of the backend and to ensure that traffic redirection to that module works + # This file ensures that traffic redirection to that module works modulenginxconf="$moduleDir/nginx.conf" # e.g. './modules/tileserver/nginx.conf' # if a nginx reverse proxy configuration exists for this module if [ -f "$modulenginxconf" ]; then # read it into memory (to be able to alter it later) modulenginxconf_content=`cat "$modulenginxconf"` else - # if no module nginx reverse configuration exists then we need to ask the user to generate a probably unsecure one - redecho " Error: there is no nginx configuration for module '$modulename' available!" - echo "I can create a default reverse proxy configuration for you but then interweb users will be able to access all resources the module '$modulename' for city '$city' exposes through its HTTP endpoint. This can be a security thread in some circumstances as it can make functions publicly accessible you normally do not want to be. There is also the eventually that this just does not work" - echo "Create default reverse proxy (full expose of the module for city to the public) [y|n]" - read $decision - if [ "$decision" = "y" ]; then - orangeecho " creating default proxy configuration for module '$modulename' of city '$city' (not recommended) ..." - modulenginxconf_content="location / { - proxy_pass http://$modulename/ ; -}" - blueecho " make sure it works as it cannot be guaranteed" - else - redecho "Aborting as a reverse proxy configuration for module '$modulename' of city '$city' is necessary! Fix this and run this script again" - exit 1 - fi + # if no module nginx reverse configuration exists then abbort + redecho " Error: there is no nginx configuration ('nginx.conf') for module '$modulename' available! Fix this and run this script again" + exit 1 fi # If that folder does not exist @@ -81,25 +74,31 @@ addModule() { # - to include all *.conf files in '$nginxcityfolder' # - and to just be applied for the domain assigned to '$city' sed "s/# modules/# real domain include location blocks for city '$city'\ninclude \/etc\/nginx\/interweb\/$city\/\*.conf\;/" "$nginxcityconf_template_interweb" | sed "s/example.org/$domain/g" > "$nginxcityconf" - else # or if in virtual domain mode then just create the file '$nginxcityconf' containing the include statement to include all *.conf files in '$nginxcityfolder' + elif ! [ -f "$nginxcityconf" ]; then # or if in virtual domain mode then just create the file '$nginxcityconf' containing the include statement to include all *.conf files in '$nginxcityfolder' echo "include /etc/nginx/interweb/$city/*.conf;" > "$nginxcityconf" + else + greenecho " already done. No need to do it again :)" fi - + # 4. Generating nginx module configuration orangeecho " copying & altering nginx module configuration for module '$modulename' to nginx server configuration for city '$city' (overwriting if already existing) ..." - # search for pattern 'http://otp:' or 'http://otp;' or 'http://otp/' - # and replace it with 'http://otp-$city:' or 'http://otp-$city;' or 'http://otp-$city/' - # e.g. 'http://otp-germany-hamburg:' or 'http://otp-germany-hamburg;' or 'http://otp-germany-hamburg/' (lower case because of '${city,,}') - modulenginxconf_content=`echo "import re + if ! [ -f "$nginxcitymoduleconf" ]; then + # search for pattern 'http://otp:' or 'http://otp;' or 'http://otp/' + # and replace it with 'http://otp-$city:' or 'http://otp-$city;' or 'http://otp-$city/' + # e.g. 'http://otp-germany-hamburg:' or 'http://otp-germany-hamburg;' or 'http://otp-germany-hamburg/' (lower case because of '${city,,}') + modulenginxconf_content=`echo "import re txt = \"\"\"$modulenginxconf_content\"\"\" -print(re.sub(\"http\:/\/(.*?)(:|;|\/)\",\"http://\\\\\\1-${city,,}\\\\\\2\", txt, re.S)) +print(re.sub(\"http\:/\/(.*?)(:|;|\/)\",\"http://${modulename}-\\\\\\1-${city,,}\\\\\\2\", txt, re.S)) " | python3` - # if in virtual domains mode - if [ "$curmode" = "$MODE_VIRTUALDOMAINS" ]; then # do the following additional modifications to all location blocks to - # change their urls e.g. from '/tileserver' to '/Germany-Hamburg/tileserver' (no lower case here) - modulenginxconf_content=`echo "$modulenginxconf_content" | sed -E "s/location \/(.+?)/location \/$city\/\1/g" | sed -E "s/return (.+?) \/(.*?)/return \1 \/$city\/\2/g"` + # if in virtual domains mode + if [ "$curmode" = "$MODE_VIRTUALDOMAINS" ]; then # do the following additional modifications to all location blocks to + # change their urls e.g. from '/tileserver' to '/Germany-Hamburg/tileserver' (no lower case here) + modulenginxconf_content=`echo "$modulenginxconf_content" | sed -E "s/location \/(.+?)/location \/$city\/\1/g" | sed -E "s/return (.+?) \/(.*?)/return \1 \/$city\/\2/g"` + fi + echo "$modulenginxconf_content" > "$nginxcitymoduleconf" + else + greenecho " already generated. No need to do it again :)" fi - echo "$modulenginxconf_content" > "$nginxcitymoduleconf" # 5. Logging orangeecho " creating logging structure ..." @@ -116,34 +115,47 @@ print(re.sub(\"http\:/\/(.*?)(:|;|\/)\",\"http://\\\\\\1-${city,,}\\\\\\2\", txt nginxcityfolder="./data/nginx/intraweb/$city" nginxcityconf="$nginxcityfolder.conf" - nginxcitymoduleconf="$nginxcityfolder/$modulename.conf" - module_container_name="${modulename}-${city,,}" # mark #001 + orangeecho " creating nginx server configuration folder for city '$city' ..." if ! [ -d "$nginxcityfolder" ]; then - orangeecho " creating nginx server configuration folder for city '$city' ..." mkdir "$nginxcityfolder" fi + orangeecho " completing nginx configuration for city '$city' ..." if ! [ -f "$nginxcityconf" ]; then - orangeecho " completing nginx configuration for city '$city' ..." - echo "include /etc/nginx/intraweb/$city/*.conf ;" > "$nginxcityconf" + echo "include /etc/nginx/intraweb/$city/*/*.conf ;" > "$nginxcityconf" + else + greenecho " not necessary as already completed :)" fi - if ! [ -f "$nginxcitymoduleconf" ]; then - orangeecho " adding intraweb server for module '$modulename' in city '$city' ..." - # take content from template in file '$nginxcityconf_template_intraweb' and modify it to - # - be just applied to a specified autogenerated intraweb domain - # - and to replace all occurrence of 'modulename-city' with the name of the container e.g. to 'tileserver-germany-hamburg' (because of lower case convertion code on mark #001 - sed -E "s/localhost/$module_container_name.localhost/" "$nginxcityconf_template_intraweb" | sed "s/modulename-city/$module_container_name/" > "$nginxcitymoduleconf" - fi + orangeecho " adding intraweb server for module '$modulename' in city '$city'" + cd modules/$modulename + allServices=`sudo docker-compose -p "$projectname" -f "${city}.yml" ps --services` + allServices=( $allServices ) + cd ../../ + # iterate through names of all services of that city in the module + for servicename in "${allServices[@]}"; do + orangeecho " - $servicename ..." + nginxcitymoduleconf="$nginxcityfolder/$servicename.conf" + if ! [ -f "$nginxcitymoduleconf" ]; then + # take content from template in file '$nginxcityconf_template_intraweb' and modify it to + # - be just applied to a specified autogenerated intraweb domain + # - and to replace all occurrence of 'modulename-city' with the name of the container e.g. to 'tileserver-tileserver-germany-hamburg' + # Syntax: -- + sed -E "s/localhost/$servicename.localhost/" "$nginxcityconf_template_intraweb" | sed "s/modulename-city/$servicename/" > "$nginxcitymoduleconf" + else + greenecho " not necessary as already existing :)" + fi + done fi + + source "lib/plugin" + invokeAllPluginsOf "add_module" + + blueecho "5. Wiring up module '$modulename' for city '$city'" + ./server "${city}" up "$modulename" greenecho "added trufi module '$modulename' to city '$city'" - echo -e "It is available somewhere under the following domains (as far as known to this script):" - echo -e "- Interweb: \033[0;34m$domain\033[0;m" - if [ "$intraweb" = "yes" ]; then - echo -e "- Intraweb: \033[0;34m$module_container_name.localhost\033[0;m" - fi } moduleNotFound="" @@ -163,6 +175,6 @@ done if ! [ -z "$moduleNotFound" ]; then redecho "Error: You specified an invalid name of a module to add to city '$city'">&2 echo "A list of modules available:" - dir ./modules + ls -p ./modules | grep -v / exit 1 -fi +fi \ No newline at end of file diff --git a/data/nginx/nginx.conf b/data/nginx/nginx.conf index 4c72237..854c1f3 100644 --- a/data/nginx/nginx.conf +++ b/data/nginx/nginx.conf @@ -11,6 +11,7 @@ events { http { + server_names_hash_bucket_size 128; # https://stackoverflow.com/questions/6477239/anonymize-ip-logging-in-nginx map $remote_addr $ip_anonymized { ~(?P\d+\.\d+\.\d+)\. $ip.0; diff --git a/docker-compose.yml b/docker-compose.yml index 07666c6..8a95e0c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,7 +1,7 @@ version: '3' services: - nginx: + chief-nginx: image: nginx #:1.15-alpine ports: - "8290:80" # HTTP @@ -11,14 +11,14 @@ services: - ./data/nginx/interweb:/etc/nginx/interweb:ro - ./data/nginx/intraweb:/etc/nginx/intraweb:ro - ./data/nginx/nginx.conf:/etc/nginx/nginx.conf:ro - - ./data/nginx/inc:/etc/nginx/inc + - ./data/nginx/inc:/etc/nginx/inc:ro - ./data/logs/nginx:/var/log/nginx/ - ./data/letsencrypt/config:/etc/letsencrypt - - ./data/certbot/www:/var/www/certbot - - ./data/maps:/var/www/maps - command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'" + #- ./data/certbot/www:/var/www/certbot + - ./data/nginx/www:/var/www/:ro + command: "nginx -g \"daemon off;\"" restart: unless-stopped networks: default: - name: trufi-server + name: trufi-server-multi diff --git a/docs/commands/README.md b/docs/commands/README.md new file mode 100644 index 0000000..6963cf0 --- /dev/null +++ b/docs/commands/README.md @@ -0,0 +1,15 @@ +# Command documentation + +See [how to run scripts on linux](https://www.cyberciti.biz/faq/howto-run-a-script-in-linux/). + +## Module management + +- [add_module](./docs/commands/add_module.md) +- [remove_module](./docs/commands/remove_module.md) +- [restart_module](./docs/commands/restart_module.md) (deprecated) + +## Other scripts + +- [server](./docs/commands/server.md) +- [certify](./docs/commands/certify.md) +- [viewlog](./docs/commands/viewlog.md) diff --git a/docs/commands/add_module.md b/docs/commands/add_module.md new file mode 100644 index 0000000..596c874 --- /dev/null +++ b/docs/commands/add_module.md @@ -0,0 +1,15 @@ +# Module management + +**It adds a module to the run configuration and starts it automatically** + +This command accepts a list of modules to add like `./add_module "Bolivia-Cochabamba" otp tileserver` or `add otp tileserver` + +- Command: `add_module []` +- Example: `add_module "Bolivia-Cochabamba" otp` +- Example (using `workon` script): `add otp` + +`add_module` will execute `./server up ` in order to start the added module. You don't have to do that in a separate step. But you will need to inform `nginx` about this. We do so by executing `./server reload nginx` which causes nginx to reload its configuration without restarting. + +## Extending this script + +Read [Extending Trufi Multi-Instance Server - Extending 'add_module' script](../extend.md#extending_add_module_remove_module_script.md). diff --git a/docs/commands/certify.md b/docs/commands/certify.md new file mode 100644 index 0000000..4a2e6a3 --- /dev/null +++ b/docs/commands/certify.md @@ -0,0 +1,29 @@ + + +# Certify + +Allows you to obtain a valid HTTPS certificate from *Let's Encrypt* for free. That certificate will be valid for [90 days](https://letsencrypt.org/2015/11/09/why-90-days.html). Certify will also install a systemd timer on your docker host to have the certificate automatically renewed. + +## Requirements + +- Debian +- Systemd +- Certbot + +## Syntax + +```bash +./certify +``` + +## Usage + +```bash +./certify Germany-Hamburg /srv/trufi/nginx/www +``` + +| Argument placeholder | Description | +| -------------------- | ------------------------------------------------------------ | +| `` | The name of the city you want to have a HTTPS certificate generated for.
Example: `Germany-Hamburg` | +| `` | The absolute path to your webroot. It specifies the directory to save the token Let's Encrypt generates for you. It is home of the folder `./well-known ` (will be created by the script>. That `` needs to be served by a web server which you set up to react when Let's Encrypt pings your domain you want to enable HTTPS on port 80 (HTTP port) for. It looks up the token to verify that the domain really belongs to you. See https://stackoverflow.com/questions/49964315/what-should-letsencrypt-certbot-autos-webroot-path-be-for-a-non-php-non-sta for better explanation. | + diff --git a/docs/commands/remove_module.md b/docs/commands/remove_module.md new file mode 100644 index 0000000..d1ff504 --- /dev/null +++ b/docs/commands/remove_module.md @@ -0,0 +1,15 @@ + # remove_module + +**To remove an module from the run configuration (removes container but will not remove any persistent files)** + +This command accepts a list of modules to remove like `./remove_module "Bolivia-Cochabamba" otp tileserver` or `remove otp tileserver` + +- Command: `remove_module []` +- Example: `remove_module "Bolivia-Cochabamba" otp` +- Example (using `workon` script): `remove otp` + +`remove_module` will execute `./server down ` in order to stop and remove the docker container belonging to the module before removing its run configuration. You don't have to do that in a separate step. But you will need to inform `nginx` about this. We do so by executing `./server reload nginx` which causes nginx to reload its configuration without restarting. + +## Extending this script + +Read [Extending Trufi Multi-Instance Server - Extending 'remove_module' script](../extend.md). diff --git a/docs/commands/restart_module.md b/docs/commands/restart_module.md new file mode 100644 index 0000000..4236f14 --- /dev/null +++ b/docs/commands/restart_module.md @@ -0,0 +1,13 @@ +# restart_module + + **To (re)start an module (just use when the module hangs or other unusual things happened)** + +*This is deprecated and will be removed soon!* + +- Command: `restart_module ` +- Example: `restart_module "Bolivia-Cochabamba" otp` +- Example (using `workon` script): `restart_module otp` + +This script is not capable of restarting modules without any downtime. + +After adding or removing a module we should advertise the change to the web server nginx. We do so by executing `./server reload nginx` which causes nginx to reload its configuration without restarting. diff --git a/docs/commands/server.md b/docs/commands/server.md new file mode 100644 index 0000000..26541e4 --- /dev/null +++ b/docs/commands/server.md @@ -0,0 +1,95 @@ +# server + +Providing control about docker containers of this backend. It ensures that they share the same project name, the same network and the same naming convention. It allows you to perform actions on a group of modules, on just a single module, to view stats and many more. It also controls the web server so you don't have to care about it as `server` is capable of detecting if the web server needs to run or not. + +## Difference to `add_module` and `remove_module` + +This script only helps you with the docker container management. It helps to create/remove/update/start/stop it and can provides stats about running ones. It also wraps up our docker container naming convention in a nice logical way. + +Whereas `add_module` only prepares a module for a particular city in a way that `server` can detect and perform actions on it. `add_module` creates the necessary nginx configuration needed by nginx so it can redirect traffic to the module properly. What `add_module` does after doing its duty is to call `./server up ` to start the docker containers of the module which is an extra so the sysadmin (you) don't have to run `server` with these arguments by themselves. + +`remove_module` does the opposite and removes the nginx configuration needed by nginx to it can redirect traffic to the module properly. What `remove_module` does before doing its duty is to call `./server down ` to remove the docker containers of the module which is an extra so the sysadmin (you) don't have to run `server` with these arguments by themselves. + +## Syntax + +```sh +./server [] [] [] +``` + +## The scopes + +The `server` script behaves differently based on how it is run: + +- **City scope (action relative to a city)** + - If you use it in a Bash modified by the `workon` script. Of course you don't have to provide the argument `` anymore as usual when using `workon`. + - If you use it in the standard Bash and you provide the argument `` +- **Global scope (action relative to all cities** + - You don't run it in a Bash modified by the `workon` script. + - You don't specify `` in the standard Bash. + +Some examples: + +```sh +. ./workon Germany-Hamburg +server up # will bring all added modules in city 'Germany-Hamburg' up (scope 'city') +``` + +```sh +./server "Germany-Hamburg" up # will bring all added modules in city 'Germany-Hamburg' up (scope 'city') +``` + +```sh +./server up # will bring all added modules in all cities up (scope 'global') +``` + +## Actions + +`server` is highly extensible. Developers can build their own actions for `server`. See [extending this script](#extending-this-script) for more info about it. Below only the actions which are included in this backend by default are documented. + +An action can run in *city scope*, *global scope* or *module scope*. The code of each action lies in `./plugins/server` where each script file contains one action e.g. the action `up` has the file `./plugins/server/up`. If `server` cannot find the action specified by the user it passes it to docker-compose. + +### Start/Update docker containers of (all) modules + +To start the modules for the first time or to update modules without any downtime. + +```bash +./server up # global scope: wires all added modules in all cities up +./server up # city scope: wires all added modules in city '' up +./server # module scope: wires up module '' in city '' +``` + +### Remove docker containers of (all) module + +```bash +./server down # global scope: turn down all added modules in all cities +./server down # city scope: turn down all added modules in city '' +./server # module scope: turn down module '' in city '' +``` + +### Stop (all) modules + +```bash +./server stop # global scope: will stop all added modules in all cities +./server stop # city scope: will stop all added modules in city '' +./server # module scope: will stop module '' in city '' +``` + +### Reload nginx configuration + +This is necessary after a call to `add_module` or `remove_module`. + +```bash +./server reload nginx # global scope (no other scopes for chiefs exist) +``` + +### View a list of running modules + +```bash +./server ls # global scope: will display all running docker containers of modules in all cities including chiefes +./server ls # city scope: will display all running docker containers of the modules in city '' including chiefes +./server # module scope: will display all running services of module '' in city '' +``` + +## Extending this script + +Read [Extending Trufi Multi-Instance Server - Extending 'server' script](../extend.md) for how to write your own actions. diff --git a/docs/commands/viewlog.md b/docs/commands/viewlog.md new file mode 100644 index 0000000..4d0a7f5 --- /dev/null +++ b/docs/commands/viewlog.md @@ -0,0 +1,43 @@ +# Viewlog + +View logs of each docker container created within this structure. + +It works only if you run `./switchLogging on` beforehand to configure logging in all modules and `./server up` to apply the changes to the running docker containers. `viewlog` does not check if you have done that before. + +## Syntax + +City scope: + +```bash +./viewlog [] [] +``` + +to view logs of a service in a module of a city. + +Chief scope: + +```bash +./viewlog chief [] +``` + +to view logs of a service in the main `docker-compose.yml`. + +`[]` are optional parameters passed to the underlying call `journalctl`. + +## Usage + +City scope: + +```bash +./viewlog Germany-Hamburg tileserver tileserver +``` + +to view logs of service `tileserver` in module `tileserver` in city `Germany-Hamburg`. + +Chief scope: + +```bash +./viewlog chief nginx +``` + +to view logs of service `nginx` which is one of the chiefs. diff --git a/docs/commands/workon.md b/docs/commands/workon.md new file mode 100644 index 0000000..324aabd --- /dev/null +++ b/docs/commands/workon.md @@ -0,0 +1,32 @@ +# Workon + +When working with modules you always have to provide them with the `Country-City` argument. `workon` removes this necessity and is useful if you need to run different commands to administer something in `Country-City`. It is also useful for people who easily confuse `Country-City` strings with each other e.g. they confuse `Germany-Hamburg` and `Germany-Flensburg` with each other. `workon` helps you to mitigate that error. It changes your current Bash session so you don't have to fill in `` each time. Also it adds [command aliases](https://www.tutorialspoint.com/unix_commands/alias.htm) to each script name to make it smoother to type. Source it by executing + +```bash +. ./workon +``` + +e.g. + +```bash +. ./workon "Germany-Hamburg" +``` + +And your bash prompt will look like this + +```bash +Germany-Hamburg@ssl='no' (MODE 'real domains') $ +``` + +This way you can double check if you're applying changes to the right city. Instead of having to type e.g. + +```bash +./add_module "Germany-Hamburg" tileserver +``` + +you just need to type + +```bash +add tileserver +``` + diff --git a/docs/extend.md b/docs/extend.md new file mode 100644 index 0000000..2d52527 --- /dev/null +++ b/docs/extend.md @@ -0,0 +1,95 @@ +# Extending Trufi Multi-Instance Server + +*Trufi Multi-Instance Server* can be extended in various ways. + +## Chiefs + +We introduce "chiefs" which are all services specified in the `docker-compose.yml` at project root. They don't share the concept of modules and cities. As they don't share this concept you cannot add/remove them in the classical way with `add_module`/`remove_module`. Instead they will be ruled by `server` which creates/removes/starts/stops them automatically depending on what you do with the modules. + +But you can add/remove chiefs as you please. Just edit `docker-compose.yml` accordingly. Command `server` cannot detect modifications automatically so you have to execute + +```bash +sudo docker-compose -p `basename "$PWD"` -f "docker-compose.yml" up --build --detach +``` + +e.g. if you changed something on the ports and you want to update the already running `chief-nginx` then execute + +```bash +sudo docker-compose -p `basename "$PWD"` -f "docker-compose.yml" up "chief-nginx" --build --detach +``` + +. + +### Change ports of `chief-nginx` + +If you change ports of `chief-nginx` you need to do some things prior to executing `up chief-nginx --build --detach` as part of the `docker-compose` command. + +If you change the HTTPS port of `chief-nginx` from `8300` to `433` you need to change the following files accordingly: + +- `./nginx/app.ssl.conf` +- all files matching `./data/nginx/interweb/*.conf`. Execute `ls -l ./data/nginx/interweb/*.conf` to receive a list of matching files. + +If you change the HTTP port of `chief-nginx` from `8290` to `80` you need to change the following files accordingly: + +- `./nginx/app.nossl.conf` +- `./nginx/app.ssl.conf` +- all files matching `./data/nginx/interweb/*.conf`. Execute `ls -l ./data/nginx/interweb/*.conf` to receive a list of matching files. + +If you change the intraweb port of `chief-nginx` from `8090` to `8080` you need to change the following files accordingly: + +- `./nginx/app.intraweb.conf` +- all files matching `./data/nginx/intraweb/*.conf`. Execute `ls -l ./data/nginx/intraweb/*.conf` to receive a list of matching files + +## Extending `server` script + +The script [server](./commands/server.md) is highly extensible. Developers can build their own actions for `server`. An action can run in *city scope*, *global scope* or *module scope* without having the developer to code support for all these scopes explicitly (mostly this will be the case). The code of each action lies in `./plugins/server` where each script file contains one action e.g. the action `up` has the file `./plugins/server/up`. If `server` cannot find the action specified by the user it passes the specified action to docker-compose. + +Scripts without any file extension like `up` ( `./plugins/server/up` ) will be considered shell files and such will be sourced. All internal variables and functions in `server` are accessible from the code in the action script file too. They share the same code space. + +### Variables + +The following variables are useful for you + +| Variable name | Description | +| -------------- | ------------------------------------------------------------ | +| citiesPerModule | Only populated after a call to function `performIteration` an associative array with `module name` as keys and the cities as their values. Contains only added modules in added cities. Example:
tileserver -> Germany-Hamburg Ghana-Accra
otp -> Germany-Hamburg Ghana-Accra | +| projectname | contains the project name used to tie all docker containers of this backend together. Used with the `-p` switch of `docker-compose` | +| curModule | Only populated when in module scope. Contains the name of the module e.g. `tileserver` and **not** `./modules/tileserver` | +| curCity | Same as city | +| city | Only populated when in city or module scope. Contains the name of the city e.g. `Germany-Hamburg` and **not** `./config/Germany-Hamburg.conf` | +| curAction | contains the name of the action the user specified on the command line. Will be the same as the name of your action script e.g. `run` and **not** `./plugins/server/up` | + +### Functions + +| Function name | Description | +| ------------------------------------------------------------ | ------------------------------------------------------------ | +| performIteration "execute"\|"noexecute" [] | Iterate through modules applying to the current scope, add them to the associative array `citiesPerModule` .
If argument "execute" has been specified then it will call function `performExecution` with the argument ``afterwards.
In case you specified `` this function will execute `performExecution ` at the end. | +| performExecution | Execute `$curAction` on cities in modules found in `citiesPerModule` if no argument has been provided (default behaviour).
If an argument has been provided then it will be considered to be a function or command and will be called as such instead of the default behaviour. That passed function will be called with the following arguments:
`"$@" "$module" "$city"` e.g. `"plesantUp" "tileserver" "Germany-Hamburg"` or with additional arguments `"logs" "search:trufi" "tileserver" Germany-Hamburg"` | +| attentionPrompt | Displays a prompt to the user urging them to accept the execution of the action. It is used when the user tries to execute something dangerous like the action `down` in any mode. Pass a reason as the argument to the function call. The reason is a string and will be displayed to the user. Write as a reason the *reason* why your code wants the user to pay more attention than usual. | + +If your action does not change the amount of running docker containers for this project then consider putting a `exit 0` at the end so `server` does not run a costly operation to determine the difference. As your action does not touch the amount the difference will never arise so we can safely skip this step thus providing a faster user experience. See end of code of plugin `ls` . + +## Extending `add_module`/`remove_module` script. + +You may want to add something while adding a module **and** removing one. There is a rule of thumb: If you write a plugin for [add_module](./commands/add_module.md) then you will also need to write one for [remove_module](./commands/remove_module). + +Extending them is easy. Just save your plugin under `./plugins/add_remove` **and** `./plugins/remove_module` respectively. And a next call to these scripts will source them if these have the file extension `sh`. Plugins will be sourced for every single module name you specified on the command line as both scripts accept a list of module names. + +### Variables + +| Variable | Description | +| ------------ | ------------------------------------------------------------ | +| `modulename` | The name of the module e.g. `tileserver` | +| `city` | The name of the city e.g. `Ghana-Accra` | +| `curmode` | The domain mode we're in. This can hold the value `real domains` or `virtual domains` | + +For all other consult the source code. + +## Functions + +| Function name | Description | +| ------------- | ---------------------------------- | +| blueecho | Prints the passed string in blue | +| orangeecho | Prints the passed string in orange | +| greenecho | Prints the passed string in green | + diff --git a/docs/intraweb.md b/docs/intraweb.md new file mode 100644 index 0000000..fb55c3b --- /dev/null +++ b/docs/intraweb.md @@ -0,0 +1,15 @@ +# Intraweb + +Once enabled the script `add_module` will add the necessary nginx configuration necessary to make this feature work. It is a complete Intraweb for sysadmins to debug/check (misbehaving) modules related to a city. A sysadmin can enjoy the beauty of the webportals of some modules without the need to expose them to the public (our provided nginx configurations for each module limit access extremely). The Intraweb is accessible at `localhost:8090` on your server. So do the well known [SSH port forwarding](https://phoenixnap.com/kb/ssh-port-forwarding) magic to access it through your webbrowser on your own machine. + +```sh +ssh @ -L 8090:127.0.0.1:8090 +``` + +e.g. + +```sh +ssh foo@example.com -L 8090:127.0.0.1:8090 +``` + +And we can type in a url following the scheme `http://--.localhost:8090/` e.g. to reach the service `tileserver-tileserver` of city `Germany-Hamburg` type `http://tileserver-tileserver-germany-hamburg.localhost:8090/`. diff --git a/init b/init index 8173184..147a274 100755 --- a/init +++ b/init @@ -57,7 +57,7 @@ for i in {1..2}; do echo -ne "showing prompt in $((maxcount - i)) ...\033[0K\r" # https://stackoverflow.com/questions/5861428/bash-script-erase-previous-line sleep 1 done -echo -e "Do you want to process? (y|n):\033[0;34m" +echo -ne "Do you want to process? (y|n):\033[0;34m " read decision checkDecision $decision "y" echo -en "\033[0;m" @@ -87,27 +87,36 @@ rm -r $module_dir/nginx-* python3 modifyComposes.py greenecho " Modules downloaded!" -blueecho "2. Specify environment" -echo -e "Choose environment [prod|dev]:\033[0;34m" +blueecho "2. Turn SSL on? This is recommended if clients directly communicate with this backend without a SSL enabled proxy between" +echo -ne "[y|n]:\033[0;34m " read decision echo -ne "\033[0;m" -if [ $decision = "prod" ]; then - environment="production" - ENVIRONMENT="PRODUCTION" - nginxconf="./nginx/app.template.conf" +if [ $decision = "y" ]; then + ssl="yes" + nginxconf="./nginx/app.ssl.conf" +elif [ $decision = "n" ]; then + ssl="no" + nginxconf="./nginx/app.nossl.conf" +else + echo "Error: invalid input" + exit 1 +fi + +blueecho "3. Turn on feature 'Intraweb' to wire up a intraweb for sysadmins with access to this server can use for troubleshooting modules etc.?" +echo "This intraweb is only accessible through SSH (thanks to SSH port forwarding)" +echo -ne "[y|n]:\033[0;34m " +read decision +echo -ne "\033[0;m" +if [ "$decision" = "y" ]; then intraweb="yes" -elif [ $decision = "dev" ]; then - environment="development" - ENVIRONMENT="DEVELOPMENT" - nginxconf="./nginx/app.dev.conf" +elif [ "$decision" = "n" ]; then intraweb="no" else - redecho "Error: invalid input" + echo "Error: invalid input" exit 1 fi -blueecho "3. Specify mode" - +blueecho "4. Specify mode" MODE_REALDOMAINS="real domains" MODE_VIRTUALDOMAINS="virtual domains" CURMODE="" @@ -153,13 +162,13 @@ if [ "$decision" = "$MODE_REALDOMAINS" ]; then elif [ "$decision" = "$MODE_VIRTUALDOMAINS" ]; then CURMODE=$MODE_VIRTUALDOMAINS else - redecho "Error: invalid input" + echo "Error: invalid input" exit 1 fi blueecho "4. Save & load configuration" # save -echo "env=\"$environment\" +echo "ssl=\"$ssl\" curmode=\"$CURMODE\" intraweb=\"$intraweb\"">"./data/instance.conf" # load @@ -169,18 +178,14 @@ blueecho "5. NGINX configuration" orangeecho " preparing nginx configuration ..." dest="./data/nginx/interweb/virtualdomains.conf" if ! [ -f "$dest" ] && [ "$CURMODE" = "$MODE_VIRTUALDOMAINS" ]; then - if [ "$enviroment" = "production" ]; then - echo "Please tell me the domain name. This can be changed later by editing the '$dest' file" - read decision - orangeecho " Setting domain '$decision' ..." - destcontent=`echo "$destcontent" | sed "s/example.org/$decision/"` - domain="$decision" - else - orangeecho " Setting domain '$hostname' since we are in 'development' mode (this can be changed later by editing the '$dest' file) ..." - destcontent=`echo "$destcontent" | sed "s/example.org/$hostname/"` - domain="$hostname" - fi - orangeecho " adding $environment virtual server ..." + echo "Please tell me the domain name. This can be changed later by editing the '$dest' file" + read decision + + orangeecho " Setting domain '$decision' ..." + destcontent=`echo "$destcontent" | sed "s/example.org/$decision/"` + domain="$decision" + + orangeecho " adding virtual server ..." sed "s/# modules/# modules of all cities all using this single virtual server configuration\ninclude \/etc\/nginx\/interweb\/*\/\*.conf\;/" "$nginxcityconf_template_interweb" | sed "s/example.org/$domain/g" > "$dest" mkdir -p "./data/logs/nginx/$domain" @@ -196,7 +201,7 @@ greenecho "[DONE] Setup for '$name' " greenecho "#################### END ################################" echo greenecho "Current configuration:" -echo "Environment : $environment" +echo "SSL enabled : $ssl" echo "Mode : $CURMODE" echo "Domain : $domain" echo "Intraweb activated: $intraweb" diff --git a/lib/env b/lib/env index 17c530e..8e506ec 100755 --- a/lib/env +++ b/lib/env @@ -5,22 +5,12 @@ source "$instanceConfigPath" MODE_REALDOMAINS="real domains" MODE_VIRTUALDOMAINS="virtual domains" -NGINX_CONFIGPATH_PRODSERVER="./nginx/app.template.conf" -NGINX_CONFIGPATH_DEVSERVER="./nginx/app.dev.conf" +NGINX_CONFIGPATH_SSLSERVER="./nginx/app.ssl.conf" +NGINX_CONFIGPATH_NOSSLSERVER="./nginx/app.nossl.conf" -if [ "$env" = "production" ]; then - nginxcityconf_template_interweb="$NGINX_CONFIGPATH_PRODSERVER" +if [ "$ssl" = "yes" ]; then + nginxcityconf_template_interweb="$NGINX_CONFIGPATH_SSLSERVER" else - nginxcityconf_template_interweb="$NGINX_CONFIGPATH_DEVSERVER" + nginxcityconf_template_interweb="$NGINX_CONFIGPATH_NOSSLSERVER" fi nginxcityconf_template_intraweb="./nginx/app.intraweb.conf" - -#MODE_VIRTUALDOMAINS_INDICATOR="./data/nginx/interweb/virtualdomains.conf" -#ENV_PROD_INDICATOR="./data/.production" -#ENV_DEV_INDICATOR="./data/.development" - -# -# -# if [ -d "$ENV_PROD_INDICATOR" ]; then -# env="production" -# fi diff --git a/lib/plugin b/lib/plugin new file mode 100644 index 0000000..1f7704a --- /dev/null +++ b/lib/plugin @@ -0,0 +1,25 @@ +#!/bin/bash +invokeSinglePlugin() { + # 0 true + # 1 false + pluginsInvoked=1 + local res=`ls "plugins/$1" | grep "$2.sh"` + if [ -n "$res" ]; then + pluginsInvoked=0 # means true + source "plugins/$1/$2.sh" + else + local res=`ls "plugins/$1" | grep "$2"` + if [ -n "$res" ]; then + pluginsInvoked=0 # means true + ./plugins/$1/$2 ${pluginargs[@]} + fi + fi +} + +invokeAllPluginsOf() { + if [ -d "plugins/$1" ]; then + for script in `ls "plugins/$1"`; do + invokeSinglePlugin "$1" "$script" + done + fi +} diff --git a/lib/utils b/lib/utils new file mode 100644 index 0000000..bfe7763 --- /dev/null +++ b/lib/utils @@ -0,0 +1,6 @@ +#!/bin/bash +removeFirstElemInArgs() { + # remove elem on first index + unset args[0] + args=( `echo "${args[@]}"` ) +} diff --git a/lib/validation b/lib/validation index f6a9345..ce235e1 100644 --- a/lib/validation +++ b/lib/validation @@ -3,17 +3,17 @@ # environment and mode args=( $@ ) source ./lib/env +source ./lib/utils # validations ## if $cityfile is empty but a valid city configuration name has been provided if [ -z "$cityfile" ] && [ -f "./config/${args[0]/".env"/""}.env" ]; then envname=${args[0]//".env"/""} - export cityfile="./config/$envname.env" + cityfile="./config/$envname.env" +# export cityfile="./config/$envname.env" - # remove elem on first index - unset args[0] - args=( `echo "${args[@]}"` ) + removeFirstElemInArgs fi ## if $cityfile contains something but the $city environment variable hasn't been populated diff --git a/modifyComposes.py b/modifyComposes.py index 8b45e1a..a091399 100755 --- a/modifyComposes.py +++ b/modifyComposes.py @@ -14,15 +14,19 @@ def orangeprint(text): sys.exit(1) orangeprint("patching modules ...") -for ext in os.listdir(modulesPath): - extDir = os.path.join(modulesPath, ext) - if os.path.exists(os.path.join(extDir, "data")): +for module in os.listdir(modulesPath): + if module.find("_") > -1: + os.rename(os.path.join(modulesPath, module), os.path.join(modulesPath, module.replace("_", "-"))) + module = module.replace("_", "-") + moduleDir = os.path.join(modulesPath, module) + if os.path.exists(os.path.join(moduleDir, "data")): print(" renaming directory 'data' to 'data_template' ...") - os.rename(os.path.join(extDir, "data"), os.path.join(extDir, "data_template")) + os.rename(os.path.join(moduleDir, "data"), os.path.join(moduleDir, "data_template")) - composefile = os.path.join(extDir, "docker-compose.yml") + composefile = os.path.join(moduleDir, "docker-compose.yml") + if os.path.exists(composefile): - orangeprint(" modifying module '{}' ('{}') ...".format(ext, extDir)) + orangeprint(" modifying module '{}' ('{}') ...".format(module, moduleDir)) sfile = open(composefile, "r") dockercompose = yaml.safe_load(sfile.read()) sfile.close() @@ -30,7 +34,12 @@ def orangeprint(text): servicesToRename = [] for service in dockercompose["services"]: + addToRenameList = False if not service.endswith("-$city_normalize"): + addToRenameList = True + if 2 > service.count(module): + addToRenameList = True + if addToRenameList: servicesToRename.append(service) print(" service '{}'".format(service)) if "volumes" in dockercompose["services"][service]: @@ -47,10 +56,13 @@ def orangeprint(text): del dockercompose["services"][service]["ports"] for service in servicesToRename: - print(" renaming to '{}' ...".format(service + "-$city_normalize")) - dockercompose["services"][service + "-$city_normalize"] = dockercompose["services"][service] + newName = module + "-" + service.replace("-$city_normalize", "") + "-$city_normalize" + print(" renaming to '{}' ...".format(newName)) + dockercompose["services"][newName] = dockercompose["services"][service] del dockercompose["services"][service] + + dockercompose["networks"]["default"]["name"] = "$projectname" - sfile = open(os.path.join(extDir, "docker-compose.yml"), "w") + sfile = open(os.path.join(moduleDir, "docker-compose.yml"), "w") sfile.write(yaml.dump(dockercompose, sort_keys=False, default_flow_style=False)) sfile.close() diff --git a/nginx/app.dev.conf b/nginx/app.nossl.conf similarity index 100% rename from nginx/app.dev.conf rename to nginx/app.nossl.conf diff --git a/nginx/app.template.conf b/nginx/app.ssl.conf similarity index 100% rename from nginx/app.template.conf rename to nginx/app.ssl.conf diff --git a/plugins/server/ls.sh b/plugins/server/ls.sh new file mode 100755 index 0000000..2003715 --- /dev/null +++ b/plugins/server/ls.sh @@ -0,0 +1,47 @@ +#!/bin/bash +# docker trufi name convention: +# Syntax : _--_ +# Example: city-services_tileserver-tileserver-germany-hamburg_1 + +# Query construction to filter container names (positive filter) +## Syntax : _--_ +## .............. +query="${projectname}_" # match $projectname by default and always + +## Syntax : _--_ +## ............ +if [ -n "$curModule" ]; then + query+="${curModule}-" # and match module $curModule e.g. match module 'tileserver' +else + query+=".*-" # and match any modules +fi + +## Syntax : _--_ +## .............. +query+=".*-" # and match any services + +## Syntax : _--_ +## ...... +if [ "$cityScope" = "city" ]; then + query+="${city,,}" # and match city ${city,,} e.g. match city 'germany-hamburg' +else + query+=".*" # and match any cities +fi + +if [ -z "$curActionArgs" ]; then + curActionArgs=(--format "\"table {{.Names}} \t {{.CreatedAt}} \t {{.Status}}\"" --no-trunc) +fi +# get status table +# get results of query, +# get chief services +# and the status table heading +#sudo docker container ls --format "table {{.Names}} \t {{.CreatedAt}} \t {{.Status}}" --no-trunc | grep "$query\|${projectname}_chief\|NAMES" +body=`eval sudo docker container ls ${curActionArgs[@]}` +heading=`echo "$body" | sed -ne "1p"` +body=`echo "$body" | grep "$query\|${projectname}_chief"` + +body=${body//chief-/"\033[0;35mchief\033[0;m-"} +echo -e "\033[1;37m${heading}\033[0;m" +echo -e "$body" + +exit 0 diff --git a/plugins/server/reload.sh b/plugins/server/reload.sh new file mode 100755 index 0000000..c1634b4 --- /dev/null +++ b/plugins/server/reload.sh @@ -0,0 +1,32 @@ +#!/bin/bash + +# reload action for 'nginx' hardcoded as this is the main module not dynamically add-/removeable +if [ "$curModule" = "nginx" ]; then + orangeecho "reloading nginx configuration without the need to restart nginx ..." + citiesPerModule["./"]="docker-compose" + curAction="exec" + curActionArgs="chief-nginx nginx -s reload" + performExecution +else + # check if a modfunctions.sh exists for the module + if [ -f "./modules/$curModule/modfunctions.sh" ]; then + source "./modules/$curModule/modfunctions.sh" + if [ -n "$modreload" ]; then + curAction="exec" + for reloadCMD in "${modreload[@]}"; do + curActionArgs="$reloadCMD" # e.g. tileserver killall -HUP tileserver-gl + # | + # --> name of the service without '-${city_normalize}' because will be appended & expanded below + #orangeecho "Executing \033[0;34m${curActionArgs}\033[0;m inside container '${curActionArgs[0]}-${city,,}' in module '$curModule' of city '$city' ..." + curActionArgs=( $curActionArgs ) + curActionArgs[0]=${curActionArgs[0]}-${city,,} # name of the service (appended & expanded) + curActionArgs="${curActionArgs[@]}" + performExecution + done + else + redecho "No reloading strategy found for module '$curModule'" + fi + fi +fi + +exit 0 diff --git a/plugins/server/up.sh b/plugins/server/up.sh new file mode 100644 index 0000000..ad198a0 --- /dev/null +++ b/plugins/server/up.sh @@ -0,0 +1,45 @@ +#!/bin/bash + +# pleasant up +pleasantUp() { + local module="$1" + local nameOfCity="$2" + # get names of all services of that city in the module + allServices=`sudo docker-compose -p "$projectname" -f "${nameOfCity}.yml" ps --services` + allServices=( $allServices ) + + # get running services of that city in the module + runningServices=`sudo docker-compose -p "$projectname" -f "${nameOfCity}.yml" ps` + length=${#runningServices[@]} + + # iterate through names of all services of that city in the module + for servicename in "${allServices[@]}"; do # e.g. tileserver-ghana-accra, tileserver-germany-hamburg + isRunning=`sudo docker container ls | grep "${projectname}_${servicename}"` # prints a status line if the container with the name '${projectname}-${servicename}' is running otherwise it prints nothing + if [ -n "$isRunning" ]; then + orangeecho " - updating docker service '$servicename' without downtime ..." + + # obtain container id of service 'servicename' of that city in the module + IFS=" " + curContainerId=( $isRunning ) # recycling variable 'isRunning' + curContainerId="${curContainerId[0]}" + + # create a new container of service 'servicename' (scale up) + sudo docker-compose -p "$projectname" -f "${nameOfCity}.yml" up --detach --scale ${allServices[$i]}=2 --no-recreate ${allServices[$i]} + + sleep 5 # TODO: healtheck instead of waiting five seconds in hope this is enough for the startup process + + # remove old container + sudo docker rm -f "$curContainerId" + + # scale back + sudo docker-compose -p "$projectname" -f "${nameOfCity}.yml" up --detach --scale ${allServices[$i]}=1 --no-recreate ${allServices[$i]} + else + orangeecho " - wiring up a docker container for service '$servicename' ..." + sudo docker-compose -p "$projectname" -f "${nameOfCity}.yml" up --build --detach + fi + done +} +# perform iteration over all affected docker-compose files +# - to populate variable 'citiesPerModule' +# - and to literate over the populated variable and call our own 'plesantUp' with each city value +performIteration "execute" pleasantUp \ No newline at end of file diff --git a/remove_module b/remove_module index 3eade5b..c2bc446 100755 --- a/remove_module +++ b/remove_module @@ -1,7 +1,12 @@ #!/bin/bash +projectname=`basename "$PWD"` + source ./lib/colorful +# Removes all occurrence of a configuration e.g. to make intraweb work +# removeNGINXconf +# e.g. removeNGINXconf intraweb tileserver removeNGINXconf() { local modulename="$2" if [ -f "./data/nginx/${1}/${city}/${modulename}.conf" ]; then @@ -9,7 +14,7 @@ removeNGINXconf() { rm "./data/nginx/${1}/${city}/${modulename}.conf" --verbose fi - if [ -d "./data/nginx/${1}/${city}" ] && [ -z `dir "./data/nginx/${1}/${city}"` ]; then + if [ -d "./data/nginx/${1}/${city}" ] && [ -z `ls "./data/nginx/${1}/${city}"` ]; then orangeecho " removing nginx configuration of city '$city' because it's empty now ..." rm -R "./data/nginx/${1}/${city}" --verbose if [ -f "./data/nginx/${1}/${city}.conf" ]; then @@ -24,17 +29,34 @@ source ./lib/validation removeModule() { local modulename="$1" - blueecho "2. deactivate module '$modulename' in city '$city'" + + blueecho "1. Turning down module '$modulename' of city '$city'" + ./server "$city" down "$modulename" + + blueecho "2. remove interweb nginx configuration" + removeNGINXconf "interweb" "$modulename" + if [ "$intraweb" = "yes" ]; then + blueecho "3. remove intraweb nginx configuration" + cd modules/$modulename + allServices=`sudo docker-compose -p "$projectname" -f "${city}.yml" ps --services` + allServices=( $allServices ) + cd ../../ + # iterate through names of all services of that city in the module + for servicename in "${allServices[@]}"; do + removeNGINXconf "intraweb" "$servicename" + done + fi + + source "lib/plugin" + invokeAllPluginsOf "remove_module" + + blueecho "4. deactivate module '$modulename' in city '$city'" ymlFile="$moduleDir/$city.yml" ymlFileInactive="$ymlFile.inactive" if [ -f "$ymlFile" ]; then orangeecho " setting module in directory '$moduleDir' to inactive ..." - - cd $moduleDir - sudo docker-compose -f "$city.yml" down - cd ../../ mv "$ymlFile" "$ymlFileInactive" --verbose if ! [ -f "$ymlFileInactive" ]; then @@ -47,14 +69,7 @@ removeModule() { else greenecho " already set to inactive. No need to do it again :)" fi - - blueecho "3. remove interweb nginx configuration" - removeNGINXconf "interweb" "$modulename" - if [ "$intraweb" = "yes" ]; then - blueecho "4. remove intraweb nginx configuration" - removeNGINXconf "intraweb" "$modulename" - fi - + greenecho "removed trufi module '$modulename' from city '$city'" } @@ -63,11 +78,6 @@ for module in "${args[@]}"; do if [ -z "$module" ]; then continue fi - #if $modulename is empty or $moduleDir does not exist - if ! [ -d "$moduleDir" ]; then - orangeecho "WARNING: Module '$module' does not exist, ignoring ..." - continue - fi echo removeModule "$module" done diff --git a/restart_module b/restart_module index a04aa9c..e2cb664 100755 --- a/restart_module +++ b/restart_module @@ -4,6 +4,8 @@ projectname=`basename "$PWD"` source ./lib/colorful +orangeecho "This script has been deprecated and will be removed in the next major release v3.0.0" + blueecho "performing some checks & normalizations ..." source ./lib/validation @@ -13,7 +15,7 @@ moduleDir="./modules/$modulename" if [ -z $modulename ] || ! [ -d "$moduleDir" ]; then redecho " Error: You must specify a valid name of an module to add to city '$city'">&2 echo " A list of modules available:" - dir ./modules + ls -p ./modules | grep -v / exit 1 elif [ -f "$moduleDir/$city.yml.inactive" ]; then redecho " Module '$modulename' inactive in city '$city'. Inactive modules cannot be restarted!" @@ -21,5 +23,5 @@ elif [ -f "$moduleDir/$city.yml.inactive" ]; then fi cd "$moduleDir" -sudo docker-compose -p "$projectname" -f "$city.yml" --env-file ../../$cityfile restart +sudo docker-compose -p "${projectname}" -f "$city.yml" --env-file ../../$cityfile restart cd ../../ diff --git a/server b/server index 8b485a3..1f0fcba 100755 --- a/server +++ b/server @@ -1,7 +1,9 @@ #!/bin/bash - source ./lib/colorful source ./lib/env +source ./lib/utils +source "lib/plugin" +# server [city] action [module] [actionargs] # Run added modules (this will be automatically normalized to 'up --detach' before passing to docker-compose) # --------------------- @@ -33,225 +35,232 @@ source ./lib/env # other commands that docker-compose accepts +# Initalize and validate +projectname=`basename "$PWD"` args=( $@ ) -operateInScope="" -declare -A added_city_modules -module_dockercompose_city="$city.yml" - -body=() -projectname=`basename "$PWD"` -action="${args[0]}" -customCode="" -customDisplay="" +declare -A citiesPerModule -# remove elem from array 'args' on first index -unset args[0] -args=( `echo "${args[@]}"` ) +## if the first argument is a valid city configuration file and provided in the form 'Country-City' or 'Country-City.env' +## e.g.: server [city] action [module] [actionargs] +## .... +cityScope="all" +envfile="${args[0]/'.env'/''}.env" +if [ -f "./config/$envfile" ]; then # export the variables in the city configuration file + export $(cat ./config/$envfile | xargs) + cityScope="city" + blueecho "performing some checks & normalizations ..." + source ./lib/validation # will re-init variable 'args' and will normalize it with self-provided function 'removeFirstElemInArgs' + curCity="$city" # only to provide analog behaviour to the comming 'curAction', 'curModule' and 'curActionArgs' variables but not needed for this architecture to work properly except for plugins using 'curCity' +fi -# if the first argument is a valid city configuration file and provided in the form 'Country-City' or 'Country-City.env' -if [ -f "./config/${args[0]/'.env'/''}.env" ]; then # export the variables in the city configuration file - export $(cat ../$envfile | xargs) +if [ ${#args[@]} -eq 0 ]; then + redecho "Error: Not enough arguments!" + echo "USAGE: ./server [city] action [module] [actionargs]" + echo "" + echo "[city] Optional, name of city e.g. Bolivia-Cochabamba" + echo "action Required, name of the action to execute/perform e.g. 'start'" + echo "[module] Optional, name of the module to execute/perform action against e.g. 'tileserver'" + echo "[actionargs] Optional, additional arguments for the action to execute/perform e.g. '--build --detach' for action 'up'" + exit 1 fi -if [ -z "$city" ]; then - maxcount=5 - for i in {0..5}; do - echo -ne "No city configuration specified! Action '$action' will be applied to all added modules in all cities in $((maxcount - i)) ...\033[0K\r" # https://stackoverflow.com/questions/5861428/bash-script-erase-previous-line - # sleep 1 - done - operateInScope="global" - cityrelatedtext="in all cities" - # list all added modules in all cities - listAddedModules() { - for item in `dir "$1"`; - do - local addedModules="" - for content in `dir "$1/$item"`; - do - if ! [ "$content" == "docker-compose.yml" ] && [[ "$content" == *.yml ]] && [ -f "$1/$item/$content" ]; then - addedModules+=" -f $content" +## e.g.: server [city] action [module] [actionargs] +## ...... +curAction="${args[0]}" +action="$curAction" +removeFirstElemInArgs +## e.g.: server [city] action [module] [actionargs] +## ........ +#if [ -n "${args[0]}" ] && [ -d "./modules/${args[0]}" ]; then +curModule="${args[0]}" +removeFirstElemInArgs +#fi +## e.g.: server [city] action [module] [actionargs] +## ............ +curActionArgs="${args[@]}" + +# Define the core + +## batch execution (based on what an iterator has found previously) +performExecution() { + if [ -z "${!citiesPerModule[@]}" ]; then # a message for developers + redecho "Trying to call 'performExecution' without running 'performExecution \"noexecute\"' before leads to this error message!" + echo "crying ..." + return + + fi + for module in "${!citiesPerModule[@]}"; do + orangeecho "performing desired action '$curAction' on module '$module' ..." + if [ -d "$module" ]; then + local oldpwd="$PWD" + cd "$module" + + local cities=( ${citiesPerModule[$module]} ) + for city in "${cities[@]}"; do + orangeecho " - in city $city ..." + if [ -n "$1" ]; then # assume a function when given + "$@" "$module" "$city"; + else + sudo docker-compose -p "${projectname}" -f "$city.yml" $curAction $curActionArgs fi done - if [ -n "$addedModules" ]; then - added_city_modules[$item]="$addedModules" - fi - done - } -else - operateInScope="city" - cityrelatedtext="in city '$city'" - # list all added modules for the current city - listAddedModules() { - for item in `dir "$1"`; - do - if [ -f "$1/$item/$module_dockercompose_city" ]; then - added_city_modules[$item]="-f $module_dockercompose_city" - fi - done - } -fi - -_reloadNginxConfig() { - orangeecho "reloading nginx configuration without the need to restart nginx ..." - sudo docker-compose -p "$projectname" -f "docker-compose.yml" exec nginx nginx -s reload -} -_donothing() { - return 0 + cd "$oldpwd" + fi + done } -_ls() { - heading=() - line=1 - IFS=" -" - result=`sudo docker container ls` - result=( $result ) - for i in "${result[@]}"; do - if (( $line > 1 )); then - belongstoproject=`echo $i | grep "$projectname" | grep "$city"` - isNginx=`echo $i | grep "$projectname" | grep "nginx"` - if [ -n "$belongstoproject" ] || [ -n "$isNginx" ]; then - body+=($i) - fi - else - heading+=($i) +## iterators for modules +### perform on all cities of a module +iterateThroughAllCitiesOfModule() { + cd "modules/$1" + addedCities="" + for composeFile in `ls | grep .yml`; do + if ! [ "$composeFile" = "docker-compose.yml" ] && [[ "$composeFile" == *.yml ]] && [ -f "$composeFile" ]; then + addedCities+="${composeFile/'.yml'/''} " fi - line=$(($line+1)) done - - printArrayLineByLine + citiesPerModule["modules/$1"]=$addedCities + cd "../../" } -# a customCode and therefore getting parameters: -# $1 name of docker-compose file -# $2 $action -_refresh() { - # amount of arguments passed lower than three then there weren't three arguments specified necessary to operate on specified docker-composes so - if [ ${#?[@]} -lt 2 ]; then - # assume operation on file docker-compose.yml in project root and do nothing - return - fi - - local dockercomposeFile=$1 - # get services - local runningServices=`sudo docker-compose -p "$projectname" -f "$dockercomposeFile" ps --services` - local runningContainerIds=`sudo docker-compose -p "$projectname" -f "$dockercomposeFile" ps --quiet` - local runningServices=( $runningServices ) - local runningContainerIds=( $runningContainerIds ) - local length=${#runningServices[@]} - - # https://github.com/docker/compose/issues/1786#issuecomment-579794865 - for ((i=0; i<${length}; i++)); do - orangeecho "Updating docker service '${runningServices[$i]}' of docker-compose file '$dockercomposeFile' ..." - sudo docker-compose -p "$projectname" -f "$dockercomposeFile" up --detach --scale ${runningServices[$i]}=2 --no-recreate ${runningServices[$i]} +### perform on all modules in a city +iterateThroughAllModulesInCity() { + cd modules + for module in `ls -p ./ | grep /`; do + module=${module/\//""} + cd "$module" - sleep 5 # TODO: healtheck instead of waiting five seconds in hope this is enough for the startup process - - sudo docker rm -f ${runningContainerIds[$i]} - sudo docker-compose -p "$projectname" -f "$dockercomposeFile" up --detach --scale ${runningServices[$i]}=1 --no-recreate ${runningServices[$i]} + if [ -f "$1.yml" ]; then + citiesPerModule["modules/$module"]="$1" + fi + cd ../ done + cd ../ } -printArrayLineByLine() { - for i in "${heading[@]}"; do - echo "$i" - done - - for i in "${body[@]}"; do - echo "$i" +### perform on all modules in all cities +iterateThroughEverything() { + for module in `ls -p modules | grep /`; do + local module=${module/\//""} + iterateThroughAllCitiesOfModule $module $1 done - } -operateOnNginx() { - orangeecho "performing desired action '$action' on docker-compose.yml $appendTXT..." - sudo docker-compose -p "$projectname" -f "docker-compose.yml" $action +### determine iteration strategy to use and call it +performIteration() { + local error="" + if [ "$cityScope" = "all" ] && [ -z "$curModule" ]; then # no city and no module have been specified (perform on all modules in all cities) + # server action + iterateThroughEverything $1 # perform on all modules in all cities + elif [ "$cityScope" = "all" ] && [ -n "$curModule" ]; then # no city but a module have been specified + # server action module [actionargs] + iterateThroughAllCitiesOfModule $curModule $1 # perform on all cities of a module + elif [ "$cityScope" = "city" ] && [ -z "$curModule" ]; then # a city but no module have been specified + # server city action + iterateThroughAllModulesInCity $city $1 # perform on all modules in current city + elif [ "$cityScope" = "city" ] && [ -n "$curModule" ]; then # a city and a module have been specified + # server city action module [actionargs] + citiesPerModule["modules/$curModule"]="$city" + else # if this else gets called then it means something is wrong with the code + redecho "Error: Action '$curAction' couldn't be run as an iteration strategy to perform couldn't be found!" + local error="ITERATIONSTRATEGYNOTFOUND" + fi + if [ "$1" = "execute" ] && [ -z "$error" ]; then + local locargs=( $@ ) + unset locargs[0] + locargs=( `echo "${locargs[@]}"` ) + performExecution ${locargs[@]} + fi } - -if ! [ "$operateInScope" = "global" ]; then - blueecho "performing some checks & normalizations ..." - source ./lib/validation -fi - -# if action 'nginx' specified then overwrite normal behaviour of this script -if [ "$action" = "nginx" ]; then - blueecho "will only operate on 'nginx'" - action="${args[@]}" - listAddedModules() { - true - } - if [ "$action" = "reload" ]; then - _reloadNginxConfig - greenecho "OK" - exit 0 +## attention please prompt +attentionPrompt() { + local reason="$1" + local cityrelatedtext="" + if [ -z "$reason" ]; then + reason="(no reason provided)" fi -fi - -if [ "$action" = "run" ] || [ "$action" = "up" ]; then - orangeecho "deprefixing command to a valid docker-compose command ..." - action="up --build --detach" - unset $2 -elif [ "$action" = "ls" ]; then - orangeecho "list the running containers of each module $cityrelatedtext all at once (nginx will be always included) ..." - action="ps" - customCode="_donothing" - customDisplay="_ls" -elif [ "$action" = "refresh" ]; then - customCode="_refresh" -elif [ "$action" = "log" ] || [ "$action" = "viewlog" ] ; then - redecho "Action '$action' no longer supported..." -fi - -listAddedModules "modules" # the directories in 'modules' represent the names of the modules - -cd modules -for module in ${!added_city_modules[@]}; do - cd "$module" - - if [ -z "$customCode" ]; then - orangeecho "performing desired action '$action' on module '$module' $cityrelatedtext ..." - #echo "sudo docker-compose -p \"$projectname\" ${added_city_modules[$module]} $action" - sudo docker-compose -p "$projectname" ${added_city_modules[$module]} $action + redecho "----------------------------------------" + echo -e "The action '$curAction' asks for\033[1;m your approval\033[0;m before continuing! There must be a\033[0;31m dangerous situation\033[0;m it wants you to be aware of" + echo -e "Reason: \033[0;34m${reason}\033[0;m" + echo -e "\033[1;37mExecution details:\033[0;m" + echo -e "Scope: \033[0;34m${cityScope}\033[0;m" + if [ "$cityScope" = "city" ]; then + echo -e "Affected city: \033[0;34m${city}\033[0;m" + local cityrelatedtext=" in city '$city'" else - eval $customCode "${added_city_modules[$module]/-f/}" "$action" + echo -e "Affected city: \033[0;34m(action will be applied to all cities when approving)\033[0;m" fi - - cd ../ -done -cd ../ - - -if [ -z "$customCode" ]; then - appendTXT="" - if [ "$action" = "down" ]; then - appendTXT="because there is no container for any city running " + if [ -n "$curModule" ]; then + echo -e "Affected module: \033[0;34m${curModule}\033[0;m" + else + echo -e "Affected module: \033[0;34m(action will be applied to all modules${cityrelatedtext} when approving)\033[0;m" fi - # `sudo docker container ls | grep "$projectname" | wc -l` - Get running containers containing $projectname in their name and count them - # sudo docker container ls - List all running containers on this system - # grep "$projectname" - only show running containers containing $projectname in their name - # wc -l - count the lines returned by the last command - - # sudo docker-compose -f "docker-compose.yml" ps -q | wc -l` - Get running containers belonging to "docker-compose.yml" and count them - # sudo docker-compose -f "docker-compose.yml" ps -q - Get the ids of the containers belonging to "docker-compose.yml" - # wc -l - count the lines returned by the last command + echo -e "Action: \033[0;34m${curAction}\033[0;m" + echo -e "Action arguments: \033[0;34m${curActionArgs}\033[0;m" + echo -e "Current date and time: \033[0;34m$(date)\033[0;m" + redecho "----------------------------------------" - if [ `sudo docker container ls | grep "$projectname" | wc -l` -gt `sudo docker-compose -f "docker-compose.yml" ps -q | wc -l` ]; then - if [ "$action" = "down" ] || [ "$action" = "stop" ]; then - blueecho "Not performing action '$action' on service 'nginx' as this is what running containers from other cities depend on!" + decide() { + echo -ne "\033[0;35mYour decision [y|n]:\033[0;m \033[0;34m" + read decision + if [ "$decision" = "n" ]; then + greenecho "Action '$curAction' has been interrupted at your request. Nothing dangerous has been executed!" + echo "Script interrupted by user" + exit 2 + elif [ "$decision" = "y" ]; then + orangeecho "Execution of action '$curAction' continues!" + return else - operateOnNginx + redecho "Invalid input. Only 'y' and 'n' (case-sensitive) counts!" + decide fi - else - operateOnNginx + } + decide +} + +# Call actions (made by plugins) +pluginargs=( ${moreargs[@]} ) +invokeSinglePlugin "server" "$curAction" +if [ $pluginsInvoked -eq 1 ]; then # a plugin hasn't been invoked + if [ "$curAction" = "down" ] || [ "$curAction" = "stop" ]; then + attentionPrompt "Are you sure that nothing depends on the modules you want to perform action '$curAction' on?" + performIteration "execute" + else # no additional behaviour defined so pass to docker-compose directly + performIteration "execute" fi -else - eval $customCode $action fi -if [ -n "$customDisplay" ]; then - eval $customDisplay -fi +# Determine if there are still running services after certain action has been performed +## Difference between 'grep "${projectname}-chief"' and '-p "${projectname}"': +## grep "${projectname}-chief": This is a combination of the project name being the same for all modules and all chiefs running under this structure AND the prefix of the service names in the docker-compose.yml +## -p "${projectname}": This is the project name of the structure itself. It is being the same for all modules and all chiefs running under this structure +runningServices=`sudo docker container ls | grep "${projectname}" | wc -l` # count running services including the chief ones +allServiceContainers=`sudo docker container ls --all | grep "${projectname}" | wc -l` # count all services including the chief ones +runningChiefs=`sudo docker container ls | grep "${projectname}_chief-" | wc -l` # count running chiefs +allChiefContainers=`sudo docker container ls --all | grep "${projectname}_chief-" | wc -l` +allChiefsAvailable=`sudo docker-compose -p "${projectname}" -f docker-compose.yml ps --services | wc -l` # count all chiefs +## if no service is running (except the chiefs) +if [ 0 -eq $((runningServices-runningChiefs)) ]; then + ### check if they have been removed completely (except the chiefs) + if [ 0 -eq $((allServiceContainers-runningChiefs)) ]; then # also stop and remove all chiefs + orangeecho "turning down chiefs as there are no docker containers left ..." + sudo docker-compose -p "${projectname}" -f "docker-compose.yml" down # so we can even turn down and remove the chief ones + else ### not all services have been removed but they have been stopped (chiefs not counted in) + orangeecho "stopping chiefs as there are only stopped containers ..." + sudo docker-compose -p "${projectname}" -f "docker-compose.yml" stop # so we just stop the chiefs + fi +else ## if services are running (chiefs not counted in) + ### if amount of chiefs having containers is greater than the amount of running ones + if [ $allChiefContainers -gt $runningChiefs ]; then + orangeecho "start chiefs as there are running docker containers ..." + sudo docker-compose -p "${projectname}" -f "docker-compose.yml" start # start them all + ### if amount of chiefs (including ones without a container) is greater than the amount of existing chief containers + elif [ $allChiefsAvailable -gt $allChiefContainers ]; then + orangeecho "turning up chiefs as there are running docker containers ..." + sudo docker-compose -p "${projectname}" -f "docker-compose.yml" up --build --detach # wire up the missing chiefs + fi +fi \ No newline at end of file diff --git a/viewlog b/viewlog index 60563b1..71c5bb6 100755 --- a/viewlog +++ b/viewlog @@ -1,39 +1,72 @@ #!/bin/bash # makes sense for production only +# ./viewlog [] -source ./lib/colorful +args=( $@ ) -blueecho "performing some checks & normalizations ..." -source ./lib/validation +source ./lib/colorful +source ./lib/utils +projectname=`basename "$PWD"` +# ./viewlog [] [] [] +# .............. +envfile="${args[0]/'.env'/''}.env" +if [ -f "./config/$envfile" ]; then # export the variables in the city configuration file + blueecho "performing some checks & normalizations ..." + source ./lib/validation +fi +# ./viewlog [] +# ............ modulename="${args[0]}" -# remove elem on first index -unset args[0] -args=( `echo "${args[@]}"` ) +removeFirstElemInArgs +if [ -n "$city" ]; then + if ! [ -d "modules/$modulename" ]; then + redecho "Error: You specified an invalid name of a module to read log from">&2 + echo "A list of modules available:" + ls -p ./modules | grep -v / + fi +fi -containername="$modulename-$city" -moreargs="false" -projectname=`basename "$PWD"` +# ./viewlog [] +# ............. +servicename="${args[0]}" +removeFirstElemInArgs + +## remove the ''' prefix from the console input '' if the user has entered it. The prefix will be automatically added later +serviceprefix="${modulename}-" +servicename=${servicename/$serviceprefix/""} -if [ -z "$modulename" ]; then - echo 'Error: extension name as param is needed (e.g. "tileserver")' >&2 - exit 1 +if [ -n "$city" ]; then + cd "modules/$modulename" + allServices=`sudo docker-compose -p "$projectname" -f "${city}.yml" ps --services` + cd "../../" +else + serviceame="${args[0]}" + removeFirstElemInArgs + allServices=`sudo docker-compose -p "$projectname" -f "docker-compose.yml" ps --services` fi - -if [ ${#args[@]} -gt 0 ]; then - moreargs="true" +if [ -z `echo $allServices | grep $servicename` ]; then + redecho "Error: You specified an invalid name of a service in module '$modulename' to read log from">&2 + echo "A list of services in that module available:" + echo "$allServices" fi -if [ "$moreargs" = "false" ]; then - args+=( --since today --follow ) -fi +# ./viewlog [] +# ............ +if [ ${#args[@]} -eq 0 ]; then + args+=( "--since today --follow" ) +fi arguments="${args[@]}" -# determining right container name -containername="${projectname}_${containername}" -containername=`sudo docker container ls --format "{{.Names}}"` -containername=`echo $containername | grep $containername` +# determining full qualified container name +containername="${projectname}_${modulename}-${servicename}" +if [ -n "$city" ]; then + containername+="-${city,,}" +fi +namesOfAllContainers=`sudo docker container ls --all --format "{{.Names}}"` +containername=`echo "$namesOfAllContainers" | grep $containername` +# Executing journalctl command to get the logs orangeecho "Executing 'sudo journalctl CONTAINER_NAME=\"${containername}\" $arguments'" sudo journalctl CONTAINER_NAME="${containername}" $arguments diff --git a/workon b/workon index 748e970..e44387b 100755 --- a/workon +++ b/workon @@ -16,7 +16,7 @@ workon_optionalvariables=( "email" ) if [ -z "$enfile" ] && [ -z "$1" ]; then redecho "No city file to load found. \033[0;mExecute '\033[0;34m. ./workon \033[0;m'" echo "List of available city files:" - dir config + ls config return fi @@ -58,7 +58,7 @@ if ! [ -z "$cityfile" ]; then fi orangeecho "modifying prompt ..." -export PS1="\033[0;32m$city@$env\033[0;34m (MODE '$curmode')\033[0;m $ " +export PS1="\033[0;32m$city@ssl='$ssl'\033[0;34m (MODE '$curmode')\033[0;m $ " orangeecho "adding command aliases ..." alias add="./add_module" alias remove="./remove_module" @@ -85,12 +85,12 @@ echo '-------------------------------------------------------------------------- echo echo -e "\033[1;34mCommand list:\033[0;m" echo -e "\033[0;34mclose\033[0;m get the prompt and the environment reverted back." -echo -e "\033[0;34madd\033[0;m alias to \033[0;36m./add_module\033[0;m" -echo -e "\033[0;34mremove\033[0;m alias to \033[0;36m./remove_module\033[0;m" -echo -e "\033[0;34mserver\033[0;m alias to \033[0;36m./server\033[0;m" -echo -e "\033[0;34mviewlog\033[0;m alias to \033[0;36m./viewlog\033[0;m" -echo -e "\033[0;34mrestart\033[0;m alias to \033[0;36m./restart_module\033[0;m" -echo -e "\033[0;34mworkon\033[0;m alias to \033[0;36m. ./workon\033[0;m (this script)" +echo -e "\033[0;34madd\033[0;m alias to \033[0;36m./add_module \"$city\"\033[0;m" +echo -e "\033[0;34mremove\033[0;m alias to \033[0;36m./remove_module \"$city\"\033[0;m" +echo -e "\033[0;34mserver\033[0;m alias to \033[0;36m./server \"$city\"\033[0;m" +echo -e "\033[0;34mviewlog\033[0;m alias to \033[0;36m./viewlog \"$city\"\033[0;m" +echo -e "\033[0;34mrestart\033[0;m alias to \033[0;36m./restart_module \"$city\"\033[0;m (deprecated)" +echo -e "\033[0;34mworkon\033[0;m alias to \033[0;36m. ./workon \"$city\"\033[0;m (this script)" echo "other commands work as expected" echo if [ -n "$errors" ]; then