Merge pull request #2 from trufi-association/develop

Prepare v2.1.0: Static rising

.gitignore

@@ -1,4 +1,4 @@
-trufi-server-modules
+trufi-*-modules
 modules
 data/nginx/interweb
 data/nginx/intraweb
@@ -7,6 +7,8 @@ data/nginx/inc/*.pem
 data/logs
 data/letsencrypt
 data/maps
+data/www
 data/certbot
 .vscode
 test
+.directory

README.md

@@ -1,15 +1,11 @@
-# trufi-server-cities
+# Trufi Multi-Instance Server
 
-A repository as a recipe to create your own production backend city for multiple Trufi Apps. This only contains the services which actually **consume & serve but <u>do not create</u>**
+This is a structure build to host services following our `modules` convention. It allows of tearing up and down modules and their services easily. The following repositories have been created by us following our convention so these are fully compatible to this structure:
 
-- the search index (consumed by module `photon`)
-- the mbtile (map tiles for your region) (consumed by module `tileserver`)
-- the static png tiles for your region) (consumed by module `static_maps`)
-- the routing graph (consumed by module `otp` - **O**pen**T**rip**P**lanner )
-
-and of course Nginx which combines these modules to make them appear as *one* with *one* HTTPS certificate, web identity and url scheme.
-
-If you actually need to create the stuff e.g. the mbtiles or the graph you better go to [Trufi Server Resources](https://github.com/trufi-association/trufi-server-resources).
+- [Trufi Server Modules](https://github.com/trufi-association/trufi-server-modules)
+  Use this to host services necessary for apps built on our Trufi Core. These modules are used by our famous and very first [Trufi App](https://trufi.app).
+- [Trufi Website Modules](https://github.com/trufi-association/trufi-website-modules)
+  Use this to host services necessary to have a project website to show potential users your app build on our Trufi Core.
 
 **We use docker but not to make this cross-platform. This is intended to be used on Linux. We only use docker to make setups equal, easy for all admins and their systems, for security and troubleshooting concerns.**
 
@@ -27,15 +23,15 @@ At last we introduce "chiefs" which are all services specified in the `docker-co
 
 ### City
 
-The config files are located inside `./config` folder, you can create a new one providing your own variables:
+The city config files are located inside `./config` folder, you can create a new one providing your own variables:
 
-| variable   | example            | description                                                  |
-| ---------- | ------------------ | ------------------------------------------------------------ |
-| city       | Bolivia-Cochabamba | Just `Country-City` name                                     |
-| domain     | cbba.trufi.app     | The domain name of the city. If you use the mode `virtual domains` then be creative as this variable will then not be of use but needs to be available |
-| otpversion | 1.5.0              | Put there `1.5.0` for regions having PTv1 schema in OSM otherwise `2.0.0` |
+| variable                  | example                   | description                                                  |
+| ------------------------- | ------------------------- | ------------------------------------------------------------ |
+| city                      | Bolivia-Cochabamba        | Just `Country-City` name (required)                          |
+| domain                    | cbba.trufi.app            | The domain name of the city. If you use the mode `virtual domains` then be creative as this variable will then not be of use but needs to be available |
+| *... (depends on module)* | *... (depends on module)* | *... (depends on module)*                                    |
 
-Create a new one based on the already existing config files to get an idea of their internal structure. Do that for each city you want to host backend services for. *You may want to remove the other configuration city files which are meant to provide examples.*
+Create a new one based on the already existing config files to get an idea of their internal structure. Do that for each city you want to host backend services for. *You may want to remove the other configuration city files which are meant to provide examples.* If there is only one city configuration file ending with `.env` left then you can use the commands without the `<name of city>` argument e.g. `add_module tileserver` or `server up tileserver`. But as soon as there are more than one then you need to specify `<name of city>` of course.
 
 ### Global
 
@@ -53,12 +49,23 @@ For other but important parameters which apply globally to all cities, we use a
 
 You can find all in the [modules](./modules) folder. Each module has a README file with more detailed info.
 
-In order to add more modules compatible to [trufi-server-modules](https://github.com/trufi-association/trufi-server-modules) like [tsm-locaco](https://github.com/trufi-association/tsm-locaco) you just have to do a `git clone` inside the `module` folder, to execute `modifyComposes.py` and to work with it as you did with the other modules pre-installed.
+In order to add more modules compatible to [trufi-server-modules](https://github.com/trufi-association/trufi-server-modules) like [tsm-locaco](https://github.com/trufi-association/tsm-locaco) you just have to do a `git clone` inside the `modules` folder, to execute `modifyComposes.py` and to work with it as you did with the other modules pre-installed.
+
+Read more about [including external modules](./docs/extend.md#external_modules)
 
 ## Setup & Maintenance
 
+### Initiate your own instance of this
+
+1. Clone of course
+2. Rename the cloned repo `trufi-server-multi` differently as you want to have a custom name describing what your clone of this structure is for. You might want to rename the folder to `website-services` to host websites and web platforms such as dashboards for your customers. Or `city-services` for services related to city governance.
+3. Clone a `modules` repository into this one e.g. https://github.com/trufi-association/trufi-website-modules or https://github.com/trufi-association/trufi-server-modules
+4. Execute `./init` and answer the setup questions
+
 ### Copying things over
 
+**TODO:** Move somewhere else
+
 Definitely you used our tools in the [Trufi Server Resources](https://github.com/trufi-server-resources) you generate all the data you need for the backend tools here. Excellent because the structure there is to 100% compactible to this one and you don't need to figure out how to copy/move things over to here. See also the concept of [Resource Binding](https://github.com/trufi-association/trufi-server-resources#main-output-folder) we introduced. But anyway now come the smart people to play and you can be one of them.
 
 To copy over the files to their appropriate location here you just need to do the following: In your very own copy of [Trufi Server Resources](https://github.com/trufi-server-resources) go to its `data` directory and copy the content of the `<Country-City>` which holds your data. Paste them into the `modules` folder of this one and accept the merge with already existing data. Or to put it into the words/world of sysadmins just execute
@@ -75,22 +82,26 @@ Then rename all `data` folders inside the modules to `data_Country-City` e.g. `d
 
 We hide all (optional) services behind a nginx proxy which **can** handle the encryption for them. To make encrypted connections to that nginx proxy possible we need to issue a HTTPS certificate. Luckily there is *Let’s Encrypt* issuing them to anyone without the need to pay. This backend relies on the SSL/TLS certificate management infrastructure of the docker host (your linux system). There are multiple ways connecting the dots.
 
-Our docker `cief-nginx` by default listens on the host port `8290` for HTTP and on `8300` for HTTPS requests.
+Our docker `cief-nginx` by default listens on the host port `8290` for HTTP and on `8300` for HTTPS requests. See [changing ports of chief-nginx](./docs/extend.md#changing_ports_of_chief-nginx) if you want to change that.
 
-#### Docker `chief-nginx` has its own HTTPS certificate store and runs on a different port
+#### An HTTP server already exists on your system
 
-This requires the `ssl` variable in `./data/instance.conf` set to `yes` before any execution of `add_module`. If you accidentally run `add_module` before doing that then use `remove_module` to remove the configuration files as `remove_module` is the opposite of `add_module`.
+This requires the `ssl` variable in `./data/instance.conf` set to `yes` before any execution of `add_module`. If you accidentally run `add_module` before doing that then use `remove_module` to remove the nginx configuration files as `remove_module` is the opposite of `add_module`.
 
-The docker container `nginx` in our provided configuration expects to find the necessary files needed to answer https requests in the host path `./data/certbot/conf` which is in its schematics equal to the well known [/etc/letsencrypt](https://eff-certbot.readthedocs.io/en/stable/using.html#where-certs). Of course this behaviour can be changed that the nginx docker container sees `/etc/letsencrypt` of your host directly although not recommended. This just adds one more security risk to manage. If you care about security you just put the files in `./data/certbot/conf`  you need for running this docker infrastructure. To put it in other words: If you have the following domains `example.com` , `example.uk` and `example.org` . The first domain `example.com` is used by your web server running on the host linux system and the others two are used by the `nginx` running in this docker infrastructure then just put the necessary files for HTTPS on `example.org` and `example.com` into  `./data/certbot/conf` . This way you have `/etc/letsencrypt` on your host for the HTTPS server running on your host system. The other one `./data/certbot/conf` for the HTTPS server running for this infrastructure.
+In any case you need to run `openssl dhparam -out ./data/nginx/inc/dhparam.pem 4096` after executing `init` to generate the dhparam file.
 
-In that case you need to run `openssl dhparam -out ./data/nginx/inc/dhparam.pem 4096` after executing `init` to generate the dhparam file.
+The docker container `chief-nginx` in our provided configuration expects to find the necessary files needed to answer https requests in the host path `./data/certbot/conf` which is in its schematics equal to the well known [/etc/letsencrypt](https://eff-certbot.readthedocs.io/en/stable/using.html#where-certs). Of course this behaviour can be changed that the nginx docker container sees `/etc/letsencrypt` of your host directly although not recommended. This just adds one more security risk to manage. If you care about security you just put the files in `./data/certbot/conf`  you need for running this docker infrastructure. To put it in other words: If you have the following domains `example.com` , `example.uk` and `example.org` . The first domain `example.com` is used by your web server listening to port 80 and the others two are used by the `nginx` running in this docker infrastructure then just put the necessary files for HTTPS on `example.uk` and `example.org` into  `./data/certbot/conf` . This way you have `/etc/letsencrypt` on your host for the HTTPS server running on your host system serving ` example.com`. The other one `./data/certbot/conf` for the HTTPS server running for this infrastructure serving `example.uk` and `example.org`.
+
+This is the setup on our own server and we provide a [working example of our nginx configuration for Let's Encrypt stuff](./docs/command/certify.md#example_nginx_configuration) you can use as an inspiration on how to do that thing.
 
 #### Central HTTPS server on your host system
 
 This requires the `ssl` variable in `./data/instance.conf` set to `no` before any execution of `add_module`. If you accidentally run `add_module` before doing that then use `remove_module` to remove the configuration files as `remove_module` is the opposite of `add_module`.
 
 This is useful if you are already providing other services in need of HTTPS to your customers e.g. you are already hosting a website. Now you want to have your server provide different services for different cities. Also this is your only option if your server is behind a firewall just letting port 80 and 443 pass through. Go setup nginx on your host which then does all the HTTPS stuff and takes care to redirect to the appropriate sub webservers based on specified parameters you have defined. This allows you to use this structure without any HTTPS configuration ( `ssl="no"` ) because encryption is handled by your HTTPS server on your host.
 
+See an [example nginx configuration only used for Let's Encrypt](./docs/commands/certify.md#example_nginx_configuration)
+
 #### Get HTTPS certificate (independent from your infrastructure)
 
 We created a highly flexible script to ease HTTPS certificate creation and it accounts for:

add_module

@@ -2,24 +2,28 @@
 
 projectname=`basename "$PWD"`
 
+source ./lib/singlenv
 source ./lib/colorful
 source ./lib/env
 
 blueecho "1. perform some checks & normalizations ..."
 source ./lib/validation
 
-addModule() {
-	local modulename="$1"
-	blueecho "2. add module '$modulename' to city '$city'"
+compileCompose() {
+	blueecho "Add module '$modulename' to city '$city'"
+
+	if ! [ -f "$moduleDir/docker-compose.yml" ]; then
+		echo "   doing this makes no sense as the template file '$moduleDir/docker-compose.yml' does not exist!"
+		return
+	fi
 
 	# 1. Generate a docker-compose for the city (if not already existing)
 	ymlFile="$moduleDir/$city.yml"
 	ymlFileInactive="$ymlFile.inactive"
 
 	if ! [ -f "$ymlFileInactive" ] && ! [ -f "$ymlFile" ]; then
 		orangeecho "   building & adding in directory '$moduleDir' ) ..."
-		export city_normalize="${city,,}"
-		(export `cat ./$cityfile | xargs`; export projectname="$projectname" 
+		(export `cat ./$cityfile | xargs`; export projectname="$projectname" ; export city_normalize="${city,,}"
 		envsubst < $moduleDir/docker-compose.yml > "$ymlFile" )
 		if ! [ -f "$ymlFile" ]; then
 			redecho "     Creating & adding operation failed! Do I have write access to '$moduleDir'?">&2
@@ -38,9 +42,11 @@ addModule() {
 	else
 		greenecho "   already added. No need to do it again :)"
 	fi
+}
 
+interwebConfiguration() {
 	# 2. Prepare interweb configuration
-	blueecho "3. add interweb nginx configuration"
+	blueecho "Add interweb nginx configuration"
 	orangeecho "   adding nginx configuration for '$modulename' in city '$city' ..."
 
 	# This folder to be created somewhere under './data' is necessary to be able to store all nginx reverse proxy configurations for '$city' e.g. 'Germany-Hamburg'
@@ -51,16 +57,21 @@ addModule() {
 	nginxcitymoduleconf="$nginxcityfolder/$modulename.conf" # e.g. './data/nginx/interweb/Germany-Hamburg/tileserver.conf'
 	# This file ensures that traffic redirection to that module works
 	modulenginxconf="$moduleDir/nginx.conf" # e.g. './modules/tileserver/nginx.conf'
-	
+
 	# if a nginx reverse proxy configuration exists for this module
 	if [ -f "$modulenginxconf" ]; then # read it into memory (to be able to alter it later)
 		modulenginxconf_content=`cat "$modulenginxconf"`
+		modulenginxconf_content=`
+export \`cat $cityfile | xargs\`
+export projectname="$projectname"
+export modulename="$modulename"
+export city_normalize="${city,,}"
+echo "$modulenginxconf_content" | envsubst`
 	else
 		# if no module nginx reverse configuration exists then abbort
-		redecho "     Error: there is no nginx configuration ('nginx.conf') for module '$modulename' available! Fix this and run this script again"
-		exit 1
+		return
 	fi
-	
+
 	# If that folder does not exist
 	if ! [ -d "$nginxcityfolder" ]; then # create it
 		orangeecho "     creating nginx server configuration folder for city '$city' ..."
@@ -81,7 +92,7 @@ addModule() {
 	fi
 
 	# 4. Generating nginx module configuration
-	orangeecho "     copying & altering nginx module configuration for module '$modulename' to nginx server configuration for city '$city' (overwriting if already existing) ..."
+	orangeecho "     copying & altering nginx module configuration for module '$modulename' to nginx server configuration for city '$city' ..."
 	if ! [ -f "$nginxcitymoduleconf" ]; then
 		# search for pattern 'http://otp:' or 'http://otp;' or 'http://otp/'
 		# and replace it with 'http://otp-$city:' or 'http://otp-$city;' or 'http://otp-$city/'
@@ -99,18 +110,28 @@ print(re.sub(\"http\:/\/(.*?)(:|;|\/)\",\"http://${modulename}-\\\\\\1-${city,,}
 	else
 		greenecho "       already generated. No need to do it again :)"
 	fi
-
+}
+
+prepareLogging() {
 	# 5. Logging
 	orangeecho "   creating logging structure ..."
 	if ! [ -d "./data/logs/nginx/$domain" ]; then
 		mkdir -p "./data/logs/nginx/$domain" --verbose
 	else
 		greenecho "     not necessary as already existing :)"
 	fi
+}
 
+intrawebConfiguration() {
 	# 6. Intraweb configuration
 	if [ "$intraweb" = "yes" ]; then
-		blueecho "4. add intraweb nginx configuration ..."
+		blueecho "Add intraweb nginx configuration ..."
+
+		if ! [ -f "$moduleDir/docker-compose.yml" ]; then
+			echo "   doing this makes no sense as the template file '$moduleDir/docker-compose.yml' does not exist!"
+			return
+		fi
+
 		orangeecho "   adding nginx configuration for '$modulename' in city '$city' ..."
 
 		nginxcityfolder="./data/nginx/intraweb/$city"
@@ -123,7 +144,7 @@ print(re.sub(\"http\:/\/(.*?)(:|;|\/)\",\"http://${modulename}-\\\\\\1-${city,,}
 
 		orangeecho "     completing nginx configuration for city '$city' ..."
 		if ! [ -f "$nginxcityconf" ]; then
-			echo "include /etc/nginx/intraweb/$city/*/*.conf ;" > "$nginxcityconf"
+			echo "include /etc/nginx/intraweb/$city/*.conf ;" > "$nginxcityconf"
 		else
 			greenecho "     not necessary as already completed :)"
 		fi
@@ -148,14 +169,23 @@ print(re.sub(\"http\:/\/(.*?)(:|;|\/)\",\"http://${modulename}-\\\\\\1-${city,,}
 			fi
 		done
 	fi
+}
+
+addModule() {
+	modulename="$1"
+
+	compileCompose
+	interwebConfiguration
+	prepareLogging
+	intrawebConfiguration
 
 	source "lib/plugin"
 	invokeAllPluginsOf "add_module"
 
-	blueecho "5. Wiring up module '$modulename' for city '$city'"
+	blueecho "Wiring up module '$modulename' for city '$city'"
 	./server "${city}" up "$modulename"
 
-	greenecho "added trufi module '$modulename' to city '$city'"
+	greenecho "Added trufi module '$modulename' to city '$city'!"
 }
 
 moduleNotFound=""
@@ -177,4 +207,4 @@ if ! [ -z "$moduleNotFound" ]; then
 	echo "A list of modules available:"
 	ls -p ./modules | grep -v /
 	exit 1
-fi
+fi

docker-compose.yml

@@ -1,24 +1,21 @@
 version: '3'
 services:
-
   chief-nginx:
-    image: nginx #:1.15-alpine
+    image: nginx
     ports:
-      - "8290:80" # HTTP
-      - "8300:443" # HTTPS
-      - "127.0.0.1:8090:8090" # intraweb port.
+    - 8290:80
+    - 8300:443
+    - 127.0.0.1:8090:8090
     volumes:
-      - ./data/nginx/interweb:/etc/nginx/interweb:ro
-      - ./data/nginx/intraweb:/etc/nginx/intraweb:ro
-      - ./data/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
-      - ./data/nginx/inc:/etc/nginx/inc:ro
-      - ./data/logs/nginx:/var/log/nginx/
-      - ./data/letsencrypt/config:/etc/letsencrypt
-      #- ./data/certbot/www:/var/www/certbot
-      - ./data/nginx/www:/var/www/:ro
-    command: "nginx -g \"daemon off;\""
+    - ./data/nginx/interweb:/etc/nginx/interweb:ro
+    - ./data/nginx/intraweb:/etc/nginx/intraweb:ro
+    - ./data/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+    - ./data/nginx/inc:/etc/nginx/inc:ro
+    - ./data/logs/nginx:/var/log/nginx/
+    - ./data/letsencrypt/config:/etc/letsencrypt
+    - ./data/nginx/www:/var/www/html:ro
+    command: nginx -g "daemon off;"
     restart: unless-stopped
-
 networks:
   default:
     name: trufi-server-multi