diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index b2767349..6a03007a 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -9,10 +9,9 @@ jobs: name: Deploy docs runs-on: ubuntu-latest steps: - - name: Checkout master - uses: actions/checkout@v1 - - - name: Deploy docs - uses: mhausenblas/mkdocs-deploy-gh-pages@master - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + - uses: actions/checkout@v2 + - uses: actions/setup-python@v2 + with: + python-version: 3.x + - run: pip3 install -r requirements-mkdocs.txt + - run: mkdocs gh-deploy --force diff --git a/.templates/wireguard/use-container-dns.sh b/.templates/wireguard/use-container-dns.sh index 18737322..7f13f219 100644 --- a/.templates/wireguard/use-container-dns.sh +++ b/.templates/wireguard/use-container-dns.sh @@ -1,8 +1,9 @@ # Forward DNS requests from remote WireGuard clients to the default # gateway on the internal bridged network that the WireGuard container -# is attached to. This results in queries being sent to any other -# container on the same internal bridged network that is listening -# on port 53 (eg PiHole, AdGuardHome or bind9). +# is attached to. The gateway routes queries out from the bridged network to +# the host's network. This results in queries being sent to any daemon or +# container that is listening on host port 53 (eg PiHole, AdGuardHome, dnsmasq +# or bind9). # # Acknowledgement: @ukkopahis diff --git a/docs/Accessing-your-Device-from-the-internet.md b/docs/Basic_setup/Accessing-your-Device-from-the-internet.md similarity index 95% rename from docs/Accessing-your-Device-from-the-internet.md rename to docs/Basic_setup/Accessing-your-Device-from-the-internet.md index 405d09ee..fca16019 100644 --- a/docs/Accessing-your-Device-from-the-internet.md +++ b/docs/Basic_setup/Accessing-your-Device-from-the-internet.md @@ -6,14 +6,14 @@ From time to time the IP address that your ISP assigns changes and it's difficul Secondly, how do you get into your home network? Your router has a firewall that is designed to keep the rest of the internet out of your network to protect you. The solution to that is a Virtual Private Network (VPN) or "tunnel". -## Dynamic DNS +## Dynamic DNS There are two parts to a Dynamic DNS service: 1. You have to register with a Dynamic DNS service provider and obtain a domain name that is not already taken by someone else. 2. Something on your side of the network needs to propagate updates so that your chosen domain name remains in sync with your router's dynamically-allocated public IP address. -### Register with a Dynamic DNS service provider +### Register with a Dynamic DNS service provider The first part is fairly simple and there are quite a few Dynamic DNS service providers including: @@ -24,7 +24,7 @@ The first part is fairly simple and there are quite a few Dynamic DNS service pr Some router vendors also provide their own built-in Dynamic DNS capabilities for registered customers so it's a good idea to check your router's capabilities before you plough ahead. -### Dynamic DNS propagation +### Dynamic DNS propagation The "something" on your side of the network propagating WAN IP address changes can be either: @@ -39,7 +39,7 @@ A behind-the-router technique usually relies on sending updates according to a s > This seems to be a problem for DuckDNS which takes a beating because almost every person using it is sending an update bang-on five minutes. -### DuckDNS client +### DuckDNS client IOTstack provides a solution for DuckDNS. The best approach to running it is: @@ -99,7 +99,7 @@ A null result indicates failure so check your work. Remember, the Domain Name System is a *distributed* database. It takes *time* for changes to propagate. The response you get from directing a query to ns1.duckdns.org may not be the same as the response you get from any other DNS server. You often have to wait until cached records expire and a recursive query reaches the authoritative DuckDNS name-servers. -#### Running the DuckDNS client automatically +#### Running the DuckDNS client automatically The recommended arrangement for keeping your Dynamic DNS service up-to-date is to invoke `duck.sh` from `cron` at five minute intervals. @@ -152,7 +152,7 @@ $ cat /dev/null >~/Logs/duck.log ### WireGuard -WireGuard is supplied as part of IOTstack. See [WireGuard documentation](https://sensorsiot.github.io/IOTstack/Containers/WireGuard.html). +WireGuard is supplied as part of IOTstack. See [WireGuard documentation](../Containers/WireGuard.md). ### PiVPN diff --git a/docs/Backup-and-Restore.md b/docs/Basic_setup/Backup-and-Restore.md similarity index 99% rename from docs/Backup-and-Restore.md rename to docs/Basic_setup/Backup-and-Restore.md index 3dae4ab1..48de688a 100644 --- a/docs/Backup-and-Restore.md +++ b/docs/Basic_setup/Backup-and-Restore.md @@ -8,6 +8,7 @@ The backup command can be executed from IOTstack's menu, or from a cronjob. To ensure that all your data is saved correctly, the stack should be brought down. This is mainly due to databases potentially being in a state that could cause data loss. There are 2 ways to run backups: + * From the menu: `Backup and Restore` > `Run backup` * Running the following command: `bash ./scripts/backup.sh` @@ -21,6 +22,7 @@ The current directory of bash must be in IOTstack's directory, to ensure that it ``` ./scripts/backup.sh {TYPE=3} {USER=$(whoami)} ``` + * Types: * 1 = Backup with Date * A tarball file will be created that contains the date and time the backup was started, in the filename. @@ -33,10 +35,12 @@ The current directory of bash must be in IOTstack's directory, to ensure that it If this parameter is not supplied when run as root, the script will ask for the username as input Backups: + * You can find the backups in the ./backups/ folder. With rolling being in ./backups/rolling/ and date backups in ./backups/backup/ * Log files can also be found in the ./backups/logs/ directory. ### Examples: + * `./scripts/backup.sh` * `./scripts/backup.sh 3` @@ -52,6 +56,7 @@ This will only produce a backup in the rollowing folder and change all the permi ## Restore There are 2 ways to run a restore: + * From the menu: `Backup and Restore` > `Restore from backup` * Running the following command: `bash ./scripts/restore.sh` @@ -64,6 +69,7 @@ There are 2 ways to run a restore: ./scripts/restore.sh {FILENAME=backup.tar.gz} {noask} ``` The restore script takes 2 arguments: + * Filename: The name of the backup file. The file must be present in the `./backups/` directory, or a subfolder in it. That means it should be moved from `./backups/backup` to `./backups/`, or that you need to specify the `backup` portion of the directory (see examples) * NoAsk: If a second parameter is present, is acts as setting the no ask flag to true. diff --git a/docs/Custom.md b/docs/Basic_setup/Custom.md similarity index 97% rename from docs/Custom.md rename to docs/Basic_setup/Custom.md index 8879f9ad..03a3508c 100644 --- a/docs/Custom.md +++ b/docs/Basic_setup/Custom.md @@ -125,7 +125,7 @@ services: environment: ``` -This will remove the default environment variables set in the template, and tell docker-compose to use the variables specified in your file. It is not mandatory that the *.env file be placed in the service's service directory, but is strongly suggested. Keep in mind the [PostBuild Script](https://sensorsiot.github.io/IOTstack/PostBuild-Script) functionality to automatically copy your *.env files into their directories on successful build if you need to. +This will remove the default environment variables set in the template, and tell docker-compose to use the variables specified in your file. It is not mandatory that the *.env file be placed in the service's service directory, but is strongly suggested. Keep in mind the [PostBuild Script](../Developers/PostBuild-Script.md) functionality to automatically copy your *.env files into their directories on successful build if you need to. ### Adding custom services diff --git a/docs/Default-Configs.md b/docs/Basic_setup/Default-Configs.md similarity index 98% rename from docs/Default-Configs.md rename to docs/Basic_setup/Default-Configs.md index 6ef88715..51b05314 100644 --- a/docs/Default-Configs.md +++ b/docs/Basic_setup/Default-Configs.md @@ -1,4 +1,4 @@ -# Build Stack Default Passwords for Services +# Default Passwords and ports Here you can find a list of the default configurations for IOTstack for quick referece. diff --git a/docs/Docker-commands.md b/docs/Basic_setup/Docker-commands.md similarity index 100% rename from docs/Docker-commands.md rename to docs/Basic_setup/Docker-commands.md diff --git a/docs/How-the-script-works.md b/docs/Basic_setup/How-the-script-works.md similarity index 100% rename from docs/How-the-script-works.md rename to docs/Basic_setup/How-the-script-works.md diff --git a/docs/Misc.md b/docs/Basic_setup/Misc.md similarity index 100% rename from docs/Misc.md rename to docs/Basic_setup/Misc.md diff --git a/docs/Native-RTL_433.md b/docs/Basic_setup/Native-RTL_433.md similarity index 100% rename from docs/Native-RTL_433.md rename to docs/Basic_setup/Native-RTL_433.md diff --git a/docs/Networking.md b/docs/Basic_setup/Networking.md similarity index 100% rename from docs/Networking.md rename to docs/Basic_setup/Networking.md diff --git a/docs/RPIEasy_native.md b/docs/Basic_setup/RPIEasy_native.md similarity index 100% rename from docs/RPIEasy_native.md rename to docs/Basic_setup/RPIEasy_native.md diff --git a/docs/Understanding-Containers.md b/docs/Basic_setup/Understanding-Containers.md similarity index 100% rename from docs/Understanding-Containers.md rename to docs/Basic_setup/Understanding-Containers.md diff --git a/docs/Basic_setup/What-is-sudo.md b/docs/Basic_setup/What-is-sudo.md new file mode 100644 index 00000000..9f1a50bf --- /dev/null +++ b/docs/Basic_setup/What-is-sudo.md @@ -0,0 +1,47 @@ +# What is sudo? + +Many first-time users of IOTstack get into difficulty by misusing the `sudo` command. The problem is best understood by example. In the following, you would expect `~` (tilde) to expand to `/home/pi`. It does: + +```bash +$ echo ~/IOTstack +/home/pi/IOTstack +``` + +The command below sends the same `echo` command to `bash` for execution. This is what happens when you type the name of a shell script. You get a new instance of `bash` to run the script: + +```bash +$ bash -c 'echo ~/IOTstack' +/home/pi/IOTstack +``` + +Same answer. Again, this is what you expect. But now try it with `sudo` on the front: + +```bash +$ sudo bash -c 'echo ~/IOTstack' +/root/IOTstack +``` + +Different answer. It is different because `sudo` means "become root, and then run the command". The process of becoming root changes the home directory, and that changes the definition of `~`. + +Any script designed for working with IOTstack assumes `~` (or the equivalent `$HOME` variable) expands to `/home/pi`. That assumption is invalidated if the script is run by `sudo`. + +Of necessity, any script designed for working with IOTstack will have to invoke `sudo` **inside** the script **when it is required**. You do not need to second-guess the script's designer. + +Please try to minimise your use of `sudo` when you are working with IOTstack. Here are some rules of thumb: + +1. Is what you are about to run a script? If yes, check whether the script already contains `sudo` commands. Using `menu.sh` as the example: + + ```bash + $ grep -c 'sudo' ~/IOTstack/menu.sh + 28 + ``` + + There are numerous uses of `sudo` within `menu.sh`. That means the designer thought about when `sudo` was needed. + +2. Did the command you **just executed** work without `sudo`? Note the emphasis on the past tense. If yes, then your work is done. If no, and the error suggests elevated privileges are necessary, then re-execute the last command like this: + + ```bash + $ sudo !! + ``` + +It takes time, patience and practice to learn when `sudo` is **actually** needed. Over-using `sudo` out of habit, or because you were following a bad example you found on the web, is a very good way to find that you have created so many problems for yourself that will need to reinstall your IOTstack. *Please* err on the side of caution! diff --git a/docs/Getting-Started.md b/docs/Basic_setup/index.md similarity index 78% rename from docs/Getting-Started.md rename to docs/Basic_setup/index.md index 9c7223f8..75b589c0 100644 --- a/docs/Getting-Started.md +++ b/docs/Basic_setup/index.md @@ -1,6 +1,6 @@ # Getting Started -## introduction to IOTstack - videos +## introduction to IOTstack - videos Andreas Spiess Video #295: Raspberry Pi Server based on Docker, with VPN, Dropbox backup, Influx, Grafana, etc: IOTstack @@ -10,27 +10,25 @@ Andreas Spiess Video #352: Raspberry Pi4 Home Automation Server (incl. Docker, O [![#352 Raspberry Pi4 Home Automation Server (incl. Docker, OpenHAB, HASSIO, NextCloud)](http://img.youtube.com/vi/KJRMjUzlHI8/0.jpg)](https://www.youtube.com/watch?v=KJRMjUzlHI8) -## assumptions +## Assumptions IOTstack makes the following assumptions: 1. Your hardware is a Raspberry Pi (typically a 3B+ or 4B). - Note: - * The Raspberry Pi Zero W2 has been tested with IOTstack. It works but the 512MB RAM means you should not try to run too many containers concurrently. + * Users have also [reported success + ](https://github.com/SensorsIot/IOTstack/issues/375) on Orange Pi + Win/Plus. -2. Your Raspberry Pi has a reasonably-recent version of 32-bit Raspberry Pi OS (aka "Raspbian") installed. You can download operating-system images: +2. Your Raspberry Pi has a reasonably-recent version of 32-bit or 64-bit Raspberry Pi OS (aka "Raspbian") installed. You can download operating-system images: * [Current release](https://www.raspberrypi.com/software/operating-systems/) + : "Raspberry Pi OS with desktop" is recommended. * [Prior releases](http://downloads.raspberrypi.org/raspios_armhf/images/) + : This offers only "Raspberry Pi OS with desktop" images. - Note: - - * If you use the first link, "Raspberry Pi OS with desktop" is recommended. - * The second link only offers "Raspberry Pi OS with desktop" images. - -3. Your operating system has been kept up-to-date with: +3. Your operating system has been updated: ```bash $ sudo apt update @@ -46,15 +44,9 @@ If the first three assumptions hold, assumptions four through six are Raspberry Please don't read these assumptions as saying that IOTstack will not run on other hardware, other operating systems, or as a different user. It is just that IOTstack gets most of its testing under these conditions. The further you get from these implicit assumptions, the more your mileage may vary. -### other platforms - -Users have reported success on other platforms, including: - -* [Orange Pi WinPlus](https://github.com/SensorsIot/IOTstack/issues/375) +## New installation -## new installation - -### automatic (recommended) +### automatic (recommended) 1. Install `curl`: @@ -82,7 +74,7 @@ Users have reported success on other platforms, including: $ docker-compose up -d ``` -### manual +### manual 1. Install `git`: @@ -122,21 +114,17 @@ Users have reported success on other platforms, including: $ docker-compose up -d ``` -### scripted +### scripted If you prefer to automate your installations using scripts, see: * [Installing Docker for IOTstack](https://gist.github.com/Paraphraser/d119ae81f9e60a94e1209986d8c9e42f#scripting-iotstack-installations). -## migrating from the old repo (gcgarner)? - -If you are still running on gcgarner/IOTstack and need to migrate to SensorsIot/IOTstack, see: +## Required system patches -* [Migrating IOTstack from gcgarner to SensorsIot](./gcgarner-migration.md). +Unless you know what you are doing, assume these are needed. -## recommended system patches - -### patch 1 – restrict DHCP +### patch 1 – restrict DHCP Run the following commands: @@ -147,7 +135,7 @@ $ sudo reboot See [Issue 219](https://github.com/SensorsIot/IOTstack/issues/219) and [Issue 253](https://github.com/SensorsIot/IOTstack/issues/253) for more information. -### patch 2 – update libseccomp2 +### patch 2 – update libseccomp2 This patch is **ONLY** for Raspbian Buster. Do **NOT** install this patch if you are running Raspbian Bullseye. @@ -162,9 +150,9 @@ PRETTY_NAME="Raspbian GNU/Linux 10 (buster)" If you see the word "buster", proceed to step 2. Otherwise, skip this patch. -#### step 2: if you are running "buster" … +#### step 2: if you are indeed running "buster" -You need this patch if you are running Raspbian Buster. Without this patch, Docker images will fail if: +Without this patch on Buster, Docker images will fail if: * the image is based on Alpine and the image's maintainer updates to [Alpine 3.13](https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.13.0#time64_requirement); and/or * an image's maintainer updates to a library that depends on 64-bit values for *Unix epoch time* (the so-called Y2038 problem). @@ -189,61 +177,14 @@ Enable by running (takes effect after reboot): echo $(cat /boot/cmdline.txt) cgroup_memory=1 cgroup_enable=memory | sudo tee /boot/cmdline.txt ``` -## a word about the `sudo` command - -Many first-time users of IOTstack get into difficulty by misusing the `sudo` command. The problem is best understood by example. In the following, you would expect `~` (tilde) to expand to `/home/pi`. It does: - -```bash -$ echo ~/IOTstack -/home/pi/IOTstack -``` - -The command below sends the same `echo` command to `bash` for execution. This is what happens when you type the name of a shell script. You get a new instance of `bash` to run the script: - -```bash -$ bash -c 'echo ~/IOTstack' -/home/pi/IOTstack -``` - -Same answer. Again, this is what you expect. But now try it with `sudo` on the front: - -```bash -$ sudo bash -c 'echo ~/IOTstack' -/root/IOTstack -``` - -Different answer. It is different because `sudo` means "become root, and then run the command". The process of becoming root changes the home directory, and that changes the definition of `~`. - -Any script designed for working with IOTstack assumes `~` (or the equivalent `$HOME` variable) expands to `/home/pi`. That assumption is invalidated if the script is run by `sudo`. - -Of necessity, any script designed for working with IOTstack will have to invoke `sudo` **inside** the script **when it is required**. You do not need to second-guess the script's designer. - -Please try to minimise your use of `sudo` when you are working with IOTstack. Here are some rules of thumb: - -1. Is what you are about to run a script? If yes, check whether the script already contains `sudo` commands. Using `menu.sh` as the example: - - ```bash - $ grep -c 'sudo' ~/IOTstack/menu.sh - 28 - ``` - - There are numerous uses of `sudo` within `menu.sh`. That means the designer thought about when `sudo` was needed. - -2. Did the command you **just executed** work without `sudo`? Note the emphasis on the past tense. If yes, then your work is done. If no, and the error suggests elevated privileges are necessary, then re-execute the last command like this: - - ```bash - $ sudo !! - ``` - -It takes time, patience and practice to learn when `sudo` is **actually** needed. Over-using `sudo` out of habit, or because you were following a bad example you found on the web, is a very good way to find that you have created so many problems for yourself that will need to reinstall your IOTstack. *Please* err on the side of caution! -## the IOTstack menu +## the IOTstack menu The menu is used to install Docker and then build the `docker-compose.yml` file which is necessary for starting the stack. > The menu is only an aid. It is a good idea to learn the `docker` and `docker-compose` commands if you plan on using Docker in the long run. -### menu item: Install Docker (old menu only) +### menu item: Install Docker (old menu only) Please do **not** try to install `docker` and `docker-compose` via `sudo apt install`. There's more to it than that. Docker needs to be installed by `menu.sh`. The menu will prompt you to install docker if it detects that docker is not already installed. You can manually install it from within the `Native Installs` menu: @@ -260,7 +201,7 @@ Note: * New menu (master branch) automates this step. -### menu item: Build Stack +### menu item: Build Stack `docker-compose` uses a `docker-compose.yml` file to configure all your services. The `docker-compose.yml` file is created by the menu: @@ -292,15 +233,15 @@ Some containers also need to be built locally. Node-RED is an example. Depending Be patient (and ignore the huge number of warnings). -### menu item: Docker commands +### menu item: Docker commands The commands in this menu execute shell scripts in the root of the project. -### other menu items +### other menu items The old and new menus differ in the options they offer. You should come back and explore them once your stack is built and running. -## switching menus +## switching menus At the time of writing, IOTstack supports three menus: @@ -334,7 +275,7 @@ $ git checkout -- .templates/mosquitto/Dockerfile When `git status` reports no more "modified" files, it is safe to switch your branch. -### current menu (master branch) +### current menu (master branch) ```bash $ cd ~/IOTstack/ @@ -343,7 +284,7 @@ $ git checkout master $ ./menu.sh ``` -### old menu (old-menu branch) +### old menu (old-menu branch) ```bash $ cd ~/IOTstack/ @@ -352,7 +293,7 @@ $ git checkout old-menu $ ./menu.sh ``` -### experimental branch +### experimental branch Switch to the experimental branch to try the latest and greatest features. @@ -377,14 +318,14 @@ Notes: * The way back is to take down your stack, restore a backup, and bring up your stack again. -## useful commands: docker & docker-compose +## useful commands: docker & docker-compose Handy rules: * `docker` commands can be executed from anywhere, but * `docker-compose` commands need to be executed from within `~/IOTstack` -### starting your IOTstack +### starting your IOTstack To start the stack: @@ -395,7 +336,7 @@ $ docker-compose up -d Once the stack has been brought up, it will stay up until you take it down. This includes shutdowns and reboots of your Raspberry Pi. If you do not want the stack to start automatically after a reboot, you need to stop the stack before you issue the reboot command. -#### logging journald errors +#### logging journald errors If you get docker logging error like: @@ -423,9 +364,9 @@ Cannot create container for service [service name here]: unknown log opt 'max-fi Logging limits were added to prevent Docker using up lots of RAM if log2ram is enabled, or SD cards being filled with log data and degraded from unnecessary IO. See [Docker Logging configurations](https://docs.docker.com/config/containers/logging/configure/) -You can also turn logging off or set it to use another option for any service by using the IOTstack `docker-compose-override.yml` file mentioned at [IOTstack/Custom](https://sensorsiot.github.io/IOTstack/Custom/). +You can also turn logging off or set it to use another option for any service by using the IOTstack `docker-compose-override.yml` file mentioned at [IOTstack/Custom](Custom.md). -### starting an individual container +### starting an individual container To start a particular container: @@ -434,7 +375,7 @@ $ cd ~/IOTstack $ docker-compose up -d «container» ``` -### stopping your IOTstack +### stopping your IOTstack Stopping aka "downing" the stack stops and deletes all containers, and removes the internal network: @@ -450,7 +391,7 @@ $ cd ~/IOTstack $ docker-compose stop ``` -### stopping an individual container +### stopping an individual container `stop` can also be used to stop individual containers, like this: @@ -480,7 +421,7 @@ $ cd ~/IOTstack $ docker-compose up -d «container» ``` -### checking container status +### checking container status You can check the status of containers with: @@ -495,7 +436,7 @@ $ cd ~/IOTstack $ docker-compose ps ``` -### viewing container logs +### viewing container logs You can inspect the logs of most containers like this: @@ -517,7 +458,7 @@ $ docker logs -f nodered Terminate with a Control+C. Note that restarting a container will also terminate a followed log. -### restarting a container +### restarting a container You can restart a container in several ways: @@ -544,9 +485,9 @@ $ cd ~/IOTstack $ docker-compose up -d --force-recreate «container» ``` -See also [updating images built from Dockerfiles](#updateDockerfile) if you need to force `docker-compose` to notice a change to a Dockerfile. +See also [updating images built from Dockerfiles](#updating-images-not-built-from-dockerfiles) if you need to force `docker-compose` to notice a change to a Dockerfile. -## persistent data +## persistent data Docker allows a container's designer to map folders inside a container to a folder on your disk (SD, SSD, HD). This is done with the "volumes" key in `docker-compose.yml`. Consider the following snippet for Node-RED: @@ -588,7 +529,7 @@ is mirrored at the same relative path **inside** the container at: /data ``` -### deleting persistent data +### deleting persistent data If you need a "clean slate" for a container, you can delete its volumes. Using InfluxDB as an example: @@ -616,9 +557,9 @@ When InfluxDB starts, it sees that the folder on right-hand-side of the volumes This is how **most** containers behave. There are exceptions so it's always a good idea to keep a backup. -## stack maintenance +## stack maintenance -### update Raspberry Pi OS +### update Raspberry Pi OS You should keep your Raspberry Pi up-to-date. Despite the word "container" suggesting that *containers* are fully self-contained, they sometimes depend on operating system components ("WireGuard" is an example). @@ -627,7 +568,7 @@ $ sudo apt update $ sudo apt upgrade -y ``` -### git pull +### git pull Although the menu will generally do this for you, it does not hurt to keep your local copy of the IOTstack repository in sync with the master version on GitHub. @@ -636,7 +577,7 @@ $ cd ~/IOTstack $ git pull ``` -### container image updates +### container image updates There are two kinds of images used in IOTstack: @@ -650,7 +591,7 @@ The easiest way to work out which type of image you are looking at is to inspect * `image:` keyword then the image is **not** built using a Dockerfile. * `build:` keyword then the image **is** built using a Dockerfile. -#### updating images not built from Dockerfiles +#### updating images not built from Dockerfiles If new versions of this type of image become available on DockerHub, your local IOTstack copies can be updated by a `pull` command: @@ -665,7 +606,7 @@ The `pull` downloads any new images. It does this without disrupting the running The `up -d` notices any newly-downloaded images, builds new containers, and swaps old-for-new. There is barely any downtime for affected containers. -#### updating images built from Dockerfiles +#### updating images built from Dockerfiles Containers built using Dockerfiles have a two-step process: @@ -685,7 +626,7 @@ Note: * You can also add nodes to Node-RED using Manage Palette. -##### when Dockerfile changes (*local* image only) +##### when Dockerfile changes (*local* image only) When your Dockerfile changes, you need to rebuild like this: @@ -697,7 +638,7 @@ $ docker system prune This only rebuilds the *local* image and, even then, only if `docker-compose` senses a *material* change to the Dockerfile. -If you are trying to force the inclusion of a later version of an add-on node, you need to treat it like a [DockerHub update](#rebuildDockerfile). +If you are trying to force the inclusion of a later version of an add-on node, you need to treat it like a [DockerHub update](#updating-images-built-from-dockerfiles). Key point: @@ -712,7 +653,7 @@ Note: $ docker-compose up --build -d nodered ``` -##### when DockerHub updates (*base* and *local* images) +##### when DockerHub updates (*base* and *local* images) When a newer version of the *base* image appears on DockerHub, you need to rebuild like this: @@ -728,7 +669,7 @@ This causes DockerHub to be checked for the later version of the *base* image, d Then, the Dockerfile is run to produce a new *local* image. The Dockerfile run happens even if a new *base* image was not downloaded in the previous step. -### deleting unused images +### deleting unused images As your system evolves and new images come down from DockerHub, you may find that more disk space is being occupied than you expected. Try running: @@ -762,9 +703,9 @@ $ docker rmi dbf28ba50432 In general, you can use the repository name to remove an image but the Image ID is sometimes needed. The most common situation where you are likely to need the Image ID is after an image has been updated on DockerHub and pulled down to your Raspberry Pi. You will find two containers with the same name. One will be tagged "latest" (the running version) while the other will be tagged "\" (the prior version). You use the Image ID to resolve the ambiguity. -### pinning to specific versions +### pinning to specific versions -See [container image updates](#imageUpdates) to understand how to tell the difference between images that are used "as is" from DockerHub versus those that are built from local Dockerfiles. +See [container image updates](#container-image-updates) to understand how to tell the difference between images that are used "as is" from DockerHub versus those that are built from local Dockerfiles. Note: @@ -820,7 +761,7 @@ To pin an image to a specific version: $ docker-compose up -d --build mosquitto ``` -## the nuclear option - use with caution +## the nuclear option - use with caution If you create a mess and can't see how to recover, try proceeding like this: @@ -840,7 +781,7 @@ In words: 4. Move your existing IOTstack directory out of the way. If you get a permissions problem: * Re-try the command with `sudo`; and - * Read [a word about the `sudo` command](#aboutSudo). Needing `sudo` in this situation is an example of over-using `sudo`. + * Read [a word about the `sudo` command](#a-word-about-the-sudo-command). Needing `sudo` in this situation is an example of over-using `sudo`. 5. Check out a clean copy of IOTstack. diff --git a/docs/Containers/AdGuardHome.md b/docs/Containers/AdGuardHome.md index 3404cac8..1fd59f19 100644 --- a/docs/Containers/AdGuardHome.md +++ b/docs/Containers/AdGuardHome.md @@ -9,7 +9,7 @@ AdGuard Home and PiHole perform similar functions. They use the same ports so you can **not** run both at the same time. You must choose one or the other. -## Quick Start +## Quick Start When you first install AdGuard Home: @@ -34,7 +34,7 @@ When you first install AdGuard Home: Port 8089 is the default administrative user interface for AdGuard Home running under IOTstack. -Port 8089 is not active until you have completed the [Quick Start](#quickStart) procedure. You must start by connecting to port 3001. +Port 8089 is not active until you have completed the [Quick Start](#quick-start) procedure. You must start by connecting to port 3001. Because of AdGuard Home limitations, you must take special precautions if you decide to change to a different port number: @@ -50,11 +50,11 @@ Because of AdGuard Home limitations, you must take special precautions if you de $ docker-compose up -d adguardhome ``` -3. Repeat the [Quick Start](#quickStart) procedure, this time substituting the new Admin Web Interface port where you see "8089". +3. Repeat the [Quick Start](#quick-start) procedure, this time substituting the new Admin Web Interface port where you see "8089". ## About port 3001:3000 -Port 3001 (external, 3000 internal) is only used during [Quick Start](#quickStart) procedure. Once port 8089 becomes active, port 3001 ceases to be active. +Port 3001 (external, 3000 internal) is only used during [Quick Start](#quick-start) procedure. Once port 8089 becomes active, port 3001 ceases to be active. In other words, you need to keep port 3001 reserved even though it is only ever used to set up port 8089. diff --git a/docs/Containers/Blynk_server.md b/docs/Containers/Blynk_server.md index 6985a867..d5bf3260 100644 --- a/docs/Containers/Blynk_server.md +++ b/docs/Containers/Blynk_server.md @@ -2,7 +2,7 @@ This document discusses an IOTstack-specific version of Blynk-Server. It is built on top of an [Ubuntu](https://hub.docker.com/_/ubuntu) base image using a *Dockerfile*. -## References +## References - [Ubuntu base image](https://hub.docker.com/_/ubuntu) at DockerHub - [Peter Knight Blynk-Server fork](https://github.com/Peterkn2001/blynk-server) at GitHub (includes documentation) @@ -18,7 +18,7 @@ Acknowledgement: - Original writeup from @877dev -## Significant directories and files +## Significant directories and files ``` ~/IOTstack @@ -56,19 +56,19 @@ Everything in ❽: * will be replaced if it is not present when the container starts; but * will never be overwritten if altered by you. -## How Blynk Server gets built for IOTstack +## How Blynk Server gets built for IOTstack -### GitHub Updates +### GitHub Updates Periodically, the source code is updated and a new version is released. You can check for the latest version at the [releases page](https://github.com/Peterkn2001/blynk-server/releases/). -### IOTstack menu +### IOTstack menu When you select Blynk Server in the IOTstack menu, the *template service definition* is copied into the *Compose* file. > Under old menu, it is also copied to the *working service definition* and then not really used. -### IOTstack first run +### IOTstack first run On a first install of IOTstack, you run the menu, choose your containers, and are told to do this: @@ -131,7 +131,7 @@ You *may* see the same pattern in *Portainer*, which reports the ***base image** > Whether you see one or two rows depends on the version of `docker-compose` you are using and how your version of `docker-compose` builds local images. -## Logging +## Logging You can inspect Blynk Server's log by: @@ -139,7 +139,7 @@ You can inspect Blynk Server's log by: $ docker logs blynk_server ``` -## Changing Blynk Server's configuration +## Changing Blynk Server's configuration The first time you launch the `blynk_server` container, the following structure will be created in the persistent storage area: @@ -158,7 +158,7 @@ $ cd ~/IOTstack $ docker-compose restart blynk_server ``` -## Getting a clean slate +## Getting a clean slate Erasing Blynk Server's persistent storage area triggers self-healing and restores known defaults: @@ -178,7 +178,7 @@ Note: $ docker-compose restart blynk_server ``` -## Upgrading Blynk Server +## Upgrading Blynk Server To find out when a new version has been released, you need to visit the [Blynk-Server releases](https://github.com/Peterkn2001/blynk-server/releases/) page at GitHub. @@ -220,11 +220,11 @@ At the time of writing, version 0.41.16 was the most up-to-date. Suppose that ve The second `prune` will only be needed if there is an old *base image* and that, in turn, depends on the version of `docker-compose` you are using and how your version of `docker-compose` builds local images. -## Using Blynk Server +## Using Blynk Server See the [References](#references) for documentation links. -### Connecting to the administrative UI +### Connecting to the administrative UI To connect to the administrative interface, navigate to: @@ -237,7 +237,7 @@ You may encounter browser security warnings which you will have to acknowledge i - username = `admin@blynk.cc` - password = `admin` -### Change username and password +### Change username and password 1. Click on Users > "email address" and edit email, name and password. 2. Save changes. @@ -248,19 +248,19 @@ You may encounter browser security warnings which you will have to acknowledge i $ docker-compose restart blynk_server ``` -### Setup gmail +### Setup gmail Optional step, useful for getting the auth token emailed to you. (To be added once confirmed working....) -### iOS/Android app setup +### iOS/Android app setup 1. When setting up the application on your mobile be sure to select "custom" setup [see](https://github.com/Peterkn2001/blynk-server#app-and-sketch-changes). 2. Press "New Project" 3. Give it a name, choose device "Raspberry Pi 3 B" so you have plenty of [virtual pins](http://help.blynk.cc/en/articles/512061-what-is-virtual-pins) available, and lastly select WiFi. 4. Create project and the [auth token](https://docs.blynk.cc/#getting-started-getting-started-with-the-blynk-app-4-auth-token) will be emailed to you (if emails configured). You can also find the token in app under the phone app settings, or in the admin web interface by clicking Users>"email address" and scroll down to token. -### Quick usage guide for app +### Quick usage guide for app 1. Press on the empty page, the widgets will appear from the right. 2. Select your widget, let's say a button. @@ -273,7 +273,7 @@ Optional step, useful for getting the auth token emailed to you. Enter Node-Red..... -### Node-RED +### Node-RED 1. Install `node-red-contrib-blynk-ws` from Manage Palette. 2. Drag a "write event" node into your flow, and connect to a debug node diff --git a/docs/Containers/Chronograf.md b/docs/Containers/Chronograf.md index c2fa8179..02d276fe 100644 --- a/docs/Containers/Chronograf.md +++ b/docs/Containers/Chronograf.md @@ -1,12 +1,12 @@ # Chronograf -## References +## References - [*influxdata Chronograf* documentation](https://docs.influxdata.com/chronograf/) - [*GitHub*: influxdata/influxdata-docker/chronograf](https://github.com/influxdata/influxdata-docker/tree/master/chronograf) - [*DockerHub*: influxdata Chronograf](https://hub.docker.com/_/chronograf) -## Kapacitor integration +## Kapacitor integration If you selected Kapacitor in the menu and want Chronograf to be able to interact with it, you need to edit `docker-compose.yml` to un-comment the lines which are commented-out in the following: @@ -28,7 +28,7 @@ $ cd ~IOTstack $ docker-compose up -d chronograf ``` -## Upgrading Chronograf +## Upgrading Chronograf You can update the container via: @@ -45,7 +45,7 @@ In words: * `docker-compose up -d` causes any newly-downloaded images to be instantiated as containers (replacing the old containers); and * the `prune` gets rid of the outdated images. -### Chronograf version pinning +### Chronograf version pinning If you need to pin to a particular version: diff --git a/docs/Containers/deconz.md b/docs/Containers/Deconz.md similarity index 100% rename from docs/Containers/deconz.md rename to docs/Containers/Deconz.md diff --git a/docs/Containers/diyHue.md b/docs/Containers/DiyHue.md similarity index 100% rename from docs/Containers/diyHue.md rename to docs/Containers/DiyHue.md diff --git a/docs/Containers/Grafana.md b/docs/Containers/Grafana.md index 824dd8e8..85e64bb6 100644 --- a/docs/Containers/Grafana.md +++ b/docs/Containers/Grafana.md @@ -15,6 +15,16 @@ The default *~/IOTstack/services/grafana/grafana.env* contains this line: Uncomment that line and change the right hand side to [your own timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). +## Adding InfluxDB datasource + +Select Data Sources -> Add data source -> InfluxDB. + +Set options: + +* HTTP / URL: `http://influxdb:8086` +* InfluxDB Details / Database: `telegraf` +* InfluxDB Details / User: `nodered` +* InfluxDB Details / Password: `nodered` ## Security diff --git a/docs/Containers/Home-Assistant.md b/docs/Containers/Home-Assistant.md index 3562ad42..0ea204c4 100644 --- a/docs/Containers/Home-Assistant.md +++ b/docs/Containers/Home-Assistant.md @@ -2,7 +2,7 @@ Home Assistant is a home automation platform. It is able to track and control all devices at your home and offer a platform for automating control. -## References +## References - [Home Assistant home page](https://www.home-assistant.io/) @@ -13,7 +13,7 @@ Home Assistant is a home automation platform. It is able to track and control al - [DockerHub](https://hub.docker.com/r/homeassistant/home-assistant/) -## Home Assistant: two versions +## Home Assistant: two versions There are two versions of Home Assistant: @@ -31,7 +31,7 @@ Technically, both versions of Home Assistant can be installed on your Raspberry IOTstack used to offer a menu entry leading to a convenience script that could install Supervised Home Assistant but that stopped working when Home Assistant changed their approach. Now, the only method supported by IOTstack is Home Assistant Container. -### Installing Home Assistant Container +### Installing Home Assistant Container Home Assistant (Container) can be found in the `Build Stack` menu. Selecting it in this menu results in a service definition being added to: @@ -56,11 +56,13 @@ $ cd ~/IOTstack $ docker-compose up -d ``` -### Installing Supervised Home Assistant +### Installing Supervised Home Assistant The direction being taken by the Home Assistant folks is to supply a ready-to-run image for your Raspberry Pi. That effectively dedicates your Raspberry Pi to Home Assistant and precludes the possibility of running alongside IOTstack and containers like Mosquitto, InfluxDB, Node-RED, Grafana, PiHole and WireGuard. -It is possible to run Supervised Home Assistant on the same Raspberry Pi as IOTstack. The recommended approach is to start from a clean slate and use [PiBuilder](https://github.com/Paraphraser/PiBuilder). +Alternatively you can try to manually install Supervised Home Assistant using their [installation instructions for advanced users](https://github.com/home-assistant/supervised-installer) and when it works, install IOTstack. In theory this should work, but isn't tested or supported. + +The recommended approach is to start from a clean slate and use [PiBuilder](https://github.com/Paraphraser/PiBuilder). When you visit the PiBuilder link you may well have a reaction like "all far too complicated" but you should try to get past that. PiBuilder has two main use-cases: @@ -104,12 +106,12 @@ The first time you use PiBuilder, the process boils down to: where «name» is the name you give to your Raspberry Pi (eg "iot-hub"). -After step 9, Supervised Home Assistant will be running. The `04_setup.sh` script also deals with the [random MACs](#aboutRandomMACs) problem. After step 11, you'll be able to either: +After step 9, Supervised Home Assistant will be running. The `04_setup.sh` script also deals with the [random MACs](#why-random-macs-are-such-a-hassle) problem. After step 11, you'll be able to either: 1. Restore a backup; or 2. Run the IOTstack menu and choose your containers. -## Why random MACs are such a hassle +## Why random MACs are such a hassle > This material was originally posted as part of [Issue 312](https://github.com/SensorsIot/IOTstack/issues/312). It was moved here following a suggestion by [lole-elol](https://github.com/lole-elol). @@ -169,7 +171,7 @@ Random MACs are not a problem for a **client** device like a phone, tablet or la It is not just configuration-time SSH sessions that break. If you decide to leave Raspberry Pi random Wifi MAC active **and** you have other clients (eq IoT devices) communicating with the Pi over WiFi, you will wrong-foot those clients each time the Raspberry Pi reboots. Data communications services from those clients will be impacted until those client devices time-out and catch up. -# Using bluetooth from the container +## Using bluetooth from the container In order to be able to use BT & BLE devices from HA integrations, make sure that bluetooth is enabled and powered on at the start of the (Rpi) host by editing `/etc/bluetooth/main.conf`: ```conf @@ -187,3 +189,116 @@ UP ... ``` ref: https://scribles.net/auto-power-on-bluetooth-adapter-on-boot-up/ + +## HTTPS with a valid SSL certificate + +Some HA integrations (e.g google assistant) require your HA API to be +accessible via https with a valid certificate. You can configure HA to do this: +[docs](https://www.home-assistant.io/docs/configuration/remote/) / +[guide](https://www.home-assistant.io/docs/ecosystem/certificates/lets_encrypt/) +or use a reverse proxy container, as described below. + +The linuxserver Secure Web Access Gateway container +([swag](https://docs.linuxserver.io/general/swag)) ([Docker hub +docs](https://hub.docker.com/r/linuxserver/swag)) will automatically generate a +SSL-certificate, update the SSL certificate before it expires and act as a +reverse proxy. + +1. First test your HA is working correctly: `http://raspberrypi.local:8123/` (assuming +your RPi hostname is raspberrypi) +2. Make sure you have duckdns working. +3. On your internet router, forward public port 443 to the RPi port 443 +4. Add swag to ~/IOTstack/docker-compose.yml beneath the `services:`-line: +``` + swag: + image: ghcr.io/linuxserver/swag + cap_add: + - NET_ADMIN + environment: + - PUID=1000 + - PGID=1000 + - TZ=Etc/UTC + - URL=.duckdns.org + - SUBDOMAINS=wildcard + - VALIDATION=duckdns + - DUCKDNSTOKEN= + - CERTPROVIDER=zerossl + - EMAIL= # required when using zerossl + volumes: + - ./volumes/swag/config:/config + ports: + - 443:443 + restart: unless-stopped +``` + Replace the bracketed values. Do NOT use any "-characters to enclose the values. + +5. Start the swag container, this creates the file to be edited in the next step: + ``` + cd ~/IOTstack && docker-compose up -d + ``` + + Check it starts up OK: `docker-compose logs -f swag`. It will take a minute or two before it finally logs "Server ready". + +6. Enable reverse proxy for `raspberrypi.local`. `homassistant.*` is already by default. and fix homeassistant container name ("upstream_app"): + ``` + sed -e 's/server_name/server_name *.local/' \ + volumes/swag/config/nginx/proxy-confs/homeassistant.subdomain.conf.sample \ + > volumes/swag/config/nginx/proxy-confs/homeassistant.subdomain.conf + ``` + +7. Forward to correct IP when target is a container running in "network_mode: + host" (like Home Assistant does): + ``` + cat << 'EOF' | sudo tee volumes/swag/config/custom-cont-init.d/add-host.docker.internal.sh + #!/bin/sh + DOCKER_GW=$(ip route | awk 'NR==1 {print $3}') + + sed -i -e "s/upstream_app .*/upstream_app ${DOCKER_GW};/" \ + /config/nginx/proxy-confs/homeassistant.subdomain.conf + EOF + sudo chmod u+x volumes/swag/config/custom-cont-init.d/add-host.docker.internal.sh + ``` + (This needs to be copy-pasted/entered as-is, ignore any "> "-prefixes printed + by bash) + +8. (optional) Add reverse proxy password protection if you don't want to rely + on the HA login for security, doesn't affect API-access: + ``` + sed -i -e 's/#auth_basic/auth_basic/' \ + volumes/swag/config/nginx/proxy-confs/homeassistant.subdomain.conf + docker-compose exec swag htpasswd -c /config/nginx/.htpasswd anyusername + ``` +9. Add `use_x_forwarded_for` and `trusted_proxies` to your homeassistant [http + config](https://www.home-assistant.io/integrations/http). The configuration + file is at `volumes/home_assistant/configuration.yaml` For a default install + the resulting http-section should be: + ``` + http: + use_x_forwarded_for: true + trusted_proxies: + - 192.168.0.0/16 + - 172.16.0.0/12 + - 10.77.0.0/16 + ``` +10. Refresh the stack: `cd ~/IOTstack && docker-compose stop && docker-compose + up -d` (again may take 1-3 minutes for swag to start if it recreates + certificates) +11. Test homeassistant is still working correctly: + `http://raspberrypi.local:8123/` (assuming your RPi hostname is + raspberrypi) +12. Test the reverse proxy https is working correctly: + `https://raspberrypi.local/` (browser will issue a warning about wrong + certificate domain, as the certificate is issued for you duckdns-domain, we + are just testing) + + Or from the command line in the RPi: + ``` + curl --resolve homeassistant..duckdns.org:443:127.0.0.1 \ + https://homeassistant..duckdns.org/ + ``` + (output should end in `if (!window.latestJS) { }`) + +13. And finally test your router forwards correctly by accessing it from + outside your LAN(e.g. using a mobile phone): + `https://homeassistant..duckdns.org/` Now the certificate + should work without any warnings. diff --git a/docs/Containers/InfluxDB.md b/docs/Containers/InfluxDB.md index 4f714771..81d1d5bc 100644 --- a/docs/Containers/InfluxDB.md +++ b/docs/Containers/InfluxDB.md @@ -1,9 +1,104 @@ # InfluxDB +A time series database. + +InfluxDB has configurable aggregation and retention policies allowing +measurement resolution reduction, storing all added data points for recent data +and only aggregated values for older data. + +To connect use: + +| Field | Default | +| --------- | ---------- | +| User | nodered | +| Password | nodered | +| URL (from other services) | http://influxdb:8086 | +| URL (on the host machine) | http://localhost:8086 | + ## References - [Docker](https://hub.docker.com/_/influxdb) - [Website](https://www.influxdata.com/) -## Security +## Setup + +To access the influx console, show current databases and database measurements: +``` +pi@raspberrypi:~/IOTstack $ docker-compose exec influxdb bash +root@6bca535a945f:/# influx +Connected to http://localhost:8086 version 1.8.10 +InfluxDB shell version: 1.8.10 +> show databases +name: databases +name +---- +_internal +telegraf +> use telegraf +Using database telegraf +> show measurements +name: measurements +name +---- +cpu +cpu_temperature +disk +diskio +etc... +``` + +To create a new database and set a limited retention policy, here for instance +any data older than 52 weeks is deleted: + +``` +> create database mydb +> show retention policies on mydb +name duration shardGroupDuration replicaN default +---- -------- ------------------ -------- ------- +autogen 0s 168h0m0s 1 true +> alter retention policy "autogen" on "mydb" duration 52w shard duration 1w replication 1 default +> show retention policies on mydb +name duration shardGroupDuration replicaN default +---- -------- ------------------ -------- ------- +autogen 8736h0m0s 168h0m0s 1 true + +``` + +Aggregation, on the other hand, is where you keep your relevant statistics, but +decrease their time-resolution and lose individual data-points. This is a much +more complicated topic and harder to configure. As such it is well outside the +scope of this guide. + + +## Reducing flash wear-out + +SSD-drives have pretty good controllers spreading out writes, so this isn't a +this isn't really a concern for them. But if you store data on a SD-card, +flash wear may cause the card to fail prematurely. Flash memory has a limited +number of erase-write cycles per physical block. These blocks may be multiple +megabytes. You can use `sudo lsblk -D` to see how big the erase granularity is +on your card. The goal is to avoid writing lots of small changes targeting the +same physical blocks. Here are some tips to mitigate SD-card wear: + +* Don't use short retention policies. This may mask heavy disk IO without + increasing disk space usage. Depending on the file system used, new data may + be written to the same flash blocks that were freed by expiration, wearing + them out. +* Take care not to add measurements too often. If possible no more often than + once a minute. Add all measurements in one operation. +* Adding measurements directly to Influxdb will cause a write on every + operation. If your client code can't aggregate multiple measurements into one + write, consider routing them via Telegraf. It has the + `flush_interval`-option, which will combine the measurements into one write. +* All InfluxDB queries are logged by default and logs are written to the + SD-card. To disable this, add to docker-compose.yml, next to the other + INFLUXDB_\* entries: + ``` + - INFLUXDB_DATA_QUERY_LOG_ENABLED=false + - INFLUXDB_HTTP_LOG_ENABLED=false + ``` + This is especially important if you plan on having Grafana or Chronograf + displaying up-to-date data on a dashboard. + +## Old-menu branch The credentials and default database name for influxdb are stored in the file called influxdb/influx.env . The default username and password is set to "nodered" for both. It is HIGHLY recommended that you change them. The environment file contains several commented out options allowing you to set several access options such as default admin user credentials as well as the default database name. Any change to the environment file will require a restart of the service. To access the terminal for influxdb execute `./services/influxdb/terminal.sh`. Here you can set additional parameters or create other databases. diff --git a/docs/Containers/Kapacitor.md b/docs/Containers/Kapacitor.md index a126675e..11296e31 100644 --- a/docs/Containers/Kapacitor.md +++ b/docs/Containers/Kapacitor.md @@ -1,12 +1,12 @@ # Kapacitor -## References +## References - [*influxdata Kapacitor* documentation](https://docs.influxdata.com/kapacitor/) - [*GitHub*: influxdata/influxdata-docker/kapacitor](https://github.com/influxdata/influxdata-docker/tree/master/kapacitor) - [*DockerHub*: influxdata Kapacitor](https://hub.docker.com/_/kapacitor) -## Upgrading Kapacitor +## Upgrading Kapacitor You can update the container via: @@ -23,7 +23,7 @@ In words: * `docker-compose up -d` causes any newly-downloaded images to be instantiated as containers (replacing the old containers); and * the `prune` gets rid of the outdated images. -### Kapacitor version pinning +### Kapacitor version pinning If you need to pin to a particular version: diff --git a/docs/Containers/MariaDB.md b/docs/Containers/MariaDB.md index 6f89b82b..c32c3437 100644 --- a/docs/Containers/MariaDB.md +++ b/docs/Containers/MariaDB.md @@ -1,3 +1,4 @@ +# MariaDB ## Source * [Docker hub](https://hub.docker.com/r/linuxserver/mariadb/) @@ -59,14 +60,16 @@ You can open a terminal session within the mariadb container via: $ docker exec -it mariadb bash ``` +To connect to the database: `mysql -uroot -p` + To close the terminal session, either: * type "exit" and press return; or * press control+d. -## Container health check +## Container health check -### theory of operation +### theory of operation A script , or "agent", to assess the health of the MariaDB container has been added to the *local image* via the *Dockerfile*. In other words, the script is specific to IOTstack. @@ -84,11 +87,11 @@ The agent is invoked 30 seconds after the container starts, and every 30 seconds mysqld is alive ``` -3. If the command returned the expected response, the agent tests the responsiveness of the TCP port the `mysqld` daemon should be listening on (see [customising health-check](#healthCheckCustom)). +3. If the command returned the expected response, the agent tests the responsiveness of the TCP port the `mysqld` daemon should be listening on (see [customising health-check](#customising-health-check)). 4. If all of those steps succeed, the agent concludes that MariaDB is functioning properly and returns "healthy". -### monitoring health-check +### monitoring health-check Portainer's *Containers* display contains a *Status* column which shows health-check results for all containers that support the feature. @@ -121,7 +124,7 @@ Possible reply patterns are: mariadb Up About a minute (unhealthy) ``` -### customising health-check +### customising health-check You can customise the operation of the health-check agent by editing the `mariadb` service definition in your *Compose* file: diff --git a/docs/Containers/Mosquitto.md b/docs/Containers/Mosquitto.md index d06e143d..282e4fd9 100644 --- a/docs/Containers/Mosquitto.md +++ b/docs/Containers/Mosquitto.md @@ -6,7 +6,7 @@ This document discusses an IOTstack-specific version of Mosquitto built on top o
-## References +## References - [*Eclipse Mosquitto* home](https://mosquitto.org) - [*GitHub*: eclipse/mosquitto](https://github.com/eclipse/mosquitto) @@ -14,7 +14,7 @@ This document discusses an IOTstack-specific version of Mosquitto built on top o - [Setting up passwords](https://www.youtube.com/watch?v=1msiFQT_flo) (video) - [Tutorial: from MQTT to InfluxDB via Node-Red](https://gist.github.com/Paraphraser/c9db25d131dd4c09848ffb353b69038f) -## Significant directories and files +## Significant directories and files ``` ~/IOTstack @@ -57,23 +57,23 @@ This document discusses an IOTstack-specific version of Mosquitto built on top o * You will normally need `sudo` to make changes in this area. * Each time Mosquitto starts, it automatically replaces anything originating in ❹ that has gone missing from ❼. This "self-repair" function is intended to provide reasonable assurance that Mosquitto will at least **start** instead of going into a restart loop. -## How Mosquitto gets built for IOTstack +## How Mosquitto gets built for IOTstack -### Mosquitto source code ([*GitHub*](https://github.com)) +### Mosquitto source code ([*GitHub*](https://github.com)) The source code for Mosquitto lives at [*GitHub* eclipse/mosquitto](https://github.com/eclipse/mosquitto). -### Mosquitto images ([*DockerHub*](https://hub.docker.com)) +### Mosquitto images ([*DockerHub*](https://hub.docker.com)) Periodically, the source code is recompiled and the resulting image is pushed to [eclipse-mosquitto](https://hub.docker.com/_/eclipse-mosquitto?tab=tags&page=1&ordering=last_updated) on *DockerHub*. -### IOTstack menu +### IOTstack menu When you select Mosquitto in the IOTstack menu, the *template service definition* is copied into the *Compose* file. > Under old menu, it is also copied to the *working service definition* and then not really used. -### IOTstack first run +### IOTstack first run On a first install of IOTstack, you run the menu, choose Mosquitto as one of your containers, and are told to do this: @@ -82,7 +82,7 @@ $ cd ~/IOTstack $ docker-compose up -d ``` -> See also the [Migration considerations](#migration) (below). +> See also the [Migration considerations](#migration-considerations) (below). `docker-compose` reads the *Compose* file. When it arrives at the `mosquitto` fragment, it finds: @@ -107,7 +107,7 @@ The *Dockerfile* begins with: FROM eclipse-mosquitto:latest ``` -> If you need to pin to a particular version of Mosquitto, the *Dockerfile* is the place to do it. See [Mosquitto version pinning](#versionPinning). +> If you need to pin to a particular version of Mosquitto, the *Dockerfile* is the place to do it. See [Mosquitto version pinning](#mosquitto-version-pinning). The `FROM` statement tells the build process to pull down the ***base image*** from [*DockerHub*](https://hub.docker.com). @@ -144,7 +144,7 @@ You *may* see the same pattern in Portainer, which reports the *base image* as " > Whether you see one or two rows depends on the version of `docker-compose` you are using and how your version of `docker-compose` builds local images. -### Migration considerations +### Migration considerations Under the original IOTstack implementation of Mosquitto (just "as it comes" from *DockerHub*), the service definition expected the configuration files to be at: @@ -205,7 +205,7 @@ Using `mosquitto.conf` as the example, assume you wish to use your existing file 5. If necessary, repeat these steps with `filter.acl`. -## Logging +## Logging Mosquitto logging is controlled by `mosquitto.conf`. This is the default configuration: @@ -248,9 +248,9 @@ $ sudo tail ~/IOTstack/volumes/mosquitto/log/mosquitto.log Logs written to `mosquitto.log` do not disappear when your IOTstack is restarted. They persist until you take action to prune the file. -## Security +## Security -### Configuring security +### Configuring security Mosquitto security is controlled by `mosquitto.conf`. These are the relevant directives: @@ -269,7 +269,7 @@ enabled | true | credentials optional | | enabled | false | credentials required | | -### Password file management +### Password file management The password file for Mosquitto is part of a mapped volume: @@ -287,7 +287,7 @@ The Mosquitto container performs self-repair each time the container is brought * If `false` then **all** MQTT requests will be rejected. -#### create username and password +#### create username and password To create a username and password, use the following as a template. @@ -303,9 +303,9 @@ $ docker exec mosquitto mosquitto_passwd -b /mosquitto/pwfile/pwfile hello world Note: -* See also [customising health-check](#healthCheckCustom). If you are creating usernames and passwords, you may also want to create credentials for the health-check agent. +* See also [customising health-check](#customising-health-check). If you are creating usernames and passwords, you may also want to create credentials for the health-check agent. -#### check password file +#### check password file There are two ways to verify that the password file exists and has the expected content: @@ -329,7 +329,7 @@ Each credential starts with the username and occupies one line in the file: hello:$7$101$ZFOHHVJLp2bcgX+h$MdHsc4rfOAhmGG+65NpIEJkxY0beNeFUyfjNAGx1ILDmI498o4cVOaD9vDmXqlGUH9g6AgHki8RPDEgjWZMkDA== ``` -#### remove entry from password file +#### remove entry from password file To remove an entry from the password file: @@ -337,7 +337,7 @@ To remove an entry from the password file: $ docker exec mosquitto mosquitto_passwd -D /mosquitto/pwfile/pwfile «username» ``` -#### reset the password file +#### reset the password file There are several ways to reset the password file. Your options are: @@ -368,7 +368,7 @@ There are several ways to reset the password file. Your options are: The result is an empty password file. -### Activate Mosquitto security +### Activate Mosquitto security 1. Use `sudo` and your favourite text editor to open the following file: @@ -411,15 +411,15 @@ There are several ways to reset the password file. Your options are: $ docker-compose restart mosquitto ``` -### Testing Mosquitto security +### Testing Mosquitto security -#### assumptions +#### assumptions 1. You have created at least one username ("hello") and password ("world"). 2. `password_file` is enabled. 3. `allow_anonymous` is `false`. -#### install testing tools +#### install testing tools If you do not have the Mosquitto clients installed on your Raspberry Pi (ie `$ which mosquitto_pub` does not return a path), install them using: @@ -427,7 +427,7 @@ If you do not have the Mosquitto clients installed on your Raspberry Pi (ie `$ w $ sudo apt install -y mosquitto-clients ``` -#### test: *anonymous access is prohibited* +#### test: *anonymous access is prohibited* Test **without** providing credentials: @@ -441,7 +441,7 @@ Note: * The error is the expected result and shows that Mosquitto will not allow anonymous access. -#### test: *access with credentials is permitted* +#### test: *access with credentials is permitted* Test with credentials @@ -454,7 +454,7 @@ Note: * The absence of any error message means the message was sent. Silence = success! -#### test: *round-trip with credentials is permitted* +#### test: *round-trip with credentials is permitted* Prove round-trip connectivity will succeed when credentials are provided. First, set up a subscriber as a background process. This mimics the role of a process like Node-Red: @@ -482,9 +482,9 @@ $ [1]+ Terminated mosquitto_sub -v -h 127.0.0.1 -p 1883 -t "/password/test" -F "%I %t %p" -u hello -P world ``` -## Container health check +## Container health check -### theory of operation +### theory of operation A script , or "agent", to assess the health of the Mosquitto container has been added to the *local image* via the *Dockerfile*. In other words, the script is specific to IOTstack. @@ -499,7 +499,7 @@ The agent is invoked 30 seconds after the container starts, and every 30 seconds * Subscribes to the same broker for the same topic for a single message event. * Compares the payload sent with the payload received. If the payloads (ie time-stamps) match, the agent concludes that the Mosquitto broker (the process running inside the same container) is functioning properly for round-trip messaging. -### monitoring health-check +### monitoring health-check Portainer's *Containers* display contains a *Status* column which shows health-check results for all containers that support the feature. @@ -545,7 +545,7 @@ Notes: * If you enable authentication for your Mosquitto broker, you will need to add `-u «user»` and `-P «password»` parameters to this command. * You should expect to see a new message appear approximately every 30 seconds. That indicates the health-check agent is functioning normally. Use control+c to terminate the command. -### customising health-check +### customising health-check You can customise the operation of the health-check agent by editing the `mosquitto` service definition in your *Compose* file: @@ -565,7 +565,7 @@ You can customise the operation of the health-check agent by editing the `mosqui Note: - * You will also need to use the same topic string in the `mosquitto_sub` command shown at [monitoring health-check](#healthCheckMonitor). + * You will also need to use the same topic string in the `mosquitto_sub` command shown at [monitoring health-check](#monitoring-health-check). 3. If you have enabled authentication for your Mosquitto broker service, you will need to provide appropriate credentials for your health-check agent: @@ -594,7 +594,7 @@ You can customise the operation of the health-check agent by editing the `mosqui You must remove the entire `healthcheck:` clause. -## Upgrading Mosquitto +## Upgrading Mosquitto You can update most containers like this: @@ -636,7 +636,7 @@ Your existing Mosquitto container continues to run while the rebuild proceeds. O The `prune` is the simplest way of cleaning up. The first call removes the old *local image*. The second call cleans up the old *base image*. Whether an old *base image* exists depends on the version of `docker-compose` you are using and how your version of `docker-compose` builds local images. -### Mosquitto version pinning +### Mosquitto version pinning If you need to pin Mosquitto to a particular version: @@ -672,7 +672,7 @@ Note: * As well as preventing Docker from updating the *base image*, pinning will also block incoming updates to the *Dockerfile* from a `git pull`. Nothing will change until you decide to remove the pin. -## About Port 9001 +## About Port 9001 Earlier versions of the IOTstack service definition for Mosquitto included two port mappings: diff --git a/docs/Containers/NextCloud.md b/docs/Containers/NextCloud.md index d88aa353..f649757f 100644 --- a/docs/Containers/NextCloud.md +++ b/docs/Containers/NextCloud.md @@ -1,6 +1,6 @@ # Nextcloud -## Service definition +## Service definition This is the **core** of the IOTstack Nextcloud service definition: @@ -54,7 +54,7 @@ Under new-menu, the menu can generate random passwords for you. You can either u The passwords need to be set before you bring up the Nextcloud service for the first time but the following initialisation steps assume you might not have done that and always start over from a clean slate. -## Initialising Nextcloud +## Initialising Nextcloud 1. Be in the correct directory: @@ -108,7 +108,7 @@ The passwords need to be set before you bring up the Nextcloud service for the f * You **can't** use a multicast domain name (eg `myrpi.local`). An mDNS name will not work until Nextcloud has been initialised! * Once you have picked a connection method, **STICK TO IT**. - * You are only stuck with this restriction until Nextcloud has been initialised. You **can** (and should) fix it later by completing the steps in ["Access through untrusted domain"](#untrustedDomain). + * You are only stuck with this restriction until Nextcloud has been initialised. You **can** (and should) fix it later by completing the steps in ["Access through untrusted domain"](#access-through-untrusted-domain). 7. On a computer that is **not** the Raspberry Pi running Nextcloud, launch a browser and point to the Raspberry Pi running Nextcloud using your chosen connection method. Examples: @@ -243,7 +243,7 @@ See also: * [Nextcloud documentation - trusted domains](https://docs.nextcloud.com/server/21/admin_manual/installation/installation_wizard.html#trusted-domains). -### Using a DNS alias for your Nextcloud service +### Using a DNS alias for your Nextcloud service The examples above include using a DNS alias (a CNAME record) for your Nextcloud service. If you decide to do that, you may see this warning in the log: @@ -261,13 +261,13 @@ You can silence the warning by editing the Nextcloud service definition in `dock Nextcloud traffic is not encrypted. Do **not** expose it to the web by opening a port on your home router. Instead, use a VPN like Wireguard to provide secure access to your home network, and let your remote clients access Nextcloud over the VPN tunnel. -## Container health check +## Container health check A script , or "agent", to assess the health of the MariaDB container has been added to the *local image* via the *Dockerfile*. In other words, the script is specific to IOTstack. -Because it is an instance of MariaDB, Nextcloud_DB inherits the health-check agent. See the [IOTstack MariaDB](https://sensorsiot.github.io/IOTstack/Containers/MariaDB/) documentation for more information. +Because it is an instance of MariaDB, Nextcloud_DB inherits the health-check agent. See the [IOTstack MariaDB](MariaDB.md) documentation for more information. -## Keeping Nextcloud up-to-date +## Keeping Nextcloud up-to-date To update the `nextcloud` container: @@ -290,7 +290,7 @@ $ docker system prune The first "prune" removes the old *local* image, the second removes the old *base* image. Whether an old *base image* exists depends on the version of `docker-compose` you are using and how your version of `docker-compose` builds local images. -## Backups +## Backups Nextcloud is currently excluded from the IOTstack-supplied backup scripts due to its potential size. diff --git a/docs/Containers/Node-RED.md b/docs/Containers/Node-RED.md index ee32ce4b..1095d409 100644 --- a/docs/Containers/Node-RED.md +++ b/docs/Containers/Node-RED.md @@ -1,13 +1,13 @@ # Node-RED -## References +## References - [nodered.org home](https://nodered.org/) - [GitHub: node-red/node-red-docker](https://github.com/node-red/node-red-docker) - [DockerHub: nodered/node-red](https://hub.docker.com/r/nodered/node-red) - [Tutorial: from MQTT to InfluxDB via Node-Red](https://gist.github.com/Paraphraser/c9db25d131dd4c09848ffb353b69038f) -## Significant files +## Significant files ``` ~/IOTstack @@ -33,7 +33,7 @@ 6. Data directory (mapped volume). 7. SSH directory (mapped volume). -## How Node-RED gets built for IOTstack +## How Node-RED gets built for IOTstack ### Node-RED source code ([GitHub](https://github.com)) @@ -64,7 +64,7 @@ You choose add-on nodes from a supplementary menu. We recommend accepting the de Key points: * Under new menu, you must press the right arrow to access the supplementary menu. Under old menu, the list of add-on nodes is displayed automatically. -* Do not be concerned if you can't find an add-on node you need in the list. You can also add nodes via Manage Palette once Node-RED is running. See [adding extra nodes](#addingExtraNodes). +* Do not be concerned if you can't find an add-on node you need in the list. You can also add nodes via Manage Palette once Node-RED is running. See [adding extra nodes](#adding-extra-nodes). Choosing add-on nodes in the menu causes the *Dockerfile* to be created. @@ -135,9 +135,9 @@ You should not remove the *base* image, even though it appears to be unused. > Whether you see one or two rows depends on the version of `docker-compose` you are using and how your version of `docker-compose` builds local images. -## Securing Node-RED +## Securing Node-RED -### Setting an encryption key for your credentials +### Setting an encryption key for your credentials After you install Node-RED, you should set an encryption key. Completing this step will silence the warning you will see when you run: @@ -192,7 +192,7 @@ $ cd ~/IOTstack $ docker-compose restart nodered ``` -### Setting a username and password for Node-RED +### Setting a username and password for Node-RED To secure Node-RED you need a password hash. Run the following command, replacing `PASSWORD` with your own password: @@ -208,7 +208,7 @@ $2a$08$gTdx7SkckJVCw1U98o4r0O7b8P.gd5/LAPlZI6geg5LRg4AUKuDhS Copy that text to your clipboard, then follow the instructions at [Node-RED User Guide - Securing Node-RED - Username & Password-based authentication](https://nodered.org/docs/user-guide/runtime/securing-node-red#usernamepassword-based-authentication). -## Referring to other containers +## Referring to other containers Node-RED can run in two modes. By default, it runs in "non-host mode" but you can also move the container to "host mode" by editing the Node-RED service definition in your *Compose* file to: @@ -220,7 +220,7 @@ Node-RED can run in two modes. By default, it runs in "non-host mode" but you ca 2. Remove the `ports` directive and the mapping of port 1880. -### When Node-RED is not in host mode +### When Node-RED is not in host mode Most examples on the web assume Node-RED and other services in the MING (Mosquitto, InfluxDB, Node-RED, Grafana) stack have been installed natively, rather than in Docker containers. Those examples typically include the loopback address + port syntax, like this: @@ -244,7 +244,7 @@ influxdb:8086 Behind the scenes, Docker maintains a table similar to an `/etc/hosts` file mapping container names to the IP addresses on the internal bridged network that are assigned, dynamically, by Docker when it spins up each container. -### When Node-RED is in host mode +### When Node-RED is in host mode This is where you use loopback+port syntax, such as the following to communicate with Mosquitto: @@ -254,7 +254,7 @@ This is where you use loopback+port syntax, such as the following to communicate What actually occurs is that Docker is listening to external port 1883 on behalf of Mosquitto. It receives the packet and routes it (layer three) to the internal bridged network, performing network address translation (NAT) along the way to map the external port to the internal port. Then the packet is delivered to Mosquitto. The reverse happens when Mosquitto replies. It works but is less efficient than when all containers are in non-host mode. -## GPIO Access +## GPIO Access To communicate with your Raspberry Pi's GPIO you need to do the following: @@ -265,7 +265,7 @@ To communicate with your Raspberry Pi's GPIO you need to do the following: $ sudo apt install pigpio python-pigpio python3-pigpio ``` -2. Install the `node-red-node-pi-gpiod` node. See [Adding extra nodes](#addingExtraNodes). It allows you to connect to multiple Pis from the same Node-RED service. +2. Install the `node-red-node-pi-gpiod` node. See [Adding extra nodes](#adding-extra-nodes). It allows you to connect to multiple Pis from the same Node-RED service. 3. Make sure that the `pigpdiod` daemon is running. The recommended method is listed [here](https://github.com/node-red/node-red-nodes/tree/master/hardware/pigpiod). In essence, you need to: - Use `sudo` to edit `/etc/rc.local`; @@ -281,7 +281,7 @@ To communicate with your Raspberry Pi's GPIO you need to do the following: 4. Drag a gpio node onto the canvas and configure it using the IP address of your Raspberry Pi (eg 192.168.1.123). Don't try to use 127.0.0.1 because that is the loopback address of the Node-RED container. -## Sharing files between Node-RED and the Raspberry Pi +## Sharing files between Node-RED and the Raspberry Pi Containers run in a sandboxed environment. A process running inside a container can't see the Raspberry Pi's file system. Neither can a process running outside a container access files inside the container. @@ -372,7 +372,7 @@ Deploying the flow and clicking on the Inject node results in the debug message You can reverse this process. Any file you place within the path `~/IOTstack/volumes/nodered/data` can be read by a "File in" node. -## Executing commands outside the Node-RED container +## Executing commands outside the Node-RED container A reasonably common requirement in a Node-RED flow is the ability to execute a command on the host system. The standard tool for this is an "exec" node. @@ -400,7 +400,7 @@ The same thing will happen if a Node-RED "exec" node executes that `grep` comman Docker doesn't provide any mechanism for a container to execute an arbitrary command **outside** of its container. A workaround is to utilise SSH. This remainder of this section explains how to set up the SSH scaffolding so that "exec" nodes running in a Node-RED container can invoke arbitrary commands **outside** container-space. -### Task Goal +### Task Goal Be able to use a Node-RED `exec` node to perform the equivalent of: @@ -413,14 +413,14 @@ where: * `«HOSTNAME»` is any host under your control (not just the Raspberry Pi running IOTstack); and * `«COMMAND»` is any command known to the target host. -### Assumptions +### Assumptions * [SensorsIot/IOTstack](https://github.com/SensorsIot/IOTstack) is installed on your Raspberry Pi. * The Node-RED container is running. These instructions are specific to IOTstack but the underlying concepts should apply to any installation of Node-RED in a Docker container. -### Executing commands "inside" a container +### Executing commands "inside" a container These instructions make frequent use of the ability to run commands "inside" the Node-RED container. For example, suppose you want to execute: @@ -466,11 +466,11 @@ You have several options: 3. Run the command from Portainer by selecting the container, then clicking the ">_ console" link. This is identical to opening a shell. -### Variable definitions +### Variable definitions You will need to have a few concepts clear in your mind before you can set up SSH successfully. I use double-angle quote marks (guillemets) to mean "substitute the appropriate value here". -#### «HOSTNAME» (required) +#### «HOSTNAME» (required) The name of your Raspberry Pi. When you first booted your RPi, it had the name "raspberrypi" but you probably changed it using `raspi-config`. Example: @@ -515,7 +515,7 @@ The user ID of the account on «HOSTNAME» where you want Node-RED flows to be a pi ``` -### Step 1: *Generate SSH key-pair for Node-RED* (one time) +### Step 1: *Generate SSH key-pair for Node-RED* (one time) Create a key-pair for Node-RED. This is done by executing the `ssh-keygen` command **inside** the container: @@ -581,7 +581,7 @@ and check to make sure that only the key(s) you wanted were added. If you do not see an indication that a key has been added, you may need to retrace your steps. -### Step 3: *Perform the recommended test* +### Step 3: *Perform the recommended test* The output above recommends a test. The test needs to be run **inside** the Node-RED container so the syntax is: @@ -602,9 +602,9 @@ If everything works as expected, you should see a list of the files in your IOTs Assuming success, think about what just happened? You told SSH **inside** the Node-RED container to run the `ls` command **outside** the container on your Raspberry Pi. You broke through the containerisation. -### Understanding what's where and what each file does +### Understanding what's where and what each file does -#### What files are where +#### What files are where Six files are relevant to Node-RED's ability to execute commands outside of container-space: @@ -617,7 +617,7 @@ Six files are relevant to Node-RED's ability to execute commands outside of cont Unless you take precautions, those keys will change whenever your Raspberry Pi is rebuilt from scratch and that **will** stop SSH from working. - You can recover by re-running [`ssh-copy-id`](#sshStep2). + You can recover by re-running [`ssh-copy-id`](#step-2-exchange-keys-with-target-hosts-once-per-target-host). * in `~/IOTstack/volumes/nodered/ssh`: @@ -632,7 +632,7 @@ Six files are relevant to Node-RED's ability to execute commands outside of cont If you lose or destroy these keys, SSH **will** stop working. - You can recover by [generating new keys](#sshStep1) and then re-running [`ssh-copy-id`](#sshStep2). + You can recover by [generating new keys](#step-1-generate-ssh-key-pair-for-node-red-one-time) and then re-running [`ssh-copy-id`](#sshStep2). - `known_hosts` @@ -654,7 +654,7 @@ Six files are relevant to Node-RED's ability to execute commands outside of cont Note that providing the correct password at the command line will auto-repair the `authorized_keys` file. -#### What each file does +#### What each file does SSH running **inside** the Node-RED container uses the Node-RED container's private key to provide assurance to SSH running **outside** the container that it (the Node-RED container) is who it claims to be. @@ -664,7 +664,7 @@ SSH running **outside** container-space uses the Raspberry Pi's private host key SSH running **inside** the Node-RED container verifies that assurance by using its copy of the Raspberry Pi's public host key stored in `known_hosts`. -### Config file (optional) +### Config file (optional) You don't **have** to do this step but it will simplify your exec node commands and reduce your maintenance problems if you do. @@ -741,7 +741,7 @@ $ sudo chown root:root config $ sudo mv config ./volumes/nodered/ssh ``` -#### Re-test with config file in place +#### Re-test with config file in place The previous test used this syntax: @@ -763,7 +763,7 @@ $ docker exec nodered ssh «HOSTNAME» ls -1 /home/pi/IOTstack The result should be the same as the earlier test. -### A test flow +### A test flow ![node-red-exec-node-ssh-test](./images/nodered-exec-node-ssh-test.jpeg) @@ -805,7 +805,7 @@ In the Node-RED GUI: The first line is the result of running the command inside the Node-RED container. The second line is the result of running the same command outside the Node-RED container on the Raspberry Pi. -### Suppose you want to add another «HOSTNAME» +### Suppose you want to add another «HOSTNAME» 1. Exchange keys with the new target host using: @@ -821,7 +821,7 @@ In the Node-RED GUI: to define the new host. Remember to use `sudo` to edit the file. There is no need to restart Node-RED or recreate the container. -## Upgrading Node-RED +## Upgrading Node-RED You can update most containers like this: @@ -863,7 +863,7 @@ Your existing Node-RED container continues to run while the rebuild proceeds. On The `prune` is the simplest way of cleaning up old images. Sometimes you need to run this twice, the first time to clean up the old local image, the second time for the old base image. Whether an old base image exists depends on the version of `docker-compose` you are using and how your version of `docker-compose` builds local images. -## Customising Node-RED +## Customising Node-RED You customise your *local* image of Node-RED by making changes to: @@ -881,7 +881,7 @@ $ docker system prune The `--build` option on the `up` command (as distinct from a `docker-compose build` command) works in this situation because you've made a substantive change to your *Dockerfile*. -### Node.js version +### Node.js version Out of the box, IOTstack starts the Node-RED *Dockerfile* with: @@ -903,9 +903,9 @@ In the unlikely event that you need to run an add-on node that needs version 10 FROM nodered/node-red:latest-10 ``` -Once you have made that change, follow the steps at [apply *Dockerfile* changes](#applyDockerfileChange). +Once you have made that change, follow the steps at [apply *Dockerfile* changes](#upgrading-node-red). -### Adding extra packages +### Adding extra packages As well as providing the Node-RED service, the nodered container is an excellent testbed. Installing the DNS tools, Mosquitto clients and tcpdump will help you to figure out what is going on **inside** container-space. @@ -927,13 +927,13 @@ RUN apk update && apk add --no-cache eudev-dev mosquitto-clients bind-tools tcpd You can add as many extra packages as you like. They will persist until you change the *Dockerfile* again. -Once you have made this change, follow the steps at [apply *Dockerfile* changes](#applyDockerfileChange). +Once you have made this change, follow the steps at [apply *Dockerfile* changes](#upgrading-node-red). -### Adding extra nodes +### Adding extra nodes You can install nodes by: -1. Adding nodes to the *Dockerfile* and then following the steps at [apply *Dockerfile* changes](#applyDockerfileChange). +1. Adding nodes to the *Dockerfile* and then following the steps at [apply *Dockerfile* changes](#upgrading-node-red). This is also what will happen if you re-run the menu and change the selected nodes, except that the menu will also blow away any other customisations you may have made to your *Dockerfile*. @@ -997,7 +997,7 @@ If you're wondering about "backup", nodes installed via: Basically, if you're running IOTstack backups then your add-on nodes will be backed-up. -### Node precedence +### Node precedence Add-on nodes that are installed via *Dockerfile* wind up at the **internal** path: @@ -1037,7 +1037,7 @@ take precedence over those installed at: Or, to put it more simply: in any contest, Manage Palette prevails over *Dockerfile*. -### Resolving node duplication +### Resolving node duplication Sometimes, even when you are 100% certain that **you** didn't do it, an add-on node will turn up in both places. There is probably some logical reason for this but I don't know what it is. diff --git a/docs/Containers/Octoprint.md b/docs/Containers/Octoprint.md index ec932c78..ef076707 100644 --- a/docs/Containers/Octoprint.md +++ b/docs/Containers/Octoprint.md @@ -1,3 +1,6 @@ +--- +title: Octoprint +--- # OctoPrint – the snappy web interface for your 3D printer ## References diff --git a/docs/Containers/openHAB.md b/docs/Containers/OpenHab.md similarity index 100% rename from docs/Containers/openHAB.md rename to docs/Containers/OpenHab.md diff --git a/docs/Containers/Pi-hole.md b/docs/Containers/Pi-hole.md index 7842f79b..1035aa93 100644 --- a/docs/Containers/Pi-hole.md +++ b/docs/Containers/Pi-hole.md @@ -1,8 +1,127 @@ # Pi-hole -Pi-hole is a fantastic utility to reduce ads +Pi-hole is a fantastic utility to reduce ads. -The interface can be found on `"your_ip":8089/admin` +The web interface can be found at `http://«your_ip»:8089/admin` +where «your_ip» can be: -Default password is `pihole`. This can be changed in the `~/IOTstack/services/pihole/pihole.env` file +* The IP address of the Raspberry Pi running Pi-hole. +* The domain name of the Raspberry Pi running Pi-hole. +* The multicast DNS name (eg "raspberrypi.local") of the Raspberry Pi running + Pi-hole. + +Default password is random, it can be changed by running: +``` +docker-compose exec pihole pihole -a -p myNewPassword +``` + +References: + +* [Pi-hole on GitHub](https://github.com/pi-hole/docker-pi-hole) +* [Pi-hole on Dockerhub](https://hub.docker.com/r/pihole/pihole) + +## Environment variables + +Environment variables govern much of Pi-hole's behaviour. If you are running new +menu (master branch), the variables are inline in `docker-compose.yml`. If you +are running old menu, the variables will be in: +`~/IOTstack/services/pihole/pihole.env` + +The first time Pi-hole is launched, it checks for the `WEBPASSWORD` environment +variable. If found, sets the initial password. + +Pi-hole supports a [long list of environment +variables](https://github.com/pi-hole/docker-pi-hole#environment-variables). + +## Using Pi-hole as your DNS resolver + +In order for the Pi-hole to ad-block or resolve anything, it needs to be +defined as the DNS server. This can either be done manually to each device or +you can define it as a DNS-nameserver for the whole LAN. + +Note that using Pi-hole for clients on your network pretty much **requires** the +Raspberry Pi running Pi-hole to have a fixed IP address. + +Assuming your RPi hostname is `raspberrypi` and has the static IP +`192.168.1.10`: + +1. Go to your network's DHCP server, usually this is your Wireless Access Point + / WLAN Router. + * Login into its web-interface + * Find where DNS servers are defined + * Change all DNS fields to `192.168.1.10` +2. All local machines have to be rebooted. Without this they will continue to + use the old DNS setting from an old DHCP lease for quite some time. + +## Adding domain names + +Login to the Pi-hole web interface: `http://raspberrypi.local:8089/admin`: + +1. Select from Left menu: Local DNS -> DNS Records +2. Enter Domain: `raspberrypi.home.arpa` and IP Address: `192.168.1.10`. Press + Add. + +Now you can use `raspberrypi.home.arpa` as the domain name for the Raspberry Pi +in your whole local network. You can also add domain names for your other +devices, provided they too have static IPs. + +The Raspberry Pi itself must also use be configured to use the Pi-hole DNS +server. This is especially important when you add your own domains names, +otherwise DNS may work differently on the Pi than on other devices. Configure +this by running: +```bash +echo "name_servers=127.0.0.1" | sudo tee -a /etc/resolvconf.conf +echo "name_servers_append=8.8.8.8" | sudo tee -a /etc/resolvconf.conf +echo "resolv_conf_local_only=NO" | sudo tee -a /etc/resolvconf.conf +sudo resolvconf -u # Ignore "Too few arguments."-complaint +``` +Quick explanation: resolv_conf_local_only is disabled and a public nameserver +is added, so that in case the Pi-hole container is stopped, the Raspberry won't +lose DNS functionality. It will just fallback to 8.8.8.8. + +### Testing & Troubleshooting + +Install dig: +``` +apt install dnsutils +``` + +Test that Pi-hole is correctly configured (should respond 192.168.1.10): +``` +dig raspberrypi.home.arpa @192.168.1.10 +``` + +To test on your desktop if your network configuration is correct, and an ESP +will resolve its DNS queries correctly, restart your desktop machine to ensure +DNS changes are updated and then use: +``` +dig raspberrypi.home.arpa +``` +This should produce the same result as the previous command. + +If this fails to resolve the IP, check that the server in the response is +`192.168.1.10`. If it's `127.0.0.xx` check `/etc/resolv.conf` begins with +`nameserver 192.168.1.10`. + +## Why .home.arpa? + +Instead of `.home.arpa` - which is the real standard, but a mouthful - you may +use `.internal`. Using `.local` would technically also work, but it should be +reserved only for mDNS use. + +## Microcontrollers + +If you want to avoid hardcoding your Raspberry Pi IP to your ESPhome devices, +you need a DNS server that will do the resolving. This can be done using the +Pi-hole container as described above. + +!!! info "`*.local` won't work for ESPhome" + + There is a special case for resolving `*.local` addresses. If you do a + `ping raspberrypi.local` on your desktop linux or the RPI, it will first + try using mDNS/bonjour to resolve the IP address raspberrypi.local. If this + fails it will then ask the DNS server. Esphome devices can't use mDNS to + resolve an IP address. You need a proper DNS server to respond to queries + made by an ESP. As such, `dig raspberrypi.local` will fail, simulating + ESPhome device behavior. This is as intended, and you should use + raspberrypi.home.arpa as the address on your ESP-device. -To enable your router to use the pihole container edit your DNS settings on your router to point to your Pi's IP address \ No newline at end of file diff --git a/docs/Containers/Portainer-ce.md b/docs/Containers/Portainer-ce.md index 3ee4484b..dbcc386d 100644 --- a/docs/Containers/Portainer-ce.md +++ b/docs/Containers/Portainer-ce.md @@ -1,11 +1,11 @@ # Portainer CE -## References +## References - [Docker](https://hub.docker.com/r/portainer/portainer-ce/) - [Website](https://www.portainer.io/portainer-ce/) -## Definition +## Definition - "#yourip" means any of the following: @@ -13,7 +13,7 @@ - the multicast domain name of your Raspberry Pi (eg `iot-hub.local`) - the domain name of your Raspberry Pi (eg `iot-hub.mydomain.com`) -## About *Portainer CE* +## About *Portainer CE* *Portainer CE* (Community Edition) is an application for managing Docker. It is a successor to *Portainer*. According to [the *Portainer CE* documentation](https://www.portainer.io/2020/08/portainer-ce-2-0-what-to-expect/) @@ -21,7 +21,7 @@ From that it should be clear that *Portainer* is deprecated and that *Portainer CE* is the way forward. -## Installing *Portainer CE* +## Installing *Portainer CE* Run the menu: @@ -40,7 +40,7 @@ Ignore any message like this: > WARNING: Found orphan containers (portainer) for this project … -## First run of *Portainer CE* +## First run of *Portainer CE* In your web browser navigate to `#yourip:9000/`: @@ -51,7 +51,7 @@ From there, you can click on the "Local" group and take a look around. One of th There are 'Quick actions' to view logs and other stats. This can all be done from terminal commands but *Portainer CE* makes it easier. -## Setting the Public IP address for your end-point +## Setting the Public IP address for your end-point If you click on a "Published Port" in the "Containers" list, your browser may return an error saying something like "can't connect to server" associated with an IP address of "0.0.0.0". @@ -79,7 +79,7 @@ Keep in mind that clicking on a "Published Port" does not guarantee that your br > All things considered, you will get more consistent behaviour if you simply bookmark the URLs you want to use for your IOTstack services. -## If you forget your password +## If you forget your password If you forget the password you created for *Portainer CE*, you can recover by doing the following: @@ -92,5 +92,5 @@ $ docker-compose start portainer-ce Then, follow the steps in: -1. [First run of *Portainer CE*](#firstRun); and -2. [Setting the Public IP address for your end-point](#setPublicIP). +1. [First run of *Portainer CE*](#first-run-of-portainer-ce); and +2. [Setting the Public IP address for your end-point](#setting-the-public-ip-address-for-your-end-point). diff --git a/docs/Containers/Prometheus.md b/docs/Containers/Prometheus.md index 1bee676f..a8af53a0 100644 --- a/docs/Containers/Prometheus.md +++ b/docs/Containers/Prometheus.md @@ -1,6 +1,6 @@ # Prometheus -## References +## References * [*Prometheus* home](https://prometheus.io) * *GitHub*: @@ -15,7 +15,7 @@ - [*CAdvisor*](https://hub.docker.com/r/zcube/cadvisor) - [*Node Exporter*](https://hub.docker.com/r/prom/node-exporter) -## Overview +## Overview Prometheus is a collection of three containers: @@ -23,11 +23,11 @@ Prometheus is a collection of three containers: * *CAdvisor* * *Node Exporter* -The [default configuration](#activeConfig) for *Prometheus* supplied with IOTstack scrapes information from all three containers. +The [default configuration](#active-configuration-file) for *Prometheus* supplied with IOTstack scrapes information from all three containers. -## Installing Prometheus +## Installing Prometheus -### *if you are running New Menu …* +### *if you are running New Menu …* When you select *Prometheus* in the IOTstack menu, you must also select: @@ -36,7 +36,7 @@ When you select *Prometheus* in the IOTstack menu, you must also select: If you do not select all three containers, Prometheus will not start. -### *if you are running Old Menu …* +### *if you are running Old Menu …* When you select *Prometheus* in the IOTstack menu, the service definition includes the three containers: @@ -44,7 +44,7 @@ When you select *Prometheus* in the IOTstack menu, the service definition includ * *prometheus-cadvisor;* and * *prometheus-nodeexporter*. -## Significant directories and files +## Significant directories and files ``` ~/IOTstack @@ -75,25 +75,25 @@ When you select *Prometheus* in the IOTstack menu, the service definition includ 5. The *working service definition* (only relevant to old-menu, copied from ❶). 6. The *Compose* file (includes ❶). 7. The *persistent storage area*. -8. The [configuration directory](#configDir). +8. The [configuration directory](#configuration-directory). -## How *Prometheus* gets built for IOTstack +## How *Prometheus* gets built for IOTstack -### *Prometheus* source code ([*GitHub*](https://github.com)) +### *Prometheus* source code ([*GitHub*](https://github.com)) The source code for *Prometheus* lives at [*GitHub* prometheus/prometheus](https://github.com/prometheus/prometheus). -### *Prometheus* images ([*DockerHub*](https://hub.docker.com)) +### *Prometheus* images ([*DockerHub*](https://hub.docker.com)) Periodically, the source code is recompiled and the resulting image is pushed to [prom/prometheus](https://hub.docker.com/r/prom/prometheus) on *DockerHub*. -### IOTstack menu +### IOTstack menu When you select *Prometheus* in the IOTstack menu, the *template service definition* is copied into the *Compose* file. > Under old menu, it is also copied to the *working service definition* and then not really used. -### IOTstack first run +### IOTstack first run On a first install of IOTstack, you run the menu, choose *Prometheus* as one of your containers, and are told to do this: @@ -124,7 +124,7 @@ The *Dockerfile* begins with: FROM prom/prometheus:latest ``` -> If you need to pin to a particular version of *Prometheus*, the *Dockerfile* is the place to do it. See [*Prometheus* version pinning](#versionPinning). +> If you need to pin to a particular version of *Prometheus*, the *Dockerfile* is the place to do it. See [*Prometheus* version pinning](#prometheus-version-pinning). The `FROM` statement tells the build process to pull down the ***base image*** from [*DockerHub*](https://hub.docker.com). @@ -158,15 +158,15 @@ You *may* see the same pattern in Portainer, which reports the *base image* as " > Whether you see one or two rows depends on the version of `docker-compose` you are using and how your version of `docker-compose` builds local images. -### Dependencies: *CAdvisor* and *Node Exporter* +### Dependencies: *CAdvisor* and *Node Exporter* The *CAdvisor* and *Node Exporter* are included in the *Prometheus* service definition as dependent containers. What that means is that each time you start *Prometheus*, `docker-compose` ensures that *CAdvisor* and *Node Exporter* are already running, and keeps them running. -The [default configuration](#activeConfig) for *Prometheus* assumes *CAdvisor* and *Node Exporter* are running and starts scraping information from those targets as soon as it launches. +The [default configuration](#active-configuration-file) for *Prometheus* assumes *CAdvisor* and *Node Exporter* are running and starts scraping information from those targets as soon as it launches. -## Configuring **Prometheus** +## Configuring **Prometheus** -### Configuration directory +### Configuration directory The configuration directory for the IOTstack implementation of *Prometheus* is at the path: @@ -181,9 +181,9 @@ That directory contains two files: If you delete either file, *Prometheus* will replace it with a default the next time the container starts. This "self-repair" function is intended to provide reasonable assurance that *Prometheus* will at least **start** instead of going into a restart loop. -Unless you [decide to change it](#environmentVars), the `config` folder and its contents are owned by "pi:pi". This means you can edit the files in the configuration directory without needing the `sudo` command. Ownership is enforced each time the container restarts. +Unless you [decide to change it](#environment-variables), the `config` folder and its contents are owned by "pi:pi". This means you can edit the files in the configuration directory without needing the `sudo` command. Ownership is enforced each time the container restarts. -#### Active configuration file +#### Active configuration file The file named `config.yml` is the active configuration. This is the file you should edit if you want to make changes. The default structure of the file is: @@ -213,7 +213,7 @@ Note: * The YAML parser used by *Prometheus* seems to be ***exceptionally*** sensitive to syntax errors (far less tolerant than `docker-compose`). For this reason, you should **always** check the *Prometheus* log after any configuration change. -#### Reference configuration file +#### Reference configuration file The file named `prometheus.yml` is a reference configuration. It is a **copy** of the original configuration file that ships inside the *Prometheus* container at the path: @@ -231,7 +231,7 @@ $ docker-compose restart prometheus $ docker logs prometheus ``` -### Environment variables +### Environment variables The IOTstack implementation of *Prometheus* supports two environment variables: @@ -241,11 +241,11 @@ environment: - IOTSTACK_GID=1000 ``` -Those variables control ownership of the [Configuration directory](#configDir) and its contents. Those environment variables are present in the standard IOTstack service definition for *Prometheus* and have the effect of assigning ownership to "pi:pi". +Those variables control ownership of the [Configuration directory](#configuration-directory) and its contents. Those environment variables are present in the standard IOTstack service definition for *Prometheus* and have the effect of assigning ownership to "pi:pi". -If you delete those environment variables from your *Compose* file, the [Configuration directory](#configDir) will be owned by "nobody:nobody"; otherwise the directory and its contents will be owned by whatever values you pass for those variables. +If you delete those environment variables from your *Compose* file, the [Configuration directory](#configuration-directory) will be owned by "nobody:nobody"; otherwise the directory and its contents will be owned by whatever values you pass for those variables. -### Migration considerations +### Migration considerations Under the original IOTstack implementation of *Prometheus* (just "as it comes" from *DockerHub*), the service definition expected the configuration file to be at: @@ -276,7 +276,7 @@ Note: * The YAML parser used by *Prometheus* is very sensitive to syntax errors. Always check the *Prometheus* log after any configuration change. -## Upgrading *Prometheus* +## Upgrading *Prometheus* You can update `cadvisor` and `nodeexporter` like this: @@ -320,7 +320,7 @@ The `prune` is the simplest way of cleaning up. The first call removes the old * > Whether an old *base image* exists depends on the version of `docker-compose` you are using and how your version of `docker-compose` builds local images. -### *Prometheus* version pinning +### *Prometheus* version pinning If you need to pin *Prometheus* to a particular version: diff --git a/docs/Containers/Python.md b/docs/Containers/Python.md index 24e613bb..7d5dc49c 100644 --- a/docs/Containers/Python.md +++ b/docs/Containers/Python.md @@ -1,12 +1,12 @@ # Python -## references +## references * [Python.org](https://www.python.org) * [Dockerhub image library](https://hub.docker.com/_/python) * [GitHub docker-library/python](https://github.com/docker-library/python) -## selecting Python in the IOTstack menu +## selecting Python in the IOTstack menu When you select Python in the menu: @@ -40,15 +40,9 @@ When you select Python in the menu: # - "external:internal" volumes: - ./volumes/python/app:/usr/src/app - networks: - - iotstack_nw ``` - Note: - - * This service definition is for "new menu" (master branch). The only difference with "old menu" (old-menu branch) is the omission of the last two lines. - -### customising your Python service definition +### customising your Python service definition The service definition contains a number of customisation points: @@ -76,7 +70,7 @@ $ cd ~/IOTstack $ docker-compose up -d python ``` -## Python - first launch +## Python - first launch After running the menu, you are told to run the commands: @@ -145,7 +139,7 @@ This is what happens: Pressing control+c terminates the log display but does not terminate the running container. -## stopping the Python service +## stopping the Python service To stop the container from running, either: @@ -163,7 +157,7 @@ To stop the container from running, either: $ docker-compose rm --force --stop -v python ``` -## starting the Python service +## starting the Python service To bring up the container again after you have stopped it, either: @@ -181,7 +175,7 @@ To bring up the container again after you have stopped it, either: $ docker-compose up -d python ``` -## Python - second-and-subsequent launch +## Python - second-and-subsequent launch Each time you launch the Python container *after* the first launch: @@ -189,7 +183,7 @@ Each time you launch the Python container *after* the first launch: 2. The `docker-entrypoint.sh` script runs and performs "self-repair" by replacing any files that have gone missing from the persistent storage area. Self-repair does **not** overwrite existing files! 3. The `app.py` Python script is run. -## when things go wrong - check the log +## when things go wrong - check the log If the container misbehaves, the log is your friend: @@ -197,7 +191,7 @@ If the container misbehaves, the log is your friend: $ docker logs python ``` -## project development life-cycle +## project development life-cycle It is **critical** that you understand that **all** of your project development should occur within the folder: @@ -207,7 +201,7 @@ It is **critical** that you understand that **all** of your project development So long as you are performing some sort of routine backup (either with a supplied script or a third party solution like [Paraphraser/IOTstackBackup](https://github.com/Paraphraser/IOTstackBackup)), your work will be protected. -### getting started +### getting started Start by editing the file: @@ -228,7 +222,7 @@ $ cd ~/IOTstack $ docker-compose restart python ``` -### reading and writing to disk +### reading and writing to disk Consider this line in the service definition: @@ -255,7 +249,7 @@ What it means is that: If your script writes into any other directory inside the container, the data will be lost when the container re-launches. -### getting a clean slate +### getting a clean slate If you make a mess of things and need to start from a clean slate, erase the persistent storage area: @@ -268,7 +262,7 @@ $ docker-compose up -d python The container will re-initialise the persistent storage area from its defaults. -### adding packages +### adding packages As you develop your project, you may find that you need to add supporting packages. For this example, we will assume you want to add "[Flask](https://pypi.org/project/Flask/)" and "[beautifulsoup4](https://pypi.org/project/beautifulsoup4/)". @@ -322,7 +316,7 @@ To make *Flask* and *beautifulsoup4* a permanent part of your container: Flask==2.0.1 ``` -5. Continue your development work by returning to [getting started](#gettingStarted). +5. Continue your development work by returning to [getting started](#getting-started). Note: @@ -346,11 +340,11 @@ Note: The `requirements.txt` file will be recreated and it will be a copy of the version in the *services* directory as of the last image rebuild. -### making your own Python script the default +### making your own Python script the default -Suppose the Python script you have been developing reaches a major milestone and you decide to "freeze dry" your work up to that point so that it becomes the default when you ask for a [clean slate](#cleanSlate). Proceed like this: +Suppose the Python script you have been developing reaches a major milestone and you decide to "freeze dry" your work up to that point so that it becomes the default when you ask for a [clean slate](#getting-a-clean-slate). Proceed like this: -1. If you have added any packages by following the steps in [adding packages](#addingPackages), run the following command: +1. If you have added any packages by following the steps in [adding packages](#adding-packages), run the following command: ```bash $ docker exec python bash -c 'pip3 freeze >requirements.txt' @@ -412,11 +406,11 @@ Suppose the Python script you have been developing reaches a major milestone and $ docker system prune -f ``` -### canning your project +### canning your project Suppose your project has reached the stage where you wish to put it into production as a service under its own name. Make two further assumptions: -1. You have gone through the steps in [making your own Python script the default](#scriptBaking) and you are **certain** that the content of `./services/python/app` correctly captures your project. +1. You have gone through the steps in [making your own Python script the default](#making-your-own-python-script-the-default) and you are **certain** that the content of `./services/python/app` correctly captures your project. 2. You want to give your project the name "wishbone". Proceed like this: @@ -456,8 +450,6 @@ Proceed like this: # - "external:internal" # - "external:internal" volumes: volumes: - ./volumes/python/app:/usr/src/app | - ./volumes/wishbone/app:/usr/src/app - networks: networks: - - iotstack_nw - iotstack_nw ``` Note: @@ -479,7 +471,7 @@ Remember: ~/IOTstack/volumes/wishbone/app ``` -## routine maintenance +## routine maintenance To make sure you are running from the most-recent **base** image of Python from Dockerhub: @@ -503,4 +495,4 @@ The old base image can't be removed until the old local image has been removed, Note: -* If you have followed the steps in [canning your project](#scriptCanning) and your service has a name other than `python`, just substitute the new name where you see `python` in the two `dockerc-compose` commands. +* If you have followed the steps in [canning your project](#canning-your-project) and your service has a name other than `python`, just substitute the new name where you see `python` in the two `dockerc-compose` commands. diff --git a/docs/Containers/Telegraf.md b/docs/Containers/Telegraf.md index 7c07fac1..b0511ef5 100644 --- a/docs/Containers/Telegraf.md +++ b/docs/Containers/Telegraf.md @@ -7,13 +7,13 @@ The purpose of the Dockerfile is to: * tailor the default configuration to be IOTstack-ready; and * enable the container to perform self-repair if essential elements of the persistent storage area disappear. -## References +## References - [*influxdata Telegraf* home](https://www.influxdata.com/time-series-platform/telegraf/) - [*GitHub*: influxdata/influxdata-docker/telegraf](https://github.com/influxdata/influxdata-docker/tree/master/telegraf) - [*DockerHub*: influxdata Telegraf](https://hub.docker.com/_/telegraf) -## Significant directories and files +## Significant directories and files ``` ~/IOTstack @@ -38,14 +38,14 @@ The purpose of the Dockerfile is to: 1. The *Dockerfile* used to customise Telegraf for IOTstack. 2. A replacement for the `telegraf` container script of the same name, extended to handle container self-repair. -3. The *additions folder*. See [Applying optional additions](#optionalAdditions). +3. The *additions folder*. See [Applying optional additions](#applying-optional-additions). 4. The *auto_include folder*. Additions automatically applied to - `telegraf.conf`. See [Automatic includes to telegraf.conf](#autoInclude). + `telegraf.conf`. See [Automatic includes to telegraf.conf](#automatic-includes-to-telegrafconf). 5. The *template service definition*. 6. The *working service definition* (only relevant to old-menu, copied from ❹). 7. The *persistent storage area* for the `telegraf` container. -8. A working copy of the *additions folder* (copied from ❸). See [Applying optional additions](#optionalAdditions). -9. The *reference configuration file*. See [Changing Telegraf's configuration](#editConfiguration). +8. A working copy of the *additions folder* (copied from ❸). See [Applying optional additions](#applying-optional-additions). +9. The *reference configuration file*. See [Changing Telegraf's configuration](#changing-telegrafs-configuration). 10. The *active configuration file*. A subset of ➒ altered to support communication with InfluxDB running in a container in the same IOTstack instance. Everything in the persistent storage area ❼: @@ -53,19 +53,19 @@ Everything in the persistent storage area ❼: * will be replaced if it is not present when the container starts; but * will never be overwritten if altered by you. -## How Telegraf gets built for IOTstack +## How Telegraf gets built for IOTstack -### Telegraf images ([*DockerHub*](https://hub.docker.com)) +### Telegraf images ([*DockerHub*](https://hub.docker.com)) Periodically, the source code is recompiled and the resulting image is pushed to [influxdata Telegraf](https://hub.docker.com/_/telegraf?tab=tags&page=1&ordering=last_updated) on *DockerHub*. -### IOTstack menu +### IOTstack menu When you select Telegraf in the IOTstack menu, the *template service definition* is copied into the *Compose* file. > Under old menu, it is also copied to the *working service definition* and then not really used. -### IOTstack first run +### IOTstack first run On a first install of IOTstack, you run the menu, choose your containers, and are told to do this: @@ -74,7 +74,7 @@ $ cd ~/IOTstack $ docker-compose up -d ``` -> See also the [Migration considerations](#migration) (below). +> See also the [Migration considerations](#migration-considerations) (below). `docker-compose` reads the *Compose* file. When it arrives at the `telegraf` fragment, it finds: @@ -99,7 +99,7 @@ The *Dockerfile* begins with: FROM telegraf:latest ``` -> If you need to pin to a particular version of Telegraf, the *Dockerfile* is the place to do it. See [Telegraf version pinning](#versionPinning). +> If you need to pin to a particular version of Telegraf, the *Dockerfile* is the place to do it. See [Telegraf version pinning](#telegraf-version-pinning). The `FROM` statement tells the build process to pull down the ***base image*** from [*DockerHub*](https://hub.docker.com). @@ -134,7 +134,7 @@ You *may* see the same pattern in *Portainer*, which reports the ***base image** > Whether you see one or two rows depends on the version of `docker-compose` you are using and how your version of `docker-compose` builds local images. -### Migration considerations +### Migration considerations Under the original IOTstack implementation of Telegraf (just "as it comes" from *DockerHub*), the service definition expected `telegraf.conf` to be at: @@ -154,9 +154,9 @@ With one exception, all prior and current versions of the default configuration > In other words, once you strip away comments and blank lines, and remove any "active" configuration options that simply repeat their default setting, you get the same subset of "active" configuration options. The default configuration file supplied with gcgarner/IOTstack is available [here](https://github.com/gcgarner/IOTstack/blob/master/.templates/telegraf/telegraf.conf) if you wish to refer to it. -The exception is `[[inputs.mqtt_consumer]]` which is now provided as an optional addition. If your existing Telegraf configuration depends on that input, you will need to apply it. See [applying optional additions](#optionalAdditions). +The exception is `[[inputs.mqtt_consumer]]` which is now provided as an optional addition. If your existing Telegraf configuration depends on that input, you will need to apply it. See [applying optional additions](#applying-optional-additions). -## Logging +## Logging You can inspect Telegraf's log by: @@ -166,7 +166,7 @@ $ docker logs telegraf These logs are ephemeral and will disappear when your Telegraf container is rebuilt. -### log message: *database "telegraf" creation failed* +### log message: *database "telegraf" creation failed* The following log message can be misleading: @@ -178,7 +178,7 @@ If InfluxDB is not running when Telegraf starts, the `depends_on:` clause in Tel What this error message *usually* means is that Telegraf has tried to communicate with InfluxDB before the latter is ready to accept connections. Telegraf typically retries after a short delay and is then able to communicate with InfluxDB. -## Changing Telegraf's configuration +## Changing Telegraf's configuration The first time you launch the Telegraf container, the following structure will be created in the persistent storage area: @@ -204,7 +204,7 @@ The file: - is created by removing all comment lines and blank lines from `telegraf-reference.conf`, leaving only the "active" configuration options, and then adding options necessary for IOTstack. - is less than 30 lines and is significantly easier to understand than `telegraf-reference.conf`. -* `inputs.docker.conf` – see [Applying optional additions](#optionalAdditions) below. +* `inputs.docker.conf` – see [Applying optional additions](#applying-optional-additions) below. The intention of this structure is that you: @@ -219,7 +219,7 @@ $ cd ~/IOTstack $ docker-compose restart telegraf ``` -### Automatic includes to telegraf.conf +### Automatic includes to telegraf.conf * `inputs.docker.conf` instructs Telegraf to collect metrics from Docker. Requires kernel control groups to be enabled to collect memory usage data. If not done during initial installation, @@ -229,9 +229,9 @@ $ docker-compose restart telegraf ``` * `inputs.cpu_temp.conf' collects cpu temperature. -### Applying optional additions +### Applying optional additions -The *additions folder* (see [Significant directories and files](#significantFiles)) is a mechanism for additional *IOTstack-ready* configuration options to be provided for Telegraf. +The *additions folder* (see [Significant directories and files](#significant-directories-and-files)) is a mechanism for additional *IOTstack-ready* configuration options to be provided for Telegraf. Currently there is one addition: @@ -249,9 +249,9 @@ $ docker-compose restart telegraf The `grep` strips comment lines and the `sudo tee` is a safe way of appending the result to `telegraf.conf`. The `restart` causes Telegraf to notice the change. -## Getting a clean slate +## Getting a clean slate -### Erasing the persistent storage area +### Erasing the persistent storage area Erasing Telegraf's persistent storage area triggers self-healing and restores known defaults: @@ -272,7 +272,7 @@ Note: $ docker-compose restart telegraf ``` -### Resetting the InfluxDB database +### Resetting the InfluxDB database To reset the InfluxDB database that Telegraf writes into, proceed like this: @@ -293,7 +293,7 @@ In words: * Delete the `telegraf` database, and then exit the CLI. * Start the Telegraf container. This re-creates the database automatically. -## Upgrading Telegraf +## Upgrading Telegraf You can update most containers like this: @@ -335,7 +335,7 @@ Your existing Telegraf container continues to run while the rebuild proceeds. On The `prune` is the simplest way of cleaning up. The first call removes the old ***local image***. The second call cleans up the old ***base image***. Whether an old ***base image*** exists depends on the version of `docker-compose` you are using and how your version of `docker-compose` builds local images. -### Telegraf version pinning +### Telegraf version pinning If you need to pin Telegraf to a particular version: diff --git a/docs/Containers/WireGuard.md b/docs/Containers/WireGuard.md index 53d0991a..ba73d6de 100644 --- a/docs/Containers/WireGuard.md +++ b/docs/Containers/WireGuard.md @@ -11,27 +11,27 @@ Assumptions: * These instructions assume that you have privileges to configure your network's gateway (router). If you are not able to make changes to your network's firewall settings, then you will not be able to finish this setup. * In common with most VPN technologies, WireGuard assumes that the WAN side of your network's gateway has a public IP address which is reachable directly. WireGuard may not work if that assumption does not hold. If you strike this problem, you have to take it up with your ISP. -## Installing WireGuard under IOTstack +## Installing WireGuard under IOTstack You increase your chances of a trouble-free installation by performing the installation steps in the following order. -### Step 1: Update your Raspberry Pi OS +### Step 1: Update your Raspberry Pi OS -To be able to run WireGuard successfully, your Raspberry Pi needs to be **fully** up-to-date. If you want to understand why, see [the read only flag](#readOnlyFlag). +To be able to run WireGuard successfully, your Raspberry Pi needs to be **fully** up-to-date. If you want to understand why, see [the read only flag](#the-read-only-flag). ```bash $ sudo apt update $ sudo apt upgrade -y ``` -### Step 2: Set up a Dynamic DNS name +### Step 2: Set up a Dynamic DNS name Before you can use WireGuard (or any VPN solution), you need a mechanism for your remote clients to reach your home router. You have two choices: 1. Obtain a permanent IP address for your home router from your Internet Service Provider (ISP). Approach your ISP if you wish to pursue this option. It generally involves additional charges. -2. Use a Dynamic DNS service. See IOTstack documentation [Accessing your device from the internet](https://sensorsiot.github.io/IOTstack/Accessing-your-Device-from-the-internet.html). The rest of this documentation assumes you have chosen this option. +2. Use a Dynamic DNS service. See IOTstack documentation [Accessing your device from the internet](../Basic_setup/Accessing-your-Device-from-the-internet.md). The rest of this documentation assumes you have chosen this option. -### Step 3: Understand the Service Definition +### Step 3: Understand the Service Definition This is the service definition *template* that IOTstack uses for WireGuard: @@ -67,11 +67,10 @@ Unfortunately, that service definition will not work "as is". It needs to be con Key points: * Everything in the `environment:` section from `SERVERURL=` down to `PEERDNS=` (inclusive) affects WireGuard's generated configurations (the QR codes). In other words, any time you change any of those values, any existing QR codes will stop working. -* WireGuard does not need to communicate directly with other Docker containers so there is no need for it to join `iotstack_nw`. -### Step 4: Decide what to configure +### Step 4: Decide what to configure -With most containers, you can continue to tweak environment variables and settings without upsetting the container's basic behaviour. WireGuard is a little different. You really need to think, carefully, about how you want to configure the service before you start. If you change your mind later, you generally have to [start from a clean slate](#cleanSlate). +With most containers, you can continue to tweak environment variables and settings without upsetting the container's basic behaviour. WireGuard is a little different. You really need to think, carefully, about how you want to configure the service before you start. If you change your mind later, you generally have to [start from a clean slate](#getting-a-clean-slate). #### Fields that you should always configure @@ -101,17 +100,32 @@ With most containers, you can continue to tweak environment variables and settin - Many examples on the web use "PEERS=n" where "n" is a number. In practice, that approach seems to be a little fragile and is not recommended for IOTstack. -#### Optional configuration - DNS resolution for peers +#### Optional configuration - DNS resolution for peers You have several options for how your remote peers resolve DNS requests: * `PEERDNS=auto` - The default value of `auto` instructs the WireGuard *service* running within the WireGuard *container* to use the same DNS as the WireGuard *container* when resolving requests from connected peers. In practice, that means the *service* directs queries to 127.0.0.11, which Docker intercepts and forwards to whichever resolvers are specified in the Raspberry Pi's `/etc/resolv.conf`. + DNS queries made on connected WireGuard clients should work as if they were + made on the host. If you [configure](Pi-hole.md#pi-hole-as-dns-server) your + ad-blocker into the host's `resolveconf.conf`, Wireguard clients will also + automatically use it. -* `PEERDNS=auto` with `custom-cont-init` + Details: The default value of `auto` instructs the WireGuard *service* + running within the WireGuard *container* to use a DNS-service, coredns, + also running in the Wireguard container. Coredns by default directs queries + to 127.0.0.11, which Docker intercepts and forwards to whichever resolvers + are specified in the Raspberry Pi's `/etc/resolv.conf`. - This configuration instructs WireGuard to forward DNS queries from remote peers to any **container** which is listening on port 53. This is the option you will want to choose if you are running an ad-blocking DNS server (eg *PiHole* or *AdGuardHome*) in a container on the same host as WireGuard, and you want your remote clients to obtain DNS resolution via the ad-blocker. +* `PEERDNS=auto` with `custom-cont-init` + + This configuration instructs WireGuard to forward DNS queries from remote + peers to any host daemon or **container** which is listening on port 53. + This is the option you will want to choose if you are running an + ad-blocking DNS server (eg *PiHole* or *AdGuardHome*) in a container on the + same host as WireGuard, and you want your remote clients to obtain DNS + resolution via the ad-blocker, but don't want your Raspberry Pi host to use + it. > Acknowledgement: thanks to @ukkopahis for developing this option. @@ -161,8 +175,11 @@ You have several options for how your remote peers resolve DNS requests: ```yml - PEERDNS=192.168.203.65 ``` + Do note that changes to `PEERDNS` will not be updated to existing clients, + and as such you may want to use `PEERDNS=auto` unless you have a very + specific requirement. -#### Optional configuration - WireGuard ports +#### Optional configuration - WireGuard ports The WireGuard service definition template follows the convention of using UDP port "51820" in three places. You can leave it like that and it will just work. There is no reason to change the defaults unless you want to. @@ -189,15 +206,15 @@ Rule #1: Rule #2: -* The *«public»* port forms part of the QR codes. If you decide to change the *«public»* port after you generate the QR codes, you will have to [start over from a clean slate](#cleanSlate). +* The *«public»* port forms part of the QR codes. If you decide to change the *«public»* port after you generate the QR codes, you will have to [start over from a clean slate](#getting-a-clean-slate). Rule #3: * Your router needs to know about both the *«public»* and *«external»* ports so, if you decide to change either of those, you must also reconfigure your router. -See [Understanding WireGuard's port numbers](#understandingPorts) if you want more information on how the various port numbers are used. +See [Understanding WireGuard's port numbers](#understanding-wireguards-port-numbers) if you want more information on how the various port numbers are used. -### Step 5: Configure WireGuard +### Step 5: Configure WireGuard There are two approaches: @@ -206,7 +223,7 @@ There are two approaches: Of the two, the first is generally the simpler and means you don't have to re-run the menu whenever you want to change WireGuard's configuration. -#### Method 1: Configure WireGuard by editing `docker-compose.yml` +#### Method 1: Configure WireGuard by editing `docker-compose.yml` 1. Run the menu: @@ -221,12 +238,12 @@ Of the two, the first is generally the simpler and means you don't have to re-ru 5. Choose Exit. 6. Open `docker-compose.yml` in your favourite text editor. 7. Navigate to the WireGuard service definition. -8. Implement the decisions you took in [decide what to configure](#configureWhat). +8. Implement the decisions you took in [decide what to configure](#step-4-decide-what-to-configure). 9. Save your work. -#### Method 2: Configure WireGuard using `compose-override.yml` +#### Method 2: Configure WireGuard using `compose-override.yml` -The [Custom services and overriding default settings for IOTstack](https://sensorsiot.github.io/IOTstack/Custom/) page describes how to use an override file to allow the menu to incorporate your custom configurations into the final `docker-compose.yml` file. +The [Custom services and overriding default settings for IOTstack](../Basic_setup/Custom.md) page describes how to use an override file to allow the menu to incorporate your custom configurations into the final `docker-compose.yml` file. You will need to create the `compose-override.yml` **before** running the menu to build your stack. If you have already built your stack, you'll have to rebuild it after creating `compose-override.yml`. @@ -236,7 +253,7 @@ You will need to create the `compose-override.yml` **before** running the menu t ~/IOTstack/compose-override.yml ``` -2. Define overrides to implement the decisions you took in [Decide what to configure](#configureWhat). For example: +2. Define overrides to implement the decisions you took in [Decide what to configure](#step-4-decide-what-to-configure). For example: ```yml services: @@ -277,7 +294,7 @@ You will need to create the `compose-override.yml` **before** running the menu t and verify that the `wireguard` service definition is as you expect. -### Step 6: Start WireGuard +### Step 6: Start WireGuard 1. To start WireGuard, bring up your stack: @@ -298,7 +315,7 @@ You will need to create the `compose-override.yml` **before** running the menu t $ docker logs wireguard ``` - See also discussion of [the read-only flag](#readOnlyFlag). + See also discussion of [the read-only flag](#the-read-only-flag). 3. Confirm that WireGuard has generated the expected configurations. For example, given the following setting in `docker-compose.yml`: @@ -341,7 +358,7 @@ You will need to create the `compose-override.yml` **before** running the menu t Notice how each element in the `PEERS=` list is represented by a sub-directory prefixed with `peer_`. You should expect the same pattern for your peers. -### Step 7: Save your WireGuard client configuration files (QR codes) +### Step 7: Save your WireGuard client configuration files (QR codes) The first time you launch WireGuard, it generates cryptographically protected configurations for your remote clients and encapsulates those configurations in QR codes. You can see the QR codes by running: @@ -382,7 +399,7 @@ In this case: Keep in mind that each QR code contains everything needed for **any** device to access your home network via WireGuard. Treat your `.png` files as "sensitive documents". -### Step 8: Configure your router with a NAT rule +### Step 8: Configure your router with a NAT rule A typical home network will have a firewall that effectively blocks all incoming attempts from the Internet to open a new connection with a device on your network. @@ -427,7 +444,7 @@ A typical configuration process goes something like this: * *Public Port* or *External Port* needs to be the value you chose for «public» in the WireGuard service definition (51820 if you didn't change it). * *Service Name* (or *Service Type*) is typically a text field, an editable menu (where you can either make a choice or type your own value), or a button approximating an editable menu. If you are given the option of choosing "WireGuard", do that, otherwise just type that name into the field. It has no significance other than reminding you what the rule is for. -### Step 9: Configure your remote WireGuard clients +### Step 9: Configure your remote WireGuard clients This is a massive topic and one which is well beyond the scope of this guide. You really will have to work it out for yourself. Start by Googling: @@ -443,7 +460,7 @@ For portable devices (eg iOS and Android) it usually boils down to: 4. Point the device's camera at the QR code. 5. Follow your nose. -## Understanding WireGuard's port numbers +## Understanding WireGuard's port numbers Here's a concrete example configuration using three different port numbers: @@ -466,7 +483,7 @@ You also need to make a few assumptions: 1. The host running the remote WireGuard client (eg a mobile phone with the WireGuard app installed) has been allocated the IP address 55.66.77.88 when it connected to the Internet over 3G/4G/5G. 2. When the remote WireGuard client initiated the session, it chose UDP port 44524 as its source port. The actual number chosen is (essentially) random and only significant to the client. 3. Your Internet Service Provider allocated the IP address 12.13.14.15 to the WAN side of your router. -4. You have done all the steps in [Set up a Dynamic DNS name](#obtainDDNS) and your WAN IP address (12.13.14.15) is being propagated to your Dynamic DNS service provider. +4. You have done all the steps in [Set up a Dynamic DNS name](#step-2-set-up-a-dynamic-dns-name) and your WAN IP address (12.13.14.15) is being propagated to your Dynamic DNS service provider. Here's a reference model to help explain what occurs: @@ -494,9 +511,9 @@ Even if you use port 51820 everywhere (the default), all this Network Address Tr This model is a slight simplification because the remote client may also be also operating behind a router performing Network Address Translation. It is just easier to understand the basic concepts if you assume the remote client has a publicly-routable IP address. -## Debugging techniques +## Debugging techniques -### Monitor WireGuard traffic between your router and your Raspberry Pi +### Monitor WireGuard traffic between your router and your Raspberry Pi If `tcpdump` is not installed on your Raspberry Pi, you can install it by: @@ -512,7 +529,7 @@ $ sudo tcpdump -i eth0 -n udp port «external» Press ctrlc to terminate the capture. -### Monitor WireGuard traffic between your Raspberry Pi and the WireGuard container +### Monitor WireGuard traffic between your Raspberry Pi and the WireGuard container First, you need to add `tcpdump` to the container. You only need to do this once per debugging session. The package will remain in place until the next time you re-create the container. @@ -528,7 +545,7 @@ $ docker exec wireguard tcpdump -i eth0 -n udp port «internal» Press ctrlc to terminate the capture. -### Is Docker listening on the Raspberry Pi's «external» port? +### Is Docker listening on the Raspberry Pi's «external» port? ```bash $ PORT=«external»; sudo nmap -sU -p $PORT 127.0.0.1 | grep "$PORT/udp" @@ -541,7 +558,7 @@ There will be a short delay. The expected answer is either: Success implies that the container is also listening. -### Is your router listening on the «public» port? +### Is your router listening on the «public» port? ```bash $ PORT=«public»; sudo nmap -sU -p $PORT downunda.duckdns.org | grep "$PORT/udp" @@ -552,7 +569,7 @@ There will be a short delay. The expected answer is either: * `«public»/udp open|filtered unknown` = router is listening * `«public»/udp closed unknown` = router is not listening -## The read-only flag +## The read-only flag The `:ro` at the end of the following line in WireGuard's service definition means "read only": @@ -568,7 +585,7 @@ Writing into `/lib/modules` is not needed on a Raspberry Pi, providing that Rasp If WireGuard refuses to install and you have good reason to suspect that WireGuard may be trying to write to `/lib/modules` then you can *consider* removing the `:ro` flag and re-trying. Just be aware that WireGuard will likely be modifying your operating system. -## Updating WireGuard +## Updating WireGuard To update the WireGuard container: @@ -584,7 +601,7 @@ $ docker-compose up -d wireguard $ docker system prune ``` -## Getting a clean slate +## Getting a clean slate If WireGuard misbehaves, you can start over from a clean slate. You *may* also need to do this if you change any of the following environment variables: diff --git a/docs/Containers/x2go.md b/docs/Containers/X2go.md similarity index 100% rename from docs/Containers/x2go.md rename to docs/Containers/X2go.md diff --git a/docs/BuildStack-RandomPassword.md b/docs/Developers/BuildStack-RandomPassword.md similarity index 100% rename from docs/BuildStack-RandomPassword.md rename to docs/Developers/BuildStack-RandomPassword.md diff --git a/docs/BuildStack-Services.md b/docs/Developers/BuildStack-Services.md similarity index 100% rename from docs/BuildStack-Services.md rename to docs/Developers/BuildStack-Services.md diff --git a/docs/Menu-System.md b/docs/Developers/Menu-System.md similarity index 100% rename from docs/Menu-System.md rename to docs/Developers/Menu-System.md diff --git a/docs/PostBuild-Script.md b/docs/Developers/PostBuild-Script.md similarity index 81% rename from docs/PostBuild-Script.md rename to docs/Developers/PostBuild-Script.md index 2bc8430e..cd3142b1 100644 --- a/docs/PostBuild-Script.md +++ b/docs/Developers/PostBuild-Script.md @@ -2,7 +2,7 @@ The postbuild bash script allows for executing arbitrary execution of bash commands after the stack has been build. ## How to use -Place a file in the main directory called `postbuild.sh`. When the buildstack [build logic](https://sensorsiot.github.io/IOTstack/Menu-System) finishes, it'll execute the `postbuild.sh` script, passing in each service selected from the buildstack menu as a parameter. This script is run each time the buildstack logic runs. +Place a file in the main directory called `postbuild.sh`. When the buildstack [build logic](../Developers/Menu-System.md) finishes, it'll execute the `postbuild.sh` script, passing in each service selected from the buildstack menu as a parameter. This script is run each time the buildstack logic runs. ## Updates The `postbuild.sh` file has been added to gitignore, so it won't be updated by IOTstack when IOTstack is updated. It has also been added to the backup script so that it will be backed up with your personal IOTstack backups. diff --git a/docs/Contributing-Services.md b/docs/Developers/index.md similarity index 51% rename from docs/Contributing-Services.md rename to docs/Developers/index.md index 4f8c4ec7..25e5e2cb 100644 --- a/docs/Contributing-Services.md +++ b/docs/Developers/index.md @@ -1,6 +1,23 @@ -# Contributing a service to IOTstack +# Contributing -On this page you can find information on how to contribute a service to IOTstack. We are generally very accepting of new services where they are useful. Keep in mind that if it is not IOTstack, selfhosted, or automation related we may not approve the PR. +## Writing documentation + +Documentation is is written as markdown, processed using mkdocs ([docs](https://www.mkdocs.org/user-guide/writing-your-docs/#writing-your-docs)) and the Material theme ([docs](https://squidfunk.github.io/mkdocs-material/reference/)). The Material theme is not just styling, but provides additional syntax extensions. + +Setup your system for mkdocs and Material: +``` +pip3 install -r requirements-mkdocs.txt +``` + +To test your local changes while writing them and before making a pull-request: +``` +cd ~/IOTstack +mkdocs serve +``` + +## Creating a new service + +In this section you can find information on how to contribute a service to IOTstack. We are generally very accepting of new services where they are useful. Keep in mind that if it is not IOTstack, selfhosted, or automation related we may not approve the PR. Services will grow over time, we may split up the buildstack menu into subsections or create filters to make organising all the services we provide easier to find. @@ -9,8 +26,8 @@ Services will grow over time, we may split up the buildstack menu into subsectio * `build.py` file is correct * Service allows for changing external WUI port from Build Stack's options menu if service uses a HTTP/S port * Use a default password, or allow the user to generate a random password for the service for initial installation. If the service asks to setup an account this can be ignored. -* Ensure [Default Configs](https://sensorsiot.github.io/IOTstack/Default-Configs) is updated with WUI port and username/password. -* Must detect port confilicts with other services on [BuildStack](https://sensorsiot.github.io/IOTstack/Menu-System) Menu. +* Ensure [Default Configs](../Basic_setup/Default-Configs.md) is updated with WUI port and username/password. +* Must detect port confilicts with other services on [BuildStack](Menu-System.md) Menu. * `Pre` and `Post` hooks work with no errors. * Does not require user to edit config files in order to get the service running. * Ensure that your service can be backed up and restored without errors or data loss. @@ -21,8 +38,9 @@ Services will grow over time, we may split up the buildstack menu into subsectio If your new service is approved and merged then congratulations! Please watch the Issues page on github over the next few days and weeks to see if any users have questions or issues with your new service. Links: -* [Default configs](https://sensorsiot.github.io/IOTstack/Default-Configs) -* [Password configuration for Services](https://sensorsiot.github.io/IOTstack/BuildStack-RandomPassword) -* [Build Stack Menu System](https://sensorsiot.github.io/IOTstack/Menu-System) -* [Coding a new service](https://sensorsiot.github.io/IOTstack/BuildStack-Services) -* [IOTstack issues](https://github.com/SensorsIot/IOTstack/issues) \ No newline at end of file + +* [Default configs](../Basic_setup/Default-Configs.md) +* [Password configuration for Services](BuildStack-RandomPassword.md) +* [Build Stack Menu System](Menu-System.md) +* [Coding a new service](BuildStack-Services.md) +* [IOTstack issues](htps://github.com/SensorsIot/IOTstack/issues) diff --git a/docs/Home.md b/docs/Home.md deleted file mode 100644 index d21ab031..00000000 --- a/docs/Home.md +++ /dev/null @@ -1,75 +0,0 @@ -# Wiki - -The README is moving to the Wiki, It's easier to add content and example to the Wiki vs the README.md - -* [Getting Started](https://sensorsiot.github.io/IOTstack/Containers/Getting-Started) -* [Updating the project](https://sensorsiot.github.io/IOTstack/Containers/Updating-the-Project) -* [How the script works](https://sensorsiot.github.io/IOTstack/Containers/How-the-script-works) -* [Understanding Containers](https://sensorsiot.github.io/IOTstack/Containers/Understanding-Containers) - -*** - -# Docker - -* [Commands](https://sensorsiot.github.io/IOTstack/Containers/Docker-commands) -* [Docker Networks](https://sensorsiot.github.io/IOTstack/Containers/Networking) - -*** - -# Containers -* [Portainer](https://sensorsiot.github.io/IOTstack/Containers/Portainer) -* [Portainer-ce](https://sensorsiot.github.io/IOTstack/Containers/Portainer-ce) -* [Portainer Agent](https://sensorsiot.github.io/IOTstack/Containers/Portainer-agent) -* [Node-RED](https://sensorsiot.github.io/IOTstack/Containers/Node-RED) -* [Grafana](https://sensorsiot.github.io/IOTstack/Containers/Grafana) -* [Mosquitto](https://sensorsiot.github.io/IOTstack/Containers/Mosquitto) -* [PostgreSQL](https://sensorsiot.github.io/IOTstack/Containers/PostgreSQL) -* [Adminer](https://sensorsiot.github.io/IOTstack/Containers/Adminer) -* [openHAB](https://sensorsiot.github.io/IOTstack/Containers/openHAB) -* [Home Assistant](https://sensorsiot.github.io/IOTstack/Containers/Home-Assistant) -* [Pi-Hole](https://sensorsiot.github.io/IOTstack/Containers/Pi-hole) -* [zigbee2MQTT](https://sensorsiot.github.io/IOTstack/Containers/Zigbee2MQTT) -* [Plex](https://sensorsiot.github.io/IOTstack/Containers/Plex) -* [TasmoAdmin](https://sensorsiot.github.io/IOTstack/Containers/TasmoAdmin) -* [RTL_433](https://sensorsiot.github.io/IOTstack/Containers/RTL_433-docker) -* [EspruinoHub (testing)](https://sensorsiot.github.io/IOTstack/Containers/EspruinoHub) -* [Next-Cloud](https://sensorsiot.github.io/IOTstack/Containers/NextCloud) -* [MariaDB](https://sensorsiot.github.io/IOTstack/Containers/MariaDB) -* [MotionEye](https://sensorsiot.github.io/IOTstack/Containers/MotionEye) -* [Blynk Server](https://sensorsiot.github.io/IOTstack/Containers/Blynk_server) -* [diyHue](https://sensorsiot.github.io/IOTstack/Containers/diyHue) -* [Python](https://sensorsiot.github.io/IOTstack/Containers/Python) -* [Heimdall](https://sensorsiot.github.io/IOTstack/Containers/Heimdall) -* [DashMachine](https://sensorsiot.github.io/IOTstack/Containers/DashMachine) -* [Homer](https://sensorsiot.github.io/IOTstack/Containers/Homer) -* [Custom containers](https://sensorsiot.github.io/IOTstack/Containers/Custom) - -*** - -# Native installs - -* [RTL_433](https://sensorsiot.github.io/IOTstack/Containers/Native-RTL_433) -* [RPIEasy](https://sensorsiot.github.io/IOTstack/Containers/RPIEasy_native) - -*** - -# Backups - -* [Docker backups](https://sensorsiot.github.io/IOTstack/Containers/Backups) -* Recovery (coming soon) -*** - -# Remote Access - -* [VPN and Dynamic DNS](https://sensorsiot.github.io/IOTstack/Containers/Accessing-your-Device-from-the-internet) -* [x2go](https://sensorsiot.github.io/IOTstack/Containers/x2go) - -*** - -# Miscellaneous - -* [log2ram](https://sensorsiot.github.io/IOTstack/Containers/Misc) -* [Dropbox-Uploader](https://sensorsiot.github.io/IOTstack/Containers/Misc) - -*** - diff --git a/docs/New-Menu-Release-Notes.md b/docs/Updates/New-Menu-Release-Notes.md similarity index 91% rename from docs/New-Menu-Release-Notes.md rename to docs/Updates/New-Menu-Release-Notes.md index 8176bd27..ab324a63 100644 --- a/docs/New-Menu-Release-Notes.md +++ b/docs/Updates/New-Menu-Release-Notes.md @@ -8,7 +8,8 @@ There are many features that are needing to be introduced into the new menu syst ## Breaking changes There are a few changes that you need to be aware of: -* Docker Environmental `*.env` files are no longer a thing by default. Everything needed is specified in the service.yml file, you can still optionally use them though either with [Custom Overrides](https://sensorsiot.github.io/IOTstack/Custom) or with the [PostBuild](https://sensorsiot.github.io/IOTstack/PostBuild-Script) script. Specific config files for certain services still work as they once did. + +* Docker Environmental `*.env` files are no longer a thing by default. Everything needed is specified in the service.yml file, you can still optionally use them though either with [Custom Overrides](../Basic_setup/Custom.md) or with the [PostBuild](../Developers/PostBuild-Script.md) script. Specific config files for certain services still work as they once did. * Python 3, pip3, PyYAML and Blessed are all required to be installed. * Not backwards compatible with old menu system. You will be able to switch back to the old menu system for a period of time by changing to the `old-menu` branch. It will be unmaintained except for critical updates. It will eventually be removed - but not before everyone is ready to leave it. @@ -26,4 +27,4 @@ There are a few changes that you need to be aware of: * Removed env files * Backup and restoring more streamlined * Documentation updated for all services -* No longer needs to be installed in the home directory `~`. \ No newline at end of file +* No longer needs to be installed in the home directory `~`. diff --git a/docs/gcgarner-migration.md b/docs/Updates/gcgarner-migration.md similarity index 90% rename from docs/gcgarner-migration.md rename to docs/Updates/gcgarner-migration.md index 2db53811..43178476 100644 --- a/docs/gcgarner-migration.md +++ b/docs/Updates/gcgarner-migration.md @@ -6,9 +6,9 @@ Migrating to SensorsIot/IOTstack was fairly easy when this repository was first The probability of conflicts developing increases as a function of time since the fork. Conflicts were and are pretty much inevitable so a more involved procedure is needed. -## Migration Steps +## Migration Steps -### Step 1 – Check your assumptions +### Step 1 – Check your assumptions Make sure that you are, *actually*, on gcgarner. Don't assume! @@ -20,7 +20,7 @@ origin https://github.com/gcgarner/IOTstack.git (push) Do not proceed if you don't see those URLs! -### Step 2 – Take IOTstack down +### Step 2 – Take IOTstack down Take your stack down. This is not *strictly* necessary but we'll be moving the goalposts a bit so it's better to be on the safe side. @@ -29,22 +29,22 @@ $ cd ~/IOTstack $ docker-compose down ``` -### Step 3 – Choose your migration method +### Step 3 – Choose your migration method There are two basic approaches to switching from gcgarner/IOTstack to SensorsIot/IOTstack: -- [Migration by changing upstream repository](#migrateChangeUpstream) -- [Migration by clone and merge](#migrateCloneMerge) +- [Migration by changing upstream repository](#migration-option-1-change-upstream-repository) +- [Migration by clone and merge](#migration-option-2-clone-and-merge) You can think of the first as "working *with* git" while the second is "using brute force". The first approach will work if you haven't tried any other migration steps and/or have not made too many changes to items in your gcgarner/IOTstack that are under git control. -If you are already stuck or you try the first approach and get a mess, or it all looks far too hard to sort out, then try the [Migration by clone and merge](#migrateCloneMerge) approach. +If you are already stuck or you try the first approach and get a mess, or it all looks far too hard to sort out, then try the [Migration by clone and merge](#migration-option-2-clone-and-merge) approach. -#### Migration Option 1 – change upstream repository +#### Migration Option 1 – change upstream repository -##### Check for local changes +##### Check for local changes Make sure you are on the master branch (you probably are so this is just a precaution), and then see if Git thinks you have made any local changes: @@ -93,7 +93,7 @@ The simplest way to deal with modified files is to rename them to move them out menu.sh.jqh ``` -##### Synchronise with gcgarner on GitHub +##### Synchronise with gcgarner on GitHub Make sure your local copy of gcgarner is in sync with GitHub. @@ -101,7 +101,7 @@ Make sure your local copy of gcgarner is in sync with GitHub. $ git pull ``` -##### Get rid of any upstream reference +##### Get rid of any upstream reference There may or may not be any "upstream" set. The most likely reason for this to happen is if you used your local copy as the basis of a Pull Request. @@ -111,7 +111,7 @@ The next command will probably return an error, which you should ignore. It's ju $ git remote remove upstream ``` -##### Point to SensorsIot +##### Point to SensorsIot Change your local repository to point to SensorsIot. @@ -119,7 +119,7 @@ Change your local repository to point to SensorsIot. $ git remote set-url origin https://github.com/SensorsIot/IOTstack.git ``` -##### Synchronise with SensorsIot on GitHub +##### Synchronise with SensorsIot on GitHub This is where things can get a bit tricky so please read these instructions carefully **before** you proceed. @@ -174,7 +174,7 @@ Auto-merging .templates/someRandomService/service.yml If you don't use `someRandomService` then you could safely ignore this on the basis that it was "probably right". However, if you did use that service and it started to misbehave after migration, you would know that the `service.yml` file was a good place to start looking for explanations. -##### Finish with a pull +##### Finish with a pull At this point, only the migrated master branch is present on your local copy of the repository. The next command brings you fully in-sync with GitHub: @@ -182,11 +182,11 @@ At this point, only the migrated master branch is present on your local copy of $ git pull ``` -#### Migration Option 2 – clone and merge +#### Migration Option 2 – clone and merge If you have been following the process correctly, your IOTstack will already be down. -##### Rename your existing IOTstack folder +##### Rename your existing IOTstack folder Move your old IOTstack folder out of the way, like this: @@ -199,7 +199,7 @@ Note: * You should not need `sudo` for the `mv` command but it is OK to use it if necessary. -##### Fetch a clean clone of SensorsIot/IOTstack +##### Fetch a clean clone of SensorsIot/IOTstack ``` $ git clone https://github.com/SensorsIot/IOTstack.git ~/IOTstack @@ -240,7 +240,7 @@ Observe what is **not** there: From this, it should be self-evident that a clean checkout from GitHub is the factory for *all* IOTstack installations, while the contents of `backups`, `services`, `volumes` and `docker-compose.yml` represent each user's individual choices, configuration options and data. -##### Merge old into new +##### Merge old into new Execute the following commands: @@ -272,7 +272,7 @@ There is no need to migrate the `backups` directory. You are better off creating $ mkdir ~/IOTstack/backups ``` -### Step 4 – Choose your menu +### Step 4 – Choose your menu If you have reached this point, you have migrated to SensorsIot/IOTstack where you are on the "master" branch. This implies "new menu". @@ -353,7 +353,7 @@ Although you can freely change branches, it's probably not a good idea to try to Even so, nothing will change **until** you run your chosen menu to completion and allow it to generate a new `docker-compose.yml`. -### Step 5 – Bring up your stack +### Step 5 – Bring up your stack Unless you have gotten ahead of yourself and have already run the menu (old or new) then nothing will have changed in the parts of your `~/IOTstack` folder that define your IOTstack implementation. You can safely: @@ -361,7 +361,7 @@ Unless you have gotten ahead of yourself and have already run the menu (old or n $ docker-compose up -d ``` -## See also +## See also There is another gist [Installing Docker for IOTstack](https://gist.github.com/Paraphraser/d119ae81f9e60a94e1209986d8c9e42f) which explains how to overcome problems with outdated Docker and Docker-Compose installations. diff --git a/docs/Updating-the-Project.md b/docs/Updates/index.md similarity index 50% rename from docs/Updating-the-Project.md rename to docs/Updates/index.md index 5d0aeff0..e659c046 100644 --- a/docs/Updating-the-Project.md +++ b/docs/Updates/index.md @@ -1,9 +1,36 @@ # Updating the project -**If you ran the git checkout -- 'git ls-files -m' as suggested in the old wiki entry then please check your duck.sh because it removed your domain and token** - Periodically updates are made to project which include new or modified container template, changes to backups or additional features. As these are released your local copy of this project will become out of date. This section deals with how to bring your project to the latest published state. +## Quick instructions + +1. backup your current settings: `cp docker-compose.yml docker-compose.yml.bak` +2. check `git status` for any local changes you may have made to project files. Save and preserve your changes by doing a commit: `git commit -a -m "local customization"`. Or revert them using: `git checkout -- path/to/changed_file`. +3. update project files from github: `git pull origin master -r` +4. get latest images from the web: `docker-compose pull` +5. rebuild localy created images from new Dockerfiles: `docker-compose build --pull --no-cache` +6. update running containers to latest: `docker-compose up --build -d` + +### Troubleshooting: if a container fails to start after update + +* try restarting the whole stack: `docker-compose restart` +* backup your stack settings: `cp docker-compose.yml docker-compose.yml.bak` +* Check log output of the failing service: `docker-compose logs *service-name*` + * try googling and fixing problems in docker-compose.yml manually. +* try recreating the failing service definition using menu.sh: + 1. `./menu.sh`, select Build Stack, unselect the failing service, press + enter to build, and then exit. + 2. `./menu.sh`, select Build Stack, select the service back again, press + enter to build, and then exit. + 3. Try starting now: `docker-compose up -d` +* Go to the [IOTStack Discord](https://discord.gg/ZpKHnks) and describe your + problem. We're happy to help. + +## Details, partly outdated + +!!! warning + If you ran `git checkout -- 'git ls-files -m'` as suggested in the old wiki entry then please check your duck.sh because it removed your domain and token + Git offers build in functionality to fetch the latest changes. `git pull origin master` will fetch the latest changes from GitHub without overwriting files that you have modified yourself. If you have done a local commit then your project may to handle a merge conflict. @@ -18,4 +45,4 @@ With the new latest version of the project you can now use the menu to build you ![image](https://user-images.githubusercontent.com/46672225/68646024-8fee2f80-0522-11ea-8b6e-f1d439a5be7f.png) -After your stack had been rebuild you can run `docker-compose up -d` to pull in the latest changes. If you have not update your images in a while consider running the `./scripts/update.sh` to get the latest version of the image from Docker hub as well \ No newline at end of file +After your stack had been rebuild you can run `docker-compose up -d` to pull in the latest changes. If you have not update your images in a while consider running the `./scripts/update.sh` to get the latest version of the image from Docker hub as well diff --git a/docs/index.md b/docs/index.md index 96f90300..425df460 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,11 +1,26 @@ +--- +title: Home +hide: + - navigation +--- # IOTStack Wiki +!!! abstract inline end "What is IOTstack" + IOTstack is a builder for docker-compose to easily make and maintain IoT + stacks on the Raspberry Pi + Welcome to the IOTstack Wiki: -* Use the list of contents at the left of this page to explore this Wiki. +* + Use the header tabs and content list at the left to explore this Wiki. + + - - If you are viewing this on a device that does not show the list by default, click the "≡" icon. +* If you are just getting started with IOTstack, see [Getting Started](Basic_setup/). + * If you're running gcgarner/IOTstack see [Migrating to SensorsIot](Updates/gcgarner-migration.md). -* If you are looking for information on a specific container, click on the "Containers" folder at the bottom of the list. +* You're always welcome to ask questions on the [IOTStack Discord](https://discord.gg/ZpKHnks). -* If you are just getting started with IOTstack, see [Getting Started](./Getting-Started.md). +* Fixes and improvements welcome, see [Contributing](Developers/) diff --git a/docs/stack-24.svg b/docs/stack-24.svg new file mode 100644 index 00000000..05794aee --- /dev/null +++ b/docs/stack-24.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/style.css b/docs/style.css new file mode 100644 index 00000000..91073fc2 --- /dev/null +++ b/docs/style.css @@ -0,0 +1,17 @@ +/* vim: set sw=2: */ + +/* hide "Made with Material" footer */ +.md-footer-meta { + display: none; +} + +@media screen and (max-width:76.25em) { + .show-when-wide-layout { + display:none + } +} +@media screen and (min-width:76.25em) { + .show-when-narrow-layout { + display:none + } +} diff --git a/mkdocs.yml b/mkdocs.yml index 3bd986db..c908ecd6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,10 +1,60 @@ site_name: IOTstack site_description: 'Docker stack for getting started on IOT on the Raspberry PI' + +# Repository +repo_url: https://github.com/SensorsIot/IOTstack +repo_name: SensorsIot/IOTstack +edit_uri: "https://github.com/SensorsIot/IOTstack/edit/master/docs" + theme: name: material + icon: + logo: octicons/stack-24 + favicon: stack-24.svg + palette: + - scheme: default + toggle: + icon: material/weather-sunny + name: Switch to dark mode + - scheme: slate + toggle: + icon: material/weather-night + name: Switch to light mode + features: + - tabs + - navigation.tabs + - navigation.tabs.sticky + - navigation.sections + plugins: - search -# - awesome-pages + - redirects: + # Forward renamed pages to avoid breaking old links. + redirect_maps: + Getting-Started.md: Basic_setup/index.md + Accessing-your-Device-from-the-internet.md: Basic_setup/Accessing-your-Device-from-the-internet.md + Backup-and-Restore.md: Basic_setup/Backup-and-Restore.md + Custom.md: Basic_setup/Custom.md + Default-Configs.md: Basic_setup/Default-Configs.md + Docker-commands.md: Basic_setup/Docker-commands.md + How-the-script-works.md: Basic_setup/How-the-script-works.md + Misc.md: Basic_setup/Misc.md + Native-RTL_433.md: Basic_setup/Native-RTL_433.md + Networking.md: Basic_setup/Networking.md + RPIEasy_native.md: Basic_setup/RPIEasy_native.md + Understanding-Containers.md: Basic_setup/Understanding-Containers.md + Updates/Updating-the-Project.md: Updates/index.md + PostBuild-Script.md: Developers/PostBuild-Script.md + BuildStack-RandomPassword.md: Developers/BuildStack-RandomPassword.md + BuildStack-Services.md: Developers/BuildStack-Services.md + Menu-System.md: Developers/Menu-System.md + Contributing-Services.md: Developers/index.md + +extra_css: + - style.css + markdown_extensions: + - admonition - pymdownx.superfences -repo_url: https://github.com/SensorsIot/IOTstack + - toc: + permalink: true diff --git a/requirements-mkdocs.txt b/requirements-mkdocs.txt new file mode 100644 index 00000000..cddd4398 --- /dev/null +++ b/requirements-mkdocs.txt @@ -0,0 +1,3 @@ +mkdocs-material +mkdocs-material-extensions +mkdocs-redirects