diff --git a/docs/Basic_setup/Networking.md b/docs/Basic_setup/Networking.md index 30f468e2..caed06d0 100644 --- a/docs/Basic_setup/Networking.md +++ b/docs/Basic_setup/Networking.md @@ -5,7 +5,9 @@ An easy way to find out your IP is by typing `ip address` in the terminal and lo Check the docker-compose.yml to see which ports have been used +
![net](https://user-images.githubusercontent.com/46672225/66702353-0bcc4080-ed07-11e9-994b-62219f50b096.png) +
## Examples - You want to connect your nodered to your mqtt server. In nodered drop an mqtt node, when you need to specify the address type `mosquitto` @@ -25,4 +27,4 @@ For adminer: ports: - 9080:8080 ``` -Port 9080 on Host Pi is mapped to port 8080 of the container. Therefore 127.0.0.1:8080 will take you to openHAB, where 127.0.0.1:9080 will take you to adminer \ No newline at end of file +Port 9080 on Host Pi is mapped to port 8080 of the container. Therefore 127.0.0.1:8080 will take you to openHAB, where 127.0.0.1:9080 will take you to adminer diff --git a/docs/Basic_setup/Understanding-Containers.md b/docs/Basic_setup/Understanding-Containers.md index 75dbec40..8f247b6f 100644 --- a/docs/Basic_setup/Understanding-Containers.md +++ b/docs/Basic_setup/Understanding-Containers.md @@ -4,7 +4,7 @@ In simple terms, Docker is a software platform that simplifies the process of bu managing and distributing applications. It does this by virtualizing the operating system of the computer on which it is installed and running. -# The Problem +## The Problem Let’s say you have three different Python-based applications that you plan to host on a single server (which could either be a physical or a virtual machine). @@ -14,7 +14,7 @@ libraries and dependencies, differ from one application to another. Since we cannot have different versions of Python installed on the same machine, this prevents us from hosting all three applications on the same computer. -# The Solution +## The Solution Let’s look at how we could solve this problem without making use of Docker. In such a scenario, we could solve this problem either by having three physical machines, or a single physical machine, which is powerful enough to host and run three virtual machines on it. @@ -34,7 +34,7 @@ This allows each container to be isolated from the other present on the same hos multiple containers with different application requirements and dependencies to run on the same host, as long as they have the same operating system requirements. -# Docker Terminology +## Docker Terminology Docker Images and Docker Containers are the two essential things that you will come across daily while working with Docker. @@ -45,7 +45,7 @@ required to run that application on Docker. On the other hand, as stated earlier, a Docker Container is a logical entity. In more precise terms, it is a running instance of the Docker Image. -# What is Docker-Compose? +## What is Docker-Compose? Docker Compose provides a way to orchestrate multiple containers that work together. Docker compose is a simple yet powerful tool that is used to run multiple containers as a single service. @@ -56,21 +56,21 @@ each separately. It wires up the networks (literally), mounts all volumes and ex The IOTstack with the templates and menu is a generator for that docker-compose service descriptor. -# How Docker Compose Works? +## How Docker Compose Works? use yaml files to configure application services (docker-compose.yaml) can start all the services with a single command ( docker-compose up ) can stop all the service with a single command ( docker-compose down ) -# How are the containers connected +## How are the containers connected The containers are automagically connected when we run the stack with docker-compose up. The containers using same logical network (by default) where the instances can access each other with the instance logical name. Means if there is an instance called *mosquitto* and an *openhab*, when openHAB instance need to access mqtt on that case the domain name of mosquitto will be resolved as the runnuning instance of mosquitto. -# How the container are connected to host machine +## How the container are connected to host machine -## Volumes +### Volumes The containers are enclosed processes which state are lost with the restart of container. To be able to persist states volumes (images or directories) can be used to share data with the host. @@ -89,7 +89,7 @@ Volumes are the preferred mechanism for persisting data generated by and used by While bind mounts are dependent on the directory structure of the host machine, volumes are completely managed by Docker. In IOTstack project uses the volumes directory in general to bind these container volumes. -## Ports +### Ports When containers running a we would like to delegate some services to the outside world, for example OpenHAB web frontend have to be accessible for users. There are several ways to achive that. One is mounting the port to the most machine, this called port binding. On that case service will have a dedicated diff --git a/docs/Basic_setup/What-is-sudo.md b/docs/Basic_setup/What-is-sudo.md index 1ba728c2..43477bb8 100644 --- a/docs/Basic_setup/What-is-sudo.md +++ b/docs/Basic_setup/What-is-sudo.md @@ -45,3 +45,15 @@ Please try to minimise your use of `sudo` when you are working with IOTstack. He ``` It takes time, patience and practice to learn when `sudo` is **actually** needed. Over-using `sudo` out of habit, or because you were following a bad example you found on the web, is a very good way to find that you have created so many problems for yourself that will need to reinstall your IOTstack. *Please* err on the side of caution! + +## Configuration + +To edit sudo functionality and permissions use: `sudo visudo` + +For instance, to allow sudo usage without prompting for a password: +```bash +# Allow members of group sudo to execute any command without password prompt +%sudo ALL=(ALL:ALL) NOPASSWD:ALL +``` + +For more information: `man sudoers` diff --git a/docs/Developers/Git-Setup.md b/docs/Developers/Git-Setup.md new file mode 100644 index 00000000..014d9628 --- /dev/null +++ b/docs/Developers/Git-Setup.md @@ -0,0 +1,178 @@ +How to setup and use git for IOTstack development. + +1. First, create a + [fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo) of + SensorsIot/IOTstack on github. And + [setup](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account) + your ssh-keys. +1. Clone your fork and setup your github username and email + ``` console + $ git clone git@github.com:/IOTstack.git + $ cd IOTstack + $ git config user.name + $ git config user.email <1234>+@users.noreply.github.com + ``` +1. Add up the SensorsIot/IOTstack upstream + ``` console + $ git remote add upstream https://github.com/SensorsIot/IOTstack.git + ``` +1. Configure for ease of operation + ``` console + $ git config fetch.prune true + $ git config remote.pushDefault origin + $ git config --add remote.origin.fetch "^refs/heads/gh-pages" + $ git config --add remote.upstream.fetch "^refs/heads/gh-pages" + $ git config branch.master.mergeoptions "--no-ff" + $ git config fetch.parallel 0 + $ git fetch --all + ``` + +## Make a pull-request + +``` mermaid +flowchart LR + upstream["upstream (SensorsIOT)"] -- "1. git fetch + git checkout -b" + --> local[local branch] + local -- "2. git commit" --> local + local -- "3. git push" --> origin["origin (your fork)"] + origin -- "3. create github pull-request" --> upstream +``` + +Please see [Contributing](index.md) for instructions on how to write commit +messages. + +``` console +$ git fetch upstream +$ git checkout -b upstream/master +...coding and testing... +$ git add +Check everything has been added: +$ git status +$ git commit +$ git push +``` +When you execute git push, its output should have a link for creating the +pull-request to github. + +## Common operations + +### Show compact history with "git lg" + +``` console +$ git config alias.lg "log --color --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit" +``` + +### Remove branches of merged pull-requests. + +When your pull-requests have been merged, their branches aren't needed anymore. +Remove them to reduce clutter and distractions. The master branch is never +deleted. + +``` console +$ git fetch --all +$ git checkout master +$ git branch -r --merged upstream/master | \ + grep -v origin/master$ | grep origin | sed 's/origin\///' | \ + xargs -I 'B' git push --delete origin B +$ git branch --merged upstream/master | grep -v " master$" | \ + xargs -I'B' git branch -d B +``` + +## Advanced topics + +### Fetch all pull-requests as branches + +This is handy for easily testing out other persons' suggested changes. The +branches are of course fetch-only, and you can't push your own commits to them. + +``` console +$ git config --add remote.upstream.fetch +refs/pull/*/head:refs/remotes/upstream/pr-* +$ git fetch upstream +``` + +*Note:* Everything below requires this. + +### Show up-to-date branches not merged + +Branches that include the latest upstream/master, but are not merged to +your current branch, are potentially mergeable pull-requests. This is useful +for identifying which pull-requests you should be able to merge without +conflict. + +``` console +$ git fetch upstream +$ git branch -r --contains upstream/master --no-merged upstream/master +``` + +### Check pull-requests on Github can be merged without conflicts + +In git, the only way to know if a branch can be merged without a conflict, is +by actually doing the merge. An alias to (re-)create a branch named +`merge-test` and do merges into it: + +``` console +$ git config alias.test-pull-request-merge $'!f() { : git merge && \ + OPENPULLS=$(curl -s \'https://api.github.com/repos/SensorsIot/IOTstack/pulls?base=master&per_page=100\' | \ + grep "^.....number" | sed -E \'s/.* ([0-9]+),/ upstream\\/pr-\\1/\') && \ + git fetch upstream && git checkout -B merge-test upstream/master && \ + git branch -r --contains upstream/master --no-merged upstream/master | \ + grep upstream/pr- | sort - <(echo "$OPENPULLS") | \ + { uniq -d; [[ "$1" ]] && echo "$1"; } | \ + xargs -I B sh -c "echo Merging B && \ + git merge --no-rerere-autoupdate --no-ff --quiet B || \ + { echo ***FAILED TO MERGE B && exit 255; };" ;}; f' +``` + + + + +Then use this alias combined with `git checkout -`, returning your working copy +back to the original branch if all merges succeeded: + +``` console +$ git test-pull-request-merge && git checkout - +``` + +This merges all branches that are: a) currently open pull requests and b) +up-to-date, i.e. contains upstream/master and c) not merged already and d) the +optional provided argument. Note: won't ignore draft pull-requests. If it +encounters a failure, it stops immediately to let you inspect the conflict. + +!!! help "Failed merge?" + + *If* there was a merge-conflict, inspect it e.g. using `git diff`, but + don't do any real work or conflict resolution in the merge-test branch. + When you have understood the merge-conflict and want to leave the + merge-test branch, abort the failed merge and switch to your actual branch: + + ``` console + $ git diff + $ git merge --abort + $ git checkout + ``` + +### Check your branch doesn't conflict with any existing pull-request + +When you intend to submit a pull-request you might want to check that it won't +conflict with any of the existing pull-requests. + +1. Commit all your changes into your pull request branch. +2. Use the alias from the previous "Test all current pull-requests..."-topic + to test merging your branch in addition to all current pull request: + + ``` console + $ git test-pull-request-merge && git checkout - + ``` + + If there is a merge-conflict, see "Failed merge?" above. diff --git a/docs/Developers/index.md b/docs/Developers/index.md index 25e5e2cb..9bed8c25 100644 --- a/docs/Developers/index.md +++ b/docs/Developers/index.md @@ -1,7 +1,25 @@ # Contributing +We welcome pull-requests. + +For larger contributions, please open an issue describing your idea. It +may provide valuable discussion and feedback. It also prevents the unfortunate +case of two persons working on the same thing. There's no need to wait for any +approval. + +!!! check "Development guidelines" + * It-just-works - use good defaults that will work well for a first time user + * Keep-it-simple - try to keep stuff beginner-friendly and don't go too + deep into advanced topics + ## Writing documentation +!!! tip inline end + For simple changes you can straight-up just use the edit link available on + every documentation page. It's the pen-icon to the right of the top + heading. Write your changes, preview everything looks as expected and + submit as proposed changes. + Documentation is is written as markdown, processed using mkdocs ([docs](https://www.mkdocs.org/user-guide/writing-your-docs/#writing-your-docs)) and the Material theme ([docs](https://squidfunk.github.io/mkdocs-material/reference/)). The Material theme is not just styling, but provides additional syntax extensions. Setup your system for mkdocs and Material: @@ -34,6 +52,47 @@ Services will grow over time, we may split up the buildstack menu into subsectio * Any configs that are required before getting the service running should be configured in the service's options menu (and a BuildStack menu Issue should be displayed if not). * Fork the repo and push the changes to your fork. Create a cross repo PR for the mods to review. We may request additional changes from you. +## Commit message + +``` +service_name: Add/Fix/Change feature or bug summary + +Optional longer description of the commit. What is changed and why it +is changed. Wrap at 72 characters. + +* You can use markdown formating as this will automatically be the + description of your pull-request. +* End by adding any issues this commit fixes, one per line: + +Fixes #1234 +Fixes #4567 +``` + +1. The first line is a short description. Keep it short, aim for 50 + characters. This is like the subject of an email. It shouldn't try to fully + or uniquely describe what the commit does. More importantly it should aim + to inform *why* this commit was made. + + `service_name` - service or project-part being changed, e.g. influxdb, + grafana, docs. Documentation changes should use the the name of the + service. Use `docs` if it's changes to general documentation. If all else + fails, use the folder-name of the file you are changing. Use lowercase. + + `Add/Fix/Change` - what type of an change this commit is. Capitalized. + + `feature or bug summary` - free very short text giving an idea of why/what. + +2. Empty line. + +3. A longer description of what and why. Wrapped to 72 characters. + + Use [github issue linking]( + https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue) + to automatically close issues when the pull-request of this commit is + merged. + +For tips on how to use git, see [Git Setup](Git-Setup.md). + ## Follow up If your new service is approved and merged then congratulations! Please watch the Issues page on github over the next few days and weeks to see if any users have questions or issues with your new service. diff --git a/docs/Updates/index.md b/docs/Updates/index.md index 020c8f1b..823adcfc 100644 --- a/docs/Updates/index.md +++ b/docs/Updates/index.md @@ -1,5 +1,101 @@ # Updating the project +There are two different sources: the IOTstack project (github.com) and +the Docker Hub (hub.docker.com). Both the initial stack creation and updates +come from these. To illustrate the steps and artifacts of the *update* process: + +``` mermaid +flowchart TD + GIT[github.com/sensorsiot/IOTstack.git] + GIT --- GITPULL([$ git pull -r]) + GITPULL --> TEMPLATES["~/IOTstack/.templates"] + TEMPLATES --- MENU([$ ./menu.sh -> Build stack]) + MENU --> COMPOSE["~/IOTstack/docker-compose.yml\n~/IOTstack/services/*/Dockerfile"] + COMPOSE --- UP(["$ docker-compose up --build -d"]) + + HUB[hub.docker.com images and tags] + HUB --- PULL([$ docker-compose pull\n$ docker-compose build --pull --no-cache]) + COMPOSE --- PULL + PULL --> CACHE[local Docker image cache] + CACHE --- UP + + UP --> CONTAINER[updated Docker containers] + + classDef command fill:#9996,stroke-width:0px + class GITPULL,MENU,UP,PULL command +``` + +!!! note "Minor details" + + In order to keep the graph simple, some minor details fudged: + + - `$ docker-compose pull` will read `docker-compose.yml`, in order to know + what image tags to check for updates. + - `docker-compose build --pull --no-cache` will use + `~/IOTstack/.templates/*/Dockerfile` and + `~/IOTstack/services/*/Dockerfile` and pull their referenced Docker + images, if there are selected services using these Dockerfiles. + +## Backup and rollback + +The usual way of backing up just your `~/IOTstack` contents isn't sufficient +for a 100% identical restore. Some containers may have local ephemeral +modifications that will be lost when they're recreated. Currently running +containers may be based on now outdated images. Recreating a container using an +old image is tricky. The local Docker image cache can't easily be restored to +the same state with old images and old tag references. The `docker pull` will +fetch the latest images, but it's not unheard of that the latest image may +break [something]( +https://github.com/node-red/node-red/issues/3461#issuecomment-1076348639). + +Thus to *guarantee* a successful rollback, you have to shutdown your RPi and +save a complete disk image backup of its storage using another machine. + +For a hobby project, not having perfect rollback may be a risk you're willing +to take. Usually container image problems have fixes/workarounds within a day. + +## Recommended: Update only Docker images + +When you built the stack using the menu, it created the Docker Compose file +`docker-compose.yml`. This file uses tag references (e.g. `:latest`) to get the +image for that tag from hub.docker.com. Thus when Docker is told to pull +images, it will download and update it's local cache to the newest image. No +need to update `docker-compose.yml` or `Dockerfile`s. + +Updating the IOTstack project templates and recreating your +`docker-compose.yml` isn't usually necessary. Doing so isn't likely to provide +much benefits, and may actually break something. A full update is only +recommended when there is a new feature or change you need. + +!!! tip "Recommended update procedure" + + 1. Shutdown your RPi, remove its storage medium and backup a full image + of the storage to another machine. Reattach your storage and power up + your RPi. To skip this step may cause a long downtime as you debug the + problem. + 2. Get latest images from the web: + ``` console + $ docker-compose pull + ``` + 3. Rebuild localy created images based on new parent images: + ``` console + $ docker-compose build --pull --no-cache + ``` + Note: this may not do anything, depending on your selected services. + 4. Update(recreate) containers that have new images: + ``` console + $ docker-compose up --build -d + ``` + +If a service fails to start after it's updated, especially if you are updating +frequently, wait for a few hours and repeat the update procedure. Sometimes bad +releases are published to hub.docker.com, but they are usually fixed in under +half a day. Of course you are always welcome to report the problem to our +[Discord](https://discord.gg/ZpKHnks) server. Usually someone else has +encountered the same problem and reported the fix. + +## Full update + Periodically updates are made to project which include new or modified container template, changes to backups or additional features. As these are released your local copy of this project will become out of date. This section deals with how to bring your project to the latest published state. !!! danger "Breaking update" @@ -10,9 +106,12 @@ Periodically updates are made to project which include new or modified container ## Quick instructions -1. backup your current settings: `cp docker-compose.yml docker-compose.yml.bak` -2. check `git status` for any local changes you may have made to project files. Save and preserve your changes by doing a commit: `git commit -a -m "local customization"`. Or revert them using: `git checkout -- path/to/changed_file`. -3. update project files from github: `git pull origin master -r` +1. shutdown your RPi, remove its storage medium and do a full image backup of + it to another machine. Reinstall your storage and power up your RPi. +2. backup your current settings: `cp docker-compose.yml docker-compose.yml.bak` +3. check `git status` for any local changes you may have made to project files, ignore any reported "Untracked files". Save and preserve your changes by doing a commit: `git commit -a -m "local customization"`. Or revert them using: `git checkout -- path/to/changed_file`. +4. update project files from github: `git pull origin master -r` +5. recreate the compose file and Dockerfile:s: `./menu.sh`, select Build Stack, don't change selections, press enter to build, and then exit. 4. get latest images from the web: `docker-compose pull` 5. rebuild localy created images from new Dockerfiles: `docker-compose build --pull --no-cache` 6. update running containers to latest: `docker-compose up --build -d` @@ -29,7 +128,7 @@ Periodically updates are made to project which include new or modified container 2. `./menu.sh`, select Build Stack, select the service back again, press enter to build, and then exit. 3. Try starting now: `docker-compose up -d` -* Go to the [IOTStack Discord](https://discord.gg/ZpKHnks) and describe your +* Go to the [IOTstack Discord](https://discord.gg/ZpKHnks) and describe your problem. We're happy to help. ## Details, partly outdated diff --git a/docs/index.md b/docs/index.md index 425df460..bd9d9427 100644 --- a/docs/index.md +++ b/docs/index.md @@ -12,13 +12,13 @@ hide: Welcome to the IOTstack Wiki: * - Use the header tabs and content list at the left to explore this Wiki. + Use the top tabs and then the left list to explore this Wiki. -* If you are just getting started with IOTstack, see [Getting Started](Basic_setup/). +* If you are just getting started with IOTstack, see [Getting Started](Basic_setup/index.md). * If you're running gcgarner/IOTstack see [Migrating to SensorsIot](Updates/gcgarner-migration.md). * You're always welcome to ask questions on the [IOTStack Discord](https://discord.gg/ZpKHnks). diff --git a/docs/style.css b/docs/style.css index 32443d97..df18e019 100644 --- a/docs/style.css +++ b/docs/style.css @@ -22,3 +22,11 @@ display:none } } + +/* Make dark-mode links just a tiny bit lighter, for better contrast + on low brightness screens. */ +@media screen { + [data-md-color-scheme="slate"] { + --md-typeset-a-color: var(--md-primary-fg-color--light) !important; + } +} diff --git a/mkdocs.yml b/mkdocs.yml index cc30ba43..9a4ba449 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -64,6 +64,13 @@ markdown_extensions: - pymdownx.highlight: pygments_lang_class: true - admonition - - pymdownx.superfences + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format + - pymdownx.tabbed: + alternate_style: true - toc: permalink: true + - md_in_html diff --git a/requirements-mkdocs.txt b/requirements-mkdocs.txt index cddd4398..11c6c2c7 100644 --- a/requirements-mkdocs.txt +++ b/requirements-mkdocs.txt @@ -1,3 +1,24 @@ -mkdocs-material -mkdocs-material-extensions -mkdocs-redirects +bracex==2.2.1 +click==8.0.4 +ghp-import==2.0.2 +importlib-metadata==4.11.2 +Jinja2==3.0.3 +Markdown==3.3.6 +MarkupSafe==2.1.0 +mergedeep==1.3.4 +mkdocs==1.2.3 +mkdocs-awesome-pages-plugin==2.7.0 +mkdocs-material==8.2.3 +mkdocs-material-extensions==1.0.3 +mkdocs-redirects==1.0.3 +packaging==21.3 +Pygments==2.11.2 +pymdown-extensions==9.2 +pyparsing==3.0.7 +python-dateutil==2.8.2 +PyYAML==6.0 +pyyaml_env_tag==0.1 +six==1.16.0 +watchdog==2.1.6 +wcmatch==8.3 +zipp==3.7.0