Publish docs with mkdocs and github actions (#243)

.github/workflows/publish-docs.yml

@@ -0,0 +1,24 @@
+name: Publish docs via GitHub Pages
+on:
+  push:
+    branches:
+      - master
+
+jobs:
+  build:
+    name: Publish docs
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout main
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Deploy docs
+        uses: mhausenblas/mkdocs-deploy-gh-pages@master
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          CUSTOM_DOMAIN: www.mirrorcache.org
+          CONFIG_FILE: mkdocs.yml
+          EXTRA_PACKAGES: build-base
+          REQUIREMENTS: docs/requirements.txt

README.asciidoc → README.md

@@ -1,4 +1,5 @@
-= Welcome to MirrorCache!
+Welcome to MirrorCache!
+-----------------------
 
 MirrorCache is a Web Server for files download, which will route download requests to an appropriate mirror.
 MirrorCache doesn't store files and instead keeps in DB list of files from the `Main Server`.
@@ -11,23 +12,23 @@ In this regard MirrorCache is a cache of (meta)information about geographical lo
 
 Output below domonstrates a cache miss, so the download request will be redirected to the `Main Server` (in this case download.opensuse.org):
 
-[source,bash]
------------------
+
+```
 > curl -I http://mirrorcache.opensuse.org/download/update/openSUSE-current/x86_64/alsa-1.1.5-lp152.8.6_lp152.9.4.1.x86_64.drpm
 HTTP/1.1 302 Found
 location: http://download.opensuse.org/update/openSUSE-current/x86_64/alsa-1.1.5-lp152.8.6_lp152.9.4.1.x86_64.drpm
 date: Wed, 29 Jul 2020 08:37:07 GMT
------------------
+```
 
 Then background jobs will collect info about the hottest misses and scan predefined mirrors for presence of these files. Further requests will be redirected to one of the mirrors that has the file:
 
-[source,bash]
------------------
+
+```
 > curl -I http://mirrorcache.opensuse.org/download/update/openSUSE-current/x86_64/alsa-1.1.5-lp152.8.6_lp152.9.4.1.x86_64.drpm
 HTTP/1.1 302 Found
 location: http://ftp.gwdg.de/pub/opensuse/update/openSUSE-current/x86_64/alsa-1.1.5-lp152.8.6_lp152.9.4.1.x86_64.drpm
 date: Wed, 29 Jul 2020 08:40:00 GMT
------------------
+```
 
 The project was implemented as a quick hack with some amount of shortcuts to make things going.
 The goal is to improve it over time with the main focus to do the job.

docs/CNAME

@@ -0,0 +1 @@
+www.mirrorcache.org

docs/assets/logo-16.png

@@ -0,0 +1 @@
+../../assets/images/logo-16.png

docs/assets/logo.svg

@@ -0,0 +1 @@
+../../assets/images/logo.svg

docs/flow.md

@@ -0,0 +1,10 @@
+---
+hide:
+  - toc
+---
+
+# Overview and concept
+
+Here is the perfect place to provide a nice picture of the process...
+
+![](assets/flow.svg)

docs/index.md

@@ -0,0 +1 @@
+../README.md

doc/mb_compare.md → docs/mb_compare.md

@@ -1,6 +1,9 @@
+title: MirrorCache vs MirrorBrain
+
 # MirrorCache (mc) concept vs MirrorBrain (mb) concept
 
-1. Scanning
+## Scanning
+
    1. mb uses daily full scan of all files on all mirrors, without regard of whether files were ever downloaded or changed recently. This may be waste of resources.
 mc scans only those locations which are needed by users and prioritize the scans according to popularity.
    1. It may take up to 24 hours before mb notices that file has gone from a mirror. Clients will be incorrectly redirected and receive 404 error.
@@ -10,21 +13,26 @@ mc has separate jobs for keeping in sync information about files on the Main Ser
    1. New files are added by scanners in mb, which currently leads to many incorrect files in database. (Potentially overload of mb DB because of incorrectly configured mirror).
 (This may be fixed in cost of added coupling on mb architecture).
 
-1. Maintenance jobs, scheduling, logging, resources and scalability
+## Jobs queue
+
    * mb relies on command line tool and cron to run jobs. User then manually reviews logs to find eventual problems (which is a challenge to do with current amount of logging).
    * mc has a popular, mature, fast and asynchronous job queue: https://docs.mojolicious.org/Minion, which includes prioritization, rescheduling, custom job queues, scalability (adding workers), manageability (assigning workers to particular queues), dashboard, search, etc.
 
-1. WWW
+## WWW
+
    * mb relies on rendering of static files generated by a cron job.
    * mc uses the mojolicious.org framework, which has delayed rendering, javascript, etc.
 
-1. Fileless architecture 
+## Fileless architecture
+
    1. Architecture
+
       * By design mb uses local files for rendering directories in WebUI and for scanner jobs to detect files present on mirror.
-It is a complication for using several geographical instances of mb, so it's possible to configure Apache plugins with some periodical sync jobs to store empty files instead of keeping full content of files. 
+It is a complication for using several geographical instances of mb, so it's possible to configure Apache plugins with some periodical sync jobs to store empty files instead of keeping full content of files.
       * mc is fileless by design, so it should be easy to spawn a new mc instance, which will become useful immediately.
 
    1. Since at the moment filename is not enough to identify the content of a file (see https://github.com/openSUSE/open-build-service/issues/6690), additional identification is required (e.g. filesize + mtime). This will complicate mb approach even more. (It will need to sync additional info besides empty files).
 
-1. Robustness
+## Robustness
+
 While mc may lead to eventual increased traffic to the main server, both mc and mb must rely on robustness of the main server.

docs/requirements.txt

@@ -0,0 +1,6 @@
+mkdocs
+mkdocs-material
+mkdocs-material-extensions
+mkdocs-minify-plugin
+mkdocs-markdownextradata-plugin
+mkdocs-git-revision-date-localized-plugin

doc/setup.md → docs/setup.md

@@ -5,6 +5,7 @@
 ### Environment variables
 
 MirrorCache can be configured with following environment variables:
+
   * MIRRORCACHE_ROOT (required): defines location of files, which needs redirection. It may be url, local folder or rsync address, e.g. `MIRRORCACHE_ROOT=http://download.opensuse.org` or `MIRRORCACHE_ROOT=/srv/mirrorcache` or `MIRRORCACHE_ROOT=rsync://user:[email protected]/module`. (Note that you must install additionally `perl-Digest-MD4` if rsync url needs password verification).
   * MIRRORCACHE_AUTH_URL (optional) may contain remote openid server url (default https://www.opensuse.org/openid/user/). if explicitly set to empty value - all login attempt will be allowed and user set to 'Demo'.
   * MIRRORCACHE_TOP_FOLDERS (space separated values) may be set to automatically redirect /folder to /download/folder.
@@ -18,6 +19,7 @@ MirrorCache can be configured with following environment variables:
 
 Without any database configuration MirrorCache will attempt to connect to database 'mirrorcache' on default PostgreSQL port 5432.
 Following variables can be used to configure database access:
+
   * MIRRORCACHE_DBUSER (default empty)
   * MIRRORCACHE_DBPASS (default empty)
   * TEST_PG or MIRRORCACHE_DSN , e.g. MIRRORCACHE_DSN='DBI:Pg:dbname=mc_dev;host=/path/to/pg'`
@@ -27,6 +29,7 @@ If neither TEST_PG nor MIRRORCACHE_DSN is defined, following variables are used:
   * MIRRORCACHE_DBPORT (default empty)
 
 ### GeoIP location
+
   * If environment variable MIRRORCACHE_CITY_MMDB or MIRRORCACHE_IP2LOCATION is defined, the app will attempt to detect country of the request and find a mirror in the same country, e.g. `MIRRORCACHE_CITY_MMDB=/var/lib/GeoIP/GeoLite2-City.mmdb` or `MIRRORCACHE_IP2LOCATION=/var/lib/GeoIP/IP2LOCATION-LITE-DB5.IPV6.BIN`.
   * See Maxmind or IP2Location website to obtain such file.
   * Additional dependencies must be installed as well for GeoIP location to work: perl modules Mojolicious::Plugin::ClientIP and MaxMind::DB::Reader :
@@ -42,10 +45,10 @@ zypper in perl-Geo-IP2Location
 
 ### Install package
 
-An example for openSUSE 15.2
+An example for openSUSE
 ```bash
-zypper ar https://mirrorcache.opensuse.org/repositories/home:andriinikitin:/MirrorCache/openSUSE_Leap_15.2 mc
-zypper --gpg-auto-import-keys --no-gpg-checks refresh
+zypper ar -f obs://openSUSE:infrastructure:MirrorCache MirrorCache
+zypper refresh -s
 zypper install MirrorCache
 
 zypper install postgresql postgresql-server
@@ -193,27 +196,29 @@ $mc/status
 
 ### Run tests from [t/environ](/t/environ) with docker, manually for debugging
 
-- Requires docker configured for non-root users
-- Available configuration:
-  - `MIRRORCACHE_CITY_MMDB` adds this environment variable inside the container and mounts it as a volume if the file exists on the host
-  - `EXPOSE_PORT` maps whatever port you need from the container to host port 80
-    ```bash
-    cd t/environ
+__Note:__ Requires docker configured for non-root users
+
+`MIRRORCACHE_CITY_MMDB` adds this environment variable inside the container and mounts it as a volume if the file exists on the host
 
-    # Just run the test:
-    ./01-smoke.sh
+`EXPOSE_PORT` maps whatever port you need from the container to host port 80
 
-    # Run the test with your own MIRRORCACHE_CITY_MMDB
-    MIRRORCACHE_CITY_MMDB=/var/lib/GeoIP/GeoLite2-City.mmdb ./01-smoke.sh
+```
+cd t/environ
+
+# Just run the test:
+./01-smoke.sh
 
-    # Run the test and keep the container, while mapping port 3110 to host port 80
-    EXPOSE_PORT=3110 ./01-smoke.sh
-    ```
+# Run the test with your own MIRRORCACHE_CITY_MMDB
+MIRRORCACHE_CITY_MMDB=/var/lib/GeoIP/GeoLite2-City.mmdb ./01-smoke.sh
+
+# Run the test and keep the container, while mapping port 3110 to host port 80
+EXPOSE_PORT=3110 ./01-smoke.sh
+```
 
-    To log in with a fake test-user, change `$mc/start` to `MIRRORCACHE_TEST_TRUST_AUTH=1 $mc/start` in your test
+To log in with a fake test-user, change `$mc/start` to `MIRRORCACHE_TEST_TRUST_AUTH=1 $mc/start` in your test
 
-    Setting `MIRRORCACHE_TEST_TRUST_AUTH` to any number > 1 will result in `current_user` being `undef`, so no fake test-user login.
-    You will only have access to some routes defined in [lib/MirrorCache/WebAPI.pm](/lib/MirrorCache/WebAPI.pm).
+Setting `MIRRORCACHE_TEST_TRUST_AUTH` to any number > 1 will result in `current_user` being `undef`, so no fake test-user login.
+You will only have access to some routes defined in [lib/MirrorCache/WebAPI.pm](/lib/MirrorCache/WebAPI.pm).
 
 **WARNING** - Be careful when working inside container:
 1. The source tree is mapped to the host, so any changes of source code inside container will be reflected on host and vice versa.

mkdocs.yml

@@ -0,0 +1,82 @@
+site_name: MirrorCache documentation
+site_url: https://www.mirrorcache.org
+site_description: Mirror redirector web service
+site_author: openSUSE contributors
+copyright: "© 2020-2021 openSUSE contributors"
+
+repo_name: openSUSE/MirrorCache
+repo_url: https://github.com/openSUSE/MirrorCache
+edit_uri: edit/master/docs/
+
+
+theme:
+  name: material
+  palette:
+    - media: "(prefers-color-scheme: light)"
+      scheme: default
+      primary: white
+      accent: green
+      toggle:
+        icon: material/toggle-switch-off-outline
+        name: Switch to dark mode
+    - media: "(prefers-color-scheme: dark)"
+      scheme: slate
+      primary: black
+      accent: green
+      toggle:
+        icon: material/toggle-switch
+        name: Switch to light mode
+  logo: assets/logo.svg
+  favicon: assets/logo-16.png
+  features:
+    - navigation.instant
+    - navigation.expand
+    - navigation.top
+    #v8.0.0 - navigation.tracking
+    #- toc.integrate
+
+plugins:
+  - search
+  - markdownextradata
+  - git-revision-date-localized
+  - minify:
+      minify_html: true
+
+markdown_extensions:
+  - admonition
+  - abbr
+  - attr_list
+  - def_list
+  - footnotes
+  - meta
+  - md_in_html
+  - toc:
+      permalink: true
+  - pymdownx.arithmatex:
+      generic: true
+  - pymdownx.betterem:
+      smart_enable: all
+  - pymdownx.caret
+  - pymdownx.details
+  - pymdownx.emoji:
+      emoji_index: !!python/name:materialx.emoji.twemoji
+      emoji_generator: !!python/name:materialx.emoji.to_svg
+  - pymdownx.highlight
+  - pymdownx.inlinehilite
+  - pymdownx.keys
+  # - pymdownx.magiclink:
+      # repo_url_shorthand: true
+      # user: openSUSE
+      # repo: MirrorCache
+  - pymdownx.mark
+  - pymdownx.smartsymbols
+  - pymdownx.superfences:
+      custom_fences:
+        - name: mermaid
+          class: mermaid-experimental
+          format: !!python/name:pymdownx.superfences.fence_code_format
+  - pymdownx.tabbed:
+      alternate_style: true
+  - pymdownx.tasklist:
+      custom_checkbox: true
+  - pymdownx.tilde