Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: remove nginx from docker compose #1815

Merged
merged 24 commits into from
Dec 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
7a5d8d1
build: remove nginx from docker compose #1602
gulfaraz Oct 22, 2024
d51c778
chore: update package lock
gulfaraz Oct 28, 2024
ab51f33
refactor: move db-init.sql to tools folder
gulfaraz Nov 28, 2024
84247d8
fix: rm web-server-network from docker-compose.ci
gulfaraz Nov 28, 2024
603a3a9
chore: expose ports
gulfaraz Nov 28, 2024
20cc17d
refactor: remove nginx/letsencrypt folder
gulfaraz Nov 28, 2024
93e27e8
refactor: use same echo message in docker.override and docker.ci
gulfaraz Nov 28, 2024
9173926
build: use cp to expose dashboard build folder
gulfaraz Nov 28, 2024
ce52063
refactor: remove root user override in docker.ci
gulfaraz Nov 28, 2024
44c75e3
fix: move raster-files.zip to azure storage
gulfaraz Nov 28, 2024
3b704b7
fix: use recursive instead of archive flag in cp
gulfaraz Nov 29, 2024
4b18118
fix: remove explicit lineclamp import
gulfaraz Nov 29, 2024
64f9bff
ci: use cache for docker builds
gulfaraz Nov 29, 2024
ee55fb7
ci: reduce build wait time to 30s
gulfaraz Nov 29, 2024
46bd348
ci: cleanup dashboard build
gulfaraz Nov 29, 2024
d9af32c
fix: use azureuser for webhook service
gulfaraz Nov 29, 2024
7228694
fix: set default server name in nginx conf
gulfaraz Nov 29, 2024
8ac4a23
fix: set correct nginx webserver root
gulfaraz Nov 29, 2024
333f130
docs: update VM setup
gulfaraz Nov 29, 2024
511fc1b
ci: build dashboard and use nginx
gulfaraz Nov 29, 2024
fc77c4d
ci: override volume to avoid permission error
gulfaraz Nov 29, 2024
d942b43
build: set owner for dist folder
gulfaraz Nov 29, 2024
7643617
fix: add geoserver volume to docker ignore
gulfaraz Nov 29, 2024
75eb409
docs: include server name to nginx setup
gulfaraz Dec 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ are connected to a database-server). For setting up a fully working version of t

1. Load initial raster data

- Get the file `raster-files.zip` from [this folder](https://rodekruis.sharepoint.com/sites/510-CRAVK-510/Gedeelde%20%20documenten/Forms/AllItems.aspx?id=%2Fsites%2F510%2DCRAVK%2D510%2FGedeelde%20%20documenten%2FIBF%20%2D%20System%2FIBF%20Portal%20Production%20Data&viewid=6b2bb429%2D8ac9%2D4abb%2D9d81%2D08c846f72f9b).
- Download [raster-files.zip](https://510ibfsystem.blob.core.windows.net/rasters/raster-files.zip)
- Unzip it in `services/API-service/geoserver-volume/raster-files` folder, such that that folder now has subfolders:
- `input`-folder: static raster files that are served through 'geoserver' to the 'IBF-dashboard'
- `mock-output`-foldermock output raster files that are used by the mock-endpoint (see below)
Expand Down
29 changes: 14 additions & 15 deletions docker-compose.ci.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,15 @@
services:
nginx:
image: nginx
ports:
- 8080:80
volumes:
- ./tools/nginx.conf:/etc/nginx/conf.d/default.conf
- ./interfaces/IBF-dashboard/www/browser:/var/www/ibf-dashboard
networks:
- api-network
restart: unless-stopped

ibf-api-service:
command: ['npm', 'run', 'start:dev']
environment:
Expand All @@ -7,19 +18,12 @@ services:
- GEOSERVER_ADMIN_PASSWORD=${GEOSERVER_ADMIN_PASSWORD}
ports:
- ${LOCAL_PORT_IBF_SERVICE}:3000
volumes: !override
- ibf-api-service-node-modules:/home/ibf/api-service/node_modules
depends_on:
- ibf-local-db
networks:
- web-server-network
- api-network
user: root

ibf-dashboard:
entrypoint: ['echo', 'Service ibf-dashboard disabled']

ibf-geoserver:
ports:
- 8081:8080

ibf-local-db:
image: postgis/postgis
Expand All @@ -28,14 +32,9 @@ services:
- POSTGRES_PASSWORD=${DB_PASSWORD}
- POSTGRES_DB=${DB_DATABASE}
volumes:
- ./database/init.sql:/docker-entrypoint-initdb.d/init.sql
# - ibf-local-db-data:/var/lib/postgresql/data
- ./tools/db-init.sql:/docker-entrypoint-initdb.d/init.sql
ports:
- 5437:5432
networks:
- api-network
restart: unless-stopped

# volumes:
# ibf-local-db-data:
# api_service_node_modules:
23 changes: 15 additions & 8 deletions docker-compose.override.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,15 @@
services:
nginx:
image: nginx
ports:
- 8080:80
volumes:
- ./tools/nginx.conf:/etc/nginx/conf.d/default.conf
- ./interfaces/IBF-dashboard/www/browser:/var/www/ibf-dashboard
networks:
- api-network
restart: unless-stopped

ibf-api-service:
command: ['npm', 'run', 'start:dev']
environment:
Expand All @@ -10,15 +21,10 @@ services:
depends_on:
- ibf-local-db
networks:
- web-server-network
- api-network

ibf-dashboard:
entrypoint: ['echo', 'Service ibf-dashboard disabled']

ibf-geoserver:
ports:
- 8081:8080
entrypoint: ['echo', 'ibf-dashboard is disabled']

ibf-local-db:
image: postgis/postgis
Expand All @@ -27,7 +33,7 @@ services:
- POSTGRES_PASSWORD=${DB_PASSWORD}
- POSTGRES_DB=${DB_DATABASE}
volumes:
- ./database/init.sql:/docker-entrypoint-initdb.d/init.sql
- ./tools/db-init.sql:/docker-entrypoint-initdb.d/init.sql
- ibf-local-db-data:/var/lib/postgresql/data
ports:
- 5437:5432
Expand All @@ -37,4 +43,5 @@ services:

volumes:
ibf-local-db-data:
api_service_node_modules:
ibf-api-service-node-modules:
ibf-dashboard-node-modules:
36 changes: 9 additions & 27 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -1,21 +1,4 @@
services:
nginx:
image: staticfloat/nginx-certbot
ports:
- 80:80
- 443:443
environment:
CERTBOT_EMAIL: ${CERTBOT_EMAIL}
ENVSUBST_VARS: FQDN
FQDN: ${FQDN}
volumes:
- ./nginx/conf.d:/etc/nginx/user.conf.d
- ibf-dashboard-root:/var/www/ibf-dashboard
- ./nginx/letsencrypt:/etc/letsencrypt
networks:
web-server-network:
restart: unless-stopped

ibf-api-service:
container_name: ibf-api-service
build:
Expand Down Expand Up @@ -47,11 +30,12 @@ services:
- EXTERNAL_API_SERVICE_URL=${EXTERNAL_API_SERVICE_URL}
- TWILIO_WHATSAPP_NUMBER=${TWILIO_WHATSAPP_NUMBER}
- API_SERVICE_URL=${API_SERVICE_URL}
ports:
- 3000:3000
volumes:
- 'api_service_node_modules:/home/ibf/api-service/node_modules'
- ibf-api-service-node-modules:/home/ibf/api-service/node_modules
- ./services/API-service:/home/ibf/api-service
networks:
web-server-network:
api-network:
restart: unless-stopped

Expand Down Expand Up @@ -83,31 +67,29 @@ services:
- SUPPORT_EMAIL_ADDRESS=${SUPPORT_EMAIL_ADDRESS}
- WHATS_NEW_URL=${WHATS_NEW_URL}
volumes:
- ibf-dashboard-root:/home/node/app/www/browser
networks:
web-server-network:
- ibf-dashboard-node-modules:/home/node/app/node_modules
- ./interfaces/IBF-dashboard/www:/home/node/app/_www

ibf-geoserver:
container_name: ibf-geoserver
image: kartoza/geoserver:2.19.2
environment:
- GEOSERVER_ADMIN_PASSWORD=${GEOSERVER_ADMIN_PASSWORD}
- JAVA_OPTS="-DALLOW_ENV_PARAMETRIZATION=true"
ports:
- 8081:8080
volumes:
- ./services/API-service/geoserver-volume/raster-files:/opt/geoserver/data_dir/workspaces/ibf-system/ibf-pipeline
- ./services/API-service/geoserver-volume/geoserver-layers:/opt/geoserver/data_dir/workspaces/ibf-system
- ./.env:/opt/geoserver/data_dir/geoserver-environment.properties
networks:
web-server-network:
api-network:
restart: unless-stopped

volumes:
ibf-dashboard-root:
api_service_node_modules:
ibf-api-service-node-modules:
ibf-dashboard-node-modules:

networks:
web-server-network:
driver: bridge
api-network:
driver: bridge
67 changes: 41 additions & 26 deletions docs/VM_SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
`touch /home/ibf-user`
6. Open `/etc/sudoers` with `sudo nano /etc/sudoers` and add these lines

```jsx
```console
# Allow members of group ibf-users to execute systemctl daemon-reload
%ibf-users ALL=NOPASSWD: /bin/systemctl daemon-reload

Expand All @@ -33,17 +33,19 @@
```

2. Install Software
1. NodeJS
[Source](https://github.com/nodesource/distributions/blob/master/README.md#installation-instructions)
1. Follow instructions in Source for Node 16
2. Verification - `node -v`
2. Docker [Source](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository)
1. Follow instructions in Source all th
2. Verification - `docker -v`
3. Allow users to access docker commands
1. `sudo usermod -aG docker <username>`
2. Verification - `grep docker /etc/group`

1. [Install NodeJS](https://github.com/nodesource/distributions/blob/master/README.md#installation-instructions)
- `node -v` to verify
2. [Install Docker](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository)
- `docker -v` to verify
- `sudo usermod -aG docker <username>` to allow users to access docker commands
- `grep docker /etc/group` to verify group members
- `sudo chmod 660 /var/run/docker.sock` to give group members access to docker socket
3. [Install Nginx](https://nginx.org/en/linux_packages.html#Ubuntu)
- `nginx -v` to verify

3. Setup IBF-system

1. `cd /home/ibf-user`
2. `git clone https://github.com/rodekruis/IBF-system.git`
3. `cd /home/ibf-user/IBF-system`
Expand All @@ -54,33 +56,46 @@
[Source](https://stackoverflow.com/a/6448326/1753041)
6. `sudo chmod -R g+rwX /home/ibf-user/IBF-system`
[Source](https://stackoverflow.com/a/6448326/1753041)
7. Setup Environment Variables
7. Create build folder
- `mkdir /home/ibf-user/IBF-system/interfaces/IBF-dashboard/www`
- `chown azureuser:ibf-users /home/ibf-user/IBF-system/interfaces/IBF-dashboard/www`
- `chmod 775 /home/ibf-user/IBF-system/interfaces/IBF-dashboard/www`
8. Setup Environment Variables
1. Create `/home/ibf-user/IBF-system/.env`
1. `cp /home/ibf-user/IBF-system/example.env /home/ibf-user/IBF-system/.env`
2. Set the appropriate values in the `.env` file
3. Load the `.env` vars by `source /home/ibf-user/IBF-system/.env`
4. Test if the vars were loaded correctly `echo $NODE_ENV`
8. Load certificate: load `DigiCertGlobalRootCA.crt.pem` in `services/API-service/cert` for connection to Azure Postgres server (if applicable)
9. `. tools/deploy.sh`
9. Load certificate: load `DigiCertGlobalRootCA.crt.pem` in `services/API-service/cert` for connection to Azure Postgres server (if applicable)
10. Configure Nginx
- `cp /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf.backup`
- `cp tools/nginx.conf /etc/nginx/conf.d/default.conf`
- Change `root`, `server_name`, and `proxy_pass` values in `/etc/nginx/conf.d/default.conf`
- Set `user` directive to `azureuser` in `/etc/nginx/nginx.conf`
- `service nginx restart` to restart nginx with new configuration
- `systemctl enable nginx` to start nginx on VM reboot
- `service nginx status` to verify
11. `source tools/deploy.sh`

4. Load base data

1. Load Geoserver source data
1. Download
[raster-files.zip](https://rodekruis.sharepoint.com/sites/510-CRAVK-510/_layouts/15/guestaccess.aspx?folderid=0fa454e6dc0024dbdba7a178655bdc216&authkey=AcqhM85JHZY8cc6H7BTKgO0&expiration=2021-08-27T22%3A00%3A00.000Z&e=MnocDf)
1. Download [raster-files.zip](https://510ibfsystem.blob.core.windows.net/rasters/raster-files.zip)
2. Unzip the files using `apt install unzip` and `unzip raster-files.zip`, into `services/API-service/geoserver-volume/raster-files/`
2. Seed database: `docker compose exec ibf-api-service npm run seed`
3. Run all mock scenarios via Swagger: `api/scripts/mock-all`

5. Setup web-hook
1. On Github
1. [Create web-hook](https://github.com/rodekruis/IBF-system/settings/hooks) to
5. Setup webhook

1. On GitHub
1. [Create webhook](https://github.com/rodekruis/IBF-system/settings/hooks) to
listen on `http://ip-address:3099/`
2. Set secret for web-hook access
2. On VM:
2. Set secret for webhook access
2. On VM
1. `sudo cp tools/webhook.service /etc/systemd/system/`
2. Set `GITHUB_WEBHOOK_SECRET` value in `/etc/systemd/system/webhook.service` as same value set in Github Webhooks
3. Verification - `ls /etc/systemd/system/`
2. Set `GITHUB_WEBHOOK_SECRET` value in `/etc/systemd/system/webhook.service` as same value set in GitHub Webhooks
3. `ls /etc/systemd/system/` to verify
4. In `/home/ibf-user/IBF-system` - `npm install github-webhook-handler`
5. `sudo systemctl enable webhook` > this makes sure the webhook also restarts again automatically on reboot
6. `sudo service webhook start`
7. Verification - `sudo service webhook status`
5. `systemctl enable webhook` to start webhook on VM reboot
6. `service webhook start` to start webhook manually
7. `service webhook status` to verify
2 changes: 1 addition & 1 deletion docs/VM_SETUP_HANDOVER.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@
4. Load initial data

1. Download
[raster-files.zip](https://rodekruis.sharepoint.com/sites/510-CRAVK-510/_layouts/15/guestaccess.aspx?folderid=0fa454e6dc0024dbdba7a178655bdc216&authkey=AcqhM85JHZY8cc6H7BTKgO0&expiration=2021-08-27T22%3A00%3A00.000Z&e=MnocDf)
[raster-files.zip](https://510ibfsystem.blob.core.windows.net/rasters/raster-files.zip)
2. Unzip the files using `apt install unzip` and `unzip raster-files.zip`, into `services/API-service/geoserver-volume/raster-files/` such that that folder now has subfolders `input`, `mock-output` and `output`.
3. Run seed script through `docker-compose exec ibf-api-service npm run seed`

Expand Down
4 changes: 0 additions & 4 deletions example.env
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
# nginx
CERTBOT_EMAIL=
FQDN=

# services/API-service
DB_HOST=ibf-local-db #docker.for.mac.localhost
DB_PORT=5432 #5437
Expand Down
7 changes: 6 additions & 1 deletion interfaces/IBF-dashboard/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,10 @@ FROM node:20-alpine

RUN apk update && apk add python3 make g++ git

RUN mkdir -p /home/node/app/node_modules && mkdir -p /home/node/app/www && chown -R node:node /home/node/app
RUN mkdir -p /home/node/app/node_modules && \
mkdir -p /home/node/app/www && \
mkdir -p /home/node/app/_www && \
chown -R node:node /home/node/app

WORKDIR /home/node/app

Expand Down Expand Up @@ -50,3 +53,5 @@ ARG WHATS_NEW_URL
ENV WHATS_NEW_URL=$WHATS_NEW_URL

RUN npm run build -- --configuration $NG_CONFIGURATION

CMD ["cp", "-r", "www/.", "_www/"]
Loading
Loading