diff --git a/.env.example b/.env.example index 31190393..dba9a3bb 100644 --- a/.env.example +++ b/.env.example @@ -1,3 +1,4 @@ +# Base APP_DIR=/srv/app # Host Ports @@ -71,6 +72,7 @@ TEST_CKAN_DATASTORE_READ_URL=postgresql://datastore_ro:datastore@db/datastore_te # Dev settings USE_HTTPS_FOR_DEV=false +CKAN_DEV_COMPOSE_SERVICE=ckan-dev # CKAN core ## If use docker-compose.ghcr.yml only "*.*.*" versions available in: https://github.com/mjanez/ckan-docker/pkgs/container/ckan-docker diff --git a/README.md b/README.md index 0c383c4c..1cc8887e 100644 --- a/README.md +++ b/README.md @@ -106,7 +106,7 @@ To verify a successful Docker installation, run `docker run hello-world` and `do versions for client and server. > [!NOTE] -> Learn more about [Docker](#docker-basic-commands)/[Docker Compose](#docker-compose-basic-commands) basic commands. +> Learn more about [Docker/Docker Compose](#docker-basic-commands) basic commands. > @@ -209,31 +209,38 @@ See [CKAN images](#5-ckan-images) for more details of what happens when using de #### Create an extension You can use the ckan [extension](https://docs.ckan.org/en/latest/extensions/tutorial.html#creating-a-new-extension) instructions to create a CKAN extension, only executing the command inside the CKAN container and setting the mounted `src/` folder as output: - docker compose -f docker-compose.dev.yml exec ckan-dev /bin/sh -c "ckan -c /srv/app/ckan.ini generate extension --output-dir /srv/app/src_extensions" - -![extension](https://user-images.githubusercontent.com/54408245/220623568-b4e074c7-6d07-4d27-ae29-35ce70961463.png) + ```bash + docker compose -f docker-compose.dev.yml exec ckan-dev /bin/sh -c "ckan -c /srv/app/ckan.ini generate extension --output-dir /srv/app/src_extensions" + ``` +Then, answer the prompts to configure the plugin: + + ```bash + Extension's name [must begin 'ckanext-']: ckanext-newextension + Author's name []: Joe Bloggs + Author's email []: joe@bloggs.com + Your Github user or organization name []: joebloggs + Brief description of the project []: test creating a new extension + List of keywords (separated by spaces) [CKAN]: ckanext-newextension + Do you want to include code examples? [y/N]: y + + Written: /srv/app/src_extensions/ckanext-newextension + ``` The new extension files and directories are created in the `/srv/app/src_extensions/` folder in the running container. They will also exist in the local src/ directory as local `/src` directory is mounted as `/srv/app/src_extensions/` on the ckan container. You might need to change the owner of its folder to have the appropiate permissions. #### Running HTTPS on development mode -#### Running HTTPS on development mode - Sometimes is useful to run your local development instance under HTTPS, for instance if you are using authentication extensions like [ckanext-saml2auth](https://github.com/keitaroinc/ckanext-saml2auth). To enable it, set the following in your `.env` file: -``` -``` - USE_HTTPS_FOR_DEV=true -``` -``` + ```ini + USE_HTTPS_FOR_DEV=true + ``` and update the site URL setting: -``` -``` - CKAN_SITE_URL=https://localhost:5000 -``` -``` + ```ini + CKAN_SITE_URL=https://localhost:5000 + ``` After recreating the `ckan-dev` container, you should be able to access CKAN at https://localhost:5000 @@ -327,7 +334,6 @@ ckan ├── setup ├── Dockerfile └── Dockerfile.dev - ``` > [!NOTE] @@ -337,6 +343,36 @@ ckan > * Shows any changes between the staging area and the repository: > `git diff --staged [file]` +#### Applying patches in dev mode +To apply patches in development mode, you would need to follow these steps: + +1. Ensure that your patches are placed in the [`ckan/patches`](/ckan/patches/) directory. The patches should be organized into subdirectories named after the package they are intended to patch (e.g., `ckan` or `ckanext-??`). Each patch file should end with the .patch extension. + + For example, your directory structure might look like this: + + ```bash + ckan + ├── patches + │ ├── ckan + │ │ ├── 01_datasets_per_page.patch + │ │ ├── 02_groups_per_page.patch + │ │ ├── 03_or_filters.patch + │ └── ckanext-harvest + │ └── 01_resubmit_objects.patch + ├── setup + ├── Dockerfile + └── Dockerfile.dev + ``` + +2. Navigate to the [`/src`](/src/) directory. + +3. Apply the patches using the patch command: + + ```bash + find /path/to/ckan/patches -name '*.patch' -exec patch -p1 < {} \; + ``` + + This command will recursively search the `/path/to/ckan/patches` directory for files ending with `.patch` and apply them using the patch command. Replace `/path/to/ckan/patches` with the actual path to your `ckan/patches` directory. ## ckan-docker addons ### Debugging @@ -547,8 +583,8 @@ PostgreSQL offers the command line tools [`pg_dump`](https://www.postgresql.org/ 0 0 * * * /path/to/your/script/ckan_backup_custom.sh ``` - > [!NOTE] - > Replace `/path/to/your/script` with the actual path to the `ckan_backup_custom.sh` script. + > [!NOTE] + > Replace `/path/to/your/script` with the actual path to the `ckan_backup_custom.sh` script. 8. Save and close the file. @@ -605,89 +641,8 @@ If need to use a backup, restore it: ```bash ckan -c ckan.ini user remove user_example` ``` - - ### Docker. Basic commands -#### Linux post-install steps -[These optional post-installation procedures](https://docs.docker.com/engine/install/linux-postinstall/) shows you how to configure your Linux host machine to work better with Docker. For example, managing docker with [a non-root user](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user). - -#### Configure Docker to start on boot -```bash -sudo systemctl enable docker - -# To disable this behavior, use disable instead. -sudo systemctl disable docker -``` - - -#### Clear all Docker unused objects (images, containers, networks, local volumes) -```bash -docker system prune # Clear all -docker system prune -a # Clear all (includes unused and dangling containers) - -# By default, volumes are not removed to prevent important data from being deleted if there is currently no container using the volume. Use the `--volumes` flag when running the command to prune volumes: `docker system prune -a --volumes` - -docker image prune # Clear unused images -docker container prune # Clear unused containers -docker volume prune # Clear unused volumes -docker network prune # Clear unused networks -``` - -### Docker Compose. Basic commands -More info about Docker Compose commands at [docker compose reference](https://docs.docker.com/compose/reference/). - -```bash -# Basic. All containers or specific container: -## Starts existing containers for a service. -docker compose start - -## Restarts existing containers/container for a service. -docker compose restart - -## Stops running containers without removing them. -docker compose stop - -## Pauses running containers of a service. -docker compose pause - -## Unpauses paused containers of a service. -docker compose unpause - -# Display the logs of a container. Is it possible to retrieve only the last n seconds or other -docker logs [--since 60s] -f - -## Lists containers. -docker compose ps - -## Remove all docker compose project -docker compose rm - - -# Build. -## Builds, (re)creates, starts, and attaches to containers for a service. -docker compose [-f ] up - -## Build & up all the containers. -docker compose [-f ] up -d --build - -## Build & up an specific container. -docker compose [-f ] up -d --build - -## To avoid using a cache of the previous build while creating a new image. -docker compose [-f ] build --no-cache - -## Build a project with a specific Docker Compose prefix. -docker compose [-f ] -p up -d --build - -## Log the build -docker compose build --no-cache &> docker_build.log - - -# Down -# Stops containers and removes containers, networks, volumes, and images created by up. -docker compose [-p ] down -``` - +For more information about Docker and Docker Compose's basic commands and post-installation procedures, see [Docker/Docker Compose Info](./doc/info_docker.md) ### Docker Compose. Configure a docker compose service to start on boot To have Docker Compose run automatically when you reboot a machine, you can follow the steps below: diff --git a/apache/Dockerfile b/apache/Dockerfile index 6c81cce3..d0b1691a 100644 --- a/apache/Dockerfile +++ b/apache/Dockerfile @@ -2,8 +2,9 @@ FROM httpd:2.4-alpine ENV CKAN_PORT=5000 ENV PYCSW_PORT=8000 -ENV PYCSW_CONTAINER_NAME=pycsw -ENV CKAN_CONTAINER_NAME=ckan +ENV PYCSW_COMPOSE_SERVICE=pycsw +ENV CKAN_COMPOSE_SERVICE=ckan +ENV CKAN_DEV_COMPOSE_SERVICE=ckan-dev ENV APACHE_PORT=80 ENV APACHE_LOG_DIR=/var/log/apache ENV APACHE_ROOT=/usr/local/apache2 @@ -12,8 +13,8 @@ ENV PROXY_SERVER_NAME=localhost ENV PROXY_CKAN_LOCATION=/catalog ENV PROXY_PYCSW_LOCATION=/csw -ENV PROXY_PYCSW_PROXY_PASS=http://${PYCSW_CONTAINER_NAME}:${PYCSW_PORT} -ENV PROXY_CKAN_PROXY_PASS=http://${CKAN_CONTAINER_NAME}:${CKAN_PORT} +ENV PROXY_PYCSW_PROXY_PASS=http://${PYCSW_COMPOSE_SERVICE}:${PYCSW_PORT} +ENV PROXY_CKAN_PROXY_PASS=http://${CKAN_COMPOSE_SERVICE}:${CKAN_PORT} RUN mkdir -p ${APACHE_LOG_DIR} diff --git a/doc/info_docker.md b/doc/info_docker.md new file mode 100644 index 00000000..f91721a5 --- /dev/null +++ b/doc/info_docker.md @@ -0,0 +1,80 @@ +# Docker. Basic commands +## Linux post-install steps +[These optional post-installation procedures](https://docs.docker.com/engine/install/linux-postinstall/) shows you how to configure your Linux host machine to work better with Docker. For example, managing docker with [a non-root user](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user). + +### Configure Docker to start on boot +```bash +sudo systemctl enable docker + +# To disable this behavior, use disable instead. +sudo systemctl disable docker +``` + +### Clear all Docker unused objects (images, containers, networks, local volumes) +```bash +docker system prune # Clear all +docker system prune -a # Clear all (includes unused and dangling containers) + +# By default, volumes are not removed to prevent important data from being deleted if there is currently no container using the volume. Use the `--volumes` flag when running the command to prune volumes: `docker system prune -a --volumes` + +docker image prune # Clear unused images +docker container prune # Clear unused containers +docker volume prune # Clear unused volumes +docker network prune # Clear unused networks +``` + +## Docker Compose. Basic commands +More info about Docker Compose commands at [docker compose reference](https://docs.docker.com/compose/reference/). + +```bash +# Basic. All containers or specific container: +## Starts existing containers for a service. +docker compose start + +## Restarts existing containers/container for a service. +docker compose restart + +## Stops running containers without removing them. +docker compose stop + +## Pauses running containers of a service. +docker compose pause + +## Unpauses paused containers of a service. +docker compose unpause + +# Display the logs of a container. Is it possible to retrieve only the last n seconds or other +docker logs [--since 60s] -f + +## Lists containers. +docker compose ps + +## Remove all docker compose project +docker compose rm + + +# Build. +## Builds, (re)creates, starts, and attaches to containers for a service. +docker compose [-f ] up + +## Build & up all the containers. +docker compose [-f ] up -d --build + +## Build & up an specific container. +docker compose [-f ] up -d --build + +## To avoid using a cache of the previous build while creating a new image. +docker compose [-f ] build --no-cache + +## Build a project with a specific Docker Compose prefix. +docker compose [-f ] -p up -d --build + +## Log the build +docker compose build --no-cache &> docker_build.log + + +# Down +# Stops containers and removes containers, networks, volumes, and images created by up. +docker compose [-p ] down +``` + diff --git a/docker-compose.apache.yml b/docker-compose.apache.yml index e76f817d..4fda7f59 100644 --- a/docker-compose.apache.yml +++ b/docker-compose.apache.yml @@ -3,6 +3,9 @@ volumes: ckan_logs: pg_data: solr_data: + pip_cache: + site_packages: + services: apache: @@ -47,12 +50,11 @@ services: - db - solr - redis - ports: - - "0.0.0.0:${CKAN_PORT_HOST}:${CKAN_PORT}" volumes: - ckan_storage:/var/lib/ckan - ckan_logs:/var/log - - /usr/lib/python3.9/site-packages + - pip_cache:/root/.cache/pip + - site_packages:/usr/lib/python3.9/site-packages restart: unless-stopped healthcheck: test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:${CKAN_PORT}"] @@ -73,8 +75,6 @@ services: - ckannet depends_on: - ckan - ports: - - "0.0.0.0:${PYCSW_PORT_HOST}:${PYCSW_PORT}" volumes: - ./log:${APP_DIR}/log - ./metadata:${APP_DIR}/metadata diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index 6d8681a2..366a5e87 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -3,6 +3,10 @@ volumes: ckan_logs: pg_data: solr_data: + pip_cache: + site_packages: + vscode_server: + services: apache: @@ -11,8 +15,11 @@ services: dockerfile: Dockerfile env_file: - .env + environment: + - CKAN_COMPOSE_SERVICE=${CKAN_DEV_COMPOSE_SERVICE} + - PROXY_CKAN_PROXY_PASS=http://${CKAN_DEV_COMPOSE_SERVICE}:${CKAN_PORT} depends_on: - - ckan-dev + - ${CKAN_DEV_COMPOSE_SERVICE} ports: - "0.0.0.0:${APACHE_PORT_HOST}:${APACHE_PORT}" restart: on-failure:3 @@ -30,7 +37,7 @@ services: options: max-size: "100m" max-file: "10" - depends_on: + links: - db - solr - redis @@ -40,8 +47,9 @@ services: - ckan_storage:/var/lib/ckan - ckan_logs:/var/log - ./src:/srv/app/src_extensions - - /usr/lib/python3.9/site-packages - - /root/.vscode-server + - pip_cache:/root/.cache/pip + - site_packages:/usr/lib/python3.9/site-packages + - vscode_server:/root/.vscode-server restart: unless-stopped healthcheck: test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:${CKAN_PORT}"] @@ -52,6 +60,11 @@ services: dockerfile: Dockerfile env_file: - .env + logging: + driver: "json-file" + options: + max-size: "100m" + max-file: "10" depends_on: - ckan-dev ports: @@ -59,7 +72,7 @@ services: volumes: - ./log:${APP_DIR}/log - ./metadata:${APP_DIR}/metadata - restart: unless-stopped + restart: on-failure:3 healthcheck: test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:${PYCSW_PORT}"] @@ -78,6 +91,11 @@ services: - DATASTORE_DB volumes: - pg_data:/var/lib/postgresql/data + logging: + driver: "json-file" + options: + max-size: "100m" + max-file: "10" restart: unless-stopped healthcheck: test: ["CMD", "pg_isready", "-U", "${POSTGRES_USER}", "-d", "${POSTGRES_DB}"] diff --git a/docker-compose.ghcr.yml b/docker-compose.ghcr.yml index d81a4ab0..94b7219e 100644 --- a/docker-compose.ghcr.yml +++ b/docker-compose.ghcr.yml @@ -3,6 +3,9 @@ volumes: ckan_logs: pg_data: solr_data: + pip_cache: + site_packages: + services: nginx: @@ -24,6 +27,7 @@ services: ports: - "0.0.0.0:${NGINX_PORT_HOST}:${NGINX_PORT}" - "0.0.0.0:${NGINX_SSLPORT_HOST}:${NGINX_SSLPORT}" + restart: on-failure:3 ckan: build: @@ -50,7 +54,8 @@ services: volumes: - ckan_storage:/var/lib/ckan - ckan_logs:/var/log - - /usr/lib/python3.9/site-packages + - pip_cache:/root/.cache/pip + - site_packages:/usr/lib/python3.9/site-packages restart: unless-stopped healthcheck: test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:${CKAN_PORT}"] @@ -71,8 +76,6 @@ services: - ckannet depends_on: - ckan - ports: - - "0.0.0.0:${PYCSW_PORT_HOST}:${PYCSW_PORT}" volumes: - ./log:${APP_DIR}/log - ./metadata:${APP_DIR}/metadata diff --git a/docker-compose.yml b/docker-compose.yml index 94e189ee..c087ac81 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -3,6 +3,8 @@ volumes: ckan_logs: pg_data: solr_data: + pip_cache: + site_packages: services: @@ -25,6 +27,7 @@ services: ports: - "0.0.0.0:${NGINX_PORT_HOST}:${NGINX_PORT}" - "0.0.0.0:${NGINX_SSLPORT_HOST}:${NGINX_SSLPORT}" + restart: on-failure:3 ckan: build: @@ -51,7 +54,8 @@ services: volumes: - ckan_storage:/var/lib/ckan - ckan_logs:/var/log - - /usr/lib/python3.9/site-packages + - pip_cache:/root/.cache/pip + - site_packages:/usr/lib/python3.9/site-packages restart: unless-stopped healthcheck: test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:${CKAN_PORT}"] @@ -65,11 +69,19 @@ services: logging: driver: "json-file" options: - max-size: "100m" - max-file: "10" - restart: unless-stopped + max-size: "100m" + max-file: "10" + networks: + - webnet + - ckannet + depends_on: + - ckan + volumes: + - ./log:${APP_DIR}/log + - ./metadata:${APP_DIR}/metadata + restart: on-failure:3 healthcheck: - test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:5000"] + test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:${PYCSW_PORT}"] db: build: @@ -114,7 +126,7 @@ services: - solr_data:/var/solr restart: unless-stopped healthcheck: - test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:8983/solr/"] + test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:${SOLR_PORT}/solr/"] redis: image: redis:${REDIS_VERSION} @@ -127,7 +139,7 @@ services: - redisnet restart: unless-stopped healthcheck: - test: [ "CMD", "redis-cli", "-e", "QUIT" ] + test: ["CMD", "redis-cli", "-e", "QUIT"] networks: webnet: @@ -137,4 +149,4 @@ networks: dbnet: internal: true redisnet: - internal: true + internal: true \ No newline at end of file diff --git a/nginx/Dockerfile b/nginx/Dockerfile index 90e9a182..afca9823 100644 --- a/nginx/Dockerfile +++ b/nginx/Dockerfile @@ -2,13 +2,13 @@ FROM nginx:stable-alpine ENV CKAN_PORT=5000 ENV PYCSW_PORT=8000 -ENV PYCSW_CONTAINER_NAME=pycsw -ENV CKAN_CONTAINER_NAME=ckan +ENV PYCSW_COMPOSE_SERVICE=pycsw +ENV CKAN_COMPOSE_SERVICE=ckan ENV PROXY_SERVER_NAME=localhost ENV PROXY_CKAN_LOCATION=/catalog ENV PROXY_PYCSW_LOCATION=/csw -ENV PROXY_PYCSW_PROXY_PASS=http://${PYCSW_CONTAINER_NAME}:${PYCSW_PORT} -ENV PROXY_CKAN_PROXY_PASS=http://${CKAN_CONTAINER_NAME}:${CKAN_PORT} +ENV PROXY_PYCSW_PROXY_PASS=http://${PYCSW_COMPOSE_SERVICE}:${PYCSW_PORT} +ENV PROXY_CKAN_PROXY_PASS=http://${CKAN_COMPOSE_SERVICE}:${CKAN_PORT} ENV NGINX_PORT=80 ENV NGINX_LOG_DIR=/var/log/nginx