Merge pull request #130 from mjanez/ckan-2.9.11

Update Docker Compose files with new volume configurations

.env.example

@@ -1,3 +1,4 @@
+# Base
 APP_DIR=/srv/app
 
 # Host Ports
@@ -71,6 +72,7 @@ TEST_CKAN_DATASTORE_READ_URL=postgresql://datastore_ro:datastore@db/datastore_te
 
 # Dev settings
 USE_HTTPS_FOR_DEV=false
+CKAN_DEV_COMPOSE_SERVICE=ckan-dev
 
 # CKAN core
 ## If use docker-compose.ghcr.yml only "*.*.*" versions available in: https://github.com/mjanez/ckan-docker/pkgs/container/ckan-docker

README.md

@@ -106,7 +106,7 @@ To verify a successful Docker installation, run `docker run hello-world` and `do
 versions for client and server.
 
 > [!NOTE]
-> Learn more about [Docker](#docker-basic-commands)/[Docker Compose](#docker-compose-basic-commands) basic commands.
+> Learn more about [Docker/Docker Compose](#docker-basic-commands) basic commands.
 > 
 
 
@@ -209,31 +209,38 @@ See [CKAN images](#5-ckan-images) for more details of what happens when using de
 #### Create an extension
 You can use the ckan [extension](https://docs.ckan.org/en/latest/extensions/tutorial.html#creating-a-new-extension) instructions to create a CKAN extension, only executing the command inside the CKAN container and setting the mounted `src/` folder as output:
 
-    docker compose -f docker-compose.dev.yml exec ckan-dev /bin/sh -c "ckan -c /srv/app/ckan.ini generate extension --output-dir /srv/app/src_extensions"
-    
-![extension](https://user-images.githubusercontent.com/54408245/220623568-b4e074c7-6d07-4d27-ae29-35ce70961463.png)
+  ```bash
+  docker compose -f docker-compose.dev.yml exec ckan-dev /bin/sh -c "ckan -c /srv/app/ckan.ini generate extension --output-dir /srv/app/src_extensions"
+  ```
 
+Then, answer the prompts to configure the plugin:
+
+  ```bash
+  Extension's name [must begin 'ckanext-']: ckanext-newextension
+  Author's name []: Joe Bloggs
+  Author's email []: [email protected]
+  Your Github user or organization name []: joebloggs
+  Brief description of the project []: test creating a new extension
+  List of keywords (separated by spaces) [CKAN]: ckanext-newextension
+  Do you want to include code examples? [y/N]: y
+
+  Written: /srv/app/src_extensions/ckanext-newextension
+  ```
 
 The new extension files and directories are created in the `/srv/app/src_extensions/` folder in the running container. They will also exist in the local src/ directory as local `/src` directory is mounted as `/srv/app/src_extensions/` on the ckan container. You might need to change the owner of its folder to have the appropiate permissions.
 
 #### Running HTTPS on development mode
-#### Running HTTPS on development mode
-
 Sometimes is useful to run your local development instance under HTTPS, for instance if you are using authentication extensions like [ckanext-saml2auth](https://github.com/keitaroinc/ckanext-saml2auth). To enable it, set the following in your `.env` file:
 
-```
-```
-  USE_HTTPS_FOR_DEV=true
-```
-```
+  ```ini
+    USE_HTTPS_FOR_DEV=true
+  ```
 
 and update the site URL setting:
 
-```
-```
-  CKAN_SITE_URL=https://localhost:5000
-```
-```
+  ```ini
+    CKAN_SITE_URL=https://localhost:5000
+  ```
 
 After recreating the `ckan-dev` container, you should be able to access CKAN at https://localhost:5000
 
@@ -327,7 +334,6 @@ ckan
 ├── setup
 ├── Dockerfile
 └── Dockerfile.dev
-
 ```
 
 > [!NOTE]
@@ -337,6 +343,36 @@ ckan
 > * Shows any changes between the staging area and the repository:
 >   `git diff --staged [file]`
 
+#### Applying patches in dev mode
+To apply patches in development mode, you would need to follow these steps:
+
+1. Ensure that your patches are placed in the [`ckan/patches`](/ckan/patches/) directory. The patches should be organized into subdirectories named after the package they are intended to patch (e.g., `ckan` or `ckanext-??`). Each patch file should end with the .patch extension.
+
+    For example, your directory structure might look like this:
+
+    ```bash
+    ckan
+    ├── patches
+    │   ├── ckan
+    │   │   ├── 01_datasets_per_page.patch
+    │   │   ├── 02_groups_per_page.patch
+    │   │   ├── 03_or_filters.patch
+    │   └── ckanext-harvest
+    │       └── 01_resubmit_objects.patch
+    ├── setup
+    ├── Dockerfile
+    └── Dockerfile.dev
+    ```
+
+2. Navigate to the [`/src`](/src/) directory.
+
+3. Apply the patches using the patch command:
+
+    ```bash
+    find /path/to/ckan/patches -name '*.patch' -exec patch -p1 < {} \;
+    ```
+
+    This command will recursively search the `/path/to/ckan/patches` directory for files ending with `.patch` and apply them using the patch command. Replace `/path/to/ckan/patches` with the actual path to your `ckan/patches` directory.
 
 ## ckan-docker addons
 ### Debugging
@@ -547,8 +583,8 @@ PostgreSQL offers the command line tools [`pg_dump`](https://www.postgresql.org/
     0 0 * * * /path/to/your/script/ckan_backup_custom.sh
     ```
 
-    > [!NOTE]
-    > Replace `/path/to/your/script` with the actual path to the `ckan_backup_custom.sh` script.
+  > [!NOTE]
+  > Replace `/path/to/your/script` with the actual path to the `ckan_backup_custom.sh` script.
   
 8. Save and close the file.
 
@@ -605,89 +641,8 @@ If need to use a backup, restore it:
    ```bash
    ckan -c ckan.ini user remove user_example`
     ```
-
-    
 ### Docker. Basic commands
-#### Linux post-install steps
-[These optional post-installation procedures](https://docs.docker.com/engine/install/linux-postinstall/) shows you how to configure your Linux host machine to work better with Docker. For example, managing docker with [a non-root user](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user).
-
-#### Configure Docker to start on boot
-```bash
-sudo systemctl enable docker
-
-# To disable this behavior, use disable instead.
-sudo systemctl disable docker
-```
-
-
-#### Clear all Docker unused objects (images, containers, networks, local volumes)
-```bash
-docker system prune # Clear all
-docker system prune -a # Clear all (includes unused and dangling containers)
-
-# By default, volumes are not removed to prevent important data from being deleted if there is currently no container using the volume. Use the `--volumes` flag when running the command to prune volumes: `docker system prune -a --volumes`
-
-docker image prune # Clear unused images
-docker container prune # Clear unused containers
-docker volume prune # Clear unused volumes
-docker network prune # Clear unused networks
-```
-
-### Docker Compose. Basic commands
-More info about Docker Compose commands at [docker compose reference](https://docs.docker.com/compose/reference/).
-
-```bash
-# Basic. All containers or specific container: <container>
-## Starts existing containers for a service.
-docker compose start <container>
-
-## Restarts existing containers/container for a service.
-docker compose restart <container>
-
-## Stops running containers without removing them.
-docker compose stop <container>
-
-## Pauses running containers of a service.
-docker compose pause <container>
-
-## Unpauses paused containers of a service.
-docker compose unpause <container>
-
-# Display the logs of a container. Is it possible to retrieve only the last n seconds or other
-docker logs [--since 60s]  <container> -f 
-
-## Lists containers.
-docker compose ps
-
-## Remove all docker compose project
-docker compose rm <container>
-
-
-# Build.
-## Builds, (re)creates, starts, and attaches to containers for a service.
-docker compose [-f <docker compose-file>] up
-
-## Build & up all the containers.
-docker compose [-f <docker compose-file>] up -d --build
-
-## Build & up an specific container.
-docker compose [-f <docker compose-file>] up -d --build <container>
-
-## To avoid using a cache of the previous build while creating a new image.
-docker compose [-f <docker compose-file>] build --no-cache
-
-## Build a project with a specific Docker Compose prefix.
-docker compose [-f <docker compose-file>] -p <my_project> up -d --build
-
-## Log the build
-docker compose build --no-cache &> docker_build.log
-
-
-# Down
-# Stops containers and removes containers, networks, volumes, and images created by up.
-docker compose [-p <my_project>] down
-```
-
+For more information about Docker and Docker Compose's basic commands and post-installation procedures, see [Docker/Docker Compose Info](./doc/info_docker.md)
 
 ### Docker Compose. Configure a docker compose service to start on boot
 To have Docker Compose run automatically when you reboot a machine, you can follow the steps below:

apache/Dockerfile

@@ -2,8 +2,9 @@ FROM httpd:2.4-alpine
 
 ENV CKAN_PORT=5000
 ENV PYCSW_PORT=8000
-ENV PYCSW_CONTAINER_NAME=pycsw
-ENV CKAN_CONTAINER_NAME=ckan
+ENV PYCSW_COMPOSE_SERVICE=pycsw
+ENV CKAN_COMPOSE_SERVICE=ckan
+ENV CKAN_DEV_COMPOSE_SERVICE=ckan-dev
 ENV APACHE_PORT=80
 ENV APACHE_LOG_DIR=/var/log/apache
 ENV APACHE_ROOT=/usr/local/apache2
@@ -12,8 +13,8 @@ ENV PROXY_SERVER_NAME=localhost
 
 ENV PROXY_CKAN_LOCATION=/catalog
 ENV PROXY_PYCSW_LOCATION=/csw
-ENV PROXY_PYCSW_PROXY_PASS=http://${PYCSW_CONTAINER_NAME}:${PYCSW_PORT}
-ENV PROXY_CKAN_PROXY_PASS=http://${CKAN_CONTAINER_NAME}:${CKAN_PORT}
+ENV PROXY_PYCSW_PROXY_PASS=http://${PYCSW_COMPOSE_SERVICE}:${PYCSW_PORT}
+ENV PROXY_CKAN_PROXY_PASS=http://${CKAN_COMPOSE_SERVICE}:${CKAN_PORT}
 
 RUN mkdir -p ${APACHE_LOG_DIR}
 

doc/info_docker.md

@@ -0,0 +1,80 @@
+# Docker. Basic commands
+## Linux post-install steps
+[These optional post-installation procedures](https://docs.docker.com/engine/install/linux-postinstall/) shows you how to configure your Linux host machine to work better with Docker. For example, managing docker with [a non-root user](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user).
+
+### Configure Docker to start on boot
+```bash
+sudo systemctl enable docker
+
+# To disable this behavior, use disable instead.
+sudo systemctl disable docker
+```
+
+### Clear all Docker unused objects (images, containers, networks, local volumes)
+```bash
+docker system prune # Clear all
+docker system prune -a # Clear all (includes unused and dangling containers)
+
+# By default, volumes are not removed to prevent important data from being deleted if there is currently no container using the volume. Use the `--volumes` flag when running the command to prune volumes: `docker system prune -a --volumes`
+
+docker image prune # Clear unused images
+docker container prune # Clear unused containers
+docker volume prune # Clear unused volumes
+docker network prune # Clear unused networks
+```
+
+## Docker Compose. Basic commands
+More info about Docker Compose commands at [docker compose reference](https://docs.docker.com/compose/reference/).
+
+```bash
+# Basic. All containers or specific container: <container>
+## Starts existing containers for a service.
+docker compose start <container>
+
+## Restarts existing containers/container for a service.
+docker compose restart <container>
+
+## Stops running containers without removing them.
+docker compose stop <container>
+
+## Pauses running containers of a service.
+docker compose pause <container>
+
+## Unpauses paused containers of a service.
+docker compose unpause <container>
+
+# Display the logs of a container. Is it possible to retrieve only the last n seconds or other
+docker logs [--since 60s]  <container> -f 
+
+## Lists containers.
+docker compose ps
+
+## Remove all docker compose project
+docker compose rm <container>
+
+
+# Build.
+## Builds, (re)creates, starts, and attaches to containers for a service.
+docker compose [-f <docker compose-file>] up
+
+## Build & up all the containers.
+docker compose [-f <docker compose-file>] up -d --build
+
+## Build & up an specific container.
+docker compose [-f <docker compose-file>] up -d --build <container>
+
+## To avoid using a cache of the previous build while creating a new image.
+docker compose [-f <docker compose-file>] build --no-cache
+
+## Build a project with a specific Docker Compose prefix.
+docker compose [-f <docker compose-file>] -p <my_project> up -d --build
+
+## Log the build
+docker compose build --no-cache &> docker_build.log
+
+
+# Down
+# Stops containers and removes containers, networks, volumes, and images created by up.
+docker compose [-p <my_project>] down
+```
+

docker-compose.apache.yml

@@ -3,6 +3,9 @@ volumes:
   ckan_logs:
   pg_data:
   solr_data:
+  pip_cache:
+  site_packages:
+
 
 services:
   apache:
@@ -47,12 +50,11 @@ services:
       - db
       - solr
       - redis
-    ports:
-      - "0.0.0.0:${CKAN_PORT_HOST}:${CKAN_PORT}" 
     volumes:
       - ckan_storage:/var/lib/ckan
       - ckan_logs:/var/log
-      - /usr/lib/python3.9/site-packages
+      - pip_cache:/root/.cache/pip
+      - site_packages:/usr/lib/python3.9/site-packages
     restart: unless-stopped
     healthcheck:
       test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:${CKAN_PORT}"]
@@ -73,8 +75,6 @@ services:
       - ckannet
     depends_on:
       - ckan
-    ports:
-      - "0.0.0.0:${PYCSW_PORT_HOST}:${PYCSW_PORT}"
     volumes:
         - ./log:${APP_DIR}/log
         - ./metadata:${APP_DIR}/metadata