Skip to content

Commit

Permalink
fixes part 3
Browse files Browse the repository at this point in the history
  • Loading branch information
mluukkai committed Mar 16, 2024
1 parent 2855ae5 commit d65bb71
Show file tree
Hide file tree
Showing 5 changed files with 78 additions and 72 deletions.
11 changes: 7 additions & 4 deletions docs/part-3/section-1.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,16 @@ We've focused on using Docker as a tool to solve various types of problems. Mean

The goal for this part is to look into some of the best container practices and improve our processes.

In [part 1](/part-1/section-3#exercises-15---16) we talked about how [Alpine Linux](https://www.alpinelinux.org/) image can be quite a bit than Ubuntu but didn't really care about why we'd choose one above the other. On top of that, we have been running the applications as [root](https://en.wikipedia.org/wiki/Superuser), i.e. the super user, which is potentially dangerous.
In [part 1](/part-1/section-3#exercises-15---16), we mentioned how the [Alpine Linux](https://www.alpinelinux.org/) image is much smaller than Ubuntu, but we didn't dive any reasoning why we might pick one over the other.

On top of that, we have been running the applications as [root](https://en.wikipedia.org/wiki/Superuser), i.e. the super user, which is potentially dangerous.

## Look into official images

Which version is considered to be the "official" version is up to the maintainers of [Docker Official Images](https://github.com/docker-library/official-images) to decide. The official images [repository](https://github.com/docker-library/official-images) contains a library of images considered official. They are introduced into the library by regular pull request processes. The extended process for verifying an image is described in the repository README.
Which version is considered to be the "official" version is up to the maintainers of [Docker Official Images](https://github.com/docker-library/official-images) to decide. The [official images repository](https://github.com/docker-library/official-images) contains a library of images considered official. They are introduced into the library by regular pull request processes. The extended process for verifying an image is described in the repository README.

Many of the most well known projects are maintained under the [docker-library](https://github.com/docker-library) organization. Those include images such as Postgres and Python. But many of them are added to the library, but are maintained in a separate organization, such as Ubuntu and [Nodejs](https://github.com/nodejs/docker-node).
Many of the most well-known projects are maintained under the [docker-library](https://github.com/docker-library) organization. Those include images such as Postgres and Python.
However, many of them are included in the library but are managed by a separate organization, like Ubuntu and [Node.js](https://github.com/nodejs/docker-node)

Let's look into the Ubuntu image on [Docker Hub](https://hub.docker.com/r/library/ubuntu/) and trace the process.

Expand Down Expand Up @@ -41,7 +44,7 @@ Notice how the file is not extracted at any point. The `ADD` instruction [docume

We could verify the checksums of the file if we were interested. For the Ubuntu image automation from the launchpad takes care of creating the PRs to docker-library and the maintainers of the official images repository verify the PRs.

You can also visit the Docker Hub page for the image tag itself, which shows the layers and warns about potential security issues. Note how many different problems it finds [here](https://hub.docker.com/layers/library/ubuntu/22.04/images/sha256-b2175cd4cfdd5cdb1740b0e6ec6bbb4ea4892801c0ad5101a81f694152b6c559?context=explore).
You can also visit the Docker Hub page for the image tag itself, which shows the layers and warns about potential security issues. You can see how many different problems it finds [here](https://hub.docker.com/layers/library/ubuntu/22.04/images/sha256-b2175cd4cfdd5cdb1740b0e6ec6bbb4ea4892801c0ad5101a81f694152b6c559?context=explore).

Now we have learned that the build processes are open and we can verify it if we have the need. In addition, we learned that there's nothing that makes the "official" images special.

Expand Down
30 changes: 15 additions & 15 deletions docs/part-3/section-2.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@ Since we cannot assume that everyone has access to their own server, we will dem

We will use [GitHub Actions](https://github.com/features/actions) to build an image and push the image to Docker Hub, and then use a project called [Watchtower](https://containrrr.dev/watchtower/) to automatically pull and restart the new image in the target machine.

As example, we will look repository [https://github.com/docker-hy/docker-hy.github.io](https://github.com/docker-hy/docker-hy.github.io), that is, the material of this course.
As an example, we will look repository [https://github.com/docker-hy/docker-hy.github.io](https://github.com/docker-hy/docker-hy.github.io), that is, the material of this course.

As was said [GitHub Actions](https://github.com/features/actions) is used to implement the first part of the deployment pipeline. The [documentation](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions) gives the following overview:

_GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. You can create workflows that build and test commit and every pull request to your repository, or deploy merged pull requests to production._

The project defines a _workflow_ with GitHub Actions that builds a Docker image and pushes it to Docker Hub every time the code is pushed to the GitHub repository.
The [project](https://github.com/docker-hy/docker-hy.github.io) defines a _workflow_ with GitHub Actions that builds a Docker image and pushes it to Docker Hub every time the code is pushed to the GitHub repository.

Let us now see how the workflow definition looks. It is stored in the file _deploy.yml_ inside the _.github/workflows_ directory:

Expand Down Expand Up @@ -61,19 +61,19 @@ jobs:
The [workflow](https://docs.github.com/en/actions/using-workflows) has two [jobs](https://docs.github.com/en/actions/using-jobs/using-jobs-in-a-workflow), we are now interested in the one that is called _publish-docker-hub_. The other job, called _deploy_ takes care of deploying the page as a GitHub page.
A job consists of series of [steps](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idsteps). Each step is a small operation or _action_ that does its part of the whole. The steps are the following
A job consists of a series of [steps](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idsteps). Each step is a small operation or _action_ that does its part of the whole. The steps are the following
- [actions/checkout@v2](https://github.com/actions/checkout) is used to check out the code from the repository
- [docker/login-action@v1](https://github.com/docker/login-action) is used to log in to Docker Hub
- [docker/build-push-action@v2](https://github.com/docker/build-push-action) is used to build the image and push it to Docker Hub
The first action was one of the ready made actions that GitHub provides. The latter two are official actions offered by Docker. See [here](https://github.com/marketplace/actions/build-and-push-docker-images) for more info about the official Docker GitHub Actions.
The first action was one of the ready-made actions that GitHub provides. The latter two are official actions offered by Docker. See [here](https://github.com/marketplace/actions/build-and-push-docker-images) for more info about the official Docker GitHub Actions.
Before the workflow will work, two [secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets) should be added to the GitHub repository: `DOCKERHUB_TOKEN` and `DOCKERHUB_USERNAME`. This is done by opening the repository in browser and first pressing *Settings* then *Secrets*. The `DOCKERHUB_TOKEN` can be created in Docker Hub from the *Account Settings / Security*.
Before the workflow will work, two [secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets) should be added to the GitHub repository: `DOCKERHUB_TOKEN` and `DOCKERHUB_USERNAME`. This is done by opening the repository in the browser and first pressing *Settings* then *Secrets*. The `DOCKERHUB_TOKEN` can be created in Docker Hub from the *Account Settings / Security*.

GitHub Actions are doing only the "first half" of the deployment pipeline: they are ensuring that every push to GitHub is built to an Docker image which is then pushed to Docker Hub.
GitHub Actions are doing only the "first half" of the deployment pipeline: they are ensuring that every push to GitHub is built to a Docker image which is then pushed to Docker Hub.

The other half of the deployment pipeline is implemented by a containerized service called [Watchtower](https://github.com/containrrr/watchtower) that is an open source project that automates the task of updating images. Watchtower will pull the source of the image (in this case Docker Hub) for changes in the containers that are running. The container that is running will be updated and automatically restarted when a new version of the image is pushed to Docker Hub. Watchtower respects tags e.g. container using ubuntu:18.04 will not be updated unless a new version of ubuntu:18.04 is released.
The other half of the deployment pipeline is implemented by a containerized service called [Watchtower](https://github.com/containrrr/watchtower) which is an open-source project that automates the task of updating images. Watchtower will pull the source of the image (in this case Docker Hub) for changes in the containers that are running. The container that is running will be updated and automatically restarted when a new version of the image is pushed to Docker Hub. Watchtower respects tags e.g. q container using ubuntu:22.04 will not be updated unless a new version of ubuntu:22.04 is released.

:::tip Security reminder: Docker Hub accessing your computer

Expand Down Expand Up @@ -102,15 +102,15 @@ One needs to be careful when starting Watchtower with _docker compose up_, sinc

:::info Exercise 3.1: Your pipeline

Create now a similar deployment pipeline for a simple NodeJS/Express app found
Create now a similar deployment pipeline for a simple Node.js/Express app found
[here](https://github.com/docker-hy/material-applications/tree/main/express-app).

Either clone the project or copy the files to your own repository. Set up similar deployment pipeline (or the "first half") using GitHub Actions that was just described. Ensure that a new image gets pushed to Docker Hub every time you push the code to GitHub (you may eg. change the message the app shows).
Either clone the project or copy the files to your own repository. Set up a similar deployment pipeline (or the "first half") using GitHub Actions that was just described. Ensure that a new image gets pushed to Docker Hub every time you push the code to GitHub (you may eg. change the message the app shows).

Note that there is importat change that you should make to the above workflow configuration, the branch should be named _main_:
Note that there is important change that you should make to the above workflow configuration, the branch should be named _main_:

```yaml
name: Release NodeJS app
name: Release Node.js app
on:
push:
Expand Down Expand Up @@ -168,7 +168,7 @@ Now your deployment pipeline is set up! Ensure that it works:

:::

:::info Exercise 3.3: Building images inside of a container
:::info Exercise 3.3: Scripting magic

Create a now script/program that downloads a repository from GitHub, builds a Dockerfile located in the root and then publishes it into the Docker Hub.

Expand All @@ -184,7 +184,7 @@ Now your deployment pipeline is set up! Ensure that it works:

:::

:::info Exercise 3.4: Building images inside of a container
:::info Exercise 3.4: Building images from inside of a container

As seen from the Docker Compose file, the Watchtower uses a volume to [docker.sock](https://stackoverflow.com/questions/35110146/can-anyone-explain-docker-sock) socket to access the Docker daemon of the host from the container:

Expand All @@ -209,10 +209,10 @@ Your Dockerized could be run like this (the command is divided into many lines f
docker run -e DOCKER_USER=mluukkai \
-e DOCKER_PWD=password_here \
-v /var/run/docker.sock:/var/run/docker.sock \
builder mluukkai/express_app mluukkai/testing2
builder mluukkai/express_app mluukkai/testing
```

Note that now the Docker Hub credentials are defined as environment variables since the script needs to login to Docker Hub for the push.
Note that now the Docker Hub credentials are defined as environment variables since the script needs to log in to Docker Hub for the push.

Submit the Dockerfile and the final version of your script.

Expand Down
60 changes: 30 additions & 30 deletions docs/part-3/section-3.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,22 @@
title: 'Using a non-root user'
---

Let's get back to the youtube-dl application, that we for last time worked with it [Part 2](http://localhost:8000/part-2/1-migrating-to-docker-compose#volumes-in-docker-compose).
Let's get back to the yt-dlp application, that we for last time worked with it [Part 2](http://localhost:8000/part-2/1-migrating-to-docker-compose#volumes-in-docker-compose).

The application could, in theory, escape the container due to a bug in Docker or Linux kernel. To mitigate this security issue we will add a non-root user to our container and run our process with that user. Another option would be to map the root user to a high, non-existing user id on the host with https://docs.docker.com/engine/security/userns-remap/, and can be used in case you must use root within the container.

The Dockerfile that we did in [Part 1](/part-1/section-4) was this:

```dockerfile
FROM ubuntu:18.04
FROM ubuntu:22.04

WORKDIR /mydir

RUN apt-get update
RUN apt-get install -y curl python
RUN curl -L https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl -o /usr/local/bin/youtube-dl
RUN chmod a+x /usr/local/bin/youtube-dl
RUN apt-get update && apt-get install -y curl python3
RUN curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp
RUN chmod a+x /usr/local/bin/yt-dlp

ENV LC_ALL=C.UTF-8

ENTRYPOINT ["/usr/local/bin/youtube-dl"]
ENTRYPOINT ["/usr/local/bin/yt-dlp"]
```

We will add an user called _appuser_ with the following command
Expand All @@ -32,43 +29,46 @@ RUN useradd -m appuser
After that we change the user with the directive [USER](https://docs.docker.com/engine/reference/builder/#user) - so all commands after this line will be executed as our new user, including the `CMD` and `ENTRYPOINT`.

```dockerfile
FROM ubuntu:18.04
FROM ubuntu:22.04

WORKDIR /usr/videos
WORKDIR /mydir

ENV LC_ALL=C.UTF-8
RUN apt-get update && apt-get install -y curl python3
RUN curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp
RUN chmod a+x /usr/local/bin/yt-dlp

RUN apt-get update
RUN apt-get install -y curl python
RUN curl -L https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl -o /usr/local/bin/youtube-dl
RUN chmod a+x /usr/local/bin/youtube-dl
RUN useradd -m appuser

USER appuser

ENTRYPOINT ["/usr/local/bin/youtube-dl"]
ENTRYPOINT ["/usr/local/bin/yt-dlp"]
```

The `WORKDIR` is renamed to /usr/videos since it makes more sense as the videos will be downloaded there. When we run this image without bind mounting our local directory:

```console
$ docker container run youtube-dl https://imgur.com/JY5tHqr
$ docker run yt-dlp https://www.youtube.com/watch?v=XsqlHHTGQrw

...

[Imgur] JY5tHqr: Downloading webpage
[download] Destination: Imgur-JY5tHqr.mp4
[download] 100% of 190.20KiB in 00:0044MiB/s ETA 00:000
ERROR: unable to open for writing: [Errno 13] Permission denied: 'Imgur-JY5tHqr.mp4.part'
[info] XsqlHHTGQrw: Downloading 1 format(s): 22
[download] Unable to open file: [Errno 13] Permission denied: 'Master’s Programme in Computer Science | University of Helsinki [XsqlHHTGQrw].mp4.part'. Retrying (1/3)...
[download] Unable to open file: [Errno 13] Permission denied: 'Master’s Programme in Computer Science | University of Helsinki [XsqlHHTGQrw].mp4.part'. Retrying (2/3)...
[download] Unable to open file: [Errno 13] Permission denied: 'Master’s Programme in Computer Science | University of Helsinki [XsqlHHTGQrw].mp4.part'. Retrying (3/3)...

ERROR: unable to open for writing: [Errno 13] Permission denied: 'Master’s Programme in Computer Science | University of Helsinki [XsqlHHTGQrw].mp4.part'
```

We'll see that our `appuser` user can not write to `/usr/videos` - this can be fixed with `chown` or not fix it at all, if the intented usage is to always have a `/usr/videos` mounted from the host. By mounting the directory the application works as intended.
We'll see that our `appuser` user has no access to write to the container filesystem. This can be fixed with `chown` or not fix it at all, if the intented usage is to always have a `/mydir` mounted from the host. By mounting the directory the application works as intended.

If we want to give the `appuser` permission to write inside the container, the permission change must be done when we are still executing as root, that is, before the directive `USER` is used to change the user:
If we want to give the `appuser` permission to write inside the container, the permission change must be done when we are still executing as root, that is, before the directive `USER` is used to change the user:

```dockerfile
FROM ubuntu:18.04
FROM ubuntu:22.04

# ...

WORKDIR /mydir

# create the appuser
RUN useradd -m appuser

Expand All @@ -78,20 +78,20 @@ RUN chown appuser .
# now we can change the user
USER appuser

ENTRYPOINT ["/usr/local/bin/youtube-dl"]
ENTRYPOINT ["/usr/local/bin/yt-dlp"]
```

## Exercise 3.5

:::caution Mandatory Exercise 3.5

In exercises [1.12](/part-1/section-6#exercises-111-114) and [1.13](/part-1/section-6#exercises-111-114) we created Dockerfiles for both example [frontend](https://github.com/docker-hy/material-applications/tree/main/example-frontend) and [backend](https://github.com/docker-hy/material-applications/tree/main/example-backend).
In exercises [1.12](/part-1/section-6#exercises-111-114) and [1.13](/part-1/section-6#exercises-111-114) we created Dockerfiles for both [frontend](https://github.com/docker-hy/material-applications/tree/main/example-frontend) and [backend](https://github.com/docker-hy/material-applications/tree/main/example-backend).

Security issues with the user being a root are serious for the example frontend and backend as the containers for web services are supposed to be accessible through the Internet.

Make sure the containers start their processes as a non-root user.
Make sure the containers start their processes as non-root user.

Backend image is based on [Alpine Linux](https://www.alpinelinux.org/), that does not support the command `useradd`. Google will surely help you a way to create user in an `alpine` based image.
The backend image is based on [Alpine Linux](https://www.alpinelinux.org/), which does not support the command `useradd`. Google will surely help you a way to create a user in an `alpine` based image.

Submit the Dockerfiles.

Expand Down
Loading

0 comments on commit d65bb71

Please sign in to comment.