ACME "Waiting for Nginx..."

[kasimrafique]

Contribution guidelines

• I've read the contribution guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... I understand that not following the below instructions will result in immediate closure and/or deletion of my issue.
• ... I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
• ... I have understood that answers are voluntary and community-driven, and not commercial support.
• ... I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description

Fresh install solely running mailcow. ACME container gets stuck on "Waiting for nginx" and cannot proceed to get ssl certs. Looked at issue #4530 which didn't help but pointed me in the right direction. I have dug around and found out the problem and will detail in a reply to this issue.

https://github.com/mailcow/mailcow-dockerized/blob/987a027339e1b91e75df608252d1e61d33803a04/data/Dockerfiles/acme/acme.sh#L126

Logs:

acme-mailcow-1  | Wed Jul 17 22:01:24 UTC 2024 - Waiting for Docker API...                                                                                                                                                                   acme-mailcow-1  | Wed Jul 17 22:01:24 UTC 2024 - Docker API OK                                                                                                                                                                               acme-mailcow-1  | Wed Jul 17 22:01:24 UTC 2024 - Waiting for Postfix...                                                                                                                                                                      acme-mailcow-1  | Wed Jul 17 22:01:25 UTC 2024 - Postfix OK                                                                                                                                                                                  acme-mailcow-1  | Wed Jul 17 22:01:25 UTC 2024 - Waiting for Dovecot...                                                                                                                                                                      acme-mailcow-1  | Wed Jul 17 22:01:25 UTC 2024 - Dovecot OK                                                                                                                                                                                  acme-mailcow-1  | Wed Jul 17 22:01:25 UTC 2024 - Waiting for database...                                                                                                                                                                     acme-mailcow-1  | Wed Jul 17 22:01:25 UTC 2024 - Database OK                                                                                                                                                                                 acme-mailcow-1  | Wed Jul 17 22:01:25 UTC 2024 - Waiting for Nginx...

Steps to reproduce:

- after generating mailcow.conf and running `docker compose up -d` self signed certificate is used,
when logging into UI and navigating to acme logs see "waiting for nginx..." 
- reloading / restarting results in same result

Which branch are you using?

master

Which architecture are you using?

ARM64 (aarch64)

Operating System:

Ubuntu 22.04 minimal

Server/VM specifications:

12gb ram, 2 cores

Is Apparmor, SELinux or similar active?

no

Virtualization technology:

no

Docker version:

Docker version 27.0.3, build 7d4bcd8

docker-compose version or docker compose version:

Docker Compose version v2.28.1

mailcow version:

2024-06c

Reverse proxy:

n/a

Logs of git diff:

n/a

Logs of iptables -L -vn:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)                                                                                                                                                                                                pkts bytes target     prot opt in     out     source               destination                                                                                                                                                               3666  937K MAILCOW    all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* mailcow */                                                                                                                                        140K 1385M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED                                                                                                                               8   640 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0                                                                                                                                                                 2311  209K ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0                                                                                                                                                                    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:123                                                                                                                                           284 16196 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22                                                                                                                                   26  1660 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited                                                                                                                                                                                                                                                                                                                                                                 Chain FORWARD (policy DROP 0 packets, 0 bytes)                                                                                                                                                                                                pkts bytes target     prot opt in     out     source               destination                                                                                                                                                              43628 8342K MAILCOW    all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* mailcow */                                                                                                                                       43628 8342K DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0                                                                                                                                                              43628 8342K DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0                                                                                                                                                 31225 7077K ACCEPT     all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED                                                                                                                      4525  330K DOCKER     all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0                                                                                                                                                             7878  935K ACCEPT     all  --  br-mailcow !br-mailcow  0.0.0.0/0            0.0.0.0/0                                                                                                                                                        4469  327K ACCEPT     all  --  br-mailcow br-mailcow  0.0.0.0/0            0.0.0.0/0                                                                                                                                                            0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED                                                                                                                            0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0                                                                                                                                                                   0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0                                                                                                                                                                 0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0                                                                                                                                                                  0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited                                                                                                                                                                                                                                                                                                                                                                 Chain OUTPUT (policy ACCEPT 99770 packets, 50M bytes)                                                                                                                                                                                         pkts bytes target     prot opt in     out     source               destination                                                                                                                                                               5536  441K InstanceServices  all  --  *      *       0.0.0.0/0            169.254.0.0/16                                                                                                                                                                                                                                                                                                                                                                                                 Chain DOCKER (2 references)                                                                                                                                                                                                                   pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                  0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.5           tcp dpt:8983                                                                                                                                   0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.249         tcp dpt:6379                                                                                                                                   0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.6           tcp dpt:3306                                                                                                                                   0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:110                                                                                                                                    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:143                                                                                                                                    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:993                                                                                                                                    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.8           tcp dpt:80                                                                                                                                     1    52 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:995                                                                                                                                   48  2848 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.8           tcp dpt:443                                                                                                                                    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:4190                                                                                                                                   0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:12345                                                                                                                                  3   164 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:25                                                                                                                                     2   100 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:465                                                                                                                                    2   104 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:587                                                                                                                                                                                                                                                                                                                                                                             Chain DOCKER-ISOLATION-STAGE-1 (1 references)                                                                                                                                                                                                 pkts bytes target     prot opt in     out     source               destination                                                                                                                                                               7878  935K DOCKER-ISOLATION-STAGE-2  all  --  br-mailcow !br-mailcow  0.0.0.0/0            0.0.0.0/0                                                                                                                                            0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0                                                                                                                                              1147K  240M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0                                                                                                                                                                                                                                                                                                                                                                                                             Chain DOCKER-ISOLATION-STAGE-2 (2 references)                                                                                                                                                                                                 pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                  0     0 DROP       all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0                                                                                                                                                                0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0                                                                                                                                                                198K   50M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0                                                                                                                                                                                                                                                                                                                                                                                                             Chain DOCKER-USER (1 references)                                                                                                                                                                                                              pkts bytes target     prot opt in     out     source               destination                                                                                                                                                              1215K  522M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0                                                                                                                                                                                                                                                                                                                                                                                                             Chain InstanceServices (1 references)                                                                                                                                                                                                         pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                  0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.0.2          owner UID match 0 tcp dpt:3260 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */                                                                                                                                                                                                                  0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.2.0/24       owner UID match 0 tcp dpt:3260 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */                                                                                                                                                                                                                  0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.4.0/24       owner UID match 0 tcp dpt:3260 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */                                                                                                                                                                                                                  0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.5.0/24       owner UID match 0 tcp dpt:3260 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */                                                                                                                                                                                                                  0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.0.2          tcp dpt:80 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */                                                                                                                                                                                                                                   1416  144K ACCEPT     udp  --  *      *       0.0.0.0/0            169.254.169.254      udp dpt:53 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */                                                                                                                                                                                                                                      0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.169.254      tcp dpt:53 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */                                                                                                                                                                                                                                      0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.0.3          owner UID match 0 tcp dpt:80 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */                                                                                                                                                                                                                    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.0.4          tcp dpt:80 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */                                                                                                                                                                                                                                   4101  296K ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.169.254      tcp dpt:80 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */                                                                                                                                                                                                                                      0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            169.254.169.254      udp dpt:67 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */                                                                                                                                                                                                                                      0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            169.254.169.254      udp dpt:69 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */                                                                                                                                                                                                                                     19  1444 ACCEPT     udp  --  *      *       0.0.0.0/0            169.254.169.254      udp dpt:123 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */                                                                                                                                                                                                                                     0     0 REJECT     tcp  --  *      *       0.0.0.0/0            169.254.0.0/16       tcp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */ reject-with tcp-reset                                                                                                                                                                                                                       0     0 REJECT     udp  --  *      *       0.0.0.0/0            169.254.0.0/16       udp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */ reject-with icmp-port-unreachable                                                                                                                                                                                                                                                                                                                                                                                                                                                    Chain MAILCOW (2 references)                                                                                                                                                                                                                  pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                  0     0 DROP       tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            0.0.0.0/0            /* mailcow isolation */

Logs of ip6tables -L -vn:

Chain INPUT (policy ACCEPT 2663 packets, 170K bytes)                                                                                                                                                                                          pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                237 14288 MAILCOW    all      *      *       ::/0                 ::/0                 /* mailcow */                                                                                                                                                                                                                                                                                                                                                                                    Chain FORWARD (policy DROP 0 packets, 0 bytes)                                                                                                                                                                                                pkts bytes target     prot opt in     out     source               destination                                                                                                                                                               8417 4503K MAILCOW    all      *      *       ::/0                 ::/0                 /* mailcow */                                                                                                                                        8426 4504K DOCKER-USER  all      *      *       ::/0                 ::/0                                                                                                                                                                    8426 4504K DOCKER-ISOLATION-STAGE-1  all      *      *       ::/0                 ::/0                                                                                                                                                       4332 4226K ACCEPT     all      *      br-mailcow  ::/0                 ::/0                 ctstate RELATED,ESTABLISHED                                                                                                                      4094  278K DOCKER     all      *      br-mailcow  ::/0                 ::/0                                                                                                                                                                     0     0 ACCEPT     all      br-mailcow !br-mailcow  ::/0                 ::/0                                                                                                                                                             4094  278K ACCEPT     all      br-mailcow br-mailcow  ::/0                 ::/0                                                                                                                                                                 0     0 ACCEPT     all      *      docker0  ::/0                 ::/0                 ctstate RELATED,ESTABLISHED                                                                                                                            0     0 DOCKER     all      *      docker0  ::/0                 ::/0                                                                                                                                                                        0     0 ACCEPT     all      docker0 !docker0  ::/0                 ::/0                                                                                                                                                                      0     0 ACCEPT     all      docker0 docker0  ::/0                 ::/0                                                                                                                                                                                                                                                                                                                                                                                                                Chain OUTPUT (policy ACCEPT 5201 packets, 577K bytes)                                                                                                                                                                                         pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                                                                                                                                                                                                                                                           Chain DOCKER (2 references)                                                                                                                                                                                                                   pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                  0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:110                                                                                                                                 0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:143                                                                                                                                 0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:993                                                                                                                                 0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::c  tcp dpt:80                                                                                                                                  0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:995                                                                                                                                 0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::c  tcp dpt:443                                                                                                                                 0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:4190                                                                                                                                0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::11  tcp dpt:25                                                                                                                                 0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::11  tcp dpt:465                                                                                                                                0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::11  tcp dpt:587                                                                                                                                                                                                                                                                                                                                                                         Chain DOCKER-ISOLATION-STAGE-1 (1 references)                                                                                                                                                                                                 pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                  0     0 DOCKER-ISOLATION-STAGE-2  all      br-mailcow !br-mailcow  ::/0                 ::/0                                                                                                                                                 0     0 DOCKER-ISOLATION-STAGE-2  all      docker0 !docker0  ::/0                 ::/0                                                                                                                                                    105K   61M RETURN     all      *      *       ::/0                 ::/0                                                                                                                                                                                                                                                                                                                                                                                                                  Chain DOCKER-ISOLATION-STAGE-2 (2 references)                                                                                                                                                                                                 pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                  0     0 DROP       all      *      br-mailcow  ::/0                 ::/0                                                                                                                                                                     0     0 DROP       all      *      docker0  ::/0                 ::/0                                                                                                                                                                        0     0 RETURN     all      *      *       ::/0                 ::/0                                                                                                                                                                                                                                                                                                                                                                                                                  Chain DOCKER-USER (1 references)                                                                                                                                                                                                              pkts bytes target     prot opt in     out     source               destination                                                                                                                                                               119K   71M RETURN     all      *      *       ::/0                 ::/0                                                                                                                                                                                                                                                                                                                                                                                                                  Chain MAILCOW (2 references)                                                                                                                                                                                                                  pkts bytes target     prot opt in     out     source               destination

Logs of iptables -L -vn -t nat:

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)                                                                                                                                                                                           pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                943 53900 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL                                                                                                                                                                                                                                                                                                                                                                    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)                                                                                                                                                                                                pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                                                                                                                                                                                                                                                           Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)                                                                                                                                                                                               pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                  0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL                                                                                                                                                                                                                                                                                                                                                                    Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)                                                                                                                                                                                          pkts bytes target     prot opt in     out     source               destination                                                                                                                                                               7301  617K MASQUERADE  all  --  *      !br-mailcow  172.22.1.0/24        0.0.0.0/0                                                                                                                                                              0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0                                                                                                                                                                 0     0 MASQUERADE  tcp  --  *      *       172.22.1.5           172.22.1.5           tcp dpt:8983                                                                                                                                           0     0 MASQUERADE  tcp  --  *      *       172.22.1.249         172.22.1.249         tcp dpt:6379                                                                                                                                           0     0 MASQUERADE  tcp  --  *      *       172.22.1.6           172.22.1.6           tcp dpt:3306                                                                                                                                           0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:110                                                                                                                                            0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:143                                                                                                                                            0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:993                                                                                                                                            0     0 MASQUERADE  tcp  --  *      *       172.22.1.8           172.22.1.8           tcp dpt:80                                                                                                                                             0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:995                                                                                                                                            0     0 MASQUERADE  tcp  --  *      *       172.22.1.8           172.22.1.8           tcp dpt:443                                                                                                                                            0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:4190                                                                                                                                           0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:12345                                                                                                                                          0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:25                                                                                                                                             0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:465                                                                                                                                            0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:587                                                                                                                                                                                                                                                                                                                                                                                     Chain DOCKER (2 references)                                                                                                                                                                                                                   pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                  0     0 RETURN     all  --  br-mailcow *       0.0.0.0/0            0.0.0.0/0                                                                                                                                                                0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0                                                                                                                                                                   0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:7654 to:172.22.1.249:6379                                                                                                                  0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:18983 to:172.22.1.5:8983                                                                                                                   0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:13306 to:172.22.1.6:3306                                                                                                                   0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:110 to:172.22.1.250:110                                                                                                                    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:143 to:172.22.1.250:143                                                                                                                    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:993 to:172.22.1.250:993                                                                                                                   18  1080 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:172.22.1.8:80                                                                                                                        1    52 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:995 to:172.22.1.250:995                                                                                                                   48  2848 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:172.22.1.8:443                                                                                                                      0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:4190 to:172.22.1.250:4190                                                                                                                  0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:19991 to:172.22.1.250:12345                                                                                                                3   164 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25 to:172.22.1.253:25                                                                                                                      2   100 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:465 to:172.22.1.253:465                                                                                                                    2   104 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:587 to:172.22.1.253:587

Logs of ip6tables -L -vn -t nat:

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)                                                                                                                                                                                           pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                  0     0 DOCKER     all      *      *       ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL                                                                                                                                                                                                                                                                                                                                                                    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)                                                                                                                                                                                                pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                                                                                                                                                                                                                                                           Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)                                                                                                                                                                                               pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                  0     0 DOCKER     all      *      *       ::/0                !::1                  ADDRTYPE match dst-type LOCAL                                                                                                                                                                                                                                                                                                                                                                    Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)                                                                                                                                                                                          pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                  0     0 MASQUERADE  all      *      !br-mailcow  fd4d:6169:6c63:6f77::/64  ::/0                                                                                                                                                              0     0 MASQUERADE  all      *      docker0  ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL                                                                                                                         0     0 MASQUERADE  all      *      !docker0  fd00:dead:beef:c0::/80  ::/0                                                                                                                                                                   0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:110                                                                                                                                      0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:143                                                                                                                                      0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:993                                                                                                                                      0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::c  fd4d:6169:6c63:6f77::c  tcp dpt:80                                                                                                                                       0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:995                                                                                                                                      0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::c  fd4d:6169:6c63:6f77::c  tcp dpt:443                                                                                                                                      0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:4190                                                                                                                                     0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::11  fd4d:6169:6c63:6f77::11  tcp dpt:25                                                                                                                                     0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::11  fd4d:6169:6c63:6f77::11  tcp dpt:465                                                                                                                                    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::11  fd4d:6169:6c63:6f77::11  tcp dpt:587                                                                                                                                                                                                                                                                                                                                                                             Chain DOCKER (2 references)                                                                                                                                                                                                                   pkts bytes target     prot opt in     out     source               destination                                                                                                                                                                  0     0 RETURN     all      br-mailcow *       ::/0                 ::/0                                                                                                                                                                     0     0 RETURN     all      docker0 *       ::/0                 ::/0                                                                                                                                                                        0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:110 to:[fd4d:6169:6c63:6f77::b]:110                                                                                                        0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:143 to:[fd4d:6169:6c63:6f77::b]:143                                                                                                        0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:993 to:[fd4d:6169:6c63:6f77::b]:993                                                                                                        0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:80 to:[fd4d:6169:6c63:6f77::c]:80                                                                                                          0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:995 to:[fd4d:6169:6c63:6f77::b]:995                                                                                                        0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:443 to:[fd4d:6169:6c63:6f77::c]:443                                                                                                        0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:4190 to:[fd4d:6169:6c63:6f77::b]:4190                                                                                                      0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:25 to:[fd4d:6169:6c63:6f77::11]:25                                                                                                         0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:465 to:[fd4d:6169:6c63:6f77::11]:465                                                                                                       0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:587 to:[fd4d:6169:6c63:6f77::11]:587

DNS check:

172.64.155.249                                                                                                                                                                                                                               104.18.32.7

[kasimrafique]

mailcow-dockerized/data/Dockerfiles/acme/acme.sh

Line 126 in 987a027

until $(curl --output /dev/null --silent --head --fail http://nginx:8081); do

The problem lies in this curl request, as when I exec into the container, I am able to ping the alias nginx - however when curling it fails to resolve the host

However, if I curl with the nginx containers internal ip, I get a response and the script would continue. So this is what is stopping the acme container from proceeding.

What I have done in the mean time is exec into the container and modify the acme.sh to use the nginx ip, and run the script within the container.

• This successfully gets a lets encrypt certificate and attempts to deploy. I had to restart the stack for the SSL to load properly.

But as the script is copied / built into the image, it goes back to "Waiting for Nginx..." after running docker compose down && docker compose up -d

---

If there was a way to persistantly edit the acme.sh this would fix my issue, or if we can figure out why nginx isn't resolving for curl

Appreciate any help / input - Thanks in advance :)

[weaseldum]

Hello,

I'm hitting the same bug on a fresh mailcow install. Same OS (ubuntu 22.04) and same architecture (ARM64). The same fix (replace "nginx" with "$nginx ip" in acme.sh worked to obtain a cert, but as stated a "docker compose down && docker compose up -d" reverts acme.sh to a broken state.
I believe there are 2 lines of code in acme.sh that need fixing, both reference "http://nginx".

[kasimrafique]

I've some more digging and found out that apparently this is the intended behaviour of curl curl/curl#11104? So unless an explicit entry is added to /etc/hosts curl does not resolve just nginx.

I think I have however found a fix:

From this thread if we use the full domain for the container so ContainerName.{COMPOSE_PROJECT_NAME}_mailcow-network i.e. nginx.mailcowdockerized_mailcow-network in place of nginx this works natively and does not rely on using the ip address of the nginx container.

[weaseldum]

I tested this fix as well and it works. Hopefully it will be merged soon.

[DerLinkman]

duplicate of #5973 #5928

A temporarily fix is given but we're not sure if we merge it or not...