Auto-renewal of Let's Encrypt fails when Jitsi Docker host already has a well-known directory setup

[Kees]

Project details

Project description (max 200 words): We are running our own Jitsi instance. This instance runs in Docker, which is easy to update and maintain in conjunction with Ansible. However, due to Docker limitations, also with IPv6, I’ve put the forum behind an Nginx reverse proxy. This proxy also handles the certificate regeneration of Discourse (the forum). Jitsi has its own mechanism to update this. The only thing I need to do is to bind the well-known ACME environment on the Docker host with the Jitsi ACME environment, so that it’s transparently shared and simply works. Now it doesn’t, because Nginx steals the request from Jitsi, so it never gets the data to finalize the ACME process.
Your personal background within the context of the project: Senior Linux system and network engineer
Estimated timeline in terms of time boxed deliverables: I checked already briefly a month ago and couldn’t quickly find the location Jitsi uses for ACME. The goal of this topic is to find this location so that I can bind it to the Docker host location of Nginx. Depending on how familiar someone is with this, I suspect this might be an afternoon of discovering the mechanisms and find a solution. Could be less, could be more.
Resources needed or already arranged: The only thing that’s needed is time, which I have to prioritize to other things at the moment. I can manually update Let’s Encrypt, no biggy. But automation is preferred. Anyone from the @it-team willing to have a look? If not, then I’ll now and then update this topic and people can always jump in with comments and feedback

This is the repo of the Jitsi Docker solution: GitHub - jitsi/docker-jitsi-meet: Jitsi Meet on Docker

[Kees]

This is where the cert is installed: docker-jitsi-meet/ssl.conf at d6fac8e0c38294463a535442bba3dd27327b966c · jitsi/docker-jitsi-meet · GitHub

So when that variable is expanded it would be /config/acme-certs/meet.tzm.community. But before it’s placed there, there is a generation process, that’s happening somewhere else. That directory is what I need to bind.

I guess if this is ran with sh -x a verbose output is given and it will probably show where the certificate verification is taking place.

I think there is no need. Just quickly logged in interactively in the container, I think this is the directory i can transparently make available:

root@455208b3d0b0:/config# realpath ./acme.sh/meet.tzm.community/meet.tzm.community.csr
/config/acme.sh/meet.tzm.community/meet.tzm.community.csr

But this is something I will have to check in practice to verify. I’m still doubting it, but acme.sh works differently than certbot, which is usually use. That one cleans up the ACME well-known directory.

But I think this is the directory that needs to be binded.

root@455208b3d0b0:/config/acme.sh# bash acme.sh --renew-all --debug 2 --home /config/acme.sh
[Fri Jun 11 13:17:43 CEST 2021] Lets find script dir.
[Fri Jun 11 13:17:43 CEST 2021] _SCRIPT_='acme.sh'
[Fri Jun 11 13:17:43 CEST 2021] _script='/config/acme.sh/acme.sh'
[Fri Jun 11 13:17:43 CEST 2021] _script_home='/config/acme.sh'
[Fri Jun 11 13:17:43 CEST 2021] Using config home:/config/acme.sh
[Fri Jun 11 13:17:43 CEST 2021] LE_WORKING_DIR='/config/acme.sh'
https://github.com/acmesh-official/acme.sh
v2.8.8
[Fri Jun 11 13:17:43 CEST 2021] Running cmd: renewAll
[Fri Jun 11 13:17:43 CEST 2021] Using config home:/config/acme.sh
[Fri Jun 11 13:17:43 CEST 2021] default_acme_server
[Fri Jun 11 13:17:43 CEST 2021] ACME_DIRECTORY='https://acme-v02.api.letsencrypt.org/directory'
[Fri Jun 11 13:17:43 CEST 2021] _ACME_SERVER_HOST='acme-v02.api.letsencrypt.org'
[Fri Jun 11 13:17:43 CEST 2021] _stopRenewOnError
[Fri Jun 11 13:17:43 CEST 2021] _set_level='2'
[Fri Jun 11 13:17:43 CEST 2021] di='/config/acme.sh/meet.tzm.community/'
[Fri Jun 11 13:17:43 CEST 2021] d='meet.tzm.community'
[Fri Jun 11 13:17:43 CEST 2021] Using config home:/config/acme.sh
[Fri Jun 11 13:17:43 CEST 2021] ACME_DIRECTORY='https://acme-v02.api.letsencrypt.org/directory'
[Fri Jun 11 13:17:43 CEST 2021] _ACME_SERVER_HOST='acme-v02.api.letsencrypt.org'
[Fri Jun 11 13:17:43 CEST 2021] DOMAIN_PATH='/config/acme.sh/meet.tzm.community'
[Fri Jun 11 13:17:43 CEST 2021] Renew: 'meet.tzm.community'
[Fri Jun 11 13:17:43 CEST 2021] Le_API='https://acme-v02.api.letsencrypt.org/directory'
[Fri Jun 11 13:17:43 CEST 2021] Using config home:/config/acme.sh
[Fri Jun 11 13:17:43 CEST 2021] ACME_DIRECTORY='https://acme-v02.api.letsencrypt.org/directory'
[Fri Jun 11 13:17:43 CEST 2021] _ACME_SERVER_HOST='acme-v02.api.letsencrypt.org'
[Fri Jun 11 13:17:43 CEST 2021] Skip, Next renewal time is: Tue Jun 15 16:50:43 UTC 2021
[Fri Jun 11 13:17:43 CEST 2021] Add '--force' to force to renew.
[Fri Jun 11 13:17:43 CEST 2021] Return code: 2
[Fri Jun 11 13:17:43 CEST 2021] Skipped meet.tzm.community
[Fri Jun 11 13:17:43 CEST 2021] _error_level='3'
[Fri Jun 11 13:17:43 CEST 2021] _set_level='2'

I’ll wait for a few weekends, then the renewal will be triggered and I might have more time, that will help with troubleshooting.

It may be fixed now with the following below in the docker-compose override file. But need to wait 5 days to do the automatic verification. Wasting too much time on this already

# Ansible managed: /home/kjong/git/tzm-ansible/roles/jitsi/templates/docker-compose.override.yml.j2 modified on 2021-05-04 10:33:01 +0200 (CEST) by kjong on ws1
version: '3'

services:
    web:
        image: aqual1te/jitsi:latest
        volumes:
            - /var/www/acme-challenge:/home/.acme

[Kees]

Just finished my actual work, so I forced acme.sh to do an update and it then complained about that port 80 is already in use. Which is true, that’s Nginx on the Docker host. So since life is one big compilation of ugly hacks, I reversed the setup. The Docker host is now the primary certificate administrator and will use certbot to update it automatically. This works, already tested.

I’ll update the Ansible templates for the Docker compose override file and then I’ll automate the symlink creation in Ansible as well for the certificate names, which differ between acme.sh and certbot.

So, Kees closed

[Kees]

And also configured in Ansible, so it’s persistent, dynamically configured and persistent after a reinstall.

[kjong@ws1 tzm-ansible (master *)]$ git diff roles/jitsi/tasks/main.yml roles/jitsi/templates/docker-compose.override.yml.j2
diff --git a/roles/jitsi/tasks/main.yml b/roles/jitsi/tasks/main.yml
index 3dd78f4..10f97ce 100644
--- a/roles/jitsi/tasks/main.yml
+++ b/roles/jitsi/tasks/main.yml
@@ -27,6 +27,23 @@
     loop: "{{ stat_jitsi_meet.results }}"
     when: not item.stat.exists
 
+  - name: stat check for let's encrypt keys
+    stat:
+      path: "/etc/letsencrypt/live/meet.tzm.community/{{ item }}"
+    register: stat_letsencrypt
+    loop:
+      - fullchain.pem
+      - privkey.pem
+
+  - name: create key symlink
+    ansible.builtin.file:
+      src: "{{ item.stat.lnk_source | basename }}"
+      dest: "{{ (item.item != 'fullchain.pem') | ternary(item.stat.lnk_source | dirname ~ '/key.pem', item.stat.lnk_source | dirname ~ '/' ~ item.item) }}"
+      state: link
+    loop: "{{ stat_letsencrypt.results }}"
+    loop_control:
+      label: "{{ item.item }}"
+
 #  - name: copy nginx config
 #    template:
 #      src: "{{ item.src }}.j2"
@@ -152,6 +169,8 @@
         value: meet.{{ lookup('neo', domain, 'tzm-community', 'apex') }}
       - option: LETSENCRYPT_EMAIL
         value: "{{ mail_admin }}"
+      - option: LETSENCRYPT_USE_STAGING
+        value: 1
       - option: ENABLE_AUTH
         value: 1
       - option: ENABLE_GUESTS
diff --git a/roles/jitsi/templates/docker-compose.override.yml.j2 b/roles/jitsi/templates/docker-compose.override.yml.j2
index d0e779b..eba36c1 100644
--- a/roles/jitsi/templates/docker-compose.override.yml.j2
+++ b/roles/jitsi/templates/docker-compose.override.yml.j2
@@ -4,3 +4,5 @@ version: '3'
 services:
     web:
         image: aqual1te/jitsi:latest
+        volumes:
+            - /etc/letsencrypt/archive:/config/acme-certs

[Bahusson (Jakub)]

Good job. I need to finish this docker course eventually. It makes so many things easier…

[Kees]

The issue is not totally resolved. I created this issue on Github: acme.sh via Docker container on custom port · Issue #3566 · acmesh-official/acme.sh · GitHub

But I also have a quick fix in mind, which is doable since I already manage a custom fork of Jitsi for the theming and custom configuration.

[Kees]

Final fix, now it’s Kees closed for real

diff --git a/roles/jitsi/tasks/main.yml b/roles/jitsi/tasks/main.yml
index ad8100e..4ce74aa 100644
--- a/roles/jitsi/tasks/main.yml
+++ b/roles/jitsi/tasks/main.yml
@@ -37,7 +37,7 @@
 
   - name: create key symlink
     ansible.builtin.file:
-      dest: "{{ (item.item != 'fullchain.pem') | ternary(item.stat.lnk_source | dirname ~ '/key.pem', item.stat.lnk_source | dirname ~ '/' ~ item.item) }}"
+      dest: "{{ (item.item == 'fullchain.pem') | ternary(item.stat.lnk_source | dirname ~ '/cert.crt', item.stat.lnk_source | dirname ~ '/cert.key' ) }}"
       src: "{{ item.stat.lnk_source | basename }}"
       state: link
     loop: "{{ stat_letsencrypt.results }}"
@@ -164,7 +164,7 @@
       - option: ENABLE_NOISY_MIC_DETECTION
         value: 1
       - option: ENABLE_LETSENCRYPT
-        value: 1
+        value: 0
       - option: LETSENCRYPT_DOMAIN
         value: meet.{{ lookup('neo', domain, 'tzm-community', 'apex') }}
       - option: LETSENCRYPT_EMAIL
diff --git a/roles/jitsi/templates/docker-compose.override.yml.j2 b/roles/jitsi/templates/docker-compose.override.yml.j2
index eba36c1..bfbb63e 100644
--- a/roles/jitsi/templates/docker-compose.override.yml.j2
+++ b/roles/jitsi/templates/docker-compose.override.yml.j2
@@ -5,4 +5,4 @@ services:
     web:
         image: aqual1te/jitsi:latest
         volumes:
-            - /etc/letsencrypt/archive:/config/acme-certs
+            - /etc/letsencrypt/archive/meet.{{ lookup('neo', domain, 'tzm-community', 'apex') }}:/config/keys