==========
== 0x2f ==
==========

Running the TrueCharts Baikal chart from Incubator

EDIT: This is better now

Thanks to changes made in https://github.com/ckulka/baikal-docker/issues/134 it’s now possible to set BAIKAL_SKIP_CHOWN=true env var through the TrueNAS “Extra Environment Variables” field. This removes the need for making a bind mount as described later in this post.

Fingers crossed for enabling this by default in the TrueCharts image: https://github.com/truecharts/charts/issues/16573

Why is this happening

It doesn’t work out of the box. You need to make a bind mount that removes one file from within the image. When trying to run the Baikal TrueCharts chart, you’ll find that the app will be stuck in the “Deploying” state. The Docker image that this chart uses (baikal-docker:nginx) works like this:

  1. On startup, run docker-entrypoint.sh
  2. This will in turn run the .sh scripts from docker-entrypoint.d/

The files in that folder are:

  • 40-disable-nginx-ipv6-if-unsupported.sh - probably does what it says on the tin, no problems with it.
  • 40-fix-baikal-file-permissions.sh - cause of Baikal not launching and being stuck in “deploying” state.
    Inside this file there’s chown -R nginx:nginx /var/www/baikal. Running that on TrueNAS takes a long time.
    TrueNAS Scale apps are ran using Kubernetes, which means apps need to start within 60 seconds and respond HTTP 200 to a health check. If not they’re marked as unhealthy and are restarted forever. The chown [...]’s slowness delays the startup of the server by more than 60 seconds, so Baikal never really gets a chance to start in time and respond to the healthcheck.
  • 40-php-fpm.sh - starts the server, this file needs to exist or else the Baikal will never launch

How to fix it

You need to somehow make sure that the /docker-entrypoint.d/40-fix-baikal-file-permissions.sh file won’t exist on Baikal container startup, so chown won’t be ran, thus preventing Biakal startup delay.

My fix is really simple. I just copied the /docker-entrypoint.d/ folder from the Docker image’s repo (links in prev. section) but WITHOUT the 40-fix-baikal-file-permissions.sh script. I put the folder somewhere in the TrueNAS filesystem. Then I create a Host Path mount on the Baikal app so that the original contents of the /docker-entrypoint.d/ folder are replaced by my version.

Step by step guide to do that:

  1. In TrueNAS’s web UI, go to the Apps view, click the Baikal app, and click the “Edit” button. I am now on https://my_truenas_url/ui/apps/installed/TRUECHARTS/incubator/baikal/edit (I’m on TrueNAS Scale 23.0.10)
  2. Go to the Storage and Presistence section, and in the subsection called Additional App Storage click the Add button
  3. In the Type of Dropdown storage, select Host Path
  4. In the Host Path field, type wherever you created your new /docker-entrypoint.d/ that’s missing the chown script. In my case that was /mnt/tank/docker-entrypoint.d
  5. In the Mount Path field, type in /docker-entrypoint.d. This is where the folder is inside the image, so our Host Path mount will replace it.

Save your chart and wait for Baikal to come back up. If it still doesn’t work, double check that you included the original 40-php-fpm.sh file from the Docker image’s repo into your /docker-entrypoint.d folder. Baikal won’t start if you forget to add this crucial file.

Ending notes

Doing better

My solution is not very sophisticated. You can probably do better if you know Kubernetes enough to edit the deployment.yml (initContainers, hooks?) of the app so that the chown script is missing on container start. That way you could avoid messing with the host path mounts.

This is as far as I got:

# run after ssh'ing to TrueNAS
k3s kubectl edit deployment/baikal --namespace ix-baikal

Ingress doesn’t work

There’s a weird required Traefik Middleware field in the Ingress section. Other charts don’t have this. I don’t know to put her so for now Baikal is without HTTPS.

The auto-permissions initContainers fails

Looks like the auto-permissions initContainer fails to set the permissions as well

root@truenas[/etc/rancher/k3s]# k3s kubectl logs $(k3s kubectl get pods -o name --namespace ix-baikal)  --namespace ix-baikal -c auto-permissions
Automatically correcting permissions...
Automatically correcting permissions for /var/www/baikal/config...
Failed setting permissions...
Automatically correcting permissions for /docker-entrypoint.d/...
/bin/sh: /usr/bin/nfs4xdr_winacl: not found
Failed setting permissions...
Automatically correcting permissions for /var/www/baikal/Specific...
/bin/sh: /usr/bin/nfs4xdr_winacl: not found
/bin/sh: /usr/bin/nfs4xdr_winacl: not found
Failed setting permissions...