It’s cold in my house. My hands are stiff. I’ve been installing and configuring the Gitlab Container Registry since last Friday, and it’s awesome. Awesome, in this sense, means that when it works, it makes my little heart go pitter patter as I imagine how much easier this will make my life. Then I return to reality, as configuring it is anything but simple.

This is some of the stuff I’ve figured out.

Stuff Missing From The Documentation

S3 Storage Driver Needs Region

Add region to the configuration or it will barf.

registry['storage'] = {
  's3' => {
    'accesskey' => 's3-access-key',
    'secretkey' => 's3-secret-key-for-access-key',
    'bucket' => 'your-s3-bucket'
    'region' => 'us-east-1'  # or wherever your bucket lives
  }
}

Less Obvious Stuff

Project Naming Matters

The documentation says that your container should be named group/project like docker/jenkins, which is fine. What isn’t clear is that it’s not the project name but the actual repository name. If your repository is named docker-jenkins, even if you’re able to pull it as docker/jenkins the push from docker will fail. Tag your container the same as your repository.

Adjust the Registry Config File

The documentation says that you want to edit gitlab.yml, but the file is now named config.yml and is located in /var/opt/gitlab/registry. We’re running behind haproxy for SSL offloading, and we send a 301 back when someone makes a request for an http scheme when they should be using https instead. Because of this, I have Gitlab configured to run on localhost over HTTP, and nginx knows that it’s behind a proxy. The registry doesn’t, so its auth schema is still pointing to an HTTP url. I don’t know how docker handles the redirect, so it’s just easier to edit this file and change the realm under auth: token: to use https://.

Restarting the Registry

The docs kindly tell us that restarting the registry is our own affair, but they don’t tell us how to do it. The entire system runs under sv, so it’s pretty easy.

If you were to pop into a shell in the container, you’d see that sv is monitoring directories under /opt/gitlab/service:

    9 ?        S      0:07 runsvdir -P /opt/gitlab/service log: .... <snip>

(All those dots are a good thing. Read the docs to know why.)

Under this directory we have a directory called registry, so we simply have to tell sv to bounce the service running there.

Here’s how it looked before:

root@git:/# ps ax | grep registry
25050 ?        Ss     0:00 runsv registry
25051 ?        S      0:00 svlogd -tt /var/log/gitlab/registry
25141 ?        Ssl    0:01 /opt/gitlab/embedded/bin/registry serve ./config.yml

Here’s the instruction:

root@git:/# sv h /opt/gitlab/service/registry

Here’s how it looks after:

root@git:/# ps ax | grep registry
25050 ?        Ss     0:00 runsv registry
25051 ?        S      0:00 svlogd -tt /var/log/gitlab/registry
27494 ?        Ssl    0:00 /opt/gitlab/embedded/bin/registry serve ./config.yml

In the instruction we told sv to send a HUP signal to the process. That’s usually enough to get something to restart, and in this case, the change in the PID shows that it was.

If you want to be more brutal, you can just kill the PID directly, and sv will restart it.

To do this without getting a shell on the container, use docker exec:

$ docker exec gitlab sv h /opt/gitlab/service/registry

Use the Gitlab URL For Authentication

Running w/ SSL offloading requires that we set the token realm in the config. If we don’t do this, it sets it to http:// instead of https:// because all of the SSL configuration in Gitlab is turned off.

Don’t point the token_realm at your registry hostname. It uses the auth engine from within Gitlab:

registry['token_realm'] = "https://gitlab.example.com"

(Arguably this should be obvious, but the documentation isn’t super clear which components are part of the registry package and which are part of the Gitlab package.)

S3 Permissions Are Different With v2

The S3 IAM user now needs various permissions from the Multipart set. See the storage driver docs for details.

Additional S3 Wonk

I have an environment in the US that uses S3 as the storage backend, storing blobs in us-east-1. It works fine.

I have another environment in EU that stores its blobs in eu-central-1. It does not work, throwing 403s when pushing.

After much futzing around, this issue let me to the following line:

registry_nginx['custom_gitlab_server_config'] = 'proxy_cache_convert_head off;'

With that change, pushing worked. Whatevz.