メモ：Harbor on Ubuntu でハマった際のメモ

Ubuntu 上にHarbor をインストールした際に幾つかハマった部分があったので、それの自分用備忘録。

前提

• Ubuntu 20.04.1 LTS
• Harbor 2.2.0

手順

• docker/docker-compose のインストール
• Let's Encrypt を用いた証明書の準備
• Harbor インストール
• トラブル - その1
• トラブル - その2

docker/docker-compose のインストール

$ uname -a
Linux harbor2 5.4.0-66-generic #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ sudo apt-get install \
 apt-transport-https \
 ca-certificates \
 curl \
 gnupg-agent \
 software-properties-common

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

$ sudo apt-key fingerprint 0EBFCD88
pub   rsa4096 2017-02-22 [SCEA]
      9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid           [ unknown] Docker Release (CE deb) <docker@docker.com>
sub   rsa4096 2017-02-22 [S]

$ sudo add-apt-repository \
 "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
 $(lsb_release -cs) \
 stable"

$ sudo apt-get update

$ sudo apt-get install docker-ce docker-ce-cli containerd.io

hello-world のdocker イメージが走るかどうか確認します。

$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:308866a43596e83578c7dfa15e27a73011bdd402185a84c5cd7f32a88b501a24
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
...SNIP...

$ sudo usermod -aG docker <user>

続いて、docker-compose をインストールします。

$ sudo curl -L "https://github.com/docker/compose/releases/download/1.28.5/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

$ sudo chmod +x /usr/local/bin/docker-compose

$ docker-compose version
docker-compose version 1.28.5, build c4eb3a1f
docker-py version: 4.4.4
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

Let's Encrypt を用いた証明書の準備

Ubuntu 上で走らせるHarbor 用の証明書の準備をします。

$ certbot --server https://acme-v02.api.letsencrypt.org/directory -d "harbor2.<MYDOMAIN>" --manual --preferred-challenges dns-01 certonly --work-dir /home/demo/lab-cert/harbor2/certbot/wd --config-dir /home/demo/lab-cert/harbor2/certbot/cfg --logs-dir /home/demo/lab-cert/harbor2/certbot/logs

DNS チャレンジが反映されるまで少し時間掛かるので、以下のコマンドで設定が正しく反映されるまで少し待ちます。

$ nslookup -q=txt _acme-challenge.harbor2.<MYDOMAIN> 8.8.8.8

Harbor の設定ファイルを作成する際にここで作成された証明書と秘密鍵は利用します。

Harbor インストール

公式の手順に従って、Harbor をインストールしておきます。

$ wget https://github.com/goharbor/harbor/releases/download/v2.2.0/harbor-offline-installer-v2.2.0.tgz

$ wget https://github.com/goharbor/harbor/releases/download/v2.2.0/harbor-offline-installer-v2.2.0.tgz.asc

$ gpg --keyserver hkps://keyserver.ubuntu.com --receive-keys 644FF454C0B4115C

$ gpg --keyserver hkps://keyserver.ubuntu.com --verify harbor-offline-installer-v2.2.0.tgz.asc

$ tar xvzf harbor-offline-installer-v2.2.0.tgz
harbor/harbor.v2.2.0.tar.gz
harbor/prepare
harbor/LICENSE
harbor/install.sh
harbor/common.sh
harbor/harbor.yml.tmpl

$ cp harbor.yml.tmpl harbor.yml

harbor.yml の hostname, certificate, private_key, harbor_admin_password を修正します。

$ vim harbor.yml

$ sudo ./install.sh --with-notary --with-trivy --with-chartmuseum
[sudo] password for demo:

[Step 0]: checking if docker is installed ...

Note: docker version: 20.10.5

[Step 1]: checking docker-compose is installed ...

Note: docker-compose version: 1.28.5

[Step 2]: loading Harbor images ...
07ed3fe22282: Loading layer [==================================================>]  34.51MB/34.51MB
632651017131: Loading layer [==================================================>]  8.071MB/8.071MB
cff019bd8e54: Loading layer [==================================================>]  3.584kB/3.584kB
db8113c9a129: Loading layer [==================================================>]   2.56kB/2.56kB
04eaffb344c9: Loading layer [==================================================>]  61.03MB/61.03MB
30932a235d0d: Loading layer [==================================================>]  61.85MB/61.85MB
Loaded image: goharbor/harbor-jobservice:v2.2.0
68170e81b04b: Loading layer [==================================================>]  34.51MB/34.51MB
c0276ff1011e: Loading layer [==================================================>]  7.815MB/7.815MB
892518eb7e09: Loading layer [==================================================>]  17.61MB/17.61MB
25f373af3c04: Loading layer [==================================================>]  4.608kB/4.608kB
df5c0f8011ee: Loading layer [==================================================>]  18.43MB/18.43MB
Loaded image: goharbor/harbor-exporter:v2.2.0
d6b0c623c73b: Loading layer [==================================================>]  4.933MB/4.933MB
494ceea2a6b4: Loading layer [==================================================>]  4.096kB/4.096kB
64e95a63b7a3: Loading layer [==================================================>]  3.072kB/3.072kB
f2c35b3b0dcd: Loading layer [==================================================>]  18.99MB/18.99MB
5c74d99fc846: Loading layer [==================================================>]  19.81MB/19.81MB
Loaded image: goharbor/registry-photon:v2.2.0
3fbc0344880d: Loading layer [==================================================>]  8.072MB/8.072MB
9c5adc52de0d: Loading layer [==================================================>]  3.584kB/3.584kB
05781011aa08: Loading layer [==================================================>]   2.56kB/2.56kB
19e4b43530bc: Loading layer [==================================================>]  53.27MB/53.27MB
9a88bba5ca8d: Loading layer [==================================================>]  5.632kB/5.632kB
7c2bf6707239: Loading layer [==================================================>]  87.55kB/87.55kB
b1aeff496e1d: Loading layer [==================================================>]  11.78kB/11.78kB
f8d3079c10d4: Loading layer [==================================================>]   54.2MB/54.2MB
eb473baf6abd: Loading layer [==================================================>]   2.56kB/2.56kB
Loaded image: goharbor/harbor-core:v2.2.0
f649b07d9770: Loading layer [==================================================>]  63.77MB/63.77MB
a1252bd74521: Loading layer [==================================================>]     80MB/80MB
12a45cabca01: Loading layer [==================================================>]  6.144kB/6.144kB
cb64020cac49: Loading layer [==================================================>]   2.56kB/2.56kB
11273c337dac: Loading layer [==================================================>]   2.56kB/2.56kB
06bf2b44257c: Loading layer [==================================================>]   2.56kB/2.56kB
ae1d550e31f7: Loading layer [==================================================>]   2.56kB/2.56kB
5418b645d05a: Loading layer [==================================================>]  11.26kB/11.26kB
Loaded image: goharbor/harbor-db:v2.2.0
165bc38d4a20: Loading layer [==================================================>]  4.926MB/4.926MB
4450dd70e473: Loading layer [==================================================>]  5.926MB/5.926MB
571aff5ac473: Loading layer [==================================================>]  14.86MB/14.86MB
7213db5cd3f6: Loading layer [==================================================>]  27.36MB/27.36MB
feb90353404b: Loading layer [==================================================>]  22.02kB/22.02kB
2bf612d23dd5: Loading layer [==================================================>]  14.86MB/14.86MB
Loaded image: goharbor/notary-server-photon:v2.2.0
75b7bc9e1233: Loading layer [==================================================>]  6.237MB/6.237MB
45cc62077a3e: Loading layer [==================================================>]  4.096kB/4.096kB
0254af6d0275: Loading layer [==================================================>]  3.072kB/3.072kB
6b42f8a7f98d: Loading layer [==================================================>]   28.3MB/28.3MB
4c3750e9c704: Loading layer [==================================================>]  11.38MB/11.38MB
2f3db0c6619f: Loading layer [==================================================>]   40.5MB/40.5MB
Loaded image: goharbor/trivy-adapter-photon:v2.2.0
bbd0a1895331: Loading layer [==================================================>]  4.933MB/4.933MB
5db7b6078317: Loading layer [==================================================>]  4.096kB/4.096kB
b2a993735d1e: Loading layer [==================================================>]  18.99MB/18.99MB
46f8d3251467: Loading layer [==================================================>]  3.072kB/3.072kB
36435ed81d46: Loading layer [==================================================>]  25.32MB/25.32MB
586ede682f3f: Loading layer [==================================================>]  45.14MB/45.14MB
Loaded image: goharbor/harbor-registryctl:v2.2.0
59cead1174d4: Loading layer [==================================================>]  35.94MB/35.94MB
8c26e21f2027: Loading layer [==================================================>]  3.072kB/3.072kB
741a65c6dac7: Loading layer [==================================================>]   59.9kB/59.9kB
438633fad008: Loading layer [==================================================>]  61.95kB/61.95kB
Loaded image: goharbor/redis-photon:v2.2.0
2fc5cd36d28c: Loading layer [==================================================>]  76.07MB/76.07MB
6a135eaee93d: Loading layer [==================================================>]  3.584kB/3.584kB
e5c3feb6aca0: Loading layer [==================================================>]  3.072kB/3.072kB
a31d1977777a: Loading layer [==================================================>]   2.56kB/2.56kB
0969721e9ff9: Loading layer [==================================================>]  3.072kB/3.072kB
e790c9ba4ed2: Loading layer [==================================================>]  3.584kB/3.584kB
ee43eb3a3893: Loading layer [==================================================>]  12.29kB/12.29kB
Loaded image: goharbor/harbor-log:v2.2.0
45a339152b94: Loading layer [==================================================>]  6.779MB/6.779MB
Loaded image: goharbor/nginx-photon:v2.2.0
e6c87254655c: Loading layer [==================================================>]  4.926MB/4.926MB
385174b02cde: Loading layer [==================================================>]  5.926MB/5.926MB
427415aeb0cc: Loading layer [==================================================>]  13.33MB/13.33MB
a46c9a86420a: Loading layer [==================================================>]  27.36MB/27.36MB
0646903e30c4: Loading layer [==================================================>]  22.02kB/22.02kB
74c332a73d82: Loading layer [==================================================>]  13.33MB/13.33MB
Loaded image: goharbor/notary-signer-photon:v2.2.0
d6c1f4fe3f89: Loading layer [==================================================>]  4.932MB/4.932MB
da140a6b9c66: Loading layer [==================================================>]  62.71MB/62.71MB
014c145ecf1c: Loading layer [==================================================>]  3.072kB/3.072kB
73ad0cb1c27d: Loading layer [==================================================>]  4.096kB/4.096kB
4d442ea85017: Loading layer [==================================================>]  63.53MB/63.53MB
Loaded image: goharbor/chartmuseum-photon:v2.2.0
c8fae5121874: Loading layer [==================================================>]  77.48MB/77.48MB
3b920f9fa989: Loading layer [==================================================>]  54.62MB/54.62MB
f156b6b2a217: Loading layer [==================================================>]   2.56kB/2.56kB
906ca23bc04b: Loading layer [==================================================>]  1.536kB/1.536kB
12b8ebf41897: Loading layer [==================================================>]  18.43kB/18.43kB
6190944c245c: Loading layer [==================================================>]  4.058MB/4.058MB
e08cb3f4e745: Loading layer [==================================================>]  278.5kB/278.5kB
Loaded image: goharbor/prepare:v2.2.0
366e44984cdc: Loading layer [==================================================>]  6.779MB/6.779MB
eb1850e4d6ec: Loading layer [==================================================>]  9.096MB/9.096MB
ecaa0fbfe5ea: Loading layer [==================================================>]  1.691MB/1.691MB
Loaded image: goharbor/harbor-portal:v2.2.0


[Step 3]: preparing environment ...

[Step 4]: preparing harbor configs ...
prepare base dir is set to /home/demo/harbor
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /data/secret/keys/secretkey
Successfully called func: create_root_cert
Successfully called func: create_root_cert
Successfully called func: create_cert
Copying certs for notary signer
Copying nginx configuration file for notary
Generated configuration file: /config/nginx/conf.d/notary.upstream.conf
Generated configuration file: /config/nginx/conf.d/notary.server.conf
Generated configuration file: /config/notary/server-config.postgres.json
Generated configuration file: /config/notary/server_env
Generated and saved secret to file: /data/secret/keys/defaultalias
Generated configuration file: /config/notary/signer_env
Generated configuration file: /config/notary/signer-config.postgres.json
Generated configuration file: /config/trivy-adapter/env
Generated configuration file: /config/chartserver/env
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir



[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating network "harbor_harbor-notary" with the default driver
Creating network "harbor_harbor-chartmuseum" with the default driver
Creating network "harbor_notary-sig" with the default driver
Creating harbor-log ... done
Creating registry      ... done
Creating harbor-portal ... done
Creating harbor-db     ... done
Creating chartmuseum   ... done
Creating redis         ... done
Creating registryctl   ... done
Creating notary-signer ... done
Creating harbor-core   ... done
Creating trivy-adapter ... done
Creating notary-server     ... done
Creating harbor-jobservice ... done
Creating nginx             ... done
✔ ----Harbor has been installed and started successfully.----

無事インストール出来ました。

トラブル - その1

trivy を用いてイメージの脆弱性スキャンをしようとすると以下の通りエラーが起きてしまい、Harbor にストアしたイメージの脆弱性スキャンが出来ませんでした。

2021-03-13T14:51:00.024Z	FATAL	failed to download vulnerability DB: failed to download vulnerability DB: failed to list releases: Get "https://api.github.com/repos/aquasecurity/trivy-db/releases": dial tcp: lookup api.github.com on 127.0.0.11:53: read udp 127.0.0.1:33885->127.0.0.11:53: i/o timeout
: general response handler: unexpected status code: 500, expected: 200: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: running trivy wrapper: running trivy: exit status 1: 

こちらのIssue を参考に、/etc/docker/daemon.json ファイルを作成し、DNS のエントリーを入れる事で解決しました。

$ cat /etc/docker/daemon.json
{
 "dns":["xxx.xxx.xxx.xxx"]
}

トラブル - その2

上の作業をする際に、 docker-compose stop で Harbor を停止させ、更に docker daemon を停止、再度立ち上げようとした際に以下のエラーが置きました。

$ systemctl status docker
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sat 2021-03-13 14:57:53 UTC; 19s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
    Process: 22302 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
   Main PID: 22302 (code=exited, status=1/FAILURE)

Mar 13 14:57:51 harbor2 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Mar 13 14:57:51 harbor2 systemd[1]: docker.service: Failed with result 'exit-code'.
Mar 13 14:57:51 harbor2 systemd[1]: Failed to start Docker Application Container Engine.
Mar 13 14:57:53 harbor2 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Mar 13 14:57:53 harbor2 systemd[1]: Stopped Docker Application Container Engine.
Mar 13 14:57:53 harbor2 systemd[1]: docker.service: Start request repeated too quickly.
Mar 13 14:57:53 harbor2 systemd[1]: docker.service: Failed with result 'exit-code'.
Mar 13 14:57:53 harbor2 systemd[1]: Failed to start Docker Application Container Engine.

$ journalctl -xe
-- A start job for unit docker.service has finished with a failure.
--
-- The job identifier is 1688 and the job result is failed.
Mar 13 14:57:53 harbor2 systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit docker.socket has entered the 'failed' state with result 'service-start-limit-hit'.
...SNIP...

こちらのIssue と、stackoverflow の記事を参考に解決出来ました。ExecStart=部分を修正しています。

$ sudo systemctl stop docker

$ sudo cp -au /var/lib/docker /var/lib/docker.bk

$ cat /etc/systemd/system/multi-user.target.wants/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecStart=/usr/bin/dockerd --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID

$ sudo systemctl start docker

$ sudo systemctl status docker
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-03-13 15:12:52 UTC; 8s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 1332 (dockerd)
      Tasks: 32
     Memory: 61.2M
...SNIP...

$ sudo docker-compose start
Starting log           ... done
Starting registry      ... done
Starting registryctl   ... done
Starting postgresql    ... done
Starting portal        ... done
Starting redis         ... done
Starting core          ... done
Starting jobservice    ... done
Starting proxy         ... done
Starting notary-signer ... done
Starting notary-server ... done
Starting trivy-adapter ... done
Starting chartmuseum   ... done

無事、Ubuntu 上でHarbor が稼働し、脆弱性スキャンも動く様になりました。