Tanzu Kubernetes Grid(TKGm) 上に Tanzu Build Service(TBS) をインストールする

この記事は既にインストールしたHarbor を利用し、Tanzu Kubernetes Grid(TKGm) 上にTBS をインストールした際のメモです。

前提

  • Harbor
  • TKGm on vSphere 環境

手順

こちらの記事を元に準備します。

TKGm Workload Cluster の準備

公式の手順に従って、TKGm on vSphere のManagement Cluster を用意します。
Workload Cluster のControl Plane、Worker ノード共に、ノードをカスタマイズしておきたかったので、.tkg/config.yaml を以下の通り変更しています。
VSPHERE_CONTROL_PLANE_NUM_CPUS: "2"
VSPHERE_CONTROL_PLANE_MEM_MIB: "4096"
VSPHERE_CONTROL_PLANE_DISK_GIB: "64"
VSPHERE_WORKER_NUM_CPUS: "2"
VSPHERE_WORKER_MEM_MIB: "8192"
VSPHERE_WORKER_DISK_GIB: "128"

その上で、tkg create でWorkload Clusterを作成します。
$ tkg create cluster devsecops --plan dev --vsphere-controlplane-endpoint-ip xxx.xxx.xxx.xxx --kubernetes-version=v1.18.10+vmware.1
$ kubectl get nodes
NAME                            STATUS   ROLES    AGE    VERSION
devsecops-control-plane-kw5wx   Ready    master   2d6h   v1.18.10+vmware.1
devsecops-md-0-7f8445b7-zgklp   Ready    <none>   2d6h   v1.18.10+vmware.1
$ kubectl get sc
NAME                PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
default (default)   csi.vsphere.vmware.com   Delete          Immediate           false                  23h

TBS のインストール

インストール時にアクセスする必要があるレジストリにdocker login しておきます。
$ docker login harbor2.<MYDOMAIN> -u <harbor-user>
$ docker login registry.pivotal.io -u <tanzu-network-user>

TBS v1.1.1 をインストールしていきますので、必要なファイルをVMware Tanzu Network からダウンロードします。ダウンロードしたファイルを展開していきます。
$ mkdir v1.1.1
$ tar xvf build-service-1.1.1.tar -C v1.1.1
images.lock
manifests/
manifests/values.star
manifests/kpack/
manifests/kpack/kp-config.yaml
manifests/kpack/registry-serviceaccount.yaml
manifests/kpack/release.yaml
manifests/kpack/registry-secret.yaml
manifests/kpack/overlay.yaml
manifests/secret-syncer/
manifests/secret-syncer/deployment.yaml
manifests/secret-syncer/rbac.yaml
manifests/ca-cert.yaml
manifests/build-service/
manifests/build-service/authenticated-role.yaml
manifests/build-service/namespace.yaml
manifests/build-service/user-role.yaml
manifests/build-service/ccb-service-account.yaml
manifests/build-service/registry-secret.yaml
manifests/build-service/admin-role.yaml
manifests/stacks-operator/
manifests/stacks-operator/release.yaml
manifests/stacks-operator/registry-secret.yaml
manifests/stacks-operator/overlay.yaml
manifests/warmer/
manifests/warmer/deployment.yaml
manifests/warmer/daemonset.yaml
manifests/warmer/rbac.yaml
manifests/pod-webhook/
manifests/pod-webhook/deployment.yaml
manifests/pod-webhook/overlay.yaml
manifests/pod-webhook/configmaps.yaml
manifests/pod-webhook/rbac.yaml
values.yaml

Carvel ツールのytt, kbld, kapp を利用しますので、事前にインストールしておきます。
また、kp CLI も必要になるので、事前にインストールしておきます。
$ kbld version
kbld version 0.29.0

Succeeded
$ ytt version
ytt version 0.31.0
$ kapp version
kapp version 0.36.0

Succeeded
$ kp version
0.2.0-build.1 e40192c

イメージのRelocate

次のコマンドでTBS インストールに利用するイメージを、Harbor レジストリにリロケートします。
$ kbld relocate -f images.lock --lock-output images-relocated.lock --repository harbor2.<MYDOMAIN>/tanzu/tbs
relocate | exporting 15 images...
relocate | will export registry.pivotal.io/build-service/kpack-build-init-windows@sha256:8a6d67db07938d1f7ee6ac711a3a8b1fefe7997e96d4b1db53570441485bc9b2
relocate | will export registry.pivotal.io/build-service/kpack-build-init@sha256:ee73f0c697c2c0f71c263cd5254fd02309fe7a07a8141290c63e37c9b401f817
relocate | will export registry.pivotal.io/build-service/kpack-completion-windows@sha256:4faaa2a46eeff8f225382e21703b0afcc3910b8453e7aa30830f3b615a0f3929
relocate | will export registry.pivotal.io/build-service/kpack-completion@sha256:798623b1715f35be5c2bf7951735c871b0ac58aaa7dc1bdd63084df7f2856a2b
relocate | will export registry.pivotal.io/build-service/kpack-controller@sha256:101143de3d1abe2d2f738ce96b5991cd085ecc6f0e8dc761b940dd1cd346b864
relocate | will export registry.pivotal.io/build-service/kpack-lifecycle@sha256:fb7e0916ea429697630743b34e858c3555ddfbb5940683754dfccd3bfa446e0a
relocate | will export registry.pivotal.io/build-service/kpack-rebase@sha256:a8c378ca28941ba8274185556aa901f9c723cbdca2b169a45ab37a17525820f5
relocate | will export registry.pivotal.io/build-service/kpack-webhook@sha256:5d053b1e85a910312cba59d6fd2019921fcd29e2854262c21e9ed1c6a0f575ff
relocate | will export registry.pivotal.io/build-service/pod-webhook@sha256:d40667df0d832800d894d4f9c1f662f96805de5dd5f6951d99ac0b56d56db2da
relocate | will export registry.pivotal.io/build-service/secret-syncer@sha256:bfe60f3ed34da87d5b07aad763512069f3b37a800a3d0b64fc89cd6d304ed195
relocate | will export registry.pivotal.io/build-service/setup-ca-certs@sha256:de52c865c102bb8ba65c2378a45ee6c080e95d7d8765f3f52555e763c797d300
relocate | will export registry.pivotal.io/build-service/sleeper@sha256:9e4b3c352ef900dd543b2ef0c2fa16041c2e1207ed356b56c9ccb00d1244ad01
relocate | will export registry.pivotal.io/build-service/smart-warmer@sha256:c8d3acb20048c7b9a4843235f785f2047007fc023c1a131b8da35f0771d28237
relocate | will export registry.pivotal.io/build-service/stackify@sha256:1b4d890741cd70eae4c7bf8fd08998f9e13de108b22973f1deb7bed2e838549f
relocate | will export registry.pivotal.io/build-service/stacks-operator-controller@sha256:0be2f03da1876d37f364094b8ca68cd0b82f086aa0f3eb9fd4463e0fb74ca052
relocate | exported 15 images
relocate | importing 15 images...
relocate | importing registry.pivotal.io/build-service/kpack-lifecycle@sha256:fb7e0916ea429697630743b34e858c3555ddfbb5940683754dfccd3bfa446e0a -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:fb7e0916ea429697630743b34e858c3555ddfbb5940683754dfccd3bfa446e0a...
relocate | importing registry.pivotal.io/build-service/stacks-operator-controller@sha256:0be2f03da1876d37f364094b8ca68cd0b82f086aa0f3eb9fd4463e0fb74ca052 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:0be2f03da1876d37f364094b8ca68cd0b82f086aa0f3eb9fd4463e0fb74ca052...
relocate | importing registry.pivotal.io/build-service/setup-ca-certs@sha256:de52c865c102bb8ba65c2378a45ee6c080e95d7d8765f3f52555e763c797d300 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:de52c865c102bb8ba65c2378a45ee6c080e95d7d8765f3f52555e763c797d300...
relocate | importing registry.pivotal.io/build-service/secret-syncer@sha256:bfe60f3ed34da87d5b07aad763512069f3b37a800a3d0b64fc89cd6d304ed195 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:bfe60f3ed34da87d5b07aad763512069f3b37a800a3d0b64fc89cd6d304ed195...
relocate | importing registry.pivotal.io/build-service/kpack-build-init-windows@sha256:8a6d67db07938d1f7ee6ac711a3a8b1fefe7997e96d4b1db53570441485bc9b2 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:8a6d67db07938d1f7ee6ac711a3a8b1fefe7997e96d4b1db53570441485bc9b2...
relocate | importing registry.pivotal.io/build-service/smart-warmer@sha256:c8d3acb20048c7b9a4843235f785f2047007fc023c1a131b8da35f0771d28237 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:c8d3acb20048c7b9a4843235f785f2047007fc023c1a131b8da35f0771d28237...
relocate | importing registry.pivotal.io/build-service/kpack-webhook@sha256:5d053b1e85a910312cba59d6fd2019921fcd29e2854262c21e9ed1c6a0f575ff -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:5d053b1e85a910312cba59d6fd2019921fcd29e2854262c21e9ed1c6a0f575ff...
relocate | importing registry.pivotal.io/build-service/pod-webhook@sha256:d40667df0d832800d894d4f9c1f662f96805de5dd5f6951d99ac0b56d56db2da -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:d40667df0d832800d894d4f9c1f662f96805de5dd5f6951d99ac0b56d56db2da...
relocate | importing registry.pivotal.io/build-service/stackify@sha256:1b4d890741cd70eae4c7bf8fd08998f9e13de108b22973f1deb7bed2e838549f -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:1b4d890741cd70eae4c7bf8fd08998f9e13de108b22973f1deb7bed2e838549f...
relocate | importing registry.pivotal.io/build-service/sleeper@sha256:9e4b3c352ef900dd543b2ef0c2fa16041c2e1207ed356b56c9ccb00d1244ad01 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:9e4b3c352ef900dd543b2ef0c2fa16041c2e1207ed356b56c9ccb00d1244ad01...
relocate | importing registry.pivotal.io/build-service/kpack-rebase@sha256:a8c378ca28941ba8274185556aa901f9c723cbdca2b169a45ab37a17525820f5 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:a8c378ca28941ba8274185556aa901f9c723cbdca2b169a45ab37a17525820f5...
relocate | importing registry.pivotal.io/build-service/kpack-build-init@sha256:ee73f0c697c2c0f71c263cd5254fd02309fe7a07a8141290c63e37c9b401f817 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:ee73f0c697c2c0f71c263cd5254fd02309fe7a07a8141290c63e37c9b401f817...
relocate | importing registry.pivotal.io/build-service/kpack-completion-windows@sha256:4faaa2a46eeff8f225382e21703b0afcc3910b8453e7aa30830f3b615a0f3929 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:4faaa2a46eeff8f225382e21703b0afcc3910b8453e7aa30830f3b615a0f3929...
relocate | importing registry.pivotal.io/build-service/kpack-controller@sha256:101143de3d1abe2d2f738ce96b5991cd085ecc6f0e8dc761b940dd1cd346b864 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:101143de3d1abe2d2f738ce96b5991cd085ecc6f0e8dc761b940dd1cd346b864...
relocate | importing registry.pivotal.io/build-service/kpack-completion@sha256:798623b1715f35be5c2bf7951735c871b0ac58aaa7dc1bdd63084df7f2856a2b -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:798623b1715f35be5c2bf7951735c871b0ac58aaa7dc1bdd63084df7f2856a2b...
relocate | imported 15 images
Succeeded

TKGm Workload Cluster にTBS インストール

リロケートしたイメージを利用し、作成したTKGm Workload Cluster 上にTBS をインストールします。
$ ytt -f values.yaml -f manifests/ -v docker_repository=harbor2.<MYDOMAIN>/tanzu/tbs -v docker_username=<harbor-user> -v docker_password=<harbor-password> | kbld -f images-relocated.lock -f- |kapp deploy -a tbs -f- -y
Target cluster 'https://xxx.xxx.xxx.xxx:6443' (nodes: devsecops-control-plane-kw5wx, 1+)
resolve | final: build-init -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:ee73f0c697c2c0f71c263cd5254fd02309fe7a07a8141290c63e37c9b401f817
resolve | final: completion -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:798623b1715f35be5c2bf7951735c871b0ac58aaa7dc1bdd63084df7f2856a2b
resolve | final: dev.registry.pivotal.io/build-service/pod-webhook@sha256:d40667df0d832800d894d4f9c1f662f96805de5dd5f6951d99ac0b56d56db2da -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:d40667df0d832800d894d4f9c1f662f96805de5dd5f6951d99ac0b56d56db2da
resolve | final: dev.registry.pivotal.io/build-service/setup-ca-certs@sha256:de52c865c102bb8ba65c2378a45ee6c080e95d7d8765f3f52555e763c797d300 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:de52c865c102bb8ba65c2378a45ee6c080e95d7d8765f3f52555e763c797d300
resolve | final: dev.registry.pivotal.io/core-deps/stackify@sha256:b69bcea0e1e1053a7ea89933fd2f26ed4b7d9b44da0425c171ce858b43a8c74a -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:1b4d890741cd70eae4c7bf8fd08998f9e13de108b22973f1deb7bed2e838549f
resolve | final: dev.registry.pivotal.io/core-deps/stacks-operator@sha256:2b6b3170a2e62358cbf300252222217974998c2338ec769836b4d44d7581b3d3 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:0be2f03da1876d37f364094b8ca68cd0b82f086aa0f3eb9fd4463e0fb74ca052
resolve | final: gcr.io/cf-build-service-public/kpack/build-init-windows@sha256:8a6d67db07938d1f7ee6ac711a3a8b1fefe7997e96d4b1db53570441485bc9b2 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:8a6d67db07938d1f7ee6ac711a3a8b1fefe7997e96d4b1db53570441485bc9b2
resolve | final: gcr.io/cf-build-service-public/kpack/build-init@sha256:ee73f0c697c2c0f71c263cd5254fd02309fe7a07a8141290c63e37c9b401f817 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:ee73f0c697c2c0f71c263cd5254fd02309fe7a07a8141290c63e37c9b401f817
resolve | final: gcr.io/cf-build-service-public/kpack/completion-windows@sha256:4faaa2a46eeff8f225382e21703b0afcc3910b8453e7aa30830f3b615a0f3929 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:4faaa2a46eeff8f225382e21703b0afcc3910b8453e7aa30830f3b615a0f3929
resolve | final: gcr.io/cf-build-service-public/kpack/completion@sha256:798623b1715f35be5c2bf7951735c871b0ac58aaa7dc1bdd63084df7f2856a2b -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:798623b1715f35be5c2bf7951735c871b0ac58aaa7dc1bdd63084df7f2856a2b
resolve | final: gcr.io/cf-build-service-public/kpack/controller@sha256:101143de3d1abe2d2f738ce96b5991cd085ecc6f0e8dc761b940dd1cd346b864 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:101143de3d1abe2d2f738ce96b5991cd085ecc6f0e8dc761b940dd1cd346b864
resolve | final: gcr.io/cf-build-service-public/kpack/lifecycle@sha256:fb7e0916ea429697630743b34e858c3555ddfbb5940683754dfccd3bfa446e0a -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:fb7e0916ea429697630743b34e858c3555ddfbb5940683754dfccd3bfa446e0a
resolve | final: gcr.io/cf-build-service-public/kpack/rebase@sha256:a8c378ca28941ba8274185556aa901f9c723cbdca2b169a45ab37a17525820f5 -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:a8c378ca28941ba8274185556aa901f9c723cbdca2b169a45ab37a17525820f5
resolve | final: gcr.io/cf-build-service-public/kpack/webhook@sha256:5d053b1e85a910312cba59d6fd2019921fcd29e2854262c21e9ed1c6a0f575ff -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:5d053b1e85a910312cba59d6fd2019921fcd29e2854262c21e9ed1c6a0f575ff
resolve | final: rebase -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:a8c378ca28941ba8274185556aa901f9c723cbdca2b169a45ab37a17525820f5
resolve | final: secret-syncer -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:bfe60f3ed34da87d5b07aad763512069f3b37a800a3d0b64fc89cd6d304ed195
resolve | final: setup-ca-certs -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:de52c865c102bb8ba65c2378a45ee6c080e95d7d8765f3f52555e763c797d300
resolve | final: sleeper -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:9e4b3c352ef900dd543b2ef0c2fa16041c2e1207ed356b56c9ccb00d1244ad01
resolve | final: stackify -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:1b4d890741cd70eae4c7bf8fd08998f9e13de108b22973f1deb7bed2e838549f
resolve | final: warmer -> harbor2.<MYDOMAIN>/tanzu/tbs@sha256:c8d3acb20048c7b9a4843235f785f2047007fc023c1a131b8da35f0771d28237

Changes

Namespace               Name                                                            Kind                            Conds.  Age  Op      Op st.  Wait to    Rs  Ri
(cluster)               build-service                                                   Namespace                       -       -    create  -       reconcile  -   -
^                       build-service-admin-role                                        ClusterRole                     -       -    create  -       reconcile  -   -
^                       build-service-admin-role-binding                                ClusterRoleBinding              -       -    create  -       reconcile  -   -
^                       build-service-authenticated-role                                ClusterRole                     -       -    create  -       reconcile  -   -
^                       build-service-authenticated-role-binding                        ClusterRoleBinding              -       -    create  -       reconcile  -   -
^                       build-service-secret-syncer-role                                ClusterRole                     -       -    create  -       reconcile  -   -
^                       build-service-secret-syncer-role-binding                        ClusterRoleBinding              -       -    create  -       reconcile  -   -
^                       build-service-user-role                                         ClusterRole                     -       -    create  -       reconcile  -   -
^                       build-service-warmer-role                                       ClusterRole                     -       -    create  -       reconcile  -   -
^                       build-service-warmer-role-binding                               ClusterRoleBinding              -       -    create  -       reconcile  -   -
^                       builders.kpack.io                                               CustomResourceDefinition        -       -    create  -       reconcile  -   -
^                       builds.kpack.io                                                 CustomResourceDefinition        -       -    create  -       reconcile  -   -
^                       cert-injection-webhook-cluster-role                             ClusterRole                     -       -    create  -       reconcile  -   -
^                       cert-injection-webhook-cluster-role-binding                     ClusterRoleBinding              -       -    create  -       reconcile  -   -
^                       clusterbuilders.kpack.io                                        CustomResourceDefinition        -       -    create  -       reconcile  -   -
^                       clusterstacks.kpack.io                                          CustomResourceDefinition        -       -    create  -       reconcile  -   -
^                       clusterstores.kpack.io                                          CustomResourceDefinition        -       -    create  -       reconcile  -   -
^                       custom-stack-editor-role                                        ClusterRole                     -       -    create  -       reconcile  -   -
^                       custom-stack-viewer-role                                        ClusterRole                     -       -    create  -       reconcile  -   -
^                       customstacks.stacks.stacks-operator.tanzu.vmware.com            CustomResourceDefinition        -       -    create  -       reconcile  -   -
^                       defaults.webhook.cert-injection.tanzu.vmware.com                MutatingWebhookConfiguration    -       -    create  -       reconcile  -   -
^                       defaults.webhook.kpack.io                                       MutatingWebhookConfiguration    -       -    create  -       reconcile  -   -
^                       images.kpack.io                                                 CustomResourceDefinition        -       -    create  -       reconcile  -   -
^                       kpack                                                           Namespace                       -       -    create  -       reconcile  -   -
^                       kpack-controller-admin                                          ClusterRole                     -       -    create  -       reconcile  -   -
^                       kpack-controller-admin-binding                                  ClusterRoleBinding              -       -    create  -       reconcile  -   -
^                       kpack-webhook-certs-mutatingwebhookconfiguration-admin-binding  ClusterRoleBinding              -       -    create  -       reconcile  -   -
^                       kpack-webhook-mutatingwebhookconfiguration-admin                ClusterRole                     -       -    create  -       reconcile  -   -
^                       metrics-reader                                                  ClusterRole                     -       -    create  -       reconcile  -   -
^                       proxy-role                                                      ClusterRole                     -       -    create  -       reconcile  -   -
^                       proxy-rolebinding                                               ClusterRoleBinding              -       -    create  -       reconcile  -   -
^                       sourceresolvers.kpack.io                                        CustomResourceDefinition        -       -    create  -       reconcile  -   -
^                       stacks-operator-manager-role                                    ClusterRole                     -       -    create  -       reconcile  -   -
^                       stacks-operator-manager-rolebinding                             ClusterRoleBinding              -       -    create  -       reconcile  -   -
^                       stacks-operator-system                                          Namespace                       -       -    create  -       reconcile  -   -
^                       validation.webhook.kpack.io                                     ValidatingWebhookConfiguration  -       -    create  -       reconcile  -   -
build-service           build-pod-image-fetcher                                         DaemonSet                       -       -    create  -       reconcile  -   -
^                       build-service-warmer-namespace-role                             Role                            -       -    create  -       reconcile  -   -
^                       build-service-warmer-namespace-role-binding                     RoleBinding                     -       -    create  -       reconcile  -   -
^                       ca-cert                                                         ConfigMap                       -       -    create  -       reconcile  -   -
^                       canonical-registry-secret                                       Secret                          -       -    create  -       reconcile  -   -
^                       cb-service-account                                              ServiceAccount                  -       -    create  -       reconcile  -   -
^                       cert-injection-webhook                                          Deployment                      -       -    create  -       reconcile  -   -
^                       cert-injection-webhook                                          Service                         -       -    create  -       reconcile  -   -
^                       cert-injection-webhook-role                                     Role                            -       -    create  -       reconcile  -   -
^                       cert-injection-webhook-role-binding                             RoleBinding                     -       -    create  -       reconcile  -   -
^                       cert-injection-webhook-sa                                       ServiceAccount                  -       -    create  -       reconcile  -   -
^                       cert-injection-webhook-tls                                      Secret                          -       -    create  -       reconcile  -   -
^                       http-proxy                                                      ConfigMap                       -       -    create  -       reconcile  -   -
^                       https-proxy                                                     ConfigMap                       -       -    create  -       reconcile  -   -
^                       no-proxy                                                        ConfigMap                       -       -    create  -       reconcile  -   -
^                       secret-syncer-controller                                        Deployment                      -       -    create  -       reconcile  -   -
^                       secret-syncer-service-account                                   ServiceAccount                  -       -    create  -       reconcile  -   -
^                       setup-ca-certs-image                                            ConfigMap                       -       -    create  -       reconcile  -   -
^                       sleeper-image                                                   ConfigMap                       -       -    create  -       reconcile  -   -
^                       warmer-controller                                               Deployment                      -       -    create  -       reconcile  -   -
^                       warmer-service-account                                          ServiceAccount                  -       -    create  -       reconcile  -   -
kpack                   build-init-image                                                ConfigMap                       -       -    create  -       reconcile  -   -
^                       build-init-windows-image                                        ConfigMap                       -       -    create  -       reconcile  -   -
^                       canonical-registry-secret                                       Secret                          -       -    create  -       reconcile  -   -
^                       canonical-registry-serviceaccount                               ServiceAccount                  -       -    create  -       reconcile  -   -
^                       completion-image                                                ConfigMap                       -       -    create  -       reconcile  -   -
^                       completion-windows-image                                        ConfigMap                       -       -    create  -       reconcile  -   -
^                       controller                                                      ServiceAccount                  -       -    create  -       reconcile  -   -
^                       kp-config                                                       ConfigMap                       -       -    create  -       reconcile  -   -
^                       kpack-controller                                                Deployment                      -       -    create  -       reconcile  -   -
^                       kpack-controller-local-config                                   Role                            -       -    create  -       reconcile  -   -
^                       kpack-controller-local-config-binding                           RoleBinding                     -       -    create  -       reconcile  -   -
^                       kpack-webhook                                                   Deployment                      -       -    create  -       reconcile  -   -
^                       kpack-webhook                                                   Service                         -       -    create  -       reconcile  -   -
^                       kpack-webhook-certs-admin                                       Role                            -       -    create  -       reconcile  -   -
^                       kpack-webhook-certs-admin-binding                               RoleBinding                     -       -    create  -       reconcile  -   -
^                       lifecycle-image                                                 ConfigMap                       -       -    create  -       reconcile  -   -
^                       rebase-image                                                    ConfigMap                       -       -    create  -       reconcile  -   -
^                       webhook                                                         ServiceAccount                  -       -    create  -       reconcile  -   -
^                       webhook-certs                                                   Secret                          -       -    create  -       reconcile  -   -
stacks-operator-system  canonical-registry-secret                                       Secret                          -       -    create  -       reconcile  -   -
^                       controller-manager                                              Deployment                      -       -    create  -       reconcile  -   -
^                       controller-manager-metrics-service                              Service                         -       -    create  -       reconcile  -   -
^                       leader-election-role                                            Role                            -       -    create  -       reconcile  -   -
^                       leader-election-rolebinding                                     RoleBinding                     -       -    create  -       reconcile  -   -
^                       stackify-image                                                  ConfigMap                       -       -    create  -       reconcile  -   -

Op:      82 create, 0 delete, 0 update, 0 noop
Wait to: 82 reconcile, 0 delete, 0 noop

4:41:29AM: ---- applying 36 changes [0/82 done] ----
4:41:29AM: create clusterrolebinding/build-service-warmer-role-binding (rbac.authorization.k8s.io/v1) cluster
4:41:29AM: create clusterrole/build-service-admin-role (rbac.authorization.k8s.io/v1) cluster
4:41:29AM: create clusterrolebinding/build-service-admin-role-binding (rbac.authorization.k8s.io/v1) cluster
4:41:29AM: create clusterrole/build-service-authenticated-role (rbac.authorization.k8s.io/v1) cluster
4:41:29AM: create namespace/build-service (v1) cluster
4:41:29AM: create clusterrolebinding/build-service-authenticated-role-binding (rbac.authorization.k8s.io/v1) cluster
4:41:29AM: create clusterrole/build-service-user-role (rbac.authorization.k8s.io/v1) cluster
4:41:29AM: create namespace/kpack (v1) cluster
4:41:29AM: create customresourcedefinition/clusterbuilders.kpack.io (apiextensions.k8s.io/v1beta1) cluster
4:41:29AM: create clusterrole/build-service-secret-syncer-role (rbac.authorization.k8s.io/v1) cluster
4:41:29AM: create clusterrolebinding/kpack-controller-admin-binding (rbac.authorization.k8s.io/v1) cluster
4:41:29AM: create clusterrole/kpack-controller-admin (rbac.authorization.k8s.io/v1) cluster
4:41:30AM: create customresourcedefinition/clusterstores.kpack.io (apiextensions.k8s.io/v1beta1) cluster
4:41:30AM: create customresourcedefinition/builds.kpack.io (apiextensions.k8s.io/v1beta1) cluster
4:41:30AM: create customresourcedefinition/builders.kpack.io (apiextensions.k8s.io/v1beta1) cluster
4:41:30AM: create mutatingwebhookconfiguration/defaults.webhook.kpack.io (admissionregistration.k8s.io/v1beta1) cluster
4:41:30AM: create validatingwebhookconfiguration/validation.webhook.kpack.io (admissionregistration.k8s.io/v1beta1) cluster
4:41:30AM: create customresourcedefinition/images.kpack.io (apiextensions.k8s.io/v1beta1) cluster
4:41:30AM: create clusterrole/kpack-webhook-mutatingwebhookconfiguration-admin (rbac.authorization.k8s.io/v1) cluster
4:41:30AM: create customresourcedefinition/sourceresolvers.kpack.io (apiextensions.k8s.io/v1beta1) cluster
4:41:30AM: create clusterrolebinding/kpack-webhook-certs-mutatingwebhookconfiguration-admin-binding (rbac.authorization.k8s.io/v1) cluster
4:41:30AM: create mutatingwebhookconfiguration/defaults.webhook.cert-injection.tanzu.vmware.com (admissionregistration.k8s.io/v1beta1) cluster
4:41:30AM: create clusterrole/cert-injection-webhook-cluster-role (rbac.authorization.k8s.io/v1) cluster
4:41:30AM: create clusterrole/custom-stack-editor-role (rbac.authorization.k8s.io/v1) cluster
4:41:30AM: create clusterrolebinding/cert-injection-webhook-cluster-role-binding (rbac.authorization.k8s.io/v1) cluster
4:41:30AM: create namespace/stacks-operator-system (v1) cluster
4:41:30AM: create clusterrole/proxy-role (rbac.authorization.k8s.io/v1) cluster
4:41:30AM: create clusterrole/metrics-reader (rbac.authorization.k8s.io/v1beta1) cluster
4:41:30AM: create clusterrolebinding/proxy-rolebinding (rbac.authorization.k8s.io/v1) cluster
4:41:30AM: create clusterrolebinding/stacks-operator-manager-rolebinding (rbac.authorization.k8s.io/v1) cluster
4:41:30AM: create clusterrole/custom-stack-viewer-role (rbac.authorization.k8s.io/v1) cluster
4:41:30AM: create clusterrole/stacks-operator-manager-role (rbac.authorization.k8s.io/v1) cluster
4:41:30AM: create clusterrole/build-service-warmer-role (rbac.authorization.k8s.io/v1) cluster
4:41:30AM: create clusterrolebinding/build-service-secret-syncer-role-binding (rbac.authorization.k8s.io/v1) cluster
4:41:31AM: create customresourcedefinition/clusterstacks.kpack.io (apiextensions.k8s.io/v1beta1) cluster
4:41:31AM: create customresourcedefinition/customstacks.stacks.stacks-operator.tanzu.vmware.com (apiextensions.k8s.io/v1beta1) cluster
...(SNIP)...
4:41:38AM: ---- waiting on 6 changes [76/82 done] ----
4:41:40AM: ok: reconcile deployment/cert-injection-webhook (apps/v1) namespace: build-service
4:41:40AM: ok: reconcile deployment/kpack-controller (apps/v1) namespace: kpack
4:41:40AM: ok: reconcile deployment/warmer-controller (apps/v1) namespace: build-service
4:41:40AM: ---- waiting on 3 changes [79/82 done] ----
4:41:41AM: ok: reconcile deployment/secret-syncer-controller (apps/v1) namespace: build-service
4:41:41AM: ---- waiting on 2 changes [80/82 done] ----
4:41:43AM: ok: reconcile deployment/controller-manager (apps/v1) namespace: stacks-operator-system
4:41:43AM: ---- waiting on 1 changes [81/82 done] ----
4:42:00AM: ok: reconcile daemonset/build-pod-image-fetcher (apps/v1) namespace: build-service
4:42:00AM: ---- applying complete [82/82 done] ----
4:42:00AM: ---- waiting complete [82/82 done] ----

Succeeded

インストール出来た事を確認します。
$ kapp list
Target cluster 'https://xxx.xxx.xxx.xxx:6443' (nodes: devsecops-control-plane-kw5wx, 1+)

Apps in namespace 'default'

Name  Namespaces                      Lcs   Lca
tbs   (cluster),build-service,kpack,  true  3m
      stacks-operator-system

Lcs: Last Change Successful
Lca: Last Change Age

1 apps

Succeeded

TBS で利用するClusterBuilder のインストール

この状態ではTBS で利用するClusterBuilder(Stack(TBSで利用するベースOSイメージ)+Store(Buildpacks の集合体)の組み合わせ)が未だインストールされていないため、TBS を利用したコンテナイメージ作成が出来ません。
ClusterBuilder をインストールする際には、VMware Tanzu Network のTBS Dependencies にあるdescriptor-xxxx.yaml を利用します。

Stack や Store のバージョンアップもこの後試したいので、少し古めのdescriptor-100.0.60.yaml を利用します。
$ cat descriptor-100.0.60.yaml
apiVersion: kp.kpack.io/v1alpha1
kind: DependencyDescriptor
defaultClusterBuilder: base
defaultStack: base
stores:
- name: default
  sources:
  - image: registry.pivotal.io/tanzu-go-buildpack/go@sha256:64ba98089642bdb1ae22fde9b12fb905e0034a2e99c4564249bbc5190dcef8f9
  - image: registry.pivotal.io/tanzu-java-buildpack/java@sha256:9c03a98642c39daad05a7803ee00ee6543aa0561cb18da169ac2d7f24a73b792
  - image: registry.pivotal.io/tanzu-nodejs-buildpack/nodejs@sha256:19bdfcee5d5cedeb5ad9fd704d52b21be286e5ce3a5c3b8cabc30204f490f8d4
  - image: registry.pivotal.io/tanzu-java-native-image-buildpack/java-native-image@sha256:6aa4a1904362624ee5937271e05fdeaa4669ac4fed808346327927c3090e280e
  - image: registry.pivotal.io/tbs-dependencies/tanzu-buildpacks_dotnet-core@sha256:303675080e1220af51fa08c7d20ba87fb7f4c662ceb756c6333fe01dec61adc1
  - image: registry.pivotal.io/tbs-dependencies/tanzu-buildpacks_php@sha256:fd5abb334f4adbcf46f42977992af145db04fb262d4c516ff4264f94e1fcd689
  - image: registry.pivotal.io/tbs-dependencies/tanzu-buildpacks_nginx@sha256:e67d5cd2e5240a9eb7a899b9b5d979ad85d0cf6c8182e15424516dac9f577371
  - image: registry.pivotal.io/tbs-dependencies/tanzu-buildpacks_httpd@sha256:34989fb8e264ccaea7916a9017b306d621b017920f71439fc515164ac0484cf5
  - image: registry.pivotal.io/tbs-dependencies/paketo-buildpacks_procfile@sha256:861d9c93924dd1db3fbe6663754da603398931cdc4dee3da5c057ffa65cb383d
stacks:
- name: tiny
  buildImage:
    image: registry.pivotal.io/tbs-dependencies/build-tiny@sha256:0e17e2dcbf4c4eb1fe15a113619a53847dc3ebf9276d6535d2abb9c9e5e60493
  runImage:
    image: registry.pivotal.io/tbs-dependencies/run-tiny@sha256:1b36a50f20b3ba648d427e6dcd813c35fee99034f479c503ad104b58f2de3c2d
- name: base
  buildImage:
    image: registry.pivotal.io/tbs-dependencies/build-base@sha256:f2b6d8e5f674a687131468640c50e3405ab6bb66ab820a0264635a0a1a35265d
  runImage:
    image: registry.pivotal.io/tbs-dependencies/run-base@sha256:64b97816ff8e96bfacd804bb994f76c93d210cee8f726ce7bdad4a1a4e858e7f
- name: full
  buildImage:
    image: registry.pivotal.io/tbs-dependencies/build-full@sha256:6f8c43753d908ca737f1620fb4a7a63bae392a4674f5ddfc9ac362b9cd6b4fab
  runImage:
    image: registry.pivotal.io/tbs-dependencies/run-full@sha256:26cf2ed7fbb994d6347de901fdddc12decfe37d3f9f38c7b3404c46533a90e32
clusterBuilders:
- name: base
  stack: base
  store: default
  order:
  - group:
    - id: tanzu-buildpacks/dotnet-core
  - group:
    - id: tanzu-buildpacks/nodejs
  - group:
    - id: tanzu-buildpacks/go
  - group:
    - id: tanzu-buildpacks/php
  - group:
    - id: tanzu-buildpacks/nginx
  - group:
    - id: tanzu-buildpacks/httpd
  - group:
    - id: tanzu-buildpacks/java-native-image
  - group:
    - id: tanzu-buildpacks/java
  - group:
    - id: paketo-buildpacks/procfile
- name: full
  stack: full
  store: default
  order:
  - group:
    - id: tanzu-buildpacks/dotnet-core
  - group:
    - id: tanzu-buildpacks/nodejs
  - group:
    - id: tanzu-buildpacks/go
  - group:
    - id: tanzu-buildpacks/php
  - group:
    - id: tanzu-buildpacks/nginx
  - group:
    - id: tanzu-buildpacks/httpd
  - group:
    - id: tanzu-buildpacks/java-native-image
  - group:
    - id: tanzu-buildpacks/java
  - group:
    - id: paketo-buildpacks/procfile
- name: tiny
  stack: tiny
  store: default
  order:
  - group:
    - id: tanzu-buildpacks/go
  - group:
    - id: tanzu-buildpacks/java-native-image
  - group:
    - id: paketo-buildpacks/procfile

kp コマンドでClusterBuilder をインストールします。
$ kp import -f descriptor-100.0.60.yaml
Importing ClusterStore 'default'...
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/tanzu-buildpacks_go@sha256:64ba98089642bdb1ae22fde9b12fb905e0034a2e99c4564249bbc5190dcef8f9'
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/tanzu-buildpacks_java@sha256:9c03a98642c39daad05a7803ee00ee6543aa0561cb18da169ac2d7f24a73b792'
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/tanzu-buildpacks_nodejs@sha256:19bdfcee5d5cedeb5ad9fd704d52b21be286e5ce3a5c3b8cabc30204f490f8d4'
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/tanzu-buildpacks_java-native-image@sha256:6aa4a1904362624ee5937271e05fdeaa4669ac4fed808346327927c3090e280e'
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/tanzu-buildpacks_dotnet-core@sha256:303675080e1220af51fa08c7d20ba87fb7f4c662ceb756c6333fe01dec61adc1'
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/tanzu-buildpacks_php@sha256:fd5abb334f4adbcf46f42977992af145db04fb262d4c516ff4264f94e1fcd689'
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/tanzu-buildpacks_nginx@sha256:e67d5cd2e5240a9eb7a899b9b5d979ad85d0cf6c8182e15424516dac9f577371'
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/tanzu-buildpacks_httpd@sha256:34989fb8e264ccaea7916a9017b306d621b017920f71439fc515164ac0484cf5'
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/paketo-buildpacks_procfile@sha256:861d9c93924dd1db3fbe6663754da603398931cdc4dee3da5c057ffa65cb383d'
Importing ClusterStack 'tiny'...
Uploading to 'harbor2.<MYDOMAIN>/tanzu/tbs'...
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/build@sha256:0e17e2dcbf4c4eb1fe15a113619a53847dc3ebf9276d6535d2abb9c9e5e60493'
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/run@sha256:1b36a50f20b3ba648d427e6dcd813c35fee99034f479c503ad104b58f2de3c2d'
Importing ClusterStack 'base'...
Uploading to 'harbor2.<MYDOMAIN>/tanzu/tbs'...
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/build@sha256:f2b6d8e5f674a687131468640c50e3405ab6bb66ab820a0264635a0a1a35265d'
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/run@sha256:64b97816ff8e96bfacd804bb994f76c93d210cee8f726ce7bdad4a1a4e858e7f'
Importing ClusterStack 'full'...
Uploading to 'harbor2.<MYDOMAIN>/tanzu/tbs'...
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/build@sha256:6f8c43753d908ca737f1620fb4a7a63bae392a4674f5ddfc9ac362b9cd6b4fab'
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/run@sha256:26cf2ed7fbb994d6347de901fdddc12decfe37d3f9f38c7b3404c46533a90e32'
Importing ClusterStack 'default'...
Uploading to 'harbor2.<MYDOMAIN>/tanzu/tbs'...
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/build@sha256:f2b6d8e5f674a687131468640c50e3405ab6bb66ab820a0264635a0a1a35265d'
	Uploading 'harbor2.<MYDOMAIN>/tanzu/tbs/run@sha256:64b97816ff8e96bfacd804bb994f76c93d210cee8f726ce7bdad4a1a4e858e7f'
Importing ClusterBuilder 'base'...
Importing ClusterBuilder 'full'...
Importing ClusterBuilder 'tiny'...
Importing ClusterBuilder 'default'...
Imported resources
$ kp clusterbuilder list
NAME       READY    STACK                          IMAGE
base       true     io.buildpacks.stacks.bionic    harbor2.<MYDOMAIN>/tanzu/tbs/base@sha256:e5bceac410f63800c0b56eb58b0e9db689165321cc78d8b856a12133da5095fe
default    true     io.buildpacks.stacks.bionic    harbor2.<MYDOMAIN>/tanzu/tbs/default@sha256:e5bceac410f63800c0b56eb58b0e9db689165321cc78d8b856a12133da5095fe
full       true     io.buildpacks.stacks.bionic    harbor2.<MYDOMAIN>/tanzu/tbs/full@sha256:be7705ece27d17f02921d5a575d2ca7ba9d4b2aba0f744ab9751fdff9afd8201
tiny       true     io.paketo.stacks.tiny          harbor2.<MYDOMAIN>/tanzu/tbs/tiny@sha256:daf377c30aaff0a8b0458be97a4c889c899d75f3e7e4cbda97a10bcf34b766a7

テスト

TBS を利用して、コンテナイメージを作成してみたいと思います。
まずは、Harbor レジストリにアクセスするためのsecret を作成します。
$ kubectl create ns demo
namespace/demo created
$ kp secret create lab-harbor2 --registry harbor2.<MYDOMAIN> --registry-user <harbor-user> -n demo
registry password:
Secret "lab-harbor2" created

コンテナイメージビルドのテストには、spring-projects/spring-petclinic を利用したいと思いますので、事前にgit clone でコードをローカル環境にダウンロードしておきます。また、この記事ではビルド済のローカルJAR ファイルを対象にしますので、事前にビルドしておきます。
~/spring-petclinic$ ./mvnw spring-boot:build-image -Dmaven.test.skip=true

ビルドが完了したら、kp CLI を利用し、TBS を用いたイメージを作成します。
$ kp image create spring-petclinic --tag harbor2.<MYDOMAIN>/tanzu/spring-petclinic --local-path spring-petclinic/target/spring-petclinic-2.3.0.BUILD-SNAPSHOT.jar -n demo --wait
Creating Image...
Uploading to 'harbor2.<MYDOMAIN>/tanzu/spring-petclinic-source'...
	Uploading 'harbor2.<MYDOMAIN>/tanzu/spring-petclinic-source@sha256:2743063f15ed963aa5dd47df7795f5b9d16c952f08fc2b23b0eb1ad8cbef2964'
Image "spring-petclinic" created
===> PREPARE
Build reason(s): CONFIG
CONFIG:
	resources: {}
	- source: {}
	+ source:
	+   registry:
	+     image: harbor2.<MYDOMAIN>/tanzu/spring-petclinic-source@sha256:2743063f15ed963aa5dd47df7795f5b9d16c952f08fc2b23b0eb1ad8cbef2964
Loading secret for "harbor2.<MYDOMAIN>" from secret "lab-harbor2" at location "/var/build-secrets/lab-harbor2"
Pulling harbor2.<MYDOMAIN>/tanzu/spring-petclinic-source@sha256:2743063f15ed963aa5dd47df7795f5b9d16c952f08fc2b23b0eb1ad8cbef2964...
Successfully pulled harbor2.<MYDOMAIN>/tanzu/spring-petclinic-source@sha256:2743063f15ed963aa5dd47df7795f5b9d16c952f08fc2b23b0eb1ad8cbef2964 in path "/workspace"
...

これだと上手く行かず、TBS でのコンテナイメージ作成が失敗します。
$ kp image list -n demo
NAME                READY      LATEST REASON    LATEST IMAGE    NAMESPACE
spring-petclinic    Unknown    CONFIG                           demo


TBS のリソースを確認してみます。
$ kubectl get all -n build-service
NAME                                            READY   STATUS             RESTARTS   AGE
pod/build-pod-image-fetcher-g2khm               5/5     Running            0          48m
pod/cert-injection-webhook-5f6d8bf4bf-nmgq4     1/1     Running            0          48m
pod/secret-syncer-controller-5bbdfbc694-hlcdq   1/1     Running            0          48m
pod/smart-warmer-image-fetcher-n4jnc            1/4     ImagePullBackOff   0          29m
pod/warmer-controller-67696b9d6c-f4psz          1/1     Running            0          48m

NAME                             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/cert-injection-webhook   ClusterIP   100.70.46.6   <none>        443/TCP   48m

NAME                                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/build-pod-image-fetcher      1         1         1       1            1           kubernetes.io/os=linux   48m
daemonset.apps/smart-warmer-image-fetcher   1         1         0       1            0           kubernetes.io/os=linux   29m

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cert-injection-webhook     1/1     1            1           48m
deployment.apps/secret-syncer-controller   1/1     1            1           48m
deployment.apps/warmer-controller          1/1     1            1           48m

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/cert-injection-webhook-5f6d8bf4bf     1         1         1       48m
replicaset.apps/secret-syncer-controller-5bbdfbc694   1         1         1       48m
replicaset.apps/warmer-controller-67696b9d6c          1         1         1       48m
$ kubectl describe pod/smart-warmer-image-fetcher-n4jnc -n build-service
Name:         smart-warmer-image-fetcher-n4jnc
Namespace:    build-service
Priority:     0
Node:         devsecops-md-0-7f8445b7-zgklp/192.168.11.55
Start Time:   Sun, 14 Mar 2021 05:01:20 +0000
Labels:       app=smart-warmer-image-fetcher
              controller-revision-hash=9756d8cb9
              pod-template-generation=4
...(SNIP)...
  Warning  Failed     31m                   kubelet            Failed to pull image "harbor2.<MYDOMAIN>/tanzu/tbs/base@sha256:e5bceac410f63800c0b56eb58b0e9db689165321cc78d8b856a12133da5095fe": rpc error: code = InvalidArgument desc = failed to pull and unpack image "harbor2.<MYDOMAIN>/tanzu/tbs/base@sha256:e5bceac410f63800c0b56eb58b0e9db689165321cc78d8b856a12133da5095fe": failed to prepare extraction snapshot "extract-631548843-h5lL sha256:66438f4383deee3ee0eed5548c27bd996bb70916716eb0149e9580cbe89ceba4": info.Labels: label key and value greater than maximum size (4096 bytes), key: containerd: invalid argument
  Warning  Failed     31m                   kubelet            Failed to pull image "harbor2.<MYDOMAIN>/tanzu/tbs/default@sha256:e5bceac410f63800c0b56eb58b0e9db689165321cc78d8b856a12133da5095fe": rpc error: code = InvalidArgument desc = failed to pull and unpack image "harbor2.<MYDOMAIN>/tanzu/tbs/default@sha256:e5bceac410f63800c0b56eb58b0e9db689165321cc78d8b856a12133da5095fe": failed to prepare extraction snapshot "extract-673558922-xPzS sha256:66438f4383deee3ee0eed5548c27bd996bb70916716eb0149e9580cbe89ceba4": info.Labels: label key and value greater than maximum size (4096 bytes), key: containerd: invalid argument
  Normal   BackOff    12m (x86 over 31m)    kubelet            Back-off pulling image "harbor2.<MYDOMAIN>/tanzu/tbs/base@sha256:e5bceac410f63800c0b56eb58b0e9db689165321cc78d8b856a12133da5095fe"
  Warning  Failed     2m4s (x127 over 31m)  kubelet            Error: ImagePullBackOff

<info.Labels: label key and value greater than maximum size (4096 bytes), key: containerd: invalid argument>というエラーメッセージが出力されています。
既に、BLOG.IK.AM さんが書いている「Tanzu Kubernetes Grid 1.2.1 (K8s 1.19.3, Containerd 1.4.1)でコンテナイメージのlabelサイズが4KB以上の場合にpullできない問題のworkaroundメモ」通りの事象に当たっています。Workload Cluster は作成済なので、Control Plane / Worker ノードそれぞれにssh ログインし、containerd の設定を変えていきます。
$ ssh capv@<control-plane-node-ip or worker-node-ip> -i <ssh-private-key-for-tkgm-cluster>
$ sudo su -
# echo '  [plugins."io.containerd.grpc.v1.cri".containerd]' >> /etc/containerd/config.toml
# echo '    disable_snapshot_annotations = true' >> /etc/containerd/config.toml
# systemctl restart containerd
# systemctl status containerd

これが終わったら、改めてkp CLI を実行し、TBS を利用してコンテナイメージを作成します。
$ kp image delete spring-petclinic -n demo
Image "spring-petclinic" deleted
$ kp image create spring-petclinic --tag harbor2.<MYDOMAIN>/tanzu/spring-petclinic --local-path ~/spring-petclinic/target/spring-petclinic-2.3.0.BUILD-SNAPSHOT.jar -n demo --wait
Creating Image...
Uploading to 'harbor2.<MYDOMAIN>/tanzu/spring-petclinic-source'...
	Uploading 'harbor2.<MYDOMAIN>/tanzu/spring-petclinic-source@sha256:2743063f15ed963aa5dd47df7795f5b9d16c952f08fc2b23b0eb1ad8cbef2964'
Image "spring-petclinic" created
===> PREPARE
Build reason(s): CONFIG
CONFIG:
	resources: {}
	- source: {}
	+ source:
	+   registry:
	+     image: harbor2.<MYDOMAIN>/tanzu/spring-petclinic-source@sha256:2743063f15ed963aa5dd47df7795f5b9d16c952f08fc2b23b0eb1ad8cbef2964
Loading secret for "harbor2.<MYDOMAIN>" from secret "lab-harbor2" at location "/var/build-secrets/lab-harbor2"
Pulling harbor2.<MYDOMAIN>/tanzu/spring-petclinic-source@sha256:2743063f15ed963aa5dd47df7795f5b9d16c952f08fc2b23b0eb1ad8cbef2964...
Successfully pulled harbor2.<MYDOMAIN>/tanzu/spring-petclinic-source@sha256:2743063f15ed963aa5dd47df7795f5b9d16c952f08fc2b23b0eb1ad8cbef2964 in path "/workspace"
===> DETECT
5 of 33 buildpacks participating
paketo-buildpacks/ca-certificates   1.0.1
paketo-buildpacks/bellsoft-liberica 6.0.0
paketo-buildpacks/executable-jar    3.1.3
paketo-buildpacks/dist-zip          2.2.2
paketo-buildpacks/spring-boot       3.5.0
===> ANALYZE
Previous image with name "harbor2.<MYDOMAIN>/tanzu/spring-petclinic" not found
===> RESTORE
===> BUILD

Paketo CA Certificates Buildpack 1.0.1
  https://github.com/paketo-buildpacks/ca-certificates
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
    Writing profile.d/helper

Paketo BellSoft Liberica Buildpack 6.0.0
  https://github.com/paketo-buildpacks/bellsoft-liberica
  Build Configuration:
    $BP_JVM_VERSION              11.*            the Java version
  Launch Configuration:
    $BPL_JVM_HEAD_ROOM           0               the headroom in memory calculation
    $BPL_JVM_LOADED_CLASS_COUNT  35% of classes  the number of loaded classes in memory calculation
    $BPL_JVM_THREAD_COUNT        250             the number of threads in memory calculation
    $JAVA_TOOL_OPTIONS                           the JVM launch flags
  BellSoft Liberica JRE 11.0.9: Contributing to layer
    Reusing cached download from buildpack
    Expanding to /layers/paketo-buildpacks_bellsoft-liberica/jre
    Adding 138 container CA certificates to JVM truststore
    Writing env.launch/BPI_APPLICATION_PATH.default
    Writing env.launch/BPI_JVM_CACERTS.default
    Writing env.launch/BPI_JVM_CLASS_COUNT.default
    Writing env.launch/BPI_JVM_SECURITY_PROVIDERS.default
    Writing env.launch/JAVA_HOME.default
    Writing env.launch/MALLOC_ARENA_MAX.default
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/active-processor-count
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/java-opts
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/link-local-dns
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/memory-calculator
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/openssl-certificate-loader
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/security-providers-configurer
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/security-providers-classpath-9
    Writing profile.d/helper
  JVMKill Agent 1.16.0: Contributing to layer
    Reusing cached download from buildpack
    Copying to /layers/paketo-buildpacks_bellsoft-liberica/jvmkill
    Writing env.launch/JAVA_TOOL_OPTIONS.append
    Writing env.launch/JAVA_TOOL_OPTIONS.delim
  Java Security Properties: Contributing to layer
    Writing env.launch/JAVA_SECURITY_PROPERTIES.default
    Writing env.launch/JAVA_TOOL_OPTIONS.append
    Writing env.launch/JAVA_TOOL_OPTIONS.delim

Paketo Executable JAR Buildpack 3.1.3
  https://github.com/paketo-buildpacks/executable-jar
    Writing env.launch/CLASSPATH.delim
    Writing env.launch/CLASSPATH.prepend
  Process types:
    executable-jar: java org.springframework.boot.loader.JarLauncher
    task:           java org.springframework.boot.loader.JarLauncher
    web:            java org.springframework.boot.loader.JarLauncher

Paketo Spring Boot Buildpack 3.5.0
  https://github.com/paketo-buildpacks/spring-boot
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_spring-boot/helper/exec.d/spring-cloud-bindings
    Writing profile.d/helper
  Web Application Type: Contributing to layer
    Servlet web application detected
    Writing env.launch/BPL_JVM_THREAD_COUNT.default
  Spring Cloud Bindings 1.7.0: Contributing to layer
    Reusing cached download from buildpack
    Copying to /layers/paketo-buildpacks_spring-boot/spring-cloud-bindings
  Image labels:
    org.opencontainers.image.title
    org.opencontainers.image.version
    org.springframework.boot.spring-configuration-metadata.json
    org.springframework.boot.version
===> EXPORT
Adding layer 'paketo-buildpacks/ca-certificates:helper'
Adding layer 'paketo-buildpacks/bellsoft-liberica:helper'
Adding layer 'paketo-buildpacks/bellsoft-liberica:java-security-properties'
Adding layer 'paketo-buildpacks/bellsoft-liberica:jre'
Adding layer 'paketo-buildpacks/bellsoft-liberica:jvmkill'
Adding layer 'paketo-buildpacks/executable-jar:class-path'
Adding layer 'paketo-buildpacks/spring-boot:helper'
Adding layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
Adding layer 'paketo-buildpacks/spring-boot:web-application-type'
Adding 1/1 app layer(s)
Adding layer 'launcher'
Adding layer 'config'
Adding layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Adding label 'org.opencontainers.image.title'
Adding label 'org.opencontainers.image.version'
Adding label 'org.springframework.boot.spring-configuration-metadata.json'
Adding label 'org.springframework.boot.version'
Setting default process type 'web'
*** Images (sha256:78f1a6620889e880037118cf03ca626b271b4f02d433deaae4d2c2cbadbb27ba):
      harbor2.<MYDOMAIN>/tanzu/spring-petclinic
      harbor2.<MYDOMAIN>/tanzu/spring-petclinic:b1.20210314.053352
===> COMPLETION
Build successful

無事、Harbor レジストリにコンテナイメージがストアされた事を確認出来ました。





まとめ

TKGm + TBS + Harbor の組み合わせで、コンテナイメージを作成し、保管する事が出来ました。この環境を利用し、TBS を利用したメモを書いていきたいと思います。

このブログの人気の投稿