ERROR 'get build args: ConfigMap "yatai-image-builder-config" not found' when publishing BentoRequest

[IooHooI]

I have installed Yatai components following these instructions:

• https://docs.yatai.io/en/latest/installation/yatai.html
• https://docs.yatai.io/en/latest/installation/yatai_image_builder.html
• https://docs.yatai.io/en/latest/installation/yatai_deployment.html

I have logged in with bentoml cloud afterwards:

bentoml cloud login --api-token [API TOKEN] --endpoint http://localhost:52348

Then I've created a Bentoml model and a Bento:

And after that I've tried to publish a new BentoRequest to build an OCI image for this Bento:

apiVersion: resources.yatai.ai/v1alpha1
kind: BentoRequest
metadata:
    name: summarization
    namespace: yatai
spec:
    bentoTag: summarization:v0.0.2  # check the tag by `bentoml list iris_classifier`

This manifest was taken from here:
https://github.com/bentoml/yatai?tab=readme-ov-file#-deploy-bento

I used only BentoRequest to see if yatai-image-builder will be able to build a docker image for the Bento.

When I applied this manifest and looked at yatai-image-builder-xxxxxxx Pod logs, I saw this:

2025-02-22T12:54:02Z	DEBUG	events	Making sure docker config secret yatai-regcred in namespace yatai	{"type": "Normal", "object": {"kind":"BentoRequest","namespace":"yatai","name":"summarization","uid":"96b3f2bf-9f1c-4689-88a0-41f952deaf72","apiVersion":"resources.yatai.ai/v1alpha1","resourceVersion":"8796521"}, "reason": "GenerateImageBuilderPod"}
2025-02-22T12:54:02Z	DEBUG	events	Docker config secret yatai-regcred in namespace yatai is ready	{"type": "Normal", "object": {"kind":"BentoRequest","namespace":"yatai","name":"summarization","uid":"96b3f2bf-9f1c-4689-88a0-41f952deaf72","apiVersion":"resources.yatai.ai/v1alpha1","resourceVersion":"8796521"}, "reason": "GenerateImageBuilderPod"}
2025-02-22T12:54:02Z	DEBUG	events	Checking image exists: cr.selcloud.ru/yatai-bentos/yatai-bentos-111111:yatai.summarization.v0.0.2	{"type": "Normal", "object": {"kind":"BentoRequest","namespace":"yatai","name":"summarization","uid":"96b3f2bf-9f1c-4689-88a0-41f952deaf72","apiVersion":"resources.yatai.ai/v1alpha1","resourceVersion":"8796523"}, "reason": "CheckingImage"}
2025-02-22T12:54:02Z	DEBUG	events	Image not exists: cr.selcloud.ru/yatai-bentos/yatai-bentos-111111:yatai.summarization.v0.0.2	{"type": "Normal", "object": {"kind":"BentoRequest","namespace":"yatai","name":"summarization","uid":"96b3f2bf-9f1c-4689-88a0-41f952deaf72","apiVersion":"resources.yatai.ai/v1alpha1","resourceVersion":"8796523"}, "reason": "CheckingImage"}
2025-02-22T12:54:02Z	DEBUG	events	Getting bento summarization:v0.0.2 from yatai service	{"type": "Normal", "object": {"kind":"BentoRequest","namespace":"yatai","name":"summarization","uid":"96b3f2bf-9f1c-4689-88a0-41f952deaf72","apiVersion":"resources.yatai.ai/v1alpha1","resourceVersion":"8796527"}, "reason": "GenerateImageBuilderPod"}
2025-02-22T12:54:02Z	DEBUG	events	Got bento summarization:v0.0.2 from yatai service	{"type": "Normal", "object": {"kind":"BentoRequest","namespace":"yatai","name":"summarization","uid":"96b3f2bf-9f1c-4689-88a0-41f952deaf72","apiVersion":"resources.yatai.ai/v1alpha1","resourceVersion":"8796527"}, "reason": "GenerateImageBuilderPod"}
2025-02-22T12:54:02Z	DEBUG	events	Getting secret yatai-api-token in namespace yatai	{"type": "Normal", "object": {"kind":"BentoRequest","namespace":"yatai","name":"summarization","uid":"96b3f2bf-9f1c-4689-88a0-41f952deaf72","apiVersion":"resources.yatai.ai/v1alpha1","resourceVersion":"8796527"}, "reason": "GenerateImageBuilderPod"}
2025-02-22T12:54:02Z	DEBUG	events	Secret yatai-api-token is found in namespace yatai, so updating it	{"type": "Normal", "object": {"kind":"BentoRequest","namespace":"yatai","name":"summarization","uid":"96b3f2bf-9f1c-4689-88a0-41f952deaf72","apiVersion":"resources.yatai.ai/v1alpha1","resourceVersion":"8796527"}, "reason": "GenerateImageBuilderPod"}
time="2025-02-22T12:54:02Z" level=info msg="Image builder is using the images {quay.io/bentoml/bento-downloader:0.0.5 quay.io/bentoml/curl:0.0.1 quay.io/bentoml/kaniko:debug quay.io/bentoml/yatai-bento-metrics-transformer:0.0.3 quay.io/bentoml/buildkit:master quay.io/bentoml/buildkit:master-rootless quay.io/bentoml/bentoml-buildah:0.0.1}"
2025-02-22T12:54:02Z	DEBUG	events	Secret yatai-api-token is updated in namespace yatai	{"type": "Normal", "object": {"kind":"BentoRequest","namespace":"yatai","name":"summarization","uid":"96b3f2bf-9f1c-4689-88a0-41f952deaf72","apiVersion":"resources.yatai.ai/v1alpha1","resourceVersion":"8796527"}, "reason": "GenerateImageBuilderPod"}
2025-02-22T12:54:02Z	DEBUG	events	Getting model summarization-model:0.0.2 from yatai service	{"type": "Normal", "object": {"kind":"BentoRequest","namespace":"yatai","name":"summarization","uid":"96b3f2bf-9f1c-4689-88a0-41f952deaf72","apiVersion":"resources.yatai.ai/v1alpha1","resourceVersion":"8796527"}, "reason": "GenerateImageBuilderPod"}
2025-02-22T12:54:03Z	DEBUG	events	Model summarization-model:0.0.2 is got from yatai service	{"type": "Normal", "object": {"kind":"BentoRequest","namespace":"yatai","name":"summarization","uid":"96b3f2bf-9f1c-4689-88a0-41f952deaf72","apiVersion":"resources.yatai.ai/v1alpha1","resourceVersion":"8796527"}, "reason": "GenerateImageBuilderPod"}
2025-02-22T12:54:03Z	DEBUG	events	Getting configmap yatai-image-builder-config from namespace yatai-image-builder	{"type": "Normal", "object": {"kind":"BentoRequest","namespace":"yatai","name":"summarization","uid":"96b3f2bf-9f1c-4689-88a0-41f952deaf72","apiVersion":"resources.yatai.ai/v1alpha1","resourceVersion":"8796527"}, "reason": "GenerateImageBuilderPod"}
2025-02-22T12:54:03Z	DEBUG	events	Configmap yatai-image-builder-config is not found in namespace yatai-image-builder	{"type": "Normal", "object": {"kind":"BentoRequest","namespace":"yatai","name":"summarization","uid":"96b3f2bf-9f1c-4689-88a0-41f952deaf72","apiVersion":"resources.yatai.ai/v1alpha1","resourceVersion":"8796527"}, "reason": "GenerateImageBuilderPod"}
2025-02-22T12:54:03Z	ERROR	Failed to reconcile BentoRequest.	{"controller": "bentorequest", "controllerGroup": "resources.yatai.ai", "controllerKind": "BentoRequest", "BentoRequest": {"name":"summarization","namespace":"yatai"}, "namespace": "yatai", "name": "summarization", "reconcileID": "60b0018f-e22b-4d3c-91c2-ab65323faccc", "bentoRequest": "summarization", "bentoRequestNamespace": "yatai", "error": "ensure image exists: generate image builder job: generate image builder pod template spec: get build args: ConfigMap \"yatai-image-builder-config\" not found", "errorVerbose": "ConfigMap \"yatai-image-builder-config\" not found\nget build args\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).generateImageBuilderPodTemplateSpec\n\t/workspace/controllers/resources/bentorequest_controller.go:2854\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).generateImageBuilderJob\n\t/workspace/controllers/resources/bentorequest_controller.go:2253\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).ensureImageExists\n\t/workspace/controllers/resources/bentorequest_controller.go:585\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).Reconcile\n\t/workspace/controllers/resources/bentorequest_controller.go:234\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\ngenerate image builder pod template spec\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).generateImageBuilderJob\n\t/workspace/controllers/resources/bentorequest_controller.go:2258\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).ensureImageExists\n\t/workspace/controllers/resources/bentorequest_controller.go:585\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).Reconcile\n\t/workspace/controllers/resources/bentorequest_controller.go:234\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\ngenerate image builder job\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).ensureImageExists\n\t/workspace/controllers/resources/bentorequest_controller.go:590\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).Reconcile\n\t/workspace/controllers/resources/bentorequest_controller.go:234\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\nensure image exists\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).Reconcile\n\t/workspace/controllers/resources/bentorequest_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
github.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).Reconcile.func1
	/workspace/controllers/resources/bentorequest_controller.go:190
github.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).Reconcile
	/workspace/controllers/resources/bentorequest_controller.go:241
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224
2025-02-22T12:54:03Z	DEBUG	events	Failed to reconcile BentoRequest: ensure image exists: generate image builder job: generate image builder pod template spec: get build args: ConfigMap "yatai-image-builder-config" not found	{"type": "Warning", "object": {"kind":"BentoRequest","namespace":"yatai","name":"summarization","uid":"96b3f2bf-9f1c-4689-88a0-41f952deaf72","apiVersion":"resources.yatai.ai/v1alpha1","resourceVersion":"8796527"}, "reason": "ReconcileError"}
2025-02-22T12:54:03Z	ERROR	Reconciler error	{"controller": "bentorequest", "controllerGroup": "resources.yatai.ai", "controllerKind": "BentoRequest", "BentoRequest": {"name":"summarization","namespace":"yatai"}, "namespace": "yatai", "name": "summarization", "reconcileID": "60b0018f-e22b-4d3c-91c2-ab65323faccc", "error": "ensure image exists: generate image builder job: generate image builder pod template spec: get build args: ConfigMap \"yatai-image-builder-config\" not found", "errorVerbose": "ConfigMap \"yatai-image-builder-config\" not found\nget build args\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).generateImageBuilderPodTemplateSpec\n\t/workspace/controllers/resources/bentorequest_controller.go:2854\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).generateImageBuilderJob\n\t/workspace/controllers/resources/bentorequest_controller.go:2253\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).ensureImageExists\n\t/workspace/controllers/resources/bentorequest_controller.go:585\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).Reconcile\n\t/workspace/controllers/resources/bentorequest_controller.go:234\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\ngenerate image builder pod template spec\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).generateImageBuilderJob\n\t/workspace/controllers/resources/bentorequest_controller.go:2258\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).ensureImageExists\n\t/workspace/controllers/resources/bentorequest_controller.go:585\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).Reconcile\n\t/workspace/controllers/resources/bentorequest_controller.go:234\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\ngenerate image builder job\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).ensureImageExists\n\t/workspace/controllers/resources/bentorequest_controller.go:590\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).Reconcile\n\t/workspace/controllers/resources/bentorequest_controller.go:234\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\nensure image exists\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).Reconcile\n\t/workspace/controllers/resources/bentorequest_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224

So the last DEBUG log was Configmap yatai-image-builder-config is not found in namespace yatai-image-builder.

I have looked at Helm Charts for all three components and did not find neither yatai-image-builder-config Configmap nor any Configmap at all.

I've looked at Reconciler source code and only found that Configmap with name yatai-image-builder-config is only being requested from yatai-image-builder namespace:

...
const (
	// nolint: gosec
	YataiImageBuilderAWSAccessKeySecretName = "yatai-image-builder-aws-access-key"
	// nolint: gosec
	YataiImageBuilderGCPAccessKeySecretName   = "yatai-image-builder-gcp-access-key"
	KubeAnnotationBentoRequestHash            = "yatai.ai/bento-request-hash"
	KubeAnnotationBentoRequestImageBuiderHash = "yatai.ai/bento-request-image-builder-hash"
	KubeAnnotationBentoRequestModelSeederHash = "yatai.ai/bento-request-model-seeder-hash"
	KubeLabelYataiImageBuilderSeparateModels  = "yatai.ai/yatai-image-builder-separate-models"
	KubeAnnotationBentoStorageNS              = "yatai.ai/bento-storage-namepsace"
	KubeAnnotationModelStorageNS              = "yatai.ai/model-storage-namepsace"
	KubeAnnotationImageInfo                   = "yatai.ai/image-info"
	StoreSchemaAWS                            = "aws"
	StoreSchemaGCP                            = "gcp"

	configCmName = "yatai-image-builder-config"
)
...

r.Recorder.Eventf(opt.BentoRequest, corev1.EventTypeNormal, "GenerateImageBuilderPod", "Configmap %s is not found in namespace %s", configCmName, configNamespace)

...

I have not found any place where this ConfigMap is created.

So, question is, cloud You please explain to me:

• where exactly this ConfigMap should have been created?
• what component should have created this ConfigMap?
• what exactly this ConfigMap should contain?

Thanks in advance!