[abctl] Error pod airbyte-abctl-airbyte-bootloader failed

[marcosmarxm]

Let's use this thread to discuss users facing this error :)

[bambangraharjo]

I'm facing this warning when installing using abctl, t3.2xlarge ec2 instance, 30GB Disk.
Installation never finish if not interrupted by me.
Installing 'airbyte/airbyte' (version: 0.524.0) Helm Chart (this may take several minutes) (12m53s)
Warning message:

 WARNING  Encountered an issue deploying Airbyte:                                                                                                                                                           
            Pod: airbyte-abctl-airbyte-bootloader.17f6f0d8905554e4
            Reason: Failed
            Message: Error: ImagePullBackOff
            Count: 6

[marcosmarxm]

Sorry delay @bambangraharjo, are you using any custom values or default installation?

[bambangraharjo]

Hi @marcosmarxm, there is a possibility because of the pull image limitation from Docker.
https://www.docker.com/increase-rate-limits/
Since I'm using anonymous usage for pulling docker images.
I'm not facing this issue again.

[bambangraharjo]

FYI, I'm using custom values when facing this issue.
But I don't think custom values are the root cause, since now it's working with the same custom values.

[marcosmarxm]

?So the solution was to login in docker hub to be able to retrieve the images

[bambangraharjo]

I'm not logged in to the docker hub, I tried the next day and this issue didn't occur again.
This is my command installation when this issue occurred.
abctl local install --values values.yaml --secret secrets.yaml --low-resource-mode --chart-version 0.524.0

[icgoco]

I'm the same...and I can't find a solution. A big disaster this update.

[marcosmarxm]

Can you share logs or steps you're doing @icgoco ?

[Coinio]

I'm getting this error. If I use the --verbose flag I get some more detail. I'm using Postgres 15.8. I get this error regardless of if I try to migrate the previous install or just do abctl local install.

  DEBUG   airbyte-bootloader: 2024-11-14 11:50:03,621 [main]    ERROR   i.a.b.Application(main):25 - Unable to bootstrap Airbyte environment.
  DEBUG   airbyte-bootloader: org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException: Migration failed !
  DEBUG   airbyte-bootloader: Caused by: org.flywaydb.core.api.FlywayException: Migration failed !
  DEBUG   airbyte-bootloader: Caused by: org.jooq.exception.DataAccessException: SQL [select id from connection where non_breaking_change_preference = cast(? as "non_breaking_change_preference_type")]; ERROR: operator does not exist: character varying = non_breaking_change_preference_type
  DEBUG   airbyte-bootloader:   Hint: No operator matches the given name and argument types. You might need to add explicit type casts.
  DEBUG   airbyte-bootloader:   Position: 64
  DEBUG   airbyte-bootloader: Caused by: org.postgresql.util.PSQLException: ERROR: operator does not exist: character varying = non_breaking_change_preference_type
  DEBUG   airbyte-bootloader:   Hint: No operator matches the given name and argument types. You might need to add explicit type casts.
  DEBUG   airbyte-bootloader:   Position: 64
  DEBUG   airbyte-bootloader:
  DEBUG   airbyte-bootloader: 2024-11-14 11:50:03,625 [Thread-18]       INFO    i.m.r.Micronaut(lambda$start$0):118 - Embedded Application shutting down
  DEBUG   helm: Add/Modify event for airbyte-abctl-airbyte-bootloader: MODIFIED
  DEBUG   helm: Pod airbyte-abctl-airbyte-bootloader running
  DEBUG   done streaming bootloader logs
  DEBUG   helm: Add/Modify event for airbyte-abctl-airbyte-bootloader: MODIFIED
  DEBUG   helm: warning: Upgrade "airbyte-abctl" failed: pre-upgrade hooks failed: 1 error occurred:
                * pod airbyte-abctl-airbyte-bootloader failed
  DEBUG   helm: release upgrade failed: pre-upgrade hooks failed: 1 error occurred:
                * pod airbyte-abctl-airbyte-bootloader failed
  ERROR   Failed to install airbyte/airbyte Helm Chart

[Coinio]

I managed to get around this if anybody else is having a similar issue.

For us it seemed to be related to a database collation mismatch (recent Postgres version upgrade), but even after I fixed all of the collation warnings on our Postgres database this issue was still present when using the abctl too. The only thing that fixed it was following the solution in the post below.

#36855 (comment)

[ramonvermeulen]

I get this error when running abctl local install --verbose on a MacBook pro m4:

  INFO    Namespace 'airbyte-abctl' already exists
  INFO    Persistent volume 'airbyte-minio-pv' already exists
  INFO    Persistent volume 'airbyte-volume-db' already exists
  INFO    Persistent volume claim 'airbyte-minio-pv-claim-airbyte-minio-0' already exists
  INFO    Persistent volume claim 'airbyte-volume-db-airbyte-db-0' already exists
  DEBUG   helm: WARNING: repository name "airbyte" already exists
  INFO    Starting Helm Chart installation of 'airbyte/airbyte' (version: 1.3.1)
  DEBUG   helm: preparing upgrade for airbyte-abctl
  DEBUG   helm: performing update for airbyte-abctl
  DEBUG   helm: creating upgraded release for airbyte-abctl
  DEBUG   helm: Starting delete for "airbyte-abctl-airbyte-env" ConfigMap
  DEBUG   helm: beginning wait for 1 resources to be deleted with timeout of 1h0m0s
  DEBUG   helm: creating 1 resource(s)
  DEBUG   helm: Starting delete for "airbyte-abctl-airbyte-secrets" Secret
  DEBUG   helm: beginning wait for 1 resources to be deleted with timeout of 1h0m0s
  DEBUG   helm: creating 1 resource(s)
  DEBUG   helm: Starting delete for "airbyte-abctl-airbyte-bootloader" Pod
  DEBUG   helm: beginning wait for 1 resources to be deleted with timeout of 1h0m0s
  DEBUG   helm: creating 1 resource(s)
  DEBUG   helm: Watching for changes to Pod airbyte-abctl-airbyte-bootloader with timeout of 1h0m0s
  DEBUG   Successfully assigned airbyte-abctl/airbyte-abctl-airbyte-bootloader to airbyte-abctl-control-plane
  DEBUG   helm: Add/Modify event for airbyte-abctl-airbyte-bootloader: ADDED
  DEBUG   helm: Pod airbyte-abctl-airbyte-bootloader pending
  DEBUG   helm: Add/Modify event for airbyte-abctl-airbyte-bootloader: MODIFIED
  DEBUG   helm: Pod airbyte-abctl-airbyte-bootloader pending
  DEBUG   helm: Add/Modify event for airbyte-abctl-airbyte-bootloader: MODIFIED
  DEBUG   helm: Pod airbyte-abctl-airbyte-bootloader pending
  DEBUG   Container image "airbyte/bootloader:1.3.1" already present on machine
  DEBUG   Created container airbyte-bootloader-container
  DEBUG   start streaming bootloader logs
  DEBUG   helm: Add/Modify event for airbyte-abctl-airbyte-bootloader: MODIFIED
  DEBUG   helm: Pod airbyte-abctl-airbyte-bootloader pending
  DEBUG   helm: Add/Modify event for airbyte-abctl-airbyte-bootloader: MODIFIED
  DEBUG   helm: warning: Upgrade "airbyte-abctl" failed: pre-upgrade hooks failed: 1 error occurred:
                * pod airbyte-abctl-airbyte-bootloader failed
  DEBUG   helm: release upgrade failed: pre-upgrade hooks failed: 1 error occurred:
                * pod airbyte-abctl-airbyte-bootloader failed
  ERROR   Failed to install airbyte/airbyte Helm Chart
  DEBUG   looking at airbyte-abctl-airbyte-bootloader
            Failed()
  DEBUG   found logs:  Unsetting empty environment variable 'AB_INSTANCE_
  DEBUG   looking at airbyte-db-0
            Running()
  DEBUG   found logs:  The files belonging to this database system will b
  DEBUG   looking at airbyte-minio-0
            Running()
  DEBUG   found logs:  Formatting 1st pool, 1 set(s), 1 drives per set.
          W
  ERROR   Unable to install Airbyte locally
  DEBUG   Event watcher completed.
  ERROR   unable to install airbyte chart: bootloader failed

  INFO    The bootloader failed to its initialization checks or migrations. Try running again with --verbose to see the full bootloader logs.

Any idea what can be causing this?

[ykeremy]

Getting the same error, clean installation on M4 pro