Could you let me know if offline deployment is possible?

[vingin007]

Topic

No response

Relevant information

Could you please tell me if this project has an offline deployment solution? My work environment is on an internal network without internet access. How can I install this project?

[szemek]

As far as I know, there is no simple offline deployment out of the box, but it should possible with extra work to change existing abctl or helm charts deployments into offline one.
Have you considered PyAirbyte https://docs.airbyte.com/using-airbyte/pyairbyte/getting-started ?

[brandybobandyistaken]

This is a shame since I was previously able to deploy on a server with no internet access using docker compose. All I had to do is scp the images needed.

[brandybobandyistaken]

I've tried bringing over the abctl executable as well as the images it seems to use, but it still tries to pull the images despite them being loaded into Docker.

[szemek]

@vingin007 @brandybobandyistaken abctl uses kind cluster (https://kind.sigs.k8s.io) under the hood.
It is possible to load cached Docker images into kind cluster. Please notice that for images from hub.docker.com you need to add docker.io/ prefix.

kind load docker-image docker.io/airbyte/server:0.63.15 --name airbyte-abctl
kind load docker-image registry.k8s.io/coredns/coredns:v1.11.1 --name airbyte-abctl

https://kind.sigs.k8s.io/docs/user/quick-start/#loading-an-image-into-your-cluster

With some customizations you should be able to use abctl with helm charts from local path instead of helm registry.
I noticed on my machine that in ~/.airbyte directory there is abctl subdirectory with .helmcache and .helmrepo, so maybe it will be even easier 🧐

.airbyte
└── abctl
    ├── .helmcache
    │   ├── airbyte-0.551.0.tgz
    │   ├── airbyte-0.594.0.tgz
    │   ├── airbyte-charts.txt
    │   ├── airbyte-index.yaml
    │   ├── ingress-nginx-4.11.2.tgz
    │   ├── nginx-charts.txt
    │   └── nginx-index.yaml
    ├── .helmrepo
    ├── abctl.kubeconfig
    └── data
        ├── airbyte-minio-pv
        └── airbyte-volume-db

[stephanbertl]

Same issues here. We are using nexus proxy for literally everything. No direct access to pypi, docker,...

Our k8s cluster works fine and we can pull images through the proxy.

Airbyte however makes requests to pypi.org and connectors.airbyte.com and that obviously fails.

So it seems it cant be used offline, because no pypi mirror or proxy can be configured for those URLs?