Large Schema Discovery for Connection. "Failed to find output files from connector within timeout of 9 minute(s). Is the connector still running? Failure origin: airbyte_platform"

[genesisdayrit]

I'm getting the following error after ~9 minutes of hanging on the fetching schema / schema discovery screen for a connection between a SQL Server (MSSQL) database as the source and Snowflake as the destination.

Error:

Discovering schema failed
Workload failed, source: unknown

Learn more
Internal message: Failed to find output files from connector within timeout of 9 minute(s). Is the connector still running?
Failure origin: airbyte_platform

Setup details:

• The schema is about 19k+ tables from an ERP system Business Central
• Both MSSQL as source and Snowflake as destination are successfully connecting on setup tests
• We are using a EC2 c5.xlarge w/ Amazon Linux and airbyte installed using abctl local install
• When running top, I'm not seeing the CPU usage % is strained and found it to be <10% usage during most periods while checking during the schema discovery process.

---

I've tried re-installing airbyte using abctl local install--values values.yaml to try to update the possible different timeout variables, with the following values.yaml variables being tested after suggestion of variables to test for increasing the timeout time using the Airbyte Community slack bot (thread) for help

global:
  env_vars:
    HTTP_IDLE_TIMEOUT: 3600s
    MICRONAUT_SERVER_MAX_REQUEST_SIZE: "104857600" # 100MB

webapp:
  ingress:
    annotations:
      nginx.ingress.kubernetes.io/proxy-send-timeout: 3600
      nginx.ingress.kubernetes.io/proxy-read-timeout: 3600

server:
  extraEnvs:
    - name: HTTP_IDLE_TIMEOUT
      value: 3600s
    - name: READ_TIMEOUT
      value: 3600s
    - name: WORKLOAD_API_READ_TIMEOUT_SECONDS
      value: "3600"
    - name: HTTP_IDLE_TIMEOUT
      value: 60m
    - name: READ_TIMEOUT
      value: 60m

worker:
  env:
    - name: DISCOVER_JOB_TIMEOUT_MINUTES
      value: "60"

This was the original error that I was getting prior to making the variable changes to values.yaml above

---

Without making any known changes to settings or values.yaml, I've seen a couple subsequent runs with the following new error after a long idling time on the fetch screen, but then returning to the original "9 minute timeout error" on next try.

I wasn't able to find a way to get the logs during the schema discovery process or if this is possible.

Anyone have ideas on the issue here or how to resolve?