Airbyte API: Request for Input on Configuration API deprecation

[nataliekwong]

We want to better understand how our users rely on our APIs to operate Airbyte and ensure we provide a great experience in using Airbyte, and are looking for your feedback to improve the experience.

A brief history:

• We released the Airbyte API in April 2023. The Airbyte API is a reliable, easy-to-use, and versioned interface for programmatically controlling the Airbyte platform with full support from the Airbyte team. The SDKs and Terraform provider all support the Airbyte API, and will not support the config API.
• We released the Config API in the early days of Airbyte and stopped supporting it in April 2023 in favor of the Airbyte API.

While the config API is a convenient way to interact with Airbyte, it is not designed to be a public-facing API and will be formally deprecated this year. The correct API to use with Airbyte is the Airbyte API. Our goal is to ensure that you have the ability to manage and run core Airbyte functionality without resorting to functionality only available in the config API.

Our ask to you:

What endpoints or data are you missing from the Airbyte API? Share with us (using the template below) what you are currently using the config API for today that you think we should support so that we can better understand your use case and build a roadmap for the API.

Examples

Please follow this template in your response:

Config API endpoint: /v1/source_definition/list
Endpoint exist in the Airbyte API? No
Use Case: I need to list all available source definitions because my app will display the connector to the final user

Config API endpoint: /v1/sources/list
Endpoint exist in the Airbyte API? Yes
Use Case: The current Airbyte API doesn't bring me all the information I need to ….

For now, we are only looking for input on usage of the Configuration API and will keep this open until July 2024. In the future, we’ll open a separate discussion on expanding API endpoints. We look forward to hearing from you!

[rensoostenbachBL]

Config API endpoint: v1/connector_builder_projects/list
Endpoint exist in the Airbyte API? No
Use Case: I need to list all connector builder projects because I am using the Configuration API to deploy custom connectors.

[rensoostenbachBL]

Config API endpoint: v1/connector_builder_projects/create
Endpoint exist in the Airbyte API? No
Use Case: I need to create a connector builder project because I am using the Configuration API to deploy custom connectors.

[rensoostenbachBL]

Config API endpoint: v1/connector_builder_projects/publish
Endpoint exist in the Airbyte API? No
Use Case: I need to publish a connector builder project because I am using the Configuration API to deploy custom connectors.

[rensoostenbachBL]

Config API endpoint: v1/declarative_source_definitions/list_manifests
Endpoint exist in the Airbyte API? No
Use Case: I need to list all declaraative source definitions because I am using the Configuration API to deploy custom connectors.

[rensoostenbachBL]

Config API endpoint: v1/declarative_source_definitions/update_active_manifest
Endpoint exist in the Airbyte API? No
Use Case: I need to update an active manifest because I am using the Configuration API to deploy custom connectors.

[rensoostenbachBL]

Config API endpoint: v1/declarative_source_definitions/create_manifest
Endpoint exist in the Airbyte API? No
Use Case: I need to create a manifest because I am using the Configuration API to deploy custom connectors.

[rensoostenbachBL]

Config API endpoint: /v1/sources/discover_schema
Endpoint exist in the Airbyte API? No
Use Case: I need to create connections that require me to discover the schema.

[nataliekwong]

Hi there! Can you accomplish this using https://reference.airbyte.com/reference/getstreamproperties? If not, can you share more about why this wouldn't work for what you're looking to do?

[rensoostenbachBL]

Hi, I'm not exactly sure if getstreamproperties would work, since it is not totally clear to me what its output is.

This is how we are currently using discover_schema:

def create_connection(
    self, source_name: str, destination_name: str, stream_names: List[str] = None
):
    """Create a connection if it doesn't already exist."""
    if not self._check_connection_exists(source_name, destination_name):
        source_id = self._get_source_id_by_name(source_name)
        destination_id = self._get_destination_id_by_name(destination_name)
        # Discover source schemas
        sources_schema_resp = self._api_call(
            "sources/discover_schema",
            "POST",
            data={"sourceId": source_id},
            config_api=True,
        )
        streams = sources_schema_resp["catalog"]["streams"]
        catalog_id = sources_schema_resp.get("catalogId")

        # Enable streams
        for stream in streams:
            if stream_names is None:
                stream["config"]["selected"] = True
            else:
                stream["config"]["selected"] = (
                    stream["stream"]["name"] in stream_names
                )

        # Create connection
        self._api_call(
            "web_backend/connections/create",
            "POST",
            data={
                "sourceId": source_id,
                "destinationId": destination_id,
                "name": f"{source_name} to {destination_name}",
                "prefix": f"{source_name}_",
                "namespaceDefinition": "destination",
                "scheduleType": "manual",
                "status": "active",
                "sourceCatalogId": catalog_id,
                "syncCatalog": {"streams": streams},
                "nonBreakingChangesPreference": "propagate_fully",
                "operations": [
                    {
                        "name": "Normalization",
                        "workspaceId": self.workspace_id,
                        "operatorConfiguration": {
                            "operatorType": "normalization",
                            "normalization": {"option": "basic"},
                        },
                    }
                ],
            },
            config_api=True,
        )

Thus, we are chaging the sources_schema_resp["catalog"]["streams"] such that we can use it in the web_backend/connections/create call.

[tomlynchRNA]

Config API endpoint: /v1/sources/check_connection
Endpoint exist in the Airbyte API? No
Use Case: We want to test the connection to a source before we try creating a connection with it.

[JonsSpaghetti]

We're looking into adding support for this in the public API directly.

Quick question to help us think about how we implement this in the public API - do you have use cases that make a check_connection call outside of the context of a Create Source/Destination call or do you always make that call right after creating the source/destination?

[jlia0]

We always make that call right after creating the source/destination, however, there should also be a need for manual / periodic check_connection because the source / destination might be unavailable or removed of the integration.

[jlia0]

This doesn't seem to be documented though, so I suspect support might disappear?

It's their internal API

[superdosh]

@JonsSpaghetti our use case does a check outside of the create as well. We want to be able to proactively indicate that an integration is unavailable.

[JonsSpaghetti]

This makes sense, thanks.

Note that support disappearing means that if you are using that internal endpoint, we can't guarantee that it will remain stable and usable and we won't document the usage of it. It doesn't mean that it will suddenly be removed - it's just that if we need to evolve the API or change it to suit one of our internal use cases, we reserve the right to do so with no warning.

You'll always be able to use that endpoint as long as you'd like, but once we add an endpoint to the public API which covers that use case, that's the one we'll officially support and keep stable for your use.

[superdosh]

Config API endpoint: /v1/connections/get
Endpoint exist in the Airbyte API? No
Use Case: I need the stream names and headers in order to show to our users. The Airbyte API has a get connection, but it doesn't include sufficient information to get the stream names and the headers for each stream. The config APIs call has the syncCatalog which contains that information.

[superdosh]

Config API endpoint: /v1/sources/check_connection
Endpoint exist in the Airbyte API? No
Use Case: I need to make sure the source is still valid before doing anything with a source. If it's not valid, I need to have our users go through a flow again to create a source. This is not available at all in the Airbyte API.

[superdosh]

Config API endpoint: /v1/jobs/get_debug_info
Endpoint exist in the Airbyte API? No
Use Case: I need to provide progress updates on a user's sync job, but the best available are the logs which come back from this debug endpoint. I can parse these logs from the config api debug endpoint to provide that. Would love if there was a cleaner way to do this.

[lraivio]

Config API endpoint: /v1/scheduler/destinations/check_connection
Endpoint exist in the Airbyte API? No
Use Case: I need to check if the configuration for a destination is correct and we can connect to the destination database before proceeding with creating the destination.

[tomlynchRNA]

Config API endpoint: /v1/source_definitions/list_latest
Endpoint exist in the Airbyte API? No
Use Case: I need to get a list of available source definitions with ID from airbyte, to display a list of available source connectors to the end user of my app

Config API endpoint: /v1/source_definition_specifications/get
Endpoint exist in the Airbyte API? No
Use Case: I need to fetch a specific source definition to get the json schema spec, so I can show a form based on the spec the user chose from the list

[evehmane]

Config API endpoint: /v1/source_definitions/list_for_workspace
Endpoint exist in the Airbyte API? No
Use Case: I need to list all available source definitions for workspace, to display a list of available source connectors to the end user of my app

[evehmane]

Config API endpoint: /v1/source_definitions/create_custom
Endpoint exist in the Airbyte API? No
Use Case: I need to create a custom source definition for the given workspace, to make custom source connectors available to the end user

[evehmane]

Config API endpoint: /v1/source_definitions/update
Endpoint exist in the Airbyte API? No
Use Case: I need to update a custom source definition to upgrade (or downgrade) the docker image version, so that the end user has the best source connector version available

[evehmane]

Config API endpoint: /v1/sources/list
Endpoint exist in the Airbyte API? Yes
Use Case: The current Airbyte API doesn't bring me source definition id for custom sources. The sourceType is returned for non-custom sources but the equivalent info for custom sources is not returned. I need this to know the type (source definition id) of all sources.

[prashant6592]

Config API endpoint: https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/sources/discover_schema
Endpoint exist in the Airbyte API? Yes
Use Case: I need the data type of column for my mysql / sql source to kick in the downstream process of transformation

[nataliekwong]

Hi there! Would https://reference.airbyte.com/reference/getstreamproperties work for what you're looking for here?

[machariamuguku]

Config API endpoint: /api/v1/workspaces/list
Endpoint exist in the Airbyte API? No
Use Case: needed for getting a list of all workspaces available for looping through for automated source definition docker image tag updates (workspace by workspace)

Config API endpoint: api/v1/source_definitions/list_for_workspace
Endpoint exist in the Airbyte API? No
Use Case: needed for getting a list of all source definitions available per workspace for automated source definition docker image tag updates

Config API endpoint: /api/v1/source_definitions/update
Endpoint exist in the Airbyte API? No
Use Case: needed for automated source definition docker image tag update

[cjwooo]

Data ingestion is a critical, user-facing section of our product. We use the Airbyte Config API to manage all aspects of source/destination definition management and connector/connection management. Almost all of our sources and destinations are using custom connector images.

Config API endpoint: /v1/workspaces/list
Endpoint exist in the Airbyte API? Yes - GET /v1/workspaces
Use Case: Find the default workspace. The Airbyte API appears to meet our requirements.

Config API endpoint: /v1/source_definition_specifications/get
Endpoint exist in the Airbyte API: No
Use Case: Fetch the specification of a source definition. Used for pre-validating a source config.

Config API endpoint: /v1/destination_definition_specifications/get
Endpoint exist in the Airbyte API: No
Use Case: Fetch the specification of a destination definition. Used for pre-validating a destination config.

Config API endpoint: /v1/source_definitions/create_custom
Endpoint exist in the Airbyte API: No
Use Case: Create source definitions for our custom sources

Config API endpoint: /v1/source_definitions/update
Endpoint exist in the Airbyte API: No
Use Case: Upgrade the image versions of our custom source definitions

Config API endpoint: /v1/source_definitions/list
Endpoint exist in the Airbyte API: No
Use Case: Check if a custom source definition exists yet, and if yes, get its ID

Config API endpoint: /v1/destination_definitions/create_custom
Endpoint exist in the Airbyte API: No
Use Case: Create source definitions for our custom destinations

Config API endpoint: /v1/destination_definitions/update
Endpoint exist in the Airbyte API: No
Use Case: Upgrade the image versions of our custom destination definitions

Config API endpoint: /v1/destination_definitions/list
Endpoint exist in the Airbyte API: No
Use Case: Check if a custom destination definition exists yet, and if yes, get its ID

Config API endpoint: /v1/sources/create
Endpoint exist in the Airbyte API: Yes - POST /v1/sources
Use Case: Create a source using our custom source definition. The Airbyte API is sufficient.

Config API endpoint: /v1/sources/update
Endpoint exist in the Airbyte API: Yes - PUT /v1/sources/{sourceId}
Use Case: Completely overwrite the configuration of an existing source. The Airbyte API is sufficient.

Config API endpoint: /v1/sources/list
Endpoint exist in the Airbyte API: Yes - GET /v1/sources
Use Case: Check if a desired source exists or needs to be created. If exists, get its source ID in order to update its config via API. The Airbyte API seems sufficient, but it would nice if the endpoint also returned each source's definition id as well.

Config API endpoint: /v1/sources/delete
Endpoint exist in the Airbyte API: Yes - DELETE /v1/sources/delete
Use Case: Delete an existing source. The Airbyte API seems sufficient.

Config API endpoint: /v1/destinations/create
Endpoint exist in the Airbyte API: Yes - POST /v1/destinations
Use Case: Create a source using our custom destination definition. The Airbyte API is sufficient.

Config API endpoint: /v1/destinations/update
Endpoint exist in the Airbyte API: Yes - PUT /v1/destinations/{destinationId}
Use Case: Completely overwrite the configuration of an existing destination. The Airbyte API is sufficient.

Config API endpoint: /v1/destinations/list
Endpoint exist in the Airbyte API: Yes - GET /v1/destinations
Use Case: Check if a desired destination exists or needs to be created. If exists, get its destination ID in order to update its config via API. The Airbyte API seems sufficient, but it would nice if the endpoint also returned each destination's definition id as well.

Config API endpoint: /v1/destinations/delete
Endpoint exist in the Airbyte API: Yes - DELETE /v1/destinations/delete
Use Case: Delete an existing source. The Airbyte API seems sufficient.

Config API endpoint: /v1/connections/list
Endpoint exist in the Airbyte API: Yes - GET /v1/connections
Use Case: Search for an existing connection with a specific name, source ID, and destination ID. The Airbyte API is sufficient.

Config API endpoint: /v1/sources/discover_schema
Endpoint exist in the Airbyte API? No
Use Case: Fetch the schema for a source in order to create the desired connection.

Config API endpoint: /v1/connections/create
Endpoint exist in the Airbyte API? Yes, but missing a feature for us - POST /v1/connections
Use Case: Create a connection. The Airbyte API is missing the ability to specify cpu/memory resource requirements.

Config API endpoint: /v1/connections/update
Endpoint exist in the Airbyte API? Yes, but missing a feature for us - PATCH /v1/connections/{connectionId}
Use Case: Update a connection. The Airbyte API is missing the ability to specify cpu/memory resource requirements.

Config API endpoint: /v1/connections/delete
Endpoint exist in the Airbyte API? Yes - DELETE /v1/connections/{connectionId}
Use Case: Delete a connection. The Airbyte API is sufficient.

Config API endpoint: /v1/state/create_or_update
Endpoint exist in the Airbyte API? No
Use Case: Make manual fixes to the incremental state of an existing connection, or copy the state from an old connection to a new connection.

Config API endpoint: /v1/state/get
Endpoint exist in the Airbyte API? No
Use Case: Fetch the current incremental state of a connection. Used for occasional connection migration tasks and for debugging our custom connectors.

Config API endpoint: /v1/connections/sync
Endpoint exist in the Airbyte API? Yes - POST /v1/jobs
Use Case: Trigger a manual sync of an existing connection. The Airbyte API is sufficient.

Config API endpoint: /v1/connections/reset
Endpoint exist in the Airbyte API? Yes - POST /v1/jobs
Use Case: Trigger a reset job for an existing connection. The Airbyte API is sufficient.

Config API endpoint: /v1/connections/status
Endpoint exist in the Airbyte API? No, but might be covered by GET /v1/jobs/{jobId}
Use Case: Get the latest status of an existing connection. Used to check if the connection had an error in the check_configuration step.

Config API endpoint: /v1/jobs/get
Endpoint exist in the Airbyte API? Yes, but missing crucial features for us - GET /v1/jobs/{jobId}
Use Case: Fetch the sync logs for a specific job attempt. The Airbyte API does not return any information on the attempts for a job, nor does it return the sync logs for those attempts. Second use case is to check the latest sync status of a connection.

Config API endpoint: /v1/workspaces/update
Endpoint exist in the Airbyte API? Yes, but missing crucial features for us - PATCH /v1/workspaces/{workspaceId}
Use Case: Configure the Airbyte workspace to send sync success/failure notifications to our backend webhook. The Airbyte API does not allow configuration of any workspace notification settings.

[nataliekwong]

Hi there! Thanks for sharing this extensive list. Can you share more about the ask for /v1/sources/discover_schema - would https://reference.airbyte.com/reference/getstreamproperties work for you as an alternative?

[cjwooo]

@nataliekwong We use the discover_schema endpoint to get the list of available streams in order to create the appropriate catalog for our database-related connections. The getstreamproperties should be sufficient for us.

[nataliekwong]

Great! Thanks for letting me know!

[hlecco]

Config API endpoint: v1/sources/check_connection
Endpoint exist in the Airbyte API? No
Use Case: Test a source after it has been created.

Config API endpoint: v1/web_backend/connections/get
Endpoint exist in the Airbyte API? No
Use Case: List available streams for a connection and check for schema changes for existing streams.

Config API endpoint: v1/web_backend/connections/update
Endpoint exist in the Airbyte API? Yes. There is a PATCH v1/connections/{connectionId} endpoint, but it does not provide the same functionality.
Use Case: Updating connections according to the response from the previous (/get) endpoint.

[nataliekwong]

Hi there! for /v1/web_backend/connections/get, can you use https://reference.airbyte.com/reference/getstreamproperties to accomplish what you're looking to do around checking for available streams & schema changes?

Also, for v1/web_backend/connections/update, can you share more about what functionality you're missing?

[MariamElZaatari]

Source Definitions

Config API endpoint: /v1/source_definitions/get
Endpoint exist in the Airbyte API? No
Use Case: Get all sources using the source definition in order to check if the source needs updating

Config API endpoint: /v1/source_definitions/create_custom
Endpoint exist in the Airbyte API? No
Use Case: Create source definitions for our custom sources

Config API endpoint: /v1/source_definitions/update
Endpoint exist in the Airbyte API? No
Use Case: Upgrade the image versions of our custom source definitions

Config API endpoint:/v1/source_definitions/list
Endpoint exist in the Airbyte API? No
Use Case: Check if a custom source definition exists yet, and if yes, get its ID

Config API endpoint:/v1/source_definitions/list_latest
Endpoint exist in the Airbyte API? No
Use Case: Get latest source definition of a custom source to update all associated sources to it

Destination Definitions

Config API endpoint: /v1/destination_definitions/list_latest
Endpoint exist in the Airbyte API? No
Use Case: Update a destination to the latest definition Id, such as GBQ

Source

Config API endpoint: /v1/sources/discover_schema
**Endpoint exist in the Airbyte API?**No
Use Case: Fetch the schema for a source in order to create the desired connection using the catalogId. Also, to get the streams of a source and enable/disable based on user preferences.

Config API endpoint: /v1/sources/search
**Endpoint exist in the Airbyte API?**No
Use Case: Search the sources using the name, we want to delete a source with a specific name.

Workspace

Config API endpoint: /v1/workspaces/update
Endpoint exist in the Airbyte API? Yes
Use Case: Configure the Airbyte workspace to send sync success/failure notifications to our backend webhook. The Airbyte API does not allow configuration of any workspace notification settings.

[nataliekwong]

Hi there! Thanks for sharing the full list, very helpful!

Regarding your ask around /v1/sources/discover_schema, would https://reference.airbyte.com/reference/getstreamproperties work for what you're looking for around the source schema and streams?

[g-jonpouliot]

Config API endpoint: /v1/jobs/get_without_logs
Endpoint exist in the Airbyte API? Yes
Use Case: We need specific data on how many rows were synced by each stream. Currently, the public API endpoint /v1/jobs/{jobId} only provides the aggregate number, not individual counts for each stream.

[nataliekwong]

Hi everyone, thanks for participating and sharing such great feedback for our team! We appreciated the detail of the use case you shared.

We have scoped our short-term API improvements here in our external roadmap. You can follow along directly in this issue: #42990

I will go ahead and close this discussion for now. Thanks again for your input!