Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source Zoom: Fix authenticator bug and add missing fields #35369

Merged
merged 40 commits into from
Mar 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
8323471
fix unit test
ChristoGrab Feb 16, 2024
9e05e01
fix auth_headers bug in authenticator
ChristoGrab Feb 16, 2024
9b06ea2
add schemas for tmp parent streams
ChristoGrab Feb 16, 2024
d0061d9
update acceptance-test-config to latest format
ChristoGrab Feb 16, 2024
73d6f9f
Merge updates from master
ChristoGrab Feb 16, 2024
6a05796
add missing fields
ChristoGrab Feb 16, 2024
0c0eeec
add breakingChange metadata and migration guide
ChristoGrab Feb 16, 2024
64ccd5e
chore: formatting and typos
ChristoGrab Feb 16, 2024
9bab8e2
Merge branch 'master' of https://github.com/airbytehq/airbyte into ch…
ChristoGrab Feb 20, 2024
9023ab6
use stream_partition in manifest
ChristoGrab Feb 21, 2024
dba01c0
Merge branch 'master' of https://github.com/airbytehq/airbyte into ch…
ChristoGrab Feb 21, 2024
f364e60
update stream schemas
ChristoGrab Feb 21, 2024
47558cc
chore: add endpoint docs for tested streams
ChristoGrab Feb 21, 2024
0f31414
address all failing QA checks in pipeline
ChristoGrab Feb 22, 2024
0f2d167
use poetry for deps and enable pypi in metadata
ChristoGrab Feb 22, 2024
1795621
bump version to 0.1.7
ChristoGrab Feb 22, 2024
2f7ad46
Merge branch 'master' into christo/linnworks-registry
ChristoGrab Feb 27, 2024
10e37b4
Fast-forward to main branch
ChristoGrab Feb 27, 2024
d2bcd9f
add expected records
ChristoGrab Feb 27, 2024
2133659
chore: format
ChristoGrab Feb 27, 2024
20a3911
update bypass_reason for ignored_field
ChristoGrab Feb 27, 2024
53711c5
add back removed fields to avoid breaking schema change
ChristoGrab Feb 27, 2024
4562fbc
chore: fix to expected records
ChristoGrab Feb 28, 2024
4b9053f
update migration guide and enable in cloud registry
ChristoGrab Feb 28, 2024
7994131
chore: actually update migration gide
ChristoGrab Feb 28, 2024
bf41020
Merge branch 'christo/linnworks-registry' of https://github.com/airby…
ChristoGrab Feb 28, 2024
210a42a
remove pendulum from deps, fix readme typo
ChristoGrab Feb 28, 2024
5c4bece
Merge branch 'master' into christo/zoom-fixes
ChristoGrab Feb 28, 2024
9896808
Merge branch 'christo/zoom-fixes' of https://github.com/airbytehq/air…
ChristoGrab Feb 28, 2024
7b38aca
update cdk dep and manifest version
ChristoGrab Feb 28, 2024
ea29e47
Merge branch 'master' into christo/zoom-fixes
ChristoGrab Feb 28, 2024
f020656
Merge branch 'master' into christo/zoom-fixes
ChristoGrab Mar 11, 2024
e1c0c2a
add scopedImpact metadata
ChristoGrab Mar 11, 2024
da8adc2
chore: fix metadata
ChristoGrab Mar 11, 2024
a2c4c14
chore: format
ChristoGrab Mar 11, 2024
94ad45e
remove required keys from schemas
ChristoGrab Mar 12, 2024
bf25287
make all strings nullable in schemas
ChristoGrab Mar 12, 2024
c2dd526
all fields nullable in schemas
ChristoGrab Mar 12, 2024
4ecd141
scrub passwords from expected_records
ChristoGrab Mar 12, 2024
4f15d46
format
ChristoGrab Mar 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 0 additions & 40 deletions airbyte-integrations/connectors/source-zoom/Dockerfile

This file was deleted.

64 changes: 57 additions & 7 deletions airbyte-integrations/connectors/source-zoom/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,19 +17,70 @@ and place them into `secrets/config.json`.
### Locally running the connector docker image


#### Build
**Via [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) (recommended):**


#### Use `airbyte-ci` to build your connector
The Airbyte way of building this connector is to use our `airbyte-ci` tool.
You can follow install instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1).
Then running the following command will build your connector:

```bash
airbyte-ci connectors --name=source-zoom build
airbyte-ci connectors --name source-zoom build
```
Once the command is done, you will find your connector image in your local docker registry: `airbyte/source-zoom:dev`.

##### Customizing our build process
When contributing on our connector you might need to customize the build process to add a system dependency or set an env var.
You can customize our build process by adding a `build_customization.py` module to your connector.
This module should contain a `pre_connector_install` and `post_connector_install` async function that will mutate the base image and the connector container respectively.
It will be imported at runtime by our build process and the functions will be called if they exist.

Here is an example of a `build_customization.py` module:
```python
from __future__ import annotations

from typing import TYPE_CHECKING

if TYPE_CHECKING:
# Feel free to check the dagger documentation for more information on the Container object and its methods.
# https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/
from dagger import Container


async def pre_connector_install(base_image_container: Container) -> Container:
return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")

async def post_connector_install(connector_container: Container) -> Container:
return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")
```

An image will be built with the tag `airbyte/source-zoom:dev`.
#### Build your own connector image
This connector is built using our dynamic built process in `airbyte-ci`.
The base image used to build it is defined within the metadata.yaml file under the `connectorBuildOptions`.
The build logic is defined using [Dagger](https://dagger.io/) [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/builds/python_connectors.py).
It does not rely on a Dockerfile.

If you would like to patch our connector and build your own a simple approach would be to:

1. Create your own Dockerfile based on the latest version of the connector image.
```Dockerfile
FROM airbyte/source-zoom:latest

COPY . ./airbyte/integration_code
RUN pip install ./airbyte/integration_code

**Via `docker build`:**
# The entrypoint and default env vars are already set in the base image
# ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
# ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
```
Please use this as an example. This is not optimized.

2. Build your image:
```bash
docker build -t airbyte/source-zoom:dev .
# Running the spec command against your patched connector
docker run airbyte/source-zoom:dev spec
```

#### Run
Then run any of the connector commands as follows:
```
Expand Down Expand Up @@ -64,4 +115,3 @@ You've checked out the repo, implemented a million dollar feature, and you're re
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
6. Pat yourself on the back for being an awesome contributor.
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.

Original file line number Diff line number Diff line change
@@ -1,44 +1,55 @@
# See [Connector Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/connector-acceptance-tests-reference)
# for more information about how to configure these tests
connector_image: airbyte/source-zoom:dev
tests:
acceptance_tests:
spec:
- spec_path: "source_zoom/spec.yaml"
tests:
- spec_path: "source_zoom/spec.yaml"
connection:
- config_path: "secrets/config.json"
status: "succeed"
- config_path: "integration_tests/invalid_config.json"
status: "failed"
tests:
- config_path: "secrets/config.json"
status: "succeed"
- config_path: "integration_tests/invalid_config.json"
status: "failed"
discovery:
- config_path: "secrets/config.json"
tests:
- config_path: "secrets/config.json"
basic_read:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
timeout_seconds: 3600
empty_streams:
- "meeting_registrants"
- "meeting_polls"
- "meeting_poll_results"
- "meeting_registration_questions"
- "webinars"
- "webinar_panelists"
- "webinar_registrants"
- "webinar_absentees"
- "webinar_polls"
- "webinar_poll_results"
- "webinar_registration_questions"
- "webinar_tracking_sources"
- "webinar_qna_results"
- "report_meetings"
- "report_meeting_participants"
- "report_webinars"
- "report_webinar_participants"
tests:
- config_path: "secrets/config.json"
expect_records:
path: "integration_tests/expected_records.jsonl"
configured_catalog_path: "integration_tests/configured_catalog.json"
timeout_seconds: 3600
empty_streams:
- name: "webinars"
bypass_reason: "Need paid Webinar license to populate data"
- name: "webinar_panelists"
bypass_reason: "Need paid Webinar license to populate data"
- name: "webinar_registrants"
bypass_reason: "Need paid Webinar license to populate data"
- name: "webinar_absentees"
bypass_reason: "Need paid Webinar license to populate data"
- name: "webinar_polls"
bypass_reason: "Need paid Webinar license to populate data"
- name: "webinar_poll_results"
bypass_reason: "Need paid Webinar license to populate data"
- name: "webinar_registration_questions"
bypass_reason: "Need paid Webinar license to populate data"
- name: "webinar_tracking_sources"
bypass_reason: "Need paid Webinar license to populate data"
- name: "webinar_qna_results"
bypass_reason: "Need paid Webinar license to populate data"
- name: "report_webinars"
bypass_reason: "Need paid Webinar license to populate data"
- name: "report_webinar_participants"
bypass_reason: "Need paid Webinar license to populate data"
full_refresh:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
ignored_fields:
"meetings":
- "start_url"
"webinars":
- "start_url"
timeout_seconds: 3600
tests:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
ignored_fields:
meetings:
- name: "start_url"
bypass_reason: "Causes sequential_read test to fail as the value is unique upon each read"
timeout_seconds: 3600
Loading
Loading