Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update #2

Merged
merged 31 commits into from
Feb 5, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
5da0a7f
ci: post direct links to html connector test reports (#252) (#263)
devin-ai-integration[bot] Jan 24, 2025
0d22bdd
feat: add help response for unrecognized slash commands (#264)
devin-ai-integration[bot] Jan 24, 2025
4ea9d94
feat(low-code): add items and property mappings to dynamic schemas (#…
lazebnyi Jan 24, 2025
b1824c6
feat: combine slash command jobs into single job steps (#266)
devin-ai-integration[bot] Jan 24, 2025
0a12a58
fix(low-code): Fix declarative low-code state migration in SubstreamP…
tolik0 Jan 24, 2025
c964574
feat(low-code concurrent): Allow async job low-code streams that are …
maxi297 Jan 24, 2025
a055174
chore: add test_response.txt to gitignore (#269)
devin-ai-integration[bot] Jan 25, 2025
0eba8a9
ci: migrate to Poetry 2.0 and remove `--no-update` references (#276)
devin-ai-integration[bot] Jan 28, 2025
d9d93ab
ci: fix poetry lock workflow for missing/blank file (#284)
devin-ai-integration[bot] Jan 28, 2025
d318618
feat(low-code concurrent): Add `use_global_cursor` flag to Concurrent…
tolik0 Jan 28, 2025
4f8e9d8
chore: add missing poetry-core constraint (<2.0), tighten python vers…
aaronsteers Jan 28, 2025
0b7d976
fix(low-code CDK): fix checkpointing for declarative streams (#177)
lazebnyi Jan 28, 2025
c0ba13f
ci: add pre-commit config for airbyte-python-cdk (#270)
devin-ai-integration[bot] Jan 29, 2025
3729574
chore: remove deprecated `distutils` references (#292)
aaronsteers Jan 29, 2025
128b678
feat: expand pre-commit config (#259)
devin-ai-integration[bot] Jan 29, 2025
ad20739
chore: remove pendulum dependency (#258)
devin-ai-integration[bot] Jan 29, 2025
3af96dc
ci: add Python 3.12 support (#291)
devin-ai-integration[bot] Jan 29, 2025
4e7f94a
feat(low-code): add use check availability flag to dynamic check (#293)
lazebnyi Jan 30, 2025
0b14379
ci: skip py3.12 in test matrix (#300)
aaronsteers Jan 30, 2025
33e9f5e
fix(concurrent-partition-cursor): Fix cursor comparison error (#298)
tolik0 Jan 30, 2025
a6d55be
fix: make new datetime parse functions more permissive (#296)
aaronsteers Jan 30, 2025
dea2cc9
feat(low-code): added json.loads to jwt authenticator (#301)
lazebnyi Jan 30, 2025
ee537af
feat: use create_concurrent_cursor_from_perpartition_cursor (#286)
artem1205 Jan 30, 2025
65e6a0d
fix: (OAuthAuthenticator) - get the `access_token`, `refresh_token`, …
bazarnov Jan 30, 2025
10a7a87
fix: Relocates `self._extract_slice_fields(stream_slice=stream_slice)…
pnilan Jan 31, 2025
e57d38a
docs: add migration note for pendulum removal (#308)
devin-ai-integration[bot] Feb 1, 2025
426ab5b
fix(low-code): add wrong dynamic stream name type validation (#305)
lazebnyi Feb 1, 2025
ef97304
feat(low-code): improve logging on async retriever errors (#307)
maxi297 Feb 2, 2025
979598c
chore: use python 3.11 for connector tests in CI (#313)
ChristoGrab Feb 4, 2025
126e233
feat: enable handling of nested fields when injecting request_option …
ChristoGrab Feb 4, 2025
ca68c5c
fix(concurrent cdk): Properly call set_initial_state() on the cursor …
brianjlai Feb 4, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/autofix-command.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ jobs:
- name: Set up Poetry
uses: Gr1N/setup-poetry@v9
with:
poetry-version: "1.8.4"
poetry-version: "2.0.1"
- name: Set up Python
uses: actions/setup-python@v5
with:
Expand Down
57 changes: 54 additions & 3 deletions .github/workflows/connector-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ concurrency:
jobs:
cdk_changes:
name: Get Changes
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
permissions:
statuses: write
pull-requests: read
Expand Down Expand Up @@ -62,7 +62,7 @@ jobs:
# Forked PRs are handled by the community_ci.yml workflow
# If the condition is not met the job will be skipped (it will not fail)
# runs-on: connector-test-large
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 360 # 6 hours
strategy:
fail-fast: false
Expand Down Expand Up @@ -96,6 +96,8 @@ jobs:
name: "Check: '${{matrix.connector}}' (skip=${{needs.cdk_changes.outputs['src'] == 'false' || needs.cdk_changes.outputs[matrix.cdk_extra] == 'false'}})"
permissions:
checks: write
contents: write # Required for creating commit statuses
pull-requests: read
steps:
- name: Abort if extra not changed (${{matrix.cdk_extra}})
id: no_changes
Expand Down Expand Up @@ -126,7 +128,23 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
python-version: "3.11"
# Create initial pending status for test report
- name: Create Pending Test Report Status
if: steps.no_changes.outputs.status != 'cancelled'
env:
GH_TOKEN: ${{ secrets.GH_PAT_MAINTENANCE_OCTAVIA }}
run: |
HEAD_SHA="${{ github.event.pull_request.head.sha || github.sha }}"
gh api \
--method POST \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
repos/${{ github.repository }}/statuses/$HEAD_SHA \
-f state="pending" \
-f description="Running connector tests..." \
-f context="${{ matrix.connector }} Test Report"

- name: Test Connector
if: steps.no_changes.outputs.status != 'cancelled'
timeout-minutes: 90
Expand Down Expand Up @@ -173,6 +191,39 @@ jobs:
echo "success=${success}" >> $GITHUB_OUTPUT
echo "html_report_url=${html_report_url}" >> $GITHUB_OUTPUT

# Update the test report status with results
- name: Update Test Report Status
if: always() && steps.no_changes.outputs.status != 'cancelled' && steps.evaluate_output.outcome == 'success'
env:
GH_TOKEN: ${{ secrets.GH_PAT_MAINTENANCE_OCTAVIA }}
run: |
HEAD_SHA="${{ github.event.pull_request.head.sha || github.sha }}"
gh api \
--method POST \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
repos/${{ github.repository }}/statuses/$HEAD_SHA \
-f state="${{ steps.evaluate_output.outputs.success == 'true' && 'success' || 'failure' }}" \
-f target_url="${{ steps.evaluate_output.outputs.html_report_url }}" \
-f description="Click Details to view the test report" \
-f context="${{ matrix.connector }} Test Report"

# Create failure status if report generation failed
- name: Create Report Generation Failed Status
if: always() && steps.no_changes.outputs.status != 'cancelled' && steps.evaluate_output.outcome != 'success'
env:
GH_TOKEN: ${{ secrets.GH_PAT_MAINTENANCE_OCTAVIA }}
run: |
HEAD_SHA="${{ github.event.pull_request.head.sha || github.sha }}"
gh api \
--method POST \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
repos/${{ github.repository }}/statuses/$HEAD_SHA \
-f state="failure" \
-f description="Failed to run connector tests." \
-f context="${{ matrix.connector }} Test Report"

# Upload the job output to the artifacts
- name: Upload Job Output
id: upload_job_output
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pdoc_preview.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
- name: Set up Poetry
uses: Gr1N/setup-poetry@v9
with:
poetry-version: "1.8.4"
poetry-version: "2.0.1"
- name: Set up Python
uses: actions/setup-python@v5
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pdoc_publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:
- name: Set up Poetry
uses: Gr1N/setup-poetry@v9
with:
poetry-version: "1.8.4"
poetry-version: "2.0.1"
- name: Set up Python
uses: actions/setup-python@v5
with:
Expand Down
13 changes: 12 additions & 1 deletion .github/workflows/poetry-lock-command.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,17 @@ jobs:
with:
pr: ${{ github.event.inputs.pr }}

- name: Check for blank or missing poetry.lock
run: |
if [ ! -s "poetry.lock" ]; then
echo "poetry.lock missing or blank. Fetching from main branch..."
git fetch origin main
git checkout origin/main -- poetry.lock
echo "Lock file restored from main."
else
echo "poetry.lock found. Proceeding."
fi

- name: Get PR info
id: pr-info
run: |
Expand Down Expand Up @@ -72,7 +83,7 @@ jobs:
- name: Set up Poetry
uses: Gr1N/setup-poetry@v9
with:
poetry-version: "1.8.4"
poetry-version: "2.0.1"
- name: Set up Python
uses: actions/setup-python@v5
with:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/pytest_fast.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
- name: Set up Poetry
uses: Gr1N/setup-poetry@v9
with:
poetry-version: "1.8.4"
poetry-version: "2.0.1"

- name: Check Poetry lock file is current
run: poetry check
Expand All @@ -44,7 +44,7 @@ jobs:
- name: Set up Poetry
uses: Gr1N/setup-poetry@v9
with:
poetry-version: "1.8.4"
poetry-version: "2.0.1"
- name: Set up Python
uses: actions/setup-python@v5
with:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/pytest_matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ jobs:
python-version: [
"3.10",
"3.11",
#'3.12', # Currently blocked by Pendulum
# "3.12", # `requests-cache` blocker: https://github.com/airbytehq/airbyte-python-cdk/issues/299
]
os: [
Ubuntu,
Expand Down Expand Up @@ -61,7 +61,7 @@ jobs:
uses: Gr1N/setup-poetry@v9
if: steps.changes.outputs.src == 'true'
with:
poetry-version: "1.8.4"
poetry-version: "2.0.1"
- name: Set up Python
uses: actions/setup-python@v5
if: steps.changes.outputs.src == 'true'
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/python_lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
- name: Set up Poetry
uses: Gr1N/setup-poetry@v9
with:
poetry-version: "1.8.4"
poetry-version: "2.0.1"
- name: Set up Python
uses: actions/setup-python@v5
with:
Expand Down
34 changes: 33 additions & 1 deletion .github/workflows/slash_command_dispatch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,9 @@ jobs:
steps:
- name: Slash Command Dispatch
id: dispatch
uses: peter-evans/slash-command-dispatch@v4
# TODO: Revert to `peter-evans/slash-command-dispatch@v4` after PR merges:
# - https://github.com/peter-evans/slash-command-dispatch/pull/372/files
uses: aaronsteers/slash-command-dispatch@aj/fix/add-dispatched-bool-output
with:
repository: ${{ github.repository }}
token: ${{ secrets.GH_PAT_MAINTENANCE_OCTAVIA }}
Expand All @@ -36,3 +38,33 @@ jobs:
comment-id: ${{ github.event.comment.id }}
body: |
> Error: ${{ steps.dispatch.outputs.error-message }}

- name: Generate help text
id: help
if: >
startsWith(github.event.comment.body, '/') &&
!steps.dispatch.outputs.dispatched
run: |
HELP_TEXT="The following slash commands are available:

- \`/autofix\` - Corrects any linting or formatting issues
- \`/test\` - Runs the test suite
- \`/poetry-lock\` - Re-locks dependencies and updates the poetry.lock file
- \`/help\` - Shows this help message"

if [[ "${{ github.event.comment.body }}" == "/help" ]]; then
echo "body=$HELP_TEXT" >> $GITHUB_OUTPUT
else
echo "body=It looks like you are trying to enter a slash command. Either the slash command is unrecognized or you don't have access to call it.

$HELP_TEXT" >> $GITHUB_OUTPUT
fi

- name: Post help message
if: >
startsWith(github.event.comment.body, '/') &&
!steps.dispatch.outputs.dispatched
uses: peter-evans/create-or-update-comment@v4
with:
comment-id: ${{ github.event.comment.id }}
body: ${{ steps.help.outputs.body }}
2 changes: 1 addition & 1 deletion .github/workflows/test-command.yml
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ jobs:
- name: Set up Poetry
uses: Gr1N/setup-poetry@v9
with:
poetry-version: "1.8.4"
poetry-version: "2.0.1"
- name: Set up Python
uses: actions/setup-python@v5
with:
Expand Down
63 changes: 63 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
exclude: |
(?x)(
# Python/system files
^.*/__init__\.py$|
^.*?/\.venv/.*$|
^.*?/node_modules/.*$|

# Generated/test files
^.*?/\.pytest_cache/.*$|
^.*?/__pycache__/.*$|
^.*?/\.mypy_cache/.*$|
^.*?/\.ruff_cache/.*$

# Package management
^.*?/poetry\.lock$|
^.*?/package-lock\.json$|
^.*?/pnpm-lock\.yaml$|

# Build and test artifacts
^.*?/build/.*$|
^.*?/dist/.*$|
^.*?/\.coverage$|
^.*?/coverage\.xml$|
)

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-added-large-files
- id: check-toml

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.8.3
hooks:
# Run the linter with repo-defined settings
- id: ruff
args: [--fix]

# Run the formatter with repo-defined settings
- id: ruff-format

- repo: https://github.com/pre-commit/mirrors-prettier
rev: v3.0.3
hooks:
- id: prettier
types_or: [json, yaml]
additional_dependencies:
- [email protected]

- repo: local
hooks:
- id: addlicense
name: Add license headers
entry: addlicense -c "Airbyte, Inc." -l apache -v -f LICENSE_SHORT
language: golang
additional_dependencies: [github.com/google/[email protected]]
files: \.py$
1 change: 1 addition & 0 deletions LICENSE_SHORT
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Copyright (c) 2025 Airbyte, Inc., all rights reserved.
6 changes: 3 additions & 3 deletions airbyte_cdk/cli/source_declarative_manifest/_run.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
import sys
import traceback
from collections.abc import Mapping
from datetime import datetime
from pathlib import Path
from typing import Any, cast

Expand All @@ -44,6 +43,7 @@
)
from airbyte_cdk.sources.declarative.yaml_declarative_source import YamlDeclarativeSource
from airbyte_cdk.sources.source import TState
from airbyte_cdk.utils.datetime_helpers import ab_datetime_now


class SourceLocalYaml(YamlDeclarativeSource):
Expand Down Expand Up @@ -101,7 +101,7 @@ def _get_local_yaml_source(args: list[str]) -> SourceLocalYaml:
type=Type.TRACE,
trace=AirbyteTraceMessage(
type=TraceType.ERROR,
emitted_at=int(datetime.now().timestamp() * 1000),
emitted_at=ab_datetime_now().to_epoch_millis(),
error=AirbyteErrorTraceMessage(
message=f"Error starting the sync. This could be due to an invalid configuration or catalog. Please contact Support for assistance. Error: {error}",
stack_trace=traceback.format_exc(),
Expand Down Expand Up @@ -191,7 +191,7 @@ def create_declarative_source(
type=Type.TRACE,
trace=AirbyteTraceMessage(
type=TraceType.ERROR,
emitted_at=int(datetime.now().timestamp() * 1000),
emitted_at=ab_datetime_now().to_epoch_millis(),
error=AirbyteErrorTraceMessage(
message=f"Error starting the sync. This could be due to an invalid configuration or catalog. Please contact Support for assistance. Error: {error}",
stack_trace=traceback.format_exc(),
Expand Down
4 changes: 2 additions & 2 deletions airbyte_cdk/connector_builder/connector_builder_handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
#

import dataclasses
from datetime import datetime
from typing import Any, List, Mapping

from airbyte_cdk.connector_builder.message_grouper import MessageGrouper
Expand All @@ -21,6 +20,7 @@
ModelToComponentFactory,
)
from airbyte_cdk.utils.airbyte_secrets_utils import filter_secrets
from airbyte_cdk.utils.datetime_helpers import ab_datetime_now
from airbyte_cdk.utils.traced_exception import AirbyteTracedException

DEFAULT_MAXIMUM_NUMBER_OF_PAGES_PER_SLICE = 5
Expand Down Expand Up @@ -114,4 +114,4 @@ def resolve_manifest(source: ManifestDeclarativeSource) -> AirbyteMessage:


def _emitted_at() -> int:
return int(datetime.now().timestamp()) * 1000
return ab_datetime_now().to_epoch_millis()
Loading
Loading