-
Notifications
You must be signed in to change notification settings - Fork 4.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add coveragerc. Documentation md file. Change version and ql level
- Loading branch information
1 parent
b2a87c7
commit 70e30bc
Showing
6 changed files
with
247 additions
and
52 deletions.
There are no files selected for viewing
3 changes: 3 additions & 0 deletions
3
airbyte-integrations/connectors/source-paypal-transaction/.coveragerc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| [run] | ||
| omit = | ||
| source_paypal_transaction/run.py |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 changes: 4 additions & 0 deletions
4
airbyte-integrations/connectors/source-paypal-transaction/unit_tests/test_source.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| from source_paypal_transaction import SourcePaypalTransaction | ||
|
|
||
| def test_source(): | ||
| assert SourcePaypalTransaction() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,78 +1,261 @@ | ||
| # Paypal Transaction | ||
| import Tabs from '@theme/Tabs'; | ||
| import TabItem from '@theme/TabItem'; | ||
|
|
||
| This page contains the setup guide and reference information for the Paypal Transaction source connector. | ||
| # Paypal | ||
|
|
||
| This page contains the setup guide and reference information for the Paypal source connector. | ||
|
|
||
| This connector uses [PayPal APIs](https://developer.paypal.com/api/rest/authentication/) OAuth 2.0 access token to authenticate requests. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| The [Paypal Transaction API](https://developer.paypal.com/docs/api/transaction-search/v1/) is used to get the history of transactions for a PayPal account. | ||
| You will need a Paypal account, which you can get following [these steps](https://developer.paypal.com/docs/platforms/get-started/) | ||
|
|
||
| In the same page, you will also find how to setup a Sandbox so you can test the connector before using it in production. | ||
|
|
||
| ## Setup guide | ||
| ### Step 1: Set up Paypal Transaction | ||
| ### Step 1: Get your Paypal secrets | ||
|
|
||
| In order to get an `Client ID` and `Secret` please go to [this](https://developer.paypal.com/docs/platforms/get-started/) page and follow the instructions. After registration you may find your `Client ID` and `Secret` [here](https://developer.paypal.com/developer/accounts/). | ||
| After creating your account you will be able to get your `Client ID` and `Secret`. You can find them in your [Apps & Credentials page](https://developer.paypal.com/dashboard/applications/live). | ||
|
|
||
| :::note | ||
|
|
||
| Our Paypal Transactions Source Connector does not support OAuth at this time due to limitations outside of our control. If OAuth for Paypal Transactions is critical to your business, [please reach out to us](mailto:[email protected]) to discuss how we may be able to partner on this effort. | ||
| ### Step 2: Set up the Paypal Transaction connector in Airbyte | ||
|
|
||
| ::: | ||
|
|
||
| ## Step 2: Set up the Paypal Transaction connector in Airbyte | ||
| 1. Log into your Airbyte account | ||
| - For Cloud [Log in here](https://cloud.airbyte.com/workspaces). | ||
|
|
||
| <!-- env:cloud --> | ||
| **For Airbyte Cloud:** | ||
| 2. In the left navigation bar, click **Sources**. | ||
|
|
||
| a. If this is your first time creating a source, use the search bar and enter **Paypal Transaction** and select it. | ||
|
|
||
| 1. [Log into your Airbyte Cloud](https://cloud.airbyte.com/workspaces) account. | ||
| 2. In the left navigation bar, click **Sources**. In the top-right corner, click **+new source**. | ||
| 3. On the Set up the source page, enter the name for the Paypal Transaction connector and select **Paypal Transaction** from the Source type dropdown. | ||
| 4. Enter your client id | ||
| 5. Enter your secret | ||
| 6. Choose if your account is sandbox | ||
| 7. Enter the date you want your sync to start from | ||
| 8. Click **Set up source**. | ||
| <!-- /env:cloud --> | ||
| b. If you already have sources configured, go to the top-right corner and click **+new source**. Then enter **Paypal Transaction** in the searech bar and select the connector. | ||
|
|
||
| 3. Set the name for your source | ||
| 4. Enter your `Client ID` | ||
| 5. Enter your `Client secret` | ||
| 6. `Start Date`: Use the provided datepicker or enter manually a UTC date and time in the format `YYYY-MM-DDTHH:MM:SSZ`. | ||
| 7. Switch ON/Off the Sandbox toggle. By defaukt the toggle is OFF, meaning it work only in a produciton environment. | ||
| 8. _(Optional) `Dispute Start Date Range`: Use the provided datepicker or enter manually a UTC date and time in the format `YYYY-MM-DDTHH:MM:SS.sssZ`. | ||
| - If you don't add a date and you sync the `lists_disputes stream`, it will use the default value of 180 days in the past to retrieve data | ||
| - It is mandatory to add the milliseconds is you enter a datetime. | ||
| - This option only works for `lists_disputes stream` | ||
|
|
||
| <!-- env:oss --> | ||
| **For Airbyte Open Source:** | ||
| 9. _(Optional)`Refresh Token`:_ You can enter manually a refresh token. Right now the stream does this automatically. | ||
| 10. _(Optional)`Number of days per request`:_ You can specify the days used by the connector when requesting data from the Paypal API. This helps in cases when you have a rate limit and you want to lower the window of retrieving data. | ||
| - Paypal has a 10K record limit per request. This option is useful if your sync is every week and you have more than 10K per week | ||
| - The default value is 7 | ||
| - This Max value you can enter is 31 days | ||
|
|
||
| 11. Click **Set up source** | ||
|
|
||
| 1. Navigate to the Airbyte Open Source dashboard | ||
| 2. Set the name for your source | ||
| 3. Enter your client id | ||
| 4. Enter your secret | ||
| 5. Choose if your account is sandbox | ||
| 6. Enter the date you want your sync to start from | ||
| 7. Click **Set up source** | ||
| <!-- /env:oss --> | ||
| :::info | ||
|
|
||
| ## Supported sync modes | ||
| By default, syncs are run with a slice period of 7 days. If you see errors with the message `Result set size is greater than the maximum limit` or an error code like `RESULTSET_TOO_LARGE`: | ||
|
|
||
| The PayPal Transaction source connector supports the following [sync modes](https://docs.airbyte.com/cloud/core-concepts#connection-sync-modes): | ||
| - Try lower the the size of the slice period in your optional parameters in your connection configuration. | ||
| - You can try to lower the scheduling sync window in case a day slice period is not enough. Lowering the sync period it may help avoid reaching the 10K limit. | ||
|
|
||
| ::: | ||
|
|
||
| | Feature | Supported? | | ||
| | :------------------------ | :--------- | | ||
| | Full Refresh Sync | Yes | | ||
| | Incremental - Append Sync | Yes | | ||
| | Namespaces | No | | ||
|
|
||
| ## Supported Streams | ||
|
|
||
| This Source is capable of syncing the following core Streams: | ||
|
|
||
| * [Transactions](https://developer.paypal.com/docs/api/transaction-search/v1/#transactions) | ||
| * [Balances](https://developer.paypal.com/docs/api/transaction-search/v1/#balances) | ||
| * [List Products](https://developer.paypal.com/docs/api/catalog-products/v1/#products_list) | ||
| * [Show Product Details](https://developer.paypal.com/docs/api/catalog-products/v1/#products_get) | ||
| * [List Disputes](https://developer.paypal.com/docs/api/customer-disputes/v1/#disputes_list) | ||
| * [Search Invoices](https://developer.paypal.com/docs/api/invoicing/v2/#invoices_search-invoices) | ||
| * [List Payments](https://developer.paypal.com/docs/api/payments/v1/#payment_list) | ||
|
|
||
|
|
||
| ### Transactions Stream | ||
|
|
||
| The below table contains the configuraiton parameters available for this connector and the default values and available features | ||
|
|
||
| | **Param/Feature** | `Transactions` | | ||
| | :-------------------------- | :------------------------ | | ||
| | `Start Date` | Timestamp with TZ (no ms) | | ||
| | `Dispute Start Date Range` | NA | | ||
| | `Refresh token` | Auto | | ||
| | `Number of days per request`| Max 31 , 7(D) | | ||
| | `Pagination Strategy` | Page Increment | | ||
| | `Page size ` | Max 500 (F) | | ||
| | `Full Refresh` | :white_check_mark: | | ||
| | `Incremental` | :white_check_mark: (D) | | ||
|
|
||
| **D:** Default configured Value | ||
|
|
||
| **F:** Fixed Value. This means it is not configurable. | ||
|
|
||
| ___ | ||
|
|
||
| ### Balances Stream | ||
|
|
||
| The below table contains the configuraiton parameters available for this connector and the default values and available features | ||
|
|
||
| | **Param/Feature** |`Balances` | | ||
| | :-------------------------- |:------------------------ | | ||
| | `Start Date` |Timestamp with TZ (no ms) | | ||
| | `Dispute Start Date Range` |NA | | ||
| | `Refresh token` |Auto | | ||
| | `Number of days per request`|NA | | ||
| | `Pagination Strategy` |NA | | ||
| | `Page size ` |NA | | ||
| | `Full Refresh` |:white_check_mark: | | ||
| | `Incremental` |:white_check_mark: (D) | | ||
|
|
||
| **D:** Default configured Value | ||
|
|
||
| **F:** Fixed Value. This means it is not configurable. | ||
|
|
||
| ___ | ||
|
|
||
|
|
||
| ### List Products Stream | ||
|
|
||
| The below table contains the configuraiton parameters available for this connector and the default values and available features | ||
|
|
||
|
|
||
| | **Param/Feature** |`List Products` | | ||
| | :-------------------------- |:------------------------ | | ||
| | `Start Date` |NA | | ||
| | `Dispute Start Date Range` |NA | | ||
| | `Refresh token` |Auto | | ||
| | `Number of days per request`|NA | | ||
| | `Pagination Strategy` |Page Increment | | ||
| | `Page size ` |Max 20 (F) | | ||
| | `Full Refresh` |:white_check_mark: (D) | | ||
| | `Incremental` |:x: | | ||
|
|
||
| **D:** Default configured Value | ||
|
|
||
| **F:** Fixed Value. This means it is not configurable. | ||
|
|
||
| :::caution | ||
|
|
||
| When configuring your stream take in consideration that the way the API works limits the speed on retreiving data. In some cases a +30K catalog retrieval could take between 10-15 minutes. | ||
|
|
||
| ::: | ||
|
|
||
| ___ | ||
|
|
||
| ### Show Products Stream | ||
|
|
||
| The below table contains the configuraiton parameters available for this connector and the default values and available features | ||
|
|
||
| | **Param/Feature** |`Show Prod. Details` | | ||
| | :-------------------------- |:------------------------ | | ||
| | `Start Date` |NA | | ||
| | `Dispute Start Date Range` |NA | | ||
| | `Refresh token` |Auto | | ||
| | `Number of days per request`|NA | | ||
| | `Pagination Strategy` |NA | | ||
| | `Page size ` |NA | | ||
| | `Full Refresh` |:white_check_mark: (D) | | ||
| | `Incremental` |:x: | | ||
|
|
||
| **D:** Default configured Value | ||
|
|
||
| ## Performance considerations | ||
| **F:** Fixed Value. This means it is not configurable. | ||
|
|
||
| Paypal transaction API has some [limits](https://developer.paypal.com/docs/integration/direct/transaction-search/) | ||
|
|
||
| * `start_date_min` = 3 years, API call lists transaction for the previous three years. | ||
| * `start_date_max` = 1.5 days, it takes a maximum of three hours for executed transactions to appear in the list transactions call. It is set to 1.5 days by default based on experience, otherwise API throw an error. | ||
| * `stream_slice_period` = 7 day, the maximum supported date range is 31 days. | ||
| * `records_per_request` = 10000, the maximum number of records in a single request. | ||
| * `page_size` = 500, the maximum page size is 500. | ||
| * `requests_per_minute` = 30, maximum limit is 50 requests per minute from IP address to all endpoint | ||
| :::caution | ||
|
|
||
| When configuring this stream consider that the parent stream paginates with 20 number of items (Max alowed page size). The Paypal API calls are not concurrent, so the time it takes depends entirely on the server side. | ||
| This stream could take a considerable time syncing, so you should consider running the sync of this and the parent stream (`list_products`) at the end of the day. | ||
| Depending on the size of the catalog it could take several hours to sync. | ||
|
|
||
| ::: | ||
|
|
||
| ___ | ||
|
|
||
| ### List Disputes Stream | ||
|
|
||
| The below table contains the configuraiton parameters available for this connector and the default values and available features | ||
|
|
||
| | **Param/Feature** |`List Disputes` | | ||
| | :-------------------------- |:------------------------ | | ||
| | `Start Date` |NA | | ||
| | `Dispute Start Date Range` |Timestamp with TZ (w/ms) | | ||
| | `Refresh token` |Auto | | ||
| | `Number of days per request`|Max 180 , 7(D) | | ||
| | `Pagination Strategy` |Page Token | | ||
| | `Page size ` |Max 50 (F) | | ||
| | `Full Refresh` |:white_check_mark: | | ||
| | `Incremental` |:white_check_mark: (D) | | ||
|
|
||
| **D:** Default configured Value | ||
|
|
||
| **F:** Fixed Value. This means it is not configurable. | ||
|
|
||
| ___ | ||
|
|
||
| ### Search Invoices Stream | ||
|
|
||
| The below table contains the configuraiton parameters available for this connector and the default values and available features | ||
|
|
||
| | **Param/Feature** |`Search Invoices` | | ||
| | :-------------------------- |:------------------------ | | ||
| | `Start Date` |Timestamp with TZ (no ms) | | ||
| | `Dispute Start Date Range` |NA | | ||
| | `Refresh token` |Auto | | ||
| | `Number of days per request`|ND | | ||
| | `Pagination Strategy` |Page Number | | ||
| | `Page size ` |Max 100 (F) | | ||
| | `Full Refresh` |:white_check_mark: (D) | | ||
| | `Incremental` |:x: | | ||
|
|
||
| **D:** Default configured Value | ||
|
|
||
| **F:** Fixed Value. This means it is not configurable. | ||
|
|
||
| **ND:** Not Defined in the source. | ||
|
|
||
|
|
||
| :::info | ||
|
|
||
| The `start_end` from the configuration, is passed to the body of the request and uses the `creation_date_range.start` and `creation_date_range.end`. More information in the [Paypal Developer API documentation](https://developer.paypal.com/docs/api/invoicing/v2/#invoices_search-invoices). | ||
|
|
||
| ::: | ||
|
|
||
|
|
||
| ___ | ||
|
|
||
| ### List Payments Stream | ||
|
|
||
| The below table contains the configuraiton parameters available for this connector and the default values and available features. | ||
|
|
||
| | **Param/Feature** |`List Payments` | | ||
| | :-------------------------- |:------------------------ | | ||
| | `Start Date` |Timestamp with TZ (no ms) | | ||
| | `Dispute Start Date Range` |NA | | ||
| | `Refresh token` |Auto | | ||
| | `Number of days per request`|NA , 7(D) | | ||
| | `Pagination Strategy` |Page Cursor | | ||
| | `Page size ` |Max 20 (F) | | ||
| | `Full Refresh` |:white_check_mark: | | ||
| | `Incremental` |:white_check_mark: (D) | | ||
|
|
||
| **D:** Default configured Value | ||
|
|
||
| **F:** Fixed Value. This means it is not configurable. | ||
|
|
||
| ___ | ||
|
|
||
| ## Performance Considerations | ||
|
|
||
| * **Data Availability:** It takes a maximum of 3 hours for executed transactions to appear in the list transactions call. | ||
| * **Number of days per request:** The maximum supported date range is 31 days. | ||
| * **Historical Data:** You can't retrieve more than 3yrs of data for the `transactions` stream. For `dispute_start_date` you can only retrieve 180 days of data (see specifications per stream) | ||
| * `records_per_request`: The maximum number of records in a single request are 10K (API Server restriction) | ||
| * `page_size`: The maximum page size is 500. This has been configured by default. | ||
| * `requests_per_minute` = The maximum limit is 50 requests per minute from IP address to all endpoint (API Server restriction). | ||
|
|
||
|
|
||
| By default, syncs are performed with a slice period of 7 days. If you see errors with the message `Result set size is greater than the maximum limit. Change the filter criteria and try again.`, lower the size of the slice period in your connection configuration. | ||
|
|
||
| ## Data type map | ||
|
|
||
|
|
@@ -83,10 +266,12 @@ By default, syncs are performed with a slice period of 7 days. If you see errors | |
| | `array` | `array` | | ||
| | `object` | `object` | | ||
|
|
||
|
|
||
| ## Changelog | ||
|
|
||
| | Version | Date | Pull Request | Subject | | ||
| |:--------|:-----------|:---------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------| | ||
| | 2.3.1 | 2024-02-07 | [34510](https://github.com/airbytehq/airbyte/pull/34510) | Silver certified. New Streams Added | | ||
| | 2.2.1 | 2024-01-11 | [34155](https://github.com/airbytehq/airbyte/pull/34155) | prepare for airbyte-lib | | ||
| | 2.2.0 | 2023-10-25 | [31852](https://github.com/airbytehq/airbyte/pull/31852) | The size of the time_window can be configured | | ||
| | 2.1.2 | 2023-10-23 | [31759](https://github.com/airbytehq/airbyte/pull/31759) | Keep transaction_id as a string and fetch data in 7-day batches | ||
|
|
||