Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Source Paypal Transactions: Silver Certification #34510

Merged
merged 40 commits into from
Feb 17, 2024
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
d82799d
Start Silver certification. Add new streams
Jgerardopine Jan 23, 2024
d1b8976
Add Search invoices stream. Parametrize some variables.
Jgerardopine Jan 24, 2024
8a0a6bb
Add new stream - Product catalog -. Add error handling for transactio…
Jgerardopine Jan 25, 2024
57a62e9
New Stream - list disputes. Add data generator to test data.
Jgerardopine Jan 26, 2024
ca8e027
Add new stream and substream - show product details.
Jgerardopine Jan 27, 2024
4d35714
Add new streams. Add seeding to some tests. Change Metadata version.
Jgerardopine Feb 1, 2024
b2a87c7
Add tests. Organize and clean some code. Commented out show_product_d…
Jgerardopine Feb 7, 2024
70e30bc
Add coveragerc. Documentation md file. Change version and ql level
Jgerardopine Feb 8, 2024
6d2a730
Update CDK version
Jgerardopine Feb 8, 2024
9fb2466
Start Silver certification. Add new streams
Jgerardopine Jan 23, 2024
d66a2b9
Add Search invoices stream. Parametrize some variables.
Jgerardopine Jan 24, 2024
9c430da
Add new stream - Product catalog -. Add error handling for transactio…
Jgerardopine Jan 25, 2024
22c4980
New Stream - list disputes. Add data generator to test data.
Jgerardopine Jan 26, 2024
85913ec
Add new stream and substream - show product details.
Jgerardopine Jan 27, 2024
42f7138
Add new streams. Add seeding to some tests. Change Metadata version.
Jgerardopine Feb 1, 2024
7c7ff74
Add tests. Organize and clean some code. Commented out show_product_d…
Jgerardopine Feb 7, 2024
6b6b9e8
Add coveragerc. Documentation md file. Change version and ql level
Jgerardopine Feb 8, 2024
2fa110b
Update CDK version
Jgerardopine Feb 8, 2024
87442ed
Merge branch 'jgpines/paypal-trans-silver-cert' of https://github.com…
Jgerardopine Feb 8, 2024
f7f5028
Fix format (airbyte-ci)
Jgerardopine Feb 8, 2024
710ad0f
Comment out Sandbox tests. Leave only configg.json for CI tests
Jgerardopine Feb 8, 2024
106c760
Add empty streams to pass CI tests using GSM sandbox credentials
Jgerardopine Feb 8, 2024
341abc3
Fix format JS. Validate records and streams
Jgerardopine Feb 8, 2024
93e7687
Fix comments. Add test and test files
Jgerardopine Feb 15, 2024
14ef96a
Update airbyte-integrations/connectors/source-paypal-transaction/CHAN…
Jgerardopine Feb 15, 2024
3c944ed
Fix conflicts
Jgerardopine Feb 16, 2024
6ad0d11
Fix merge conflicts Poetry. Update versions in poetry.
Jgerardopine Feb 16, 2024
5ce5dbd
Merge branch 'master' into jgpines/paypal-trans-silver-cert
Jgerardopine Feb 16, 2024
dd2987c
regenerate poetry.lock with poetry 1.6.1
alafanechere Feb 16, 2024
3a1ae80
Merge branch 'master' into jgpines/paypal-trans-silver-cert
Jgerardopine Feb 16, 2024
7494eca
Update airbyte-integrations/connectors/source-paypal-transaction/pypr…
Jgerardopine Feb 16, 2024
f413325
Add end date to spec(optional). Change python to poetry in ReadME file
Jgerardopine Feb 16, 2024
c516725
Add pattern to end_date
Jgerardopine Feb 16, 2024
ee42253
Change congig[] to config.get and add dat-time
Jgerardopine Feb 16, 2024
a9bc636
Merge branch 'master' into jgpines/paypal-trans-silver-cert
Jgerardopine Feb 16, 2024
3891edd
Update airbyte-integrations/connectors/source-paypal-transaction/READ…
Jgerardopine Feb 16, 2024
d6f252f
Update invoices.py
Jgerardopine Feb 16, 2024
c197b07
Update payments_generator.py
Jgerardopine Feb 16, 2024
9b8276d
Update paypal_transaction_generator.py
Jgerardopine Feb 16, 2024
85e5080
Fix Python format (airbyte-ci)
Jgerardopine Feb 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[run]
omit =
source_paypal_transaction/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,12 @@

## 0.1.0
Source implementation with support of Transactions and Balances streams

## 1.0.0
Mark Client ID and Client Secret as required files

## 2.1.0
Migration to Low code

## 2.3.1
Jgerardopine marked this conversation as resolved.
Show resolved Hide resolved
Adding New Streams - Payments, Disputes, Invoices, Product Catalog
Original file line number Diff line number Diff line change
@@ -1,69 +1,79 @@
# See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference)
# for more information about how to configure these tests
# Make sure the paths you have in each path matches with your data.
#For multiple env testing, you can duplicate the tests and change the path to the proper credentials file
Jgerardopine marked this conversation as resolved.
Show resolved Hide resolved
connector_image: airbyte/source-paypal-transaction:dev
test_strictness_level: high
acceptance_tests:
spec:
tests:
#Test with Prod credentials (Make sure you purt the right ones)
- spec_path: "source_paypal_transaction/spec.yaml"
config_path: secrets/config_oauth.json
config_path: secrets/config.json
backward_compatibility_tests_config:
disable_for_version: "0.1.13"
connection:
tests:
- config_path: secrets/config_oauth.json
#Test With Prod Credentials
- config_path: secrets/config.json
status: succeed
- config_path: secrets/config_oauth_sandbox.json
status: succeed
- config_path: integration_tests/invalid_config.json
status: failed
- config_path: integration_tests/invalid_config_oauth.json
#Test with Invalid Credentials
- config_path: integration_tests/sample_files/invalid_config.json
status: failed
#Test with Sandbox Credentials
# - config_path: secrets/config_sandbox.json
# status: succeed
discovery:
tests:
- config_path: secrets/config_oauth.json
- config_path: secrets/config.json
backward_compatibility_tests_config:
disable_for_version: "2.0.0" # Change in cursor field for transactions stream
basic_read:
tests:
- config_path: secrets/config_oauth.json
ignored_fields:
balances:
- name: last_refresh_time
bypass_reason: "field changes during every read"
#Test Prod Environment - Uncomment and change according to your prod setup
#Change the expected records, remember to align them with the timeframe you have selected
#Do not select streams that take more than 5 mins to load data as that can lead to timeouts
#You can comment the lines if you are sure you have data for the below streams.
- config_path: secrets/config.json
empty_streams:
- name: transactions
bypass_reason: "can not populate"
timeout_seconds: 1200
expect_records:
path: "integration_tests/expected_records.jsonl"
extra_fields: no
exact_order: no
extra_records: yes
- config_path: secrets/config_oauth_sandbox.json
- name: show_product_details
bypass_reason: "Products may not exist"
- name: list_products
bypass_reason: "Product List may be too big causing timeout errors"
- name: search_invoices
bypass_reason: "Order makes the diff fail."
#Have to add for testing PR CI.
- name: list_disputes
bypass_reason: "Disputes may not exist."
ignored_fields:
balances:
- name: last_refresh_time
bypass_reason: "field changes during every read"
timeout_seconds: 1200
list_products:
- name: description
bypass_reason: "Sometimes it is not contained in the response"
timeout_seconds: 3200
expect_records:
path: "integration_tests/expected_records_sandbox.jsonl"
extra_fields: no
exact_order: no
extra_records: yes
fail_on_extra_columns: false
path: "integration_tests/sample_files/expected_records_sandbox.jsonl"
extra_fields: yes
exact_order: yes
extra_records: no
fail_on_extra_columns: False
incremental:
tests:
- config_path: secrets/config_oauth.json
configured_catalog_path: integration_tests/configured_catalog.json
- config_path: secrets/config.json
configured_catalog_path: integration_tests/incremental_catalog.json
future_state:
future_state_path: integration_tests/abnormal_state.json
future_state_path: integration_tests/sample_files/abnormal_state.json
skip_comprehensive_incremental_tests: true
full_refresh:
tests:
- config_path: secrets/config_oauth.json
- config_path: secrets/config.json
configured_catalog_path: integration_tests/full_refresh_catalog.json
ignored_fields:
balances:
- name: last_refresh_time
bypass_reason: "field changes during every read"
configured_catalog_path: integration_tests/configured_catalog.json
list_products:
- name: description
bypass_reason: "Sometimes it is not contained in the response"
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.

# REQUIREMENTS:
# 1. Put your sandbox credentials in ../secrets/config.json (Create them if it doesn't exist).
# Use the following body (change all the values):
# {
# "client_id": "YOUT_CLIENT_ID",
# "client_secret": "YOUR_SECRET_CLIENT_ID",
# "start_date": "2021-06-01T00:00:00Z",
# "end_date": "2024-06-10T00:00:00Z",
# "is_sandbox": true
# }

# HOW TO USE:
# To create a new payment: python script_name.py create
# To update an existing dispute:
# python disputes_generator.py update DISPUTE_ID ''[{"op": "replace", "path": "/reason", "value": "The new reason"}]'
# To update a dispute status
# python update_dispute.py require-evidence DISPUTE_ID SELLER_EVIDENCE

import base64
import json
import sys

import requests


# Function to get a PayPal OAuth token
def get_paypal_token(client_id, secret_id):
url = "https://api-m.sandbox.paypal.com/v1/oauth2/token"
headers = {
"Content-Type": "application/x-www-form-urlencoded",
"Authorization": "Basic " + base64.b64encode(f"{client_id}:{secret_id}".encode()).decode(),
}
payload = {"grant_type": "client_credentials"}
response = requests.post(url=url, data=payload, headers=headers)
return response.json().get("access_token")


def update_dispute(token, dispute_id, updates):
"""Update a PayPal dispute."""
url = f"https://api-m.paypal.com/v1/customer/disputes/{dispute_id}"
headers = {"Content-Type": "application/json", "Authorization": f"Bearer {token}"}
response = requests.patch(url, headers=headers, json=updates)
print("RESPONSE: ", response.text)
return response.json()


def require_evidence(token, dispute_id, action):
"""Require evidence for a PayPal dispute."""
url = f"https://api-m.paypal.com/v1/customer/disputes/{dispute_id}/require-evidence"
headers = {"Content-Type": "application/json", "Authorization": f"Bearer {token}"}
payload = {"action": action}
response = requests.post(url, headers=headers, json=payload)
print("RESPONSE: ", response.text)
return response.json()


def read_json(filepath):
with open(filepath, "r") as f:
return json.loads(f.read())


def main():
Jgerardopine marked this conversation as resolved.
Show resolved Hide resolved

operation = sys.argv[1]

CREDS = read_json("../secrets/config.json")
client_id = CREDS.get("client_id")
secret_id = CREDS.get("client_secret")
token = get_paypal_token(client_id, secret_id)

if operation == "update":
dispute_id = sys.argv[2]
updates = json.loads(sys.argv[3]) # Expecting JSON string as the third argument
update_response = update_dispute(token, dispute_id, updates)
print("Update Response:", update_response)

elif sys.argv[1] == "require-evidence":
dispute_id = sys.argv[2]
action = sys.argv[3] # Either 'BUYER_EVIDENCE' or 'SELLER_EVIDENCE'
evidence_response = require_evidence(token, dispute_id, action)
print("Evidence Requirement Response:", evidence_response)
else:
print("Invalid command. Use 'create', 'update', or 'require-evidence'.")


if __name__ == "__main__":
main()
Loading
Loading