Feat/add new streams (#8)

README.md

@@ -57,12 +57,51 @@ To get your API Token, follow the [official documentation](https://apidocs.getre
 
 You can easily run `tap-getresponse` by itself or in a pipeline using [Meltano](https://meltano.com/).
 
-### Executing the Tap Directly
+
+### Execute in a Meltano pipeline
+
+#### Install the project
+
+```bash
+meltano install
+```
+
+#### Select Entities and Attributes to Extract
+
+```bash
+meltano select tap-getresponse <entity> <attribute>
+meltano select tap-getresponse --exclude <entity> <attribute>
+
+# For example:
+meltano select tap-getresponse webinars "*"
+```
+
+Verify that only the intended entities and attributes are now selected using `meltano select --list`:
+```bash
+meltano select tap-getresponse --list
+```
+
+#### Run the pipeline
+
+Run your newly added GetResponse extractor and chosen loader in a pipeline using `meltano run`:
+```bash
+meltano run tap-getresponse <loader>
+
+# For example:
+meltano run tap-getresponse target-jsonl
+```
+
+There is also the `meltano elt` (or `meltano el`) command which is a more rigid command for running only EL pipelines.
+
+Or directly using the `meltano invoke`, which only executes a single plugin at a time. This can be useful for debugging the extractor (`meltano invoke tap-getresponse`).
 
 ```bash
-tap-getresponse --version
-tap-getresponse --help
-tap-getresponse --config CONFIG --discover > ./catalog.json
+# Test invocation:
+meltano invoke tap-getresponse --version
+# OR run a test `elt` pipeline:
+meltano elt tap-getresponse <loader>
+# Example
+meltano elt tap-getresponse target-jsonl
 ```
 
 ## Developer Resources
@@ -112,14 +151,6 @@ cd tap-getresponse
 meltano install
 ```
 
-Now you can test and orchestrate using Meltano:
-
-```bash
-# Test invocation:
-meltano invoke tap-getresponse --version
-# OR run a test `elt` pipeline:
-meltano elt tap-getresponse target-jsonl
-```
 
 ### SDK Dev Guide
 

meltano.yml

@@ -1,9 +1,17 @@
 version: 1
 send_anonymous_usage_stats: true
 project_id: tap-getresponse
-default_environment: test
+default_environment: dev
 environments:
-  - name: test
+  - name: dev
+    config:
+      plugins:
+        extractors:
+          - name: tap-getresponse
+            # select:
+            #   - contact_activities.*
+  - name: uat
+  - name: prod
 plugins:
   extractors:
     - name: tap-getresponse

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "tap-getresponse"
-version = "0.0.1"
+version = "0.0.2"
 description = "`tap-getresponse` is a Singer tap for GetResponse, built with the Meltano Singer SDK."
 readme = "README.md"
 authors = ["Leo Grosjean <[email protected]>"]

tap_getresponse/client.py

@@ -128,6 +128,7 @@ def prepare_request_payload(
         # TODO: Delete this method if no payload is required. (Most REST APIs.)
         return None
 
+    # TODO: implement status code check from headers
     def parse_response(self, response: requests.Response) -> t.Iterable[dict]:
         """Parse the response and return an iterator of result records.
 
@@ -139,20 +140,3 @@ def parse_response(self, response: requests.Response) -> t.Iterable[dict]:
         """
         # TODO: Parse response body and return a set of records.
         yield from extract_jsonpath(self.records_jsonpath, input=response.json())
-
-    def post_process(
-        self,
-        row: dict,
-        context: dict | None = None,  # noqa: ARG002
-    ) -> dict | None:
-        """As needed, append or transform raw data to match expected structure.
-
-        Args:
-            row: An individual record from the stream.
-            context: The stream context.
-
-        Returns:
-            The updated record dictionary, or ``None`` to skip the record.
-        """
-        # TODO: Delete this method if not needed.
-        return row

tap_getresponse/streams/__init__.py

@@ -0,0 +1,13 @@
+from tap_getresponse.streams.campaigns import CampaignDetailsStream, CampaignsStream
+from tap_getresponse.streams.contacts import (
+    ContactActivitiesStream,
+    ContactDetailsStream,
+    ContactsStream,
+)
+from tap_getresponse.streams.newsletters import (
+    NewsletterActivitiesStream,
+    NewsletterDetailsStream,
+    NewslettersStream,
+)
+from tap_getresponse.streams.sms import SmsStream
+from tap_getresponse.streams.webinars import WebinarsStream