Source: Twilio - Fix Pagination for 'TwilioStream'

[agarctfi]

What

Some Twilio streams are having issues paginating and seem only to be grabbing the first page.
https://github.com/airbytehq/oncall/issues/7272

How

I updated a typo about how we grab the nex_page_url, which should enable pagination again for the impacted streams.

I added these debug print statements:

        print(f"stream_data: {stream_data}")
        next_page_uri = stream_data.get("next_page_uri")
        print(f"next_page_uri: {next_page_uri}")

I also ran a read test:
poetry run python main.py read --config secrets/config.json --catalog integration_tests/constant_records_catalog.json --debug > test_1.log

Here is the output:

stream_data: {
  "meta":
    {
      "page": 0,
      "page_size": 100,
      "first_page_url": "https://conversations.twilio.com/v1/Conversations/CH0ed7b4c3498e455a96fa09fcccee720e/Messages?PageSize=100&Page=0",
      "previous_page_url": None,
      "url": "https://conversations.twilio.com/v1/Conversations/CH0ed7b4c3498e455a96fa09fcccee720e/Messages?PageSize=100&Page=0",
      "next_page_url": None,
      "key": "messages",
    },
  "messages":
    [
      {
        "body": "Ahoy there",
        "index": 0,
        "author": "smee",
        "date_updated": "2023-04-01T12:37:19Z",
        "media": None,
        "participant_sid": None,
        "conversation_sid": "CH0ed7b4c3498e455a96fa09fcccee720e",
        "account_sid": "ACdade166c12e160e9ed0a6088226718fb",
        "delivery": None,
        "url": "https://conversations.twilio.com/v1/Conversations/CH0ed7b4c3498e455a96fa09fcccee720e/Messages/IMd28bbec7d60f4c9b84595170871c6f28",
        "date_created": "2023-04-01T12:37:19Z",
        "content_sid": None,
        "sid": "IMd28bbec7d60f4c9b84595170871c6f28",
        "attributes": "{}",
        "links":
          {
            "delivery_receipts": "https://conversations.twilio.com/v1/Conversations/CH0ed7b4c3498e455a96fa09fcccee720e/Messages/IMd28bbec7d60f4c9b84595170871c6f28/Receipts",
            "channel_metadata": "https://conversations.twilio.com/v1/Conversations/CH0ed7b4c3498e455a96fa09fcccee720e/Messages/IMd28bbec7d60f4c9b84595170871c6f28/ChannelMetadata",
          },
      },
    ],
}


next_page_uri: None

I think we need to change stream_data.get("next_page_uri") to stream_data.get("next_page_url")

Review guide

Check streams.py & ensure there aren't other typos with next_page_url

User Impact

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 23, 2025 0:42am

[agarctfi]

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (236180e)

[agarctfi]

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (03fb632)

[agarctfi]

Regression ran without state:
https://storage.cloud.google.com/airbyte-ci-reports-multi/airbyte-ci/connectors/test/manual/agarctf_source-twillio_pagination-issue/1737597535/03fb632255a11dc16f10b86e4f946a1fd19cf021/source-twilio/0.11.16/output.html

With State:
https://storage.cloud.google.com/airbyte-ci-reports-multi/airbyte-ci/connectors/test/manual/agarctf_source-twillio_pagination-issue/1737589261/236180e4bbd1cd6993182cd85f91d98c3fe0c011/source-twilio/0.11.16/output.html

Tests look good (we get more records on the reported streams) except for the alerts stream. On the regression, it shows no records loaded.

I've added debug statements, but I don't see how we get zero regression records. I tested on the cloud with the current version and dev version and got the same amount of records still. 74

I think it will be worth running another test with page_size = 20 to see if the error is happening during pagination, though.