Syncing API Data with Custom Update Frequencies for Efficient Application Development

[marckustech]

Greetings!

I recently discovered AirByte. It's amazing! I've only been using it for a few minutes and love it. I want to build amazing applications and APIs using AirByte from now on.

We're currently trying to determine if AirByte is the best tool for our needs. I'm currently working on our company's API, which relies on a third-party provider for financial information. We have a limit of about 3,000 requests per minute, which prevents us from updating real-time data for our customers. We need a solution to efficiently sync this API data with our database.

I initially tried coding the solution myself, but the scope was too large to complete in a reasonable timeframe. While researching online, I found AirByte, which seems to be exactly what I need. However, I haven't fully tested it yet, and I'm juggling multiple tasks.

My question is: can AirByte sync data between our provider's API and our database (e.g., PostgreSQL or MySQL)? Specifically, I need to know if it's possible to set different update frequencies for different endpoints. For example, Endpoints 1 and 2 updates every minute, Endpoint 3 every hour, Endpoint 4 in real-time, and Endpoints 5 and 6 every month. Can AirByte handle these varying update schedules and real-time data synchronization?

Additionally, our API provider, Financial Modernity Prep, offers multiple endpoints, some requiring different stock symbols as parameters (e.g., AAPL, TSLA, PPYL over 50k). Since there are numerous symbols, manually configuring each endpoint is impractical. What is the best approach to dynamically manage these endpoints in AirByte, considering the different parameters and ensuring efficient data extraction? I would appreciate any guidance on setting this up and best practices.

In the meantime, I'll continue exploring AirByte, but I'm excited about its potential for our projects. Thank you for creating such an amazing tool! I am excited to hear back from you soon.

Best regards

[NAjustin]

There's a lot you can do within the no-code Builder UI, and it would probably work well for your use case.

When it comes to syncing on different schedules, that isn't currently implemented within a single connector, but you could just have multiple connectors set up for the same source/destination (one for each schedule you need)—they can still drop the tables into the same database, but this isolates the scheduling.

As a side-note, your use of the term "real-time" is the only real concern I have with what you posted. By nature, Airbyte is an ETL-focused tool, built around concepts like scheduling. While you can scheduling with high frequency (or even trigger the jobs via Airbyte's API), I would describe that as a "near-real-time" use case. In reality, very few analysts need true real-time data outside of things like algorithmic trading (the term is thrown around far too much)—so that may not be an issue. But if you're really looking for true real-time, you should probably be looking more at things like streaming data feeds or webhooks that take a scheduler or sync interval out of play. (But I digress)

As for rate limits, if the API supports traditional back-off headers, Airbyte can use these—it's just a couple clicks to set up in Builder under Error Handling. They're working on adding more back-off/rate-limiting strategies as well.

When it comes to the different stock symbols, you could cerate a stream that hits an endpoint that lists the symbols, filters it by whatever criteria you want, and then set up a child stream that then goes and fetches the details from another endpoint based on those results. It's also possible to set up custom inputs where you could provide a list of options when the connection is created and then reference those to pull the details. All that is possible in the no-code builder today.

It's also possible to use Airbyte's API to configure jobs, so you could create and trigger them from your own system if needed (passing any configuration you need that way). We wrap Airbyte functionality from within our SaaS app, bubbling up sync statuses and allowing users to manually trigger things and such (and then we trigger all jobs based on our own internal events versus running them on schedules).

All that to say that other than how you define real-time, you're very unlikely to have a lot of REST API use cases that you won't be able to solve for in the tool.