AsyncDefog

[Muhammad18557]

For postgres and redshift, we have the asynchronous asyncpg to replace psycopg2
For mysql, we have aiomysql to replace mysql.converter
For snowflake, we use the same connector as it also offers a cur.execute_async method- we then poll the status of the query id to see if its done running otherwise just simulate an async sleep using await asyncio.sleep(1) .
For sqlserver, we use aioodbc which has the exact same signature as pyodbc but is asynchronous and must be awaited.

However, for Databricks and BigQuery, no native asynchronous alternatives were found. To mimic asynchronous behavior for these databases, we are using asyncio.to_thread to handle blocking operations in a non-blocking manner.

Methods of AsyncDefog have the exact same signature (name, params) but have to be awaited. The initial methods like save_connection_json, check_db_creds, to_base64_creds, from_base64_creds are kept as it is (not made async) but do let me know if those also need to be async and that will be a 3 mins fix (because it will just need a wrapper off its parent class's methods into an async function i.e async def function(): return self.function()).

create_table_ddl and create_ddl_from_metadata also made async under async_admin_methods for the sake of consistency even though they are not I/O bound or db intensive.

[rishsriv]

Thanks Abdullah! This looks very, very good. And +1 for the tests.

Could you please add native asynchronous mode for the following:

• Snowflake: example here
• SQLServer, by using aioodbc. Example here

After that, I can test and merge!

[Muhammad18557]

Thanks for pointing this out! Will add native support for these! Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Rishabh Srivastava ***@***.***> Sent: Tuesday, September 10, 2024 10:03:11 PM To: defog-ai/defog-python ***@***.***> Cc: Muhammad Abdullah ***@***.***>; Author ***@***.***> Subject: Re: [defog-ai/defog-python] AsyncDefog (PR #61) - External Email - @rishsriv approved this pull request. Thanks Abdullah! This looks very, very good. And +1 for the tests. Could you please add native asynchronous mode for the following: * Snowflake: example here<https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-example#submitting-an-asynchronous-query> * SQLServer, by using aioodbc. Example here<mkleehammer/pyodbc#901 (comment)> After that, I can test and merge! — Reply to this email directly, view it on GitHub<#61 (review)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AZGPXGWBY5KVWH5A6W4EVDLZV33Z7AVCNFSM6AAAAABN6POFK2VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEOJSGYZTKMBRGE>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[rishsriv]

Finished testing and this looks great Abdullah! Thanks for adding this much needed feature!