Integrate SQLAlchemy for db conn management and introduce new SqlStorage abstraction

README.md

@@ -57,52 +57,75 @@ In another terminal:
     curl http://localhost:8000/info
 
 
-### Setting up Postgres
-
-A Postgres-backed *AnyVar* installation may use any Postgres instance, local
-or remote.  The following instructions are for using a docker-based
-Postgres instance.
-
-First, run the commands in [README-pg.md](src/anyvar/storage/README-pg.md). This will create and start a local Postgres docker instance.
-
-Next, run the commands in [postgres_init.sql](src/anyvar/storage/postgres_init.sql). This will create the `anyvar` user with the appropriate permissions and create the `anyvar` database.
-
-### Setting up Snowflake
-A Snowflake-backed *AnyVar* installation may use any Snowflake database schema.
+### SQL Database Setup
+A Postgres or Snowflake database may be used with *AnyVar*.  The Postgres database
+may be either local or remote.  Use the  `ANYVAR_STORAGE_URI` environment variable
+to define the database connection URL.  *AnyVar* uses [SQLAlchemy 1.4](https://docs.sqlalchemy.org/en/14/index.html) 
+to provide database connection management.  The default database connection URL
+is `"postgresql://postgres@localhost:5432/anyvar"`.
+
+The database integrations can be modified using the following parameters:
+* `ANYVAR_SQL_STORE_BATCH_LIMIT` - in batch mode, limit VRS object upsert batches 
+to this number; defaults to `100,000`
+* `ANYVAR_SQL_STORE_TABLE_NAME` - the name of the table that stores VRS objects; 
+defaults to `vrs_objects`
+* `ANYVAR_SQL_STORE_MAX_PENDING_BATCHES` - the maximum number of pending batches 
+to allow before blocking; defaults to `50`
+* `ANYVAR_SQL_STORE_FLUSH_ON_BATCHCTX_EXIT` - whether or not flush all pending 
+database writes when the batch manager exists; defaults to `True`
+
+The Postgres and Snowflake database connectors utilize a background thread 
+to write VRS objects to the database when operating in batch mode (e.g. annotating 
+a VCF file).  Queries and statistics query only against the already committed database 
+state.  Therefore, queries issued immediately after a batch operation may not reflect 
+all pending changes if the `ANYVAR_SQL_STORE_FLUSH_ON_BATCHCTX_EXIT` parameter is sett
+to `False`.
+
+#### Setting up Postgres
+The following instructions are for using a docker-based Postgres instance.
+
+First, run the commands in [README-pg.md](src/anyvar/storage/README-pg.md). 
+This will create and start a local Postgres docker instance.
+
+Next, run the commands in [postgres_init.sql](src/anyvar/storage/postgres_init.sql). 
+This will create the `anyvar` user with the appropriate permissions and create the 
+`anyvar` database.
+
+#### Setting up Snowflake
 The Snowflake database and schema must exist prior to starting *AnyVar*.  To point
 *AnyVar* at Snowflake, specify a Snowflake URI in the `ANYVAR_STORAGE_URI` environment
 variable.  For example:
-
-    snowflake://my-sf-acct/?database=sf_db_name&schema=sd_schema_name&user=sf_username&password=sf_password
-
+```
+snowflake://sf_username:@sf_account_identifier/sf_db_name/sf_schema_name?password=sf_password
+```
 [Snowflake connection parameter reference](https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-api)
 
-When running interactively and connecting to a Snowflake account that utilizes federated authentication or SSO, add
-the parameter `authenticator=externalbrowser`.  Non-interactive execution in a federated authentication or SSO environment
-requires a service account to connect.  Connections using an encrypted or unencrypted private key are also supported by 
-specifying the parameter `private_key=path/to/file.p8`.  The key material may be URL-encoded and inlined in the connection URI, 
+When running interactively and connecting to a Snowflake account that utilizes 
+federated authentication or SSO, add the parameter `authenticator=externalbrowser`.  
+Non-interactive execution in a federated authentication or SSO environment
+requires a service account to connect.  Connections using an encrypted or unencrypted 
+private key are also supported by specifying the parameter `private_key=path/to/file.p8`.
+The key material may be URL-encoded and inlined in the connection URI, 
 for example: `private_key=-----BEGIN+PRIVATE+KEY-----%0AMIIEvAIBA...`
 
-
 Environment variables that can be used to modify Snowflake database integration:
-* `ANYVAR_SNOWFLAKE_STORE_BATCH_LIMIT` - in batch mode, limit VRS object upsert batches to this number; defaults to `100,000`
-* `ANYVAR_SNOWFLAKE_STORE_TABLE_NAME` - the name of the table that stores VRS objects; defaults to `vrs_objects`
-* `ANYVAR_SNOWFLAKE_STORE_MAX_PENDING_BATCHES` - the maximum number of pending batches to allow before blocking; defaults to `50`
 * `ANYVAR_SNOWFLAKE_STORE_PRIVATE_KEY_PASSPHRASE` - the passphrase for an encrypted private key
-
-NOTE: If you choose to create the VRS objects table in advance, the minimal table specification is as follows:
+* `ANYVAR_SNOWFLAKE_BATCH_ADD_MODE` - the SQL statement type to use when adding new VRS objects, one of:
+    * `merge` (default) - use a MERGE statement.  This guarantees that duplicate VRS IDs will
+    not be added, but also locks the VRS object table, limiting throughput.
+    * `insert_notin` - use INSERT INTO vrs_objects SELECT FROM tmp WHERE vrs_id NOT IN (...).
+    This narrows the chance of duplicates and does not require a table lock.
+    * `insert` - use INSERT INTO.  This maximizes throughput at the cost of not checking for
+    duplicates at all.
+
+If you choose to create the VRS objects table in advance, the minimal table specification is as follows:
 ```sql
 CREATE TABLE ... (
     vrs_id VARCHAR(500) COLLATE 'utf8',
     vrs_object VARIANT
 )
 ```
 
-NOTE: The Snowflake database connector utilizes a background thread to write VRS objects to the database when operating in batch
-mode (e.g. annotating a VCF file).  Queries and statistics query only against the already committed database state.  Therefore,
-queries issued immediately after a batch operation may not reflect all pending changes.
-
-
 ## Deployment
 
 NOTE: The authoritative and sole source for version tags is the

setup.cfg

@@ -9,7 +9,7 @@ install_requires =
     uvicorn
     ga4gh.vrs[extras]~=2.0.0a5
     psycopg[binary]
-    snowflake-connector-python~=3.4.1
+    snowflake-sqlalchemy~=1.5.1
 
 [options.package_data]
 * =

src/anyvar/anyvar.py

@@ -28,8 +28,7 @@ def create_storage(uri: Optional[str] = None) -> _Storage:
     * PostgreSQL
     `postgresql://[username]:[password]@[domain]/[database]`
     * Snowflake
-    `snowflake://[account_identifier].snowflakecomputing.com/?[param=value]&[param=value]...`
-    `snowflake://[account_identifier]/?[param=value]&[param=value]...`
+    `snowflake://[user]:@[account]/[database]/[schema]?[param=value]&[param=value]...`
     """
     uri = uri or os.environ.get("ANYVAR_STORAGE_URI", DEFAULT_STORAGE_URI)