Basic implementation for DuckDB as PG Engine #437
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
goldmedal
force-pushed
the
feature/duckdb-as-pg-engine-basic
branch
2 times, most recently
from
83e9dd0 to
5192bcd
Compare
|
We can use |
goldmedal
force-pushed
the
feature/duckdb-as-pg-engine-basic
branch
from
5810eaf to
8835b0e
Compare
grieve54706
reviewed
Jan 25, 2024
accio-cache/src/main/java/io/accio/cache/DuckdbRecordIterator.java
Outdated
Show resolved
Hide resolved
|
|
||
| import java.util.List; | ||
|
|
||
| public interface PgMetastore |
There was a problem hiding this comment.
This class will be implemented by many data sources different from PG. Should it not be with the Pg prefix to avoid confusion?
There was a problem hiding this comment.
The PG prefix means PG Wire Protocol which is responsible to execute the metadata query. Metastore means where we store the meta data.
|
|
||
| public class DuckDBMetadata | ||
| implements Metadata | ||
| implements Metadata, PgMetastore |
There was a problem hiding this comment.
Could you help me know what different between Metadata and PgMetastore? I saw they have same method.
There was a problem hiding this comment.
Metadata is an interface to access data source. PgMetastore is also use to access data source but the data source is used to execute PG metadata query. In our desgin,level-1 and level-2 will be executed by PgMetastore. level-3 will be exectued by the normal metadata.
accio-main/src/main/java/io/accio/main/pgcatalog/PgCatalogManager.java
Outdated
Show resolved
Hide resolved
accio-main/src/main/java/io/accio/main/pgcatalog/function/DataSourceFunctionRegistry.java
Show resolved
Hide resolved
accio-main/src/main/java/io/accio/main/pgcatalog/regtype/AbstractRegObjectFactory.java
Outdated
Show resolved
Hide resolved
accio-tests/src/test/java/io/accio/testing/bigquery/TestReloadCache.java
Outdated
Show resolved
Hide resolved
accio-tests/src/test/java/io/accio/testing/TestMDLResource.java
Outdated
Show resolved
Hide resolved
goldmedal
force-pushed
the
feature/duckdb-as-pg-engine-basic
branch
3 times, most recently
from
2b440fd to
f62127a
Compare
goldmedal
force-pushed
the
feature/duckdb-as-pg-engine-basic
branch
from
45f47fe to
286531f
Compare
grieve54706
reviewed
Jan 30, 2024
accio-base/src/main/java/io/accio/base/client/duckdb/DuckdbClient.java
Outdated
Show resolved
Hide resolved
accio-base/src/main/java/io/accio/base/client/jdbc/BaseJdbcRecordIterator.java
Show resolved
Hide resolved
accio-base/src/main/java/io/accio/base/sqlrewrite/analyzer/ExpressionTypeAnalyzer.java
Outdated
Show resolved
Hide resolved
accio-base/src/main/java/io/accio/base/sqlrewrite/analyzer/StatementAnalyzer.java
Outdated
Show resolved
Hide resolved
accio-base/src/test/java/io/accio/base/sqlrewrite/analyzer/TestStatementAnalyzer.java
Outdated
Show resolved
Hide resolved
accio-main/src/main/java/io/accio/main/wireprotocol/WireProtocolSession.java
Show resolved
Hide resolved
accio-main/src/main/java/io/accio/main/wireprotocol/WireProtocolSession.java
Outdated
Show resolved
Hide resolved
accio-main/src/main/java/io/accio/main/wireprotocol/WireProtocolSession.java
Outdated
Show resolved
Hide resolved
accio-tests/src/test/java/io/accio/testing/bigquery/AbstractCacheTest.java
Outdated
Show resolved
Hide resolved
grieve54706
reviewed
Jan 30, 2024
accio-base/src/test/java/io/accio/base/sqlrewrite/analyzer/TestStatementAnalyzer.java
Outdated
Show resolved
Hide resolved
accio-main/src/main/java/io/accio/main/wireprotocol/WireProtocolSession.java
Outdated
Show resolved
Hide resolved
accio-tests/src/test/java/io/accio/testing/bigquery/AbstractCacheTest.java
Outdated
Show resolved
Hide resolved
grieve54706
reviewed
Jan 31, 2024
accio-base/src/test/java/io/accio/base/sqlrewrite/analyzer/TestStatementAnalyzer.java
Outdated
Show resolved
Hide resolved
accio-main/src/main/java/io/accio/main/wireprotocol/WireProtocolSession.java
Show resolved
Hide resolved
goldmedal
force-pushed
the
feature/duckdb-as-pg-engine-basic
branch
from
17c925e to
e6f1384
Compare
goldmedal
added a commit
that referenced
this pull request
Feb 7, 2024
goldmedal
added a commit
that referenced
this pull request
Feb 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Previously, we executed all SQL on the data source side. Accio serves as a pure SQL conversion layer. However, it introduces challenges when it comes to supporting new data source types. Implementing a data source connector requires extensive SQL rewriting or the implementation of specific SQL dialects. Each data source presents unique issues, making it cumbersome to expand our usage efficiently.
In this PR, we present an architecture to invoke DuckDB as a single PG engine. Run every PG-related query in DuckDB and others in DataSource.
How to do
DuckDB is a highly PG compatibility SQL engine
It implements many pg_catalog view and information_schema view. We directly sync the MDL to DuckDB, then we can invoke those schemas.
3 Level Query Flow
The main purpose to implement PG Wire Protocol is enhancing our client ecosystem. We won't expose those PG usage for user common using. We only care how the SQL behavior of BI tools or PG drivers. In the past experience, the most PG-related SQL is used to get the metadata. It means we don't need to execute all query in the data source side.
Level 1 - Metastore Full supported
The SQL is full supported by DuckDB without any rewrite.
Level 2 - Metastore Semi supported
The SQL isn't supported by DuckDB but it's related to PG Metadata. We should do some rewrite for it.
Leve 3 - Data source
The SQL is used to query real data. We should execute it in Data source side.
New Configuration
duckdb.max-concurrent-metadata-queriesWe uses HikariCP as our connection pool to avoid to create connection repeatedly. This config is used to control how many max queries Accio will keep.