forked from ibis-project/ibis
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(flink): implement a minimal PyFlink
Backend
* build(flink): update extra requirements and update * feat(flink): implement a minimal PyFlink `Backend` * fix(flink): pass unqualified name to DatabaseTable * fix(flink): ignore `limit` to fix `Expr.execute()` * test(flink): configure `pytest` plugin for PyFlink * ci(flink): exclude Flink backend from some linting * ci(flink): post-install `apache-flink` for testing * ci(flink): test Python 3.10 (supported by PyFlink) * test(flink): add `TestConf`, and enable test suite * test(flink): re-install pandas~=1.5 for ArrowDtype * feat(flink): implement `has_operation` for backend * ci(flink): remove the separate `test_flink` config * chore(flink): remove join tests and snapshot filess * test(flink): mark unnest operation not implemented * fix(flink): `deps` should be import names, not pip * test(flink): ignore deprecation warning, from Beam * ci(flink): quote reserved words year, month, table * test(flink): make warnings filter match more beams * ci(flink): mark Flink backend tests as serial-only * ci: turn on verbose mode in serial backend testing * test(flink): remove header for Flink backend tests * ci: use platform-agnostic way to delete first line * ci: swap double with single quotes for here-string * ci(flink): exclude Python 3.11 tests for `Backend` * revert: turn off verbose mode in serial backend testing Refs: ad387c1 * test(flink): mark STRUCT (i.e. ROW) type `notimpl` * test(flink): specify unimplemented join operations * chore: exclude flink from the operation support matrix for now * feat(flink): implement a minimal PyFlink `Backend` * ci: don't unnecessarily generate Python 3.10 stuff * test(flink): mark `notyet` for semi and anti joins * chore: undo artifacts of merge --------- Co-authored-by: Phillip Cloud <[email protected]>
- Loading branch information
Showing
15 changed files
with
173 additions
and
254 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| DROP TABLE IF EXISTS functional_alltypes; | ||
|
|
||
| CREATE TABLE functional_alltypes ( | ||
| id INT, | ||
| bool_col BOOLEAN, | ||
| tinyint_col TINYINT, | ||
| smallint_col SMALLINT, | ||
| int_col INT, | ||
| bigint_col BIGINT, | ||
| float_col FLOAT, | ||
| double_col DOUBLE, | ||
| date_string_col VARCHAR, | ||
| string_col VARCHAR, | ||
| timestamp_col TIMESTAMP, | ||
| `year` INT, | ||
| `month` INT | ||
| ) WITH ( | ||
| 'connector' = 'filesystem', | ||
| 'path' = 'file:///{data_dir}/csv/functional_alltypes.csv', | ||
| 'format' = 'csv', | ||
| 'csv.ignore-parse-errors' = 'true' | ||
| ); | ||
|
|
||
| DROP TABLE IF EXISTS diamonds; | ||
|
|
||
| CREATE TABLE diamonds ( | ||
| carat DOUBLE, | ||
| cut VARCHAR, | ||
| color VARCHAR, | ||
| clarity VARCHAR, | ||
| depth DOUBLE, | ||
| `table` DOUBLE, | ||
| price BIGINT, | ||
| x DOUBLE, | ||
| y DOUBLE, | ||
| z DOUBLE | ||
| ) WITH ( | ||
| 'connector' = 'filesystem', | ||
| 'path' = 'file:///{data_dir}/csv/diamonds.csv', | ||
| 'format' = 'csv', | ||
| 'csv.ignore-parse-errors' = 'true' | ||
| ); | ||
|
|
||
| DROP TABLE IF EXISTS batting; | ||
|
|
||
| CREATE TABLE batting ( | ||
| playerID VARCHAR, | ||
| yearID BIGINT, | ||
| stint BIGINT, | ||
| teamID VARCHAR, | ||
| lgID VARCHAR, | ||
| G BIGINT, | ||
| AB BIGINT, | ||
| R BIGINT, | ||
| H BIGINT, | ||
| X2B BIGINT, | ||
| X3B BIGINT, | ||
| HR BIGINT, | ||
| RBI BIGINT, | ||
| SB BIGINT, | ||
| CS BIGINT, | ||
| BB BIGINT, | ||
| SO BIGINT, | ||
| IBB BIGINT, | ||
| HBP BIGINT, | ||
| SH BIGINT, | ||
| SF BIGINT, | ||
| GIDP BIGINT | ||
| ) WITH ( | ||
| 'connector' = 'filesystem', | ||
| 'path' = 'file:///{data_dir}/csv/batting.csv', | ||
| 'format' = 'csv', | ||
| 'csv.ignore-parse-errors' = 'true' | ||
| ); | ||
|
|
||
| DROP TABLE IF EXISTS awards_players; | ||
|
|
||
| CREATE TABLE awards_players ( | ||
| playerID VARCHAR, | ||
| awardID VARCHAR, | ||
| yearID BIGINT, | ||
| lgID VARCHAR, | ||
| tie VARCHAR, | ||
| notes VARCHAR | ||
| ) WITH ( | ||
| 'connector' = 'filesystem', | ||
| 'path' = 'file:///{data_dir}/csv/awards_players.csv', | ||
| 'format' = 'csv', | ||
| 'csv.ignore-parse-errors' = 'true' | ||
| ); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
19 changes: 0 additions & 19 deletions
19
.../backends/flink/tests/snapshots/test_join/test_join_then_filter_no_column_overlap/out.sql
This file was deleted.
Oops, something went wrong.
18 changes: 0 additions & 18 deletions
18
.../backends/flink/tests/snapshots/test_join/test_mutate_then_join_no_column_overlap/out.sql
This file was deleted.
Oops, something went wrong.
21 changes: 0 additions & 21 deletions
21
ibis/backends/flink/tests/snapshots/test_join/test_mutating_join/inner/out.sql
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.