Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding support for Non-Genuine mongodb like DocumentDB or CosmosDB #181 #182

Closed
wants to merge 37 commits into from
Closed
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
85d9777
Create __init__.py
BobCashStory Jul 7, 2020
5ec5e6c
Add files via upload
BobCashStory Jul 7, 2020
1660d55
Create documentdb_translator.py
BobCashStory Jul 7, 2020
2048da4
Create documentdb_connector.py
BobCashStory Jul 7, 2020
3bf7805
Rename toucan_connectors/documentDB/__init__.py to toucan_connectors/…
BobCashStory Jul 7, 2020
1190ea2
Delete documentDB.jpg
BobCashStory Jul 7, 2020
10399e7
Update and rename toucan_connectors/documentDB/documentdb_translator.…
BobCashStory Jul 7, 2020
c951d49
Add files via upload
BobCashStory Jul 7, 2020
7441e78
switch from facet to group
riderx Jul 8, 2020
46b45e8
use same naming as toucan
BobCashStory Jul 8, 2020
68c9e74
Update __init__.py
riderx Jul 8, 2020
dd0011b
fix typo
riderx Jul 8, 2020
9e54742
use $lookup instead of $$ROOT
riderx Jul 8, 2020
6f70160
remove old $$ROOT
riderx Jul 8, 2020
fce6c14
test a more optimisez way
riderx Jul 8, 2020
9073afc
Update documentdb_connector.py
riderx Jul 8, 2020
d6a1b50
Update documentdb_connector.py
riderx Jul 8, 2020
db78fee
Update documentdb_connector.py
riderx Jul 8, 2020
d716112
Update documentdb_connector.py
riderx Jul 8, 2020
084ac2c
Update documentdb_connector.py
riderx Jul 8, 2020
3c0fdfb
Update documentdb_connector.py
riderx Jul 8, 2020
9bb8e38
Update documentdb_connector.py
riderx Jul 8, 2020
43d31a0
Update documentdb_connector.py
riderx Jul 8, 2020
46e0518
Update documentdb_connector.py
riderx Jul 8, 2020
39c201c
Update documentdb_connector.py
riderx Jul 8, 2020
2789a1e
MongoConnector as source of DocumentDBConnector
riderx Jul 13, 2020
22909df
Update documentdb_connector.py
riderx Jul 13, 2020
e05fc74
Update documentdb_connector.py
riderx Jul 13, 2020
f74c79d
add tests
riderx Jul 13, 2020
1734f6f
refactor: :ok_hand: remove documentdb connector, detect documentdb in…
riderx Jul 13, 2020
32b01a0
fix: :bug: remove typo
riderx Jul 13, 2020
9d814a0
refactor: :recycle: make code for documentdb simpler
riderx Jul 13, 2020
0ed48f3
refactor: :recycle: create function for is_non_geniune
riderx Jul 13, 2020
6b6b11e
remove useless attribute
riderx Jul 13, 2020
6e4a3ed
refactor: :recycle: use more standar way to detect non-genuine db
riderx Jul 13, 2020
f6a79f2
fix: :bug: error typo return of is_non_geniune was reversed
riderx Jul 13, 2020
1e58bc6
update geniune. in genuine
riderx Jul 20, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions toucan_connectors/mongo/mongo_connector.py
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,28 @@ def get_df(self, data_source, permissions=None):
data_source.query = apply_permissions(data_source.query, permissions)
return self._retrieve_data(data_source)

def is_non_genuine(self):
# Detect Non-Genuine mongoDB like documentDB or cosmosDB like in mongo compas https://github.com/mongodb-js/data-service/blob/master/lib/instance-detail-helper.js
is_genuine = True

# cosmosDB detection
try:
build_info = self.client.admin.command("buildinfo")
if build_info is None or buildInfo.get('_t') is not None:
is_genuine = False
except:
is_genuine = False

# documentDB detection
try:
get_cmd_line_opts = client.admin.command("getCmdLineOpts")
if get_cmd_line_opts is None:
is_genuine = False
except:
is_genuine = False

return not is_genuine

@decorate_func_with_retry
def get_slice(
self,
Expand All @@ -216,6 +238,16 @@ def get_slice(
) -> DataSlice:
# Create a copy in order to keep the original (deepcopy-like)
data_source = MongoDataSource.parse_obj(data_source)
if self.is_non_genuine() is True:
# specific code for Non-Genuine mongoDB who don't have $facet like documentDB or CosmosDB
total_count = MAX_COUNTED_ROWS
if offset:
data_source.query.append({'$skip': offset})
if limit is not None:
data_source.query.append({'$limit': limit})
df = self.get_df(data_source, permissions)
return DataSlice(df, total_count)

if offset or limit is not None:
data_source.query = apply_permissions(data_source.query, permissions)
data_source.query = normalize_query(data_source.query, data_source.parameters)
Expand Down