Skip to content

Commit

Permalink
Remove deprecated module (#297)
Browse files Browse the repository at this point in the history
* Remove deprecated module

Since we are doing a major version bump, we might as well remove the
deprecated `middleware` module.

* remove tests too

* Fix more tests
  • Loading branch information
mathialo authored Feb 15, 2024
1 parent b2f9392 commit bfa7bd7
Show file tree
Hide file tree
Showing 5 changed files with 4 additions and 112 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ Changes are grouped as follows
- The queues now require to be set up with a max size. The max upload latencey is removed.
As long as you use the queue in as a context (ie, using `with FileUploadQueue(...) as queue:`) you should not have to change anything in your code. The behaviour of the queue will change, it will most likely be much faster, but it will not require any changes from you as a user of the queue.

### Removed

* The deprecated `middleware` module has been removed.

### Added

* You can now use `Path` as a type in your config files
Expand Down
36 changes: 0 additions & 36 deletions cognite/extractorutils/middleware.py

This file was deleted.

1 change: 0 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,6 @@ decorator = "^5.1.1"
more-itertools = "^10.0.0"
typing-extensions = ">=3.7.4, <5"
python-dotenv = "^1.0.0"
jq = [{version = "^1.3.0", platform = "darwin"}, {version = "^1.3.0", platform = "linux"}]
azure-identity = "^1.14.0"
azure-keyvault-secrets = "^4.7.0"

Expand Down
44 changes: 0 additions & 44 deletions tests/tests_unit/test_middleware.py

This file was deleted.

31 changes: 0 additions & 31 deletions tests/tests_unit/test_uploader_extractor.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@

from cognite.client import CogniteClient
from cognite.client.data_classes import Row
from cognite.extractorutils.middleware import JQMiddleware
from cognite.extractorutils.uploader import EventUploadQueue, RawUploadQueue, TimeSeriesUploadQueue
from cognite.extractorutils.uploader_extractor import UploaderExtractor, UploaderExtractorConfig
from cognite.extractorutils.uploader_types import Event, InsertDatapoints, RawRow
Expand Down Expand Up @@ -117,33 +116,3 @@ def test_handle_timeseries(self, MockCogniteClient):
{"externalId": "some-other-id", "datapoints": tss[1].datapoints},
]
)

@patch("cognite.client.CogniteClient")
def test_middleware_jq(self, MockCogniteClient):
client: CogniteClient = MockCogniteClient()

ex = UploaderExtractor[UploaderExtractorConfig](
name="ext_extractor4", description="testing jq middleware", config_class=UploaderExtractorConfig
)

ex.raw_queue = RawUploadQueue(client)

jq_payload = """
{
"new_foo": .foo,
"new_baz": .baz,
"new_val": "some val"
}
"""
ex.middleware = [JQMiddleware(jq_rules=jq_payload)]
r = Row("foo", columns={"foo": "bar", "baz": "bax"})
row = RawRow(db_name="some-db", table_name="some-table", row=r)
ex.handle_output(row)

ex.raw_queue.upload() # just to clear the queue

assert (
r.columns.get("new_foo") == "bar"
and r.columns.get("new_baz") == "bax"
and r.columns.get("new_val") == "some val"
)

0 comments on commit bfa7bd7

Please sign in to comment.