Merge remote-tracking branch 'origin/master' into hosted-extractors-p…

…art2 # Conflicts: # CHANGELOG.md # cognite/client/_api/hosted_extractors/__init__.py # cognite/client/_api/hosted_extractors/sources.py # cognite/client/_api_client.py # cognite/client/_version.py # cognite/client/data_classes/hosted_extractors/__init__.py # cognite/client/data_classes/hosted_extractors/sources.py # cognite/client/testing.py # docs/source/hosted_extractors.rst # pyproject.toml # tests/tests_integration/test_api/test_hosted_extractors/test_sources.py # tests/tests_unit/test_base.py
cognitedata · Sep 5, 2024 · 5f2fa66 · 5f2fa66
2 parents 452d7eb + 8a8ed12
commit 5f2fa66
Show file tree

Hide file tree

Showing 50 changed files with 1,533 additions and 228 deletions.
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -12,7 +12,7 @@ jobs:
       - uses: actions/checkout@v4
       - uses: ./.github/actions/setup
         with:
-          extras: '-E pandas'
+          extras: "-E pandas"
       - name: Linting and static code checks
         run: pre-commit run --all-files
 
@@ -46,13 +46,13 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, windows-latest]
-        python-version: ['3.8', '3.11', '3.12']  # TODO: 3.9, 3.10 (requires a lot of work for FakeCogResGen for tests)
+        python-version: ["3.8", "3.11", "3.12"] # TODO: 3.9, 3.10 (requires a lot of work for FakeCogResGen for tests)
     steps:
       - uses: actions/checkout@v4
       - uses: ./.github/actions/setup
         with:
           python_version: ${{ matrix.python-version }}
-          extras: '-E all'
+          extras: "-E all"
 
       - name: Test full
         env:
@@ -64,7 +64,8 @@ jobs:
           COGNITE_PROJECT: python-sdk-test
           COGNITE_BASE_URL: https://greenfield.cognitedata.com
           COGNITE_CLIENT_NAME: python-sdk-integration-tests
-        run: pytest tests --durations=10 --cov --cov-report term --cov-report xml:coverage.xml -n8 --dist loadscope --reruns 2 --maxfail 20
+        # Testpaths are defined in the pytest.ini file:
+        run: pytest --durations=10 --cov --cov-report term --cov-report xml:coverage.xml -n8 --dist loadscope --reruns 2 --maxfail 20
 
       - uses: codecov/codecov-action@v4
         if: matrix.os == 'windows-latest' && matrix.python-version == '3.8'

diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
@@ -25,7 +25,8 @@ jobs:
           COGNITE_PROJECT: python-sdk-test
           COGNITE_BASE_URL: https://greenfield.cognitedata.com
           COGNITE_CLIENT_NAME: python-sdk-integration-tests
-        run: pytest tests --durations=10 --cov --cov-report term --cov-report xml:coverage.xml -n8 --dist loadscope --reruns 2 --maxfail 20
+        # Testpaths are defined in the pytest.ini file:
+        run: pytest --durations=10 --cov --cov-report term --cov-report xml:coverage.xml -n8 --dist loadscope --reruns 2 --maxfail 20
 
       - uses: codecov/codecov-action@v4
         with:

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -1,7 +1,7 @@
 ---
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.6.2
+    rev: v0.6.3
     hooks:
       - id: ruff
         args:

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -17,18 +17,60 @@ Changes are grouped as follows
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
 
-## [7.55.2] - 2024-08-29
+## [7.58.6] - 2024-09-05
+### Fixed
+- Data modeling convenience filter `SpaceFilter` now allows listing of global nodes by using `equals`
+  (when a single space is requested (requirement)). This also affects the `space` parameter to e.g.
+  `client.data_modeling.instances.list(...)`
+
+## [7.58.5] - 2024-09-04
 ### Added
-- [Feature Preview - beta] Support for `client.hosted_extractors.sources`.
+- Data modeling filters now support properties that are lists.
+### Fixed
+- Read-only properties on CogniteAssetApply (root, path and last_updated_time) are now removed.
+
+## [7.58.4] - 2024-09-03
+### Fixed
+- The deserialization `datetime` properties in `TypedNode`/`TypedEdge` now correctly handles truncated milliseconds.
+
+## [7.58.3] - 2024-09-03
+### Fixed
+- The parameter `query` is now optional in `client.data_modeling.instances.search(...)`.
+
+## [7.58.2] - 2024-09-03
+### Added
+- [Feature Preview - alpha] Support for `client.hosted_extractors.sources`.
+
+## [7.58.1] - 2024-09-03
+### Fixed
+- [Feature Preview - beta] data workflows: `workflowExecutionId` in `cognite.client.data_classes.workflows.WorkflowTriggerRun`
+ can be null or missing, as according to the API spec.
+
+## [7.58.0] - 2024-09-03
+### Added
+- Data Workflows: add support for `SubworkflowReferenceParameters` subworkflow task type. Allowing embedding other workflows into a workflow.
+
+## [7.57.0] - 2024-09-03
+### Added
+- Add a `load` method to CogniteClient, ClientConfig, and CredenitalProvider (and all it's subclasses).
+- Add `apply_settings` method to `global_config` to pass in a dict of settings
+
+## [7.56.0] - 2024-09-02
+### Added
+- Support for referencing files by instance id when running diagrams.detect
+
+## [7.55.2] - 2024-08-29
+### Fixed
+- Turn workflow_orchestration into data_workflows and add trigger doc, fix attribute names in data classes
 
 ## [7.55.1] - 2024-08-29
 ### Fixed
-- Missing exports for workflow triggers 
+- Missing exports for workflow triggers
 
 ## [7.55.0] - 2024-08-23
 ### Added
 - Support for creating a session using a one-shot token in the `client.iam.session.create` method.
-- Parameter `nonce` to the `client.functions.call()` and `client.workflow.executions.run()` methods to allow passing 
+- Parameter `nonce` to the `client.functions.call()` and `client.workflow.executions.run()` methods to allow passing
   a custom nonce instead of letting the SDK generate it from your current credentials.
 
 ## [7.54.19] - 2024-08-23

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -95,6 +95,16 @@ To speed up test runs pass the following arguments (this will parallelize across
 pytest -n4 --dist loadscope tests
 ```
 
+#### Unit Tests for Examples in Documentation
+
+For code examples defined in *docstrings* the doctest library is used and docstring tests are defined in `tests/tests_unit/test_docstring_examples.py`. Some docstring code examples may require patching which should be done here.
+
+For any code examples written directly in `docs/source` we are using the [sphinx doctest extension](https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html) with pytest. See the `docs/source/quickstart.rst` for an example of a unit test that is setup to use some fixtures defined through pytest (`docs/source/conftest.py`). To run all the tests defined in docs run:
+
+```
+pytest docs
+```
+
 ### Documentation
 
 Build html files of documentation locally by running

diff --git a/cognite/client/_api/data_modeling/instances.py b/cognite/client/_api/data_modeling/instances.py
@@ -973,7 +973,8 @@ def apply(
     def search(
         self,
         view: ViewId,
-        query: str,
+        query: str | None = None,
+        *,
         instance_type: Literal["node"] = "node",
         properties: list[str] | None = None,
         target_units: list[TargetUnit] | None = None,
@@ -988,7 +989,8 @@ def search(
     def search(
         self,
         view: ViewId,
-        query: str,
+        query: str | None = None,
+        *,
         instance_type: Literal["edge"],
         properties: list[str] | None = None,
         target_units: list[TargetUnit] | None = None,
@@ -1003,7 +1005,8 @@ def search(
     def search(
         self,
         view: ViewId,
-        query: str,
+        query: str | None = None,
+        *,
         instance_type: type[T_Node],
         properties: list[str] | None = None,
         target_units: list[TargetUnit] | None = None,
@@ -1018,7 +1021,8 @@ def search(
     def search(
         self,
         view: ViewId,
-        query: str,
+        query: str | None = None,
+        *,
         instance_type: type[T_Edge],
         properties: list[str] | None = None,
         target_units: list[TargetUnit] | None = None,
@@ -1032,7 +1036,7 @@ def search(
     def search(
         self,
         view: ViewId,
-        query: str,
+        query: str | None = None,
         instance_type: Literal["node", "edge"] | type[T_Node] | type[T_Edge] = "node",
         properties: list[str] | None = None,
         target_units: list[TargetUnit] | None = None,
@@ -1046,7 +1050,7 @@ def search(
 
         Args:
             view (ViewId): View to search in.
-            query (str): Query string that will be parsed and used for search.
+            query (str | None): Query string that will be parsed and used for search.
             instance_type (Literal["node", "edge"] | type[T_Node] | type[T_Edge]): Whether to search for nodes or edges.
             properties (list[str] | None): Optional array of properties you want to search through. If you do not specify one or more properties, the service will search all text fields within the view.
             target_units (list[TargetUnit] | None): Properties to convert to another unit. The API can only convert to another unit if a unit has been defined as part of the type on the underlying container being queried.
@@ -1103,7 +1107,9 @@ def search(
         else:
             raise ValueError(f"Invalid instance type: {instance_type}")
 
-        body = {"view": view.dump(camel_case=True), "query": query, "instanceType": instance_type_str, "limit": limit}
+        body: dict[str, Any] = {"view": view.dump(camel_case=True), "instanceType": instance_type_str, "limit": limit}
+        if query:
+            body["query"] = query
         if properties:
             body["properties"] = properties
         if include_typing:

diff --git a/cognite/client/_api/diagrams.py b/cognite/client/_api/diagrams.py
@@ -15,6 +15,7 @@
     FileReference,
     T_ContextualizationJob,
 )
+from cognite.client.data_classes.data_modeling import NodeId
 from cognite.client.exceptions import CogniteAPIError, CogniteMissingClientError
 from cognite.client.utils._experimental import FeaturePreviewWarning
 from cognite.client.utils._text import to_camel_case
@@ -92,24 +93,37 @@ def _list_from_instance_or_list(
     def _process_file_ids(
         ids: Sequence[int] | int | None,
         external_ids: SequenceNotStr[str] | str | None,
+        instance_ids: Sequence[NodeId] | NodeId | None,
         file_references: Sequence[FileReference] | FileReference | None,
-    ) -> list[dict[str, int | str | dict[str, int]] | dict[str, str] | dict[str, int]]:
+    ) -> list[
+        dict[str, int | str | dict[str, str] | dict[str, int]]
+        | dict[str, dict[str, str]]
+        | dict[str, str]
+        | dict[str, int]
+    ]:
         ids = DiagramsAPI._list_from_instance_or_list(ids, int, "ids must be int or list of int")
         external_ids = cast(
             SequenceNotStr[str],
             DiagramsAPI._list_from_instance_or_list(external_ids, str, "external_ids must be str or list of str"),
         )
+        instance_ids = DiagramsAPI._list_from_instance_or_list(
+            instance_ids, NodeId, "instance_ids must be NodeId or list of NodeId"
+        )
         file_references = DiagramsAPI._list_from_instance_or_list(
             file_references, FileReference, "file_references must be FileReference or list of FileReference"
         )
         # Handle empty lists
-        if not (external_ids or ids or file_references):
+        if not (external_ids or ids or instance_ids or file_references):
             raise ValueError("No ids, external ids or file references specified")
 
         id_objs = [{"fileId": id} for id in ids]
         external_id_objs = [{"fileExternalId": external_id} for external_id in external_ids]
+        instance_id_objs = [
+            {"fileInstanceId": instance_id.dump(camel_case=True, include_instance_type=False)}
+            for instance_id in instance_ids
+        ]
         file_reference_objects = [file_reference.to_api_item() for file_reference in file_references]
-        return [*id_objs, *external_id_objs, *file_reference_objects]
+        return [*id_objs, *external_id_objs, *instance_id_objs, *file_reference_objects]
 
     @overload
     def detect(
@@ -120,6 +134,7 @@ def detect(
         min_tokens: int = 2,
         file_ids: int | Sequence[int] | None = None,
         file_external_ids: str | SequenceNotStr[str] | None = None,
+        file_instance_ids: NodeId | Sequence[NodeId] | None = None,
         file_references: list[FileReference] | FileReference | None = None,
         pattern_mode: bool = False,
         configuration: dict[str, Any] | None = None,
@@ -136,6 +151,7 @@ def detect(
         min_tokens: int = 2,
         file_ids: int | Sequence[int] | None = None,
         file_external_ids: str | SequenceNotStr[str] | None = None,
+        file_instance_ids: NodeId | Sequence[NodeId] | None = None,
         file_references: list[FileReference] | FileReference | None = None,
         pattern_mode: bool = False,
         configuration: DiagramDetectConfig | dict[str, Any] | None = None,
@@ -152,6 +168,7 @@ def detect(
         min_tokens: int = 2,
         file_ids: int | Sequence[int] | None = None,
         file_external_ids: str | SequenceNotStr[str] | None = None,
+        file_instance_ids: NodeId | Sequence[NodeId] | None = None,
         file_references: list[FileReference] | FileReference | None = None,
         pattern_mode: bool = False,
         configuration: DiagramDetectConfig | dict[str, Any] | None = None,
@@ -165,6 +182,7 @@ def detect(
         min_tokens: int = 2,
         file_ids: int | Sequence[int] | None = None,
         file_external_ids: str | SequenceNotStr[str] | None = None,
+        file_instance_ids: NodeId | Sequence[NodeId] | None = None,
         file_references: list[FileReference] | FileReference | None = None,
         pattern_mode: bool | None = None,
         configuration: DiagramDetectConfig | dict[str, Any] | None = None,
@@ -184,7 +202,8 @@ def detect(
             min_tokens (int): Minimal number of tokens a match must be based on
             file_ids (int | Sequence[int] | None): ID of the files, should already be uploaded in the same tenant.
             file_external_ids (str | SequenceNotStr[str] | None): File external ids, alternative to file_ids and file_references.
-            file_references (list[FileReference] | FileReference | None): File references (id or external_id), and first_page and last_page to specify page ranges per file. Each reference can specify up to 50 pages. Providing a page range will also make the page count of the document a part of the response.
+            file_instance_ids (NodeId | Sequence[NodeId] | None): Files to detect in, specified by instance id.
+            file_references (list[FileReference] | FileReference | None): File references (id, external_id or instance_id), and first_page and last_page to specify page ranges per file. Each reference can specify up to 50 pages. Providing a page range will also make the page count of the document a part of the response.
             pattern_mode (bool | None): If True, entities must be provided with a sample field. This enables detecting tags that are similar to the sample, but not necessarily identical. Defaults to None.
             configuration (DiagramDetectConfig | dict[str, Any] | None): Additional configuration for the detect algorithm. See `DiagramDetectConfig` class documentation and `beta API docs <https://api-docs.cognite.com/20230101-beta/tag/Engineering-diagrams/operation/diagramDetect/#!path=configuration&t=request>`_.
             multiple_jobs (bool): Enables you to publish multiple jobs. If True the method returns a tuple of DetectJobBundle and list of potentially unposted files. If False it will return a single DiagramDetectResults. Defaults to False.
@@ -258,7 +277,7 @@ def detect(
 
             Check the documentation for `DiagramDetectConfig` for more information on the available options.
         """
-        items = self._process_file_ids(file_ids, file_external_ids, file_references)
+        items = self._process_file_ids(file_ids, file_external_ids, file_instance_ids, file_references)
         entities = [
             entity.dump(camel_case=True) if isinstance(entity, CogniteResource) else entity for entity in entities
         ]