[ProofPoint TAP] implement application #3343

flavienSindou · 2025-01-28T16:52:04Z

Proposed changes

Create connector application using adapter to get data from the Proofpoint V2 API client and process via processor to generate report

Related issues

[Proofpoint TAP] Developing a connector #1538

Checklist

I consider the submitted work as finished
I tested the code for its functionality using different use cases
I added/update the relevant documentation (either on github or on notion)
Where necessary I refactored code to improve the overall quality

Further comments

N/A

external-import/proofpoint-tap/proofpoint_tap/domain/models/octi/common.py

external-import/proofpoint-tap/proofpoint_tap/client_api/v2/threat.py

external-import/proofpoint-tap/proofpoint_tap/client_api/v2/campaign.py

external-import/proofpoint-tap/proofpoint_tap/client_api/common.py

flavienSindou · 2025-01-28T16:56:11Z

external-import/proofpoint-tap/proofpoint_tap/adapters/campaign.py

Core of the PR

external-import/proofpoint-tap/pyproject.toml

external-import/proofpoint-tap/tests/test_client_api/test_common.py

external-import/proofpoint-tap/tests/test_client_api/test_v2/test_campaign.py

external-import/proofpoint-tap/tests/test_client_api/test_v2/test_forensics.py

external-import/proofpoint-tap/tests/test_client_api/test_v2/test_threat.py

Powlinett

Some typos and missing decorators prevent the connector to launch (see comments).

I fixed these issues locally and was able to:

make the tests pass ⚠️ (some counters need to be incremented by one since the TLP markings have been added to the bundle)
run connector locally with a .config.yml file ✅
run connector locally with a .env file ✅
build the docker image with a .env file ✅
see no errors during ingestion ✅

Despite the complexity of the product, the code is pretty straightforward and each file's responsibility well defined 👏👏👏

Powlinett · 2025-02-03T10:39:57Z

external-import/proofpoint-tap/proofpoint_tap/ports/config.py

+    @abstractmethod
+    def _name(self) -> str: ...
+
+    @_make_error_handler("Unable to retrieve connector name in config")


missing @property decorator:

Powlinett · 2025-02-03T10:40:33Z

external-import/proofpoint-tap/proofpoint_tap/ports/config.py

+    @abstractmethod
+    def _scope(self) -> str: ...
+
+    @_make_error_handler("Unable to retrieve connector scope in config")


missing @property decorator:

Powlinett · 2025-02-03T10:43:09Z

external-import/proofpoint-tap/proofpoint_tap/ports/config.py

+
+    @property
+    @_make_error_handler("Unable to retrieve OpenCTI Token in config")
+    def token(self) -> SecretStr:


I couldn't manage to launch the connector with SecretStr type here ☹️

I had to change it for str:

Thank for yu remark !
In my opinion SecretStr is used to remind the developper he/she is manipulating a secret and not a simple string and I should keep it this way here.
To fix the issue you highlighted, I've altered the to_dict method to handle this case if it is ok for you :)

Powlinett · 2025-02-03T13:52:20Z

external-import/proofpoint-tap/tests/test_domain/test_use_cases/test_ingest_incident.py

+        == 1
+    )
+    ## Total 8 entities
+    assert len(entities) == 8  # noqa: S101


need to add +1 for the TLP marking entity

Powlinett · 2025-02-03T13:52:41Z

external-import/proofpoint-tap/tests/test_domain/test_use_cases/test_ingest_incident.py

+        == 1
+    )
+    ## Total 22 entities
+    assert len(entities) == 22  # noqa: S101


need to add +1 for the TLP marking entity

Powlinett · 2025-02-03T13:59:17Z

external-import/proofpoint-tap/.env.sample

+CONNECTOR_NAME=ProofPointTAP
+CONNECTOR_SCOPE=report
+CONNECTOR_DURATION_PERIOD=PT12H
+CONNECTOR_LOG_LEVEL=warning


Should be replaced by warn, otherwise a ValueError is raised.

Powlinett · 2025-02-03T14:04:06Z

external-import/proofpoint-tap/.env.sample

+TAP_MARKING_DEFINITION=white
+TAP_EXPORT_CAMPAIGNS=True
+TAP_EXPORT_EVENTS=True
+TAP_EVENTS_TYPES=issues


typo: in config adapter, the expected variable is TAP_EVENTS_TYPES (plural)

Powlinett · 2025-02-03T14:14:47Z

external-import/proofpoint-tap/app.py

+                self._logger.warning(
+                    "[CONNECTOR] Connector acquisition SINCE parameter overwritten",
+                    {
+                        # "previous": str(self.config.tap.export_since),


comment to remove?

Powlinett · 2025-02-03T14:20:19Z

external-import/proofpoint-tap/app.py

+            self._log_error(f"Data retrieval error: {str(e)}")
+            return False
+
+    def work(self) -> None:


It's a good idea to call it work instead of process 👍

Thank you !
I still call _process the ETL steps though 🤔

That's true 😅 maybe _send or _import/_export could do the trick... but the most important IMHO is to not use process_message method's name anymore, because the method wasn't processing any messages ^^

Powlinett · 2025-02-03T14:24:01Z

external-import/proofpoint-tap/proofpoint_tap/domain/models/octi/observables.py

-                score=self.score,
-                created_by_ref=self.author.id if self.author else None,
-            ),
+            custom_properties=self.custom_properties_to_stix(),
        )

    def to_indicator(


I don't remember if I mentioned it in the previous PR but these are very convenient methods 👍

…1939413125

…1939413684

…1939161808

…1939160846

…1939423039

…1939430391

…1939447426

flavienSindou

Thanks for your remarks and tips @Powlinett !

I think I've fixed the issues you highlighted. Could you have a glance please ?

flavienSindou added 11 commits January 28, 2025 17:28

refacto

4403a5a

reafcato

b28f482

refacto

bc01d90

refacto

e22e74d

feat : add ports

dbf72c4

feat: implement adapters

b1c9dfa

refacto

ddfa56d

refcato

66bd4eb

feat: implement app

527d8f7

doc: readme and examples

5cc082a

feat: add dockerfile

58f509e

flavienSindou requested review from Powlinett, Megafredo and helene-nguyen January 28, 2025 16:52

flavienSindou self-assigned this Jan 28, 2025

fix doc

82ec820

flavienSindou commented Jan 28, 2025

View reviewed changes

flavienSindou added 6 commits January 28, 2025 18:04

fix

c22e429

tests

dd3dd12

tests fix

48e4159

feat: link Observable and indicator with based-on relationship

4d0a0c5

style

7c35c2e

refacto: remove since parameter capability following product team remark

a01b0a4

helene-nguyen linked an issue Jan 30, 2025 that may be closed by this pull request

[Proofpoint TAP] Developing a connector #1538

Closed

flavienSindou added 6 commits January 30, 2025 21:40

feat: add cnofig vaer to handle 2 use cases

06782ae

feat implement ingest incidents use case

6b038be

feat: add events adapter

d7175bc

test: add use case unit test

4b5b895

deps: limit tests requirements

228efc1

doc: update example config file

866c268

flavienSindou added 3 commits January 30, 2025 21:44

feat: add 2nd use case in app

42e70fb

doc: update Readme

0c05445

fix: add tlp to bundle

a15eff5

Powlinett requested changes Feb 3, 2025

View reviewed changes

flavienSindou added 8 commits February 4, 2025 09:42

https://github.com/OpenCTI-Platform/connectors/pull/3343#discussion_r…

d3eded6

…1939413125

https://github.com/OpenCTI-Platform/connectors/pull/3343#discussion_r…

55dd9dd

…1939413684

https://github.com/OpenCTI-Platform/connectors/pull/3343#discussion_r…

b7637c2

…1939161808

https://github.com/OpenCTI-Platform/connectors/pull/3343#discussion_r…

ed43f22

…1939160846

https://github.com/OpenCTI-Platform/connectors/pull/3343#discussion_r…

71312eb

…1939423039

https://github.com/OpenCTI-Platform/connectors/pull/3343#discussion_r…

bbd038e

…1939430391

https://github.com/OpenCTI-Platform/connectors/pull/3343#discussion_r…

5d088d2

…1939447426

chore: style

ea1b801

flavienSindou requested a review from Powlinett February 4, 2025 09:13

flavienSindou commented Feb 4, 2025

View reviewed changes

Powlinett approved these changes Feb 4, 2025

View reviewed changes

flavienSindou merged commit a002ffd into feat/1538-proofpoint-tap-connector Feb 4, 2025
3 of 4 checks passed

flavienSindou deleted the feat/1538-proofpoint-tap-connector-application branch February 4, 2025 12:54

flavienSindou mentioned this pull request Feb 4, 2025

[Proofpoint TAP] Init #3387

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ProofPoint TAP] implement application #3343

[ProofPoint TAP] implement application #3343

flavienSindou commented Jan 28, 2025

flavienSindou Jan 28, 2025

Powlinett left a comment

Powlinett Feb 3, 2025

flavienSindou Feb 4, 2025

Powlinett Feb 3, 2025

flavienSindou Feb 4, 2025

Powlinett Feb 3, 2025

flavienSindou Feb 4, 2025

Powlinett Feb 3, 2025

flavienSindou Feb 4, 2025

Powlinett Feb 3, 2025

flavienSindou Feb 4, 2025

Powlinett Feb 3, 2025

flavienSindou Feb 4, 2025

Powlinett Feb 3, 2025

flavienSindou Feb 4, 2025

Powlinett Feb 3, 2025 •

edited

Loading

flavienSindou Feb 4, 2025

Powlinett Feb 3, 2025

flavienSindou Feb 4, 2025

Powlinett Feb 4, 2025

Powlinett Feb 3, 2025

flavienSindou left a comment

[ProofPoint TAP] implement application #3343

[ProofPoint TAP] implement application #3343

Conversation

flavienSindou commented Jan 28, 2025

Proposed changes

Related issues

Checklist

Further comments

Choose a reason for hiding this comment

Powlinett left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Powlinett Feb 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flavienSindou left a comment

Choose a reason for hiding this comment

Powlinett Feb 3, 2025 •

edited

Loading