Secrets cannot be pushed to the vault (external) in the data transfer request

[FaheemBhatti]

Description

We have configured a central vault that is integrated with multiple TX-EDC instances. They successfully read and write secrets in the provider-case. However, in a transfer scenario, when the consumer attempts to push the relevant generated secret (EDR-tokens) to the vault, it fails with the logs below [1].

Even when roles are reversed (i.e., the provider is made the consumer), the same issue persists. The producer can push secrets, but the consumer is unable to do so.

This suggests that either a configuration setting or an issue in the EDC implementation is preventing the consumer from writing secrets to the vault. Extensive testing confirms that our vault configuration does not contain any errors or misconfigurations, indicating the issue lies within the EDC connector behavior or setup.

Steps to Reproduce

1. Configure the central vault and verify that it is recognized by the EDC connector [2].
2. Set up a provider connector and ensure it can push secrets to the vault.
3. Initiate a transfer request where the consumer needs to push a generated secret to the vault.
4. Observe that the consumer fails to push the secret, while the provider succeeds.
5. Swap roles (make the provider a consumer) and repeat the process.
6. Note that the issue persists regardless of role reversal.

Expected Behavior

The provider and consumer connectors should be able to push secrets to the vault as required.

Actual Behavior

The consumer cannot push secrets to the vault, while the provider can do so without issues.
Role reversal does not resolve the problem.

Impact

The issue prevents the successful completion of transfer requests, potentially blocking further development.
Security risks may arise if the required secrets cannot be stored appropriately.

Possible Causes

A configuration setting within the EDC connector may restrict the consumer’s ability to push secrets.
An underlying bug in the EDC implementation affects secret storage for consumer connectors.

Attachments

[1]

DEBUG 2025-02-26T09:59:37.361257198 TransferProcess 0656575f-bc80-4bd9-bf2b-35ffba6b2556 is now in state STARTED
SEVERE 2025-02-26T09:59:37.496805894 [EDR Receiver] Failed to process event TransferProcessStarted: Failed to set secret with status 404

[2]
Vault Configuration

vault: injector: enabled: true server: dev: enabled: false devRootToken: "" postStart: # must be set externally! hashicorp: url: "${vaultUrl}" token: "" timeout: 30 healthCheck: enabled: false standbyOk: false paths: secret: /v1/secret/data/ssh-internal-test-entity-1 health: /v1/sys/health

[3]
Provider Logs:
DEBUG 2025-02-27T09:19:12.522956157 Policy Definition created trial-usage-policy-v1 DEBUG 2025-02-27T09:19:24.563750522 [DataPlaneSelectorManagerImpl] DataPlaneInstance ssh-internal-test-entity-1-tractusx-connector-dataplane is now in state AVAILABLE DEBUG 2025-02-27T09:19:44.014331975 Policy Definition created trial-access-policy-v1 DEBUG 2025-02-27T09:20:25.026951524 [DataPlaneSelectorManagerImpl] DataPlaneInstance ssh-internal-test-entity-1-tractusx-connector-dataplane is now in state AVAILABLE DEBUG 2025-02-27T09:21:12.078014221 DSP: Incoming CatalogRequestMessage for class org.eclipse.edc.connector.controlplane.catalog.spi.Catalog process DEBUG 2025-02-27T09:21:12.080009544 DIM Token expired, need to refresh. DEBUG 2025-02-27T09:21:25.495155716 [DataPlaneSelectorManagerImpl] DataPlaneInstance ssh-internal-test-entity-1-tractusx-connector-dataplane is now in state AVAILABLE DEBUG 2025-02-27T09:22:25.927895356 [DataPlaneSelectorManagerImpl] DataPlaneInstance ssh-internal-test-entity-1-tractusx-connector-dataplane is now in state AVAILABLE DEBUG 2025-02-27T09:22:29.518181665 DSP: Incoming ContractRequestMessage for class org.eclipse.edc.connector.controlplane.contract.spi.types.negotiation.ContractNegotiation process DEBUG 2025-02-27T09:22:29.846457222 [PROVIDER] ContractNegotiation 24041fab-3c07-47fd-98c4-9d2230a3d8e7 is now in state REQUESTED. DEBUG 2025-02-27T09:22:30.387115126 [ProviderContractNegotiationManagerImpl] ContractNegotiation 24041fab-3c07-47fd-98c4-9d2230a3d8e7 is now in state AGREEING DEBUG 2025-02-27T09:22:30.395427177 ContractNegotiation: ID 24041fab-3c07-47fd-98c4-9d2230a3d8e7. [Provider] send agreement DEBUG 2025-02-27T09:22:32.595735755 ContractNegotiation: ID 24041fab-3c07-47fd-98c4-9d2230a3d8e7. [Provider] send agreement DEBUG 2025-02-27T09:22:32.601472117 [ProviderContractNegotiationManagerImpl] ContractNegotiation 24041fab-3c07-47fd-98c4-9d2230a3d8e7 is now in state AGREED DEBUG 2025-02-27T09:22:33.990168902 DSP: Incoming ContractAgreementVerificationMessage for class org.eclipse.edc.connector.controlplane.contract.spi.types.negotiation.ContractNegotiation process: 24041fab-3c07-47fd-98c4-9d2230a3d8e7 DEBUG 2025-02-27T09:22:34.206582332 [PROVIDER] ContractNegotiation 24041fab-3c07-47fd-98c4-9d2230a3d8e7 is now in state VERIFIED. DEBUG 2025-02-27T09:22:34.529865523 [ProviderContractNegotiationManagerImpl] ContractNegotiation 24041fab-3c07-47fd-98c4-9d2230a3d8e7 is now in state FINALIZING DEBUG 2025-02-27T09:22:34.533286215 ContractNegotiation: ID 24041fab-3c07-47fd-98c4-9d2230a3d8e7. [Provider] send finalization DEBUG 2025-02-27T09:22:35.200038664 ContractNegotiation: ID 24041fab-3c07-47fd-98c4-9d2230a3d8e7. [Provider] send finalization DEBUG 2025-02-27T09:22:35.204445312 [ProviderContractNegotiationManagerImpl] ContractNegotiation 24041fab-3c07-47fd-98c4-9d2230a3d8e7 is now in state FINALIZED DEBUG 2025-02-27T09:22:35.800261802 DSP: Incoming TransferRequestMessage for class org.eclipse.edc.connector.controlplane.transfer.spi.types.TransferProcess process DEBUG 2025-02-27T09:22:36.021637359 TransferProcess 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba is now in state INITIAL DEBUG 2025-02-27T09:22:36.629228066 [TransferProcessManagerImpl] TransferProcess 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba is now in state PROVISIONING DEBUG 2025-02-27T09:22:36.632513557 TransferProcess: ID 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba. Provisioning DEBUG 2025-02-27T09:22:36.635333955 [TransferProcessManagerImpl] TransferProcess 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba is now in state PROVISIONED DEBUG 2025-02-27T09:22:36.640538378 [TransferProcessManagerImpl] TransferProcess 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba is now in state STARTING DEBUG 2025-02-27T09:22:36.644220354 TransferProcess: ID 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba. Initiate data flow DEBUG 2025-02-27T09:22:37.124132025 TransferProcess: ID 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba. send transfer start to https://ssh-internal-test-entity-2.c-27d7c36.kyma.ondemand.com/api/v1/dsp DEBUG 2025-02-27T09:22:37.721826342 TransferProcess: ID 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba. send transfer start to https://ssh-internal-test-entity-2.c-27d7c36.kyma.ondemand.com/api/v1/dsp DEBUG 2025-02-27T09:22:37.726254402 [TransferProcessManagerImpl] TransferProcess 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba is now in state STARTED DEBUG 2025-02-27T09:22:37.729009535 [PolicyMonitorManagerImpl] PolicyMonitorEntry 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba is now in state STARTED DEBUG 2025-02-27T09:23:26.421139588 [DataPlaneSelectorManagerImpl] DataPlaneInstance ssh-internal-test-entity-1-tractusx-connector-dataplane is now in state AVAILABLE

Consumer Logs
DEBUG 2025-02-27T09:22:28.502433365 [ConsumerContractNegotiationManagerImpl] ContractNegotiation 0884e9b8-2324-485e-a1e1-83be6b018137 is now in state INITIAL DEBUG 2025-02-27T09:22:29.116193541 [ConsumerContractNegotiationManagerImpl] ContractNegotiation 0884e9b8-2324-485e-a1e1-83be6b018137 is now in state REQUESTING DEBUG 2025-02-27T09:22:29.118479353 ContractNegotiation: ID 0884e9b8-2324-485e-a1e1-83be6b018137. [Consumer] send request DEBUG 2025-02-27T09:22:29.853328119 ContractNegotiation: ID 0884e9b8-2324-485e-a1e1-83be6b018137. [Consumer] send request DEBUG 2025-02-27T09:22:29.857236556 [ConsumerContractNegotiationManagerImpl] ContractNegotiation 0884e9b8-2324-485e-a1e1-83be6b018137 is now in state REQUESTED DEBUG 2025-02-27T09:22:31.935490343 DSP: Incoming ContractAgreementMessage for class org.eclipse.edc.connector.controlplane.contract.spi.types.negotiation.ContractNegotiation process: 0884e9b8-2324-485e-a1e1-83be6b018137 DEBUG 2025-02-27T09:22:32.589113221 [CONSUMER] ContractNegotiation 0884e9b8-2324-485e-a1e1-83be6b018137 is now in state AGREED. DEBUG 2025-02-27T09:22:33.526501566 [ConsumerContractNegotiationManagerImpl] ContractNegotiation 0884e9b8-2324-485e-a1e1-83be6b018137 is now in state VERIFYING DEBUG 2025-02-27T09:22:33.528853606 ContractNegotiation: ID 0884e9b8-2324-485e-a1e1-83be6b018137. [consumer] send verification DEBUG 2025-02-27T09:22:34.209714199 ContractNegotiation: ID 0884e9b8-2324-485e-a1e1-83be6b018137. [consumer] send verification DEBUG 2025-02-27T09:22:34.213095443 [ConsumerContractNegotiationManagerImpl] ContractNegotiation 0884e9b8-2324-485e-a1e1-83be6b018137 is now in state VERIFIED DEBUG 2025-02-27T09:22:34.917026506 DSP: Incoming ContractNegotiationEventMessage for class org.eclipse.edc.connector.controlplane.contract.spi.types.negotiation.ContractNegotiation process: 0884e9b8-2324-485e-a1e1-83be6b018137 DEBUG 2025-02-27T09:22:35.196518122 [CONSUMER] ContractNegotiation 0884e9b8-2324-485e-a1e1-83be6b018137 is now in state FINALIZED. DEBUG 2025-02-27T09:22:35.198020827 [TransferProcessManagerImpl] TransferProcess 33e9af61-db7b-4e07-b6eb-185f889a923e is now in state INITIAL DEBUG 2025-02-27T09:22:35.198266064 Transfer with id TransferProcess{id='33e9af61-db7b-4e07-b6eb-185f889a923e', state=INITIAL, stateTimestamp=2025-02-27T09:22:35.197Z} initiated DEBUG 2025-02-27T09:22:35.455266474 [TransferProcessManagerImpl] TransferProcess 33e9af61-db7b-4e07-b6eb-185f889a923e is now in state PROVISIONING DEBUG 2025-02-27T09:22:35.459572171 TransferProcess: ID 33e9af61-db7b-4e07-b6eb-185f889a923e. Provisioning DEBUG 2025-02-27T09:22:35.462380725 [TransferProcessManagerImpl] TransferProcess 33e9af61-db7b-4e07-b6eb-185f889a923e is now in state PROVISIONED DEBUG 2025-02-27T09:22:35.467384535 [TransferProcessManagerImpl] TransferProcess 33e9af61-db7b-4e07-b6eb-185f889a923e is now in state REQUESTING DEBUG 2025-02-27T09:22:35.470194244 TransferProcess: ID 33e9af61-db7b-4e07-b6eb-185f889a923e. send transfer request to https://ssh-internal-test-entity-1.c-27d7c36.kyma.ondemand.com/api/v1/dsp DEBUG 2025-02-27T09:22:36.027359127 TransferProcess: ID 33e9af61-db7b-4e07-b6eb-185f889a923e. send transfer request to https://ssh-internal-test-entity-1.c-27d7c36.kyma.ondemand.com/api/v1/dsp DEBUG 2025-02-27T09:22:36.029982617 [TransferProcessManagerImpl] TransferProcess 33e9af61-db7b-4e07-b6eb-185f889a923e is now in state REQUESTED DEBUG 2025-02-27T09:22:37.483675107 DSP: Incoming TransferStartMessage for class org.eclipse.edc.connector.controlplane.transfer.spi.types.TransferProcess process: 33e9af61-db7b-4e07-b6eb-185f889a923e DEBUG 2025-02-27T09:22:37.719637502 TransferProcess 33e9af61-db7b-4e07-b6eb-185f889a923e is now in state STARTED SEVERE 2025-02-27T09:22:37.72420183 [EDR Receiver] Failed to process event TransferProcessStarted: Failed to set secret with status 404 DEBUG 2025-02-27T09:23:09.922612763 [DataPlaneSelectorManagerImpl] DataPlaneInstance ssh-internal-test-entity-2-tractusx-connector-dataplane is now in state AVAILABLE

[4]
Also, a new secret is added to the vault in the provider's secret data space