Add service subscription configuration

[brothercorvo]

Description

Currently, a digitalpy_service is hardcoded to subscribe to two topics,

1. the command topic which allows the service manager to send command
2. the response topic which is the generic topic to which the integration manager publishes all responses.

This approach is problematic for two reasons:

1. Overhead and inefficiency: All services receive all responses, leading to unnecessary processing and potential performance degradation.
2. Complexity in integration manager: The logic in the integration manager to decide whether to publish a response or not becomes overly complex and difficult to maintain.

Solutions

To address these issues, I propose modifying the service configuration to include a list of specific topics to which the service should subscribe. This allows for more granular control and avoids the unnecessary overhead of receiving all responses. An example service configuration might look like this:

[dp_helloworld.simple_tcp]
__class = digitalpy.core.service_management.domain.model.service_configuration.ServiceConfiguration
status = STOPPED
name = SimpleTCPService
port = 8443
host = 0.0.0.0
protocol = TCPNetwork
flows = [Flow1, Flow2, Flow3] # this is the new attribute

Each flow is assumed to end with a standard "done" topic, which is dynamically subscribed to by the service. For example:

[Flow1]
?Flow1?Push
Subject?Flow1?DoWork
??Done # this is the generic final topic

[Flow2]
?Flow2?Push
Subject?Flow2?DoDifferentWork
??Done

Benefits

• Improved performance: Services will only receive the responses they are interested in, reducing unnecessary message processing.
• Simpler logic in the integration manager: By having services subscribe only to specific topics, the logic in the integration manager becomes simpler and more maintainable.
• Easier scalability: Adding new flows or services becomes easier, as topics can be added dynamically in the configuration without needing to modify the core service or integration manager logic.

Tasks

Beta Give feedback

1. Implement default service subscription to configured flow
2. Implement API endpoint to add topic subscription to service
3. Implement API endpoint to remove topic subscription from service
4. Update generator to add Done topic by default

Once this is done this feature can be marked complete

[brothercorvo]

each service MUST be able to:

• subscribe and subscribe to ANY topic
• retrieve a list of subscribed topics

while this list can be initially part of the service configuration, when the service is installed it should be moved to a DB

if the name of the flow is equal to the name of the topic a service can simply react to action 'done', in alternative the topic name should be added to the key

[naman108]

Ok the best way I can think to solve this is by a bit of a complex topic implementation. In essence it will follow the sequence:

1. Receive request in API Service
2. Send request to subject
3. Send request from subject to integration manager
4. Send request from integration manager to service management core service
5. Send command from service management core service to digitalpy service with a config in the format COMMAND_<service_id>_<command>, the service will be subscribed to the topic COMMAND_<service_id>
6. The service will process the command and if there is a response it will send the response with the same config name, the same id and the action done to the integration manager.
7. The integration manager will publish the response to the service management core service
8. The service management core service will send this response to the command response handler
9. Command response handler will push the response with the same action to the integration manager
10. The integration manager will publish the response
11. The API Service will send the response