Opni CPU Inferencing Service

Opni CPU Inferencing Service

Description

The Opni CPU Inferencing Service will infer on logs using just a CPU if the user has removed the GPU from their cluster.

Programming Languages

• Python

Diagram

[to do]

Responsibilities

• Inference on logs using a CPU
• Send over inferred logs to the workload DRAIN service to be put in the cache.

Input and output interfaces

Input

Component	Type	Description
model_update	Nats subject	When a new Deep Learning model has been trained and is ready for inferencing, the CPU inferencing service will receive a message to download the latest model..
opnilog_workload_logs	Nats subject	The CPU inferencing service is subscribed to the opnilog_workload_logs to receive the logs that need to be inferred on.

Output

Component	Type	Description
gpu_service_inference	Nats request/reply subject	When the CPU service receives logs that need to be inferred on, it will submit a request to the training controller service through this Nats subject to determine if the GPU controller can be used.
model_inferenced_workload_logs	Nats subject	Once the CPU inferencing service has inferred on the logs that were a part of the payload, it will publish the results back to the model_inferenced_workload_logs Nats subject which is to be received by the workload DRAIN service.

Restrictions/limitations

Performance issues

• CPU inferencing is not as fast as GPU inferencing.

Test plan

• Unit tests
• Integration tests
• e2e tests
• Manual testing