Fix cleanup leak in pipeline collectors

[zwpaper]

cleanup is shadowed by the later ones, especially the GPU Collector cleanup, which never has a chance to be executed.

this may be related to #215

[nvvfedorov]

Thank you for your contribution. Can you add a unit/integration test or add steps on how the change can be tested?

[zwpaper]

hi @nvvfedorov, as the current code base, it is not easy to add tests for this.

maybe we refactor the NewMetricsPipeline(c *Config) (*MetricsPipeline, func(), error) to

type DCGMCollectorConstructor func([]Counter, *Config, dcgm.Field_Entity_Group) (*DCGMCollector, func(), error)

NewMetricsPipeline(c *Config, NewDCGMCollector DCGMCollectorConstructor) (*MetricsPipeline, func(), error)

so that we can pass a mock DCGMCollectorConstructor into it and test the cleanups

[nvvfedorov]

hi @nvvfedorov, as the current code base, it is not easy to add tests for this.

maybe we refactor the NewMetricsPipeline(c *Config) (*MetricsPipeline, func(), error) to

type DCGMCollectorConstructor func([]Counter, *Config, dcgm.Field_Entity_Group) (*DCGMCollector, func(), error)

NewMetricsPipeline(c *Config, NewDCGMCollector DCGMCollectorConstructor) (*MetricsPipeline, func(), error)

so that we can pass a mock DCGMCollectorConstructor into it and test the cleanups

This makes sense. Let's make the code better.

[zwpaper]

Hi @nvvfedorov, tests added, PTAL

[nvvfedorov]

Please put temporary files into the temporary directory. See my comment.

[zwpaper]

Hi @nvvfedorov, I have updated the code, PTAL

[nvvfedorov]

LGTM

[zwpaper]

Hi @nvvfedorov, I have updated to sign the commits, can you help merge this PR, or do we need to assign another maintainer?