Adding support for CUDA_VISIBLE_DEVICES

[lipengfeizju]

• CodeCarbon version: 2.4.2
• Python version: 3.9.19
• Operating System: Rocky Linux release 8.8 (Green Obsidian)

Description

In our university's cluster, our goal is to measure the energy consumption of a deep learning model. The server uses SLURM system and we only get 1 A100 ( out of 8 GPUs). The GPU power measurement from codecarbon is about all 8 GPUs, instead of the GPU we have been allocated.

Techinically speaking, I guess in line 184 of codecarbon/core/gpu.py, it queries all GPUs from nvml instead of focusing on the GPU we are actually using. To get a more accurate measurement, it would be better to only look up the power consumption related to CUDA_VISIBLE_DEVICES.

Similar discussion about this topic can also be found here in pynvml.

So is it possible to add a new feature to support measurements focusing on CUDA_VISIBLE_DEVICES? I think this is important for deep learning applications, since the other non-visiable devices are usually unrelated to the power consumption of the DL applications.

Thank you again for providing the code base for carbon measurement.

[inimaz]

Hello @lipengfeizju! Thanks for using codecarbon.
I didn't know about this var CUDA_VISIBLE_DEVICES. If they end up implementing the function in pynvml it would be useful to use it indeed.

In the meantime, in codecarbon there is a way to filter the GPUs that are tracked if you provide their gpu_id. https://github.com/mlco2/codecarbon/blob/master/codecarbon/emissions_tracker.py#L191
So maybe you can get the id via nvidia-smi of the ones you want to use and do

EmissionsTracker(
             ...
             gpu_ids ="0,3,4"
)

Is this what you need?

[lipengfeizju]

Thanks! That's exactly what I need.

[lipengfeizju]

Sorry to reopen the issue again, is it possible to measure the power of several specific CPU cores? (Maybe just like we do for the GPU ids)

[benoit-cty]

Maybe we could initialize gpu_ids with os.environ("CUDA_VISIBLE_DEVICES") ?

For CPU node could you open another issue and provide the codecarbon debug logs ? Because it's not possible yet but maybe we could imagine a way to do it.

[lipengfeizju]

Thanks!