Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extra_loggers is not used to log metrics or hyperparameters #12046

Open
chajath opened this issue Feb 4, 2025 · 0 comments
Open

extra_loggers is not used to log metrics or hyperparameters #12046

chajath opened this issue Feb 4, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@chajath
Copy link

chajath commented Feb 4, 2025

Describe the bug

When I declare a dummy json stdout logger:

from lightning.fabric.loggers import logger
import lightning.pytorch.loggers.logger
from lightning.pytorch.utilities import rank_zero

import simplejson as json


rank_zero_only = rank_zero.rank_zero_only
Logger = lightning.pytorch.loggers.logger.Logger
rank_zero_experiment = logger.rank_zero_experiment


class MyLogger(Logger):
  """Logger override example."""

  @property
  def name(self):
    return "MyLogger"

  @property
  @rank_zero_experiment
  def experiment(self):
    # Return the experiment object associated with this logger.
    pass

  @property
  def version(self):
    # Return the experiment version, int or str.
    return "0.1"

  @rank_zero_only
  def log_hyperparams(self, params):
    print()
    print(json.dumps(params, ignore_nan=True))

  @rank_zero_only
  def log_metrics(self, metrics, step):
    print()
    print(json.dumps({metrics: metrics, step: step}, ignore_nan=True))

And wire them up to the NeMoLogger:

logger = nl.NeMoLogger(ckpt=ckpt, extra_loggers=[MyLogger()])

...
  llm.pretrain(
...
      log=logger,
  )

I don't find json dumps anywhere in the stdout log, suggesting these log methods are not called.

Steps/Code to reproduce bug
Follow the Logger example in the above section and see if the log outputs are printed.

Expected behavior
Called are dumped as jsonl in stdout.

Environment overview (please complete the following information)

  • Environment location: GKE
  • Method of NeMo install: docker container (tag 24.12)

Environment details
NeMo container 24.12

@chajath chajath added the bug Something isn't working label Feb 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant