[BUG] LightGBM MLFlow autolog not logging metrics

[tom-ph]

SynapseML version

1.0.1

System information

• Language version (e.g. python 3.8, scala 2.12): python 3.10.12, scala 2.12
• Spark Version (e.g. 3.2.3): 3.4.1
• Spark Platform (e.g. Synapse, Databricks): Databricks

Describe the problem

After setup of
spark.mlflow.pysparkml.autolog.logModelAllowlistFile /path/to/allowlist.txt
and running
mlflow.pyspark.ml.autolog()
model metrics are not logged, only parameters and artifacts.

Code to reproduce issue

import mlflow

mlflow.pyspark.ml.autolog()

train_data = spark.table("schema.train_data")

params = {
    "objective": "mse", 
    "learningRate": 0.03,
    "numIterations": 1000,
    "seed": 42,
    "earlyStoppingRound": 50,
    "validationIndicatorCol": "is_validation",
}

model = LightGBMRegressor(**params).fit(train_data)

Other info / logs

No response

What component(s) does this bug affect?

• area/cognitive: Cognitive project
• area/core: Core project
• area/deep-learning: DeepLearning project
• area/lightgbm: Lightgbm project
• area/opencv: Opencv project
• area/vw: VW project
• area/website: Website
• area/build: Project build system
• area/notebooks: Samples under notebooks folder
• area/docker: Docker usage
• area/models: models related issue

What language(s) does this bug affect?

• language/scala: Scala source code
• language/python: Pyspark APIs
• language/r: R APIs
• language/csharp: .NET APIs
• language/new: Proposals for new client languages

What integration(s) does this bug affect?

• integrations/synapse: Azure Synapse integrations
• integrations/azureml: Azure ML integrations
• integrations/databricks: Databricks integrations