I am trying to log the loss and auc for all 3 of my datasets - train, validation and test. The datamodule defines the 3 loaders and I finally invoke the model as:

trainer.fit(model,datamodule)
trainer.test(model,datamodule)

This ends up creating 2 different log files - 1 with train and validation metrics, and 1 with test metrics. Is there a way to collect these together? Context is that I am using the mlflow logger and would like to see all 3 sets of metrics on the same chart, but because of this behavior, it posts 2 different sets of metrics with the same run id, creating 2 different graphs.

I am trying to log the loss and auc for all 3 of my datasets - train, validation and test. The datamodule defines the 3 loaders and I finally invoke the model as:

trainer.fit(model,datamodule)
trainer.test(model,datamodule)

This ends up creating 2 different log files - 1 with train and validation metrics, and 1 with test metrics. Is there a way to collect these together? Context is that I am using the mlflow logger and would like to see all 3 sets of metrics on the same chart, but because of this behavior, it posts 2 different sets of metrics with the same run id, creating 2 different graphs.

• pytorch
• pytorch-lightning

Share Improve this question Follow edited Mar 12 at 9:41 Matt Pitkin 6,64511 gold badge2525 silver badges4545 bronze badges asked Mar 11 at 19:54 ApurvaApurva 17322 silver badges88 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

It looks like lightning creates a different runid for each invocation. However, if I noticed if I start a run (in my case by logging hyperparameters):

mlflow_logger = MLFlowLogger(
    experiment_name=MLFLOW_EXPERIMENT_NAME,
    tracking_uri=MLFLOW_TRACKING_URI,
    run_name=MLFLOW_RUN_NAME,
)
mlflow_logger.log_hyperparams(hyperparams)

then lightning reuses the same runId for emitting metrics for fit and test stages.