Huggingface trainer has parameter compute_metrics. This function however receives only predictions and labels as its input:

def compute_metric(pred_label):
  pred, label = pred_label

  return my_metric(pred, label)

This is sufficient for computing say Acuraccy, but unsuitable for more complex metrics like NDCG, since NDCG also requires information about group_id.

The typical example is information retrieval, where you benchmark set of queries (group_ids) each with different order of preferred results.

Is there a way to access information about group_ids in compute_metrics?

Furthermore, is there a way to access eval dataset name inside compute_metrics?