newsreclib.metrics package

Submodules

newsreclib.metrics.base module

class newsreclib.metrics.base.CustomRetrievalMetric(empty_target_action: str = 'neg', ignore_index: Optional[int] = None, **kwargs: Any)

Bases: Metric, ABC

Works with binary target data. Accepts float predictions from a model output.

As input to forward and update the metric accepts the following input:

• preds (Tensor): A float tensor of shape (N, ...)

• cand_aspects (Tensor): A long tensor of shape (N, ...)

• clicked_aspects (Tensor): A long tensor of shape (N, ...)

• cand_indexes (Tensor): A long tensor of shape (N, ...) which indicate to which query a prediction belongs

• hist_indexes (Tensor): A long tensor of shape (N, ...) which indicate to which user a target belongs

Note

cand_indexes, preds and cand_aspects must have the same dimension and will be flatten

to single dimension once provided.

Note

Predictions will be first grouped by cand_indexes and then the real metric, defined by overriding the _metric method, will be computed as the mean of the scores over each query.

As output to forward and compute the metric returns the following output:

• metric (Tensor): A tensor as computed by _metric if the number of positive targets is at least 1, otherwise behave as specified by self.empty_target_action.

Parameters:

• empty_target_action –

  Specify what to do with queries that do not have at least a positive or negative (depend on metric) target. Choose from:

  • 'neg': those queries count as 0.0 (default)

  • 'pos': those queries count as 1.0

  • 'skip': skip those queries; if all queries are skipped, 0.0 is returned

  • 'error': raise a ValueError

• ignore_index – Ignore predictions where the target is equal to this number.

• kwargs – Additional keyword arguments, see Metric kwargs for more info.

Raises:

• ValueError – If empty_target_action is not one of error, skip, neg or pos.

• ValueError – If ignore_index is not None or an integer.

cand_aspects: List[Tensor]

cand_indexes: List[Tensor]

clicked_aspects: List[Tensor]

compute() → Tensor

First concat state cand_indexes, hist_indexes, preds, cand_aspects, and clicked_aspects since they were stored as lists.

After that, compute list of groups that will help in keeping together predictions about the same query. Finally, for each group compute the _metric if the number of positive targets is at least 1, otherwise behave as specified by self.empty_target_action.

full_state_update: bool = False

higher_is_better: bool = True

hist_indexes: List[Tensor]

is_differentiable: bool = False

preds: List[Tensor]

update(preds: Tensor, cand_aspects: Tensor, clicked_aspects: Tensor, cand_indexes: Tensor, hist_indexes: Tensor) → None

Check shape, check and convert dtypes, flatten and add to accumulators.

newsreclib.metrics.diversity module

class newsreclib.metrics.diversity.Diversity(num_classes: int, empty_target_action: str = 'neg', ignore_index: Optional[int] = None, top_k: Optional[int] = None, **kwargs: Any)

Bases: RetrievalMetric

Implementation of the Aspect-based Diversity.

Reference: Iana, Andreea, Goran Glavaš, and Heiko Paulheim. “Train Once, Use Flexibly: A Modular Framework for Multi-Aspect Neural News Recommendation.” arXiv preprint arXiv:2307.16089 (2023). https://arxiv.org/pdf/2307.16089.pdf

For further details, please refer to the paper

full_state_update: bool = False

higher_is_better: bool = True

is_differentiable: bool = False

newsreclib.metrics.functional module

newsreclib.metrics.functional.diversity(preds: Tensor, target: Tensor, num_classes: int, top_k: Optional[int] = None) → Tensor

Computes Aspect-based Diversity.

Reference: Iana, Andreea, Goran Glavaš, and Heiko Paulheim. “Train Once, Use Flexibly: A Modular Framework for Multi-Aspect Neural News Recommendation.” arXiv preprint arXiv:2307.16089 (2023). https://arxiv.org/pdf/2307.16089.pdf

Parameters:

• preds – Estimated probabilities of each candidate news to be clicked.

• target – Ground truth about the aspect \(A_p\) of the news being relevant or not.

• num_classes – Number of classes of the aspect \(A_p\).

• top_k – Consider only the top k elements for each query (default: None, which considers them all).

Returns:

A single-value tensor with the aspect-based diversity (\(D_{A_p}\)) of the predictions preds wrt the labels target.

newsreclib.metrics.functional.generalized_jaccard(pred: Tensor, target: Tensor) → Tensor

Computes the Generalized Jaccard metric.

Reference: Bonnici, Vincenzo. “Kullback-Leibler divergence between quantum distributions, and its upper-bound.” arXiv preprint arXiv:2008.05932 (2020).

Parameters:

• preds – Estimated probability distribution.

• target – Target probability distribution.

Returns:

A single-value tensor with the generalized Jaccard of the predictions preds wrt the labels target.

newsreclib.metrics.functional.harmonic_mean(scores: Tensor) → Tensor

Computes the harmonic mean of N scores.

Parameters:

scores – A tensor of scores.

Returns:

A single-value tensor with the harmonic mean of the scores.

newsreclib.metrics.functional.personalization(preds: Tensor, predicted_aspects: Tensor, target_aspects: Tensor, num_classes: int, top_k: Optional[int] = None) → Tensor

Computes Aspect-based Personalization.

Reference: Iana, Andreea, Goran Glavaš, and Heiko Paulheim. “Train Once, Use Flexibly: A Modular Framework for Multi-Aspect Neural News Recommendation.” arXiv preprint arXiv:2307.16089 (2023). https://arxiv.org/pdf/2307.16089.pdf

Parameters:

• preds – Estimated probabilities of each candidate news to be clicked.

• predicted_aspects – Aspects of the news predicted to be clicked.

• target_aspects – Ground truth about the aspect \(A_p\) of the news being relevant or not.

• num_classes – Number of classes of the aspect \(A_p\).

• top_k – Consider only the top k elements for each query (default: None, which considers them all).

Returns:

A single-value tensor with the aspect-based personalization (\(PS_{A_p}\)) of the predictions preds and predicted_aspects wrt the labels target_aspects.

newsreclib.metrics.personalization module

class newsreclib.metrics.personalization.Personalization(num_classes: int, empty_target_action: str = 'neg', ignore_index: Optional[int] = None, top_k: Optional[int] = None, **kwargs: Any)

Bases: CustomRetrievalMetric

Implementation of the Aspect-based Personalization.

Reference: Iana, Andreea, Goran Glavaš, and Heiko Paulheim. “Train Once, Use Flexibly: A Modular Framework for Multi-Aspect Neural News Recommendation.” arXiv preprint arXiv:2307.16089 (2023). https://arxiv.org/pdf/2307.16089.pdf

For further details, please refer to the paper

full_state_update: bool = False

higher_is_better: bool = True

is_differentiable: bool = False

newsreclib.metrics.utils module

newsreclib.metrics.utils.get_metric(metric_name: str, metric_params: Optional[Dict[str, Union[str, int]]])

Returns a metric object for the specified name.

Parameters:

• metric_name – Name of the metric.

• metric_params – Dictionary of parameters for instantiating the metric object.

Module contents