Collection

Metric collections and helpers for dynamic per-field evaluation.

Functions:

Name	Description
_expand_field_by_key_values	Expand nested dict-like fields into generated top-level fields.

Classes:

Name	Description
MetricCollection	Aggregate multiple child metrics behind one metric interface.
MetricCollectionWithFieldDiscoveryAndGrouping	Lazily create per-field metrics while discovering or expanding fields.

MetricCollection(metrics=None, sort_fields=False)

Bases: Metric, Generic[T]

A metric that aggregates multiple sub-metrics.

Parameters:

Name	Type	Description	Default
metrics	dict[str, T] | None	Optional mapping of metric names to metric instances.	None
sort_fields	bool	Whether computed results should be emitted in sorted field order.	False

Source code in src/kibad_llm/metrics/collection.py

def __init__(self, metrics: dict[str, T] | None = None, sort_fields: bool = False) -> None:
    """Initialize the metric collection.

    Args:
        metrics: Optional mapping of metric names to metric instances.
        sort_fields: Whether computed results should be emitted in sorted field order.
    """
    super().__init__()
    self.metrics: dict[str, T] = metrics or dict()
    self.sort_fields = sort_fields

add_metric(name, metric)

Adds a new metric to the collection.

Source code in src/kibad_llm/metrics/collection.py

def add_metric(self, name: str, metric: T) -> None:
    """Adds a new metric to the collection."""
    if name in self.metrics:
        raise ValueError(f"Metric {name} already exists")
    self.metrics[name] = metric

reset()

Resets all sub-metrics.

Source code in src/kibad_llm/metrics/collection.py

def reset(self) -> None:
    """Resets all sub-metrics."""
    for metric in self.metrics.values():
        metric.reset()

MetricCollectionWithFieldDiscoveryAndGrouping(metric_class, fields=None, subfield_keys=None, subfield_values=None, sort_fields=False, field_overrides=None, **kwargs)

Bases: MetricCollection[T2], Generic[T2]

A metric collection that discovers fields dynamically and can group nested entries.

This collection creates per-field metrics lazily during _update. Fields can either be taken from an explicit allowlist or, on each update, discovered from the union of prediction and reference keys. Additionally, configured dict-like fields can be expanded into generated top-level fields such as field.A&B before the underlying single-field metrics are updated. During that expansion, the configured grouping keys are used to derive the generated field names and are removed from the scored payload, while subfield_values can optionally restrict which of the remaining nested values are compared. Additional keyword arguments passed to __init__ are forwarded to each lazily created per-field metric.

Parameters:

Name	Type	Description	Default
metric_class	type[T2]	Metric class used to instantiate field-specific metrics.	required
fields	list[str] | None	Optional allowlist of fields to evaluate. If omitted, fields are discovered from the union of keys present in each prediction/reference pair.	None
subfield_keys	dict[str, list[str]] | None	Optional mapping describing how nested entries are split into generated fields.	None
subfield_values	dict[str, list[str]] | None	Optional mapping restricting which nested values are kept after field expansion.	None
sort_fields	bool	Whether computed results should be emitted in sorted field order.	False
field_overrides	dict[str, dict[str, Any]] | None	Optional mapping of field names to keyword arguments for each per-field metric.	None
**kwargs		Additional keyword arguments forwarded to each created metric instance.	{}

Source code in src/kibad_llm/metrics/collection.py

def __init__(
    self,
    metric_class: type[T2],
    fields: list[str] | None = None,
    subfield_keys: dict[str, list[str]] | None = None,
    subfield_values: dict[str, list[str]] | None = None,
    sort_fields: bool = False,
    field_overrides: dict[str, dict[str, Any]] | None = None,
    **kwargs,
) -> None:
    """Initialize the field-discovering metric collection.

    Args:
        metric_class: Metric class used to instantiate field-specific metrics.
        fields: Optional allowlist of fields to evaluate. If omitted, fields are discovered
            from the union of keys present in each prediction/reference pair.
        subfield_keys: Optional mapping describing how nested entries are split into generated
            fields.
        subfield_values: Optional mapping restricting which nested values are kept after field
            expansion.
        sort_fields: Whether computed results should be emitted in sorted field order.
        field_overrides: Optional mapping of field names to keyword arguments for each per-field metric.
        **kwargs: Additional keyword arguments forwarded to each created metric instance.
    """
    self.metric_class = metric_class
    self.fields = fields
    self.subfield_keys = subfield_keys
    self.subfield_values = subfield_values
    self.field_overrides = field_overrides or {}
    self.metric_kwargs = kwargs
    super().__init__(sort_fields=sort_fields)