Metric

Provides the Metric base class, which defines the interface of metrics.

Classes:

Name	Description
Metric	Base class, defining the interface of metrics.

Metric

Simple base class / interface definition for metrics.

Methods:

Name	Description
reset	To reset all internal states. Must be implemented by the subclass.
update	Calls the private _update method, which must be implemented by the subclass. Used to update internal states with new data.
compute	Calls the private _compute method, which must be implemented by the subclass. Used to compute and return the metric results.
show_result	Utility method to print the metric result in a readable format.

reset()

Reset all internal states.

Raises:

Type	Description
NotImplementedError	Subclasses must implement this method.

Source code in src/kibad_llm/metric.py

def reset(self) -> None:
    """Reset all internal states.

    Raises:
        NotImplementedError: Subclasses must implement this method.
    """
    raise NotImplementedError("Subclasses must implement this method.")

update(prediction, reference, record_id=None)

Calls the private _update method to update internal states with new data.

Parameters:

Name	Type	Description	Default
prediction	Any	Predictions made in a dataset.	required
reference	Any	Gold data to compare the predictions to.	required
record_id	Hashable | None	Id to tag the comparison with.	None

Source code in src/kibad_llm/metric.py

def update(self, prediction: Any, reference: Any, record_id: Hashable | None = None) -> None:
    """Calls the private _update method to update internal states with new data.

    Args:
        prediction: Predictions made in a dataset.
        reference: Gold data to compare the predictions to.
        record_id: Id to tag the comparison with.
    """
    self._update(prediction=prediction, reference=reference, record_id=record_id)

compute(*args, reset=True, **kwargs)

Wrapper for _compute.

Parameters:

Name	Type	Description	Default
*args	Any	Positional args forwarded to the _compute implementation.	()

Other Parameters:

Name	Type	Description
reset	bool	If True, uses reset to reset the internal state after computing the result.
**kwargs	Any	Keyword args forwarded to the _compute implementation.

Returns:

Type	Description
dict[str, Any]	result of the evaluation.

Source code in src/kibad_llm/metric.py

def compute(self, *args, reset: bool = True, **kwargs) -> dict[str, Any]:
    """Wrapper for `_compute`.

    Args:
        *args (Any): Positional args forwarded to the `_compute` implementation.

    Keyword Args:
        reset (bool): If `True`, uses [`reset`][..reset] to reset the internal state after computing the result.
        **kwargs (Any): Keyword args forwarded to the `_compute` implementation.

    Returns:
        result of the evaluation.
    """
    result = self._compute(*args, **kwargs)
    if reset:
        self.reset()
    return result

show_result(result=None, reset=True)

Utility method to print the metric result in a readable format.

Parameters:

Name	Type	Description	Default
result	dict[str, Any] | None	Dict to pretty-print as JSON string or None to compute the result with compute	None
reset	bool	Whether to reset after using compute to obtain result, if the result arg was None.	True

Source code in src/kibad_llm/metric.py

def show_result(self, result: dict[str, Any] | None = None, reset: bool = True) -> None:
    """Utility method to print the metric result in a readable format.

    Args:
        result: Dict to pretty-print as JSON string or None to compute the result with [`compute`][..compute]
        reset: Whether to reset after using [`compute`][..compute] to obtain result, if the result arg was None.
    """
    if result is None:
        result = self.compute(reset=reset)

    logger.info(f"Evaluation results:\n{self._format_result(result)}")