Confusion matrix

Confusion-matrix metrics built on shared tp/fp/fn state.

Classes:

Name	Description
ConfusionMatrix	Build a confusion matrix from the shared tp/fp/fn entry state for one field.
ConfusionMatrixCollection	Build confusion matrices for multiple fields with optional field discovery and grouped-field expansion.

ConfusionMatrix(show_as_markdown=False, unassignable_label='UNASSIGNABLE', undetected_label='UNDETECTED', **kwargs)

Bases: MetricWithTpFpFnEntries

Build an alignment-averaged confusion matrix from tp/fp/fn entry state.

In multi-label settings, unmatched gold and predicted labels do not define a unique off-diagonal confusion matrix. For example, if one record contains a missed gold label A and an extra predicted label B, the tp/fp/fn state alone does not tell us whether this should be accounted for as A -> B, as A -> UNDETECTED plus UNASSIGNABLE -> B, or as part of another possible alignment.

This metric therefore uses an alignment-averaged accounting rule per record:

• exact true-positive labels are counted deterministically on the diagonal;
• unmatched gold and predicted labels are distributed over all compatible partial one-to-one alignments between false negatives and false positives;
• all compatible partial alignments are weighted equally;
• unmatched gold labels that are not aligned to a prediction contribute to undetected_label;
• unmatched predicted labels that are not aligned to a gold label contribute to unassignable_label.

The resulting off-diagonal entries are expected ambiguous error mass under this uniform partial-alignment assumption. They are useful for exploratory error analysis, but they are not directly observed misclassification counts and do not provide statistical significance or uncertainty estimates.

Warning

Because the metric operates on sets, duplicate predicted or gold labels are collapsed in multi-label settings per record.

Warning

Off-diagonal entries indicate possible label-shift mass under the accounting rule, not evidence that a particular label shift actually occurred.

Parameters:

Name	Type	Description	Default
show_as_markdown	bool	Whether compute() should log the resulting confusion matrix as a markdown table.	False
unassignable_label	str	Label used on the gold axis for predicted labels that remain unaligned to any gold label.	'UNASSIGNABLE'
undetected_label	str	Label used on the prediction axis for gold labels that remain unaligned to any prediction.	'UNDETECTED'

Other Parameters:

Name	Type	Description
field		Optional field to extract from dictionary inputs.
flatten_dicts		Whether nested dictionaries should be flattened before comparison.
ignore_subfields		Optional subfields to ignore when hashing dictionary values.
ignore_missing_entries		Whether one-sided empty entries should be skipped.

Source code in src/kibad_llm/metrics/confusion_matrix.py

def __init__(
    self,
    show_as_markdown: bool = False,
    unassignable_label: str = "UNASSIGNABLE",
    undetected_label: str = "UNDETECTED",
    **kwargs: Any,
):
    """Initialize the confusion-matrix metric.

    Args:
        show_as_markdown: Whether `compute()` should log the resulting confusion
            matrix as a markdown table.
        unassignable_label: Label used on the gold axis for predicted labels that
            remain unaligned to any gold label.
        undetected_label: Label used on the prediction axis for gold labels that
            remain unaligned to any prediction.

    Keyword Args:
        field: Optional field to extract from dictionary inputs.
        flatten_dicts: Whether nested dictionaries should be flattened before
            comparison.
        ignore_subfields: Optional subfields to ignore when hashing dictionary
            values.
        ignore_missing_entries: Whether one-sided empty entries should be skipped.
    """
    super().__init__(**kwargs)
    self.unassignable_label = unassignable_label
    self.undetected_label = undetected_label
    self.show_as_markdown = show_as_markdown

ConfusionMatrixCollection(**kwargs)

Bases: MetricCollectionWithFieldDiscoveryAndGrouping[ConfusionMatrix]

Build confusion matrices for multiple fields at once.

The collection lazily creates one ConfusionMatrix per field and inherits optional dynamic field discovery plus grouped-field expansion from MetricCollectionWithFieldDiscoveryAndGrouping. Nested dict-like fields can therefore be expanded into generated field names such as organism_trends.Amphibien&Wald before each per-field confusion matrix is updated.

Attributes:

Name	Type	Description
fields		Explicit field names to evaluate, or None to discover them dynamically.
subfield_keys		Optional rules for expanding nested dict-like fields into generated fields.
subfield_values		Optional rules restricting which nested values are compared after expansion.
metric_kwargs		Keyword arguments forwarded to the per-field ConfusionMatrix instances.

Other Parameters:

Name	Type	Description
fields		Optional allowlist of fields to evaluate. If omitted, fields are discovered from the union of keys present in each prediction/reference pair.
subfield_keys		Optional mapping describing how nested entries are split into generated fields.
subfield_values		Optional mapping restricting which nested values are kept after field expansion.
sort_fields		Whether to sort the fields in the output. Defaults to False.
show_as_markdown		Whether each per-field confusion matrix should be logged as a markdown table when computed.
unassignable_label		Label used on the gold axis for false positives.
undetected_label		Label used on the prediction axis for false negatives.
flatten_dicts		Whether nested dictionaries should be flattened before comparison.
ignore_subfields		Optional subfields to ignore when hashing dictionary payloads.
ignore_missing_entries		Whether one-sided empty entries should be skipped.

Source code in src/kibad_llm/metrics/confusion_matrix.py

def __init__(
    self,
    **kwargs,
) -> None:
    """Initialize a multi-field confusion-matrix collection.

    Keyword Args:
        fields: Optional allowlist of fields to evaluate. If omitted, fields are discovered
            from the union of keys present in each prediction/reference pair.
        subfield_keys: Optional mapping describing how nested entries are split into generated
            fields.
        subfield_values: Optional mapping restricting which nested values are kept after field
            expansion.
        sort_fields: Whether to sort the fields in the output. Defaults to False.
        show_as_markdown: Whether each per-field confusion matrix should be logged as a markdown
            table when computed.
        unassignable_label: Label used on the gold axis for false positives.
        undetected_label: Label used on the prediction axis for false negatives.
        flatten_dicts: Whether nested dictionaries should be flattened before comparison.
        ignore_subfields: Optional subfields to ignore when hashing dictionary payloads.
        ignore_missing_entries: Whether one-sided empty entries should be skipped.
    """
    super().__init__(metric_class=ConfusionMatrix, **kwargs)

num_partial_matchings(m, n) cached

Return the number of partial one-to-one alignments between two item sets.

A partial alignment links k of m gold-only items to k of n prediction-only items, for any k from 0 to min(m, n). The selected items are matched bijectively.

The number of compatible partial alignments is:

sum_k binom(m, k) * binom(n, k) * k!

where k = 0 corresponds to the alignment in which no gold-only item is paired with a prediction-only item. In the confusion-matrix computation below, this count is used only as an accounting device for distributing ambiguous error mass.

Source code in src/kibad_llm/metrics/confusion_matrix.py

@cache
def num_partial_matchings(m: int, n: int) -> int:
    """Return the number of partial one-to-one alignments between two item sets.

    A partial alignment links `k` of `m` gold-only items to `k` of `n` prediction-only
    items, for any `k` from 0 to `min(m, n)`. The selected items are matched
    bijectively.

    The number of compatible partial alignments is:

        sum_k binom(m, k) * binom(n, k) * k!

    where `k = 0` corresponds to the alignment in which no gold-only item is paired
    with a prediction-only item. In the confusion-matrix computation below, this count
    is used only as an accounting device for distributing ambiguous error mass.
    """
    return sum(comb(m, k) * comb(n, k) * factorial(k) for k in range(min(m, n) + 1))