Add event-based metrics - #77
Conversation
Reviewer's GuideThis PR implements event-based metrics in audmetric by adding a new event_confusion_matrix with onset/offset/duration tolerance matching, and derived functions for precision, recall, F-score, and unweighted average F-score. It also fixes existing per-class metric implementations, updates project dependencies and documentation, and introduces comprehensive tests with sed-eval reference data. ER diagram for event-based confusion matrix output structureerDiagram
EVENT_LABEL {
int id
string name
}
CONFUSION_MATRIX {
int true_positive
int false_positive
int false_negative
int confused
}
EVENT_LABEL ||--o{ CONFUSION_MATRIX : "has metrics for"
CONFUSION_MATRIX {
int event_label_id
int predicted_label_id
int count
}
EVENT_LABEL ||--o{ CONFUSION_MATRIX : "is predicted as"
Class diagram for new event-based metric functions in audmetricclassDiagram
class audmetric {
}
class event_confusion_matrix {
+event_confusion_matrix(truth: pd.Series, prediction: pd.Series, labels: Sequence, onset_tolerance: float, offset_tolerance: float, duration_tolerance: float, normalize: bool) list
}
class event_precision_per_class {
+event_precision_per_class(truth: pd.Series, prediction: pd.Series, labels: Sequence, zero_division: float, onset_tolerance: float, offset_tolerance: float, duration_tolerance: float) dict
}
class event_recall_per_class {
+event_recall_per_class(truth: pd.Series, prediction: pd.Series, labels: Sequence, zero_division: float, onset_tolerance: float, offset_tolerance: float, duration_tolerance: float) dict
}
class event_fscore_per_class {
+event_fscore_per_class(truth: pd.Series, prediction: pd.Series, labels: Sequence, zero_division: float, propagate_nans: bool, onset_tolerance: float, offset_tolerance: float, duration_tolerance: float) dict
}
class event_unweighted_average_fscore {
+event_unweighted_average_fscore(truth: pd.Series, prediction: pd.Series, labels: Sequence, zero_division: float, propagate_nans: bool, onset_tolerance: float, offset_tolerance: float, duration_tolerance: float) float
}
audmetric <|-- event_confusion_matrix
audmetric <|-- event_precision_per_class
audmetric <|-- event_recall_per_class
audmetric <|-- event_fscore_per_class
audmetric <|-- event_unweighted_average_fscore
event_fscore_per_class o-- event_precision_per_class
event_fscore_per_class o-- event_recall_per_class
event_precision_per_class o-- event_confusion_matrix
event_recall_per_class o-- event_confusion_matrix
event_unweighted_average_fscore o-- event_fscore_per_class
Class diagram for updated precision_per_class implementationclassDiagram
class precision_per_class {
+precision_per_class(truth: Sequence, prediction: Sequence, labels: Sequence, zero_division: float) dict
}
class confusion_matrix {
+confusion_matrix(truth: Sequence, prediction: Sequence, labels: Sequence) list
}
precision_per_class o-- confusion_matrix
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
🚀 New features to boost your workflow:
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey there - I've reviewed your changes and they look great!
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location> `audmetric/core/api.py:825-831` </location>
<code_context>
+ duration_tolerance=duration_tolerance,
+ )
+ fscore = {}
+ for label, p, r in zip(labels, precision.values(), recall.values()):
+ if p * r == 0:
+ fscore[label] = 0.0
+ elif not propagate_nans and (
+ (p == 0.0 and np.isnan(r)) or (r == 0.0 and np.isnan(p))
+ ):
</code_context>
<issue_to_address>
**suggestion:** F-score calculation may not handle NaN propagation as expected.
Please review the NaN handling logic, particularly for cases where only precision or recall is NaN, to ensure all scenarios are addressed and the behavior is consistent.
```suggestion
fscore = {}
for label, p, r in zip(labels, precision.values(), recall.values()):
if np.isnan(p) or np.isnan(r):
if propagate_nans:
fscore[label] = np.nan
else:
fscore[label] = 0.0
elif p * r == 0:
fscore[label] = 0.0
```
</issue_to_address>
### Comment 2
<location> `audmetric/core/api.py:949-950` </location>
<code_context>
+ np.seterr(**old_settings)
+ precision[np.isnan(precision)] = zero_division
+
+ # The event based confusion matrix also as a row/column for the "non-event" class,
+ # so we only include metrics for the actual labels
+ return {label: float(r) for label, r in zip(labels, precision[: len(labels)])}
+
</code_context>
<issue_to_address>
**nitpick:** Comment refers to event-based confusion matrix in non-event context.
Clarify or remove this comment in non-event-based metric functions to prevent confusion about the confusion matrix type.
</issue_to_address>
### Comment 3
<location> `tests/test_event_based.py:309-313` </location>
<code_context>
+ normalize=False,
+ )
+ np.testing.assert_equal(confusion, expected_conf)
+ norm_confusion = audmetric.event_confusion_matrix(
+ truth,
+ prediction,
+ labels,
+ onset_tolerance=onset_tol,
+ offset_tolerance=offset_tol,
+ duration_tolerance=duration_tol,
+ normalize=True,
+ )
+ for i, row in enumerate(confusion):
+ total_sum = sum(row)
+ if total_sum != 0:
+ for j, col in enumerate(row):
+ assert norm_confusion[i][j] == col / total_sum
+
+ rpc = audmetric.event_recall_per_class(
</code_context>
<issue_to_address>
**suggestion (testing):** Normalization logic is tested, but floating point comparison may be fragile.
Use np.testing.assert_almost_equal or pytest.approx for comparing normalized values to handle floating point precision issues.
```suggestion
for i, row in enumerate(confusion):
total_sum = sum(row)
if total_sum != 0:
for j, col in enumerate(row):
np.testing.assert_almost_equal(norm_confusion[i][j], col / total_sum)
```
</issue_to_address>
### Comment 4
<location> `tests/test_event_based.py:348-357` </location>
<code_context>
+ )
+ np.testing.assert_equal(fpc, expected_fpc)
+
+ uaf = audmetric.event_unweighted_average_fscore(
+ truth,
+ prediction,
+ labels,
+ zero_division=zero_division,
+ onset_tolerance=onset_tol,
+ offset_tolerance=offset_tol,
+ duration_tolerance=duration_tol,
+ )
+ np.testing.assert_equal(uaf, expected_f)
+
+
</code_context>
<issue_to_address>
**suggestion (testing):** Unweighted average F-score is tested, but consider adding a test for NaN propagation.
Please add a test case that checks the behavior of propagate_nans when some classes have NaN precision or recall, comparing results for propagate_nans=True and False.
</issue_to_address>
### Comment 5
<location> `tests/test_event_based.py:397-405` </location>
<code_context>
+ matrix[labels.index(truth_label)][labels.index(pred_label)] += 1
+
+ # Fill in remaining errors that have no confusions
+ for i, label in enumerate(labels):
+ n_label_truth = len(truth[truth == label])
+ # Count any ground truth segments that have no overlapping prediction at all
</code_context>
<issue_to_address>
**suggestion (testing):** Strong validation of confusion matrix statistics against reference implementation.
Additionally, assert that Ntp + Nfn + Nfp equals the total event count per label to detect accounting errors.
</issue_to_address>
### Comment 6
<location> `audmetric/core/api.py:453` </location>
<code_context>
return eer, Stats(fmr, fnmr, thresholds, threshold)
+def event_confusion_matrix(
+ truth: pd.Series,
+ prediction: pd.Series,
</code_context>
<issue_to_address>
**issue (complexity):** Consider refactoring the event-based metric code by extracting overlap checks, matching logic, and per-class metric calculations into dedicated helper functions.
Here are three small refactorings that will dramatically reduce the size and nesting of your new event‐based code while preserving 100% of its functionality:
1) Extract the “does-this-pair overlap under tolerances?” logic into a helper:
```python
def _segments_overlap(
start_t, end_t, start_p, end_p,
onset_tol, offset_tol, duration_tol
) -> bool:
if onset_tol is not None:
if abs(start_t - start_p) > onset_tol:
return False
# compute the effective offset tolerance
eff_off = offset_tol or 0.0
if duration_tol is not None:
eff_off = max(eff_off, duration_tol * (end_t - start_t))
if abs(end_t - end_p) > eff_off:
return False
return True
```
2) Pull the bipartite‐matching + confusion‐matrix update into a helper:
```python
from scipy.sparse import csr_array
from scipy.sparse.csgraph import maximum_bipartite_matching
def _match_and_accumulate(
hit_matrix: np.ndarray,
overlap_matrix: np.ndarray,
ground_labels: list,
pred_labels: list,
label_to_index: dict,
cm: list[list[int]]
):
# first match correct‐label hits
graph = csr_array(hit_matrix & overlap_matrix)
matches = maximum_bipartite_matching(graph)
for p_i, t_i in enumerate(matches):
if t_i >= 0:
idx = label_to_index[ground_labels[t_i]]
cm[idx][idx] += 1
overlap_matrix[t_i, :] = False
overlap_matrix[:, p_i] = False
# then match leftover overlaps with wrong labels
graph2 = csr_array(overlap_matrix)
for p_i, t_i in enumerate(maximum_bipartite_matching(graph2)):
if t_i >= 0:
i = label_to_index[ground_labels[t_i]]
j = label_to_index[pred_labels[p_i]]
cm[i][j] += 1
```
3) Consolidate your four “per_class” functions (precision/recall/fscore/UAF) into one generic extractor:
```python
import numpy as np
def _event_metric_per_class(
truth, prediction, labels, zero_division,
onset_tolerance, offset_tolerance, duration_tolerance,
axis: int # 0=precision, 1=recall
) -> dict[str, float]:
cm = np.array(
event_confusion_matrix(
truth, prediction, labels,
onset_tolerance=onset_tolerance,
offset_tolerance=offset_tolerance,
duration_tolerance=duration_tolerance,
)
)
totals = cm.sum(axis=axis)
vals = cm.diagonal() / totals
vals = np.nan_to_num(vals, nan=zero_division)
return {lab: float(vals[i]) for i, lab in enumerate(labels)}
# Then your functions become:
def event_precision_per_class(...):
return _event_metric_per_class(..., axis=0)
def event_recall_per_class(...):
return _event_metric_per_class(..., axis=1)
def event_fscore_per_class(...):
p = event_precision_per_class(...)
r = event_recall_per_class(...)
return {k: (2*p[k]*r[k]/(p[k]+r[k]) if p[k]+r[k]>0 else 0.0) for k in p}
def event_unweighted_average_fscore(...):
f = list(event_fscore_per_class(...).values())
return float(np.nanmean(f))
```
• This pulls all overlap checks, matching and per‐class maths into tiny focused units
• Your big `event_confusion_matrix` loop now just calls `_segments_overlap` and `_match_and_accumulate`
• Docstrings and argument parsing remain exactly the same
– you’ve reduced nesting, removed duplication, and made each piece easy to test in isolation.
</issue_to_address>
### Comment 7
<location> `tests/test_event_based.py:309-313` </location>
<code_context>
</code_context>
<issue_to_address>
**issue (code-quality):** Avoid loops in tests. ([`no-loop-in-tests`](https://docs.sourcery.ai/Reference/Rules-and-In-Line-Suggestions/Python/Default-Rules/no-loop-in-tests))
<details><summary>Explanation</summary>Avoid complex code, like loops, in test functions.
Google's software engineering guidelines says:
"Clear tests are trivially correct upon inspection"
To reach that avoid complex code in tests:
* loops
* conditionals
Some ways to fix this:
* Use parametrized tests to get rid of the loop.
* Move the complex logic into helpers.
* Move the complex part into pytest fixtures.
> Complexity is most often introduced in the form of logic. Logic is defined via the imperative parts of programming languages such as operators, loops, and conditionals. When a piece of code contains logic, you need to do a bit of mental computation to determine its result instead of just reading it off of the screen. It doesn't take much logic to make a test more difficult to reason about.
Software Engineering at Google / [Don't Put Logic in Tests](https://abseil.io/resources/swe-book/html/ch12.html#donapostrophet_put_logic_in_tests)
</details>
</issue_to_address>
### Comment 8
<location> `tests/test_event_based.py:311-313` </location>
<code_context>
</code_context>
<issue_to_address>
**issue (code-quality):** Avoid conditionals in tests. ([`no-conditionals-in-tests`](https://docs.sourcery.ai/Reference/Rules-and-In-Line-Suggestions/Python/Default-Rules/no-conditionals-in-tests))
<details><summary>Explanation</summary>Avoid complex code, like conditionals, in test functions.
Google's software engineering guidelines says:
"Clear tests are trivially correct upon inspection"
To reach that avoid complex code in tests:
* loops
* conditionals
Some ways to fix this:
* Use parametrized tests to get rid of the loop.
* Move the complex logic into helpers.
* Move the complex part into pytest fixtures.
> Complexity is most often introduced in the form of logic. Logic is defined via the imperative parts of programming languages such as operators, loops, and conditionals. When a piece of code contains logic, you need to do a bit of mental computation to determine its result instead of just reading it off of the screen. It doesn't take much logic to make a test more difficult to reason about.
Software Engineering at Google / [Don't Put Logic in Tests](https://abseil.io/resources/swe-book/html/ch12.html#donapostrophet_put_logic_in_tests)
</details>
</issue_to_address>
### Comment 9
<location> `tests/test_event_based.py:312-313` </location>
<code_context>
</code_context>
<issue_to_address>
**issue (code-quality):** Avoid loops in tests. ([`no-loop-in-tests`](https://docs.sourcery.ai/Reference/Rules-and-In-Line-Suggestions/Python/Default-Rules/no-loop-in-tests))
<details><summary>Explanation</summary>Avoid complex code, like loops, in test functions.
Google's software engineering guidelines says:
"Clear tests are trivially correct upon inspection"
To reach that avoid complex code in tests:
* loops
* conditionals
Some ways to fix this:
* Use parametrized tests to get rid of the loop.
* Move the complex logic into helpers.
* Move the complex part into pytest fixtures.
> Complexity is most often introduced in the form of logic. Logic is defined via the imperative parts of programming languages such as operators, loops, and conditionals. When a piece of code contains logic, you need to do a bit of mental computation to determine its result instead of just reading it off of the screen. It doesn't take much logic to make a test more difficult to reason about.
Software Engineering at Google / [Don't Put Logic in Tests](https://abseil.io/resources/swe-book/html/ch12.html#donapostrophet_put_logic_in_tests)
</details>
</issue_to_address>
### Comment 10
<location> `tests/test_event_based.py:397-405` </location>
<code_context>
</code_context>
<issue_to_address>
**issue (code-quality):** Avoid loops in tests. ([`no-loop-in-tests`](https://docs.sourcery.ai/Reference/Rules-and-In-Line-Suggestions/Python/Default-Rules/no-loop-in-tests))
<details><summary>Explanation</summary>Avoid complex code, like loops, in test functions.
Google's software engineering guidelines says:
"Clear tests are trivially correct upon inspection"
To reach that avoid complex code in tests:
* loops
* conditionals
Some ways to fix this:
* Use parametrized tests to get rid of the loop.
* Move the complex logic into helpers.
* Move the complex part into pytest fixtures.
> Complexity is most often introduced in the form of logic. Logic is defined via the imperative parts of programming languages such as operators, loops, and conditionals. When a piece of code contains logic, you need to do a bit of mental computation to determine its result instead of just reading it off of the screen. It doesn't take much logic to make a test more difficult to reason about.
Software Engineering at Google / [Don't Put Logic in Tests](https://abseil.io/resources/swe-book/html/ch12.html#donapostrophet_put_logic_in_tests)
</details>
</issue_to_address>
### Comment 11
<location> `audmetric/core/api.py:453` </location>
<code_context>
def event_confusion_matrix(
truth: pd.Series,
prediction: pd.Series,
labels: Sequence[object] = None,
*,
onset_tolerance: float | None = 0.2,
offset_tolerance: float | None = 0.2,
duration_tolerance: float | None = None,
normalize: bool = False,
) -> list[list[int | float]]:
r"""Event-based confusion.
This metric compares not only the labels of prediction and ground truth,
but also the time windows they occur in.
Each event is considered to be correctly identified
if the predicted label is the same as the ground truth label,
and if the onset is within the given ``onset_tolerance`` (in seconds)
and the offset is within the given ``offset_tolerance`` (in seconds).
Additionally to the ``offset_tolerance``,
one can also specify the ``duration_tolerance``,
to ensure that the offset occurs
within a certain proportion of the reference event duration.
If a prediction fulfills the ``duration_tolerance``
but not the ``offset_tolerance`` (or vice versa),
it is still considered to be an overlapping segment.
:footcite:`Mesaros2016`
The resulting confusion matrix has one more row and and one more column
than there are labels.
The last row/column corresponds to the absence of any event.
This allows to distinguish between segments that overlap but have differing labels,
and false negatives that have no overlapping predicted segment
as well as false positives that have no overlapping ground truth segment.
.. footbibliography::
Args:
truth: ground truth labels with a segmented index conform to `audformat`_
prediction: predicted labels with a segmented index conform to `audformat`_
labels: included labels in preferred ordering.
If no labels are supplied,
they will be inferred from
:math:`\{\text{prediction}, \text{truth}\}`
and ordered alphabetically
onset_tolerance: the onset tolerance in seconds.
If the predicted segment's onset does not occur within this time window
compared to the ground truth segment's onset,
it is not considered correct
offset_tolerance: the offset tolerance in seconds.
If the predicted segment's offset does not occur within this time window
compared to the ground truth segment's offset,
it is not considered correct,
unless the ``duration_tolerance`` is specified and fulfilled
duration_tolerance: the duration tolerance as a measure of proportion
of the ground truth segment's total duration.
If the ``offset_tolerance`` is not fulfilled,
and the predicted segment's offset does not occur within this time window
compared to the ground truth segment's offset,
it is not considered correct
normalize: normalize confusion matrix over the rows
Returns:
event confusion matrix
Raises:
ValueError: if ``truth`` or ``prediction``
do not have a segmented index conform to `audformat`_
Examples:
>>> truth = pd.Series(
... index=audformat.segmented_index(
... files=["f1.wav"] * 4,
... starts=[0, 0.1, 0.2, 0.3],
... ends=[0.1, 0.2, 0.3, 0.4],
... ),
... data=["a", "a", "b", "b"],
... )
>>> prediction = pd.Series(
... index=audformat.segmented_index(
... files=["f1.wav"] * 4 + ["f2.wav"],
... starts=[0, 0.09, 0.2, 0.31, 0.0],
... ends=[0.1, 0.2, 0.3, 0.41, 1.0],
... ),
... data=["a", "b", "a", "b", "b"],
... )
>>> event_confusion_matrix(
... truth, prediction, onset_tolerance=0.02, offset_tolerance=0.02
... )
[[1, 1, 0], [1, 1, 0], [0, 1, 0]]
.. _audformat: https://audeering.github.io/audformat/data-format.html
"""
if not audformat.is_segmented_index(truth) or not audformat.is_segmented_index(
prediction
):
raise ValueError(
"For event-based metrics, the truth and prediction "
"should be a pandas Series with a segmented index conform to audformat."
)
if labels is None:
labels = infer_labels(truth.values, prediction.values)
# Confusion matrix of event labels + "no event" label
matrix = [[0 for _ in range(len(labels) + 1)] for _ in range(len(labels) + 1)]
# Code based on 'optimal' event matching
# at https://github.com/TUT-ARG/sed_eval/blob/0cb1b6d11ceec4fe500cc9b31079c9d8666ed6eb/sed_eval/sound_event.py#L1108
for file, file_truth in truth.groupby(level=IndexField.FILE):
file_pred = prediction[
prediction.index.get_level_values(IndexField.FILE) == file
]
n_truth = len(file_truth)
n_pred = len(file_pred)
# Matrix storing whether there is an overlap
# between each truth segment and each predicted segment
overlap_matrix = np.ones((n_truth, n_pred), dtype=bool)
hit_matrix = np.zeros((n_truth, n_pred), dtype=bool)
for i, ((_, start_truth, end_truth), label_truth) in enumerate(
# file_truth.sort_index().items()
file_truth.items()
):
start_truth = start_truth.total_seconds()
end_truth = end_truth.total_seconds()
duration_truth = end_truth - start_truth
for j, ((_, start_pred, end_pred), label_pred) in enumerate(
# file_pred.sort_index().items()
file_pred.items()
):
start_pred = start_pred.total_seconds()
end_pred = end_pred.total_seconds()
# Condition 1: labels are the same
hit_matrix[i, j] = label_truth == label_pred
# Condition 2: onset is within the allowed tolerance
if onset_tolerance is not None:
overlap_matrix[i, j] *= (
math.fabs(start_truth - start_pred) <= onset_tolerance
)
# Condition 3: offset is within (absolute) offset tolerance
# or (if provided) within duration proportion based offset tolerance
if offset_tolerance is not None or duration_tolerance is not None:
actual_offset_tolerance = 0
if offset_tolerance is not None:
actual_offset_tolerance = offset_tolerance
if duration_tolerance is not None:
actual_offset_tolerance = max(
duration_tolerance * duration_truth, actual_offset_tolerance
)
offset_match = (
math.fabs(end_truth - end_pred) <= actual_offset_tolerance
)
overlap_matrix[i, j] *= offset_match
hit_matrix *= overlap_matrix
# Get optimal matching between prediction and ground truth
# when there are multiple possibilities
graph = csr_array(hit_matrix)
matches = maximum_bipartite_matching(graph)
# Store leftover overlapping segments that have not been matched
leftover_overlap_matrix = overlap_matrix.copy()
for pred_i, truth_i in enumerate(matches):
if truth_i != -1:
truth_label = file_truth.iloc[truth_i]
label_index = labels.index(truth_label)
# Mark in leftover overlap matrix
# that this segment is already covered
leftover_overlap_matrix[truth_i, :] = False
leftover_overlap_matrix[:, pred_i] = False
# Add to respective label in total confusion matrix
matrix[label_index][label_index] += 1
# Get optimal matching between prediction and ground truth segments
# that have differing labels and that do not yet have a match
leftover_graph = csr_array(leftover_overlap_matrix)
confused_matching = maximum_bipartite_matching(leftover_graph)
for pred_i, truth_i in enumerate(confused_matching):
if truth_i != -1:
pred_label = file_pred.iloc[pred_i]
truth_label = file_truth.iloc[truth_i]
# Increase counter in confusion matrix for this confused match
matrix[labels.index(truth_label)][labels.index(pred_label)] += 1
# Fill in remaining errors that have no confusions
for i, label in enumerate(labels):
n_label_truth = len(truth[truth == label])
# Count any ground truth segments that have no overlapping prediction at all
n_missed = n_label_truth - sum(matrix[i][: len(labels)])
matrix[i][-1] += n_missed
n_label_pred = len(prediction[prediction == label])
# Count any predictions that have no overlap with ground truth segments
n_extra = n_label_pred - sum([matrix[j][i] for j in range(len(labels))])
matrix[-1][i] = n_extra
if normalize:
for idx, row in enumerate(matrix):
if np.sum(row) != 0:
row_sum = float(np.sum(row))
matrix[idx] = [x / row_sum for x in row]
return matrix
</code_context>
<issue_to_address>
**issue (code-quality):** Low code quality found in event\_confusion\_matrix - 8% ([`low-code-quality`](https://docs.sourcery.ai/Reference/Default-Rules/comments/low-code-quality/))
<br/><details><summary>Explanation</summary>The quality score for this function is below the quality threshold of 25%.
This score is a combination of the method length, cognitive complexity and working memory.
How can you solve this?
It might be worth refactoring this function to make it shorter and more readable.
- Reduce the function length by extracting pieces of functionality out into
their own functions. This is the most important thing you can do - ideally a
function should be less than 10 lines.
- Reduce nesting, perhaps by introducing guard clauses to return early.
- Ensure that variables are tightly scoped, so that code using related concepts
sits together within the function rather than being scattered.</details>
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Change handling of nan propagation Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
|
Great, a new metric. Let me start asking a few questions in order to understand it better.
|
Yes, so the metrics at
This is the case where the one additional row and column of the confusion matrix comes into play. They represent the "absence of any event". So if we have a ground truth segment with label, and no overlapping prediction, we get an entry in the last column of the confusion matrix, marking that there is an absence of a predicted event. And similarly, if there is a prediction segment with no overlapping ground truth, we add an entry in the last row of the confusion matrix, marking that there is an absence of the ground truth. This also means that the bottom and right-most element of the confusion matrix is not used. I guess it would make more sense to have a non-zero value for the default onset and offset values. Since you would almost never have the case that the segments overlap perfectly. |
Non-overlapping is still not obvious to me. If I understand it correctly the following two cases would both be counted as failed prediction as the onset and offset times are not matched: But in the second case I would assume we have much more confusion than in the first case.
|
Yes.
I agree that it looks like there would be a confusion there, but by this metric, it would not be counted as any confusions, if the onset/offset condition isn't fulfilled. I think the motivation for defining the metric like this is that if you have some overlapping ground truth segments like this.: Then it is more likely that the model confused the |
Good that you reminded me of |
I think that would be nice to have, yes! I was planning to present the results the same way. |
|
I will prepare an |
|
I added the
The later could be solved by adding some extra code to Here is a code example to test the current stage. You can do it with: $ uv run --with "audplot @ git+ssh://git@github.com/audeering/audplot.git@confusion-matrix-metric" --with ipython ipythonfrom functools import partial
import pandas as pd
import audformat
from audmetric import event_confusion_matrix
import audplot
truth = pd.Series(
index=audformat.segmented_index(
files=["f1.wav"] * 4,
starts=[0, 0.1, 0.2, 0.3],
ends=[0.1, 0.2, 0.3, 0.4],
),
data=["a", "a", "b", "b"],
)
prediction = pd.Series(
index=audformat.segmented_index(
files=["f1.wav"] * 4 + ["f2.wav"],
starts=[0, 0.09, 0.2, 0.31, 0.0],
ends=[0.1, 0.2, 0.3, 0.41, 1.0],
),
data=["a", "b", "a", "b", "b"],
)
cm = partial(event_confusion_matrix, onset_tolerance=0.02, offset_tolerance=0.02)
audplot.confusion_matrix(truth, prediction, metric=cm)All of the problems would not exist if we would support |
I had a look and remember why I resisted to go into this direction previously: in |
|
Cool, thanks!
Ah ok, I didn't think about that... I guess there is also the question what label should be used for the last row/column if it is not provided. For the other event-based metrics to be correct, we need to have the number of false negatives and false positives from the "no event" column and row. Currently we derive those numbers from the total number of predictions/ground truth per class at the end of the
Maybe that is indeed the easiest solution, if we can do it in a compatible way.. It would also make it possible to set the desired labels for the "no event" class.
I see. Normalizing the confusion matrix in retrospect sounds doable, but there might be other factors that make this more complicated. |
|
There is one existing event based metric already: |
Hm, I guess it would be more related to a frame-level evaluation, or what they call "Segment-based" at It is a bit misleading that the metrics added here work on time-stamped segments, and the other |
|
How did you generated the expected test results under |
Indeed, they are generated by the script at |
Co-authored-by: Hagen Wierstorf <hwierstorf@audeering.com>
2 participants
This adds event-based metrics as described in Metrics for Polyphonic Sound Event Detection and implemented by sed-eval.
Compared to regular multi-label classification metrics, these metrics also check whether the time window of the predicted segment matches the ground truth segment. Often, there is also a certain onset and offset tolerance that is allowed, so segments don't need to have exactly the same time stamps to be considered correct.
The main computation happens in
audmetric.event_confusion_matrix():which is based on the code at https://github.com/TUT-ARG/sed_eval/blob/0cb1b6d11ceec4fe500cc9b31079c9d8666ed6eb/sed_eval/sound_event.py#L1108
but also takes note of which specific confusions happened.
Using the results from the confusion matrix, we can compute
audmetric.event_fscore_per_class(),audmetric.event_precision_per_class(),audmetric.event_recall_per_class(), andaudmetric.event_unweighted_average_fscore().audmetric.event_fscore_per_class()
audmetric.event_precision_per_class()
audmetric.event_recall_per_class()
audmetric.event_unweighted_average_fscore()
Summary by Sourcery
Implement event-based metrics in audmetric: confusion matrix, per-class precision, recall, F-score, and unweighted average F-score with onset/offset/duration tolerances, update dependencies and add supporting tests.
New Features:
Enhancements:
Build:
Documentation:
Tests:
Summary by Sourcery
Implement event-based polyphonic sound event detection metrics with optimal event matching and time tolerance thresholds, and extend the library with corresponding per-class precision, recall, F-score, and unweighted average F-score functions.
New Features:
Enhancements:
Build:
Documentation:
Tests: