Skip to content

Add Hard Voting and Median Ensembler as well as new Task Evaluator - #482

Open
Heptasphere wants to merge 3 commits into
autogluon:mainfrom
Heptasphere:simple_ensembler_integration
Open

Add Hard Voting and Median Ensembler as well as new Task Evaluator#482
Heptasphere wants to merge 3 commits into
autogluon:mainfrom
Heptasphere:simple_ensembler_integration

Conversation

@Heptasphere

Copy link
Copy Markdown

First PR for #334.

Description of changes

Two new ensemblers and a task evaluator were added:

  • MedianEnsembler: Ensembler that uses the median rule (robust soft voting in
  • HardVotingEnsembler: Ensembler for classification only that uses hard
  • NoPreprocessingTaskEvaluator: Task Evaluator allowing ensemblers that use all predicted classes in multiclass training/prediction, even for fast metrics, not just extracted true classes.

NoPreprocessingTaskEvaluator doesn't preprocess the base learner predictions before training / prediction, it only preprocesses the ensembler results for error estimation.

Testing

All tests are in tests/tabarena/simulation/test_ensembler.py

Basic tests for the format of the ensemblers were added. Tests for their results on a toy example were also added.

Test to check that the new task evaluator had similar results to TaskEvaluator on a simple ensembler (TopKAverageEnsembler) was added.

  • In the case of multiclass, small changes to the metric value were expected due to the particularities of log_loss preprocessing. We use the original float32 rather than the preprocessed float64, leading to a $2^{-23} \approx 10^{-6}$ estimation difference for the metric.
  • This task evaluator is not expected to work well with Ensemble Selection.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@LennartPurucker LennartPurucker added the meta Anything related to analysing Metadata and Meta-Learning label Aug 31, 2026
@Heptasphere Heptasphere changed the title Add Hard Voting and Mean Ensembler as well as new Task Evaluator Add Hard Voting and Median Ensembler as well as new Task Evaluator Aug 31, 2026
@Heptasphere

Copy link
Copy Markdown
Author

The code for the MedianEnsembler could easily have been made into a general code for min/max/median/whatever rule through giving a function as argument and normalizing for binary classification (unnecessary for median due to its symmetry).

I considered that median is already not very popular so any other rule might remain forever unused. (That, and I'm always a bit paranoid about letting a method defined by the user be an input for anything.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

meta Anything related to analysing Metadata and Meta-Learning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants