Skip to content

Core Components

Benchmark

Bases: BenchmarkEntity

Benchmark class represents a benchmark in the LunaBench system.

This class is responsible for managing benchmark-related operations, including creating and deleting benchmarks. It provides methods for interacting with the benchmark data and executing benchmark runs.

create(name: str) -> Benchmark staticmethod

Create a new benchmark with the given name.

The name for a benchmark must be unique. The returned Benchmark object can be used to interact and configure the new benchmark.

Parameters:

  • name (str) –

    The name of the new benchmark.

Returns:

  • Benchmark –

    The newly created Benchmark object.

open(name: str) -> Benchmark staticmethod

Load a benchmark if it exists, otherwise create a new one.

Parameters:

  • name (str) –

    The name of the benchmark.

Returns:

  • Benchmark –

    The loaded or newly created Benchmark object.

load(name: str) -> Benchmark staticmethod

Load a benchmark from the database by its name.

Parameters:

  • name (str) –

    The name of the benchmark to load.

Returns:

load_all() -> list[Benchmark] staticmethod

Load all benchmarks from the database.

Loading all benchmarks from the database can be a slow operation and should be used sparingly.

Returns:

  • list[Benchmark] –

    A list of Benchmark objects representing all benchmarks in the database. If no benchmarks are found, an empty list is returned.

reset(*, mode: ResetLevel | Literal['All', 'Unfinished', 'Failed']) -> None

Clear results for the benchmark.

Removes algorithm, metric, and feature results from the database. Metric results are cascaded (cleared along with algorithms). After the operation, the entity is reloaded from the database so its in-memory state is fully consistent.

Parameters:

  • mode (ResetLevel | Literal['All', 'Unfinished', 'Failed']) –

    ResetLevel.ALL or "All" clears all results unconditionally. ResetLevel.UNFINISHED or "Unfinished" clears only non-DONE results (includes failed). ResetLevel.FAILED or "Failed" clears only FAILED results. No default — must be explicitly provided.

set_modelset(modelset: str | ModelSet) -> None

Set the modelset for the benchmark.

This method sets the modelset for the benchmark. Changing the modelset can affect the results of the benchmark. Therfore its recommended to not change the modelset after the benchmark has been created. If it is necessary, the results of the benchmark should be deleted and the benchmark itself should be re-run.

Parameters:

  • modelset (str | ModelSet) –

    Set the modelset for the benchmark to this modelset. It can be the name of the modelset or the modelset itself.

remove_modelset() -> None

Remove the modelset from the benchmark.

This method removes the modelset from the benchmark. If the modelset is not set, this method does nothing. After removing the modelset, the results of the benchmark may be invalid.j

get_feature(name: str) -> FeatureEntity

Get a feature by its name from a benchmark.

If the feature is not present, an error will be raised.

Parameters:

  • name (str) –

    The name of the feature to be retrieved.

Raises:

  • DataNotExistError –

    Raised if its name couldn't retrieve the feature.

add_feature(name: str, feature: BaseFeature) -> FeatureEntity

Add a feature to the benchmark with a given name.

This method adds a feature to the benchmark. The name must be unique within the benchmark. When the benchmark is rerun, the feature will be used to calculate the metrics for each algorithm result.

Also, the feature must be defined in the registry. If this isn't the case, an error will be received. To fix this, please check the documentation on how to do this.

Parameters:

  • name (str) –

    Name of the feature to add.

  • feature (BaseFeature) –

    The feature to add.

Returns:

  • Feature –

    The added feature.

remove_feature(feature: str | FeatureEntity) -> None

Remove a feature from the benchmark.

Parameters:

  • feature (str | FeatureEntity) –

    The name of the feature to remove or the feature object itself. Make sure to use the FeatureUserModel object and not only an IFeature object. This is important because the feature name is used to identify the feature.

get_metric(name: str) -> MetricEntity

Get a metric by its name from a benchmark.

If the metric is not present, an error will be raised.

Parameters:

  • name (str) –

    The name of the metric to be retrieved.

Raises:

  • DataNotExistError –

    Raised if its name couldn't retrieve the metric.

add_metric(name: str, metric: BaseMetric) -> MetricEntity

Add a metric to the benchmark with a given name.

This method adds a metric to the benchmark. The name must be unique within the benchmark. When the benchmark is rerun, the metric will be calculated for each algorithm result.

Also, the metric must be defined in the registry. If this isn't the case, an error will be received. To fix this, please check the documentation on how to do this.

Parameters:

  • name (str) –

    The name of the metric to add.

  • metric (BaseMetric) –

    An instance of the metric to add.

Returns:

  • Metric –

    The added metric.

remove_metric(metric: str | MetricEntity) -> None

Remove a metric from the benchmark.

Parameters:

  • metric (str | MetricEntity) –

    The name of the metric to remove or the metric object itself. Make sure to use the MetricUserModel object and not only an IMetric object. This is important because the metric name is used to identify the metric.

get_algorithm(name: str) -> AlgorithmEntity

Get an algorithm by its name from a benchmark.

If the algorithm is not present, an error will be raised.

Parameters:

  • name (str) –

    The name of the algorithm to be retrieved.

Raises:

  • DataNotExistError –

    Raised if its name couldn't retrieve the feature.

add_algorithm(name: str, algorithm: IAlgorithm[Any] | BaseAlgorithmSync | BaseAlgorithmAsync[Any]) -> AlgorithmEntity

Add an algorithm to the benchmark with a given name.

This method adds an algorithm to the benchmark. The name must be unique within the benchmark. When the benchmark is rerun, the results for this algorithm will be calculated.

Also, the algorithm must be defined in the registry. If this isn't the case, an error will be received. To fix this, please check the documentation on how to do this.

Parameters:

Returns:

  • AlgorithmEntity –

    The added algorithm.

remove_algorithm(algorithm: str | AlgorithmEntity) -> None

Remove an algorithm from the benchmark.

Parameters:

  • algorithm (str | AlgorithmEntity) –

    The name of the algorithm to remove or the algorithm object itself. Make sure to use the AlgorithmUserModel object and not only an IAlgorithm object. This is important because the algorithm name is used to identify the algorithm.

get_plot(name: str) -> PlotEntity

Get a plot by its name from a benchmark.

If the plot is not present, an error will be raised.

Parameters:

  • name (str) –

    The name of the algorithm to be retrieved.

Raises:

  • DataNotExistError –

    Raised if its name couldn't retrieve the plot.

add_plot(name: str, plot: BasePlot) -> PlotEntity

Add a plot to the benchmark with a given name.

This method adds a plot to the benchmark. The name must be unique within the benchmark. When the benchmark is rerun, the results for this plot will be calculated.

Also, the plot must be defined in the registry. If this isn't the case, an error will be received. To fix this, please check the documentation on how to do this.

Parameters:

  • name (str) –

    The name of the plot to add.

  • plot (BasePlot) –

    The plot to add.

Returns:

  • Plot –

    The added plot.

remove_plot(plot: str | PlotEntity) -> None

Remove a plot from the benchmark.

Parameters:

  • plot (str | Plot) –

    The name of the plot to remove or the plot object itself. Make sure to use the Plot object and not only an IPlot object. This is important because the plot name is used to identify the plot.

run_features() -> None

Calculate all configured features for all models of this benchmark.

run_algorithms() -> None

Calculate all configured features for all models of this benchmark.

run_plots() -> None

Execute all plots registered in the benchmark.

Iterates through all plots in the benchmark, validates each plot against the benchmark data, and executes the plot generation. Each plot is validated before execution to ensure required data (metrics, features, etc.) is available. Plot execution is sequential and follows the order defined in the benchmark configuration.

Raises:

  • RuntimeError –

    If plot validation or execution fails. The RuntimeError wraps the underlying error, which may be PlotRunError (for validation failures) or UnknownLunaBenchError (for unexpected execution errors). Only raised in FAIL_ON_ERROR mode; in CONTINUE_ON_ERROR mode, errors are logged as warnings instead.

Notes

In FAIL_ON_ERROR mode, the method stops at the first validation or execution error. In CONTINUE_ON_ERROR mode, errors are logged and execution continues with remaining plots.

add_dependencies() -> None

Add any required dependencies for the benchmark execution.

run(*, retry_uncompleted: bool = False) -> None

Execute the benchmark.

Parameters:

  • retry_uncompleted (bool, default: False ) –

    If True, clear only non-DONE (uncompleted) results before running so that failed or incomplete components are retried while DONE results are preserved. Defaults to False.

export(exporter: Exporter[T]) -> T

Export all benchmark results using the given exporter strategy.

Builds a full BenchmarkResultContainer (all feature, metric, and algorithm results of this benchmark) and hands it to the exporter. Any object implementing the Exporter protocol can be used, so custom export formats do not require changes to this class::

benchmark.export(CsvExporter(delimiter=";"))  # built-in exporter
benchmark.export(MyArrowExporter())  # custom exporter

Parameters:

  • exporter (Exporter[T]) –

    The exporter strategy that converts benchmark results into the target format.

Returns:

  • T –

    The exported payload, e.g. str for CSV/JSON or a pd.DataFrame for the DataFrame exporter.

to_csv(path: str | Path | None = None, *, delimiter: str = ',', quoting: CsvQuoting = 'minimal', include_solution: bool = False) -> str | None

Render all benchmark results as CSV.

Convenience wrapper for self.export(CsvExporter(...)). Mirrors pandas.DataFrame.to_csv: if path is given, the CSV is written to that file and None is returned; otherwise the CSV is returned as a string.

Parameters:

  • path (str | Path | None, default: None ) –

    File path to write the CSV to. If None, the CSV is returned as a string instead. Defaults to None.

  • delimiter (str, default: ',' ) –

    Field delimiter. Defaults to ",".

  • quoting (CsvQuoting, default: 'minimal' ) –

    Quoting style: "minimal", "all", "nonnumeric", or "none". Defaults to "minimal".

  • include_solution (bool, default: False ) –

    Whether to include the serialized solution column (base64-encoded). Defaults to False.

Returns:

  • str | None –

    The benchmark results rendered as CSV, or None if written to path.

to_json(path: str | Path | None = None, *, indent: int | None = None, orient: JsonOrient = 'records', include_solution: bool = False) -> str | None

Render all benchmark results as JSON.

Convenience wrapper for self.export(JsonExporter(...)). Mirrors pandas.DataFrame.to_json: if path is given, the JSON is written to that file and None is returned; otherwise the JSON is returned as a string.

Parameters:

  • path (str | Path | None, default: None ) –

    File path to write the JSON to. If None, the JSON is returned as a string instead. Defaults to None.

  • indent (int | None, default: None ) –

    Indentation width for pretty-printing; None for compact output. Defaults to None.

  • orient (JsonOrient, default: 'records' ) –

    JSON layout passed to DataFrame.to_json. Defaults to "records".

  • include_solution (bool, default: False ) –

    Whether to include the serialized solution column (base64-encoded). Defaults to False.

Returns:

  • str | None –

    The benchmark results rendered as JSON, or None if written to path.

to_dataframe(*, include_solution: bool = False) -> pd.DataFrame

Return all benchmark results as a single DataFrame.

Convenience wrapper for self.export(DataFrameExporter(...)): algorithm results form the row spine (one row per (algorithm, model)), metrics merge on (algorithm, model), and features merge on model. Feature values are repeated across algorithms for the same model since features are model-level.

Parameters:

  • include_solution (bool, default: False ) –

    Whether to include the serialized solution as a solution column. Defaults to False.

Returns:

  • DataFrame –

    A DataFrame with columns algorithm, model, plus one column per result field of each feature and metric.

list_feature_classes() -> list[type[BaseFeature]]

Return the feature classes registered on this benchmark.

list_metrics_classes() -> list[type[BaseMetric]]

Return the metric classes registered on this benchmark.

list_plots_classes() -> list[type[BasePlot]]

Return the plot classes registered on this benchmark.

list_algorithms() -> list[tuple[type[BaseAlgorithmSync | BaseAlgorithmAsync[Any]], dict[str, Any]]]

Return the algorithm classes registered on this benchmark.

ModelSet

Bases: ModelSetEntity

Set of models.

Represents a collection of models with operations for creating, loading, adding, removing, and deleting models.

Attributes:

  • id (int) –

    The unique identifier for the model set.

  • name (str) –

    The name of the model set.

  • models (list[ModelMetadata]) –

    A list of ModelData objects representing the models in this set.

create(modelset_name: str) -> ModelSet staticmethod

Create a new model set with the given dataset name.

Creates a new model set using the provided dataset name and a model set creation use case.

Parameters:

  • modelset_name (str) –

    The name of the dataset.

Returns:

  • ModelSet –

    An instance of ModelSet representing the successfully created model set.

load(name: str) -> ModelSet staticmethod

Load a model set by its ID.

Retrieves a model set from the database using its unique identifier.

Parameters:

  • name (str) –

    The unique name of the model set to load.

Returns:

load_all() -> list[ModelSet] staticmethod

Load all model sets from the database.

Retrieves all model sets stored in the database.

Returns:

load_all_models() -> list[ModelMetadata] staticmethod

Load all models from the database.

Retrieves all models stored in the database, regardless of which model set they belong to.

Returns:

  • list[ModelMetadata] –

    A list of ModelData objects representing all models in the database.

add(model: Model | Iterable[Model]) -> None

Add a model to this model set.

Adds the specified model to this model set and updates the model set's state.

Parameters:

  • model (Model | Iterable[Model]) –

    The model to add to this model set. If it is an iterable of models (e.g. list, tuple, generator, iterator), all models will be added.

remove_model(model: Model) -> None

Remove a model from this model set.

Removes the specified model from this model set and updates the model set's state.

Parameters:

  • model (Model) –

    The model to remove from this model set.

delete() -> None

Delete this model set from the database.

Permanently removes this model set from the database.