pipeline.experiment

Classes

Experiment

A class representing an experimental setup for evaluating explanation methods on specific dataset and

Module Contents

A class representing an experimental setup for evaluating explanation methods on specific dataset and neural network models.

It should be ensured that the class corresponding to the dataset contains the generate_model() method in order for a model to be generated in the case that no model is defined for the experiment at initialization.

data

The class of the dataset used for experiments.

Type:: torch.utils.data.Dataset

models

List of neural network models to apply the explanation methods on.

Type:: list | NoneType

methods

List of explanation methods to apply and evaluate.

Type:: list

metrics

List of evaluation metrics to compute.

Type:: list

seeds

List of random seeds to use for the instantiation of the dataset.

Type:: list

method_seeds

List of random seeds to use for explanation methods.

Type:: list

data_params

Additional parameters to be passed to the instantiation of the dataset.

Type:: dict

Initializes an Experiment instance.

Parameters:

data (Any) – The dataset or data class to be used for experiments. Can be passed in as an instantiated object or as a subclass of torch.utils.data.Dataset.
models (list | NoneType | torch.nn.Module) – List of neural network models to apply the explanation methods on.
methods (Any) – List of explanation methods to apply and evaluate.
metrics (Any | NoneType, optional) – List of evaluation metrics to compute. Defaults to None.
seeds (int | list) – List of random seeds to use for the instantiation of the dataset. Defaults to 0.
method_seeds (int | list) – List of random seeds to use for explanation methods. Defaults to 0.
data_params (dict | NoneType, optional) – Additional parameters to be passed during the instantiation of the dataset. Defaults to None.
name (str, optional) – string representing name of the experiment.

Raises:

Exception – If input to any attribute initialization method is invalid.

seeds

data

data_params = None

models

metrics = [None]

methods

method_seeds

exp_name = None

get_data(seed: int) → Any

Returns the dataset instance generated with the specified seed.

Parameters:: seed (int) – the seed for instantiating the dataset.
Returns:: The dataset instance with the specified seed.
Return type:: torch.utils.data.Dataset

get_models(data_instance: Any) → List[torch.nn.Module]

Returns the list of neural networks to apply the explanation methods on.

A default neural network compatible with the given dataset will be generated if the Experiment object has None as its models.

Parameters:: data_instance (torch.utils.data.Dataset) – The dataset instance.
Returns:: List of neural networks to apply the explanation methods on.
Return type:: list

get_methods(data_instance: Any) → List[Any]

Returns the list of explanation methods to apply and evaluate.

Parameters:: data_instance (torch.utils.data.Dataset) – The dataset instance and a placeholder to keep input standardized.
Returns:: List of explanation methods to apply and evaluate.
Return type:: list

get_metrics(data_instance: Any) → List[Any]

Returns the list of evaluation metrics to compute.

Parameters:: data_instance (torch.utils.data.Dataset) – The dataset instance.
Returns:: List of evaluation metrics to compute.
Return type:: list

_init_seeds(seeds: int) → List[int]

Initializes the seeds attribute and transforms the input to the desired datatype.

Parameters:: seeds (int | list) – Random seeds to use for data.
Returns:: List of random seeds to use for the instantiation of the dataset.
Return type:: list
Raises:: Exception – If input to seeds initialization is not an integer or an Iterable of integers.

_init_data(data: Any) → Any

Initializes the data attribute.

Parameters:: data (type) – The instantiated dataset or data class.
Returns:: The dataset or data class.
Return type:: torch.utils.data.Dataset | type
Raises:: Exception – If input to data initialization is neither a torch.utils.data.Dataset instance or subclass of torch.utils.data.Dataset.

_init_methods(methods: Any) → Any

Initializes the methods attribute.

Parameters:: methods (list | NoneType) – List of explanation methods.
Returns:: List of explanation methods.
Return type:: list
Raises:: Exception – If input to methods initialization is None.

_init_metrics(metrics: Any) → Any

Initializes the metrics attribute.

Parameters:: metrics (list | NoneType) – List of evaluation metrics.
Returns:: List of evaluation metrics.
Return type:: list
Raises:: Exception – If input to metrics initialization is None and the dataset does not provide a default metric.

_init_models(models: Any) → Any

Initializes the models attribute and transforms it to the desired datatype.

Parameters:: models (list | torch.nn.Module) – Neural network models.
Returns:: List of neural network models.
Return type:: list
Raises:: Exception – If input to models initialization is not an torch.nn.Module object or Iterable of torch.nn.Module objects.

_init_data_params(data_params: Dict) → Dict

Initializes the data_params attribute.

Parameters:: data_params (dict) – Additional parameters for the instantiation of the dataset.
Returns:: Dictionary of additional data parameters.
Return type:: dict
Raises:: Exception – If input to data parameters initialization is not a dictionary.

_init_method_seeds(method_seeds: int | collections.abc.Iterable[int]) → collections.abc.Iterable[int]

Initializes the method seeds attribute and transforms the input to the desired datatype.

Parameters:: method_seeds (int | list) – Random seeds to use for applying the explanation methods.
Returns:: List of random seeds for applying the explanation methods.
Return type:: list
Raises:: Exception – If input to method seeds initialization is not an integer nor an Iterable of integers.

_verify_metric(metrics: Any) → None

Verifies whether evaluation metrics have unique labels and each of them is in a valid datatype.

Parameters:: metrics (list) – A list of evaluation metrics.
Raises:: Exception – If evaluation metric is defined in an invalid datatype or if evaluation metrics do not have unique labels.