pipeline ======== .. py:module:: pipeline Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/pipeline/experiment/index /autoapi/pipeline/pipeline/index /autoapi/pipeline/results/index Attributes ---------- .. autoapisummary:: pipeline.methods Classes ------- .. autoapisummary:: pipeline.BasePipeline pipeline.Pipeline pipeline.ExperimentPipeline pipeline.Example pipeline.Results pipeline.Experiment Functions --------- .. autoapisummary:: pipeline.to_device Package Contents ---------------- .. py:class:: BasePipeline(batch_size: int = 50, default_target: Optional[Any] = None, results: Optional[xaiunits.pipeline.results.Results] = None, n_examples: Optional[int] = None) A base class for creating an end-to-end pipeline object that applies feature attribution methods and evaluates. This is a base class is intended to support other pipeline classes that are build upon it. Explanation methods to be ran and evaluated must contain the .attribute method. The pipeline uses batching when applying the explanation methods and evaluating them over the evaluation metrics. The results of the explanation method's performance will be processed and cleaned by within the pipeline, and is easily callable by the user. Non-deterministic explanation methods are accomodated too, where multiple trials runs is possible with the pipeline. Alongside evaluation performance, runtime performance is measured in the pipeline too. For customisation of the explanation methods and evaluation metrics, applying a wrapper to them may be necessary prior to inputing to the pipeline. .. attribute:: batch_size Number of data samples to be processed by the models and for the explanation methods to be applied on within each batch. :type: int .. attribute:: default_target Type or values of the target that is expected to be outputed by the models. :type: Any .. attribute:: n_examples Number of worst and best performing data samples stored based on evaluation metric scores. :type: int | NoneType .. attribute:: results Instance of Results for storing and processing the evaluation results. :type: pipeline.Results | NoneType Initializes a BasePipeline object. :param batch_size: Number of data samples to be processed by the models and for the explanation methods to be applied on within each batch. Defaults to 50. :type batch_size: int :param default_target: Type or values of the target that is expected to be outputed by the models. Defaults to None. :type default_target: Any, optional :param n_examples: Number of worst and best performing data samples stored based on evaluation metric scores. Defaults to None. :type n_examples: int | NoneType, optional :param results: Instance of Results for storing and processing the evaluation results. Defaults to None. :type results: pipeline.Results | NoneType .. py:attribute:: batch_size :value: 50 .. py:attribute:: default_target :value: None .. py:attribute:: n_examples :value: None .. py:attribute:: results .. py:method:: _init_attr(attr_input: collections.abc.Iterable) -> List Initializes the attributes by checking its type and converts it into a list. :param attr_input: The object or collection of objects relevant to the attribute of interest. :type attr_input: Iterable | NoneType | Any :returns: A list containing all the elements of the input. :rtype: list .. py:method:: _single_explanation_attribute(data: Any, model: Any, method: Any, metrics: Any, method_seed: Optional[int] = None, device: Optional[torch.device] = None, trial_group_name: Optional[str] = None) -> None Computes the score of a single explanation method on a neural network and evaluates the explanation method with respect to all evaluation metrics. The explanation method is applied over the dataset by batches, where the a batch with size batch_size is generated by the dataset iteratively and the attribution score for it is computed. Depending on the explanation method the attribution score may be fixed or unfixed for each sample within the same batch. The total runtime of applying the explanation method to each batch will be measured. Evaluation metrics are similarly calculated individually for the attribution score of each batch. If the n_samples attribute is not None, the batches that give the best and worst n_samples number of evaluation metric scores will be stored. The evaluation and runtime measurements will be stored inside the results attribute in batches as a dictionary. :param data: Dataset of inputs :type data: torch.util.data.Dataset :param model: Neural network to for the explanation method to be applied on :type model: torch.nn.Module :param method: Explanation method to be applied to the model and evaluated. :type method: methods.methods_wrapper.Wrapper :param metrics: List of evaluation metrics to be applied. :type metrics: list :param method_seed: The seed set for reproducibility of the explanation method. It will not be set if it is None. Defaults to None. :type method_seed: int | NoneType, optional :param device: The device which the objects will be stored on. Defaults to None. :type device: torch.device, optional :param trial_group_name: string representing name of the trial group name :type trial_group_name: str .. py:method:: _apply_attribution_method(feature_inputs: torch.Tensor, method_instance: Any, batch_results: Dict) -> Any Computes the attribution evaluation for the given feature inputs using the explanation method of interest. The time it takes to compute the feature attributions is appended to the results for the current batch of data. :param feature_inputs: The input features for the current batch. :type feature_inputs: torch.Tensor :param method_instance: Explanation method used to compute the feature attributions. :type method_instance: Any :param batch_results: Recorded results for the current batch of data. :type batch_results: dict :returns: The computed feature attributions for the given feature inputs. :rtype: torch.Tensor .. py:method:: _store_top_n(n: int, key: Tuple, batch_scores: torch.Tensor, attribute: torch.Tensor, batch_info: Tuple, example_type: str) -> None Keeps a running heap of the top n examples with the highest scores, for each key. Supports both 'max' / 'min' example_type, for the n highest / lowest scores. .. rubric:: Notes - If `batch_scores` is a single value, it is repeated to match the batch size. - The heap of examples are stored in `results.examples`. :param n: Number of examples to store. :type n: int :param key: The key to identify which (method, model, metric) these examples are for. :type key: tuple :param batch_scores: A tensor of scores for the current batch of examples. :type batch_scores: torch.Tensor :param attribute: The attributes tensor for the current batch of examples. :type attribute: torch.Tensor :param batch_info: A tuple containing feature inputs, labels, targets, and context for the batch. :type batch_info: tuple :param example_type: Specifies the type of scoring mechanism ('max' or 'min') to use for ranking. :type example_type: str .. py:method:: unpack_batch(batch: Union[torch.Tensor, Tuple], device: Optional[torch.device] = None) -> Tuple Unpacks a batch into (feature_inputs, labels, context) depending on the format. :param batch: A batch of data which can be a single tensor (feature_inputs), a tuple of two tensors (feature_inputs and labels), or a tuple of three elements (feature_inputs, labels, and context). :type batch: torch.Tensor | Tuple :param device: The device which the objects will be stored on. Defaults to None. :type device: torch.device, optional :raises TypeError: If dataset is not in a supported format. .. py:method:: set_default_target(model: Any, feature_inputs: torch.Tensor, y_labels: Any) -> Any Calculates the target based on self.default_target. Two special keywords: - If self.default_target is 'y_labels', returns the y_labels - If self.default_target is 'predicted_class', returns y=model(feature_inputs) Three other standard options: - If self.default_target is None, returns None. - If self.default_target is an integer, returns that integer. - self.target may also be a tuple or tensor matching the batch size. These are the standard options that will be used by the default evaluation methods. If an alternative option is needed, the user can override the evaluation metric function. :param model: The model to use for predictions if `self.default_target` is 'predicted_class'. :type model: torch.nn.Module :param feature_inputs: The input features to the model. :type feature_inputs: torch.Tensor :param y_labels: The actual labels for the input features, if `self.default_target` is 'y_labels' :type y_labels: torch.Tensor :raises ValueError: If the value of default_target is invalid. .. py:method:: _all_models_explanation_attributes(data: List[Any], models: List[Any], methods: List[Any], metrics: List[Any], method_seeds: List[int], device: Optional[torch.device] = None, trial_group_name: Optional[str] = None) -> None Applies every explanation methods on each of the neural network models and they are evaluated against the evalaution metrics. :param data: Dataset the neural netowrk models are operating on. :type data: torch.utils.data.Dataset :param models: List of neural network models for the explanation methods to be applied on. :type models: list :param methods: List of explanation methods to apply and evaluated. :type methods: list :param metrics: List of evaluation metrics to apply. :type metrics: list :param method_seeds: List of random seeds for applying the explanation method. :type method_seeds: list :param device: The device which the objects will be stored on. Defaults to None. :type device: torch.device, optional :param trial_group_name: string representing name of the trial group name :type trial_group_name: str .. py:class:: Pipeline(models: Any, datas: Any, methods: Any, metrics: Optional[Any] = None, method_seeds: Optional[int] = None, batch_size: int = 50, default_target: Optional[str] = None, results: Optional[xaiunits.pipeline.results.Results] = None, n_examples: Optional[int] = None, name: Optional[int] = None) Bases: :py:obj:`BasePipeline` A pipeline that evaluates the performance of the explanation methods on the neural network models with respect to evaluation metrics of interest. Single or multiple models, Single or multiple datasets, explanation methods, evaluation methods are supported. Explanation methods must contain the .attribute method. The pipeline uses batching when applying the explanation methods and evaluating them over the evaluation metrics. The results of the explanation method's performance will be processed and cleaned by within the pipeline, and is easily callable by the user. Non-deterministic explanation methods are accomodated too, where multiple trials runs is possible with the pipeline. If no evaluation metric is provided explicitly as an argument to the instantiation of a Pipeline object, then a default metric will be used from one of the provided datasets, subject to the availability of it in any of the datasets. Alongside evaluation performance, runtime performance is measured in the pipeline too. For customization of the explanation methods and evaluation metrics, applying a wrapper to them may be necessary prior to inputting to the pipeline. Inherits from: BasePipeline: Base class for setting up a pipeline that applies feature attribution methods and evaluates. .. attribute:: models List of neural network models for the explanation methods to be applied on. :type: list .. attribute:: datas Dataset object for generating the data samples that are compatible to the neural networks provided. :type: list .. attribute:: methods List of explanation methods to be evaluated for their performance on producing attribution scores. :type: list .. attribute:: metrics List of evaluation metrics for the explanation methods to be evaluated against. :type: list .. attribute:: method_seed List of seeds used for repeating and replicating the results over a single or multiple trials. :type: list Initializes a Pipeline object. All models, explanation methods, evaluation metrics, method seed inputs will be forced to be list that contains all the relevant items. Labels for the evaluation metrics will be generated based on methods.methods_wrapper.MetricsWrapper.__name__ and must be unique. :param models: Single or iterable collection of neural network models for explanation methods to be applied on. :type models: torch.nn.Module | Iterable :param data: Dataset object for generating the data samples that are compatible to the neural networks provided. :type data: torch.utils.data.Dataset | Iterable :param methods: Single or iterable collection of explanation methods to be evaluated. :type methods: methods.methods_wrapper.Wrapper | captum.attr._utils.attribution.Attribution | Iterable :param metrics: Single or iterable collection of evaluation metrics used for evaluating explanation methods. Defaults to None. :type metrics: methods.methods_wrapper.MetricsWrapper | Iterable | None :param method_seed: Seeds for replicating explanation methods results over multiple trials. Defaults to None, and a random seed will be picked where no replicability will be enforced. :type method_seed: int | Iterable, optional :param batch_size: Number of data samples to be processed by the models and for the explanation methods to be applied on within each batch. Defaults to 50. :type batch_size: int :param default_target: Type or values of the target that is expected to be outputted by the models. Defaults to None. :type default_target: Any, optional :param model_names: Single or iterable collection of model name used the purpose of distinguishing between models. Defaults to None. :type model_names: str | Iterable, optional :param results: Instance of Results for storing and processing the evaluation results. Defaults to None, and an empty Results instance will be used. :type results: Results, optional :param n_examples: Number of worst and best performing data samples stored based on evaluation metric scores. Defaults to None. :type n_examples: int, optional :param name: string representing name of the trial group. :type name: str, optional :raises Exception: If the collection of evaluation metrics do not all have unique names. .. py:attribute:: models .. py:attribute:: datas .. py:attribute:: methods .. py:attribute:: method_seed .. py:attribute:: trial_group_name :value: None .. py:method:: run(device: Optional[torch.device] = None) -> xaiunits.pipeline.results.Results Alias for explanation_attribute method. Runs the pipeline and returns the results. :param device: The device which the objects will be stored on. Defaults to None. :type device: torch.device, optional :returns: the evaluation results from running the pipeline. :rtype: pipeline.Results .. py:method:: explanation_attribute(device: Optional[torch.device] = None) -> xaiunits.pipeline.results.Results Applies every explanation methods on each of the neural network models and dataset, which are then evaluated against the evaluation metrics. If an evaluation metric is not wrapped as a methods.methods_wrapper.Wrapper instance, manual wrapping will be performed. The explanation methods will be repeated a variable number of times depending on the method_seed attribute. All evaluation and runtime results will be stored within the results attribute in its unprocessed form. Each of the neural network models will be forced to evaluation mode. :param device: The device which the objects will be stored on. Defaults to None. :type device: torch.device, optional :returns: the evaluation results from running the pipeline. :rtype: pipeline.Results .. py:method:: _init_none_metric() -> Any Returns the default metric from any of the dataset inputted, if available. It iterates over the provided data and checks if any of them have a default metric attribute. If a default metric is found, it returns that metric. Otherwise, it raises an exception. :returns: The class that wraps the default metric from one of the datasets inputed. :rtype: type :raises Exception: If none of the provided datasets have a default metric. .. py:class:: ExperimentPipeline(experiments: Union[xaiunits.pipeline.experiment.Experiment, List[xaiunits.pipeline.experiment.Experiment]], batch_size: int = 50, default_target: Optional[Any] = None, results: Optional[xaiunits.pipeline.results.Results] = None, n_examples: Optional[int] = None) Bases: :py:obj:`BasePipeline` A pipeline that evaluates the performance of the explanation methods on the neural network models with respect to evaluation metrics of interest. This pipeline only expects a list of Experiment Class, in which data, models, explanation methods, evaluation metric and seeds are specified. Inherits from: BasePipeline: Base class for setting up a pipeline that applies feature attribution methods and evaluates. .. attribute:: experiments List experiments to be ran. :type: list[pipeline.Experiment] Initializes an ExperimentPipeline object. :param experiments: List of experiments to be run. :type experiments: List[Experiment] :param batch_size: Number of data samples to be processed by the models and for the explanation methods to be applied on within each batch. Defaults to 50. :type batch_size: int :param default_target: Type or values of the target that is expected to be outputed by the models. Defaults to None. :type default_target: Any, optional :param results: Instance of Results for storing and processing the evaluation results. Defaults to None, and an empty Results instance will be used. :type results: Results, optional :param n_examples: Number of worst and best performing data samples stored based on evaluation metric scores. Defaults to None. :type n_examples: int, optional .. py:attribute:: experiments .. py:method:: run(device: Optional[torch.device] = None) -> xaiunits.pipeline.results.Results Alias for explanation_attribute method. Runs the pipeline and returns the results. :param device: The device which the objects will be stored on. Defaults to None. :type device: torch.device, optional :returns: the evaluation results from running the pipeline. :rtype: pipeline.Results .. py:method:: explanation_attribute(device: Optional[torch.device] = None) -> xaiunits.pipeline.results.Results Using the items stored in each Experiment object, apply every explanation methods on each of the neural network models, which are then evaluated against the evalaution metrics. Depending on the type of object that is given to represent the dataset, multiple datasets will be instantiated using the various seeds provided, or simply using the single dataset object supplied. If an evaluation metric is not wrapped as a methods.methods_wrapper.Wrapper instance, manual wrapping will be peformed. The explanation methods will be repeated a variable number of times depending on the method_seed attribute. All evaluation and runtime results will be stored within the results attribute in its unprocessed form. Each of the neural network models will be forced to evaluation mode. :param device: The device which the objects will be stored on. Defaults to None. :type device: torch.device, optional :returns: the evaluation results from running the pipeline. :rtype: pipeline.Results .. py:function:: to_device(data: Any, device: Optional[torch.device] = None) -> Any Moves data to device if the data is a tensor or a dict of tensors. :param data: Data to be moved. :type data: Any :param device: Device to move the data to. :type device: torch.device :returns: Data object that is stored in the specified device. :rtype: Any .. py:data:: methods .. py:class:: Example Stores a datapoint with its attributions and score, for ranking top n examples. .. py:attribute:: score_for_ranking :type: float .. py:attribute:: score :type: float .. py:attribute:: attribute :type: torch.Tensor .. py:attribute:: feature_inputs :type: torch.Tensor .. py:attribute:: y_labels :type: torch.Tensor .. py:attribute:: target :type: int | None .. py:attribute:: context :type: dict | None .. py:attribute:: example_type :type: str .. py:method:: __post_init__() -> None Adjusts the `score_for_ranking` to be negative for "min" examples. This is so that we can use heapq, a min heap, as a max heap. :raises ValueError: If the value of example_type is not accepted. .. py:class:: Results Object that records and processes the results of experiments from a subclass of BasePipeline. .. attribute:: data List of all stored experiment results from a pipeline object. :type: list .. attribute:: metrics List of the names of all evaluation metrics recorded in data. :type: list .. attribute:: examples Dictionary containing the data samples that output the maximum or minimum scores with respect to the evaluation metrics of interest. :type: dict Initializes a Results object. .. py:attribute:: raw_data :value: [] .. py:attribute:: examples .. py:method:: append(incoming) Appends the new result to the collection of results stored. :param incoming: New result to be appended to the existing results. :type incoming: dict .. py:property:: data :type: pandas.DataFrame Processes the raw_data list into a DataFrame, flattening over the batch dimension. For each data instance, the value under the 'attr_time' column will be the time it took for the batch, it belongs to, to compute its respective attribution scores, divided by the size of the batch. :returns: The processed results, one row per datapoint. :rtype: pandas.DataFrame .. py:method:: process_data() -> None Convenience method for accessing the self.data property. :returns: The processed results, one row per datapoint. :rtype: pandas.DataFrame .. py:method:: print_stats(metrics: Optional[List[Any]] = None, stat_funcs: List[str] = ['mean', 'std'], index: List[str] = ['data', 'model', 'method'], initial_mean_agg: List[str] = ['batch_id', 'batch_row_id'], time_unit_measured: str = 'dataset', decimal_places: int = 3, column_index: List = []) -> pandas.DataFrame Prints the results in the form of a pivot table for each of the statistic required. The indices of the printed table correspond to the model and explanation method, and the columns of the printed table correpond to the evaluation metrics. The values of the printed table are the statistics calculated across the number of experiment trials. When mean and standard deviation are needed, a single pivot table will be printed that records both of them. :param metrics: A list of the names of all metrics to be printed. Defaults to None. :type metrics: list, optional :param stat_funcs: A list of aggregation functions that are required to be printed. It can be a str if only a single type of aggregation is required. Supports the same preset and custom aggregation functions as supported by pandas. Defaults to ['mean', 'std']. :type stat_funcs: list | str :param index: A list of the names of the columns to be used as indices in the pivot table. Defaults to ['data', 'model', 'method'] :type index: list :param initial_mean_agg: A list of the columns to be used for the initial mean. For example, if we want to calculate the standard deviation between experiments, we take the mean over batch_id and batch_row_id first, then calculate the standard deviation. Defaults to ['batch_id', 'batch_row_id']. :type initial_mean_agg: list :param time_unit_measured: The unit for which the time to perform the attribution method is calculated and aggregated. Defaults to 'dataset', and only 3 values are supported inputs: - 'dataset': time needed to apply explanation method on each dataset - 'batch': time needed to apply explanation method on each batch - 'instance': time needed to apply explanation method on each data instance. It is estimate derived from the batch time. :type time_unit_measured: str :param decimal_places: The decimal places of the values displayed. Defaults to 3 decimal places. :type decimal_places: int :param column_index: A list of column names to unpivot, in addition to Stats and Metrics. Defaults to [], so just Stats and Metrics are used as columns. :type column_index: list .. py:method:: print_all_results() -> None Prints all the data as a wide table. .. py:method:: save(filepath) Saves the data stored as a .pkl file. :param filepath: Path for the .pkl file to be saved in. :type filepath: str .. py:method:: load(filepath: str, overwrite: bool = False) -> None Loads the data from a .pkl file. The data loaded will be concatenated with the existing data store in the object if and only if overwrite is False. :param filepath: Path of the .pkl file to load the data from. :type filepath: str :param overwrite: True if and only if the data loaded overwrites any existing data stored. Defaults to False. :type overwrite: bool .. py:method:: print_max_examples() -> None Prints in descending order the collection of examples that give the maximum evaluation metric scores. .. py:method:: print_min_examples() -> None Prints in ascending order the collection of examples that give the minimum evaluation metric scores. .. py:method:: _attr_time_summing(time_unit_measured: str) -> pandas.DataFrame Aggregates attribution time based on the specified time unit and formats the data in a correct format with the aggregated attribution times as part of the metrics. :param time_unit_measured: The unit for which the time to perform the attribution method is calculated and aggregated. Only allows inputs "dataset", "batch", "instance". :type time_unit_measured: str :returns: pandas.DataFrame object with aggregated attribution time results formatted correctly. :rtype: pandas.DataFrame :raises Exception: If an invalid time unit is provided. .. py:class:: Experiment(data: Any, models: Union[List[torch.nn.Module], torch.nn.Module, None], methods: Union[Any, List[Any]], metrics: Optional[Union[Any, List[Any]]] = None, seeds: int = 0, method_seeds: int = 0, data_params: Optional[Dict] = None, name: Optional[str] = None) A class representing an experimental setup for evaluating explanation methods on specific dataset and neural network models. It should be ensured that the class corresponding to the dataset contains the generate_model() method in order for a model to be generated in the case that no model is defined for the experiment at initialization. .. attribute:: data The class of the dataset used for experiments. :type: torch.utils.data.Dataset .. attribute:: models List of neural network models to apply the explanation methods on. :type: list | NoneType .. attribute:: methods List of explanation methods to apply and evaluate. :type: list .. attribute:: metrics List of evaluation metrics to compute. :type: list .. attribute:: seeds List of random seeds to use for the instantiation of the dataset. :type: list .. attribute:: method_seeds List of random seeds to use for explanation methods. :type: list .. attribute:: data_params Additional parameters to be passed to the instantiation of the dataset. :type: dict Initializes an Experiment instance. :param data: The dataset or data class to be used for experiments. Can be passed in as an instantiated object or as a subclass of torch.utils.data.Dataset. :type data: Any :param models: List of neural network models to apply the explanation methods on. :type models: list | NoneType | torch.nn.Module :param methods: List of explanation methods to apply and evaluate. :type methods: Any :param metrics: List of evaluation metrics to compute. Defaults to None. :type metrics: Any | NoneType, optional :param seeds: List of random seeds to use for the instantiation of the dataset. Defaults to 0. :type seeds: int | list :param method_seeds: List of random seeds to use for explanation methods. Defaults to 0. :type method_seeds: int | list :param data_params: Additional parameters to be passed during the instantiation of the dataset. Defaults to None. :type data_params: dict | NoneType, optional :param name: string representing name of the experiment. :type name: str, optional :raises Exception: If input to any attribute initialization method is invalid. .. py:attribute:: seeds .. py:attribute:: data .. py:attribute:: data_params :value: None .. py:attribute:: models .. py:attribute:: metrics :value: [None] .. py:attribute:: methods .. py:attribute:: method_seeds .. py:attribute:: exp_name :value: None .. py:method:: get_data(seed: int) -> Any Returns the dataset instance generated with the specified seed. :param seed: the seed for instantiating the dataset. :type seed: int :returns: The dataset instance with the specified seed. :rtype: torch.utils.data.Dataset .. py:method:: get_models(data_instance: Any) -> List[torch.nn.Module] Returns the list of neural networks to apply the explanation methods on. A default neural network compatible with the given dataset will be generated if the Experiment object has None as its models. :param data_instance: The dataset instance. :type data_instance: torch.utils.data.Dataset :returns: List of neural networks to apply the explanation methods on. :rtype: list .. py:method:: get_methods(data_instance: Any) -> List[Any] Returns the list of explanation methods to apply and evaluate. :param data_instance: The dataset instance and a placeholder to keep input standardized. :type data_instance: torch.utils.data.Dataset :returns: List of explanation methods to apply and evaluate. :rtype: list .. py:method:: get_metrics(data_instance: Any) -> List[Any] Returns the list of evaluation metrics to compute. :param data_instance: The dataset instance. :type data_instance: torch.utils.data.Dataset :returns: List of evaluation metrics to compute. :rtype: list .. py:method:: _init_seeds(seeds: int) -> List[int] Initializes the seeds attribute and transforms the input to the desired datatype. :param seeds: Random seeds to use for data. :type seeds: int | list :returns: List of random seeds to use for the instantiation of the dataset. :rtype: list :raises Exception: If input to seeds initialization is not an integer or an Iterable of integers. .. py:method:: _init_data(data: Any) -> Any Initializes the data attribute. :param data: The instantiated dataset or data class. :type data: type :returns: The dataset or data class. :rtype: torch.utils.data.Dataset | type :raises Exception: If input to data initialization is neither a torch.utils.data.Dataset instance or subclass of torch.utils.data.Dataset. .. py:method:: _init_methods(methods: Any) -> Any Initializes the methods attribute. :param methods: List of explanation methods. :type methods: list | NoneType :returns: List of explanation methods. :rtype: list :raises Exception: If input to methods initialization is None. .. py:method:: _init_metrics(metrics: Any) -> Any Initializes the metrics attribute. :param metrics: List of evaluation metrics. :type metrics: list | NoneType :returns: List of evaluation metrics. :rtype: list :raises Exception: If input to metrics initialization is None and the dataset does not provide a default metric. .. py:method:: _init_models(models: Any) -> Any Initializes the models attribute and transforms it to the desired datatype. :param models: Neural network models. :type models: list | torch.nn.Module :returns: List of neural network models. :rtype: list :raises Exception: If input to models initialization is not an torch.nn.Module object or Iterable of torch.nn.Module objects. .. py:method:: _init_data_params(data_params: Dict) -> Dict Initializes the data_params attribute. :param data_params: Additional parameters for the instantiation of the dataset. :type data_params: dict :returns: Dictionary of additional data parameters. :rtype: dict :raises Exception: If input to data parameters initialization is not a dictionary. .. py:method:: _init_method_seeds(method_seeds: Union[int, collections.abc.Iterable[int]]) -> collections.abc.Iterable[int] Initializes the method seeds attribute and transforms the input to the desired datatype. :param method_seeds: Random seeds to use for applying the explanation methods. :type method_seeds: int | list :returns: List of random seeds for applying the explanation methods. :rtype: list :raises Exception: If input to method seeds initialization is not an integer nor an Iterable of integers. .. py:method:: _verify_metric(metrics: Any) -> None Verifies whether evaluation metrics have unique labels and each of them is in a valid datatype. :param metrics: A list of evaluation metrics. :type metrics: list :raises Exception: If evaluation metric is defined in an invalid datatype or if evaluation metrics do not have unique labels.