> ## Documentation Index
> Fetch the complete documentation index at: https://nixtlaverse.nixtla.io/llms.txt
> Use this file to discover all available pages before exploring further.

> BaseAuto class for hyperparameter optimization in NeuralForecast. Integrates Optuna, HyperOpt, Dragonfly through Ray for automated model tuning with cross-validation.

# Hyperparameter Optimization | NeuralForecast

Machine Learning forecasting methods are defined by many hyperparameters that
control their behavior, with effects ranging from their speed and memory
requirements to their predictive performance. For a long time, manual
hyperparameter tuning prevailed. This approach is time-consuming, **automated
hyperparameter optimization** methods have been introduced, proving more
efficient than manual tuning, grid search, and random search.<br /><br /> The
`BaseAuto` class offers shared API connections to hyperparameter optimization
algorithms like
[Optuna](https://docs.ray.io/en/latest/tune/examples/bayesopt_example.html),
[HyperOpt](https://docs.ray.io/en/latest/tune/examples/hyperopt_example.html),
[Dragonfly](https://docs.ray.io/en/releases-2.7.0/tune/examples/dragonfly_example.html)
among others through `ray`, which gives you access to grid search, bayesian
optimization and other state-of-the-art tools like
hyperband.

Comprehending the impacts of hyperparameters is still a
precious skill, as it can help guide the design of informed hyperparameter
spaces that are faster to explore automatically.

<img src="https://mintcdn.com/nixtla/ldwvWbCUC65OBWwN/neuralforecast/imgs_models/data_splits.png?fit=max&auto=format&n=ldwvWbCUC65OBWwN&q=85&s=958e6fb0a17a49b3c2516c1582a45e2f" alt="" width="1075" height="420" data-path="neuralforecast/imgs_models/data_splits.png" />

*Figure 1. Example of dataset split (left), validation (yellow) and test (orange). The hyperparameter optimization guiding signal is obtained from the validation set.*

##

### `BaseAuto`

```python theme={null}
BaseAuto(
    cls_model,
    h,
    loss,
    valid_loss,
    config,
    search_alg=BasicVariantGenerator(random_state=1),
    num_samples=10,
    time_budget=None,
    refit_with_val=False,
    verbose=False,
    alias=None,
    backend="ray",
    callbacks=None,
    ray_options=None,
    optuna_options=None,
    cpus=None,
    gpus=None,
)
```

Bases: <code>[LightningModule](#pytorch_lightning.LightningModule)</code>

Class for Automatic Hyperparameter Optimization, it builds on top of `ray` to
give access to a wide variety of hyperparameter optimization tools ranging
from classic grid search, to Bayesian optimization and HyperBand algorithm.

The validation loss to be optimized is defined by the `config['loss']` dictionary
value, the config also contains the rest of the hyperparameter search space.

It is important to note that the success of this hyperparameter optimization
heavily relies on a strong correlation between the validation and test periods.

**Parameters:**

| Name | Type | Description | Default |
| - | - | - | - |
| `cls_model` | <code>PyTorch/PyTorchLightning model</code> | See `neuralforecast.models` [collection here](./models.html). | *required* |
| `h` | <code>[int](#int)</code> | Forecast horizon | *required* |
| `loss` | <code>PyTorch module</code> | Instantiated train loss class from [losses collection](./losses.pytorch.html). | *required* |
| `valid_loss` | <code>PyTorch module</code> | Instantiated valid loss class from [losses collection](./losses.pytorch.html). | *required* |
| `config` | <code>[dict](#dict) or [callable](#callable)</code> | Dictionary with ray.tune defined search space or function that takes an optuna trial and returns a configuration dict. The config must include every parameter of the underlying model that has no default value (e.g. `input_size`, and `n_series` for multivariate models), either as a fixed value or as a search variable. `h`, `loss`, and `valid_loss` are injected automatically and must not be set in `config`. | *required* |
| `search_alg` | <code>ray.tune.search variant or optuna.sampler</code> | For ray see [https://docs.ray.io/en/latest/tune/api\_docs/suggestion.html](https://docs.ray.io/en/latest/tune/api_docs/suggestion.html) For optuna see [https://optuna.readthedocs.io/en/stable/reference/samplers/index.html](https://optuna.readthedocs.io/en/stable/reference/samplers/index.html). | <code>[BasicVariantGenerator](#ray.tune.search.basic_variant.BasicVariantGenerator)(random\_state=1)</code> |
| `num_samples` | <code>[int](#int)</code> | Number of hyperparameter optimization steps/samples. | <code>10</code> |
| `time_budget` | <code>[int](#int)</code> | Time budget in seconds for the hyperparameter search. | <code>None</code> |
| `refit_with_val` | <code>[bool](#bool)</code> | Refit of best model should preserve val\_size. | <code>False</code> |
| `verbose` | <code>[bool](#bool)</code> | Track progress. | <code>False</code> |
| `alias` | <code>[str](#str)</code> | Custom name of the model. | <code>None</code> |
| `backend` | <code>[str](#str)</code> | Backend to use for searching the hyperparameter space, can be either 'ray' or 'optuna'. | <code>'ray'</code> |
| `callbacks` | <code>list of callable</code> | List of functions to call during the optimization process. ray reference: [https://docs.ray.io/en/latest/tune/tutorials/tune-metrics.html](https://docs.ray.io/en/latest/tune/tutorials/tune-metrics.html) optuna reference: [https://optuna.readthedocs.io/en/stable](https://optuna.readthedocs.io/en/stable) | <code>None</code> |
| `ray_options` | <code>[RayOptions](#neuralforecast.common._base_auto.RayOptions)</code> | Container for Ray-only options. See `RayOptions` for the supported fields (`run_config`, `scheduler`, `cpus`, `gpus`). Only used with `backend='ray'`. | <code>None</code> |
| `optuna_options` | <code>[OptunaOptions](#neuralforecast.common._base_auto.OptunaOptions)</code> | Container for Optuna-only options. See `OptunaOptions` for the supported fields (`study_kwargs`, `create_study_kwargs`). Only used with `backend='optuna'`. | <code>None</code> |
| `cpus` | | No longer supported as of v3.2.0. Pin neuralforecast to v3.1.9, or pass `ray_options=RayOptions(cpus=...)` instead. | <code>None</code> |
| `gpus` | | No longer supported as of v3.2.0. Pin neuralforecast to v3.1.9, or pass `ray_options=RayOptions(gpus=...)` instead. | <code>None</code> |

#### `BaseAuto.fit`

```python theme={null}
fit(
    dataset, val_size=0, test_size=0, random_seed=None, distributed_config=None
)
```

BaseAuto.fit

Perform the hyperparameter optimization as specified by the BaseAuto configuration
dictionary `config`.

The optimization is performed on the `TimeSeriesDataset` using temporal cross validation with
the validation set that sequentially precedes the test set.

**Parameters:**

| Name | Type | Description | Default |
| - | - | - | - |
| `dataset` | <code>NeuralForecast's `TimeSeriesDataset`</code> | NeuralForecast's `TimeSeriesDataset` see details [here](./tsdataset.html) | *required* |
| `val_size` | <code>[int](#int)</code> | Size of temporal validation set (needs to be bigger than 0). | <code>0</code> |
| `test_size` | <code>[int](#int)</code> | Size of temporal test set (default 0). | <code>0</code> |
| `random_seed` | <code>[int](#int)</code> | Random seed for hyperparameter exploration algorithms, not yet implemented. | <code>None</code> |

**Returns:**

| Name | Type | Description |
| - | - | - |
| `self` | | Fitted instance of `BaseAuto` with best hyperparameters and results. |

#### `BaseAuto.predict`

```python theme={null}
predict(dataset, step_size=1, h=None, **data_kwargs)
```

BaseAuto.predict

Predictions of the best performing model on validation.

**Parameters:**

| Name | Type | Description | Default |
| - | - | - | - |
| `dataset` | <code>NeuralForecast's `TimeSeriesDataset`</code> | NeuralForecast's `TimeSeriesDataset` see details [here](./tsdataset.html) | *required* |
| `step_size` | <code>[int](#int)</code> | Steps between sequential predictions, (default 1). | <code>1</code> |
| `h` | <code>[int](#int)</code> | Prediction horizon, if None, uses the model's fitted horizon. Defaults to None. | <code>None</code> |
| `**data_kwarg` | | Additional parameters for the dataset module. | *required* |

**Returns:**

| Name | Type | Description |
| - | - | - |
| `y_hat` | | Numpy predictions of the `NeuralForecast` model. |

### Usage Example

```python theme={null}
class RayLogLossesCallback(tune.Callback):
    def on_trial_complete(self, iteration, trials, trial, **info):
        result = trial.last_result
        print(40 * '-' + 'Trial finished' + 40 * '-')
        print(f'Train loss: {result["train_loss"]:.2f}. Valid loss: {result["loss"]:.2f}')
        print(80 * '-')
```

```python theme={null}
config = {
    "hidden_size": tune.choice([512]),
    "num_layers": tune.choice([3, 4]),
    "input_size": 12,
    "max_steps": 10,
    "val_check_steps": 5
}
auto = BaseAuto(h=12, loss=MAE(), valid_loss=MSE(), cls_model=MLP, config=config, num_samples=2, cpus=1, gpus=0, callbacks=[RayLogLossesCallback()])
auto.fit(dataset=dataset)
y_hat = auto.predict(dataset=dataset)
assert mae(Y_test_df['y'].values, y_hat[:, 0]) < 200
```

```python theme={null}
def config_f(trial):
    return {
        "hidden_size": trial.suggest_categorical('hidden_size', [512]),
        "num_layers": trial.suggest_categorical('num_layers', [3, 4]),
        "input_size": 12,
        "max_steps": 10,
        "val_check_steps": 5
    }

class OptunaLogLossesCallback:
    def __call__(self, study, trial):
        metrics = trial.user_attrs['METRICS']
        print(40 * '-' + 'Trial finished' + 40 * '-')
        print(f'Train loss: {metrics["train_loss"]:.2f}. Valid loss: {metrics["loss"]:.2f}')
        print(80 * '-')
```

```python theme={null}
auto2 = BaseAuto(h=12, loss=MAE(), valid_loss=MSE(), cls_model=MLP, config=config_f, search_alg=optuna.samplers.RandomSampler(), num_samples=2, backend='optuna', callbacks=[OptunaLogLossesCallback()])
auto2.fit(dataset=dataset)
assert isinstance(auto2.results, optuna.Study)
y_hat2 = auto2.predict(dataset=dataset)
assert mae(Y_test_df['y'].values, y_hat2[:, 0]) < 200
```

### References

* [James Bergstra, Remi Bardenet, Yoshua Bengio, and Balazs Kegl
  (2011). “Algorithms for Hyper-Parameter Optimization”. In: Advances
  in Neural Information Processing Systems. url:
  https://proceedings.neurips.cc/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf](https://proceedings.neurips.cc/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf)
* [Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger,
  Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnabas
  Poczos, Eric P. Xing (2019). “Tuning Hyperparameters without Grad
  Students: Scalable and Robust Bayesian Optimisation with Dragonfly”.
  Journal of Machine Learning Research. url:
  https://arxiv.org/abs/1903.06694](https://arxiv.org/abs/1903.06694)
* [Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh,
  Ameet Talwalkar (2016). “Hyperband: A Novel Bandit-Based Approach to
  Hyperparameter Optimization”. Journal of Machine Learning Research.
  url:
  https://arxiv.org/abs/1603.06560](https://arxiv.org/abs/1603.06560)


## Related topics

- [Hyperparameter Optimization | NeuralForecast](/neuralforecast/docs/capabilities/hyperparameter_tuning.html.md)
- [Hyperparameter optimization | MLForecast](/mlforecast/docs/how-to-guides/hyperparameter_optimization.html.md)
- [Adding Models to NeuralForecast](/neuralforecast/docs/tutorials/adding_models.html.md)
- [End to End Walkthrough | NeuralForecast](/neuralforecast/docs/tutorials/getting_started_complete.html.md)
- [Weighting Timesteps | NeuralForecast](/neuralforecast/docs/tutorials/weighting_timesteps.html.md)