> ## Documentation Index
> Fetch the complete documentation index at: https://nixtlaverse.nixtla.io/llms.txt
> Use this file to discover all available pages before exploring further.

> PyTorch Dataset and DataLoader classes for time series. TimeSeriesDataset and TimeSeriesDataModule for efficient batch processing with Lightning integration.

# PyTorch Dataset/Loader

## Torch Time Series Dataset

### `TimeSeriesLoader`

```python theme={null}
TimeSeriesLoader(dataset, **kwargs)
```

Bases: <code>[DataLoader](#torch.utils.data.DataLoader)</code>

TimeSeriesLoader DataLoader.

Small change to PyTorch's Data loader.
Combines a dataset and a sampler, and provides an iterable over the given dataset.

The class `~torch.utils.data.DataLoader` supports both map-style and
iterable-style datasets with single- or multi-process loading, customizing
loading order and optional automatic batching (collation) and memory pinning.

**Parameters:**

| Name | Type | Description | Default |
| - | - | - | - |
| `dataset` | | Dataset to load data from. | *required* |
| `batch_size` | <code>[int](#int)</code> | How many samples per batch to load. Defaults to 1. | *required* |
| `shuffle` | <code>[bool](#bool)</code> | Set to True to have the data reshuffled at every epoch. Defaults to False. | *required* |
| `sampler` | <code>[Sampler](#Sampler) or [Iterable](#Iterable)</code> | Defines the strategy to draw samples from the dataset. | *required* |
| `drop_last` | <code>[bool](#bool)</code> | Set to True to drop the last incomplete batch. Defaults to False. | *required* |
| `**kwargs` | | Additional keyword arguments for DataLoader. | <code>{}</code> |

### `BaseTimeSeriesDataset`

```python theme={null}
BaseTimeSeriesDataset(
    temporal_cols, max_size, min_size, y_idx, static=None, static_cols=None
)
```

Bases: <code>[Dataset](#torch.utils.data.Dataset)</code>

Base class for time series datasets.

**Parameters:**

| Name | Type | Description | Default |
| - | - | - | - |
| `temporal_cols` | | Column names for temporal features. | *required* |
| `max_size` | <code>[int](#int)</code> | Maximum size of time series. | *required* |
| `min_size` | <code>[int](#int)</code> | Minimum size of time series. | *required* |
| `y_idx` | <code>[int](#int)</code> | Index of target variable. | *required* |
| `static` | <code>[Optional](#typing.Optional)</code> | Static features array. | <code>None</code> |
| `static_cols` | <code>[Optional](#typing.Optional)</code> | Column names for static features. | <code>None</code> |

### `LocalFilesTimeSeriesDataset`

```python theme={null}
LocalFilesTimeSeriesDataset(
    files_ds,
    temporal_cols,
    id_col,
    time_col,
    target_col,
    last_times,
    indices,
    max_size,
    min_size,
    y_idx,
    static=None,
    static_cols=None,
)
```

Bases: <code>[BaseTimeSeriesDataset](#neuralforecast.tsdataset.BaseTimeSeriesDataset)</code>

Time series dataset that loads data from local files.

**Parameters:**

| Name | Type | Description | Default |
| - | - | - | - |
| `files_ds` | <code>[List](#typing.List)\[[str](#str)]</code> | List of file paths. | *required* |
| `temporal_cols` | | Column names for temporal features. | *required* |
| `id_col` | <code>[str](#str)</code> | Name of ID column. | *required* |
| `time_col` | <code>[str](#str)</code> | Name of time column. | *required* |
| `target_col` | <code>[str](#str)</code> | Name of target column. | *required* |
| `last_times` | | Last time for each time series. | *required* |
| `indices` | | Series indices. | *required* |
| `max_size` | <code>[int](#int)</code> | Maximum size of time series. | *required* |
| `min_size` | <code>[int](#int)</code> | Minimum size of time series. | *required* |
| `y_idx` | <code>[int](#int)</code> | Index of target variable. | *required* |
| `static` | <code>[Optional](#typing.Optional)</code> | Static features array. | <code>None</code> |
| `static_cols` | <code>[Optional](#typing.Optional)</code> | Column names for static features. | <code>None</code> |

#### `LocalFilesTimeSeriesDataset.from_data_directories`

```python theme={null}
from_data_directories(
    directories,
    static_df=None,
    exogs=[],
    id_col="unique_id",
    time_col="ds",
    target_col="y",
)
```

Create dataset from data directories.

Expects directories to be a list of directories of the form \[unique\_id=id\_0, unique\_id=id\_1, ...].
Each directory should contain the timeseries corresponding to that unique\_id, represented as a
pandas or polars DataFrame. The timeseries can be entirely contained in one parquet file or
split between multiple, but within each parquet files the timeseries should be sorted by time.

**Parameters:**

| Name | Type | Description | Default |
| - | - | - | - |
| `directories` | | List of directory paths. | *required* |
| `static_df` | <code>[Optional](#typing.Optional)</code> | Static features DataFrame. | <code>None</code> |
| `exogs` | <code>[List](#typing.List)</code> | List of exogenous variable names. Defaults to \[]. | <code>\[]</code> |
| `id_col` | <code>[str](#str)</code> | Name of ID column. Defaults to "unique\_id". | <code>'unique\_id'</code> |
| `time_col` | <code>[str](#str)</code> | Name of time column. Defaults to "ds". | <code>'ds'</code> |
| `target_col` | <code>[str](#str)</code> | Name of target column. Defaults to "y". | <code>'y'</code> |

**Returns:**

| Name | Type | Description |
| - | - | - |
| `LocalFilesTimeSeriesDataset` | | Dataset created from directories. |

### `TimeSeriesDataset`

```python theme={null}
TimeSeriesDataset(
    temporal, temporal_cols, indptr, y_idx, static=None, static_cols=None
)
```

Bases: <code>[BaseTimeSeriesDataset](#neuralforecast.tsdataset.BaseTimeSeriesDataset)</code>

Time series dataset implementation.

**Parameters:**

| Name | Type | Description | Default |
| - | - | - | - |
| `temporal` | | Temporal data array. | *required* |
| `temporal_cols` | | Column names for temporal features. | *required* |
| `indptr` | | Index pointers for time series grouping. | *required* |
| `y_idx` | <code>[int](#int)</code> | Index of target variable. | *required* |
| `static` | <code>[Optional](#typing.Optional)</code> | Static features array. | <code>None</code> |
| `static_cols` | <code>[Optional](#typing.Optional)</code> | Column names for static features. | <code>None</code> |

#### `TimeSeriesDataset.append`

```python theme={null}
append(futr_dataset)
```

Add future observations to the dataset.

**Parameters:**

| Name | Type | Description | Default |
| - | - | - | - |
| `futr_dataset` | <code>[TimeSeriesDataset](#neuralforecast.tsdataset.TimeSeriesDataset)</code> | Future dataset to append. | *required* |

**Returns:**

| Name | Type | Description |
| - | - | - |
| `TimeSeriesDataset` | <code>[TimeSeriesDataset](#neuralforecast.tsdataset.TimeSeriesDataset)</code> | Copy of dataset with future observations appended. |

**Raises:**

| Type | Description |
| - | - |
| <code>[ValueError](#ValueError)</code> | If datasets have different number of groups. |

#### `TimeSeriesDataset.trim_dataset`

```python theme={null}
trim_dataset(dataset, left_trim=0, right_trim=0)
```

Trim temporal information from a dataset.

Returns temporal indexes \[t+left:t-right] for all series.

**Parameters:**

| Name | Type | Description | Default |
| - | - | - | - |
| `dataset` | | Dataset to trim. | *required* |
| `left_trim` | <code>[int](#int)</code> | Number of observations to trim from the left. Defaults to 0. | <code>0</code> |
| `right_trim` | <code>[int](#int)</code> | Number of observations to trim from the right. Defaults to 0. | <code>0</code> |

**Returns:**

| Name | Type | Description |
| - | - | - |
| `TimeSeriesDataset` | | Trimmed dataset. |

**Raises:**

| Type | Description |
| - | - |
| <code>[Exception](#Exception)</code> | If trim size exceeds minimum series length. |

### `TimeSeriesDataModule`

```python theme={null}
TimeSeriesDataModule(
    dataset,
    batch_size=32,
    valid_batch_size=1024,
    drop_last=False,
    shuffle_train=True,
    **dataloaders_kwargs
)
```

Bases: <code>[LightningDataModule](#pytorch_lightning.LightningDataModule)</code>

PyTorch Lightning data module for time series datasets.

**Parameters:**

| Name | Type | Description | Default |
| - | - | - | - |
| `dataset` | <code>[BaseTimeSeriesDataset](#neuralforecast.tsdataset.BaseTimeSeriesDataset)</code> | Time series dataset. | *required* |
| `batch_size` | <code>[int](#int)</code> | Batch size for training. Defaults to 32. | <code>32</code> |
| `valid_batch_size` | <code>[int](#int)</code> | Batch size for validation. Defaults to 1024. | <code>1024</code> |
| `drop_last` | <code>[bool](#bool)</code> | Whether to drop the last incomplete batch. Defaults to False. | <code>False</code> |
| `shuffle_train` | <code>[bool](#bool)</code> | Whether to shuffle training data. Defaults to True. | <code>True</code> |
| `**dataloaders_kwargs` | | Additional keyword arguments for data loaders. | <code>{}</code> |

### Example

```python theme={null}
import lightning.pytorch as L
import torch.utils.data as data
from pytorch_lightning.demos.boring_classes import RandomDataset

class MyDataModule(L.LightningDataModule):
    def prepare_data(self):
        # download, IO, etc. Useful with shared filesystems
        # only called on 1 GPU/TPU in distributed
        ...

    def setup(self, stage):
        # make assignments here (val/train/test split)
        # called on every process in DDP
        dataset = RandomDataset(1, 100)
        self.train, self.val, self.test = data.random_split(
            dataset, [80, 10, 10], generator=torch.Generator().manual_seed(42)
        )

    def train_dataloader(self):
        return data.DataLoader(self.train)

    def val_dataloader(self):
        return data.DataLoader(self.val)

    def test_dataloader(self):
        return data.DataLoader(self.test)

    def on_exception(self, exception):
        # clean up state after the trainer faced an exception
        ...

    def teardown(self):
        # clean up state after the trainer stops, delete files...
        # called on every process in DDP
        ...*
```

```python theme={null}
# To test correct future_df wrangling of the `update_df` method
# We are checking that we are able to recover the AirPassengers dataset
# using the dataframe or splitting it into parts and initializing.
```


## Related topics

- [PyTorch Losses](/neuralforecast/losses.pytorch.html.md)
- [Adding Models to NeuralForecast](/neuralforecast/docs/tutorials/adding_models.html.md)
- [NLinear](/neuralforecast/models.nlinear.html.md)
- [DLinear](/neuralforecast/models.dlinear.html.md)
- [SOFTS](/neuralforecast/models.softs.html.md)