Data Loader module

nnodely.operators.loader.Loader.getSamples(self, dataset: str, *, index: int | None = None, window: int = 1) → dict

Retrieves a window of samples from a given dataset.

Parameters:

  • dataset (str) – The name of the dataset to retrieve samples from.

  • index (int, optional) – The starting index of the samples. If None, a random index is chosen. Default is None.

  • window (int, optional) – The number of consecutive samples to retrieve. Default is 1.

Returns:

A dictionary containing the retrieved samples. The keys are input names, and the values are lists of samples.

Return type:

dict

Raises:

ValueError – If the dataset is not loaded.

Examples

model = Modely()
model.loadData('dataset_name')
samples = model.getSamples('dataset_name', index=10, window=5)
nnodely.operators.loader.Loader.filterData(self, filter_function: Callable, dataset_name: str | None = None) → None

Filters the data in the dataset using the provided filter function.

Parameters:

  • filter_function (Callable) – A function that takes a sample as input and returns True if the sample should be kept, and False if it should be removed.

  • dataset_name (str or None, optional) – The name of the dataset to filter. If None, all datasets are filtered. Default is None.

Examples

model = Modely()
model.loadData('dataset_name', 'path/to/data')

def filter_fn(sample):
    return sample['input1'] > 0

model.filterData(filter_fn, 'dataset_name')
nnodely.operators.loader.Loader.resamplingData(self, df: DataFrame, *, scale: float = 1000000000.0) → None

Resamples the DataFrame to a specified sample time.

Parameters:

  • df (pd.DataFrame) – The DataFrame to resample.

  • scale (float, optional) – The scale factor to convert the sample time to nanoseconds. Default is 1e9

Returns:

The resampled DataFrame.

Return type:

pd.DataFrame

Raises:

TypeError – If the DataFrame does not contain a time column or if the time column is not in datetime format.

Examples

model = Modely()
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'time': np.array(range(60), dtype=np.float32),
    'x': np.array(10*[10] + 20*[20] + 30*[30], dtype=np.float32)
})

resampled_df = model.resamplingData(df, scale=1e9)
nnodely.operators.loader.Loader.loadData(self, name: str, source: str | dict | DataFrame, *, format: list | None = None, skiplines: int = 0, delimiter: str = ',', header: int | str | Sequence | None = None, resampling: bool = False) → None

Loads data into the model. The data can be loaded from a directory path containing the csv files or from a crafted dataset.

Parameters:

  • name (str) – The name of the dataset.

  • source (str or list or pd.DataFrame) – The source of the data. Can be a directory path containing the csv files or a custom dataset provided as a dictionary or a pandas DataFrame.

  • format (list or None, optional) – The format of the data. When loading multiple csv files the format parameter will define how to read each column of the file. Default is None.

  • skiplines (int, optional) – The number of lines to skip at the beginning of the file. Default is 0.

  • delimiter (str, optional) – The delimiter used in the data files. Default is ‘,’.

  • header (list or None, optional) – The header of the data files. Default is None.

Raises:

ValueError – If the network is not neuralized. If the delimiter is not valid.

Examples

Load data from files:

import numpy as np
from nnodely.basic.model import Modely
from nnodely.layers.input import Input
from nnodely.layers.output import Output
from nnodely.layers.fir import Fir

x = Input('x')
y = Input('y')
out = Output('out', Fir(x.tw(0.05)))

test = Modely(visualizer=None)
test.addModel('example_model', out)
test.neuralizeModel(0.01)

data_struct = ['x', '', 'y']
test.loadData(name='example_dataset', source='path/to/data', format=data_struct)

Load data from a crafted dataset:

x = Input('x')
y = Input('y')
out = Output('out', Fir(x.tw(0.05)))

test = Modely(visualizer=None)
test.addModel('example_model', out)
test.neuralizeModel(0.01)

data_x = np.array(range(10))
dataset = {'x': data_x, 'y': (2*data_x)}
test.loadData(name='example_dataset', source=dataset)

For more examples of how to use the data loader, please refer to the relative tutorial.