hpelm package¶
Submodules¶
hpelm.elm module¶
Created on Mon Oct 27 17:48:33 2014
@author: akusok

class
hpelm.elm.
ELM
(inputs, outputs, classification='', w=None, batch=1000, accelerator=None, precision='double', norm=None, tprint=5)[source]¶ Bases:
object
Interface for training Extreme Learning Machines (ELM).
Parameters:  inputs (int) – dimensionality of input data, or number of data features
 outputs (int) – dimensionality of output data, or number of classes
 classification ('c'/'wc'/'ml', optional) – train ELM for classfication (‘c’) / weighted classification (‘wc’) / multilabel classification (‘ml’). For weighted classification you can provide weights in w. ELM will compute and use the corresponding classification error instead of Mean Squared Error.
 w (vector, optional) – weights vector for weighted classification, lenght (outputs * 1).
 batch (int, optional) – batch size for data processing in ELM, reduces memory requirements. Does not work for model structure selection (validation, crossvalidation, LeaveOneOut). Can be changed later directly as a class attribute.
 accelerator ("GPU"/"basic", optional) – type of accelerated ELM to use: None, ‘GPU’, ‘basic’, ...
 precision (optional) – data precision to use, supports signle (‘single’, ‘32’ or numpy.float32) or double (‘double’, ‘64’ or numpy.float64). Single precision is faster but may cause numerical errors. Majority of GPUs work in single precision. Default: double.
 norm (double, optinal) – L2normalization parameter, None gives the default value.
 tprint (int, optional) – ELM reports its progess every tprint seconds or after every batch, whatever takes longer.
Class attributes; attributes that simply store initialization or train() parameters are omitted.

nnet
¶ object
Implementation of neural network with computational methods, but without complex logic. Different implementations are given by different classes: for Python, for GPU, etc. See
hpelm.nnets
folder for particular files. You can implement your own computational algorithm by inheriting fromhpelm.nnets.SLFN
and overwriting some methods.

flist
¶ list of strings
Awailable types of neurons, use them when adding new neurons.
Note
Below the ‘matrix’ type means a 2dimensional Numpy.ndarray.

add_data
(X, T)[source]¶ Feed new training data (X,T) to ELM model in batches; does not solve ELM itself.
Helper method that updates intermediate solution parameters HH and HT, which are used for solving ELM later. Updates accumulate, so this method can be called multiple times with different parts of training data. To reset accumulated training data, use ELM.nnet.reset().
For training an ELM use ELM.train() instead.
Parameters:  X (matrix) – input training data
 T (matrix) – output training data

add_neurons
(number, func, W=None, B=None)[source]¶ Adds neurons to ELM model. ELM is created empty, and needs some neurons to work.
Add neurons to an empty ELM model, or add more neurons to a model that already has some.
Random weights W and biases B are generated automatically if not provided explicitly. Maximum number of neurons is limited by the available RAM and computational power, a sensible limit would be 1000 neurons for an average size dataset and 15000 for the largest datasets. ELM becomes slower after 3000 neurons because computational complexity is proportional to a qube of number of neurons.
This method checks and prepares neurons, they are actually stored in solver object.
Parameters:  number (int) – number of neurons to add
 func (string) – type of neurons: “lin” for linear, “sigm” or “tanh” for nonlinear, “rbf_l1”, “rbf_l2” or “rbf_linf” for radial basis function neurons.
 W (matrix, optional) – random projection matrix size (inputs * number). For ‘rbf_‘ neurons, W stores centroids of radial basis functions in transposed form.
 B (vector, optional) – bias vector of size (number * 1), a 1dimensional Numpy.ndarray object. For ‘rbf_‘ neurons, B gives widths of radial basis functions.

confusion
(T, Y)[source]¶ Computes confusion matrix for classification.
Confusion matrix \(C\) such that element \(C_{i,j}\) equals to the number of observations known to be class \(i\) but predicted to be class \(j\).
Parameters:  T (matrix) – true outputs or classes, size (N * outputs)
 Y (matrix) – predicted outputs by ELM model, size (N * outputs)
Returns: conf – confusion matrix, size (outputs * outputs)
Return type: matrix

error
(T, Y)[source]¶ Calculate error of model predictions.
Computes Mean Squared Error (MSE) between model predictions Y and true outputs T. For classification, computes misclassification error. For multilabel classification, correct classes are all with Y>0.5.
For weighted classification the error is an average weighted True Positive Rate, or percentage of correctly predicted samples for each class, multiplied by weight of that class and averaged. If you want something else, just write it yourself :) See https://en.wikipedia.org/wiki/Confusion_matrix for details.
Another option is to use scikitlearn’s performance metrics. Transform Y and T into scikit’s format by
y_true = T.argmax[1]
,y_pred = Y.argmax[1]
. http://scikitlearn.org/stable/modules/classes.html#modulesklearn.metricsParameters:  T (matrix) – true outputs.
 Y (matrix) – ELM model predictions, can be computed with predict() function.
Returns: e – MSE for regression / classification error for classification.
Return type: double

load
(fname)[source]¶ Load ELM model data from a file.
Load requires an
ELM
object, and it uses solver type, precision and batch size from that ELM object.Parameters: fname (string) – filename to load model from.

predict
(X)[source]¶ Predict outputs Y for the given input data X.
Parameters: X (matrix) – input data of size (N * inputs) Returns: Y – output data or predicted classes, size (N * outputs). Return type: matrix

project
(X)[source]¶ Get ELM’s hidden layer representation of input data.
Parameters: X (matrix) – input data, size (N * inputs) Returns: H – hidden layer representation matrix, size (N * number_of_neurons) Return type: matrix

save
(fname)[source]¶ Save ELM model with current parameters.
Model does not save a particular solver, precision batch size. They are obtained from a new ELM when loading the model (so one can switch to another solver, for instance).
Also ranking and max number of neurons are not saved, because they are runtime training info irrelevant after the training completes.
Parameters: fname (string) – filename to save model into.

train
(X, T, *args, **kwargs)[source]¶ Universal training interface for ELM model with model structure selection.
Model structure selection takes more time and requires all data to fit into memory. Optimal pruning (‘OP’, effectively an L1regularization) takes the most time but gives the smallest and best performing model. Choosing a classification forces ELM to use classification error in model structure selection, and in error() method output.
Parameters:  X (matrix) – input data matrix, size (N * inputs)
 T (matrix) – outputs data matrix, size (N * outputs)
 'V'/'CV'/'LOO' (sting, choose one) – model structure selection: select optimal number of neurons using a validation set (‘V’), crossvalidation (‘CV’) or LeaveOneOut (‘LOO’)
 'OP' (string, use with 'V'/'CV'/'LOO') – choose best neurons instead of random ones, training takes longer; equivalent to L1regularization
 'c'/'wc'/'ml'/'r' (string, choose one) – train ELM for classification (‘c’), classification with weighted classes (‘wc’), multilabel classification (‘ml’) with several correct classes per data sample, or regression (‘r’) without any classification. In classification, number of outputs is the number of classes; correct class(es) for each sample has value 1 and incorrect classes have 0. Overwrites parameters given an ELM initialization time.
Keyword Arguments:  Xv (matrix, use with ‘V’) – validation set input data, size (Nv * inputs)
 Tv (matrix, use with ‘V’) – validation set outputs data, size (Nv * outputs)
 k (int, use with ‘CV’) – number of splits for crossvalidation, k>=3
 kmax (int, optional, use with ‘OP’) – maximum number of neurons to keep in ELM
 batch (int, optional) – batch size for ELM, overwrites batch size from the initialization
Returns: e – test error for crossvalidation, computed from one separate test chunk in each
split of data during the crossvalidation procedure
Return type: double, for ‘CV’
hpelm.hp_elm module¶
Created on Mon Oct 27 17:48:33 2014
@author: akusok

class
hpelm.hp_elm.
HPELM
(inputs, outputs, classification='', w=None, batch=1000, accelerator=None, precision='double', norm=None, tprint=5)[source]¶ Bases:
hpelm.elm.ELM
Interface for training HighPerformance Extreme Learning Machines (HPELM).
Parameters:  inputs (int) – dimensionality of input data, or number of data features
 outputs (int) – dimensionality of output data, or number of classes
 classification ('c'/'wc'/'ml', optional) – train ELM for classfication (‘c’) / weighted classification (‘wc’) / multilabel classification (‘ml’). For weighted classification you can provide weights in w. ELM will compute and use the corresponding classification error instead of Mean Squared Error.
 w (vector, optional) – weights vector for weighted classification, lenght (outputs * 1).
 batch (int, optional) – batch size for data processing in ELM, reduces memory requirements. Does not work for model structure selection (validation, crossvalidation, LeaveOneOut). Can be changed later directly as a class attribute.
 accelerator (string, optional) – type of accelerated ELM to use: None, ‘GPU’, ...
 precision (optional) – data precision to use, supports signle (‘single’, ‘32’ or numpy.float32) or double (‘double’, ‘64’ or numpy.float64). Single precision is faster but may cause numerical errors. Majority of GPUs work in single precision. Default: double.
 norm (double, optinal) – L2normalization parameter, None gives the default value.
 tprint (int, optional) – ELM reports its progess every tprint seconds or after every batch, whatever takes longer.
Class attributes; attributes that simply store initialization or train() parameters are omitted.

nnet
¶ object
Implementation of neural network with computational methods, but without complex logic. Different implementations are given by different classes: for Python, for GPU, etc. See
hpelm.nnets
folder for particular files. You can implement your own computational algorithm by inheriting fromhpelm.nnets.SLFN
and overwriting some methods.

flist
¶ list of strings
Awailable types of neurons, use them when adding new neurons.
Note
The ‘hdf5’ type denotes a name of HDF5 file type with a single 2dimensional array inside. HPELM uses PyTables interface to HDF5: http://www.pytables.org/. For HDF5 array examples, see http://www.pytables.org/usersguide/libref/homogenous_storage.html. Array name is irrelevant, but there must be only one array per HDF5 file.
A 2dimensional Numpy.ndarray can also be used.

add_data
(fX, fT, istart=0, icount=inf, fHH=None, fHT=None)[source]¶ Feed new training data (X,T) to HPELM model in batches: does not solve ELM itself.
This method prepares an intermediate solution data, that takes the most time. After that, obtaining the solution is fast.
The intermediate solution consists of two matrices: HH and HT. They can be in memory for a model computed at once, or stored on disk for a model computed in parts or in parallel.
For iterative solution, provide file names for ondisk matrices in the input parameters fHH and fHT. They will be created if they don’t exist, or new results will be merged with the existing ones. This method is multiprocesssafe for parallel writing into files fHH and fHT, that allows you to easily compute ELM in parallel. The multiprocesssafeness uses Python module ‘fasteners’ and a lock file, which is named fHH+’.lock’ and fHT+’.lock’.
Parameters:  fX (hdf5) – (part of) input training data size (N * inputs)
 fT (hdf5) –
 istart (int, optional) – index of first data sample to use from fX, istart < N. If not given, all data from fX is used. Sample with index istart is used for training, indexing is 0based.
 icount (int, optional) – number of data samples to use from fX, starting from istart, automatically adjusted to istart + icount <= N. If not given, all data starting from start is used. The last sample used for training is istart`+`icount1, so you can index data as: istart_1=0, icount_1=1000; istart_2=1000, icount_2=1000; istart_3=2000, icount_3=1000, ...
 fHT (fHH,) – file names for storing HH and HT matrices. Files are created if they don’t exist, or new result is added to the existing files if they exist. Parallel writing to the same fHH, fHT files is multiprocesssafe, made specially for parallel training of HPELM. Another use is to split a very long training of huge ELM into smaller parts, so the training can be interrupted and resumed later.

add_data_async
(fX, fT, istart=0, icount=inf, fHH=None, fHT=None)[source]¶ Version of add_data() with asyncronous I/O. See add_data() for reference.
Spawns new processes using Python’s multiprocessing module, and requires more memory than nonasync version.

error
(fT, fY, istart=0, icount=inf)[source]¶ Calculate error of model predictions of HPELM.
Computes Mean Squared Error (MSE) between model predictions Y and true outputs T. For classification, computes misclassification error. For multilabel classification, correct classes are all with Y>0.5.
For weighted classification the error is an average weighted True Positive Rate, or percentage of correctly predicted samples for each class, multiplied by weight of that class and averaged. If you want something else, just write it yourself :) See https://en.wikipedia.org/wiki/Confusion_matrix for details.
Parameters:  fT (hdf5) – hdf5 filename with true outputs
 fY (hdf5) – hdf5 filename with predicted outputs
 istart (int, optional) – index of first data sample to use from fX, istart < N. If not given, all data from fX is used. Sample with index istart is used for training, indexing is 0based.
 icount (int, optional) – number of data samples to use from fX, starting from istart, automatically adjusted to istart + icount <= N. If not given, all data starting from start is used. The last sample used for training is istart`+`icount1, so you can index data as: istart_1=0, icount_1=1000; istart_2=1000, icount_2=1000; istart_3=2000, icount_3=1000, ...
Returns: e – MSE for regression / classification error for classification.
Return type: double

predict
(fX, fY=None, istart=0, icount=inf)[source]¶ Iterative predict outputs and save them to HDF5, can use custom range.
Parameters:  fX (hdf5) – hdf5 filename or Numpy matrix with input data from which outputs are predicted
 fY (hdf5) – hdf5 filename or Numpy matrix to store output data into, if ‘None’ then Numpy matrix is generated automatically.
 istart (int, optional) – index of first data sample to use from fX, istart < N. If not given, all data from fX is used. Sample with index istart is used for training, indexing is 0based.
 icount (int, optional) – number of data samples to use from fX, starting from istart, automatically adjusted to istart + icount <= N. If not given, all data starting from start is used. The last sample used for training is istart`+`icount1, so you can index data as: istart_1=0, icount_1=1000; istart_2=1000, icount_2=1000; istart_3=2000, icount_3=1000, ...

predict_async
(fX, fY, istart=0, icount=inf)[source]¶ Version of predict() with asyncronous I/O. See predict() for reference.
Spawns new processes using Python’s multiprocessing module, and requires more memory than nonasync version.

project
(fX, fH=None, istart=0, icount=inf)[source]¶ Iteratively project input data from HDF5 into HPELM hidden layer, and save in another HDF5.
Parameters:  fX (hdf5) – hdf5 filename or Numpy matrix with input data to project
 fH (hdf5) – hdf5 filename or Numpy matrix to store projected inputs, if ‘None’ then Numpy matrix is generated automatically.
 istart (int, optional) – index of first data sample to use from fX, istart < N. If not given, all data from fX is used. Sample with index istart is used for training, indexing is 0based.
 icount (int, optional) – number of data samples to use from fX, starting from istart, automatically adjusted to istart + icount <= N. If not given, all data starting from start is used. The last sample used for training is istart`+`icount1, so you can index data as: istart_1=0, icount_1=1000; istart_2=1000, icount_2=1000; istart_3=2000, icount_3=1000, ...

solve_corr
(fHH, fHT)[source]¶ Solves an ELM model with the given (covariance) fHH and (correlation) fHT HDF5 files.
Parameters:  fHH (hdf5) – an hdf5 file with intermediate solution data
 fHT (hdf5) – an hdf5 file with intermediate solution data

train
(fX, fT, *args, **kwargs)[source]¶ Universal training interface for HPELM model.
Always trains a basic ELM model without model structure selection. L2regularization is available as norm parameter at HPELM initialization. Number of neurons selection with validation set for trained HPELM is available in train_hpv() method.
Parameters:  fX (hdf5) – input data on disk, size (N * inputs)
 fT (hdf5) – outputs data on disk, size (N * outputs)
 'c'/'wc'/'ml' (string, choose one) – train HPELM for classification (‘c’), classification with weighted classes (‘wc’) or multilabel classification (‘ml’) with several correct classes per data sample. In classification, number of outputs is the number of classes; correct class(es) for each sample has value 1 and incorrect classes have 0.
Keyword Arguments:  istart (int, optional) – index of first data sample to use from fX, istart < N. If not given, all data from fX is used. Sample with index istart is used for training, indexing is 0based.
 icount (int, optional) – number of data samples to use from fX, starting from istart, automatically adjusted to istart + icount <= N. If not given, all data starting from start is used. The last sample used for training is istart`+`icount1, so you can index data as: istart_1=0, icount_1=1000; istart_2=1000, icount_2=1000; istart_3=2000, icount_3=1000, ...
 batch (int, optional) – batch size for ELM, overwrites batch size from the initialization

train_async
(fX, fT, *args, **kwargs)[source]¶ Training HPELM with asyncronous I/O, good for network drives, etc. See train() for reference.
Spawns new processes using Python’s multiprocessing module.

validation_corr
(fHH, fHT, fXv, fTv, steps=10)[source]¶ Quick batch error evaluation with different numbers of neurons on a validation set.
Only feasible implementation of model structure selection with HPELM. This method makes a single pass over the validation data, computing errors for all numbers of neurons at once. It requires HDF5 files with matrices HH and HT: fHH and fHT, obtained from add_data(..., fHH, fHT) method.
The method writes the best solution to the HPELM model.
Parameters:  fHH (string) – name of HDF5 file with HH matrix
 fHT (string) – name of HDF5 file with HT matrix
 fXv (string) – name of HDF5 file with validation dataset inputs
 fTv (string) – name of HDF5 file with validation dataset outputs
 steps (int or vector) – amount of different numbers of neurons to test, choosen uniformly on a logarithmic scale from 3 to number of neurons in HPELM. Can be given exactly as a vector.
Returns: Ls – numbers of neurons used by validation_corr() method errs (vector): corresponding errors for number of neurons in Ls, with classification error if model
is run for classification
confs (list of matrix): list of confusion matrices corresponding to elements in Ls (empty for regression)
Return type: vector
hpelm.mss_cv module¶
Created on Mon Oct 27 17:48:33 2014
@author: akusok

hpelm.mss_cv.
train_cv
(self, X, T, k)[source]¶ Model structure selection with crossvalidation.
Trains ELM, crossvalidates model and sets an optimal validated solution.
Parameters:  self (ELM) – ELM object that calls train_v()
 X (matrix) – training set inputs
 T (matrix) – training set outputs
 k (int) – number of parts to split the dataset into, k2 parts are used for training and 2 parts are left out: 1 for validation and 1 for test; repeated k times until all parts have been left out for validation and test, and results averaged over these k repetitions.
Returns: err_t – error for the optimal model, computed in the ‘crosstesting’ manner on data part which is not used for training or validation
Return type: double
hpelm.mss_loo module¶
Created on Mon Oct 27 17:48:33 2014
@author: akusok

hpelm.mss_loo.
train_loo
(self, X, T)[source]¶ Model structure selection with LeaveOneOut (LOO) validation.
Trains ELM, validates model with LOO and sets an optimal validated solution. Effect is similar to crossvalidation with k==N, but ELM has explicit formula of solution for LOO without iterating k times.
Parameters:  self (ELM) – ELM object that calls train_v()
 X (matrix) – training set inputs
 T (matrix) – training set outputs
hpelm.mss_v module¶
Created on Mon Oct 27 17:48:33 2014
@author: akusok

hpelm.mss_v.
train_v
(self, X, T, Xv, Tv)[source]¶ Model structure selection with a validation set.
Trains ELM, validates model and sets an optimal validated solution.
Parameters:  self (ELM) – ELM object that calls train_v()
 X (matrix) – training set inputs
 T (matrix) – training set outputs
 Xv (matrix) – validation set inputs
 Tv (matrix) – validation set outputs