TfELM : Extreme Learning Machines Framework with Python and TensorFlow

This framework provides a comprehensive set of tools and utilities for implementing and experimenting with Extreme Learning Machines using Python and TensorFlow. ELMs are a type of machine learning algorithm known for their simplicity, efficiency, and scalability, making them ideal for various applications, including classification, regression, and feature learning tasks.

Table of Contents

• Introduction
• Features
• Documentation
• Installation
• Usage
• Examples
• License

Introduction

Extreme Learning Machines (ELMs) represent a class of feedforward neural networks initially proposed by Huang et al. in 2006. Traditional neural networks typically involve iterative optimization methods to learn both input weights and hidden layer biases, leading to computationally intensive training processes. In contrast, ELMs revolutionize this paradigm by adopting a simplified approach where input weights and biases are randomly initialized, and only the output weights are learned through a single linear regression. This architectural choice not only drastically reduces training time but also alleviates computational complexity, rendering ELMs highly efficient for tackling large-scale problems.

Basic Extreme Learning Machine concept

Let's denote the input data matrix as $X$ of size $N \times M$, where $N$ represents the number of samples and $M$ denotes the number of features. The hidden layer of an ELM consists of $K$ hidden neurons with random input weights $W$ of size $M \times K$ and biases $b$ of size $1 \times K$. The output weights $beta$ are learned through a linear regression approach. Given the activation function $g(\cdot)$, the output of the hidden layer can be computed as:

$H = g(X\mathbf{W} + \mathbf{b})$

Where $H$ is the hidden layer output matrix of size $N \times K$. The output weights $beta$ can then be determined by solving the linear regression problem:

$beta = (H^TH)^{-1}H^TY $

Where $Y$ is the target output matrix of size $N \times L$, with $L$ representing the number of output neurons.

Framework Overview

This repository offers a versatile and user-friendly framework for implementing and experimenting with ELMs using Python and TensorFlow. Researchers and practitioners alike can leverage this framework to explore novel machine learning techniques or seek efficient solutions to real-world problems. With its powerful toolkit for building and evaluating ELM models, this repository aims to facilitate seamless experimentation and deployment of ELM-based solutions across various domains.

Scarcity of ELM Implementations and the Need for a Comprehensive Framework

Despite the effectiveness and efficiency of Extreme Learning Machines (ELMs), there is a noticeable scarcity of comprehensive implementations in the machine learning community. Existing implementations are often fragmented, with individual researchers providing standalone implementations tailored to specific research papers. These implementations lack the versatility required for broader applications and may not always be reliable, with some failing to produce accurate results. Moreover, many existing implementations are solely available in MATLAB, restricting accessibility and hindering collaboration in the open-source community.

To address these limitations, this framework fills a crucial gap by providing a comprehensive and user-friendly toolkit for implementing and experimenting with ELMs using Python and TensorFlow. Leveraging TensorFlow for computation offers several advantages, including the ability to harness the power of GPUs for accelerated training. TensorFlow's efficient GPU utilization significantly speeds up computation, making it ideal for handling large-scale datasets and computationally intensive tasks. Additionally, the framework supports both GPU and CPU execution, ensuring compatibility with various computing environments.

Furthermore, the framework facilitates model persistence by enabling users to save and load ELM models using the standard HDF5 format provided by TensorFlow. This feature ensures seamless model deployment and sharing, allowing researchers and practitioners to collaborate and build upon each other's work effectively.

Features

Overview

• Comprehensive Framework: In response to the scarcity of comprehensive implementations for Extreme Learning Machines (ELMs), this framework provides a robust and all-encompassing solution. Unlike fragmented implementations scattered across various research papers, this framework offers a unified toolkit tailored for versatile applications.
• Efficient Training with TensorFlow: Harnessing the power of TensorFlow, the framework ensures efficient training of ELM models. TensorFlow's GPU acceleration significantly speeds up computation, making it suitable for handling large-scale datasets and time-sensitive applications. Moreover, the framework supports both GPU and CPU execution, ensuring compatibility with diverse computing environments.
• Versatile Applications: ELMs are renowned for their versatility, capable of tackling a wide range of machine learning tasks, including classification, regression, and feature learning. With this framework, users can effortlessly apply ELMs to various real-world problems, thanks to its user-friendly interface and extensive functionalities.
• Model Persistence and Portability: The framework enables seamless model persistence by allowing users to save and load ELM models using the standard HDF5 format provided by TensorFlow. This feature facilitates model deployment, sharing, and collaboration, empowering researchers and practitioners to build upon each other's work effectively.
• Modular Design and Extensibility: Designed with a modular architecture, the framework empowers users to customize and extend various components of the ELM model effortlessly. This flexibility ensures adaptability to diverse research requirements and facilitates the integration of novel techniques and algorithms.
• Comprehensive Documentation and Examples: Detailed documentation and usage examples accompany the framework, providing users with comprehensive guidance on understanding and implementing ELMs. These resources facilitate rapid onboarding and empower users to leverage the full potential of the framework in their research and applications.

Implemented Variants of Extreme Learning Machines

1. Basic Extreme Learning Machine (ELM):

   • Implementation with 108 various activation functions available for regression and classification tasks. Integration with scikit-learn functions like cross_val_score for straightforward accessibility.

2. Constrained Extreme Learning Machine (CELM):

   • Specifically designed for imbalanced datasets, generates weights based on class differences in the input data rather than random initialization.

3. Deep Extreme Learning Machine (DeepELM):

   • Incorporates multiple layers, with each layer consisting of a type of ELM layer, enabling deep learning capabilities.

4. Deep Representation Extreme Learning Machine (DrELM):

   • Utilizes random projection on input data, followed by shift and kernel function application to obtain beta, forming a block that can be stacked together for multilayer architecture.

5. Enhanced Deep Representation Extreme Learning Machine (EHDrELM):

   • Incorporates skip connections from residual networks to consider the output of preceding blocks in calculating shifted mapping.

6. Graph Regularized Extreme Learning Machine Autoencoder (GELM-AE):

   • Learns feature representation through unsupervised learning using the regularized Graph Laplacian concept.

7. Kernel Extreme Learning Machine (KELM):

   • Utilizes Mercer Kernels to generate hidden layer output without defining the number of neurons, with support for various kernel functions and kernel matrix approximations using the Nystrom method.

8. Local Receptive Field Extreme Learning Machine (LRF-ELM):

   • Implements randomly generated convolutional hidden nodes for feature mapping, suitable for classification tasks.

9. Metaheuristic Algorithms for Extreme Learning Machine Optimization (MA-ELM):

   • Optimizes random weights for specific tasks using Metaheuristic Algorithms from the comprehensive mealpy package, enabling efficient population-based metaheuristic optimization.

10. Multi-layer Extreme Learning Machine (ML-ELM):

    • Consists of multiple layers of ELM units (AE-ELM) for feature extraction followed by a final ELM layer for classification, adaptable to various layer configurations.

For all the mentioned methods, there is support for saving and loading mechanisms using HDF5 format. Additionally, they are compatible with cross_val_score, providing both predict and predict_proba functionalities for ROC/AUC curve calculations. Each algorithm runs on TensorFlow, ensuring efficient computation and compatibility with various computing environments.

Documentation

Documentation is available at the provided link.

Installation

PyPI

To install the ELM Framework using PyPI, simply enter:

pip install TfELM

Git

To install the ELM Framework using this git repository, clone it to your local machine:

git clone https://github.com/KStruniawski/TfELM.git

Then, navigate to the cloned directory and install the required dependencies using pip:

cd elm-framework
pip install -r requirements.txt

Usage

In this package, the folder structure is organized to facilitate easy navigation and utilization of different components. The Data folder contains exemplary datasets sourced from UCI, providing users with readily available data for experimentation. Within the Examples directory, users can find sample code demonstrating the application of various Extreme Learning Machine (ELM) methods. The Layers folder houses ELM layers, which can be directly employed or, preferably, utilized through the Models directory. The Models directory is structured following well-established patterns, facilitating usage thro