Titans Transformer Implementation
This repository contains an experimental implementation of the Titans Transformer architecture for sequence modeling tasks. The code is a personal exploration and may include errors or inefficiencies as I am currently in the learning stage. It is inspired by the ideas presented in the original Titans Transformer paper, and I highly recommend referring to the paper for accurate and detailed information.
Overview
The Titans Transformer introduces a novel architecture designed to enhance long-sequence modeling by incorporating a memory mechanism. This repository includes:
- A custom implementation of the Titans Transformer.
- Benchmarking code comparing the Titans Transformer with a standard Transformer.
- Training and evaluation scripts on the Wikitext-2 dataset.
- Visualization of benchmark results.
Note: This repository is for educational and experimental purposes only and is not a production-ready implementation.
Features
- Titans Transformer Architecture: Implements memory mechanisms for improved sequence modeling.
- Standard Transformer: Baseline implementation of the original Transformer for comparison.
- Benchmarking: Evaluates inference time and perplexity across different sequence lengths.
- Training: Customizable training loop with data preprocessing, batching, and evaluation.
Prerequisites
Dependencies
Ensure you have the following installed:
- Python 3.8+
- PyTorch
- Datasets
- Matplotlib
You can install the required libraries using:
pip install torch datasets matplotlib
Usage
Clone the Repository
git clone https://github.com/codewithdark-git/titans-transformer.git
cd titans-transformer
Run Training
To train the Titans Transformer model on the Wikitext-2 dataset, execute the training script:
python train_titans_transformer.py
Benchmark Models
To compare the Titans Transformer and Standard Transformer, run: