VistAI

Project Structure

VistAI/
├── data/                    # Dataset and processed images
├── models/                  # Pretrained models and embeddings
│   ├── clip_model/          # CLIP model weights
│   ├── dqn_model/           # Trained DQN models
│   └── embeddings/          # Image embeddings and FAISS index
├── src/                     # Source code
│   ├── app.py               # Flask application
│   ├── visual_search.py     # CLIP-based visual search
│   ├── rl_recommender.py    # DQN-based recommendation
│   ├── data_preprocessing.py# Image preprocessing utilities
│   └── utils.py             # Helper functions
├── static/                  # Static files (CSS, JS, uploads)
├── templates/               # HTML templates
├── tests/                   # Test files
├── requirements.txt         # Project dependencies
├── README.md                # Project Documentation

Methodology

1. Visual Search with CLIP

• Utilizes OpenAI's CLIP model to generate image embeddings
• Implements FAISS for efficient similarity search
• Returns visually similar products based on input image

2. Reinforcement Learning Recommendations

• Implements Deep Q-Network (DQN) for personalized recommendations
• State space: CLIP embeddings of products
• Action space: Product indices
• Reward function: Positive for correct matches, negative for mismatches

Implementation Details

Visual Search Pipeline

1. Image Preprocessing: Resize and normalize input images
2. Feature Extraction: Generate embeddings using CLIP
3. Similarity Search: Find nearest neighbors using FAISS
4. Result Ranking: Sort results by similarity score

RL Recommendation System

1. Environment Setup: Custom Gym environment for product recommendations
2. Model Training: Train DQN on product embeddings

Challenges Faced

1. Performance Optimization:

   • Implemented FAISS for efficient similarity search
   • Optimized CLIP model loading and inference

2. Data Processing:

   • Handled varying image sizes and formats
   • Implemented efficient batch processing

3. Model Training:

   • Addressed exploration-exploitation tradeoff in RL
   • Implemented reward shaping for better convergence

Results

• Visual Search: High accuracy in finding visually similar products
• Recommendation Quality: Improved user engagement with personalized suggestions
• Performance: Fast response time for real-time recommendations

Team

• AI Engineers: Zeeshan Ahmad(Team Lead), M Abeer(Team Member)
• Web Developer: Muhammad Hamza Sirang(Team Member)

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• OpenAI for the CLIP model
• Facebook Research for FAISS
• Stable Baselines3 team
• The open-source community for valuable libraries and tools