Machine Learning Algorithms: KNN, SVM, GBT, and XGBoost

This repository contains projects demonstrating the implementation and explanation of key machine learning algorithms: K-Nearest Neighbors (KNN), Support Vector Machines (SVM), Gradient Boosted Trees (GBT), and Extreme Gradient Boosting (XGBoost). These projects showcase my understanding and skills in applying these algorithms to various datasets using Python.

This was done for the AI course in 2024 at the University of Tehran under the instruction of Professor Khoshnevisan.

Overview

Part I: K-Nearest Neighbors (KNN) Algorithm

• Explanation and implementation of KNN.
• Numerical example demonstrating how KNN works.
• Discussion on determining the number of nearest neighbors (k) for predictions.

Part II: Support Vector Machines (SVM) Algorithm

• Explanation of SVM fundamentals, including margin maximization and kernel usage.
• Implementation of SVM with Radial Basis Function (RBF) kernel.
• Comparison of hard margin and soft margin classification.

Part III: Gradient Boosted Trees (GBT) Algorithm

• Explanation of the gradient boosting technique.
• Implementation of GBT, detailing the sequential training process and residual correction.
• Discussion on cross-validation and iteration determination.

Part IV: Extreme Gradient Boosting (XGBoost) Algorithm

• Overview of XGBoost and its improvements over traditional gradient boosting.
• Explanation of approximate and histogram-based splitting methods.
• Implementation of XGBoost, focusing on regularization, handling sparse data, and out-of-core computing.

Key Takeaways

• KNN: Understand how to implement and use KNN for classification and regression.
• SVM: Learn about margin maximization, kernel tricks, and how to apply SVM to nonlinear datasets.
• GBT: Gain insights into the gradient boosting technique and its application in ensemble learning.
• XGBoost: Explore the advanced features of XGBoost for efficient and accurate model training.

Resources

To access the used resources such as images and datasets, you can visit this Google Drive.

Contact

Feel free to reach out to me via LinkedIn or Email for any questions or collaborations.

License

This project is licensed under the MIT License - see the LICENSE file for details.