cuda-learning

LEARNING PATH - From Basics to Advanced CUDA Programming

This structured learning path guides you through the essential steps required to become proficient in CUDA programming, starting from foundational programming knowledge to advanced GPU computing concepts. The path emphasizes building a strong base in programming, understanding data structures, mastering C++, and diving into GPU architecture and CUDA-specific optimizations. Resources include both English and Polish materials, offering flexibility based on language preference.

1. C Programming:
   Begin with C programming if you are unfamiliar with it. A solid understanding of C is mandatory before transitioning to C++ programming.

   • The C Programming Language (ANSI C) by Brian Kernighan and Dennis Ritchie
   • Practicing C programming GPT-2 from Scratch in C - Part 1 and GPT-2 from Scratch in C - Part 2
   • 🇵🇱 Podstawy programowania. Język C
   • 🇵🇱 Zaawansowane programowanie w języku C

2. Data Structures:
   Learn essential data structures and algorithms, a prerequisite for effective problem-solving and programming.

   • 🎥C++ Data Structures & Algorithms + LEETCODE Exercises
   • Data Structures and Algorithms -> Leetcode
   • 📖 Introduction to Algorithms, fourth edition by Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest and Clifford Stein
   • 🇵🇱 📖 Algorytmy, struktury danych i techniki programowania by Paweł Wróblewski
   • 🇵🇱 📖 C++. Algorytmy i struktury danych by Adam Drozdek

3. C++ Programming:
   Master C++ programming as it serves as a foundation for CUDA development.

   • 🎥 Beginning C++ Programming - From Beginner to Beyond
   • 🎥 Back to Basics
   • 📖Modern C++ Tutorial: C++11/14/17/20 On the Fly
   • 🇵🇱 🎥 C++ od Podstaw do Eksperta

This comprehensive learning path equips you with the skills needed to progress from foundational programming to advanced CUDA development, paving the way for a career in GPU-accelerated computing.

Matmul

This section focuses on understanding the fundamentals and optimization of matrix multiplication (Matmul), a cornerstone operation in CUDA programming and high-performance computing (HPC). The provided resources cover both CPU implementations and GPU optimizations, including the use of Tensor Cores on architectures like Ampere and Ada. These materials are essential for building a strong foundation in writing optimized CUDA code.

• **Matmul theory

  • Matrix Calculus (for Machine Learning and Beyond)

• Matmul on CPU: Analysis of efficient matrix multiplication implementations on CPUs, with detailed examples of optimizations:

  • Beating OpenBLAS in FP32 Matrix Multiplication: A Full Walkthrough
  • Fast Multidimensional Matrix Multiplication on CPU from Scratch
  • Optimizing Matrix Multiplication
  • Matrix Multiplication on CPU

• CUDA Matmul Optimizations:

  • Beating cuBLAS in Single-Precision General Matrix Multiplication
  • GPU H100: Outperforming cuBLAS on H100: a Worklog
  • Ada Architecture: Implementing a fast Tensor Core matmul on the Ada Architecture
  • Ampere Architecture: How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog
  • CUDA Matrix Multiplication Optimization
  • Learning CUDA by optimizing matrix-vector multiplication (SGEMV) for cuBLAS-like performance - A worklog
  • Matrix Multiplication with CUDA — A basic introduction to the CUDA programming model
  • Inconsistency in GEMM Performance
  • Fastest GPU kernels, written from scratch.

These resources provide a comprehensive theoretical and practical foundation in matrix multiplication, enabling you to master CUDA learning and better understand algorithm optimization in GPU environments.

Softmax

• Learning CUDA by optimizing softmax: A worklog
• How to write a fast Softmax CUDA kernel?
• How to Implement an Efficient Softmax CUDA Kernel— OneFlow Performance Optimization
• Optimizing Softmax on CUDA: A Dive into High-Performance Kernels

Sort

• Performance Optimization of torch.sort on GPU

GPU programming resources

Description of the Section: GPU Programming Resources

This section provides a curated collection of resources for learning, exploring, and mastering GPU programming. It covers various aspects of GPU development, including community engagement, architectural insights, tutorials, example implementations, benchmarking, and advanced tools. These resources cater to developers at different expertise levels, offering a pathway to build and optimize high-performance GPU applications.

---

1. Communities

Engage with fellow developers and experts in the field of GPU programming:

• NVIDIA CUDA Forum
• CUDA-MODE

---

2. GPU Architectures

Understand the underlying architecture of GPUs to optimize code efficiently:

• Ampere Architecture
• Ada Architecture
• Hopper Architecture
• Grace-Hopper Architecture
• GPUs Go Brrr

---

3. Tutorials

Learn the practical aspects of GPU programming with these tutorials:

• GPU Puzzles - to solve -> Solved
• [Accurate Timing of CUDA Kernels in PyTorch](https://www.speechmatics.com/company/articles-and-news/timi