Awesome Deep Learning Resources

This is a rough list of my favorite deep learning resources. It has been useful to me for learning how to do deep learning, I use it for revisiting topics or for reference. I (Guillaume Chevalier) have built this list and got through all of the content listed here, carefully.

Contents

• Trends
• Online classes
• Books
• Posts and Articles
• Practical resources
  • Librairies and Implementations
  • Some Datasets
• Other Math Theory
  • Gradient Descent Algorithms and optimization
  • Complex Numbers & Digital Signal Processing
• Papers
  • Recurrent Neural Networks
  • Convolutional Neural Networks
  • Attention Mechanisms
  • Other
• YouTube and Videos
• Misc. Hubs and Links
• License

Trends

Here are the all-time Google Trends, from 2004 up to now, September 2017:

You might also want to look at Andrej Karpathy's new post about trends in Machine Learning research.

I believe that Deep learning is the key to make computers think more like humans, and has a lot of potential. Some hard automation tasks can be solved easily with that while this was impossible to achieve earlier with classical algorithms.

Moore's Law about exponential progress rates in computer science hardware is now more affecting GPUs than CPUs because of physical limits on how tiny an atomic transistor can be. We are shifting toward parallel architectures [read more]. Deep learning exploits parallel architectures as such under the hood by using GPUs. On top of that, deep learning algorithms may use Quantum Computing and apply to machine-brain interfaces in the future.

I find that the key of intelligence and cognition is a very interesting subject to explore and is not yet well understood. Those technologies are promising.

Online Classes

• DL&RNN Course - I created this richely dense course on Deep Learning and Recurrent Neural Networks.
• Machine Learning by Andrew Ng on Coursera - Renown entry-level online class with certificate. Taught by: Andrew Ng, Associate Professor, Stanford University; Chief Scientist, Baidu; Chairman and Co-founder, Coursera.
• Deep Learning Specialization by Andrew Ng on Coursera - New series of 5 Deep Learning courses by Andrew Ng, now with Python rather than Matlab/Octave, and which leads to a specialization certificate.
• Deep Learning by Google - Good intermediate to advanced-level course covering high-level deep learning concepts, I found it helps to get creative once the basics are acquired.
• Machine Learning for Trading by Georgia Tech - Interesting class for acquiring basic knowledge of machine learning applied to trading and some AI and finance concepts. I especially liked the section on Q-Learning.
• Neural networks class by Hugo Larochelle, Université de Sherbrooke - Interesting class about neural networks available online for free by Hugo Larochelle, yet I have watched a few of those videos.
• GLO-4030/7030 Apprentissage par réseaux de neurones profonds - This is a class given by Philippe Giguère, Professor at University Laval. I especially found awesome its rare visualization of the multi-head attention mechanism, which can be contemplated at the slide 28 of week 13's class.
• Deep Learning & Recurrent Neural Networks (DL&RNN) - The most richly dense, accelerated course on the topic of Deep Learning & Recurrent Neural Networks (scroll at the end).

Books

• Clean Code - Get back to the basics you fool! Learn how to do Clean Code for your career. This is by far the best book I've read even if this list is related to Deep Learning.
• Clean Coder - Learn how to be professional as a coder and how to interact with your manager. This is important for any coding career.
• How to Create a Mind - The audio version is nice to listen to while commuting. This book is motivating about reverse-engineering the mind and thinking on how to code AI.
• Neural Networks and Deep Learning - This book covers many of the core concepts behind neural networks and deep learning.
• Deep Learning - An MIT Press book - Yet halfway through the book, it contains satisfying math content on how to think about actual deep learning.
• Some other books I have read - Some books listed here are less related to deep learning but are still somehow relevant to this list.

Posts and Articles

• Predictions made by Ray Kurzweil - List of mid to long term futuristic predictions made by Ray Kurzweil.
• The Unreasonable Effectiveness of Recurrent Neural Networks - MUST READ post by Andrej Karpathy - this is what motivated me to learn RNNs, it demonstrates what it can achieve in the most basic form of NLP.
• Neural Networks, Manifolds, and Topology - Fresh look on how neurons map information.
• Understanding LSTM Networks - Explains the LSTM cells' inner workings, plus, it has interesting links in conclusion.
• Attention and Augmented Recurrent Neural Networks - Interesting for visual animations, it is a nice intro to attention mechanisms as an example.
• Recommending music on Spotify with deep learning - Awesome for doing clustering on audio - post by an intern at Spotify.
• Announcing SyntaxNet: The World’s Most Accurate Parser Goes Open Source - Parsey McParseface's birth, a neural syntax tree parser.
• Improving Inception and Image Classification in TensorFlow - Very interesting CNN architecture (e.g.: the inception-style convolutional layers is promising and efficient in terms of reducing the number of parameters).
• WaveNet: A Generative Model for Raw Audio - Realistic talking machines: perfect voice generation.
• François Chollet's Twitter - Author of Keras - has interesting Twitter posts and innovative ideas.
• Neuralink and the Brain’s Magical Future - Thought provoking article about the future of the brain and brain-computer interfaces.
• Migrating to Git LFS for Developing Deep Learning Applications with Large Files - Easily manage huge files in your private Git projects.

Practical Resources

Librairies and Implementations

• [Neuraxle, a framwework for machine learning