Introduction

There is awesome books, lecture and surveys, Deep Learning Bible, you can read this book while reading following papers.

You can read or take these courses while reading following papers.

Books

Deep Learning

[1] Bengio, Yoshua, Ian J. Goodfellow, and Aaron Courville. “Deep learning.” An MIT Press book. (2015) (http://www.deeplearningbook.org/) (Deep Learning Bible, you can read this book while reading following papers.) It covers the concepts and the math behind DL algorithms perfectly. I don’t recommend starting with this book since it’s really hard. But if you want to strengthen your knowledge after completing the courses above, this book is prefect.

[2] Hands-on machine learning with Scikit-learn Keras and TensorFlow by Aurelion Geron published by O’Reilley
This book is an awesome resource for learning ML and DL and also learning to code and implement the algorithms. I’d recommend starting with this if you’re more comfortable with books than courses.

[3] Dive into Deep Learning
This is an awesome reference for both getting into the math and the code for Deep Learning. It contains code examples and implementations in all popular DL frameworks (PyTorch, Tensorflow, and MXNET)
It’s available online for free and constantly updated and involves all the newest material on Deep Learning.
If you’ve got the time, I definitely suggest reading this. I’m actually starting to read it for upgrading my coding knowledge. [4] Neural Networks and Deep Learning(Michael Nielsen) - Neural networks, a beautiful biologically-inspired programming paradigm which enables a computer to learn from observational data - Deep learning, a powerful set of techniques for learning in neural networks

Mathematics

[1] [Mathematics for Machine Learning] (https://mml-book.github.io/) Fast and efficient way

[2] MIT OCW Linear Algebra 18.06 YouTube Playlist There’s also this legendary course on Linear Algebra, taught by Prof. Gilbert Strang at MIT, and it’s publicly accessible. Well, I’d really recommend watching this course if you’re really into math and want to learn a whole lot more about linear algebra, and you’ve got the time too. It’s definitely more than enough for starting ML, but if you feel like learning more, go for it: MIT OCW Linear Algebra 18.06 YouTube Playlist

[3] Probability and Statistics for Engineers and Scientists” by Walpole, Mayers, Ye. More deep and academic way:
If you would like to dive deeper into the world of probability and statistics, I’d suggest the book

Surveys

[1] LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. “Deep learning.” Nature 521.7553 (2015): 436-444. [pdf] (Three Giants’ Survey)

Lecture

[1] Machine Learning Specialization by Andrew Ng
This is probably the most popular ML course on the internet and A LOT of people have started their path into ML using it. It’s also the most popular and I guess the highest ranked course on Coursera (4.9/5). This Specialization is made of 3 courses covering the main parts of Machine Learning and by the end of it, you’ll have a good understanding of ML Algorithms and how to implement and use them in python.

[2] More deep, academic course: Stanford CS229 Machine Learning
This is the Machine Learning course taught at Stanford University, recorded in the class and uploaded on YouTube. Well, as I said before, I’m a fan of more deep academic courses and this is THE COURSE to go with if you’re like me. It involves a lot more math and details on ML concepts and algorithms and of course is more difficult to follow, but if you think you’ll be ok with the huge math and stuff and you won’t run away halfway through the course, don’t even hesitate to start with this one. The videos are uploaded online on YouTube and the course material is accessible from the course website. Two versions are available online, one from the Autumn 2018 (Andrew Ng) semester and one from Summer 2019 (Anand Avati). The first one is taught by Andrew Ng, the same instructor as the Coursera ML course introduced above, and the latter one is taught by Anand Avati, Andrew’s Ph.D. student. Choosing between the two is more a personal preference, I myself love Andrew’s way of teaching and I’m more comfortable with it.
Although, Anand Avati’s course is newer and covers more subjects. It even involves the math required for the course in the first three lectures.

[3] Easier to follow (Probably more popular): Deep Learning Specialization offered by DeepLearning.AI taught by Andrew Ng
This is a 5-course specialization, covering almost everything you need to understand Deep Learning and its ways.

[4] More deep, academic course: Stanford CS231n: Deep Learning for Computer Vision
I actually started Deep Learning with this course, and I’ve got to say, it’s THE BEST COURSE to start with if you’re ok to get a little deeper into the field like me. It is more focused on Deep Learning applications in Computer Vision, but it also covers ALL the basic and necessary aspects of Deep Learning too. So you should not worry about it being for Computer Vision at all. As a matter of fact, I watched the whole DL Specialization mentioned above too, after finishing this course, and I already knew all the stuff taught in the Specialization (and more) from this course. It even involves some Neural Network architectures mostly used in NLP. The only part of the Coursera Specialization that teaches more than this is the 5th course (Sequence Models) which is more focused on NLP. Its only drawback is that the available lecture videos are from the 2017 class, and it doesn’t cover some new topics like transformers. But if you’re interested enough, you’ll learn that new stuff on your own. (There’s also CS231n’s new semester’s course notes available which you can keep reading from those to learn the new methods too) After CS231n, I’d recommend CS224n if you’re interested in Natural Language Processing and want to get deep in that field.

Papers

So now is the paper.

This roadmap is constructed in accordance with the following four guidelines:

From outline to detail
From old to state-of-the-art
from generic to specific areas
focus on state-of-the-art

Milestone for 100 days

Basic DL Architecture

Convolutional Neural Networks (CNNs)
1. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. “Gradient-Based Learning Applied to Document Recognition” (1998)
2. Key Concepts: Convolutional layers, pooling layers, fully connected layers, and early applications to digit recognition (MNIST dataset).
Recurrent Neural Networks (RNNs)
1. Hochreiter, S., & Schmidhuber, J. “Long Short-Term Memory” (1997)
AlexNet
1. Krizhevsky, A., Sutskever, I., & Hinton, G. E. “ImageNet Classification with Deep Convolutional Neural Networks” (2012)
2. Key Concepts: Deeper networks, ReLU activation, dropout regularization, and large-scale image classification (ImageNet dataset).
GoogLeNet(Inception)
1. Szegedy, C., et al. “Going Deeper with Convolutions” (2014)
2. Key Concepts: Inception modules, dimensionality reduction, and efficiency in computation.
VGGNet
1. Simonyan, K., & Zisserman, A. “Very Deep Convolutional Networks for Large-Scale Image Recognition” (2014)
2. Key Concepts: Simplicity in architecture with deeper layers, use of smaller 3x3 convolution filters.
BN-Inception
1. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015)
Inception-v2~ v3
1. Rethinking the Inception Architecture for Computer Vision (2016)
ResNet
1. He, K., Zhang, X., Ren, S., & Sun, J. “Deep Residual Learning for Image Recognition” (2015)
2. Key Concepts: Residual blocks, solving vanishing gradient problem, and very deep networks (e.g., ResNet-50, ResNet-101).
DenseNet
1. Densely Connected Convolutional Networks(2017)
2. Key Concepts: Dense connections, feature reuse, and efficient gradient flow.
Inception-v4
1. Inception-ResNet and the Impact of Residual Connections on Learning(2016)
GANs
1. Goodfellow, I., et al. “Generative Adversarial Nets” (2014)
Word2Vec
1. Mikolov, T., et al. “Efficient Estimation of Word Representations in Vector Space” (2013)
Seq2Seq
1. Sutskever, I., Vinyals, O., & Le, Q. V. “Sequence to Sequence Learning with Neural Networks” (2014)
Attention Mechanism
1. Bahdanau, D., Cho, K., & Bengio, Y. “Neural Machine Translation by Jointly Learning to Align and Translate” (2014)
Transformers
1. Vaswani, A., et al. “Attention is All You Need” (2017)
BERT
1. Devlin, J., et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2018)
GPT
1. Radford, A., et al. “Improving Language Understanding by Generative Pre-Training” (2018)
EfficientNet
1. Tan, M., & Le, Q. V. “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks” (2019)
MobileNet
1. Howard, A. G., et al. “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications” (2017)
2. Key Concepts: Depthwise separable convolutions, lightweight models for mobile and embedded vision applications.
DALL-E - Ramesh, A., et al. “Zero-Shot Text-to-Image Generation” (2021)

Advanced DL Architecture

Foundation for Generative Models

Transformers

Language Models

Segmentation

Image Generative Models

Diffusion Models

Diffusion Models Manipulation

Neural Radiance Fields

3D Reconstruction

Implicit Representations

3D Generative Models

3D Scene Generation

SLAM

Dynamic Reconstruction

Motion Generation

Correspondences

RAFT [ECCV 2020]
PIPs [ECCV 2022]
[SIFT [IJCV 2004]](https://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf
LoFTR [ CVPR 2021]
DKM [CVPR 2023]

🪴 Jihee's Blog

Explorer

100 days final