100-day roadmap for studying multimodal deep learning

Certainly! Here is a concise 100-day roadmap for studying multimodal deep learning, focusing on the names of the key papers you should read:

Week 1-2: Foundational Papers

1. Deep Visual-Semantic Alignments
2. Show and Tell
3. DeViSE
4. VQA

Week 3-4: Multimodal Representation Learning

5. ViLBERT
6. LXMERT
7. VisualBERT

Week 5-6: Unified Multimodal Models

8. CLIP
9. ALIGN

Week 7-8: Multimodal Transformers

10. M3P
11. FLAVA

Week 9-10: Advanced Applications

12. DALL-E
13. VATT
14. Perceiver IO

Week 11-12: Recent Trends (2023-2024)

15. Flamingo
16. Gato

Week 13-14: Early Multimodal Learning

17. Multimodal Learning with Deep Boltzmann Machines
18. Multimodal Learning with Multiple Modalities

Week 15-16: Multimodal Representation Learning (Cont.)

19. ImageBERT
20. VideoBERT

Week 17-18: Audio-Visual Models

21. AV-BERT
22. L3-Net

Week 19-20: Self-Supervised Multimodal Learning

23. MIL-NCE
24. MMV

Week 21-22: Graph-Based Multimodal Learning

25. MM-Graph
26. MultiGraph

Week 23-24: Multimodal Generative Models

27. StyleGAN-T
28. Multimodal Variational Autoencoders

Week 25-26: Multimodal Machine Translation

29. Multimodal Transformer Networks for End-to-End Sign Language Production
30. OpenNMT-Multi

Week 27-28: Multimodal Sentiment Analysis

31. CMU-MOSEI
32. Multimodal Transformer for Unaligned Multimodal Language Sequences

Week 29-30: Multimodal Neural Machine Translation

33. NMT with Visual Attention
34. Multimodal Neural Machine Translation with Embedding Prediction

Week 31-32: Healthcare and Biomedical Applications

35. Multimodal Data Fusion for Healthcare Applications
36. MedFuse: Multimodal Representation Learning for Medical Data

Week 33-34: Advanced Techniques

37. Cross-Modal Attention
38. Co-Attention Networks

Week 35-36: Multimodal Fusion Techniques

39. Tensor Fusion Network
40. Dynamic Multimodal Fusion with BERT

Week 37-38: Temporal Multimodal Models

41. Temporal Multimodal Learning with Attention
42. MM-TCN: Multi-Modal Temporal Convolution Network

Week 39-40: Visual Question Answering (Advanced)

43. MAC Networks
44. BAN: Bilinear Attention Networks

Week 41-42: Multimodal Dialogue Systems

45. M2M: Towards Multimodal to Multimodal Dialogue Systems
46. Multimodal Transformer for End-to-End Multimodal Dialog

Week 43-44: Multimodal Embeddings

47. Unicoder-VL
48. VisualBERT: A Simple and Performant Baseline for Vision and Language

Week 45-46: Multimodal Learning in Robotics

49. Perception as Generative Reasoning
50. Multimodal Sensor Fusion for Object Recognition in Robotic Systems

Week 47-48: Multimodal Video Understanding

51. HERO: Hierarchical Encoder for Video+Language Tasks
52. ActBERT: Learning Global-Local Video-Text Representations

Week 49-50: Evaluation and Benchmarking

53. GLUE: A Multi-Task Benchmark and Analysis Platform
54. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Week 51-52: Multimodal Learning Trends

55. UniT: Unified Transformer for Multimodal Multitask Learning
56. LXMERT: Learning Cross-Modality Encoder Representations from Transformers (Revisit)
57. FLAVA: A Foundational Language and Vision Alignment Model (Revisit)
58. Flamingo: A Visual Language Model for Few-Shot Learning (Revisit)
59. Gato: A Generalist Agent (Revisit)
60. CLIP (Revisit)
61. ALIGN (Revisit)

This roadmap will guide you through the key milestones in multimodal deep learning research over 100 days, focusing on one or two papers per day.