Advanced Deep Learning Systems (ADLS)

Course code: ELEC70109/EE9-AML3-10/EE9-AO25

Important dates

• 12th Feb, Practical 1 (Lab 1 and 2) and Practical 2 (Lab 3 and 4) submission (New date with one-week extension)
• 9th Feb, Team project sign-up ends
• TBC, but around 12th Feb, Midterm lab oral for both practicals
• TBC, but around 18th Mar, Show me your code session
• 28th Mar, Report submission and pull request finalization

Full dates

Week Number	Date	Schedule
2	15th Jan	Lecture 1, 2, 3 and 4, Team Signup starts
3	22nd Jan	Practical 1 lab
4	29th Jan	Practical 2 lab
5	5th Feb	Lecture 5, 6 and 7
6	12th Feb	Lecture 8, 9 and 10
7	19th Feb	Lecture 11, 12 and Team Project Meetings
8	26th Feb	Team Project Lab 1
9	4th Mar	Team Project Lab 2
10	11th Mar	Team Project Lab 3
11	18th Mar	Show me your code
End of term	28th Mar	Final Report and Pull Request Finalization

Lecture 1: An Introduction to Modern Deep Learning Systems and Frameworks

• Slides
• Git tutorial
• Dark Silicon and the End of Multicore Scaling
• Managing Wire Delay in Large Chip-Multiprocessor Caches

Lecture 2: MASE: Abstractions, Optimizations and Implementations

• Slides
• Troch FX
• MASE

Lecture 3: An Introduction to Practical 1

• Slides

Lecture 4: An Introduction to Practical 2

• Slides

Lecture 5: Understanding the Workloads: Part 1

• Slides
• Convolution and DeConvolution animation
• Convolution padding and striding animation
• Deep Residual Learning for Image Recognition
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Deconvolutional Networks
• The amazing power of word vectors
• Attention Is All You Need
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Lecture 6: Understanding the Workloads: Part 2

• Slides
• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Learning Transferable Visual Models From Natural Language Supervision
• Segment Anything
• WaveNet: A Generative Model for Raw Audio
• Robust Speech Recognition via Large-Scale Weak Supervision

Lecture 7: Architectural Optimizations

• Slides
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Searching for MobileNetV3
• MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
• Longformer: The Long-Document Transformer

Lecture 8: Automated Machine Learning – An Introduction to Network Architecture Search

• Slides
• Learning Transferable Architectures for Scalable Image Recognition
• Exploring Randomly Wired Neural Networks for Image Recognition
• DARTS: Differentiable Architecture Search
• Probabilistic Dual Network Architecture Search on Graphs
• Once-for-All: Train One Network and Specialize it for Efficient Deployment
• Neural Architecture Search without Training
• Speedy Performance Estimation for Neural Architecture Search

Lecture 9: Network Compression – Part 1

• Slides
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Progressive Skeletonization: Trimming more fat from a network at initialization

Lecture 10: Network Compression – Part 2

• Slides
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• TinyBERT: Distilling BERT for Natural Language Understanding
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Distilling the Knowledge in a Neural Network
• Fast Algorithms for Convolutional Neural Networks
• EIE: Efficient Inference Engine on Compressed Deep Neural Network

Lecture 11: Computation Graph and Operator-level Optimization

• Slides
• TASO: The Tensor Algebra SuperOptimizer for Deep Learning
• Tensorflow Grappler
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• MLIR

Lecture 12: Distributed Training and Inference –>

• Slides
• Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Horovod: fast and easy distributed deep learning in TensorFlow

Labs

Labs are hosted on MASE Wiki

Assessment Information

The assessment contains two practicals (15% each), and a team project with two people (70%). If you used Colab Pro, we are wiling to reimburse that cost, please keep a copy of your receipt.

Practical 1 Submission (15%)

• Deadline (5th Feb, submission on BlackBoard): A Markdown file: with all answers (plots, tables …) of the questions and optional questions in Lab 1 and Lab2. The file should be named as practical1_your_college_short_code.md (eg. practical1_yz10513.md).
• Corresponding code in your forked repository (will be checked in the lab oral, no submission needed). But you can include code snippets in the submitted markdown file.
• Deadline (TBC, but around 5th Feb) Lab Oral.

Practical 2 Submission (15%)

• Deadline (5th Feb, submission on BlackBoard): A Markdown file: with all answers (plots, tables …) of the questions and optional questions in Lab 3 and Lab4. The file should be named as practical2_your_college_short_code.md (eg. practical2_yz10513.md).
• Corresponding code in your forked repository (will be checked in the lab oral, no submission needed). But you can include code snippets in the submitted markdown file.
• Deadline (TBC, but around 5th Feb) Lab Oral.

Team Project (70%)

• List of projects
• Project Teams
• Project Allocation
• A roadmap meeting with a GTA (not assessed).
• Deadline (18th March): A show me your code lab oral (assessed).
• Deadline (28th March): Coursework code submission as a pull request on Github (assessed). The PR should also have a clear documentation of the functionality that you have implemented and its corresponding tests. The tests should be integrated into the existing testing framework.
• Deadline (28th March, PDF attached in the PR): 8-page Report (assessed). The report should mainly focus on an evaluation of your implementation. The description of functionality and testing has being described in the PR already, so this 8-page report should include the experiments you have designed to explore certain trade-offs and properties of your optimization.