Advanced Deep Learning Systems (ADLS)

Course code: ELEC70109/EE9-AML3-10/EE9-AO25

---

Important dates

• 7th Feb 14th Feb (extended), Practical submission
• 9th Feb, Team project sign-up ends
• 10th Feb 17th Feb (extended), Midterm lab oral
• 17th Mar, Show me your code session
• 27th Mar, Report submission and pull request finalization

---

Full dates

Week Number	Date	Schedule
2	16th Jan	Lectures (1 and 2, Room 407A, 11am-1pm), Team signup starts
3	20nd Jan	Lectures (3 and 4, Room 508B, 10am-noon) and lab sessions (Room 407, 4pm-6pm)
4	27th Jan	lab session (Room 407A, 4pm-6pm)
5	3rd Feb	lab session (Room 407A, 4pm-6pm)
5	6th Feb	lab session (Room 407A, 11am-1pm)
6	10th Feb	lab session (extra) (Room 403B, 4pm-6pm)
6	13th Feb	Lectures (5 and 6, Room 407A, 11am-1pm)
7	17th Feb	Lectures (7 and 8, Room 508B, 10am-noon)
7	17th Feb	Lab oral (extended ddl, room and time tbc)
7	20th Feb	Lectures (9 and 10, Room 407A, 11am-1pm)
8	24th Feb	Lectures (11 and 12, Room 407A, 11am-1pm)
8	27th Feb	Team Project Lab (Project scoping)
9	3rd Mar	Team Project Lab
10	10th Mar	Team Project Lab
11	17th Mar	Show me your code
End of term	27th Mar	Final report and code submission

---

Lecture 1: Introduction

• Slides
• Git tutorial
• Dark Silicon and the End of Multicore Scaling
• Managing Wire Delay in Large Chip-Multiprocessor Caches

Lecture 2: MASE: Abstractions, Optimizations and Implementations

• Slides
• Troch FX
• MASE

Lecture 3: An Introduction to Labs

• Slides

Lecture 4: An Introduction to Labs (2)

• Slides

Lecture 5: Understanding the workload

• Slides
• Convolution and DeConvolution animation
• Convolution padding and striding animation
• Deep Residual Learning for Image Recognition
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Deconvolutional Networks
• The amazing power of word vectors
• Attention Is All You Need
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Lecture 6: Understanding the workload (2)

• Slides
• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Learning Transferable Visual Models From Natural Language Supervision
• Segment Anything
• WaveNet: A Generative Model for Raw Audio
• Robust Speech Recognition via Large-Scale Weak Supervision

Lecture 7: Architectural Optimizations

• Slides
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Searching for MobileNetV3
• MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
• Longformer: The Long-Document Transformer

Lecture 8: Automated Machine Learning – An Introduction to Network Architecture Search

• Slides
• Learning Transferable Architectures for Scalable Image Recognition
• Exploring Randomly Wired Neural Networks for Image Recognition
• DARTS: Differentiable Architecture Search
• Probabilistic Dual Network Architecture Search on Graphs
• Once-for-All: Train One Network and Specialize it for Efficient Deployment
• Neural Architecture Search without Training
• Speedy Performance Estimation for Neural Architecture Search

Lecture 9: Network Compression

• Slides
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Progressive Skeletonization: Trimming more fat from a network at initialization

Lecture 10: Network Compression (2)

• Slides
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• TinyBERT: Distilling BERT for Natural Language Understanding
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Distilling the Knowledge in a Neural Network
• Fast Algorithms for Convolutional Neural Networks
• EIE: Efficient Inference Engine on Compressed Deep Neural Network

Lecture 11: Computation Graph and Operator-level Optimization

• Slides
• TASO: The Tensor Algebra SuperOptimizer for Deep Learning
• Tensorflow Grappler
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• MLIR

Lecture 12: Distributed Training and Inference

• Slides
• Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Horovod: fast and easy distributed deep learning in TensorFlow

Labs

Labs are hosted on MASE Wiki

Assessment Information

The assessment contains practicals (30% in total), and a team project with two people (70%). If you used Colab Pro, we are wiling to reimburse that cost, please keep a copy of your receipt.

Practical Submission (30%)

• Deadline (7th Feb, email to a.zhao@imperial.ac.uk): A Markdown file: with all answers (plots, tables …) of the questions and optional questions in labs. The file should be named as adls_labs_your_college_short_code.md (eg. adls_labs_yz10513.md).
• Corresponding code in your forked repository (will be checked in the lab oral, no submission needed). But you can include code snippets in the submitted markdown file.
• Deadline (10th Feb) Lab Oral.

Team Project (70%)

• A team of 3-4 people
• Team making link
• ADLS Teams (Final)
• Team Project Requirements
• Past project lists
• A roadmap meeting with Aaron (Booking link).
• Deadline (17th March): A show me your code lab oral (assessed).
• Deadline (27th March): Coursework code submission as a pull request or an open repository on Github (assessed). The PR or repository should also have a clear documentation of the functionality that you have implemented and its corresponding tests. There should also be testing in place.
• Deadline (27th March, PDF attached in the PR): 6-page Report (assessed). The report should mainly focus on an evaluation of your implementation. The description of functionality and testing have being described in the PR or code repository already, so this 6-page report should include only the experiments you have designed to explore certain trade-offs and properties of your optimization.