Five Months with Dive into Deep Learning

I read a few books every year.

I do not set reading goals. When I come across a good book, I usually read it a few times, quietly enjoy it, declare that I have learned a lot, and leave it at that. But I have just finished the first book that made me want to write something afterward: the second, PyTorch edition of Dive into Deep Learning. It took me five months. The online version is at https://zh.d2l.ai.

The image below shows its outline. I did not understand these topics before; I was simply interested. Of course, even after studying them, I still do not really understand them. I first bought the hardcover, then found that the online edition was all I needed. It even includes an environment for running the code. I think of the physical copy as my contribution to the tuition bill.

What moved me most was one of the authors, Mu Li, who also teaches the accompanying video course. He was already successful in academia and industry. While teaching it, he was a senior principal scientist at AWS; he has since left to start his own business. He must have had a demanding job, and he also had two children to look after. He made all 1,055 slides himself. He livestreamed and recorded the lessons in his garage, either after his children went to sleep or while someone else looked after them: 34 lessons, totaling 51 hours. He answered nearly a thousand questions live. In a few recordings, I could hear his children crying in the background, yet he kept teaching with a smile. Those moments brought a lump to my throat. I have two children too, and I know how hard it is to carve out that kind of time.

More than a thousand people attended each livestream on average, and viewers watched the recordings for over an hour on average—something that made him very happy. I watched the recordings. At first, I found it hard to believe so many people were studying something so unfashionable. As I kept watching, I understood why. Sharing knowledge out of sheer dedication is a wonderful thing. Thank you, Mu Li.

Mu Li, a principal scientist at Amazon and one of the creators of the AI framework Apache MXNet, earned his undergraduate degree at Shanghai Jiao Tong University and his PhD at Carnegie Mellon University. He has held research assistant, assistant professor, and adjunct teaching positions at the Hong Kong University of Science and Technology, UC Berkeley, and Stanford. He joined Amazon as a principal scientist in 2016, leading a team at AWS working full-time on MXNet. He is also passionate about making AI accessible, coauthoring Dive into Deep Learning and teaching livestreamed courses on Bilibili.

Mu Li's biography, from an encyclopedia entry

The book is highly technical. Even its outline would probably put off anyone who was not interested. What I want to record and share is its greatest strength: hands-on practice. Without false modesty, my interest in computing and my university major—information security—gave me only the barest knowledge of programming. Really, just a little, and very superficial. Even so, I managed to work through the theory, however imperfectly, set up a deep learning environment with CUDA and PyTorch, run every piece of code in the book, and rent GPUs for some of the larger algorithms and models. Short-term GPU rental is not that expensive these days. Across several months of very inefficient experimentation, I spent only a few hundred yuan. I ran into plenty of walls, and each one taught me something.

I am also grateful to be learning at this moment in time. Without GPT, someone at my level could never have studied this material effectively on their own. I do not mean becoming an expert; I mean being able to keep going at all. To help myself study, I created a dedicated custom GPT. It knew how limited my background was and patiently explained concepts with example after example until I could work through the problem. Here is the prompt for my GPT tutor.

You are an expert in deep learning with extensive theoretical and engineering experience. Your task is to guide a beginner using PyTorch, with Dive into Deep Learning.pdf in the knowledge base as the textbook. Answer the questions they encounter while studying and help them learn more quickly.

Keep the following points in mind when answering:

1. **Be patient and thorough**: Answer each question patiently and in detail.

2. **Use accessible analogies**: Explain complex concepts with familiar examples from everyday life, without leaving out the underlying theory.

3. **Be rigorous and authoritative**: Make sure every answer is rigorous and authoritative.

**Note**: Do not include code unless an implementation is specifically requested. Answer in Chinese by default.

The prompt for the custom GPT that guided me throughout my studies

Because I could not make sense of Transformers, I also sat down to read a research paper properly for the first time: Attention Is All You Need. I thought the title was a bit clickbait-y, and I still did not really understand it after reading it, but that will not stop me from showing off here. In truth, a classic paper like this deserves to have every sentence understood. It amazes me that, back in 2017, these researchers could already see so much of where the technology and its applications were heading. Of course, looking back at Transformers now, it is easy to say how brilliant they were.

Even after reading the book twice, I have barely scratched the surface. Given my starting point, perhaps five readings would get me through the door. That would be something. If I do read it five times, each pass will be harder than the last. I will try when I have time—but no boasting before I have actually done it. In middle age, uninterrupted time is hard to come by. For those five months, I hardly used my phone on weekends. I am also grateful to my family for letting me do a little less childcare so I could just about finish.

Reading is like running. If you keep at it with attention, you can enter a particular state of flow. For me, that state is happy and relaxed—and within reach.

Keep trying, keep going. I wanted to record this moment, and perhaps offer a little encouragement along the way.