# Is Training AI Really That Hard? I Tried a 26M-Parameter Model Willdom Lee · 2025-01-29 [HTML](https://willdomlee.com/en/articles/training-a-tiny-model) · en > Think of a large model as a complex machine with countless knobs. Training uses large amounts of carefully prepared data to adjust those knobs, gradually finding a better configuration. The values of the knobs are the model's parameters. Training aims to improve them so the model can understand and generate text and answer questions more accurately. > > That description may not be entirely rigorous, but at least it is easy to understand. ## 1. Why try this experiment? - The holiday gave me some uninterrupted time, and DeepSeek was everywhere in the news over Chinese New Year. Instead of standing on the bank wishing for fish, I decided to make a net—and join in the experimentation myself. - In 2024, I had worked my way, somewhat imperfectly, through [Dive into Deep Learning](https://willdomlee.com/en/articles/learning-deep-learning.html). At the time, I had only run the individual code examples in the book, without taking the next step. Then I came across MiniMind, an open-source project that connected closely with what the book taught, and decided to try it. ![](https://willdomlee.com/assets/a/training-a-tiny-model/01.webp) ## 2. Thoughts after the experiment - Hands-on work is always the most efficient way to learn. Once you have some basics, you can confidently jump ahead. Gradually closing the gaps in your knowledge is how you make progress. Learning does not have to follow a prescribed sequence; you can even aim straight for the advanced material. - Use AI deeply and consistently in everyday life. Learn to handle the major tools well, develop an efficient approach of your own, and make them a natural part of your creative work and workflows. I see more people around me using AI, which is encouraging. If you keep forgetting to use it, subscribe to the most expensive plan—it would be a shame to waste the money. If you spent $200 a month on ChatGPT Pro, I suspect you would make yourself use it for a while. I am joking, of course. In 2024, I used ChatGPT's voice mode and Projects to practice English on my commute. It opened up a completely new learning experience for me, and I enjoyed it. I will share more when I have time. - High-quality private knowledge bases will become increasingly important for individuals and organizations alike. Building them is hard, tedious work—a classic case of doing what is difficult but right, and sticking with it. AI-ready organizations will have a crucial advantage. To me, their defining features include high-quality private knowledge, mature core processes, and continuous data governance. When building an AI-ready organization, the AI technology itself may be the least of your worries. - Interest matters. It can make you put your phone down and do math instead. In the photo below, I am working through basic derivatives and differentials with pen and paper to understand gradient calculations. Interest also makes learning enjoyable and rewarding rather than a burden. ![](https://willdomlee.com/assets/a/training-a-tiny-model/02.webp) ## 3. Parts of the process worth recording - MiniMind has only 26M parameters—million, not billion. That is 26 million, about 1/7,000 the size of GPT-3, whose parameters number in the hundreds of billions. I wrote no original code for this experiment. Everything was based on @jingyaogong's open-source MiniMind project, with parameter adjustments and a few small code changes. I relied heavily on AI for guidance and troubleshooting. ![](https://willdomlee.com/assets/a/training-a-tiny-model/03.webp) - I again rented a server from AutoDL, which I am comfortable with—this is not an ad. Its ready-made deep learning environments saved a lot of preparation time. You still need to know how to deal with resource download problems, though, or you may get stuck on the very first command. The entire experiment cost 50 yuan in server rental. Good value. AWS is another good option. - I selected GPUs as needed. During preparation, such as downloading resources and debugging code, I ran the server without a GPU, which cost very little. Once the code worked, I rented six RTX 4090s and used them for training to save time. I did not finish every training epoch; I ran two, taking a few hours. Seeing the experiment work was enough. ![](https://willdomlee.com/assets/a/training-a-tiny-model/04.webp) Returning to the analogy at the start, the experiment went roughly like this: - Set up the environment: I will skip the details. - Download the project files: clone MiniMind from GitHub, including its model architecture, training scripts, and configuration. This gives us an adjustable intelligent machine, but it has not been trained, so its knobs—the parameters—are still set randomly. - Download and process the dataset: the data is like a textbook. A good textbook and one full of errors will produce very different learning outcomes. Following the documentation, I downloaded a dataset from Hugging Face and processed it into the format needed for training, much like organizing the teaching materials. Our textbook contained 4.3 GB of text—not enormous, but not tiny either. ![](https://willdomlee.com/assets/a/training-a-tiny-model/05.webp) I was curious about what was actually in this textbook, so I looked at the first few lines. As shown below, it was ordinary written language. ![](https://willdomlee.com/assets/a/training-a-tiny-model/06.webp) - Start pretraining: this is the foundation-building stage. We feed the prepared material to the model—for example, the first part of a sentence—and ask it to predict what follows. We keep adjusting the knobs to reduce prediction errors. Over time, it learns language structure, grammar, and even some common knowledge, rather like a child gradually moving from babble to coherent speech. Since this was an experiment, I stopped after about two hours, by which point the knobs had been adjusted many times. - Evaluate the pretrained model: I asked a few questions as a mock exam. The results were not great. It could produce language but was prone to nonsense, still only half-literate, as the screenshot shows. ![](https://willdomlee.com/assets/a/training-a-tiny-model/07.webp) - Continue with supervised fine-tuning (SFT): the pretrained model had learned language but was not yet precise enough. Fine-tuning is specialist practice on better, task-focused data so its answers better match expectations. It is like revising the key topics for an exam after learning the basics. Out of curiosity, I looked at this specialist textbook too: several gigabytes of questions and answers. Does that not look like exam revision material? See below. ![](https://willdomlee.com/assets/a/training-a-tiny-model/08.webp) - Evaluate the fine-tuned model: I did not complete all 20 epochs, only one, which took around an hour on six GPUs. Even so, the model's answers and reasoning became noticeably better, as shown below. ![](https://willdomlee.com/assets/a/training-a-tiny-model/09.webp) - That ended the experiment. I was still curious about the output parameter file, so I opened it. That led into deep learning frameworks and Transformers, which I do not understand very well yet. I will stop there for now. ![](https://willdomlee.com/assets/a/training-a-tiny-model/10.webp) I ran into plenty of problems along the way. With AI's help, repeated trips to the documentation, close comparisons of the code, and some last-minute study of key concepts, I eventually solved them all. MiniMind's author has done a generous job: the code is comprehensive and accessible, and there is an accompanying Bilibili video ([watch on Bilibili ↗](https://www.bilibili.com/video/BV1Ld4fe9EZC/)). **“I once spent a whole day thinking, but it was worth less than a brief moment of study.” My knowledge may be limited, but I will keep going. Today is the first day of the Lunar New Year. Happy New Year, friends!** --- Originally published in Chinese on WeChat, 南山独唱团. [Read the original](https://mp.weixin.qq.com/s/0sRuM9a38-QghXbdGEd_OA).