How to finetune LLMs on custom data domains using CPT | Chapter 1, SLM Training

In this video, we are finetuning a local language model with Continued Pretraning. The goal of CPT is to turn a base language model into a domain expert in a specific topic. For this example, I am picking the SmolLM-135M parameter model. Our target domain is new Arxiv papers on deep learning and AI. To achieve this, we will be using Unsloth and Huggingface! Github repo: https://github.com/avbiswas/finetuning_recipes The repo is still under construction. It will evolve as the course expands and moves into SFT, RL, etc. Follow me on Twitter: https://x.com/neural_avb To join our Patreon, visit: https://www.patreon.com/NeuralBreakdownwithAVB Members get access to everything behind-the-scenes that goes into producing my videos - including code. Plus, it supports the channel in a big way and helps to pay my bills. Check Paper Breakdown here: paperbreakdown.com Paper Breakdown code: CPT50 Get 50% discount on Student tier. Unsloth datasets guide: https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/datasets-guide Unsloth notebooks: https://unsloth.ai/docs/get-started/unsloth-notebooks SmolLM-135M: https://huggingface.co/HuggingFaceTB/SmolLM-135M Course Video 1/this one (CPT): https://youtu.be/B8Ur62D3J3U Course Video 2 (SFT): https://youtu.be/gvZIUEL6Ruc Course Video 3 (DPO): https://youtu.be/qQwg2nFtZ1Y Low-level "from scratch" Finetuning tutorial: https://youtu.be/bZcKYiwtw1I