Notebooks#
Notebook 1: Transformer-based NLP tools; and how to fine-tune a text classifier.#
This course introduces you to Transformers, a class of deep learning neural networks based on the Transformer architecture. Although this architecture was introduced relatively recently [VSP+17], history did not start in 2017. Many of the components in and ideas behind the Transformer have a history that harks back to the first wave of neural network research [MP43, MP69, Ros58], as well as the second wave [Elm90, RMtPRG86].
In some of the following notebooks we follow, broadly, the book Natural Language Processing with Transformers, by 🤗 staff members Lewis Tunstall, Leandro von Werra, and Thomas Wolf. See the full notebook collection for this book.
Notebook 2: The Transformer Anatomy Lesson: miniGPT#
This notebook accompanies lecture 2. In this notebook you implement
and train a full transformer encoder, miniGPT, in PyTorch. Running and
understanding the notebook closely helps to understand the inner
workings of a transformer, which you typically do not see when working
with the high-level transformer library from Hugging Face.
Notebook 3: The Transformer Anatomy Lesson: Visualizing attention#
This notebook also accompanies lecture 2. In this notebook we use the bertviz library to visualize internals of bert-base-uncased, a pretrained BERT model.
Notebook 4: Instruction tuning, Tools and Agents#
This notebook accompanies Lecture 3: Transformers recap, In-context learning, Post-training, Agents.
LLM output can be used to trigger external instruments, such as web search, a calculator, etc. The output of running these external instruments then can be fed back to the LLM and may further condition its text generation. We will play with agents and tools with the smolagents library.
In the second part of the notebook we explore instruction tuning: finetuning an LLM to be prepared to link instructions to the proper answer type, rather than just generating plausible text.
Notebook 5: Efficient LLM Inference#
In this notebook we exemplify two efficient inference approaches: int8 quantization and speculative decoding. This is one of two notebooks complementing Lecture 4: Efficient LLMs.
Notebook 6: Fine tuning of Transformers models using PEFT (Parameter Efficient Fine-Tuning) techniques#
This notebook, adapted from this Hugging Face blog and complementing Lecture 4: Efficient LLMs, the task of finetuning a sequence-to-sequence Transformer on dialogue summarization in the domain of customer service conversations on Twitter, forms the backdrop for a dive into parameter-efficient fine-tuning with LoRA and QLoRA.
Notebook 7: Reasoning with LLMs#
Accompanying Lecture 7: Reasoning in LLMs, this Notebook walks you through inference using Qwen-2.5-1.5B and DeepSeek-R1-Distill-Qwen-1.5B. You are introduced in comparing direct versus chain-of-thought reasoning, with math problem solving as the test domain.
Notebook 8: Benchmarking LLMs#
This notebook accompanies Lecture 5: Benchmarking LLMs. In this Notebook we are going to benchmark pre-trained LLMs on standard benchmark sets on Hugging Face Datasets. We are going to work with the GLUE (General Language Understanding Evaluation) NLP benchmark set. The notebook’s goal is to finetune an LLM on one of the GLUE Benchmark tasks and achieve a non-trivial performance compared to scores on the GLUE Benchmark Leaderboard.
Notebook 9: Text generation with GPT#
In Lecture 1: Introduction and a brief history of neural networks we talk about different types of transformer-based models. But before we dive in, we wanted to take some time to look at one particular type of models in more detail: models trained for text generation. These are the models that became most prominent recently, because they lie at the core of all the amazing and popular large language models that have been taking the world by storm in the last couple of years (think ChatGPT, Claude, LLaMa, DeepSeek etc.).
In this notebook we explore text generation with GPT-style Transformers. It loosely follows Chapter 5 of the Natural Language Processing With Transformers book, and its associated notebook. This notebook introduces you to some concepts related to generating text, and features two exercises:
The first exercise focuses on hyperparameters for text generation (in particular, temperature).
The second exercise is about testing the capacities of GPT-2, the predecessor of GPT-3 and 4, in your native language.
We are using the GPT2 model on Hugging Face. If your runtime engine allows it, you may also try the larger models GPT-Medium, GPT-Large, or GPT-XL. Performance quality tends to go up with larger models.
Notebook 10: Build your own Diffusion Language Model#
In Guest Lectures Ruurd Kuiper talked about diffusion-based LLMs. In this notebook you get to build your own diffusion-based LLM, starting with the finetuning of a regular Transformer LLM, Llama 3.2 1B.