📋 Main Topics¶
Introduction to Fine-Tuning and RLHF - What is fine-tuning and how does it differ from pretraining? - Why is RLHF needed? - How RLHF improves language model alignment with human preferences
Reinforcement Learning and Transfer Learning - Basics of Reinforcement Learning (RL) and key concepts - Difference between Supervised Learning, RL, and Unsupervised Learning - Transfer Learning: Adapting pre-trained models to new tasks
Fine-Tuning Techniques - Full fine-tuning vs. parameter-efficient fine-tuning (LoRA, QLoRA) - Challenges of fine-tuning large models - Practical fine-tuning strategies
Low-Rank Adaptation (LoRA) - What is LoRA and its benefits? - How LoRA reduces computational cost - Implementing LoRA in practice
Reinforcement Learning from Human Feedback (RLHF) Pipeline - Pretraining → Reward modeling → RLHF training - Role of human annotators in creating preference data - Comparison of RLHF with standard RL techniques
🧠Class Activity - Labs¶
- De-toxifying LLM with RLHF
📚 Recommended Readings¶
- Illustrating Reinforcement Learning from Human Feedback (RLHF)
- RLHF 101: A Technical Dive into RLHF
- Practical Tips for Fine-Tuning LLMs Using LoRA
- Deep RL Course - Hugging Face
- What is Reinforcement Learning? (MathWorks)
🎥 Recommended Videos¶
- Reinforcement Learning from Human Feedback: From Zero to ChatGPT Watch on YouTube