Building Large Language Models from Scratch (Raschka, 2024)

(edits in progress)

Chap 1: Understanding LLMs

• Basics
• Applications
• Building & Using
• Transformer architecture
• Example datasets
• GPT architecture
• LLM buildplan

Chap 2: Text data

• Word embeddings
• Text tokens
• Tokens --> token IDs
• Special context tokens
• Byte pair encoding
• Sliding-window data sampling
• Token embeddings
• Word position encoding

Chap 3: Attention

• The long-sequence problem
• Capturing data dependencies with attention
• Self-attention
  • with trainable weights
  • weights for all input tokens
• Self-attention with trainable weights
  • Computation
  • Python class definition
• Causal attention
  • masking
  • dropout
  • Python class definition
• Multihead attention
  • stacking single-head attention layers
  • weight splits

Chap 4: GPT from scratch

• Architecture code
• Layer normalization
• Feed-forward nets with GELU (Gaussian error linear unit) activations
• Shortcut connections
• Attention & linear layers in a transformer block
• Model code
• Generating text

Chap5: Pretraining - unlabeled data

• Evaluating generative text models
  • Using GPT to generate text
  • Text generation loss
  • Training & validation set loss
• LLM training
• Reducing randomness
  • Temperature scaling
  • top-k sampling
  • Modifying the text generator
• PyTorch: model file load/save
• Loading pretrained weights from OpenAI

Chap6: Finetuning for classification

• Instruction- vs Classification-finetuning
• Dataset prep
• Dataloaders
• Initializing with pretrained weights
• Classification head
• Classification loss & accuracy
• Finetuning - supervised data
• LLM as a spam classifier

Chap7: Instruction finetuning

• TODO

Appendix A: Intro to PyTorch

• What is PyTorch?
  • 3 core components
  • deep learning, defined
  • installation
• Tensors
  • scalars, vectors, matrices, tensors
  • datatypes
  • common tensor ops
• Model as computation graphs
• Auto differentiation
• Designing multilayer neural nets
• Designing data loaders
• Typical training loops
• Model load/save
• GPUs and training performance
  • PyTorch on GPUs
  • Single-GPU training
  • Multi-GPU training
    • Selecting available GPUs
• Resources
• Exercise answers

Appendix B: References

• Chapter 1

  • Bloomberg GPT
  • Medical Q&A with LLMs
  • Attention is all you need
  • BERT (original encoder-style transformer)
  • decoder-style GPT3 model
  • original vision transformer
  • RWKV
  • Hyena hierarchy
  • Mamba
  • Llama2
  • "The Pile" text dataset
  • GPT-3 finetuning

• Chapter 2

  • Machine Learning Q and AI (2023) by Sebastian Raschka
  • byte pair encoding
  • OpenAI byte pair encoding tokenizer for GPT-2
  • OpenAI byte pair encoding, illustrated
  • BPE tokenizer from scratch (Karpathy)
  • SentencePiece Tokenizer and Detokenizer
  • Fast WordPiece Tokenization

• Chapter 3

  • Bahdanau attention
  • self-attention as scaled dot-product attention
  • FlashAttention
  • FlashAttention-2
  • PyTorch self-attention and causal attention
  • PyTorch MultiHeadAttention
  • Dropout
  • Simplifying Transformer Blocks

• Chapter 4

  • Layer Normalization
  • Pre-LayerNorm
  • ResiDual
  • Root Mean Square Layer Normalization
  • GELU (Gaussian Error Linear Unit) activation
  • GPT-2—124M, 355M, 774M, and 1.5B parameters
  • GPT-3 (OpenAI)
  • GPT-3 Technical Overview (Lambda Labs)
  • NanoGPT
  • Feedforward computation

• Chapter 5

  • loss functions & log transformations
  • Pythia
  • OLMo
  • Project Gutenberg for LLM training
  • Simple and Scalable Pretraining Strategies
  • BloombergGPT
  • GaLore optimizer
  • GaLore code repo
  • Dolma: an Open Corpus of Three Trillion Tokens
  • The Pile
  • RefinedWeb Dataset for Falcon LLM
  • RedPajama
  • FineWeb dataset from CommonCrawl
  • top-k sampling
  • Beam search (not cover in chapter 5)

• Chapter 6

  • Finetuning Transformers
  • Finetuning LLMs
  • More spam classification experiments
  • Binary classification using a single output node
  • Imbalanced-learn user guide
  • spam email classification dataset
  • BERT: Pre-training of Deep Bidirectional Transformers
  • RoBERTa
  • IMDB Movie Reviews sentiment
  • causal mask removal
  • LLM2Vec

Appendix C: Exercise Solutions

(see jupyter notebook)

Appendix D: Training loop bells & whistles

• Learning rate warmup
• Cosine decay
• Gradient clipping
• Modified training function

Appendix E: parameter-efficient finetuning with LoRA

• Intro
• Dataset prep
• Model init
• LoRA
  • listing e.5 implementation
  • image e.3
  • listing e.6 LinearWithLora layer to replace linear layers
  • image e.4 architecture
  • listing e.7 - finetuning with LoRA layers